Log classification method, system, electronic device and storage medium

Abstract

This invention provides a log classification method, system, electronic device, and storage medium. The method involves extracting target logs from raw logs; constructing word vectors for the target logs; inputting the word vectors into a pre-trained reinforcement learning agent to determine the weight of each target word in the target log based on the word vectors; wherein the reinforcement learning agent is trained using historical logs on a bidirectional long short-term memory neural network; determining the target weight of the target log based on the weight of each target word, and using a pre-trained log classification model to determine the log category corresponding to the target log based on the target weight; wherein the log classification model is trained using historical logs pre-labeled with log categories on a neural network.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to a log classification method, system, electronic device, and storage medium. Background Technology

[0002] Computers generate numerous log files during operation, containing crucial information. Properly utilizing this log information can help system administrators troubleshoot computer problems. However, log files often contain a wide variety of information. To quickly and accurately diagnose computer faults, log files can be pre-categorized, and then the corresponding categories of log information can be analyzed to rapidly identify and resolve the problem.

[0003] In existing technologies, log rules are manually set based on the patterns of various abnormal events. Log information in the log file is then matched against these rules to determine the category of the log information. However, this method requires manual filtering, matching logs of the same category, mining rules for the same category of logs, and performing matching. This process is not only time-consuming and inefficient, but also results in low accuracy of the classification results. Summary of the Invention

[0004] In view of this, the present invention provides a log classification method, system, electronic device and storage medium, with the aim of short time consumption, high efficiency and high accuracy of classification results.

[0005] The first aspect of this invention discloses a log classification method, the method comprising:

[0006] Extract the target log from the raw log;

[0007] Word vectors are constructed from the target log to obtain the word vectors corresponding to the target log;

[0008] The word vectors are input into a pre-trained reinforcement learning agent, so that the reinforcement learning agent determines the weight corresponding to each target word in the target log based on the word vectors; wherein, the reinforcement learning agent is trained using a bidirectional long short-term memory neural network to be trained using historical logs;

[0009] Based on the weight of each target word in the target log, the target weight of the target log is determined, and a pre-trained log classification model is used to determine the log category corresponding to the target log based on the target weight of the target log; wherein, the log classification model is obtained by training a neural network to be trained using historical logs with pre-labeled log categories.

[0010] Optionally, extracting the target log from the original log includes:

[0011] Based on the log extraction tool and the pre-set log template, structured target logs are extracted from the raw logs.

[0012] Optionally, the step of constructing word vectors for the target log to obtain the word vectors corresponding to the target log includes:

[0013] Determine whether the text length of the target log is greater than the target text length;

[0014] If the text length of the target log is greater than the target text length, word vectors are constructed for the target log using word2vec to obtain the word vectors corresponding to the target log;

[0015] If the text length of the target log is not greater than the target text length, word vectors are constructed for the target log according to the target word vector construction method to obtain the word vectors corresponding to the target log.

[0016] Optionally, the step of training the bidirectional long short-term memory neural network to be trained using historical logs to obtain a reinforcement learning agent includes:

[0017] Obtain historical logs and extract the target historical log from the historical logs;

[0018] Word vectors are constructed from the target historical logs to obtain the word vectors of the target historical logs; wherein, the word vectors of the target historical logs include the vectors of each word arranged in sequence;

[0019] The word vectors of the target historical log are input into the bidirectional long short-term memory neural network to be trained, so that the bidirectional long short-term memory neural network to be trained determines the weight of each word in the target historical log based on the word vectors of the target historical log, and the anomaly detection evaluation function constructed based on the weight of each word adjusts the parameters of the bidirectional long short-term memory neural network to be trained in reverse until the bidirectional long short-term memory neural network to be trained converges, thus obtaining the reinforcement learning agent;

[0020] The weight of the first word in the target historical log is determined based on the word vector of the first word, and the weight of the nth word in the target historical log is determined based on the weight of the (n-1)th word and the word vector of the nth word, where n is greater than 1.

[0021] Optionally, the word vectors of the target log include vectors corresponding to each target word arranged in sequence. The step of inputting the word vectors of the target log into a pre-trained reinforcement learning agent, so that the reinforcement learning agent determines the weight corresponding to each target word in the target log based on the word vectors of the target log, includes:

[0022] The word vectors of the target log are input into a pre-trained reinforcement learning agent, so that the reinforcement learning agent determines the weight of the s-th target word based on the vector of the s-th target word and the weight of the (s-1)-th target word in the target log, until the weight of each target word in the target log is determined.

[0023] Where s is greater than or equal to 1, and when s equals 1, the weight of the (s-1)th target word is zero.

[0024] A second aspect of the present invention discloses a log classification system, the system comprising:

[0025] The first extraction unit is used to extract the target log from the original log;

[0026] The first word vector construction unit is used to construct word vectors for the target log to obtain the word vectors corresponding to the target log.

[0027] A weight determination unit is used to input the word vectors into a pre-trained reinforcement learning agent, so that the reinforcement learning agent determines the weight corresponding to each target word in the target log based on the word vectors; wherein, the reinforcement learning agent is obtained by training a bidirectional long short-term memory neural network to be trained using historical logs;

[0028] The log category determination unit is used to determine the target weight of the target log based on the weight of each target word in the target log, and to determine the log category corresponding to the target log based on the target weight using a pre-trained log classification model; wherein, the log classification model is obtained by training a neural network to be trained using historical logs with pre-labeled log categories.

[0029] Optionally, the first extraction unit includes:

[0030] The first extraction subunit is used to extract structured target logs from the raw logs based on the log extraction tool and the pre-set log template.

[0031] Optionally, the first word vector construction unit includes:

[0032] The judgment unit is used to determine whether the text length of the target log is greater than the target text length;

[0033] The first word vector construction subunit is used to construct word vectors for the target log according to word2vec if the text length of the target log is greater than the target text length, thereby obtaining the word vectors corresponding to the target log.

[0034] The second word vector construction subunit is used to construct word vectors for the target log according to the target word vector construction method if the text length of the target log is not greater than the target text length, thereby obtaining the word vectors corresponding to the target log.

[0035] A third aspect of the present invention discloses an electronic device, comprising: a processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is configured to call and execute a program stored in the memory; and the memory is configured to store the program, the program being configured to implement the log classification method disclosed in the first aspect of the present invention.

[0036] The fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for performing the log classification method disclosed in the first aspect of the present invention.

[0037] This invention provides a log classification method, system, electronic device, and storage medium. The method extracts target logs from raw logs; constructs word vectors for the target logs, obtaining word vectors corresponding to each word in the target log; the word vectors are then input into a reinforcement learning agent pre-trained on a bidirectional long short-term memory neural network using historical logs. The reinforcement learning agent determines the weight of each word in the target log based on the word vectors, and determines the target weight of the target log based on the weight of each word. Finally, the target weight of the target log is input into a log classification model pre-trained on a corresponding neural network using historical logs labeled with log labels. The log classification model then determines the type of the target log based on the target weight. The entire log classification process requires no manual intervention, thus solving the problems of existing technologies that require manual screening, matching of logs of the same type, and rule mining and matching of logs of the same type, which is time-consuming, inefficient, and results in low accuracy. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a log classification method provided in an embodiment of the present invention;

[0040] Figure 2 An example diagram of a raw log provided for an embodiment of the present invention;

[0041] Figure 3 An example diagram of a target log provided in an embodiment of the present invention;

[0042] Figure 4 This is a training example diagram of a reinforcement learning agent provided in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the structure of a log classification system provided in an embodiment of the present invention;

[0044] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed Implementation

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0047] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0048] It should be noted that the terms "a" and "a plurality of" used in this invention disclosure are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0049] As can be seen from the background technology above, existing log categorization methods involve manually setting corresponding log rules based on the patterns of various abnormal events, and then matching the log information in the log file with these rules to determine the category of the log information. However, this method requires manual screening, matching logs of the same category, mining rules for the same category of logs, and performing matching work. This is not only time-consuming and inefficient, but also results in low accuracy of the classification results.

[0050] With the development of artificial intelligence, engineers have begun to introduce machine learning and deep learning for log classification, such as time series prediction and text classification. Since logs are also a type of text, engineers have also incorporated natural language processing into log classification. These methods of using artificial intelligence for log classification are known as Intelligent Operations and Maintenance (AIOps). AIOps is an operations and maintenance approach that automatically learns and summarizes rules from massive amounts of operational data using machine learning algorithms to make decisions. It can quickly analyze and process large amounts of data, derive effective operational and maintenance decisions, execute automated scripts, and achieve overall system operation and maintenance, ensuring system stability.

[0051] However, while machine learning can overcome some of the shortcomings of human rules for log classification, it has limited ability to extract features from data and cannot handle a wide variety of logs, especially those that are not particularly distinctive or have few features, resulting in low accuracy in log classification.

[0052] While deep learning can address the limitations of machine learning in feature processing, it also leads to low accuracy in log classification because the importance of words in logs varies and the classification results are often influenced by these words.

[0053] Therefore, this invention provides a log classification method, system, electronic device, and storage medium. The method extracts target logs from raw logs; constructs word vectors for the target logs to obtain corresponding word vectors, thus addressing the problem in existing technologies where machine learning has limited feature extraction capabilities, making it unable to handle diverse log types and resulting in low accuracy in log classification; inputs the obtained word vectors into a reinforcement learning agent pre-trained on a bidirectional long short-term memory neural network using historical logs, enabling the reinforcement learning agent to determine the weight of each word in the target log using the word vectors, thus addressing the problem in existing technologies where the importance of words in logs is not considered, leading to low accuracy in log classification; determines the target weight of the target log based on the weight of each word in the target log; and finally inputs the target weight of the target log into a log classification model pre-trained on a corresponding neural network using historical logs labeled with log categories, so that the log classification model determines the type of the target log based on the target weight. The entire log classification process requires no manual intervention, thus solving the problems in existing technologies where manual screening, matching of logs of the same type, and rule mining and matching of logs of the same type are time-consuming, inefficient, and result in low accuracy.

[0054] See Figure 1 The diagram illustrates a flowchart of a log classification method provided by an embodiment of the present invention. The log classification method specifically includes the following steps:

[0055] S101: Extract the target log from the raw log.

[0056] In this embodiment of the application, analysis of historical logs reveals that the logs are semi-structured. That is, the logs contain some variables, some fixed-structure information, and some words that do not have any practical meaning. When performing computer fault analysis, only the fixed-structure information in the logs is needed. Therefore, a corresponding log template can be set in advance so that structured logs can be extracted from the logs according to the pre-set log template.

[0057] To extract structured logs more quickly and conveniently, appropriate log extraction tools can also be used.

[0058] For example, the log extraction tool could be the FT-Tree tool.

[0059] It should be noted that variables in the log can be response time, server IP, etc.; words without practical meaning can be "the", "a", etc.; structured logs can contain the type and name of the event, and structured logs can represent the basic information of the log.

[0060] During the specific execution step S101, the original logs to be classified can be obtained so that structured target logs can be extracted from the original logs according to the log extraction tool and the pre-set log template.

[0061] For example, see Figure 2 In the various original logs shown, the numerical portions represent variables, while the remaining portions constitute the structured logs. Using log extraction tools and pre-set log templates, variable portions and meaningless words can be effectively removed from the original logs, extracting the structured target log for each original log. Figure 3 As shown.

[0062] S102: Construct word vectors for the target log to obtain the word vectors corresponding to the target log; wherein, the word vectors include the vectors corresponding to each word in the target log.

[0063] Research has found that common methods for constructing word vectors include one-hot encoding, matrix factorization, and word2vec. However, when the log text is long, using one-hot encoding can lead to the curse of dimensionality and semantic gaps. Using matrix factorization requires SVD (Single-Side Graph) solutions, which are computationally expensive, further complicating the word vector construction process. Word2vec, on the other hand, can accurately represent the relationships between log contexts and can also represent vectors from a larger bag-of-words structure. Therefore, word2vec is a viable option for constructing word vectors when the log text is long.

[0064] In the specific execution step S102, after extracting the target log from the original log, it can be first determined whether the text length of the target log is greater than the target text length; if the text length of the target log is greater than the target text length, word vectors are constructed for the target log according to word2vec to obtain the word vectors corresponding to the target log; if the text length of the target log is not greater than the target text length, word vectors are constructed for the target log according to the target word vector construction method to obtain the word vectors corresponding to the target log.

[0065] The word vectors corresponding to the target logs include the vectors corresponding to each target word arranged in sequence.

[0066] It should be noted that the target word vector can be constructed using one-hot encoding, matrix factorization, etc., and can be set according to the actual application. This application does not limit the implementation method.

[0067] S103: Input the word vectors into the pre-trained reinforcement learning agent so that the reinforcement learning agent can determine the weight corresponding to each target word in the target log based on the word vectors.

[0068] Research has shown that after constructing word vectors from logs using word2vec, the exploration space is often very large if the vectors of each word in the logs are enumerated. Therefore, the weights of each word vector in the logs can be optimized. The weight optimization of each word can be abstracted into a sequence problem, so a corresponding reinforcement learning agent can be constructed to determine the weight of each word.

[0069] In this embodiment of the application, the bidirectional long short-term memory neural network can be trained in advance using historical logs to obtain a reinforcement learning agent; so that after obtaining the target word vector of the target log, the word vector of the target log is input into the pre-trained reinforcement learning agent, and the reinforcement learning agent determines the target weight corresponding to each word in the target log based on the word vector of each word in the input target log.

[0070] Optionally, the process of pre-training a bidirectional long short-term memory neural network (BSSN) using historical logs to obtain a reinforcement learning agent can be as follows: Obtain historical logs and extract target historical logs from them; construct word vectors for the target historical logs; wherein the word vectors of the target historical logs include the vectors of each word arranged in sequence; input the word vectors of the target historical logs into the BSSN to be trained, so that the BSSN to be trained determines the weight of each word in the target historical logs based on the word vectors, and adjusts the parameters of the BSSN to be trained in reverse using an anomaly detection evaluation function constructed based on the weights of each word, until the BSSN to be trained converges, thus obtaining the reinforcement learning agent; wherein the weight of the first word in the target historical logs is determined based on the word vector of the first word, and the weight of the nth word in the target historical logs is determined based on the weight of the (n-1)th word and the word vector of the nth word, where n is greater than 1.

[0071] For example, such as Figure 4 As shown, the word vectors of the target historical log include the vectors of n words arranged in sequence. These n word vectors are input into the reinforcement learning agent to be trained. The agent determines the weight λ1 of the first word based on its vector S1, and determines the weight λ2 of the second word based on its weight λ1 and the vector S2 of the second word. This process is repeated until the weight λ1 of the nth word is determined. nThe process continues until the reinforcement learning agent converges. A corresponding anomaly detection evaluation function is constructed based on the weights of each word. This function, along with its reward and parameter combination, is used to adjust the parameters of the reinforcement learning agent under training until convergence is achieved. The reinforcement learning agent under training is a bidirectional long short-term memory neural network, and its parameter combination consists of the weights of each word in the target historical log.

[0072] In the specific execution of step S103, after obtaining the word vector of the target log, the word vector of the target log can be further input into the trained reinforcement learning agent, so that the reinforcement learning agent can determine the weight of the s-th target word based on the vector of the s-th target word and the weight of the (s-1)-th target word in the target log, until the weight of each target word in the target log is determined; where s is greater than or equal to 1, and when s equals 1, the weight of the (s-1)-th target word is zero.

[0073] S104: Determine the target weight of the target log based on the weight of each word in the target log, and use a pre-trained log classification model to determine the log category corresponding to the target log based on the target weight of the target log.

[0074] In this embodiment, the log category corresponding to each historical log can be pre-labeled, and the target historical log can be extracted from the historical log; word vectors are constructed on the target historical log to obtain the word vectors of the target historical log, so that the word vectors of the target historical log can be input into the reinforcement learning agent to obtain the weight of each word in the target historical log.

[0075] The weights of each word in the target historical log are multiplied to obtain the weight of the target historical log. This weight is then input into the neural network to be trained. The neural network determines the log category corresponding to the historical log based on its weight. The training objective is to make the determined log category as close as possible to the labeled log category. The parameters of the neural network to be trained are adjusted until the neural network converges, thus obtaining the log classification model.

[0076] It should be noted that the neural network to be trained can be a deep neural network (DNN).

[0077] In the specific execution step S104, after obtaining the weight of each target word in the target log, the weights of each target word in the target log can be multiplied to obtain the target weight of the target log; the target weight of the target log is input into the pre-trained log classification model so that the log classification model can determine the log category corresponding to the target log based on the target weight of the target log.

[0078] This invention provides a log classification method that extracts target logs from raw logs; constructs word vectors for the target logs, obtaining word vectors corresponding to the target logs; wherein the word vectors include vectors corresponding to each word in the target logs; inputs the obtained word vectors into a reinforcement learning agent pre-trained on a bidirectional long short-term memory neural network using historical logs, so that the reinforcement learning agent determines the weight corresponding to each word in the target logs based on the word vectors, and determines the target weight of the target logs based on the weights of each word in the target logs; finally, inputs the target weight of the target logs into a log classification model pre-trained on a corresponding neural network using historical logs labeled with log categories, so that the log classification model determines the type of the target logs based on the target weights. The entire log classification process does not require manual intervention, thus solving the problems of existing technologies that require manual screening, matching of logs of the same type, mining rules of logs of the same type and performing matching work, which is not only time-consuming and inefficient, but also has low accuracy in classification results.

[0079] Based on the log classification method disclosed in the embodiments of the present invention, the embodiments of the present invention also disclose a log classification system, such as... Figure 5 As shown, the log classification system includes:

[0080] The first extraction unit 51 is used to extract the target log from the original log;

[0081] The first word vector construction unit 52 is used to construct word vectors for the target log and obtain the word vectors corresponding to the target log.

[0082] The weight determination unit 53 is used to input word vectors into a pre-trained reinforcement learning agent so that the reinforcement learning agent can determine the weight of each target word in the target log based on the word vectors; wherein, the reinforcement learning agent is obtained by the training unit using historical logs to train a bidirectional long short-term memory neural network to be trained;

[0083] The log category determination unit 54 is used to determine the target weight of the target log based on the weight of each target word in the target log, and to determine the log category corresponding to the target log based on the target weight of the target log using a pre-trained log classification model; wherein, the log classification model is obtained by training the neural network to be trained using historical logs with pre-labeled log categories.

[0084] The specific principles and execution processes of each unit in the log classification system disclosed in the above embodiments of the present invention are the same as those in the log classification method disclosed in the above embodiments of the present invention. Please refer to the corresponding parts in the log classification method disclosed in the above embodiments of the present invention, and they will not be repeated here.

[0085] This invention provides a log classification system that extracts target logs from raw logs; constructs word vectors for the target logs, obtaining word vectors corresponding to the target logs; wherein the word vectors include the vectors corresponding to each word in the target logs; the obtained word vectors are then input into a reinforcement learning agent pre-trained on a bidirectional long short-term memory neural network using historical logs, so that the reinforcement learning agent determines the weights corresponding to each word in the target logs based on the word vectors, and determines the target weights of the target logs based on the weights of each word in the target logs; finally, the target weights of the target logs are input into a log classification model pre-trained on a corresponding neural network using historical logs labeled with log categories, so that the log classification model determines the type of the target logs based on the target weights. The entire log classification process does not require manual intervention, thus solving the problems of existing technologies that require manual screening, matching of logs of the same type, mining rules for logs of the same type and performing matching work, which is not only time-consuming and inefficient, but also results in low accuracy of classification results.

[0086] Optionally, the first extraction unit includes:

[0087] The first extraction subunit is used to extract structured target logs from the raw logs based on the log extraction tool and the pre-set log template.

[0088] Optionally, the first word vector construction unit includes:

[0089] The judgment unit is used to determine whether the text length of the target log is greater than the target text length;

[0090] The first word vector construction subunit is used to construct word vectors for the target log based on word2vec if the text length of the target log is greater than the target text length, thereby obtaining the word vectors corresponding to the target log.

[0091] The second word vector construction subunit is used to construct word vectors for the target log according to the target word vector construction method if the length of the target log text is not greater than the target text length, thereby obtaining the word vectors corresponding to the target log.

[0092] Optional training units include:

[0093] The second extraction unit is used to obtain historical logs and extract the target historical logs from the historical logs;

[0094] The second word vector construction unit is used to construct word vectors for the target historical log, thereby obtaining the word vectors of the target historical log; wherein, the word vectors of the target historical log include the vectors of each word arranged in sequence;

[0095] The training subunit is used to input the word vectors of the target historical log into the bidirectional long short-term memory neural network to be trained, so that the bidirectional long short-term memory neural network to be trained can determine the weight of each word in the target historical log based on the word vectors of the target historical log, and adjust the parameters of the bidirectional long short-term memory neural network to be trained in reverse based on the anomaly detection evaluation function constructed based on the weight of each word, until the bidirectional long short-term memory neural network to be trained converges, thus obtaining the reinforcement learning agent;

[0096] The weight of the first word in the target historical log is determined based on the word vector of the first word, and the weight of the nth word in the target historical log is determined based on the weight of the (n-1)th word and the word vector of the nth word, where n is greater than 1.

[0097] Optionally, the word vectors of the target log include vectors corresponding to each target word arranged in sequence, and the weight determination unit includes:

[0098] The weight determination subunit is used to input the word vectors of the target log into the pre-trained reinforcement learning agent, so that the reinforcement learning agent determines the weight of the s-th target word based on the vector of the s-th target word and the weight of the (s-1)-th target word in the target log, until the weight of each target word in the target log is determined; where s is greater than or equal to 1, and when s equals 1, the weight of the (s-1)-th target word is zero.

[0099] This application provides an electronic device, such as... Figure 6 As shown, the electronic device includes a processor 601 and a memory 602. The memory 602 is used to store program code and data for log classification, and the processor 601 is used to call the program instructions in the memory to execute the steps shown in the log classification method in the above embodiment.

[0100] This application provides a storage medium including a stored program, wherein, when the program runs, it controls the device where the storage medium is located to execute the log classification method shown in the above embodiments.

[0101] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0102] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0103] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0104] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A log classification method, characterized in that, The method includes: Extract the target log from the raw log; Word vectors are constructed from the target log to obtain the word vectors corresponding to the target log; The word vectors are input into a pre-trained reinforcement learning agent, so that the reinforcement learning agent determines the weight corresponding to each target word in the target log based on the word vectors; wherein, the reinforcement learning agent is trained using a bidirectional long short-term memory neural network to be trained using historical logs; Based on the weight of each target word in the target log, the target weight of the target log is determined, and a pre-trained log classification model is used to determine the log category corresponding to the target log based on the target weight of the target log; wherein, the log classification model is obtained by training a neural network to be trained using historical logs with pre-labeled log categories.

2. The method according to claim 1, characterized in that, The extraction of the target log from the original log includes: Based on the log extraction tool and the pre-set log template, structured target logs are extracted from the raw logs.

3. The method according to claim 1, characterized in that, The step of constructing word vectors for the target log to obtain the word vectors corresponding to the target log includes: Determine whether the text length of the target log is greater than the target text length; If the text length of the target log is greater than the target text length, word vectors are constructed for the target log using word2vec to obtain the word vectors corresponding to the target log; If the text length of the target log is not greater than the target text length, word vectors are constructed for the target log according to the target word vector construction method to obtain the word vectors corresponding to the target log.

4. The method according to claim 1, characterized in that, The process of training a bidirectional long short-term memory neural network using historical logs to obtain a reinforcement learning agent includes: Obtain historical logs and extract the target historical log from the historical logs; Word vectors are constructed from the target historical logs to obtain the word vectors of the target historical logs; wherein, the word vectors of the target historical logs include the vectors of each word arranged in sequence; The word vectors of the target historical log are input into the bidirectional long short-term memory neural network to be trained, so that the bidirectional long short-term memory neural network to be trained determines the weight of each word in the target historical log based on the word vectors of the target historical log, and the anomaly detection evaluation function constructed based on the weight of each word adjusts the parameters of the bidirectional long short-term memory neural network to be trained in reverse until the bidirectional long short-term memory neural network to be trained converges, thus obtaining the reinforcement learning agent; The weight of the first word in the target historical log is determined based on the word vector of the first word, and the weight of the nth word in the target historical log is determined based on the weight of the (n-1)th word and the word vector of the nth word, where n is greater than 1.

5. The method according to claim 1, characterized in that, The word vectors of the target log include vectors corresponding to each target word arranged in sequence. The step of inputting the word vectors of the target log into a pre-trained reinforcement learning agent, so that the reinforcement learning agent determines the weight corresponding to each target word in the target log based on the word vectors of the target log, includes: The word vectors of the target log are input into a pre-trained reinforcement learning agent, so that the reinforcement learning agent determines the weight of the s-th target word based on the vector of the s-th target word and the weight of the (s-1)-th target word in the target log, until the weight of each target word in the target log is determined. Where s is greater than or equal to 1, and when s equals 1, the weight of the (s-1)th target word is zero.

6. A log classification system, characterized in that, The system includes: The first extraction unit is used to extract the target log from the original log; The first word vector construction unit is used to construct word vectors for the target log to obtain the word vectors corresponding to the target log. A weight determination unit is used to input the word vectors into a pre-trained reinforcement learning agent, so that the reinforcement learning agent determines the weight corresponding to each target word in the target log based on the word vectors; wherein, the reinforcement learning agent is obtained by training a bidirectional long short-term memory neural network to be trained using historical logs; The log category determination unit is used to determine the target weight of the target log based on the weight of each target word in the target log, and to determine the log category corresponding to the target log based on the target weight using a pre-trained log classification model; wherein, the log classification model is obtained by training a neural network to be trained using historical logs with pre-labeled log categories.

7. The system according to claim 6, characterized in that, The first extraction unit includes: The first extraction subunit is used to extract structured target logs from the raw logs based on the log extraction tool and the pre-set log template.

8. The system according to claim 6, characterized in that, The first word vector construction unit includes: The judgment unit is used to determine whether the text length of the target log is greater than the target text length; The first word vector construction subunit is used to construct word vectors for the target log according to word2vec if the text length of the target log is greater than the target text length, thereby obtaining the word vectors corresponding to the target log. The second word vector construction subunit is used to construct word vectors for the target log according to the target word vector construction method if the text length of the target log is not greater than the target text length, thereby obtaining the word vectors corresponding to the target log.

9. An electronic device, characterized in that, include: A processor and a memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the log classification method as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for performing the log classification method as described in any one of claims 1-5.

Images