Log data processing method, device and equipment, and storage medium
By clustering and deep learning models to process log data, the problem of low efficiency in log data processing under traditional operation and maintenance methods is solved, and efficient anomaly detection and online monitoring are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2021-12-28
- Publication Date
- 2026-07-07
AI Technical Summary
Traditional operation and maintenance methods struggle to efficiently process massive amounts of storage system log data, resulting in a heavy workload for operation and maintenance personnel. Existing algorithms are ineffective with large data volumes and cannot perform fine-grained anomaly detection.
Log keys and their category identifiers are obtained by clustering log data. A target deep learning model is then used to process the log category sequence to predict subsequent log category identifiers and compare them to detect anomalies.
It enables efficient detection of log data anomalies, reduces the workload of operations and maintenance personnel, and improves the system's online monitoring and anomaly detection capabilities.
Smart Images
Figure CN114328106B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of operation and maintenance technology, and more specifically, to a log data processing method, apparatus, electronic device, and readable storage medium. Background Technology
[0002] Operations and maintenance (O&M) involves the management and upkeep of networks, servers, and software, reducing failure rates and improving equipment efficiency. Traditionally, most O&M work was done manually by O&M personnel. However, with the rapid expansion of internet businesses and the continuous increase in labor costs, this manual O&M approach is struggling to keep pace with the times.
[0003] Storage systems are the foundation of cloud computing products, requiring the ability to access massive amounts of files and handle large-scale concurrent access, while also ensuring stability, reliability, and hardware fault tolerance. Operations and maintenance (O&M) is the most direct and effective way to ensure stability; however, since storage systems often consist of hundreds of hosts and thousands of physical storage nodes, and even a medium-sized storage system contains massive amounts of logs and metrics data, relying solely on O&M personnel to manually maintain the storage system would consume a significant amount of human resources.
[0004] System logs record the system's status and various important events, helping operations and maintenance personnel to debug performance, locate faults, and perform root cause analysis. Proper use of system logs enables effective online monitoring and anomaly detection.
[0005] As mentioned above, how to process log data to detect system anomalies has become an urgent problem to be solved.
[0006] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0007] The purpose of this disclosure is to provide a log data processing method, apparatus, electronic device, and readable storage medium for detecting system anomalies.
[0008] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.
[0009] According to one aspect of this disclosure, a log data processing method is provided, comprising: obtaining a log key and its corresponding category identifier, wherein the log key is obtained by clustering first log data according to target frequency words; obtaining a log category sequence of log data to be processed based on the log key and its corresponding category identifier, wherein the log category sequence of the log data to be processed includes a first log category identifier and a second log category identifier, wherein the first log category identifier is ranked before the second log category identifier in the log category sequence of the log data to be processed; processing the log category sequence of the log data to be processed using a target deep learning model to obtain a predicted subsequent log category identifier of the first log category identifier; and comparing the second log category identifier with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal.
[0010] According to one embodiment of this disclosure, the log key is obtained by clustering the first log data according to target frequency words, including: dividing the first log data into multiple log data, the first log data including the log data to be processed; clustering single log data containing the same target frequency words in the multiple log data of the first log data into the same category, and obtaining the same target frequency words in each category of log data as the log key; numbering the log categories of the various types of log data to obtain the category identifier corresponding to the log key.
[0011] According to an embodiment of this disclosure, the method further includes: counting the frequency of each of the multiple words in the first log data; obtaining multiple second preset frequency thresholds arranged in ascending order; determining the first preset frequency threshold from the multiple second preset frequency thresholds, so that if a word in the first log data with a frequency higher than the first preset frequency threshold is taken as the target frequency word, the number of log categories obtained by clustering multiple log data of the first log data according to the target frequency word is within a preset range.
[0012] According to one embodiment of this disclosure, obtaining a log category sequence of log data to be processed based on the log key and its corresponding category identifier includes: dividing the log data to be processed into multiple log data according to thread identifiers; matching each log data of the log data to be processed with the log key to obtain a category identifier corresponding to each log data of the log data to be processed; sorting the category identifiers corresponding to each log data according to the thread identifier in chronological order to obtain a log category sequence of the log data to be processed; and processing the log category sequence of the log data to be processed using a target deep learning model, including: obtaining a target sequence length using a random number generation method; dividing the log category sequence of the log data to be processed into multiple log sessions to be processed according to the target sequence length, wherein each log session to be processed includes a first log category identifier and a second log category identifier, the first log category identifier being a first log category identifier sequence including multiple log category identifiers; inputting the first log category identifier sequence into the target deep learning model, and predicting subsequent log category identifiers of the first log category identifier sequence using the target deep learning model.
[0013] According to one embodiment of the present disclosure, the target deep learning model includes a target transformer network and a target fully connected layer, wherein the target transformer network includes a word embedding layer, an encoder, and a decoder;
[0014] The process of inputting the first log category identifier sequence into the target deep learning model and predicting the subsequent log category identifiers of the first log category identifier sequence through the target deep learning model includes: inputting the first log category identifier sequence into the target transformer network and obtaining word vectors of the first log category identifier sequence through the word embedding layer; performing positional encoding on the word vectors of the first log category identifier sequence to obtain encoder input vectors; sequentially passing the encoder input vectors through the encoder and the decoder for encoding and decoding processing to obtain decoder output vectors; and normalizing the decoder output vectors after passing them through the target fully connected layer to obtain prediction weight vectors for subsequent log category identifiers.
[0015] According to one embodiment of this disclosure, the first log data includes non-abnormal log data; the method further includes: pre-training an initial deep learning model using the non-abnormal log data to obtain a pre-trained deep learning model; and fine-tuning the pre-trained deep learning model using the non-abnormal log data to obtain the target deep learning model.
[0016] According to one embodiment of this disclosure, the initial deep learning model includes an initial transformer network and an initial fully connected layer; the pre-trained deep learning model includes a pre-trained transformer network and the initial fully connected layer; pre-training the initial deep learning model using the non-abnormal log data includes: performing mask prediction and adjacency prediction based on the non-abnormal log data through the initial transformer network, and updating the initial transformer network according to the mask prediction results and adjacency prediction results to obtain the pre-trained transformer network; fine-tuning the pre-trained deep learning model using the non-abnormal log data to obtain the target deep learning model includes: obtaining the non-abnormal log data according to the log key and its corresponding category identifier. The process involves: obtaining a log category sequence from the non-abnormal log data; acquiring a training log session from the log category sequence of the non-abnormal log data, wherein the training log session includes a first log category identifier and a second log category identifier; inputting the first log category identifier sequence of the training log session into the pre-trained deep learning model, and using the pre-trained deep learning model to predict the subsequent log category identifiers of the first log category identifier sequence of the training log session, thereby obtaining the predicted subsequent log category identifiers of the training log session; using the second log category identifier of the training log session as a label, and updating the pre-trained deep learning model based on the predicted subsequent log category identifiers of the training log session according to a normalized cross-entropy loss function, thereby obtaining the target deep learning model.
[0017] According to another aspect of this disclosure, a log data processing apparatus is provided, comprising: an acquisition module for acquiring a log key and its corresponding category identifier, wherein the log key is obtained by clustering first log data according to target frequency words; an acquisition module for acquiring a log category sequence of log data to be processed based on the log key and its corresponding category identifier, wherein the log category sequence of the log data to be processed includes a first log category identifier and a second log category identifier, wherein the first log category identifier is ranked before the second log category identifier in the log category sequence of the log data to be processed; a processing module for processing the log category sequence of the log data to be processed using a target deep learning model to obtain a predicted subsequent log category identifier of the first log category identifier; and a detection module for comparing the second log category identifier with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal.
[0018] According to an embodiment of this disclosure, the obtaining module is further configured to: divide the first log data into multiple log data entries, the first log data including the log data to be processed; cluster single log data entries containing the same target frequency word among the multiple log data entries of the first log data into the same category, and obtain the same target frequency word among the log data of each category as the log key; number the log categories of the log data of each category, and obtain the category identifier corresponding to the log key.
[0019] According to an embodiment of this disclosure, the obtaining module is further configured to: count the frequency of each of the multiple words in the first log data; obtain multiple second preset frequency thresholds arranged in ascending order; determine the first preset frequency threshold from the multiple second preset frequency thresholds, so that if a word in the first log data with a frequency higher than the first preset frequency threshold is taken as the target frequency word, the number of log categories obtained by clustering multiple log data of the first log data according to the target frequency word is within a preset range.
[0020] According to an embodiment of this disclosure, the obtaining module is further configured to: divide the log data to be processed into multiple log data according to thread identifiers; match each log data of the log data to be processed with the log key to obtain the category identifier corresponding to each log data of the log data to be processed; sort the category identifiers corresponding to each log data according to the thread identifier in chronological order to obtain the log category sequence of the log data to be processed; the processing module is further configured to: obtain the target sequence length using a random number generation method; divide the log category sequence of the log data to be processed into multiple log sessions to be processed according to the target sequence length, wherein the log sessions to be processed include the first log category identifier and the second log category identifier, the first log category identifier being a first log category identifier sequence including multiple log category identifiers; input the first log category identifier sequence into the target deep learning model, and predict the subsequent log category identifiers of the first log category identifier sequence through the target deep learning model.
[0021] According to an embodiment of this disclosure, the target deep learning model includes a target transformer network and a target fully connected layer. The target transformer network includes a word embedding layer, an encoder, and a decoder. The processing module is further configured to: input the first log category identifier sequence into the target transformer network, and obtain word vectors of the first log category identifier sequence through the word embedding layer; perform positional encoding on the word vectors of the first log category identifier sequence to obtain an encoder input vector; sequentially pass the encoder input vector through the encoder and the decoder for encoding and decoding processing to obtain a decoder output vector; and normalize the decoder output vector after passing it through the target fully connected layer to obtain a prediction weight vector for subsequent log category identifiers.
[0022] According to one embodiment of this disclosure, the first log data includes non-abnormal log data; the apparatus further includes: a training module, configured to pre-train an initial deep learning model using the non-abnormal log data to obtain a pre-trained deep learning model; and to fine-tune the pre-trained deep learning model using the non-abnormal log data to obtain the target deep learning model.
[0023] According to one embodiment of this disclosure, the initial deep learning model includes an initial transformer network and an initial fully connected layer; the pre-trained deep learning model includes a pre-trained transformer network and the initial fully connected layer; the training module is further configured to: perform mask prediction and adjacency prediction based on the non-abnormal log data through the initial transformer network, and update the initial transformer network according to the mask prediction result and adjacency prediction result to obtain the pre-trained transformer network; obtain the log category sequence of the non-abnormal log data according to the log key and its corresponding category identifier; and obtain training logs from the log category sequence of the non-abnormal log data. The training log session includes a first log category identifier and a second log category identifier. The first log category identifier sequence of the training log session is input into the pre-trained deep learning model. The pre-trained deep learning model predicts the subsequent log category identifiers of the first log category identifier sequence of the training log session to obtain the predicted subsequent log category identifiers of the training log session. The second log category identifier of the training log session is used as a label. Based on the predicted subsequent log category identifiers of the training log session, the pre-trained deep learning model is updated according to a normalized cross-entropy loss function to obtain the target deep learning model.
[0024] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor, when executing the executable instructions, implements any of the methods described above.
[0025] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, which, when executed by a processor, implement any of the methods described above.
[0026] The log data processing method provided in the embodiments of this disclosure obtains log keys by clustering the first log data according to target frequency words. Then, based on the log keys and their corresponding category identifiers, a log category sequence of the log data to be processed, including a first log category identifier and a second log category identifier arranged in sequence, is obtained. The log category sequence of the log data to be processed is processed by a target deep learning model to obtain a predicted subsequent log category identifier of the first log category identifier. The second log category identifier is then compared with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal. This enables efficient detection of log data anomalies, thereby enabling effective online monitoring and anomaly detection of the system.
[0027] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description
[0028] The above and other objects, features and advantages of this disclosure will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.
[0029] Figure 1 A schematic diagram of a system structure according to an embodiment of this disclosure is shown.
[0030] Figure 2 A flowchart of a log data processing method according to an embodiment of this disclosure is shown.
[0031] Figure 3 It shows Figure 2 The step S202 shown is a schematic diagram of the processing procedure in one embodiment.
[0032] Figure 4 according to Figure 2 A flowchart illustrating a method for obtaining target frequency words in one embodiment is shown.
[0033] Figure 5 It shows Figure 2 The steps S204 and S206 shown are schematic diagrams of the processing procedure in one embodiment.
[0034] Figure 6 It shows Figure 5 The step S512 shown is a schematic diagram of the processing procedure in one embodiment.
[0035] Figure 7 It is based on Figure 2 and Figure 6 A schematic diagram of a converter network structure is shown.
[0036] Figure 8 It is based on Figure 2 and Figure 6 The diagram illustrates an implementation of log anomaly detection using a deep learning model.
[0037] Figure 9 according to Figures 6 to 8 A schematic diagram of the training process for a deep learning model is shown in one embodiment.
[0038] Figure 10 A block diagram of a log data processing apparatus according to an embodiment of the present disclosure is shown.
[0039] Figure 11 A block diagram of another log data processing apparatus according to an embodiment of the present disclosure is shown.
[0040] Figure 12 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation
[0041] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.
[0042] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.
[0043] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. The symbol " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0044] In this disclosure, unless otherwise expressly specified and limited, the term "connection" and similar terms should be interpreted broadly, for example, it can refer to an electrical connection or the ability to communicate with each other; it can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.
[0045] Principal Component Analysis (PCA) is used in some related technologies for anomaly detection in logs. After eigenvalue decomposition, PCA obtains eigenvectors that reflect different directions of variance variation in the original data. The eigenvalues represent the magnitude of the variance in the corresponding direction, and the magnitude of the variance can be used to determine whether a sample is an outlier. Before analyzing system logs using PCA, log keys are first grouped by session, and then the number of times each log key value appears in each session is calculated. The session vector has a size of n, corresponding to the number of times each log key appears in that session. This forms a matrix where each column is a log key and each row is a session vector. PCA uses the eigenvectors obtained after eigenvalue decomposition for anomaly detection.
[0046] Other related technologies employ the N-gram model algorithm for anomaly detection in logs. N-gram is an algorithm based on a statistical language model. Its basic idea is to select the content of the text according to a sliding window of size N, forming a sequence of byte segments (grams) of length N. The frequency of occurrence of all grams is statistically analyzed, and the probability of the text content appearing is determined based on the statistical information. First, the log keys are grouped by session, with each session being a sequence of log keys. The correlation between log keys is statistically analyzed using the N-gram model, and the conditional probability is shown in the following formula (1).
[0047]
[0048] Then, the probability of the log sequence appearing in the session is obtained based on the statistical probability. If the probability is lower than a certain threshold, the session is considered to be abnormal.
[0049] Currently, most log data processing uses algorithms such as PCA and N-gram. However, as the amount of data increases, these algorithms will face three serious shortcomings:
[0050] 1. Parameters are manually adjusted, and clustering results rely on human experience;
[0051] 2. It can only perform simple clustering and cannot perform more refined operations;
[0052] 3. The large number of calculation results does not effectively reduce the workload of maintenance personnel.
[0053] In recent years, with the rapid development of artificial intelligence (AI) technology, people have begun to try applying AI to the field of operations and maintenance (O&M), giving rise to AIOps (Artificial Intelligence for IT Operations). Based on existing O&M data (such as logs, monitoring information, and application information), machine learning can be used to solve problems that traditional O&M cannot address. AI algorithms, through a training process, can learn complex patterns from massive amounts of data, helping O&M personnel troubleshoot faults and quickly locate anomalies in storage systems.
[0054] Therefore, this disclosure provides a log data processing method. The method involves clustering the first log data according to target frequency words to obtain log keys. Then, based on the log keys and their corresponding category identifiers, a log category sequence of the log data to be processed, including a first log category identifier and a second log category identifier arranged in sequence, is obtained. A target deep learning model is used to process the log category sequence of the log data to be processed to obtain a predicted subsequent log category identifier for the first log category identifier. Finally, the second log category identifier is compared with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal. This enables efficient detection of log data anomalies, thereby facilitating effective online monitoring and anomaly detection of the system.
[0055] Figure 1 An exemplary system architecture 10 is shown that can be applied to the log data processing methods or log data processing apparatus of this disclosure.
[0056] like Figure 1As shown, system architecture 10 may include terminal device 102, network 104, and server 106. Terminal device 102 may be various electronic devices with a display screen and supporting input and output, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc. Network 104 is used as a medium to provide a communication link between terminal device 102 and server 106. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables. Server 106 may be a server or server cluster providing various services, such as a backend processing server, database server, etc.
[0057] Terminal device 102 can interact with server 106 via network 104 to receive or send data. For example, terminal device 102 can obtain log keys and their corresponding category identifiers from server 106 via network 104. Alternatively, terminal device 102 can obtain the log category sequence of the log data to be processed based on the obtained log keys and their corresponding category identifiers, and then upload the log category sequence of the log data to be processed to server 106 via network 104. Or, server 106 can process the log category sequence of the log data to be processed using a target deep learning model to obtain the predicted subsequent log category identifier of the first log category identifier, and transmit the predicted subsequent log category identifier of the first log category identifier to terminal device 102 via network 104.
[0058] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0059] Figure 2 This is a flowchart illustrating a log data processing method according to an exemplary embodiment. For example... Figure 2 The method shown can be applied, for example, to the server side of the above system, or to the terminal devices of the above system.
[0060] refer to Figure 2 The method 20 provided in this embodiment may include the following steps.
[0061] In step S202, the log key and its corresponding category identifier are obtained. The log key is obtained by clustering the first log data according to the target frequency words.
[0062] In some embodiments, the first log data may include log data to be processed. The first log data used for clustering to obtain log keys may be the full log data, that is, it may include non-abnormal log data, or it may include log data to be processed that may include abnormal data.
[0063] In some embodiments, high-frequency words (keys) (i.e., target frequency words) in log data can be considered to reflect the classification structure of the log data and are suitable as the skeleton structure of the logs. Logs with the same skeleton (which may include one or more keys) can be clustered into one category, and these skeletons of different categories are called log keys. Each log key corresponds to a log category identifier. The implementation method for obtaining log keys by clustering the first log data according to the target frequency words can be referred to Figure 3 and Figure 4 .
[0064] In step S204, the log category sequence of the log data to be processed is obtained according to the log key and its corresponding category identifier. The log category sequence of the log data to be processed includes a first log category identifier and a second log category identifier. In the log category sequence of the log data to be processed, the first log category identifier is placed before the second log category identifier.
[0065] In some embodiments, each log key can be assigned a log category identifier (e.g., a number), and the corresponding log key category identifiers can form a log category sequence of the log data to be processed based on the order in which each log entry appears. For detailed implementation methods, please refer to... Figure 5 .
[0066] In step S206, the log category sequence of the log data to be processed is processed by the target deep learning model to obtain the predicted subsequent log category identifier of the first log category identifier.
[0067] In some embodiments, for example, the target deep learning model may include a target transformer network and a target fully connected layer. The target transformer network includes a word embedding layer, an encoder, and a decoder. The network structure can be referred to... Figure 7 and Figure 8 For a detailed implementation of predicting log category sequences in the log data to be processed using a target deep learning model, please refer to [link / reference]. Figure 6 and Figure 7 .
[0068] In step S208, the second log category identifier is compared with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal.
[0069] In some embodiments, the predicted result can be compared with the actual result. If they are different, the log data corresponding to the second log category identifier is considered to be abnormal. If they are the same, the log is considered to be normal.
[0070] According to the log data processing method provided in this disclosure, log keys are obtained by clustering the first log data according to target frequency words. Then, a log category sequence of the log data to be processed, including a first log category identifier and a second log category identifier arranged in sequence, is obtained based on the log keys and their corresponding category identifiers. The log category sequence of the log data to be processed is processed by a target deep learning model to obtain a predicted subsequent log category identifier of the first log category identifier. The second log category identifier is then compared with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal. This enables efficient detection of log data anomalies, thereby enabling effective online monitoring and anomaly detection of the system.
[0071] Figure 3 It shows Figure 2 The step S202 shown is a schematic diagram of the processing procedure in one embodiment. (See attached diagram.) Figure 3 As shown in the embodiments of this disclosure, before step S202 above, the method provided by the embodiments of this disclosure may further include the following steps.
[0072] Step S302: Divide the first log data into multiple log data entries.
[0073] In some embodiments, log data can be separated according to thread identifier (id), and each thread's log can be treated as a single original log data.
[0074] Step S304: Cluster the single log data containing the same target frequency word from multiple log data in the first log data into the same category, and obtain the same target frequency word in each category of log data as the log key.
[0075] In some embodiments, it can be determined whether or not each log entry needs to be tokenized, depending on the actual situation. When selecting log data, logs from a normally functioning storage system should be chosen. Some logs frequently combine several words, so the English words in the logs should be tokenized. Furthermore, depending on the specific meaning of the log, it should be considered whether certain punctuation marks should be used as tokenization markers. For example, when the log data is a continuous string, each log entry can be tokenized as a single entity. For instance, "objectsmisplaced" can be tokenized as "objects" and "misplaced", and "HEALTH_WARN" can be tokenized as "HEALTH" and "WARN".
[0076] In some embodiments, system logs in unstructured text data format can be clustered to extract log keys. Target frequency words can be extracted from each log entry. Single log entries containing the same target frequency words in both content and order are clustered into the same category, and the sequence of target frequency words in each category is obtained as the log key. The method for obtaining target frequency words can be found in [reference needed]. Figure 4 .
[0077] Step S306: Number the log categories of various log data to obtain the category identifier corresponding to the log key.
[0078] In some embodiments, for example, log categories of various log data can be numbered using 1, 2, 3... to obtain category identifiers corresponding to log keys as 001, 002...100...
[0079] Figure 4 according to Figure 2 A flowchart illustrating a method for obtaining target frequency words in one embodiment is shown. Figure 4 As shown in the embodiments of this disclosure, before step S202 above, the method provided by the embodiments of this disclosure may further include the following steps.
[0080] Step S402: Calculate the frequency of each word in the first log data.
[0081] Step S404: Obtain multiple second preset frequency thresholds arranged in ascending order.
[0082] Step S406: Determine a first preset frequency threshold from multiple second preset frequency thresholds, so that if words appearing in the first log data with a frequency higher than the first preset frequency threshold are taken as target frequency words, the number of log categories obtained by clustering multiple log data of the first log data according to the target frequency words is within a preset range.
[0083] In some embodiments, p (p is a positive integer) thresholds can be manually set based on the word frequency of each word in the log, dividing the words in the log into p+1 levels. Levels [1, q] are selected as low-frequency (q is a positive integer), and levels [q, p+1] are selected as high-frequency words. These high-frequency words are the target frequency words. The value of q can be intelligently selected based on the number of clusters, controlling the log category corresponding to the log key to a certain preset range. For example, for the following log data:
[0084] login.py[line:30]-INFO:Aclient is logining
[0085] login.py[line:37]-ERROR: The user:cinder driver is illegal
[0086] login.py[line:30]-INFO: A client is login
[0087] login.py[line:37]-ERROR: The user:xueqiang isillegal
[0088] The content before the colon can be considered as high-frequency words, which are the target frequency words for forming the log key; "cinderdriver" and "xueqiang" are low-frequency words, representing the specific values of the log; the rest are medium-frequency words. When the number of categories obtained by clustering the logs with the above high-frequency words is small, these medium-frequency words can be converted into high-frequency words to increase the number of log categories.
[0089] In some embodiments, 20% of the total number of log entries can be used as a threshold, meaning that 20% of the total number of log entries represents the number of log categories obtained through clustering. This can also be modified according to the actual situation; for example, the set can also be used as the backbone for clustering the logs, thus obtaining more types of categories.
[0090] Log data is heavily influenced by programmers, resulting in somewhat arbitrary writing and making it difficult to describe using a fixed structure. Furthermore, the log structures within the same type of log data exhibit significant repetition. According to the method provided in this disclosure, for unstructured log data that differs from natural language text data, statistical methods are employed for analysis. Log keys identifying log categories are extracted from the log data based on word frequency, enabling more accurate identification of log data for prediction by deep learning models.
[0091] Figure 5 It shows Figure 2 The diagram illustrates steps S204 and S206 in one embodiment. Figure 5 As shown in the embodiments of this disclosure, steps S204 and S206 may further include the following steps.
[0092] Step S502: Divide the log data to be processed into multiple log data according to the thread identifier.
[0093] Step S504: Match each log data entry of the log data to be processed with the log key to obtain the category identifier corresponding to each log data entry of the log data to be processed.
[0094] Step S506: Sort the category identifiers corresponding to each log data according to the thread identifier in chronological order to obtain the log category sequence of the log data to be processed.
[0095] Step S508: Obtain the length of the target sequence using a random number generation method.
[0096] Step S510: Divide the log category sequence of the log data to be processed into multiple log sessions to be processed according to the target sequence length. Each log session to be processed includes a first log category identifier and a second log category identifier. The first log category identifier is a sequence of first log category identifiers that includes multiple log category identifiers.
[0097] In some embodiments, log sequences can be truncated to different lengths, so that the session length is randomly distributed between [16, 31].
[0098] Step S512: Input the first log category identifier sequence into the target deep learning model, and use the target deep learning model to predict the subsequent log category identifiers of the first log category identifier sequence.
[0099] Figure 6 It shows Figure 5 The step S512 shown is a schematic diagram of the processing procedure in one embodiment. (See attached diagram.) Figure 6 As shown in the present embodiment, step S512 may further include the following steps.
[0100] Step S602: Input the first log category identifier sequence into the target transformer network and obtain the word vector of the first log category identifier sequence through the word embedding layer.
[0101] Step S604: Position encoding is performed on the word vectors of the first log category identifier sequence to obtain the encoder input vector.
[0102] Step S606: The encoder input vector is sequentially encoded and decoded by the encoder and decoder to obtain the decoder output vector.
[0103] Step S608: The decoder output vector is normalized after passing through the target fully connected layer to obtain the prediction weight vector of the subsequent log category identifier.
[0104] In some embodiments, refer to Figure 7 , Figure 7 It is based on Figure 2 and Figure 6 A schematic diagram of a converter network structure is shown. Figure 7As shown, in the Transformer network, input 702 first passes through word embedding (S7002) to obtain word vectors, with a word vector dimension of, for example, 256; then it passes through position encoding (S7004). Since the log sequence only has relative distances, sine and cosine can be used as position information for position encoding; then it passes through the encoder and decoder sequentially to obtain output 704. The encoder and decoder can include one or more layers of Nx, with the encoder and decoder having the same number of Nx layers, for example, two layers each, for a total of four layers. At each Nx layer, it passes through a multi-head attention network (S7006), summation and normalization (S7008), a feedforward network (S7010), and summation and normalization again (S7012).
[0105] In some embodiments, Figure 8 It is based on Figure 2 and Figure 6 This diagram illustrates an implementation of log anomaly detection using a deep learning model. Figure 8 As shown, for example, during the prediction phase, the first n-1 log sequences of the session to be detected, Log1, Log2...Log1, can be input into the input layer 802. n-1 After passing through multiple Trm(Transformer)(8042,8044) neurons, the first output (T) CLS The result enters the fully connected layer 806, then undergoes Softmax normalization, and is output as the prediction result in the output layer 808. An XOR operation is performed with the actual nth log number of the session to be tested (S8002). A result of 1 indicates an anomaly, and a result of 0 indicates normal operation. During anomaly localization, logs from different machines can be batch-input into the model according to their sequence numbers. The location of the fault can be determined based on the sequence number of the detected anomaly.
[0106] Figure 9 according to Figures 6 to 8 A schematic diagram illustrating the training process of a deep learning model in one embodiment is shown. Figure 9 As shown in the embodiments of this disclosure, before step S206 above, the method provided by the embodiments of this disclosure may further include the following steps.
[0107] Step S902: Use non-abnormal log data to pre-train the initial deep learning model to obtain the pre-trained deep learning model.
[0108] In some embodiments, refer to Figure 8 The initial deep learning model may include an initial transformer network and an initial fully connected layer; the pre-trained deep learning model may include a pre-trained transformer network and an initial fully connected layer.
[0109] Step S9022: Based on non-abnormal log data, perform mask prediction and adjacency prediction through the initial transformer network, and update the initial transformer network according to the mask prediction results and adjacency prediction results to obtain the pre-trained transformer network.
[0110] In some embodiments, the initial transformer network can be pre-trained using a sequence of log key numbers (i.e., log category sequences) corresponding to non-abnormal log data, primarily training word vectors. This pre-training can be divided into two processes: a masked language model and next sentence prediction. Specifically, several log key numbers are masked in a conversation, and predictions are made for these masked words, as well as determining whether word vectors in a sequence are adjacent. A [CLS] marker can be added to the beginning of the input sequence.
[0111] Step S904: Fine-tune the pre-trained deep learning model using non-abnormal log data to obtain the target deep learning model.
[0112] Step S9042: Obtain the log category sequence of non-abnormal log data based on the log key and its corresponding category identifier.
[0113] Step S9044: Obtain a training log session from the log category sequence of non-abnormal log data. The training log session includes a first log category identifier and a second log category identifier.
[0114] Step S9046: Input the first log category identifier sequence of the training log session into the pre-trained deep learning model, and use the pre-trained deep learning model to predict the subsequent log category identifiers of the first log category identifier sequence of the training log session to obtain the predicted subsequent log category identifiers of the training log session.
[0115] Step S9048: Use the second log category identifier of the training log session as the label, and update the pre-trained deep learning model based on the predicted subsequent log category identifier of the training log session according to the normalized cross-entropy loss function to obtain the target deep learning model.
[0116] In some embodiments, the model can be fine-tuned based on pre-training to train the classification model, referring to... Figure 8 Input layer 802 takes the first n-1 (n is a positive integer) log sequences Log1, Log2...Log1 from the training log session. n-1 The last log entry n As a labeled training model, we take the first output (T) of the model. CLSAfter entering the fully connected layer 806, and then normalizing it using Softmax, the loss function (Loss, L) can use the normalized cross-entropy:
[0117]
[0118] Where N represents the number of input batches, for example, 20 sequences; M represents the number of categories corresponding to log keys, for example, 100 log keys; i and j are both positive integers; This predicts the probability that the nth log key corresponds to any of the various log keys from index 1 to M, where y is the category identifier corresponding to the actual log key; count(y j ==1) indicates how many log keys with corresponding sequence number j actually exist, meaning how many times each log key was predicted, and the prediction probabilities of each key are averaged.
[0119] According to the method provided in this disclosure, by studying the logs of storage products, and considering the characteristics of logs such as high repetition, relatively fixed writing rules, strong temporal sequence, and topological correlation, the concept of log keys is proposed, and a log key extraction algorithm based on word frequency is designed. To make the model more suitable for our data, an improved Transformer model is used for log prediction, making the data more temporally relevant, the training speed faster, and the cross-entropy more accurate. An automatic anomaly detection system with artificial intelligence is designed, which can be used in a multi-node maintained storage system based on the characteristics of distributed storage and can perform root cause localization.
[0120] Figure 10 This is a block diagram illustrating a log data processing apparatus according to an exemplary embodiment. Figure 10 The device shown can be applied, for example, to the server side of the above system, or to the terminal device of the above system.
[0121] refer to Figure 10 The apparatus 100 provided in this embodiment may include an acquisition module 1002, a obtaining module 1004, a processing module 1006, and a detection module 1008.
[0122] The acquisition module 1002 can be used to acquire log keys and their corresponding category identifiers. The log keys are obtained by clustering the first log data according to the target frequency words.
[0123] The obtaining module 1004 can be used to obtain the log category sequence of the log data to be processed based on the log key and its corresponding category identifier. The log category sequence of the log data to be processed includes a first log category identifier and a second log category identifier. In the log category sequence of the log data to be processed, the first log category identifier is placed before the second log category identifier.
[0124] The processing module 1006 can be used to process the log category sequence of the log data to be processed through the target deep learning model to obtain the predicted subsequent log category identifier of the first log category identifier.
[0125] The detection module 1008 can be used to compare the second log category identifier with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal.
[0126] Figure 11 This is a block diagram illustrating another log data processing apparatus according to an exemplary embodiment. Figure 11 The device shown can be applied, for example, to the server side of the above system, or to the terminal device of the above system.
[0127] refer to Figure 11 The apparatus 110 provided in this embodiment may include an acquisition module 1102, an acquisition module 1104, a training module 1106, a processing module 1108, and a detection module 1110.
[0128] The acquisition module 1102 can be used to acquire log keys and their corresponding category identifiers. The log keys are obtained by clustering the first log data according to the target frequency words.
[0129] The obtaining module 1104 can be used to obtain the log category sequence of the log data to be processed based on the log key and its corresponding category identifier. The log category sequence of the log data to be processed includes a first log category identifier and a second log category identifier. In the log category sequence of the log data to be processed, the first log category identifier is placed before the second log category identifier.
[0130] The module 1104 can also be used to: divide the first log data into multiple log data, the first log data including log data to be processed; cluster single log data containing the same target frequency word in the multiple log data of the first log data into the same category, and obtain the same target frequency word in each category of log data as the log key; number the log categories of each category of log data, and obtain the category identifier corresponding to the log key.
[0131] The module 1104 can also be used to: count the frequency of each of the multiple words in the first log data; obtain multiple second preset frequency thresholds arranged in ascending order; determine a first preset frequency threshold from the multiple second preset frequency thresholds, so that if words in the first log data with a frequency higher than the first preset frequency threshold are taken as target frequency words, the number of log categories obtained by clustering multiple log data of the first log data according to the target frequency words is within a preset range.
[0132] The module 1104 can also be used to: divide the log data to be processed into multiple log data according to the thread identifier; match each log data of the log data to be processed with the log key to obtain the category identifier corresponding to each log data of the log data to be processed; and sort the category identifiers corresponding to each log data according to the thread identifier in chronological order to obtain the log category sequence of the log data to be processed.
[0133] The first log data may include non-abnormal log data.
[0134] The training module 1106 can be used to pre-train an initial deep learning model using non-abnormal log data to obtain a pre-trained deep learning model; and to fine-tune the pre-trained deep learning model using non-abnormal log data to obtain a target deep learning model.
[0135] A pre-trained deep learning model can include a pre-trained transformer network and an initial fully connected layer.
[0136] The training module 1106 can also be used for: performing mask prediction and adjacency prediction based on non-abnormal log data through an initial transformer network, and updating the initial transformer network according to the mask prediction results and adjacency prediction results to obtain a pre-trained transformer network; obtaining a log category sequence of non-abnormal log data based on log keys and their corresponding category identifiers; obtaining a training log session from the log category sequence of non-abnormal log data, the training log session including a first log category identifier and a second log category identifier; inputting the first log category identifier sequence of the training log session into a pre-trained deep learning model, and predicting the subsequent log category identifiers of the first log category identifier sequence of the training log session through the pre-trained deep learning model to obtain the predicted subsequent log category identifiers of the training log session; using the second log category identifier of the training log session as a label, updating the pre-trained deep learning model according to the normalized cross-entropy loss function based on the predicted subsequent log category identifiers of the training log session to obtain the target deep learning model.
[0137] The initial deep learning model may include an initial transformer network and an initial fully connected layer.
[0138] The processing module 1108 can be used to process the log category sequence of the log data to be processed through the target deep learning model to obtain the predicted subsequent log category identifier of the first log category identifier.
[0139] A target deep learning model can include a target transformer network and a target fully connected layer. The target transformer network includes a word embedding layer, an encoder, and a decoder.
[0140] The processing module 1108 can also be used to: obtain the target sequence length using a random number generation method; divide the log category sequence of the log data to be processed into multiple log sessions to be processed according to the target sequence length, wherein each log session to be processed includes a first log category identifier and a second log category identifier, wherein the first log category identifier is a first log category identifier sequence including multiple log category identifiers; input the first log category identifier sequence into a target deep learning model, and predict the subsequent log category identifiers of the first log category identifier sequence through the target deep learning model.
[0141] The processing module 1108 can also be used to: input the first log category identifier sequence into the target transformer network, and obtain the word vector of the first log category identifier sequence through the word embedding layer; perform position encoding on the word vector of the first log category identifier sequence to obtain the encoder input vector; encode and decode the encoder input vector sequentially through the encoder and decoder to obtain the decoder output vector; and normalize the decoder output vector after passing it through the target fully connected layer to obtain the prediction weight vector of the subsequent log category identifier.
[0142] The detection module 1110 can be used to compare the second log category identifier with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal.
[0143] The specific implementation of each module in the device provided in this embodiment can be referred to the content of the above method, and will not be repeated here.
[0144] Figure 12 A schematic diagram of the structure of an electronic device according to an embodiment of this disclosure is shown. It should be noted that... Figure 12 The devices shown are merely examples of computer systems and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.
[0145] like Figure 12 As shown, device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1202 or a program loaded from storage section 1208 into random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for the operation of device 1200. CPU 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.
[0146] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1212, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.
[0147] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1212. When the computer program is executed by central processing unit (CPU) 1201, it performs the functions defined above in the system of this disclosure.
[0148] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0150] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including an acquisition module, a processing module, and a detection module. The names of these modules do not necessarily limit the module itself; for example, a data acquisition module can also be described as "a module that acquires relevant data from a connected server."
[0151] In another aspect, this disclosure also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:
[0152] The process involves obtaining log keys and their corresponding category identifiers. Log keys are obtained by clustering the first log data according to target frequency words. A log category sequence of the log data to be processed is then obtained based on the log keys and their corresponding category identifiers. This sequence includes a first log category identifier and a second log category identifier, with the first log category identifier preceding the second. A target deep learning model is then used to process the log category sequence of the log data to be processed, obtaining a predicted subsequent log category identifier for the first log category identifier. Finally, the second log category identifier is compared with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal.
[0153] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.
Claims
1. A log data processing method, characterized in that, include: Obtain log keys and their corresponding category identifiers. The log keys are obtained by clustering the first log data according to target frequency words. The first log data includes non-abnormal log data. The log category sequence of the log data to be processed is obtained based on the log key and its corresponding category identifier. The log category sequence of the log data to be processed includes a first log category identifier and a second log category identifier. In the log category sequence of the log data to be processed, the first log category identifier is placed before the second log category identifier. Based on the non-abnormal log data, mask prediction and adjacency prediction are performed through the initial transformer network in the initial deep learning model. The initial transformer network is then updated according to the mask prediction results and adjacency prediction results to obtain the pre-trained transformer network, thereby obtaining the pre-trained deep learning model. The pre-trained deep learning model is fine-tuned using the non-abnormal log data to obtain the target deep learning model; The target deep learning model processes the log category sequence of the log data to be processed to obtain the predicted subsequent log category identifier of the first log category identifier; The second log category identifier is compared with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal.
2. The method according to claim 1, characterized in that, The log key is obtained by clustering the first log data according to target frequency words, including: The first log data is divided into multiple log data entries, and the first log data includes the log data to be processed. Cluster the single log data containing the same target frequency word from multiple log data in the first log data into the same category, and obtain the same target frequency word in each category of log data as the log key; The log categories of the various log data are numbered to obtain the category identifier corresponding to the log key.
3. The method according to claim 2, characterized in that, Also includes: Calculate the frequency of each word in the first log data; Obtain multiple second preset frequency thresholds arranged in ascending order; A first preset frequency threshold is determined from a plurality of second preset frequency thresholds, such that if a word appearing in the first log data with a frequency higher than the first preset frequency threshold is taken as the target frequency word, the number of log categories obtained by clustering multiple log data of the first log data according to the target frequency word is within a preset range.
4. The method according to claim 1, characterized in that, Based on the log key and its corresponding category identifier, a log category sequence of the log data to be processed is obtained, including: The log data to be processed is divided into multiple log data entries according to the thread identifier; Match each log data entry of the log data to be processed with the log key to obtain the category identifier corresponding to each log data entry of the log data to be processed; Based on the thread identifier, the category identifiers corresponding to each log data are sorted in chronological order to obtain the log category sequence of the log data to be processed; The log category sequence of the log data to be processed is processed using a target deep learning model, including: The length of the target sequence is obtained using random number generation methods; The log category sequence of the log data to be processed is divided into multiple log sessions to be processed according to the target sequence length. Each log session to be processed includes a first log category identifier and a second log category identifier. The first log category identifier is a first log category identifier sequence that includes multiple log category identifiers. The first log category identifier sequence is input into the target deep learning model, and the target deep learning model is used to predict the subsequent log category identifiers of the first log category identifier sequence.
5. The method according to claim 4, characterized in that, The target deep learning model includes a target transformer network and a target fully connected layer. The target transformer network includes a word embedding layer, an encoder, and a decoder. The first log category identifier sequence is input into the target deep learning model, and the target deep learning model predicts the subsequent log category identifiers of the first log category identifier sequence, including: The first log category identifier sequence is input into the target transformer network, and the word vector of the first log category identifier sequence is obtained through the word embedding layer; Position encoding is performed on the word vectors of the first log category identifier sequence to obtain the encoder input vector; The encoder input vector is sequentially encoded and decoded by the encoder and the decoder to obtain the decoder output vector. The decoder output vector is passed through the target fully connected layer and then normalized to obtain the prediction weight vector of the subsequent log category identifier.
6. The method according to claim 5, characterized in that, The initial deep learning model includes an initial fully connected layer; The pre-trained deep learning model includes a pre-trained transformer network and the initial fully connected layer; Fine-tuning the pre-trained deep learning model using the non-abnormal log data to obtain the target deep learning model includes: The log category sequence of the non-abnormal log data is obtained based on the log key and its corresponding category identifier; A training log session is obtained from the log category sequence of the non-abnormal log data, and the training log session includes a first log category identifier and a second log category identifier; The first log category identifier sequence of the training log session is input into the pre-trained deep learning model, and the subsequent log category identifier of the first log category identifier sequence of the training log session is predicted by the pre-trained deep learning model to obtain the predicted subsequent log category identifier of the training log session. Using the second log category identifier of the training log session as a label, the pre-trained deep learning model is updated according to the normalized cross-entropy loss function based on the predicted subsequent log category identifier of the training log session to obtain the target deep learning model.
7. A log data processing device, characterized in that, include: The acquisition module is used to acquire log keys and their corresponding category identifiers. The log keys are obtained by clustering the first log data according to target frequency words. The first log data includes non-abnormal log data. The obtaining module is used to obtain a log category sequence of log data to be processed based on the log key and its corresponding category identifier. The log category sequence of log data to be processed includes a first log category identifier and a second log category identifier. In the log category sequence of log data to be processed, the first log category identifier is placed before the second log category identifier. The training module is used to perform mask prediction and adjacency prediction based on the non-abnormal log data through the initial transformer network in the initial deep learning model, and update the initial transformer network according to the mask prediction results and adjacency prediction results to obtain the pre-trained transformer network, so as to obtain the pre-trained deep learning model. The pre-trained deep learning model is fine-tuned using the non-abnormal log data to obtain the target deep learning model; The processing module is used to process the log category sequence of the log data to be processed through the target deep learning model to obtain the predicted subsequent log category identifier of the first log category identifier; The detection module is used to compare the second log category identifier with the predicted subsequent log category identifier to detect whether the log data corresponding to the second log category identifier is abnormal.
8. An electronic device, comprising: A memory, a processor, and executable instructions stored in the memory and executable in the processor, characterized in that the processor, when executing the executable instructions, implements the method as described in any one of claims 1-6.
9. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that, When the executable instructions are executed by the processor, they implement the method as described in any one of claims 1-6.