Log dual-mode anomaly detection method and system based on full semantic information
Through the log dual-mode exception detection method based on full semantic information, the semantic information of log data is extracted using the FPL algorithm and the BERT model, and combined with the dual-mode network and softmax layer, it solves the problem of accurately analyzing new logs and obtaining global local areas in the existing technology. The feature problem realizes efficient abnormal detection of massive log data.
Patent Information
- Application Number
- CN202411377810.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-11-01
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing log exception detection methods cannot accurately analyze foreign words in new logs, and cannot obtain global and local features in the text at the same time, resulting in insufficient detection effectiveness and breadth.
The log dual-mode exception detection method based on full semantic information is adopted, and the original log data is structured through the FPL algorithm. The semantic information is extracted using the pre-trained BERT model, and feature synthesis is performed through the dual-mode network (Mogrifier LSTM and TextCNN). Combined with the softmax layer to perform exception probability calculation.
It realizes semantic transformation and feature extraction of massive log data, can be applied to different types of log data, has high detection effectiveness and breadth, and can detect global and local exceptions at the same time.
Smart Images

Figure CN118885362A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of log anomaly detection, and in particular to a log dual-mode anomaly detection method and system based on full semantic information. Background Art
[0002] With the rapid development of science and technology, the service architecture of software systems is becoming increasingly complex, and the speed of user growth is unprecedented. While software systems bring convenience, they also face greater risks and challenges. System operation failures, hacker attacks, network congestion, etc. have put forward higher requirements on system performance and operation and maintenance capabilities. Minor failures in the software system will cause huge losses to users and enterprises. Log anomaly detection can also serve as a powerful assistant for development and maintenance personnel. Some fault handling requires a high level of experience for maintenance personnel. Anomaly detection technology can assist developers in their daily work through experience learning and accumulation. Research on log anomaly detection technology can not only discover system software problems, but also timely discover dangerous system vulnerabilities, minimize the adverse effects on users and enterprises, effectively ensure the security of network system software, and improve security and reliability.
[0003] Log anomaly detection takes the log data in the software system as the research object, and uses computer technology to judge the anomaly of the log at the next moment. The basic research steps include: log parsing, feature extraction and anomaly detection. The research methods include rule-based detection methods, machine learning-based detection methods and deep learning-based detection methods. Among them, in the early days, rule-based methods were mainly used for detection, which relied on the experience of development and maintenance personnel; machine learning-based methods used the template obtained after log parsing to convert the count vector, and then used similarity calculation, cluster analysis and other methods to judge anomalies; at present, deep learning-based detection methods are the most widely used, and commonly used detection models include: LSTM, Bi-GRU and Transformer, etc., gradually focusing on the contextual relationship and semantic information of log sequences. However, the existing anomaly detection methods still have defects, mainly reflected in: the inability to accurately analyze foreign words in new logs; only performing well on a single data set, and cannot be applied to different types of log data; common TextCNN, Transformer, LSTM and other models only focus on the global features or local features of the log sequence unilaterally, and cannot simultaneously obtain global and local features in the text.
[0004] Therefore, how to improve the effectiveness and extensiveness of log anomaly detection has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] In order to improve the effectiveness and extensiveness of log anomaly detection, the present application provides a log dual-mode anomaly detection method and system based on full semantic information.
[0006] In the first aspect, the present application provides a log dual-mode anomaly detection method based on full semantic information, which adopts the following technical solutions: A log dual-mode anomaly detection method based on full semantic information, comprising: Acquire original log data, and perform structured processing on the original log data through an FPL algorithm to generate target data; Obtain a pre-trained BERT model, and extract semantic information from the target data using the pre-trained BERT model to generate a semantic extraction result; Performing feature synthesis in the semantic extraction result and generating a synthesis result through a dual-mode network; The abnormal probability is calculated by combining the comprehensive result with the softmax layer to determine whether the original log data is abnormal.
[0007] Optionally, the step of obtaining original log data and performing structured processing on the original log data by using an FPL algorithm to generate target data includes: Acquire original log data, perform log segmentation on the original log data and obtain a first processing result; Performing character recognition on the first processing result to generate a second processing result; Unifying the format of the second processing result to generate a third processing result; Sub-word tagging is performed on the third processing result to generate target data.
[0008] Optionally, the step of performing character recognition on the first processing result to generate a second processing result includes: Obtaining a preset character discrimination strategy, wherein the preset character discrimination strategy includes: a time discrimination strategy, a process number discrimination strategy, and a log content discrimination strategy; Determining a target time data set in the first processing result by using the time discrimination strategy; Determining a target process number data set in the first processing result by using the process number identification strategy; Determining a target content data set in the first processing result by using the log content identification strategy; A second processing result is generated according to the target time data set, the target process number data set, and the target content data set.
[0009] Optionally, the step of performing subword tagging on the third processing result to generate target data includes: Establishing a vocabulary according to the third processing result; Obtaining all vocabulary information in the vocabulary table and splitting according to all vocabulary information to determine word frequency; Calculate score information of consecutive byte pairs according to the word frequencies to determine a merging strategy; Target data is determined in the vocabulary according to the merge strategy.
[0010] Optionally, the step of obtaining a pre-trained BERT model and extracting semantic information from the target data using the pre-trained BERT model to generate a semantic extraction result includes: Obtain a pre-trained BERT model and determine semantic information and position information in the target data; Determining a vector list in the pre-trained BERT model according to the semantic information and the position information; The vector list is input into a target encoder set in a parallel input manner to generate a semantic extraction result.
[0011] Optionally, the step of performing feature synthesis in the semantic extraction result and generating a synthesis result through a dual-mode network includes: Inputting the semantic extraction result into the Mogrifier LSTM model to extract global features to obtain a first extraction result; Inputting the semantic extraction result into a TextCNN model to extract local features to obtain a second extraction result; Perform feature synthesis based on the first extraction result and the second extraction result to generate a synthesis result.
[0012] Optionally, the step of calculating the abnormal probability by combining the comprehensive result with the softmax layer to determine whether the original log data is abnormal includes: Calculating a first feature score corresponding to the first extraction result and a second feature score corresponding to the second extraction result in the comprehensive result; determining a relevance score based on the first feature score and the second feature score; Normalizing the relevance scores through a softmax layer to generate attention weights; The abnormal probability is calculated by combining the attention weight with the back propagation algorithm to determine whether the original log data is abnormal.
[0013] In a second aspect, the present application provides a log dual-mode anomaly detection system based on full semantic information, and the log dual-mode anomaly detection system based on full semantic information includes: A target data module is used to obtain original log data and perform structured processing on the original log data through an FPL algorithm to generate target data; A semantic extraction result module, used to obtain a pre-trained BERT model, and extract semantic information from the target data through the pre-trained BERT model to generate a semantic extraction result; A comprehensive result module, used for performing feature synthesis in the semantic extraction result through a dual-mode network and generating a comprehensive result; The abnormality judgment module is used to calculate the abnormality probability through a softmax layer combined with the comprehensive result to determine whether the original log data is abnormal.
[0014] In a third aspect, the present application provides a computer device, comprising: a memory and a processor, wherein the processor executes the method described above when running computer instructions stored in the memory.
[0015] In a fourth aspect, the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method described above.
[0016] In summary, the present application includes the following beneficial technical effects: This application obtains the original log data and performs structured processing on the original log data through the FPL algorithm to generate target data; obtains the pre-trained BERT model to extract semantic information in the target data to generate semantic extraction results; performs feature synthesis in the semantic extraction results and generates comprehensive results through a dual-mode network; and performs abnormal probability calculation in combination with the comprehensive results through a softmax layer to determine whether the original log data is abnormal. For the massive logs generated in the software system, anomaly judgment is performed through semantic transformation and feature extraction. Compared with the existing anomaly detection methods, it can make full use of the semantic information of the log text, detect global and local anomalies in dual mode, and can be applied to log data generated by various software systems, with high detection effectiveness and wide application. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present application; Figure 2 This is a flow chart of the first embodiment of the log dual-mode anomaly detection method based on full semantic information of the present application; Figure 3 This is a flow chart of the first embodiment of the log dual-mode anomaly detection method based on full semantic information of the present application; Figure 4It is a structural block diagram of the first embodiment of the log dual-mode anomaly detection system based on full semantic information of the present application. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below through the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0019] Reference Figure 1 , Figure 1 A schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present application.
[0020] like Figure 1 As shown, the computer device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0021] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0022] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a log dual-mode anomaly detection program based on full semantic information.
[0023] exist Figure 1In the computer device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the present application can be set in the computer device, and the computer device calls the log dual-mode anomaly detection program based on full semantic information stored in the memory 1005 through the processor 1001, and executes the log dual-mode anomaly detection method based on full semantic information provided in the embodiment of the present application.
[0024] The present application embodiment provides a log dual-mode anomaly detection method based on full semantic information, referring to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the log dual-mode anomaly detection method based on full semantic information of the present application.
[0025] In this embodiment, the log dual-mode anomaly detection method based on full semantic information includes the following steps: Step S10: Acquire original log data, and perform structured processing on the original log data through the FPL algorithm to generate target data.
[0026] It should be noted that the FPL algorithm is a research method that applies pairwise learning to federated learning. It allows users to control the degree of sharing of their sensitive data to balance privacy protection and model effect.
[0027] Furthermore, in order to make the generated target data more reasonable and accurate, the steps of obtaining original log data and performing structured processing on the original log data through the FPL algorithm to generate target data include: obtaining original log data, segmenting the original log data and obtaining a first processing result; performing character recognition on the first processing result to generate a second processing result; unifying the format of the second processing result to generate a third processing result; and performing sub-word marking on the third processing result to generate target data.
[0028] In the specific implementation, the original log data is structured through the FPL algorithm, which mainly includes log segmentation, character identification, unified format and subword tagging.
[0029] It should be noted that the step of performing character recognition on the first processing result to generate the second processing result includes: obtaining a preset character recognition strategy, the preset character recognition strategy includes: a time recognition strategy, a process number recognition strategy and a log content recognition strategy; Determining a target time data set in the first processing result by a time discrimination strategy; Determine the target process number data set in the first processing result through the process number identification strategy; determine the target content data set in the first processing result through the log content identification strategy; generate the second processing result according to the target time data set, the target process number data set and the target content data set.
[0030] In a specific implementation, character discrimination refers to the use of a comprehensive discriminator to retain valid information in the log text, including: setting a time discriminator to judge the log time in the log text, and deleting it after it is judged to be a time; setting a process number discriminator to judge the log process number, and deleting it after it is judged to be a process number; setting a log content discriminator to judge the core message content in the log text, and retaining it after it is judged to be the core message content.
[0031] It should be noted that the unified format refers to the use of regular expressions to unify the character case formats.
[0032] It can be understood that the step of performing sub-word tagging on the third processing result to generate target data includes: establishing a vocabulary based on the third processing result; obtaining all vocabulary information in the vocabulary and splitting according to all vocabulary information to determine word frequency; calculating score information of consecutive byte pairs according to the word frequency to determine a merging strategy; and determining target data in the vocabulary according to the merging strategy.
[0033] In a specific implementation, subword tagging refers to using Wordpiece to tag subwords in valid log text, including the following steps: Vocabulary construction, building an initial character vocabulary that contains all characters in the corpus as separate subwords; Split characters, split words into character sequences, and count word frequencies ; Calculate the scores of consecutive byte pairs and merge the words with the largest scores: ; Choose the sentence language model likelihood The largest subword is added to the vocabulary; for the input log text, it is set to , including n subwords in total, Represents a subword. Assuming that each character exists independently, the language model likelihood value of sentence S is: When the adjacent subwords x and y are merged, the likelihood value of sentence S changes to: Repeat the step of merging the words with the largest scores until the word list size reaches the set threshold. The specific value of the set threshold is set according to the specific usage, and this embodiment does not limit it here.
[0034] Step S20: Obtain a pre-trained BERT model, and extract semantic information from the target data using the pre-trained BERT model to generate a semantic extraction result.
[0035] It should be noted that the steps of obtaining a pre-trained BERT model and extracting semantic information in the target data through the pre-trained BERT model to generate a semantic extraction result include: obtaining a pre-trained BERT model, determining semantic information and position information in the target data; determining a vector list in the pre-trained BERT model according to the semantic information and position information; and inputting the vector list into the target encoder set through parallel input to generate a semantic extraction result.
[0036] In the specific implementation, semantic information extraction is performed through the BERT model, including the following sub-steps: Step S21: The input log is represented by a word list, and the word list is defined as ,in l is the set log length; Step S22: Combine the semantic information and location information of the log words and obtain a vector list from the pre-trained BERT model , where is , d represents the maximum embedding dimension of the hidden layer, and l represents the length of the maximum input sequence; Step S23: Input the m log semantic vectors into multiple transform encoders in parallel, pass through the multi-head attention layer and the feedforward network layer, and then pass through the residual network and normalization operation to obtain the final semantic vector The specific steps include: Step S231: Log input data is represented as , the calculation process is: Calculating multi-head attention scores : in, The calculation of is as follows: in: , , is the learnable parameter matrix, They are vectors The dimension of i represents , and h represents the number of attention heads.
[0037] Step S232: Input to the feedforward neural network layer for training, and perform linear transformation through the ReLU function: in, are all trainable parameters. is the dimension of the feedforward neural network.
[0038] Step S30: Perform feature synthesis in the semantic extraction result through the dual-mode network and generate a synthesis result.
[0039] Furthermore, in order to improve the effectiveness of generating comprehensive results, the steps of performing feature integration in the semantic extraction results and generating comprehensive results through the dual-mode network include: The semantic extraction result is input into the Mogrifier LSTM model to extract global features to obtain the first extraction result; the semantic extraction result is input into the TextCNN model to extract local features to obtain the second extraction result; the features are integrated according to the first extraction result and the second extraction result to generate a comprehensive result.
[0040] In a specific implementation, the dual-mode network used in this embodiment refers to extracting global features through the Mogrifier LSTM model and extracting local features through the TextCNN model.
[0041] It should be noted that feature extraction through the dual-mode network model includes the following sub-steps: S31, input the semantic vector into the Mogrifier LSTM model to extract global features and obtain global features: ; S31 includes the following sub-steps: S311, input vector Through Mogrifier iteraction, iterative calculation is performed and the vector after interaction is obtained. and the hidden layer output : in, is the Sigmoid function, is the Hadamard product of matrices.
[0042] S322, input the new input vector and hidden state obtained by the interaction of S311 into the LSTM cell unit, and calculate the cell state and the hidden layer output : Among them, the calculation of cell state c is as follows: Among them, the hidden layer output The calculation of is as follows: in, They are the forget gate, input gate, new memory gate, and output gate in the LSTM unit. are the cell state and hidden layer output at the previous moment, are the learnable parameters and weights respectively.
[0043] S32, input the semantic vector into the TextCNN model to extract local features and obtain local features: ; S32 includes the following sub-steps: S321, the semantic vector is The matrix form is input to the embedding layer; S322, setting , performs convolution operation with the input vector matrix to extract local features from different dimensions: , , , where the convolution kernel sizes used are (2, d), (3, d), and (4, d); S323, the maximum pooling method is used to maximize the feature vector and retain the key features to obtain , , ; S324, concatenate the obtained vectors to obtain .
[0044] Step S40: Calculate the abnormal probability through the softmax layer combined with the comprehensive results to determine whether the original log data is abnormal.
[0045] In a specific implementation, the step of calculating the abnormality probability through a softmax layer combined with the comprehensive result to determine whether the original log data is abnormal includes: calculating the first feature score corresponding to the first extraction result in the comprehensive result and calculating the second feature score corresponding to the second extraction result; determining the correlation score according to the first feature score and the second feature score; normalizing the correlation score through a softmax layer to generate an attention weight; and calculating the abnormality probability through a back propagation algorithm combined with the attention weight to determine whether the original log data is abnormal.
[0046] In the specific implementation, feature synthesis is performed through the self-attention module, including the following sub-steps: S41, calculate the scores of the input global features and local features respectively, and use the local features as the , for each pair Calculate the score , used to indicate Correlation between: in, is the learned weight matrix.
[0047] S42, normalize the score through the softmax function to get the attention weight : The attention weight represents degree of attention.
[0048] S43, use the attention weight to weight the original sequence and get the attention convergence representation of each element: in, is the learned weight matrix.
[0049] It should be noted that after determining the attention convergence representation, the softmax layer is used to output abnormal results. It includes the following sub-steps: S51, input the feature vector F into the fully connected layer for calculation, and obtain R : S52, use the softmax layer to calculate the abnormal probability to determine whether the log is abnormal: in, Indicates the probability that the log is judged as abnormal. 、 are the weights and biases of the model, respectively, which are updated through the back-propagation algorithm.
[0050] It should be noted that if Figure 3 As shown in the overall framework diagram, this embodiment uses the FPL (Full semantic parsing method for logs) algorithm to vectorize the original log data, convert the unstructured log text into a semantic vector with full semantic information, use a comprehensive processor to unify the data format, use subword tokenization to process words, and use the BERT model to convert the sequence vector. FPL does not use the conventional template extraction method, and can fully retain the effective information of the log text, so as to extract the semantic vector with full semantic information. Based on this, a dual-mode network structure is used to detect anomalies in log sequences. The dual-mode network structure consists of a Mogrifier LSTM model and a TextCNN model, and uses a self-attention mechanism for noise reduction. The Mogrifier LSTM model can capture context-related and irrelevant information in the log sequence and encode it in a global mode, while the TextCNN model performs convolution based on convolution kernels of different sizes, which can capture local key information in the log sequence and encode it in a local mode; the addition of the self-attention mechanism can combine the advantages of the two modes, dynamically combine global dependency with local key point information, and effectively judge log anomalies.
[0051] This embodiment obtains the original log data and performs structured processing on the original log data through the FPL algorithm to generate target data; obtains the pre-trained BERT model to extract semantic information in the target data to generate semantic extraction results; performs feature synthesis in the semantic extraction results and generates comprehensive results through the dual-mode network; and performs abnormal probability calculation in combination with the comprehensive results through the softmax layer to determine whether the original log data is abnormal. For the massive logs generated in the software system, abnormality judgment is performed through semantic transformation and feature extraction. Compared with the existing abnormality detection methods, it can make full use of the semantic information of the log text, perform global and local abnormality detection in dual mode, and can be applied to log data generated by various software systems, with high detection effectiveness and wide application.
[0052] In addition, an embodiment of the present application also proposes a computer-readable storage medium, on which is stored a program for log dual-mode anomaly detection based on full semantic information. When the program for log dual-mode anomaly detection based on full semantic information is executed by a processor, the steps of the method for log dual-mode anomaly detection based on full semantic information as described above are implemented.
[0053] Reference Figure 4 , Figure 4This is a structural block diagram of the first embodiment of the log dual-mode anomaly detection system based on full semantic information of the present application.
[0054] like Figure 4 As shown, the log dual-mode anomaly detection system based on full semantic information proposed in the embodiment of the present application includes: The target data module 10 is used to obtain the original log data and perform structured processing on the original log data through the FPL algorithm to generate target data; A semantic extraction result module 20 is used to obtain a pre-trained BERT model, and extract semantic information from the target data through the pre-trained BERT model to generate a semantic extraction result; A comprehensive result module 30, used for performing feature synthesis in the semantic extraction result through a dual-mode network and generating a comprehensive result; The abnormality judgment module 40 is used to calculate the abnormality probability through the softmax layer combined with the comprehensive result to determine whether the original log data is abnormal.
[0055] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present application. In specific applications, technicians in this field can make settings as needed, and the present application does not impose any limitation on this.
[0056] This embodiment obtains the original log data and performs structured processing on the original log data through the FPL algorithm to generate target data; obtains the pre-trained BERT model to extract semantic information in the target data to generate semantic extraction results; performs feature synthesis in the semantic extraction results and generates comprehensive results through the dual-mode network; and performs abnormal probability calculation in combination with the comprehensive results through the softmax layer to determine whether the original log data is abnormal. For the massive logs generated in the software system, abnormality judgment is performed through semantic transformation and feature extraction. Compared with the existing abnormality detection methods, it can make full use of the semantic information of the log text, perform global and local abnormality detection in dual mode, and can be applied to log data generated by various software systems, with high detection effectiveness and wide application.
[0057] In one embodiment, the target data module 10 is also used to obtain original log data, perform log segmentation on the original log data and obtain a first processing result; perform character recognition on the first processing result to generate a second processing result; perform format unification on the second processing result to generate a third processing result; and perform sub-word tagging on the third processing result to generate target data.
[0058] In one embodiment, the target data module 10 is also used to obtain a preset character recognition strategy, which includes: a time recognition strategy, a process number recognition strategy, and a log content recognition strategy; determine a target time data set in the first processing result through the time recognition strategy; determine a target process number data set in the first processing result through the process number recognition strategy; determine a target content data set in the first processing result through the log content recognition strategy; and generate a second processing result based on the target time data set, the target process number data set, and the target content data set.
[0059] In one embodiment, the target data module 10 is further used to establish a vocabulary table according to the third processing result; obtain all vocabulary information in the vocabulary table and split it according to all vocabulary information to determine the word frequency; calculate the score information of consecutive byte pairs according to the word frequency to determine the merging strategy; determine the target data in the vocabulary table according to the merging strategy.
[0060] In one embodiment, the semantic extraction result module 20 is also used to obtain a pre-trained BERT model, determine semantic information and position information in the target data; determine a vector list in the pre-trained BERT model based on the semantic information and position information; and input the vector list into the target encoder set through parallel input to generate a semantic extraction result.
[0061] In one embodiment, the comprehensive result module 30 is also used to input the semantic extraction result into the Mogrifier LSTM model to extract global features to obtain a first extraction result; input the semantic extraction result into the TextCNN model to extract local features to obtain a second extraction result; perform feature synthesis based on the first extraction result and the second extraction result and generate a comprehensive result.
[0062] In one embodiment, the comprehensive result module 30 is also used to calculate the first feature score corresponding to the first extraction result and the second feature score corresponding to the second extraction result in the comprehensive result; determine the correlation score based on the first feature score and the second feature score; normalize the correlation score through a softmax layer to generate an attention weight; and calculate the abnormality probability through a back propagation algorithm combined with the attention weight to determine whether the original log data is abnormal.
[0063] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present application. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.
[0064] In addition, for technical details not described in detail in this embodiment, please refer to the method for log dual-mode anomaly detection based on full semantic information provided in any embodiment of the present application, which will not be repeated here.
[0065] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0066] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.
[0068] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the protection scope of the present application.
Claims
1. A log dual-mode anomaly detection method based on full semantic information, characterized in that: include: Acquire original log data, and perform structured processing on the original log data through an FPL algorithm to generate target data; Obtain a pre-trained BERT model, and extract semantic information from the target data using the pre-trained BERT model to generate a semantic extraction result; Performing feature synthesis in the semantic extraction result and generating a synthesis result through a dual-mode network; The abnormal probability is calculated by combining the comprehensive result with the softmax layer to determine whether the original log data is abnormal.
2. The log dual-mode anomaly detection method based on full semantic information according to claim 1 is characterized in that: The step of obtaining the original log data and performing structured processing on the original log data by using the FPL algorithm to generate target data includes: Acquire original log data, perform log segmentation on the original log data and obtain a first processing result; Performing character recognition on the first processing result to generate a second processing result; Unifying the format of the second processing result to generate a third processing result; Sub-word tagging is performed on the third processing result to generate target data.
3. The log dual-mode anomaly detection method based on full semantic information according to claim 2 is characterized in that: The step of performing character recognition on the first processing result to generate a second processing result includes: Obtaining a preset character discrimination strategy, wherein the preset character discrimination strategy includes: a time discrimination strategy, a process number discrimination strategy, and a log content discrimination strategy; Determining a target time data set in the first processing result by using the time discrimination strategy; Determining a target process number data set in the first processing result by using the process number identification strategy; Determining a target content data set in the first processing result by using the log content identification strategy; A second processing result is generated according to the target time data set, the target process number data set, and the target content data set.
4. The log dual-mode anomaly detection method based on full semantic information according to claim 2 is characterized in that: The step of performing subword tagging on the third processing result to generate target data comprises: Establishing a vocabulary according to the third processing result; Obtaining all vocabulary information in the vocabulary table and splitting according to all vocabulary information to determine word frequency; Calculate score information of consecutive byte pairs according to the word frequencies to determine a merging strategy; Target data is determined in the vocabulary according to the merge strategy.
5. The log dual-mode anomaly detection method based on full semantic information according to claim 1 is characterized in that: The step of obtaining a pre-trained BERT model and extracting semantic information from the target data using the pre-trained BERT model to generate a semantic extraction result includes: Obtain a pre-trained BERT model and determine semantic information and position information in the target data; Determining a vector list in the pre-trained BERT model according to the semantic information and the position information; The vector list is input into a target encoder set in a parallel input manner to generate a semantic extraction result.
6. The log dual-mode anomaly detection method based on full semantic information according to claim 1 is characterized in that: The step of synthesizing features in the semantic extraction result and generating a synthesis result by using a dual-mode network includes: Inputting the semantic extraction result into the Mogrifier LSTM model to extract global features to obtain a first extraction result; Inputting the semantic extraction result into a TextCNN model to extract local features to obtain a second extraction result; Perform feature synthesis based on the first extraction result and the second extraction result to generate a synthesis result.
7. The log dual-mode anomaly detection method based on full semantic information according to claim 6 is characterized in that: The step of calculating the abnormal probability by combining the comprehensive result with the softmax layer to determine whether the original log data is abnormal includes: Calculating a first feature score corresponding to the first extraction result and a second feature score corresponding to the second extraction result in the comprehensive result; determining a relevance score based on the first feature score and the second feature score; Normalizing the relevance scores through a softmax layer to generate attention weights; The abnormal probability is calculated by combining the attention weight with the back propagation algorithm to determine whether the original log data is abnormal.
8. A log dual-mode anomaly detection system based on full semantic information, characterized in that: The log dual-mode anomaly detection system based on full semantic information includes: A target data module is used to obtain original log data and perform structured processing on the original log data through an FPL algorithm to generate target data; A semantic extraction result module, used to obtain a pre-trained BERT model, and extract semantic information from the target data through the pre-trained BERT model to generate a semantic extraction result; A comprehensive result module, used for performing feature synthesis in the semantic extraction result through a dual-mode network and generating a comprehensive result; The abnormality judgment module is used to calculate the abnormality probability through a softmax layer combined with the comprehensive result to determine whether the original log data is abnormal.
9. A computer device, characterized in that: The device comprises: a memory and a processor, and when the processor runs the computer instructions stored in the memory, the processor executes the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Distributed system anomaly detection method based on log spatio-temporal feature analysis
CN116167370A
Log anomaly detection method based on semantic vectorization representation
CN117688488A
Log anomaly detection method and system based on attention and Bi-LSTM
CN118211592A
Log analysis system employing long short-term memory recurrent neural networks
US20210287068A1
Cited By
False data injection attack position detection method and device for smart power grid, and medium
CN119598451A
Log anomaly detection method and device, equipment and storage medium
CN119806966A
Computer operation and maintenance system and method based on artificial intelligence optimization
CN120872649A
Medical sensing equipment production log analysis method and device based on big data
CN121051078A
