Abnormal log detection method and device, electronic equipment, storage medium and product

Abstract

The invention relates to the technical field of computers, and particularly provides an abnormal log detection method and device, electronic equipment, a storage medium and a product. The method comprises the following steps: acquiring to-be-detected log data, and acquiring a word vector of the log data and a position code corresponding to the word vector; the position code is used for representing a sequence position and / or an entry position of a word vector in the log data, then encoding the word vector and the position code to obtain a log semantic vector, and then processing the log semantic vector by using a pre-trained anomaly detection model to obtain an anomaly detection result. And obtaining an anomaly detection result corresponding to the log data. In conclusion, the technical scheme provided by the invention can improve the detection accuracy of the abnormal log.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an anomaly log detection method and apparatus, electronic equipment, storage medium and product. Background Technology

[0002] Logs are crucial information reflecting services and recording user actions, playing a vital role in service security monitoring, information tracing, and system service maintenance. As the user base of platform systems and applications continues to grow, the amount of logs is also increasing exponentially, generating massive amounts of data daily. Relying solely on manual analysis to detect abnormal logs is far from sufficient to meet current needs. Related technologies typically involve first parsing the logs and then using anomaly detection algorithms to detect abnormal logs. However, these anomaly detection algorithms often fail to consider the temporal changes in logs, resulting in low accuracy in anomaly detection. Summary of the Invention

[0003] This disclosure is made in view of the above-mentioned problems. This disclosure provides an anomaly log detection method and apparatus, electronic device, storage medium, and product to improve the accuracy of anomaly log detection.

[0004] According to one aspect of this disclosure, an anomaly log detection method is provided, comprising: Obtain the log data to be tested; Obtain the word vectors of the log data and the corresponding position codes of the word vectors; the position codes are used to characterize the sequence position and / or word position of the word vectors in the log data. The word vectors and the position codes are encoded to obtain the log semantic vector; The log semantic vector is processed using a pre-trained anomaly detection model to obtain the anomaly detection result corresponding to the log data.

[0005] According to another aspect of this disclosure, an anomaly log detection apparatus is provided, comprising: The first acquisition unit is used to acquire the log data to be detected; The second acquisition unit is used to acquire the word vectors of the log data and the position codes corresponding to the word vectors; the position codes are used to characterize the sequence position and / or word position of the word vectors in the log data. An encoding unit is used to encode the word vector and the position encoding to obtain a log semantic vector; The detection unit is used to process the log semantic vector using a pre-trained anomaly detection model to obtain the anomaly detection result corresponding to the log data.

[0006] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in any of the above embodiments.

[0007] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program / instructions thereon, which, when executed by a processor, implement the methods described in any of the above embodiments.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the methods described in any of the above embodiments.

[0009] As will be described in detail below, an abnormal log detection method, apparatus, electronic device, storage medium, and product according to embodiments of this disclosure involves segmenting the acquired log data to be detected to obtain word vectors. Furthermore, a positional code corresponding to each word vector is obtained. The positional code can be used to characterize the sequence position and / or word position of the word vector in the log data; in other words, the positional code can characterize the temporal dependency relationship of the word vectors. Thus, through encoding processing, log semantic vectors that more closely match the semantic meaning of the real log data are obtained. Then, a pre-trained anomaly detection model is used to process the log semantic vectors to obtain the anomaly detection result corresponding to the log data. In this process, the anomaly detection model can capture the temporal dependency and semantic relationship between log data based on the input data, which helps improve the accuracy of identifying abnormal logs. In summary, the technical solution provided by this disclosure can improve the detection accuracy of abnormal logs.

[0010] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0011] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 This disclosure provides a flowchart illustrating an abnormal log detection method.

[0013] Figure 2 This is a schematic diagram of word vector and position encoding provided in an embodiment of the present disclosure.

[0014] Figure 3 This is a schematic diagram of a word segmentation process provided in an embodiment of the present disclosure.

[0015] Figure 4 This is a schematic diagram of the architecture of a conversion encoder provided in an embodiment of this disclosure.

[0016] Figure 5 This is a schematic diagram of an adversarial training process provided in an embodiment of the present disclosure.

[0017] Figure 6 This is a flowchart illustrating another abnormal log detection method provided in an embodiment of this disclosure.

[0018] Figure 7 This diagram illustrates a comparison of the results of anomaly log detection on an HDFS dataset using embodiments of this disclosure and related technologies.

[0019] Figure 8 This diagram illustrates a comparison of the results of anomaly log detection on the BGL dataset using embodiments of this disclosure and related technologies.

[0020] Figure 9 This is an architecture diagram of an anomaly detection system provided in an embodiment of this disclosure.

[0021] Figure 10 This is a hierarchical architecture diagram of an anomaly detection system provided in an embodiment of this disclosure.

[0022] Figure 11 This is a schematic diagram illustrating a visualization method for anomaly detection results provided in an embodiment of this disclosure.

[0023] Figure 12 This is a structural block diagram of an anomaly log detection device provided in an embodiment of the present disclosure.

[0024] Figure 13 This is a hardware block diagram of an electronic device provided in an embodiment of the present disclosure.

[0025] Figure 14 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this disclosure. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0027] To address the issue of low detection accuracy of anomalous logs in related technologies, this disclosure provides a novel design concept: utilizing a pre-trained anomaly detection model for the identification and detection of anomalous logs. Furthermore, the input data of the anomaly detection model includes not only log data (i.e., word vectors) but also positional encodings that can represent temporal dependencies. Combining these two encodings yields log semantic vectors that represent semantic and temporal dependencies. Thus, the anomaly detection model can capture the temporal and semantic dependencies within the log semantic vectors, improving the accuracy of anomalous log identification. Moreover, this disclosure further enhances the robustness and adaptability of the anomaly detection model through adversarial pre-training, expanding its ability to identify unseen anomalous logs and reducing missed detections. Furthermore, this disclosure can optimize the word segmentation method based on word segmentation probability, reducing segmentation errors and further improving identification accuracy. The details are explained below.

[0028] This disclosure provides a method for detecting abnormal logs. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating an abnormal log detection method provided in an embodiment of this disclosure. Figure 1 As shown, the method includes: S102, Obtain the log data to be detected.

[0029] This disclosure does not limit the application scenarios of log data, which can be log data in any wireless system. For example, it can include, but is not limited to: log data in cloud service scenarios (such as mobile cloud storage, 139 email, and address book services, used to provide users with cloud service functions such as data storage and information synchronization); log data in communication service scenarios (such as voicemail and multi-number services, providing users with a richer communication experience); log data in capability open platforms (such as internet billing, unified authentication, big data services, providing third-party developers with capability openness and supporting the development of various applications and services); log data in support service scenarios (such as life services, mobile WLAN, testing centers, and DreamNet, used to provide users with life convenience, network access, performance testing, and other services), etc., and is not exhaustive.

[0030] Furthermore, the log data involved in the embodiments of this disclosure may include one or more of real-time log data and historical log data. Real-time log data refers to log data acquired in real time; while historical log data is generally previously acquired and pre-stored historical log data. In other words, the technical solution provided by the embodiments of this disclosure can be applied to real-time scenarios for anomaly detection of real-time log data, as well as to anomaly detection and troubleshooting of historical data, or it can be combined with some historical data to achieve anomaly detection of real-time data. Further details are omitted.

[0031] For real-time anomaly log detection scenarios, this disclosure also provides a possible method for real-time log acquisition: using a big data log collection tool to listen for and acquire real-time log data; and using a distributed message queue to consume the real-time log data, so that the consumed real-time log data can be used as log data to be detected for anomaly detection.

[0032] The big data log collection tool can be any monitoring tool capable of real-time listening and data acquisition, such as the Flume component. The distributed message queue can include, but is not limited to, Kafka queues, which can achieve ordered output and processing of real-time messages based on the message queue format. This can be used for peak shaving and valley filling to balance read and write speeds, ensuring that real-time data can be processed and consumed in real time. In a specific implementation of this embodiment, the Flume component can be used at the data collection layer to monitor the real-time data of the target system (which can be any of the business systems in the aforementioned scenarios). The collected data is consumed by Kafka and stored in the storage system, which can include, but is not limited to, distributed storage systems, such as the Hadoop Distributed File System (HDFS). Based on the above processing, the real-time log data stored in the storage system can then be processed sequentially according to the subsequent procedures.

[0033] S104, obtain the word vectors of the log data and the corresponding position codes of the word vectors; the position codes are used to characterize the sequence position and / or word position of the word vectors in the log data.

[0034] In this process, log data is typically text data, and this step is used to split the text data into multiple word vectors. For example, if the log data is "my dog ​​is cute, he likes playing.", based on this, we can obtain... Figure 2 The 11 word vectors shown. Figure 2 This is a schematic diagram illustrating a word vector and positional encoding method provided in an embodiment of this disclosure. For example... Figure 2As shown, the word vectors obtained from word segmentation can include not only text word vectors but also word vectors related to text punctuation marks. For example, text vector CLS represents the beginning of a sentence, and text vector SEP represents the end of a sentence punctuation mark. It should be understood that this is just an example, and in the relevant scenarios, other characters or text vectors can also be used to represent text punctuation marks, but we will not go into details.

[0035] The embodiments of this disclosure can employ any word segmentation method for word segmentation processing, such as word segmentation algorithms, large language models, machine learning models, etc., without any particular limitations. Furthermore, as described in the foregoing design concept section, this disclosure can improve the word segmentation method to make the word segmentation results more accurate and reasonable, as will be explained in detail below.

[0036] In this disclosure, positional encoding is used to characterize the sequence position and / or term position of a word vector in log data. In other words, positional encoding is used to indicate the temporal positional relationship of a word vector in log data. Sequence position refers to the position of the text sequence to which the word vector belongs in the log data, where the text sequence can specifically be a short sentence or a long sentence. Term position refers to the position of the word vector in the log data, for example, the position of the word vector when it is sequentially ordered within the log data.

[0037] Therefore, the positional encoding involved in the embodiments of this disclosure may include, but is not limited to, at least one of the following: sequence positional encoding and word positional encoding. Sequence positional encoding is used to characterize the position of the text sequence to which the word vector belongs in the log data; word positional encoding is used to characterize the position of the word vector in the log data.

[0038] by Figure 2 For example. Figure 2 In the example shown, a word vector (which can be denoted as Token Embeddings) corresponds to a sequence position (which can be denoted as Segment Embeddings) and a word position (which can be denoted as Position Embeddings). The sequence positions are denoted by E. A With E B It represents the temporal dependency between different short sentences (i.e., the text sequence to which they belong), while the word position is specifically used to represent the temporal dependency between word vectors.

[0039] S106, the word vectors and positional encodings are encoded to obtain the log semantic vector.

[0040] This step is used to combine temporal dependencies to determine a log semantic vector that better reflects the true meaning of the log data. In other words, the log semantic vector can reflect the temporal dependencies between data to a certain extent.

[0041] In practice, this can be achieved using a transformation encoder. In one possible embodiment, a transformation encoder can be used to encode word vectors and positional codes to obtain log semantic vectors. Transformation encoders can be of various types, and this disclosure is not limited in this regard.

[0042] In one exemplary embodiment, the transformation encoder may include, but is not limited to, an advanced language model (Bidirectional Encoder Representations from Transformers, BERT). BERT relies on the Transformer architecture and pays more attention to the contextual relationship of each word in the text, enabling it to capture semantic and contextual information in text data more accurately. Thus, by encoding the aforementioned processed log data (i.e., word vectors and positional encoding) using a transformation encoder based on the BERT architecture, high-quality semantic vectors can be generated, which is more conducive to providing a data foundation for the accurate identification of subsequent abnormal logs.

[0043] S108 uses a pre-trained anomaly detection model to process the log semantic vectors and obtain the anomaly detection results corresponding to the log data.

[0044] In the specific implementation of anomaly log detection, this disclosure uses a pre-trained anomaly detection model. This model can be one or more, including but not limited to machine learning models and large language models. For example, the anomaly detection model can adopt a deep learning architecture, such as the SparseGroupBlock model architecture. This architecture can reduce model computation and storage overhead while preserving model performance as much as possible by managing grouping parameters (Group) and sparsening redundant parameters (Sparse). It should be understood that this is only an example, and custom choices or designs can be made in actual scenarios.

[0045] Specifically, the input data of the anomaly detection model is: log semantic vectors, as described above, without further elaboration. The output data of the anomaly detection model is the anomaly detection result, which is used to indicate abnormal logs in the log data. This embodiment of the disclosure does not limit the form of the anomaly detection result. For example, it may only output the identified abnormal logs and / or the identification information of the abnormal logs; or it may output the anomaly probability of all log data, and further mark abnormal logs whose anomaly probability exceeds a threshold; or it may only output normal logs. These various output formats can be used individually or in combination, and the method is not limited.

[0046] Based on the aforementioned processing, this disclosure can supplement temporal information from the sequence dimension and / or word vector dimension, thereby strengthening the word order relationships and semantic connections between sentences and words within the log data, providing more comprehensive and in-depth log structure information for subsequent anomaly detection tasks. In this way, the anomaly detection model can capture the temporal dependencies between log data based on the positional encoding information embedded in the log semantic vector. This allows the anomaly detection model to understand both the semantic content of individual log entries and accurately grasp the temporal logic between log sequences, significantly improving the ability to identify complex anomaly logs and ultimately enhancing detection accuracy.

[0047] In summary, this disclosure performs word segmentation on the acquired log data to be detected to obtain word vectors. Furthermore, it obtains the positional encoding corresponding to each word vector. The positional encoding can be used to represent the sequence position and / or word position of the word vector in the log data. In other words, the positional encoding can represent the temporal dependencies of the word vectors. Through encoding processing, log semantic vectors that more closely match the semantic meaning of the real log data are obtained. Then, by using a pre-trained anomaly detection model to process the log semantic vectors, anomaly detection results corresponding to the log data can be obtained. In this process, the anomaly detection model can capture the temporal dependencies and semantic relationships between log data based on the input data, which helps improve the accuracy of identifying anomalous logs. Therefore, the technical solution provided by this disclosure can improve the detection accuracy of anomalous logs.

[0048] In this embodiment of the disclosure, when performing word segmentation on log data, the following implementation methods can be used: standardize the log data to obtain standardized text; obtain multiple word segmentation methods corresponding to the standardized text; perform word segmentation on the standardized text according to the word segmentation method with the highest word segmentation probability to obtain word vectors; wherein, the word segmentation probability is related to the sub-word probability of each word vector corresponding to the word segmentation method.

[0049] The standardization process can include, but is not limited to, one or more of the following: data cleaning, encoding conversion, format conversion, etc., without exhaustive or restrictive examples. Data cleaning is used to remove redundant data and reduce noise interference. The specific data types cleaned are not limited and can be customized for specific scenarios. For example, non-character markers (such as numbers, special symbols, and other redundant information) in system logs can be cleaned and filtered to simplify the original log data and reduce noise interference. Encoding conversion is used to convert log data into a unified format to avoid interference caused by inconsistent formats. For example, log data can be uniformly encoded into the Normalization Form Compatibility Composition (NFKC) defined by the Unicode standard. This is only an example and is not intended to limit this application; uniformly converting log data into other text formats is also possible and will not be elaborated further. This disclosure does not limit the storage format of log data and can be applied to log data stored in any format in any scenario. Through encoding conversion, the differences between log data of different formats can be eliminated, providing data support for subsequent processing.

[0050] It should be understood that a text can be segmented in multiple ways. This disclosure uses segmentation probability as the basis for selecting the segmentation method, choosing the one with the highest probability. Specifically, a segmentation method corresponds to a segmentation probability, meaning the segmentation probability represents the probability of segmenting according to the corresponding method. In practical scenarios, the segmentation probability is generally related to the sub-word probabilities of the word vectors corresponding to the segmentation method. For example, the segmentation probability can be the product of the sub-word probabilities of each word vector corresponding to the segmentation method, or it can be the sum or weighted sum of the sub-word probabilities of each word vector corresponding to the segmentation method, or it can be the average or weighted average of the sub-word probabilities of each word vector corresponding to the segmentation method, etc., without exhaustive enumeration.

[0051] In one possible embodiment, the method further includes: obtaining the word vector and sub-word probability corresponding to each word segmentation method; for any word segmentation method, obtaining the product of the sub-word probabilities of each word vector corresponding to the word segmentation method to obtain the word segmentation probability.

[0052] The probability of a word vector can be obtained in various ways. For example, it can be determined by a pre-trained neural network model, or by looking up a table (a pre-set table maintains the probability of each word vector corresponding to a word), or by calculating it using a pre-set word segmentation algorithm. There are no particular restrictions on this.

[0053] In addition, word segmentation can also be achieved through pre-trained word segmentation models in other implementation methods. In this case, word segmentation is performed on the log data to obtain word vectors, including: standardizing the log data to obtain standardized text (refer to the previous text, which will not be repeated here), and then using a pre-trained word segmentation model to process the standardized text to obtain word vectors; wherein, the word segmentation model is used to segment the standardized text according to the segmentation method with the highest segmentation probability.

[0054] There are no particular limitations on the type of word segmentation model involved in the embodiments of this disclosure. It can be an algorithm model, such as the Unigram algorithm model of SentencePiece, or it can be a neural network model or a large language model. There are no particular limitations on this.

[0055] Furthermore, regardless of the type of word segmentation model, it needs to be pre-trained before being used for word segmentation processing. This allows the word segmentation model to learn to perform word segmentation processing in the way that has the highest segmentation probability. For example, in one embodiment, the method further includes: pre-training the word segmentation model until the model converges; wherein the model convergence condition includes: the word loss value corresponding to the segmentation method with the highest segmentation probability is minimized.

[0056] You can refer to this. Figure 3 The word segmentation process is shown below. Figure 3 As shown, the log data can first be standardized (i.e., normalized), for example, to NFKC Unicode format. Then, the standardized log data is used to pre-train a word segmentation model (e.g., a unigram algorithm model), with the standardized log data as the input. The output data is the word segmentation result, i.e., multiple word vectors, denoted as: ,in, =1…… , Let be the total number of word vectors. Based on this, the probability of each word vector corresponding to a sub-word can be calculated, denoted as . In addition, it is also possible to obtain the current log data using the current word segmentation method. The probability of word segmentation corresponding to the word segmentation process is denoted as . Then the following relationship can be satisfied between them: That is, the segmentation probability is the product of the probabilities of each sub-word. Based on different segmentation methods, the segmentation probability corresponding to each method can be obtained. Therefore, for log data... In this regard, the optimal word segmentation method is the one with the highest segmentation probability, which satisfies the following relationship: In addition, log data needs to be obtained during the preprocessing of the word segmentation model. The loss for each word vector converges when the loss value corresponding to the optimal word segmentation method is minimized. For example, the loss function... The following relationship can be satisfied: Thus, through multiple rounds of training, the word segmentation model reaches convergence, and the model can then be saved. Subsequently, when word segmentation is needed, the log data only needs to be standardized, and then the processed data can be directly input into the pre-trained word segmentation model. The model will then process the data (which can be understood as a decoding / encoder process) to obtain the word segmentation results output by the model. Specifically, this means segmenting the data using the method with the highest segmentation probability and the lowest word vector loss to obtain multiple word vectors.

[0057] Based on the above processing, the technical solution provided in this disclosure optimizes the word segmentation process. In the initial stage of initial processing of log data, i.e., in the semantic vector generation stage, the word segmentation process is performed using the method with the highest segmentation probability. For example, the Unigram algorithm in the SentencePiece library is used to segment the preprocessed log data. Based on this, the word segmentation process no longer relies on a preset vocabulary, but can automatically learn the segmentation probability of sub-words through data statistics. This can overcome the difficulties in segmenting out-of-vocabulary (OOV) and low-frequency words, effectively handle out-of-vocabulary cases such as rare terms and spelling variations, reduce word segmentation errors, and improve word segmentation accuracy. This provides a better data foundation for subsequent processing and is conducive to further improving the accuracy of abnormal log identification.

[0058] In this embodiment of the disclosure, considering that the word vectors after word segmentation only retain log information but do not carry the temporal information in the log data, this disclosure introduces a position embedding mechanism to supplement the temporal dependency between word vectors from two dimensions. That is, position encoding may include, but is not limited to: sequence position encoding and / or word position encoding.

[0059] Thus, when obtaining the positional encoding corresponding to each word vector, the following process can be performed: obtain the sequence position and word position corresponding to each word vector; then, based on the two-layer positional embedding mechanism, encode the sequence position and word position to obtain the positional encoding.

[0060] Specifically, the two-layer positional embedding mechanism refers to supplementing word vectors with temporal information from two dimensions: sequence position and word position. Word positional encoding is used to assign an embedding vector to each word vector in a single log data entry based on its position in the sentence, thereby reinforcing the word order relationship within the sentence; while sequence positional encoding is used to assign an embedding vector to the position of a single log data entry in the overall sequence (e.g., which sentence it belongs to), thereby capturing the relative positional relationship between the current log data and the log data before and after it.

[0061] In one possible implementation, the positional encoding corresponding to the word vectors can be matrix-encoded to obtain a positional matrix corresponding to the log data, which is then used to represent the positional encoding. Specifically, different functions can be used to represent positional relationships during positional encoding. For example, for any two adjacent elements in the positional encoding, the preceding element is encoded using a first function, and the following element is encoded using a second function. In practical scenarios, the first function and the second function are different; they can be any custom function type. For example, in one implementation, the first function is a sine function, and the second function is a cosine function; or, in another implementation, the first function is a cosine function, and the second function is a sine function.

[0062] For example, if log data is segmented into N word vectors, then the corresponding N positions need to be encoded. The encoded data can form a... The position matrix. Wherein, Right now This is used to represent the total dimension of the positional embedding. For example, this disclosure performs positional embedding from two dimensions: sequence position and term position. =2. Where, Represents the grouping index of the matrix, which is related to... The value is related. This location matrix can include multiple elements, each corresponding to the location encoding result of the location code. For example, element... Indicates the first The word vector is embedded in the first... The encoded value at each position; and the element Indicates the first The word vector is embedded in the first... The encoded value at each position.

[0063] Therefore, when performing position encoding, each position code can be represented by a set of paired elements (i.e., paired encoded values). Characterized by ). For example, It can be represented as: ;and It can be represented as: Based on this, for any th For a word vector, it can be encoded by a pair of elements: an even number of bits. With odd bit encoding Distinguishing and representing is then performed. That is, each positional encoding has a unique pair of labels, so that the anomaly detection model can learn and capture the temporal dependencies between different word vectors based on the temporal patterns carried by these encoded data.

[0064] Thus, this disclosure, through a two-layer position embedding mechanism, enables the anomaly detection model to understand both the semantic content of a single log data entry and the temporal logic of the log sequence, significantly improving the ability to identify complex anomaly logs and providing more comprehensive and in-depth log structure information for subsequent anomaly detection tasks, ultimately improving detection performance.

[0065] For details, please refer to Figure 3 As shown, each word vector can correspond to a sequence position code and a word position code. Then, they can be encoded using a transformation encoder to obtain log semantic vectors, enabling the anomaly detection model to identify or learn the temporal and semantic dependencies between them based on these log semantic vectors.

[0066] The transformation encoder involved in this disclosure can have various possible types. Taking the BERT model as an example, this disclosure provides a possible architecture for a transformation encoder. See also... Figure 4 ,like Figure 4 As shown, the transformation encoder includes at least one transformation coding layer (also called a transformation encoder, which can be denoted as TransformerEocoder). It should be understood that there can be one or more transformation coding layers, connected sequentially. The input data of the first transformation coding layer is the input data of the transformation encoder (i.e., word vectors and positional encoding), and the output data of the last transformation coding layer is the output data of the transformation encoder, which can also be used as the input data for subsequent anomaly detection models. Figure 4 As shown, the input data enters the first conversion encoding layer, where the input data is composed of word vectors and positional encodings (set word positional encoding, sequence positional encoding). Figure 4 For ease of understanding, the structure of the first transformation coding layer is shown in detail, and a total of 12 transformation coding layers are briefly shown in numbered form.

[0067] Each of the transformational coding layers comprises, in sequence, a multi-head attention layer (which can be denoted as a Multi-Head Attention layer), a first connection layer (which can be denoted as a first Add&Norm layer), a feedforward network layer (which can be denoted as a FeedForward layer), and a second connection layer (which can be denoted as a second Add&Norm layer). The multi-head attention layer is used to learn the correlation between word vectors based on a multi-self-attention mechanism; the first connection layer is used to perform residual connection and normalization processing on the output data of the multi-head attention layer; the feedforward network layer is used to perform fully connected computation; and the second connection layer is used to perform residual connection and normalization processing on the output data of the feedforward network layer.

[0068] Taking the first transformational encoding layer as an example, the input data first passes through a Multi-Head Attention layer. In this layer, a multi-attention mechanism is used to learn and capture the correlations between word vectors, such as semantic and temporal correlations. Afterward, the data continues into the first connected layer. In this layer, based on a residual network analogy used in computer vision, the sum of the input data is used for layer normalization, thus solving the vanishing gradient problem. Then, the data enters a feedforward network layer for forward computation, and then connects to a second connected layer to avoid gradient vanishing through Add & Norm processing. In addition, such as... Figure 4 As shown, when the first and second connection layers perform fully connected computation, their input data includes not only the output data of the previous layer, but also the initial data before the fully connected computation (i.e., the input data of the previous layer). For example, the input data of the first connection layer includes the output data and input data of the multi-head attention layer; the input data of the second connection layer includes the output data and input data of the feedforward network layer, which will not be elaborated further.

[0069] Afterwards, multiple conversion encoding layers are required, for example... Figure 4 The data undergoes a 12-layer Transformer encoding process. Afterward, a log semantic vector can be output. In a preferred embodiment, the output vector can be extracted from the penultimate layer to form the log semantic vector and then output. This approach avoids overfitting caused by the last layer's output data being too close to the target, and also avoids significant bias caused by earlier layers not fully learning semantic and temporal relationships. In contrast, the data from the penultimate layer provides a more valuable semantic representation, which is more conducive to improving the accuracy of the recognition results. Of course, this is not intended to limit the solution; in another embodiment, the output vector can be directly extracted from the last layer to form the log semantic vector and then output.

[0070] The anomaly detection model provided in this disclosure requires pre-training before it can be used to identify anomaly logs. In real-world scenarios, the pre-training process may include, but is not limited to, the model training process. Furthermore, some embodiments also involve incremental training processes in the model application scenario, which will not be elaborated upon here. The following description uses the model training process as an example.

[0071] Specifically, in determining the model structure (e.g.) Figure 4 Based on the above, training samples need to be determined. Training samples can be, but are not limited to, log data. Training samples generally appear in pairs, including input samples and output samples. The input sample is a log semantic vector; in practical scenarios, log samples can be processed in the manner described above (e.g., word segmentation, positional encoding, and comprehensive encoding to obtain the log semantic vector), meaning the training samples and actual data are processed using the same data processing procedures and standards. The output sample is used to represent the anomaly detection result. Based on different representations of the anomaly detection result, the output sample can also have similar variations, as described above, and will not be repeated here. After determining the training samples, the anomaly detection model can be pre-trained using the training samples until the preset model convergence condition is reached. The model convergence condition can be customized, and for example, it can include, but is not limited to, one or more of the following: the number of training rounds reaches a preset round threshold, the training duration reaches a preset training duration, and the loss function reaches a preset loss threshold. There are no particular restrictions on this.

[0072] To further improve the robustness of the anomaly detection model, this disclosure also provides a novel pre-training method: adversarial training (or adversarial pre-training). Adversarial training can be implemented based on regularization, i.e., regularized adversarial training, such as, but not limited to, Clever adversarial training.

[0073] In one embodiment, the anomaly detection model can be obtained based on adversarial pre-training; wherein, adversarial pre-training includes at least one of the following: adversarial pre-training based on input perturbation and adversarial pre-training based on output perturbation.

[0074] Adversarial pre-training refers to adding adversarial perturbations to the training samples and incorporating them into the model training process. This ensures that the output of the anomaly detection model changes less even with the addition of adversarial perturbations, thereby improving the model's stability and robustness, enhancing generalization ability, avoiding overfitting, and ultimately strengthening the model's robustness. Adversarial training can be implemented based on input perturbations and / or output perturbations. Input perturbation means adding adversarial perturbation samples to the input while keeping the output samples unchanged; output perturbation means keeping the input samples unchanged while adding adversarial perturbation samples to the output.

[0075] Specifically, adversarial pre-training based on output perturbation can include the following process: using training samples to perform adversarial pre-training on the anomaly detection model until a preset model convergence condition is reached. For any round of adversarial pre-training, refer to... Figure 5 As shown, the process includes the following steps: obtaining the basic loss (also called the original loss) between the output data of the input sample and the output sample; obtaining the output perturbation corresponding to the output sample; obtaining the adversarial loss between the output data of the input sample and the output perturbation; determining the model loss based on the basic loss and the adversarial loss; and updating the model parameters of the anomaly detection model based on the model loss.

[0076] For cases involving output perturbations, model convergence conditions may include, but are not limited to, model loss being less than or equal to a preset threshold; where model loss characterizes the degree of difference between the output sample and the output perturbation. In practical scenarios, the KL divergence equation (i.e., Kullback-Leibler Divergence) can be used, along with Adam W optimization, to calculate the model loss, thereby further improving the anomaly detection model's recognition capability.

[0077] For example, the first task corresponding to the anomaly log detection task can be... The training samples of a group are abstractly represented as .in, For the input sample, and For output samples, 1~ Integers between [a certain number] Let be the number of training samples. For the case of output perturbation, the output perturbation can be represented as... At this point, the input sample corresponding to the output perturbation is still... Therefore, its model loss can be expressed as: Based on this, the model loss is less than or equal to the preset threshold, which means that the maximum difference between the output sample and the output perturbation is less than or equal to the preset threshold. This means that the anomaly detection model still has good anomaly recognition ability in the presence of interference.

[0078] Furthermore, adversarial pre-training based on input perturbations can include the following processes: adversarially pre-training the anomaly detection model using training samples until a preset model convergence condition is reached. Each round of adversarial pre-training can include the following processes: obtaining the basic loss between the input sample's output data and the output sample; obtaining the input perturbation corresponding to the input sample; obtaining the adversarial loss between the output perturbation and the input perturbation; determining the model loss based on the basic loss and the adversarial loss; and updating the model parameters of the anomaly detection model based on the model loss.

[0079] For input perturbations, the model convergence condition may include, but is not limited to, the model loss being less than or equal to a preset threshold. The model loss is related to the perturbation intensity and the degree of output difference, where the output difference is the difference between the output data of the input sample and the output data of the input perturbation, and the perturbation intensity is the difference between the input sample and the input perturbation. This convergence condition means that even with a certain degree of input perturbation, the anomaly detection model can still output similar results, thus improving model stability and anti-interference performance. In practical scenarios, the KL divergence equation (Kullback-Leibler Divergence) can still be used, with Adam W optimization added, to calculate the model loss, further improving the anomaly detection model's recognition ability.

[0080] For example, the first task corresponding to the anomaly log detection task can be... The training samples of a group are abstractly represented as .in, For the input sample, and For output samples, 1~ Integers between [a certain number] Let be the number of training samples. Based on this, for the case of input perturbation, the input perturbation can be expressed as: After processing by the anomaly detection model, its output perturbation can be expressed as: ,in, This represents the current model parameters of the anomaly detection model. Therefore, its model loss can be expressed as: ;in, It can be used to characterize the degree of difference between the input sample and the input perturbation, that is, to characterize the perturbation strength; while This can be used to characterize the degree of difference between the output data of the input sample and the output data of the input perturbation.

[0081] Based on the aforementioned processing, the anomaly detection model has good robustness and anti-interference performance. Even if the server or system frequently undergoes system updates and iterations, causing the logs to change continuously, it can still achieve high accuracy in anomaly log identification. It can adapt to the anomaly log detection needs of agile development scenarios, has a wide range of applications, and is highly flexible.

[0082] The abnormal log detection method according to embodiments of the present disclosure has been described above with reference to the accompanying drawings. To more clearly illustrate this solution, a brief exemplary description is now provided in conjunction with a real-world scenario.

[0083] For example, you can refer to Figure 6 .like Figure 6As shown, the abnormal log detection method can include two stages of processing: a log semantic vector generation stage and an abnormal log detection stage.

[0084] In the log semantic vector generation stage, this disclosure first standardizes the log data, such as splitting the raw log data and removing non-character markers. Then, it performs word segmentation (which can also be understood as sub-word partitioning) to obtain word vectors (also called word embedding vectors). Furthermore, it obtains the positional encodings corresponding to the word vectors, namely, the term positional encoding and the sequence positional encoding. These are then encoded by a transformation encoder to obtain the log semantic vector.

[0085] The next step is the anomaly log detection phase. During this phase, model training (or pre-training) can be performed beforehand. Once the model is trained, anomaly log detection and identification can be performed directly. Figure 6 The training process of the anomaly detection model is shown. For example... Figure 6 As shown, the log semantic vector obtained after the aforementioned processing (as an input sample) can be input into the anomaly detection model, thereby obtaining the output data directly output by the model. Based on this, the original loss between the output data and the output sample (i.e., the basic loss mentioned above) can be obtained. In addition, perturbations can be added to the output data, thereby obtaining the adversarial loss between the output perturbation and the output sample. Combining the two, the model loss can be obtained. Then, the model parameters are updated based on the model loss, and the gradient is eliminated to obtain the pre-trained anomaly detection model.

[0086] Then, this anomaly detection model can be used to detect abnormal logs. In practical applications, the log semantic vector generation stage can still refer to... Figure 6 The process is as shown. In the anomaly log detection stage, the generated log semantic vector is directly input into the anomaly detection model to achieve anomaly log detection and output the anomaly detection result. Further details are omitted.

[0087] Based on this, you can refer to Figure 7 and Figure 8 This document specifically illustrates a comparison of the effectiveness of the anomaly log detection method provided in this embodiment with detection methods in related technologies. Figure 7 This shows a comparison of performance results for testing the Hadoop Distributed File System Log Dataset (HDFS). Figure 8This paper presents a performance comparison of the Blue Gene / LSupercomputer Log Dataset (BGL). The performance comparison parameters used were precision, recall, and F1 score, where the F1 score is a combined score of precision and recall. Figure 7 and Figure 8 The related technologies employed include the following models: Support Vector Machine (SVM), Logistic Regression (LR), Principal Component Analysis (PCA), Integrated Model (IM), DeepLog log anomaly detection model, PLELog log anomaly detection model, LogRobust log anomaly detection model, Attention-Based Bidirectional Long Short-Term Memory (Att-Bi-LSTM) model, Log-based Convolutional Neural Network (LogCNN) model, LightLog log anomaly detection model, Causal Convolutional Long Short-Term Memory (CausalConvLSTM) model, and Convolutional Neural Network - Bidirectional Long Short-Term Memory (CNN-BiLSTM) model.

[0088] from Figure 7 and Figure 8 As can be seen, traditional machine learning models (such as SVM, LR, PCA, and IM) perform well on the simple HDFS dataset, with supervised models achieving precision and recall rates exceeding 95%. However, these models rely solely on log event count vector features and cannot handle new logs. Their performance drops significantly on the more complex BGL dataset, with unsupervised models achieving an F1 score of around 50%, revealing their weakness in handling unstable system logs. In contrast, the anomaly log detection method provided in this disclosure generally outperforms the aforementioned traditional machine learning models in anomaly detection coverage on the BGL dataset. This is because the BGL dataset has higher dimensionality, and machine learning methods have limitations when processing large amounts of feature data.

[0089] Furthermore, while models based on single deep learning networks (such as DeepLog and PLELog) outperform traditional machine learning methods (such as SVM, LR, PCA, and IM), they fail to accurately detect anomalous logs when faced with high-dimensional log datasets because they only learn discrete local features. These models, in their pursuit of extracting deep features, excessively increase network depth, leading to excessively long training times, overfitting, and performance degradation. Log data exhibits strong temporal correlation and long-distance dependencies. Some of the aforementioned models fail to consider temporal features or suffer from long-distance forgetting problems. The anomalous log detection method provided in this disclosure effectively extracts these features through data augmentation and data cleaning. Moreover, baseline models use word embedding vectors to process both log keys and parameters simultaneously, making it difficult to capture the semantic features of sentence context in logs. The anomalous log detection method provided in this disclosure solves this problem by employing both word position embedding and sequence position embedding methods, thus enabling more accurate capture of the semantic features of sentence context in logs and further improving the accuracy of anomalous log detection.

[0090] Furthermore, this disclosure also provides a system architecture scheme for the abnormal log detection method, please refer to... Figure 9 and Figure 10 .

[0091] like Figure 9 As shown, the abnormal log detection system includes several stages: data collection, model training, real-time detection, and visualization.

[0092] During the data collection phase, unified collection of distributed logs can be achieved through listening tools (such as Flume components), and the logs can be streamed to a distributed message queue (such as a Kafka queue).

[0093] During the model training phase, the labeled HDFS dataset is used as training samples, and offline training (e.g., adversarial training) is completed using a deep learning framework (such as PyTorch), generating a trained model file (in .pt format). The trained anomaly detection model can then be deployed online for anomaly log recognition.

[0094] During the real-time detection phase, log data from distributed message queues (such as Kafka queues) is synchronized to HDFS persistent storage and simultaneously flows into real-time computing components (such as Spark Streaming). These components then execute the aforementioned anomaly detection model to perform real-time detection and identification of the log data. The data is then stored in a data warehouse for hierarchical storage and other operations. Taking Spark Streaming as an example, in the specific anomaly log detection process, the trained model is distributed to the compute nodes (Executors) of the Spark cluster for streaming detection of real-time logs. Anomaly detection results are written back to the corresponding Kafka layer. Spark's Resilient Distributed Dataset (RDD) is used to perform real-time statistics on metrics such as anomaly ratio and detection latency. Furthermore, storage (such as Redis) is used to cache detection rules and model parameters, ensuring the efficient operation of the detection process. In addition, detection results and statistical metrics can be synchronized to a real-time search engine (such as ElasticSearch) and combined with online analytical processing (OLAP) tools to achieve multi-dimensional data analysis. Moreover, by continuously monitoring the log files of edge device hardware and software systems, any changes and new data generated are collected in real-time and stored in HDFS.

[0095] In the visualization stage, backend development frameworks (such as the Spring Boot backend framework) are combined with data visualization tools (such as Echarts) to display results such as abnormal trends and model performance in chart form, supporting real-time monitoring.

[0096] Figure 10 Accordingly, it shows Figure 9 The system architecture shown is a layered design scheme. For example... Figure 10 As shown, the system is structured in sequence as a data layer, logic layer, control layer, and display layer. The data layer, which can be represented as a Mapper, interacts with the real-time search engine and is used to map and retrieve data. The logic layer, represented as a Service, is used to implement specific business logic processing. The control layer, represented as a Controller, is used to receive and forward data. The display layer deploys the front-end framework and is used for human-computer interaction and visualizing data.

[0097] Based on the aforementioned system architecture, this disclosure can detect real-time log data generated by the target system, and statistically summarize and visualize abnormal log data from multiple time granularities. In this way, administrators can not only view the total number of abnormal logs generated at the current moment, but also grasp the specific number of abnormal logs at each moment of the day. For example, using the line chart component of a data visualization tool (such as Echarts), the changing trend of log anomalies for the day can be plotted and compared with the changing trend of the previous day, helping administrators to more accurately grasp the current operating status trend of the system, so as to promptly monitor and handle possible major unexpected situations, such as... Figure 11 As shown. When the set threshold is reached or a critical information error alarm is triggered, the administrator will be notified (e.g., via email).

[0098] In summary, this disclosure focuses on the design and implementation of a data-augmented log detection method, and improves the accuracy of abnormal log identification through multiple aspects such as unfamiliar word processing, temporal feature capture, and unknown anomaly identification. Specifically, it addresses the issue of unfamiliar word segmentation accuracy in the semantic vector generation stage. For example, it uses the Unigram algorithm from the SentencePiece library to segment the preprocessed log data. In this way, this disclosure can automatically learn the segmentation probability of each word vector through data statistics without relying on a pre-set vocabulary, effectively handling uncommon terms, spelling variations, and other non-lexical cases, and reducing segmentation errors. Before inputting the log data into BERT for semantic encoding, this disclosure also introduces positional embedding technology. By adding time position information (such as timestamp offsets and periodic markers) to each element (i.e., word vector) in the log sequence, the subsequent anomaly detection model can capture the periodic patterns of log events (such as daily early morning system inspection logs and weekly backup task records), improving the ability to model temporal dependencies. Furthermore, to address newly emerging log events (unknown anomaly patterns), adversarial training methods, such as Clever, are introduced. By generating adversarial examples with subtle perturbations, the anomaly detection model is forced to learn more robust semantic features, expanding the semantic coverage of the embedded vectors, enhancing its ability to identify unseen log patterns, and reducing missed detections.

[0099] In summary, this disclosure not only improves the accuracy and adaptability of log detection but also significantly enhances the ability to identify system anomalies, better handling complex situations in log data and meeting log detection needs in different scenarios. Furthermore, this disclosure improves the robustness of identifying unknown anomaly patterns. Existing technologies often suffer from insufficient semantic feature learning when facing newly emerging log events, resulting in weak ability to identify unfamiliar log patterns and a tendency to miss detections. The adversarial training mechanism introduced in this disclosure significantly reduces the probability of missed detections. In addition, this disclosure expands the semantic coverage and enhances adaptability. Existing technologies are insufficiently adaptable to dynamically changing log events, with limited semantic coverage of embedding vectors, making it difficult to handle situations such as new function logs and update operation logs. This disclosure expands the semantic coverage of embedding vectors through adversarial training, enabling the anomaly detection model to better adapt to frequent changes in log events with version iterations, maintaining high recognition accuracy and reducing detection lag when facing new types of logs. This disclosure also strengthens the anomaly detection model's ability to handle complex scenarios. Existing technologies often fail to effectively extract key anomaly information from complex log data when processing new log events due to a lack of targeted feature learning mechanisms. This disclosure, however, utilizes an adversarial training mechanism to enable the anomaly detection model to delve deeper into the semantic relationships within the log data. Even in complex scenarios where new log events differ significantly from known patterns, it can quickly and accurately identify anomalies, thus improving the reliability of the log detection system in practical applications.

[0100] This disclosure also provides an anomaly log detection device. Figure 12 This is a structural block diagram of an anomaly log detection device provided in an embodiment of the present disclosure, such as... Figure 12 As shown, the anomaly log detection device 1200 includes: The first acquisition unit 1210 is used to acquire the log data to be detected; The second acquisition unit 1220 is used to acquire the word vectors of the log data and the position codes corresponding to the word vectors; the position codes are used to characterize the sequence position and / or word position of the word vectors in the log data. Encoding unit 1230 is used to encode the word vector and the position encoding to obtain a log semantic vector; The detection unit 1240 is used to process the log semantic vector using a pre-trained anomaly detection model to obtain the anomaly detection result corresponding to the log data.

[0101] In one exemplary embodiment, the location encoding includes at least one of the following: Sequence position encoding is used to characterize the position of the text sequence to which the word vector belongs in the log data; The term position encoding is used to characterize the position of the term vector in the log data.

[0102] In one exemplary embodiment, the second acquisition unit 1220 is specifically used for: Obtain the sequence position and word position corresponding to each word vector; Based on a two-layer position embedding mechanism, the sequence position and the word position are encoded to obtain the position code.

[0103] In one exemplary embodiment, the detection unit 1240 is further configured to: The anomaly detection model is pre-trained adversarially using training samples until a preset model convergence condition is met; the training samples are log data. The adversarial pre-training includes at least one of the following: adversarial pre-training based on input perturbation and adversarial pre-training based on output perturbation.

[0104] In one exemplary embodiment, if the adversarial pre-training is based on output perturbation, during any round of adversarial pre-training, the detection unit 1240 is specifically used for: Obtain the fundamental loss between the output data of the input sample and the output sample; Obtain the output perturbation corresponding to the output sample; Obtain the adversarial loss between the output data of the input sample and the output perturbation; Based on the basic loss and the adversarial loss, determine the model loss; Based on the model loss, the model parameters of the anomaly detection model are updated.

[0105] In one exemplary embodiment, the model convergence condition includes: the model loss is less than or equal to a preset threshold; If the adversarial pre-training is based on output perturbation, the model loss is used to characterize the degree of difference between the output sample and the output perturbation; Alternatively, if the adversarial pre-training is based on output perturbation, the model loss is related to the perturbation strength and the degree of output difference, wherein the degree of output difference is the degree of difference between the output data of the input sample and the output data of the input perturbation; and the perturbation strength is the degree of difference between the input sample and the input perturbation.

[0106] In one exemplary embodiment, the encoding unit 1230 is specifically used to: encode the word vector and the position encoding using a transformation encoder to obtain a log semantic vector; The transformation encoder includes at least one transformation coding layer; each transformation coding layer includes a multi-head attention layer, a first connection layer, a feedforward network layer, and a second connection layer connected in sequence. The multi-head attention layer is used to learn the correlation between word vectors based on a multi-self attention mechanism; The first connection layer is used to perform residual connection and normalization processing on the output data of the multi-head attention layer; The feedforward network layer is used to perform fully connected computations; The second connection layer is used to perform residual connection and normalization processing on the output data of the feedforward network layer.

[0107] In one exemplary embodiment, the second acquisition unit 1220 is specifically used for: The log data is standardized to obtain standardized text; Obtain multiple word segmentation methods corresponding to the standardized text; The standardized text is segmented according to the segmentation method with the highest segmentation probability to obtain the word vector; wherein, the segmentation probability is related to the sub-word probability of each word vector corresponding to the segmentation method.

[0108] In one exemplary embodiment, the second acquisition unit 1220 is further configured to: Obtain the word vectors and sub-word probabilities corresponding to each word segmentation method; For any of the word segmentation methods, the product of the sub-word probabilities of each word vector corresponding to the word segmentation method is obtained to obtain the word segmentation probability.

[0109] In one exemplary embodiment, the second acquisition unit 1220 is specifically used for: The log data is standardized to obtain standardized text; The standardized text is processed using a pre-trained word segmentation model to obtain the word vectors; wherein the word segmentation model is used to segment the standardized text according to the segmentation method with the highest segmentation probability.

[0110] In one exemplary embodiment, the second acquisition unit 1220 is further configured to: The word segmentation model is pre-trained until it converges; wherein the convergence condition of the model includes: the word loss value corresponding to the word segmentation method with the highest probability is the smallest.

[0111] In one exemplary embodiment, the first acquisition unit 1210 is specifically used for: Use big data log collection tools to listen for and acquire real-time log data; The real-time log data is consumed using a distributed message queue, so that the consumed real-time log data is used as the log data to be detected for anomaly detection.

[0112] For details not covered, please refer to the previous text; they will not be repeated here.

[0113] Figure 13 This is a hardware block diagram of an electronic device provided according to an embodiment of the present disclosure. The electronic device 1300 according to an embodiment of the present disclosure includes at least a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the anomaly log detection method described in any of the above embodiments.

[0114] Figure 13 The illustrated electronic device 1300 specifically includes a central processing unit (CPU) 1301, a graphics processing unit (GPU) 1302, and a memory 1303. These units are interconnected via a bus 1304. The CPU 1301 and / or GPU 1302 can function as the aforementioned processor, and the memory 1303 can function as the aforementioned memory storing computer-readable instructions. Furthermore, the electronic device 1300 may also include a communication unit 1305, a storage unit 1306, an output unit 1307, an input unit 1308, and an external device 1309, all of which are also connected to the bus 1304.

[0115] Figure 14 This is a schematic diagram of a computer-readable storage medium provided according to an embodiment of the present disclosure. The computer-readable storage medium according to an embodiment of the present disclosure stores a computer program / instructions (including but not limited to computer-readable instructions). Specifically, as shown... Figure 14 As shown, a computer-readable storage medium 1400 stores computer-readable instructions 1401. When executed by a processor, this computer program / instruction implements the anomaly log detection method described in any of the preceding embodiments of this disclosure. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0116] This disclosure further provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the anomaly log detection method described in any of the preceding embodiments of this disclosure.

[0117] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0118] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0119] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0120] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0121] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0122] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0123] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An anomaly log detection method, characterized in that, include: Obtain the log data to be tested; Obtain the word vectors of the log data and the position codes corresponding to the word vectors; The positional encoding is used to characterize the sequence position and / or entry position of the word vector in the log data; The word vectors and the position codes are encoded to obtain the log semantic vector; The log semantic vector is processed using a pre-trained anomaly detection model to obtain the anomaly detection result corresponding to the log data.

2. The method according to claim 1, characterized in that, The location encoding includes at least one of the following: Sequence position encoding is used to characterize the position of the text sequence to which the word vector belongs in the log data; The term position encoding is used to characterize the position of the term vector in the log data.

3. The method according to claim 1, characterized in that, The step of obtaining the word vectors of the log data and the positional encodings corresponding to the word vectors includes: Obtain the sequence position and word position corresponding to each word vector; Based on a two-layer position embedding mechanism, the sequence position and the word position are encoded to obtain the position code.

4. The method according to claim 3, characterized in that, For any two adjacent elements in the positional encoding, the preceding element is encoded using a first function, and the following element is encoded using a second function. Wherein, the first function is a sine function and the second function is a cosine function; or, the first function is a cosine function and the second function is a sine function.

5. The method according to claim 1, characterized in that, The method further includes: performing adversarial pre-training on the anomaly detection model using training samples until a preset model convergence condition is reached; the training samples are log data. The adversarial pre-training includes at least one of the following: adversarial pre-training based on input perturbation and adversarial pre-training based on output perturbation.

6. The method according to claim 5, characterized in that, If the adversarial pre-training is based on output perturbation, any round of adversarial pre-training process includes: Obtain the fundamental loss between the input sample's output data and the output sample; Obtain the output perturbation corresponding to the output sample; Obtain the adversarial loss between the output data of the input sample and the output perturbation; Based on the basic loss and the adversarial loss, the model loss is determined; The model parameters of the anomaly detection model are updated based on the model loss.

7. The method according to claim 6, characterized in that, The model convergence condition includes: the model loss is less than or equal to a preset threshold; If the adversarial pre-training is based on output perturbation, the model loss is used to characterize the degree of difference between the output sample and the output perturbation; Alternatively, if the adversarial pre-training is based on output perturbation, the model loss is related to the perturbation strength and the degree of output difference, wherein the degree of output difference is the degree of difference between the output data of the input sample and the output data of the input perturbation; and the perturbation strength is the degree of difference between the input sample and the input perturbation.

8. The method according to claim 1, characterized in that, The process of encoding the word vector and the positional encoding to obtain the log semantic vector includes: The word vectors and position codes are encoded using a transformation encoder to obtain log semantic vectors; The transformation encoder includes at least one transformation coding layer; each transformation coding layer includes a multi-head attention layer, a first connection layer, a feedforward network layer, and a second connection layer connected in sequence. The multi-head attention layer is used to learn the correlation between word vectors based on a multi-self attention mechanism; The first connection layer is used to perform residual connection and normalization processing on the output data of the multi-head attention layer; The feedforward network layer is used to perform fully connected computations; The second connection layer is used to perform residual connection and normalization processing on the output data of the feedforward network layer.

9. The method according to claim 1, characterized in that, The step of obtaining the word vectors of the log data includes: The log data is standardized to obtain standardized text; Obtain multiple word segmentation methods corresponding to the standardized text; The standardized text is segmented according to the segmentation method with the highest segmentation probability to obtain the word vector; wherein, the segmentation probability is related to the sub-word probability of each word vector corresponding to the segmentation method.

10. The method according to claim 9, characterized in that, The method further includes: Obtain the word vectors and sub-word probabilities corresponding to each word segmentation method; For any of the word segmentation methods, the product of the sub-word probabilities of each word vector corresponding to the word segmentation method is obtained to obtain the word segmentation probability.

11. The method according to claim 1, characterized in that, The step of obtaining the word vectors of the log data includes: The log data is standardized to obtain standardized text; The standardized text is processed using a pre-trained word segmentation model to obtain the word vectors; wherein the word segmentation model is used to segment the standardized text according to the segmentation method with the highest segmentation probability.

12. The method according to claim 11, characterized in that, The method further includes: The word segmentation model is pre-trained until it converges; wherein the convergence condition includes: the word loss value corresponding to the word segmentation method with the highest probability is the smallest.

13. The method according to claim 1, characterized in that, The acquisition of the log data to be detected includes: Use big data log collection tools to listen for and acquire real-time log data; The real-time log data is consumed using a distributed message queue, so that the consumed real-time log data is used as the log data to be detected for anomaly detection.

14. An anomaly log detection device, characterized in that, include: The first acquisition unit is used to acquire the log data to be detected; The second acquisition unit is used to acquire the word vectors of the log data and the position codes corresponding to the word vectors; The positional encoding is used to characterize the sequence position and / or entry position of the word vector in the log data; An encoding unit is used to encode the word vector and the position encoding to obtain a log semantic vector; The detection unit is used to process the log semantic vector using a pre-trained anomaly detection model to obtain the anomaly detection result corresponding to the log data.

15. The apparatus according to claim 14, characterized in that, The location encoding includes at least one of the following: Sequence position encoding is used to characterize the position of the text sequence to which the word vector belongs in the log data; The term position encoding is used to characterize the position of the term vector in the log data.

16. The apparatus according to claim 14, characterized in that, The second acquisition unit is specifically used for: Obtain the sequence position and word position corresponding to each word vector; Based on a two-layer position embedding mechanism, the sequence position and the word position are encoded to obtain the position code.

17. The apparatus according to claim 14, characterized in that, The detection unit is also used for: The anomaly detection model is pre-trained adversarially using training samples until a preset model convergence condition is met; the training samples are log data. The adversarial pre-training includes at least one of the following: adversarial pre-training based on input perturbation and adversarial pre-training based on output perturbation.

18. The apparatus according to claim 17, characterized in that, If the adversarial pre-training is based on output perturbation, in any round of adversarial pre-training, the detection unit is specifically used for: Obtain the fundamental loss between the input sample's output data and the output sample; Obtain the output perturbation corresponding to the output sample; Obtain the adversarial loss between the output data of the input sample and the output perturbation; Based on the basic loss and the adversarial loss, the model loss is determined; The model parameters of the anomaly detection model are updated based on the model loss.

19. The apparatus according to claim 17, characterized in that, The model convergence condition includes: the model loss is less than or equal to a preset threshold; If the adversarial pre-training is based on output perturbation, the model loss is used to characterize the degree of difference between the output sample and the output perturbation; Alternatively, if the adversarial pre-training is based on output perturbation, the model loss is related to the perturbation strength and the degree of output difference, wherein the degree of output difference is the degree of difference between the output data of the input sample and the output data of the input perturbation; and the perturbation strength is the degree of difference between the input sample and the input perturbation.

20. The apparatus according to claim 15, characterized in that, The encoding unit is specifically used to: encode the word vector and the position encoding using a transformation encoder to obtain a log semantic vector; The transformation encoder includes at least one transformation coding layer; each transformation coding layer includes a multi-head attention layer, a first connection layer, a feedforward network layer, and a second connection layer connected in sequence. The multi-head attention layer is used to learn the correlation between word vectors based on a multi-self attention mechanism; The first connection layer is used to perform residual connection and normalization processing on the output data of the multi-head attention layer; The feedforward network layer is used to perform fully connected computations; The second connection layer is used to perform residual connection and normalization processing on the output data of the feedforward network layer.

21. The apparatus according to claim 15, characterized in that, The second acquisition unit is specifically used for: The log data is standardized to obtain standardized text; The standardized text is processed using a pre-trained word segmentation model to obtain the word vectors; wherein the word segmentation model is used to segment the standardized text according to the segmentation method with the highest segmentation probability.

22. The apparatus according to claim 21, characterized in that, The second acquisition unit is further configured to: The word segmentation model is pre-trained until it converges; wherein the convergence condition includes: the word loss value corresponding to the word segmentation method with the highest probability is the smallest.

23. The apparatus according to claim 15, characterized in that, The first acquisition unit is specifically used for: Use big data log collection tools to listen for and acquire real-time log data; The real-time log data is consumed using a distributed message queue, so that the consumed real-time log data is used as the log data to be detected for anomaly detection.

24. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-13.

25. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-13.

26. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-13.

Images