Abnormity detection method, device, equipment, medium and product

Abstract

The invention discloses an anomaly detection method and device, equipment, a medium and a product, and the method comprises the steps: obtaining initial log data, and carrying out the analysis processing of the initial log data, and obtaining intermediate log data; performing data conversion processing and clustering processing on the intermediate log data to obtain target log data; and inputting the target log data into a completely trained anomaly detection model to obtain data anomaly information. According to the technical scheme, the anomaly detection efficiency and accuracy of the log data in the power database system are improved, and the safety and stability of the power database system are effectively enhanced.

Description

Technical Field

[0001] The present invention relates to the field of anomaly detection technology, and in particular to an anomaly detection method, device, equipment, medium and product. Background Art

[0002] There are a lot of log data in the power database system that have not been fully analyzed. These data contain key information, such as operation type, timestamp, user information, affected database objects, details of the executed operation, and the returned result status, etc. This information is crucial for understanding the operating status of the database system, user behavior patterns, and potential security threats.

[0003] In order to ensure the safe and stable operation of the power database system, it is necessary to conduct in-depth analysis of the log data in the system in order to promptly discover and respond to potential abnormal behaviors. Traditional manual analysis and troubleshooting methods not only have the problem of low analysis and troubleshooting efficiency, but also have difficulty in ensuring the comprehensiveness and accuracy of the analysis. Therefore, an automated anomaly detection method is urgently needed. Summary of the invention

[0004] The present invention provides an anomaly detection method, device, equipment, medium and product, which improve the anomaly detection efficiency and accuracy of log data in an electric power database system, and effectively enhance the security and stability of the electric power database system.

[0005] In a first aspect, an embodiment of the present disclosure provides an anomaly detection method, comprising:

[0006] Acquire initial log data, and parse and process the initial log data to obtain intermediate log data;

[0007] Performing data conversion processing and clustering processing on the intermediate log data to obtain target log data;

[0008] The target log data is input into a well-trained anomaly detection model to obtain data anomaly information.

[0009] In a second aspect, an embodiment of the present disclosure provides an abnormality detection device, including:

[0010] A data acquisition module is used to acquire initial log data and parse and process the initial log data to obtain intermediate log data;

[0011] A data processing module, used for performing data conversion processing and clustering processing on the intermediate log data to obtain target log data;

[0012] The anomaly detection module is used to input the target log data into a well-trained anomaly detection model to obtain data anomaly information.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute an abnormality detection method provided by the above-mentioned first aspect embodiment.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement an anomaly detection method provided in the embodiment of the first aspect above when executed.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements an anomaly detection method provided in the embodiment of the first aspect above.

[0019] An anomaly detection method, device, equipment, medium and product of an embodiment of the present invention include obtaining initial log data, parsing the initial log data to obtain intermediate log data; performing data conversion and clustering on the intermediate log data to obtain target log data; inputting the target log data into a well-trained anomaly detection model to obtain data anomaly information. The above technical solution improves the efficiency and accuracy of anomaly detection of log data in the power database system.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of an anomaly detection method provided by Embodiment 1 of the present invention;

[0023] Figure 2is a flow chart of an anomaly detection method provided by Embodiment 2 of the present invention;

[0024] Figure 3 This is an example diagram of initial log data parsing in an anomaly detection method provided in Embodiment 2 of the present invention;

[0025] Figure 4 This is an example diagram of the formation of a target log vector in an anomaly detection method provided in the second embodiment of the present invention;

[0026] Figure 5 is a structural schematic diagram of an abnormality detection device provided in Embodiment 3 of the present invention;

[0027] Figure 6 It is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second" and "target" in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] In order to ensure the safe and stable operation of the power database system, the log data in the system must be deeply analyzed in order to promptly detect and respond to potential abnormal behaviors. Due to the complexity of log data, including multiple formats and semantic content, an efficient model is needed to accurately extract and parse this information. In particular, there may be multiple languages ​​in the log data, such as a mixture of Chinese and English, as well as multiple data formats, which requires the model to have strong semantic understanding and format adaptability. In addition, the volume of log data is usually very large, and tens of thousands of log records may be generated every day. These records need to be monitored and analyzed in real time so that immediate action can be taken when anomalies occur.

[0031] Therefore, an automated anomaly detection method is needed that can quickly process large amounts of data and accurately identify abnormal patterns. This requires the model to have not only efficient computing capabilities, but also strong learning capabilities to learn and infer potential abnormal patterns from limited data. The model must be able to process and parse various complex log formats, extract key information, and convert it into a format that can be analyzed. At the same time, the model also needs to be able to understand the context and semantics of the log entries in order to accurately identify abnormal behaviors that deviate from normal behavior patterns.

[0032] Therefore, in the prior art, supervised learning methods are used for anomaly detection. Supervised learning methods perform well in log anomaly detection, but they rely on a large amount of manually annotated training data. The annotation process is time-consuming and costly, which limits the feasibility of the model in practical applications. In addition, the existing methods may be affected by inaccurate or inconsistent annotated data. If there are errors or deviations in the annotation process, the patterns learned by the model will also be inaccurate, which will directly lead to a decrease in detection performance. Unsupervised and semi-supervised methods reduce the model's dependence on manually annotated information. However, the existing unsupervised and semi-supervised methods only use traditional machine learning methods of non-sequential models, and do not use historical anomalies to improve their detection capabilities, resulting in a decrease in recognition accuracy. In addition, most of the existing general log processing methods do not pay attention to self-assigned information, but only process the identified log templates. Good results can be achieved in processing ordinary logs. However, in the scenario of processing highly professional logs, such methods are prone to lose a large amount of professional field information contained in the self-assigned information, and only process a large number of repeated templates, resulting in a decrease in recognition effect.

[0033] The anomaly detection method provided by the present invention aims to solve the limitations of existing log anomaly detection methods, especially in reducing the reliance on manually labeled data, improving detection accuracy, and making full use of self-assigned information. Based on the characteristics of business scenario logs, a domain-based analysis and time series model is proposed to reduce the reliance on manually labeled data, while improving the accuracy, efficiency and robustness of anomaly detection, providing a more reliable and efficient log anomaly detection solution for the power database system.

[0034] Embodiment 1

[0035] Figure 1 This is a flowchart of an anomaly detection method provided in Example 1 of the present invention. This embodiment is applicable to the situation where anomaly detection is performed on log data. The method can be executed by an anomaly detection device, and the anomaly detection device can be implemented in the form of hardware and / or software.

[0036] like Figure 1 As shown, the method includes:

[0037] S101, obtaining initial log data, and parsing the initial log data to obtain intermediate log data.

[0038] In this embodiment, the initial log data can be understood as non-redundant log data obtained from the power grid system after data cleaning. The initial log data is at least one of the operation type, operation time, operation user, operation result, query statement, platform and risk level. The intermediate log data can be understood as the data obtained after parsing and splitting the initial log data, including at least template data and self-assigned data. The format of the intermediate log data is incompatible with the anomaly detection model and cannot be directly used as the input of the anomaly detection model.

[0039] Specifically, multiple log entries are obtained from the power database system, such as initial log data such as operation type, operation time, operation user, operation result, query statement, platform and risk level, as well as related abnormal label data. For each initial log data, log analysis is performed on the initial log data based on the Drain log parsing model to obtain the template data and self-assigned data of the initial log data, that is, the intermediate log data.

[0040] S102: Perform data conversion and clustering processing on the intermediate log data to obtain target log data.

[0041] In this embodiment, the target log data can be understood as the intermediate log data after optimization processing, and its data format can be directly input into the anomaly detection model. The target log data at least includes a log vector and label data corresponding to the vector.

[0042] Specifically, for each intermediate log data, the template data and self-assigned data in the intermediate log data are converted into corresponding vector data respectively. After the vector conversion is completed, the template vector of the template data and the self-assigned vector of the self-assigned data are concatenated based on the self-attention mechanism to form a complete log vector. And the log vectors are clustered by the abnormal label data related to the initial log data to obtain the label data of each log vector, and then the target log data is obtained.

[0043] S103: Input the target log data into a well-trained anomaly detection model to obtain data anomaly information.

[0044] In this embodiment, the anomaly detection model can be understood as a model for detecting whether log data has anomalies, which is a two-layer bidirectional gated recurrent unit neural network (Gated Recurrent Unit, GRU). Data anomaly information can be understood as information characterizing the anomalies existing in log data.

[0045] Specifically, a two-layer bidirectional gated recurrent unit neural network is pre-established and trained to obtain a fully trained anomaly detection model, and the target log data is input into the anomaly detection model as input information to obtain data anomaly information output by the model. The two-layer bidirectional gated recurrent unit partially alleviates the problems of long-term dependence and gradient disappearance / explosion by adding parameters and changing the structure, which can improve the security and stability of the database in the business system.

[0046] If the data anomaly information indicates that the target log data is abnormal, an alarm message is generated and sent to the associated user end or uploaded to the corresponding server end.

[0047] An anomaly detection method provided by an embodiment of the present invention includes obtaining initial log data, parsing the initial log data to obtain intermediate log data; performing data conversion and clustering processing on the intermediate log data to obtain target log data; and inputting the target log data into a well-trained anomaly detection model to obtain data anomaly information. The above technical solution improves the efficiency and accuracy of anomaly detection of log data in the power database system, and effectively enhances the security and stability of the power database system.

[0048] As a first optional embodiment of this embodiment, it also includes:

[0049] A neural network model is established and trained to obtain a fully trained anomaly detection model, wherein the anomaly detection model at least includes an update gate and a reset gate.

[0050] In this embodiment, in order to capture the characteristics of the log sequence (target log data) and extract potential information, an anomaly detection model implemented by a two-layer bidirectional GRU is established. GRU is a variant of a recurrent neural network (RNN) for processing sequence data, which aims to solve the gradient vanishing and gradient explosion problems that are prone to occur when traditional RNN is trained on long sequence data, and improves the learning efficiency and performance of the model on long sequences. The two-layer bidirectional GRU model provided in this embodiment introduces an update gate and a reset gate to control the flow of information. The update gate determines how the hidden state should be updated. It is a value between 0 and 1. A value close to 1 indicates that most of the hidden state of the previous time step is retained, and a value close to 0 indicates that most of the information of the current time step is used; the reset gate determines the degree of influence of the previous hidden state on the current time step. It is also a value between 0 and 1. A value close to 1 indicates that the previous state has a greater influence on the current state, and a value close to 0 indicates that the previous state is ignored.

[0051] Due to the gating mechanism of GRU, the gradient can be transferred between time steps more effectively, thus reducing the problem of gradient vanishing. Compared with the Long Short-Term Memory (LSTM), GRU has fewer parameters because it does not have the cell state in LSTM, but only the hidden state, which makes GRU more concise in terms of parameter quantity. Through the gating mechanism, GRU can effectively avoid the problem of gradient vanishing, allowing the model to learn long-distance dependencies.

[0052] Specifically, the parameters of GRU are as follows:

[0053] z t =σ(W z x t +U z h t-1 +b z ),r t =σ(W r x t +U r h t-1 +b r )

[0054]

[0055] Among them, x t represents the input of the model, h t represents the state that can be used for downstream tasks, that is, the label data corresponding to the log data. σ represents the sigmoid activation function, ⊙ represents element-by-element multiplication, and w z 、w r 、w h 、uz 、u r 、u h is the trainable matrix, b z , b r , b h is the bias matrix. Compared with the traditional recurrent neural network (RNN), GRU introduces an update gate z t To control how much information from the previous state should be retained and how much new information from the candidate state should be accepted. A reset gate r is also introduced t To control whether the calculation of the candidate state depends on the previous state h t-1 Its input includes log vectors and label data corresponding to the log vectors to form supervised training. In essence, it is crucial to identify truly abnormal log sequences, because normal sequences are usually composed of noise. To solve this problem, this embodiment adds a layer of weights after the conventional GRU, and at each time step, the hidden state h i Assign weights. Specifically, the importance of each step is calculated as follows, where μ represents the trainable weight at time step t:

[0056]

[0057] The final prediction output P(normal, anomalous) is given by the following formula, where W n is the trainable weight matrix, for A concrete representation of time.

[0058]

[0059] Embodiment 2

[0060] Figure 2 It is a flowchart of an anomaly detection method provided in the second embodiment of the present invention. This embodiment is a further optimization of any of the above embodiments and can be applied to the situation of performing anomaly detection on log data. The method can be executed by an anomaly detection device, and the anomaly detection device can be implemented in the form of hardware and / or software.

[0061] like Figure 2 As shown, the method includes:

[0062] S201. Obtain power log data, perform data cleaning on the power log data, and obtain initial log data and abnormal label data.

[0063] In this embodiment, the power log data can be understood as the audit log directly obtained from the power database system, including log entries such as operation type, operation time, operation user, operation result, query statement, platform, and risk level. Abnormal label data can be understood as labels that characterize abnormalities in the power log data.

[0064] Specifically, the existing database audit logs in the power grid business data, that is, the power log data, were exported through the database. Through certain data cleaning, the highly repeated descriptive information in multiple power log data in the entire table was removed to obtain the cleaned initial log data. The initial log data is complete power log data or power log data with some fields cleaned out. Abnormal label data is added to the initial log data with some fields cleaned out.

[0065] Exemplarily, 30,667 log entries were sorted out, of which about 6.95% were abnormal, which is consistent with the usual proportion of abnormalities in the general log field.

[0066] S202: Perform data separation processing on the initial log data to obtain template data and self-assigned data, and use the template data and self-assigned data as intermediate log data.

[0067] In this embodiment, the template data can be understood as the template part of the initial log data, which is a fixed field in the data. The self-assigned data can be understood as the customizable variable part of the initial log data. For example, the name is a, where the name is the template data and a is the self-assigned data.

[0068] Specifically, the initial log data is separated and processed through the Drain log parsing model to determine the template data in the initial log data, and the self-assigned data is determined based on the initial log data as a whole and the template data, and finally the template data and the self-assigned data are used as intermediate log data. Among them, the Drain log parsing model is a stream-based log parsing framework that can parse and process large-scale log data streams in real time, using a fixed-length tree model. It can convert unstructured log information into structured log information, and automatically process log data to improve the speed and accuracy of parsing. The core idea of ​​Drain is to decompose the log parsing process into multiple stages, each stage is responsible for processing different parts of the log data, so as to achieve efficient data stream processing.

[0069] Exemplarily, the Drain algorithm includes: using a preset regular expression to remove self-assigned information in log entries; classifying by the length of the log after preprocessing, assuming that log entries with the same log event may have the same length; classifying by the first several words of the log. There is a hyperparameter depth here to limit the depth of the Drain tree. It is assumed that the word at the beginning of the log message is more likely to be a constant. If the currently judged word is a number, it is assigned to the wildcard node. The reason is that the number of words in the log is limited, and the number of variable parts that cannot be removed in preprocessing such as numeric fields is inestimable; classifying log entries into leaf nodes. After completing the above steps, the log entries are classified into different log templates with the same beginning to form a library. In order to ensure that log entries with similar semantics are assigned to the same template, the similarity with all existing templates is calculated by comparing words bit by bit. Finally, Drain assigns all log entries to a corresponding template, which can represent all logs in its library. The Drain algorithm can be regarded as a classification tree, which divides logs into different groups one by one according to different standards.

[0070] Optionally, the initial log data is processed for data separation to obtain template data and self-assigned data, including:

[0071] S2021. Determine the data length of the initial log data, and determine the log start word corresponding to the data length.

[0072] In this embodiment, the log start word generally refers to a specific word or phrase in the log used to mark the beginning of a log record, such as tftp, warning, etc.

[0073] Specifically, multiple initial log data are split to obtain a single initial log data. Since log entries with the same log event may have the same length when classified according to the first few words of the log, the data length length:x is determined for each initial log data. Each data length has a corresponding log starting word, and the log starting word corresponding to the initial log data is determined at the data length corresponding to the initial log data.

[0074] For example, Figure 3 is an example diagram of an anomaly detection method provided in Embodiment 2 of the present invention involving initial log data parsing, such as Figure 3As shown, the set consisting of multiple initial log data is split into single initial log data, and the data length of each initial log data is determined. The initial log data is classified according to the data length. If the data length is 7, it is divided into the range of data length 7, and further classified based on the log starting word. Under the classification of data length 7, if the log starting word of the initial log data is warning, the initial log data is divided into the classification of the log starting word warning. Among several initial log data, for example, the initial log data numbered 1, 2, 5...29... are divided into the template of length:7-warning, forming a corresponding leaf node set. After the above classification tree process, the log entries under the same startup are classified into different log templates.

[0075] It is understandable that since the number of words in the log is limited, the variable part that cannot be deleted in the preprocessing (not the self-assigned information), such as the number of numeric fields, cannot be calculated. Therefore, if it is determined that the word in the initial log data is a number, it is assigned to the wildcard node.

[0076] S2022: Determine a log template corresponding to the log start word, match the semantic data of the initial log data with the log template, and determine the template data of the initial log data.

[0077] In this embodiment, the log template can be understood as a predefined log template having multiple template types, each template type corresponding to a word combination. The semantic data of the initial log data can be understood as the word combination of the initial log data.

[0078] Specifically, each log starting word has its corresponding multiple log templates, which are combinations of different words in different orders. In order to ensure that semantically similar log entries are assigned to the same template, words are compared bit by bit to calculate the similarity with all existing templates. Therefore, after determining the log starting word corresponding to the initial log data, the semantic data of the initial log data is compared with the multiple log templates corresponding to the log starting word to determine the similarity between the semantic data and the log template. If the similarity exceeds a set similarity threshold, the log template is determined as the template data of the initial log data. If the similarity does not exceed the set similarity threshold, a new template corresponding to the semantic data is created for the initial log data.

[0079] S2023. Determine the portion of the initial log data excluding the template data as self-assigned data.

[0080] In this embodiment, since the initial log data is formed by the combination of the template part and the self-assigned part, after the template data is determined through the above steps, the template data is stripped from the initial log data, and the remaining part of the initial log data is the self-assigned data. In addition to user-defined data, the self-assigned data also includes keywords as syntactic connections, and natural language segments (such as log descriptions and comments).

[0081] Through the above log-driven template parsing process, we can obtain log template parsing results related to actual application scenarios. Separating template information and self-assigned information in logs helps to design a model that can distinguish data with different information densities.

[0082] S203 , converting the template data and the self-assigned data in the intermediate log data respectively to obtain corresponding template vectors and self-assigned vectors.

[0083] In this embodiment, the template vector can be understood as template data after vector conversion, and the self-assigned vector can be understood as self-assigned data after vector conversion.

[0084] Specifically, the template data is used as input, and the template vector corresponding to the template data is generated through the pre-trained GloVe (Global Vectors for Word Representation) word embedding technology. The self-assigned data is used as input, and the self-assigned vector corresponding to the self-assigned data is generated through the CodeBERT model, a pre-trained model based on the Transformer architecture.

[0085] S204: Concatenate the template vector and the self-assigned vector to obtain a target log vector.

[0086] In this embodiment, Figure 4 is an example diagram of the formation of a target log vector in an anomaly detection method provided in the second embodiment of the present invention, such as Figure 4 As shown in the figure, after the initial log data is split and transformed into a template vector and a self-assigned vector, the template vector and the self-assigned vector are concatenated through a multi-head attention mechanism to obtain the final target log vector.

[0087] Specifically, we introduce a parallel multi-head attention mechanism and design a log encoding adapter with a parallel structure. For multiple log messages that are close in time, we use the term frequency-inverse document frequency (TF-IDF) as the weighting basis to concatenate the template vector and the self-assigned vector. The adapter uses the scaled dot product attention formula The final target log vector is calculated. In the formula, Softmax is a normalized exponential function, h represents the number of attention heads, d represents the input dimension of the self-assigned data, and K T is the matrix transpose, φ p is the template vector.

[0088] Specifically, Figure 4 As shown in the figure, the template vector is input into the log encoding adapter, and the self-assigned vector is input into the log encoding adapter based on the query V, key K and value Q dimensions. The log encoding adapter performs corresponding linear transformation on the self-assigned vector of each dimension. The content of the self-assigned vector after linear transformation is processed by the scaled dot-product attention mechanism ScaledDot-Product Attention to achieve further weighted processing of the self-assigned vector. The adapter runs in parallel with the log attention layer and the feedforward layer, which enhances the generalization ability of the model in different log sources. The processed self-assigned vector and the template vector are spliced, and the spliced ​​vector data is processed by the normalized exponential function SoftMax to obtain the final target log vector.

[0089] The log encoding adapter based on the parallel multi-head attention mechanism adopted in this embodiment utilizes the template information and self-assigned information ignored by the log pattern analysis, enhances the representation ability of log sequence modeling, and reduces the information loss usually caused by log template analysis. The target log data formed can be directly applied to the anomaly detection model, thereby improving the query detection efficiency.

[0090] S205 . Based on the abnormal label data corresponding to the initial log data, clustering processing is performed on each target log vector to obtain vector label data of each target log vector.

[0091] In this embodiment, the vector label data may be understood as information used to describe whether an abnormality exists in the log vector.

[0092] Specifically, the anomaly detection task involves semi-supervised learning, and a large number of unlabeled sequences are mixed with normal and abnormal samples, which makes it impossible to clearly classify normal and abnormal samples of a specific log block. To overcome this challenge, this embodiment adopts a compromise approach to assign a floating-point probability number to each log block, indicating that the block does not completely belong to an abnormal sequence or a normal sequence.

[0093] Specifically, for the target log data corresponding to the initial log data with abnormal label data, a probability is directly assigned based on the abnormal label data, for example, 1, and these labels are propagated to the unlabeled data through clustering, so that each target log data has its corresponding floating point probability number.

[0094] Specifically, through the Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN), based on d mreach-k (a,b)=max{core k (a),core k (b), dist(a,b)} organizes multiple target log data into a hierarchical tree structure according to time series to identify the clustering structure in the data. In the formula, dmreach-k is defined as the mutual reachability distance, and core k is the core distance, which represents the vector distance between the labeled data k and the unlabeled data a and b, and dist is the vector distance between the two labels. Using the reachable distance as the metric, the sample space of the log is regarded as a fully connected graph, where the edge weight is the reachable distance between two points; all edges in the graph are sorted in ascending order, and the sequence is traversed to gradually connect the endpoints of the edges, eventually forming a hierarchical binary tree; starting from the root node, outliers in the coarse cluster are removed, and overly large clusters are divided into several smaller clusters. Floating-point probability number allocation is implemented for each target log data.

[0095] Among them, when traversing clusters, the evaluation criterion λ is used to measure whether the cluster merger is a cluster. If λ is less than the set threshold, the stability is weaker, and the cluster is separated. If λ is greater than the set threshold, the stability is stronger, and the cluster is aggregated. The calculation method of λ is as follows:

[0096]

[0097] Among them, d_(mreach-k) is the distance metric calculated after considering the factor of mreach; mreach represents a variable related to the clustering effect or process and can be customized; k represents the number of clusters.

[0098] Defining λ birth is the λ value when the current node is split, λ death is the λ value when the current node is split into two child nodes, λ p is the λ value when each node under the current node is separated. p =λ death When , two child nodes are formed; when λ p ∈[λ birth ,λ death ], outliers are separated. The stability of a cluster is defined as follows:

[0099]

[0100] The leaf nodes are considered as clusters, and the entire tree is traversed from bottom to top. If the stability of the current node is less than the sum of the stabilities of its child nodes, the stability of the node is set to the sum of the stabilities of its child nodes, otherwise the current node is identified as a cluster. When the traversal reaches the root node, the process ends and the resulting cluster is used as the clustering result. Log blocks belonging to the same cluster are merged into the same marked log data.

[0101] The mutual reach distance is used to weigh the distribution of probability floating point numbers to estimate the anomaly probability of unlabeled data. This process not only provides the encoding of log information, but also enriches the relevant data of anomalies, which is conducive to training anomaly detection models.

[0102] S206: Determine the target log vector and the vector label data as target log data.

[0103] In this embodiment, after the aggregation processing in the above steps, each target log vector has its corresponding vector label data, and the two are associated, so the target log vector and its corresponding vector label data are determined as the target log data.

[0104] S207: Input the target log data into a well-trained anomaly detection model to obtain data anomaly information.

[0105] An anomaly detection method provided by an embodiment of the present invention includes acquiring power log data, performing data cleaning on the power log data to obtain initial log data and abnormal label data; performing data separation processing on the initial log data to obtain template data and self-assigned data, and using the template data and self-assigned data as intermediate log data; performing conversion processing on the template data and self-assigned data in the intermediate log data respectively to obtain corresponding template vectors and self-assigned vectors; performing splicing processing on the template vectors and the self-assigned vectors to obtain target log vectors; performing clustering processing on each target log vector based on the abnormal label data corresponding to the initial log data to obtain vector label data of each target log vector; determining the target log vector and the vector label data as target log data; and inputting the target log data into a well-trained anomaly detection model to obtain data anomaly information. The above technical scheme not only makes full use of the structured information of the log template, but also conducts in-depth semantic analysis of the self-assigned information in the log entry, significantly improving the accuracy and efficiency of anomaly detection; by combining template analysis and semantic representation, the scheme provides a template-parsed log anomaly detection method for electrical database systems (Template-Parsed Log Anomaly Detection for Electrical Database Systems, TPLAD), which can automatically detect abnormal patterns including runtime errors, unauthorized access and data leakage, thereby providing a strong guarantee for the stability and security of the database system; based on the domain-specific analysis method, the self-assigned parameter information is incorporated into the learning and inference process, which is equivalent to incorporating specific and professional content instead of just using a general template, so that the abnormal information can be more accurately identified, and the precision of anomaly detection is improved; the ability to identify anomalies is improved, and the recall rate of anomaly detection is improved; at the same time, the self-assigned information and time series model that represent the professional information in the field are used, which not only pays attention to the domain knowledge unique to the business scenario, but also pays attention to the log context where the anomaly is located, so that the anomaly can be accurately determined, the F1 score of anomaly detection is improved, and the accuracy of anomaly detection for power log data is comprehensively improved.

[0106] Embodiment 3

[0107] Figure 5 Schematic diagram of the structure of an abnormality detection device provided by Embodiment 3 of the present invention. Figure 5 As shown, the device comprises:

[0108] The data acquisition module 31 is used to acquire initial log data and parse the initial log data to obtain intermediate log data;

[0109] The data processing module 32 is used to perform data conversion processing and clustering processing on the intermediate log data to obtain target log data;

[0110] The anomaly detection module 33 is used to input the target log data into a well-trained anomaly detection model to obtain data anomaly information.

[0111] The anomaly detection device adopted in the technical solution improves the efficiency and accuracy of anomaly detection of log data in the power database system, and effectively enhances the security and stability of the power database system.

[0112] Optionally, the data acquisition module 31 includes:

[0113] A data cleaning unit, used to obtain power log data, perform data cleaning on the power log data, and obtain initial log data and abnormal label data;

[0114] The log separation unit is used to perform data separation processing on the initial log data to obtain template data and self-assigned data, and use the template data and self-assigned data as intermediate log data.

[0115] Optional, log separation unit, specifically used for:

[0116] Determine the data length of the initial log data, and determine the log start word corresponding to the data length;

[0117] Determine a log template corresponding to the log start word, match the semantic data of the initial log data with the log template, and determine the template data of the initial log data;

[0118] The portion of the initial log data excluding the template data is determined as self-assigned data.

[0119] Optionally, the data processing module 32 is specifically used for:

[0120] Respectively converting the template data and the self-assigned data in the intermediate log data to obtain corresponding template vectors and self-assigned vectors;

[0121] Concatenate the template vector and the self-assigned vector to obtain a target log vector;

[0122] Based on the abnormal label data corresponding to the initial log data, clustering processing is performed on each of the target log vectors to obtain vector label data of each of the target log vectors;

[0123] The target log vector and the vector label data are determined as target log data.

[0124] Optionally, the device further comprises a model building module, which is used to:

[0125] A neural network model is established and the neural network model is trained to obtain a fully trained anomaly detection model, wherein the anomaly detection model at least includes an update gate and a reset gate.

[0126] Optionally, the initial log data is at least one of an operation type, an operation time, an operation user, an operation result, a query statement, a platform, and a risk level.

[0127] The anomaly detection device provided in the embodiment of the present invention can execute the anomaly detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0128] Embodiment 4

[0129] Figure 6 : is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0130] like Figure 6 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0131] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0132] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as an anomaly detection method.

[0133] In some embodiments, the anomaly detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the anomaly detection method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the anomaly detection method in any other suitable manner (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0139] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0140] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0141] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An anomaly detection method, characterized in that: include: Acquire initial log data, and parse and process the initial log data to obtain intermediate log data; Performing data conversion processing and clustering processing on the intermediate log data to obtain target log data; The target log data is input into a well-trained anomaly detection model to obtain data anomaly information.

2. The method according to claim 1, characterized in that: The obtaining of initial log data and parsing of the initial log data to obtain intermediate log data includes: Acquire power log data, perform data cleaning on the power log data, and obtain initial log data and abnormal label data; The initial log data is subjected to data separation processing to obtain template data and self-assigned data, and the template data and self-assigned data are used as intermediate log data.

3. The method according to claim 2, characterized in that The data separation process is performed on the initial log data to obtain template data and self-assigned data, including: Determine the data length of the initial log data, and determine the log start word corresponding to the data length; Determine a log template corresponding to the log start word, match the semantic data of the initial log data with the log template, and determine the template data of the initial log data; The portion of the initial log data excluding the template data is determined as self-assigned data.

4. The method according to claim 1, characterized in that: The performing data conversion processing and clustering processing on the intermediate log data to obtain target log data includes: Respectively converting the template data and the self-assigned data in the intermediate log data to obtain corresponding template vectors and self-assigned vectors; Concatenate the template vector and the self-assigned vector to obtain a target log vector; Based on the abnormal label data corresponding to the initial log data, clustering processing is performed on each of the target log vectors to obtain vector label data of each of the target log vectors; The target log vector and the vector label data are determined as target log data.

5. The method according to claim 1, characterized in that Also includes: A neural network model is established and the neural network model is trained to obtain a fully trained anomaly detection model, wherein the anomaly detection model at least includes an update gate and a reset gate.

6. The method according to claim 1, characterized in that The initial log data is at least one of an operation type, an operation time, an operation user, an operation result, a query statement, a platform, and a risk level.

7. An abnormality detection device, characterized in that: include: A data acquisition module is used to acquire initial log data and parse and process the initial log data to obtain intermediate log data; A data processing module, used for performing data conversion processing and clustering processing on the intermediate log data to obtain target log data; The anomaly detection module is used to input the target log data into a well-trained anomaly detection model to obtain data anomaly information.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute an abnormality detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement an abnormality detection method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements an anomaly detection method according to any one of claims 1 to 6.

Images