Abnormality detection system and method, electronic equipment and storage medium

Abstract

The invention provides an exception detection system and method, electronic equipment and a storage medium, and relates to the field of artificial intelligence, the exception detection system comprises a plurality of detection modules, different exception detection operations are preset for different detection modules, the detection modules can be used for detecting operation logs of a target system from different dimensions, and the detection efficiency is improved. And obtaining detection identification data generated by different detection modules. Besides, the anomaly detection system further comprises a model detection module, detection identification data generated by each detection module can form an identification sequence, anomaly detection is carried out on the identification sequence by utilizing a pre-training neural network model, and a corresponding overall anomaly value of the target system in a preset time period is obtained. Therefore, the pre-trained neural network model can be utilized to carry out comprehensive detection on the detection results of different detection modules to obtain a more comprehensive and reliable comprehensive detection result, so that the method can better adapt to multi-dimensional anomalies in a high-frequency and complex environment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an anomaly detection system, method, electronic device and storage medium. Background Art

[0002] In modern large-scale computing environments, the large amount of logs generated by the system provide basic data for fault detection and system monitoring. However, traditional rule-based log analysis methods can only provide limited detection capabilities and cannot cope with multi-dimensional anomalies in high-frequency and complex environments.

[0003] In view of this, how to provide a more efficient and accurate log anomaly detection system is a technical problem that technical personnel in this field need to solve. Summary of the invention

[0004] The purpose of the present invention is to provide an anomaly detection system, method, electronic device and storage medium, which can adopt different anomaly detection operations to detect the operation log of the target system, and can further use a pre-trained neural network model to perform comprehensive detection on the detection results obtained by different anomaly detection operations, so as to obtain more comprehensive and reliable comprehensive detection results, thereby being more adaptable to multi-dimensional anomalies in high-frequency and complex environments.

[0005] In order to solve the above technical problems, the present invention provides an anomaly detection system, comprising:

[0006] Multiple detection modules, different detection modules are preset with different anomaly detection operations, and are used to perform the anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, convert the anomaly value into a discrete anomaly value, and add an identifier corresponding to the anomaly detection operation to the discrete anomaly value to obtain detection identifier data;

[0007] A model detection module is used to form an identification sequence by combining the detection identification data, and to perform anomaly detection on the identification sequence using a pre-trained neural network model to obtain an overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal based on the overall abnormality value; wherein the pre-trained neural network model has learned a preset baseline identification sequence, and the overall abnormality value represents the degree to which the identification sequence deviates from the baseline identification sequence.

[0008] Optionally, the detection module includes:

[0009] A log sequence detection module, configured to convert the operation log into a log sequence; perform anomaly detection on the log sequence using a pre-trained log sequence detection model to obtain a first anomaly value; wherein the pre-trained log sequence detection model has learned a preset normal log sequence, and the first anomaly value represents the degree to which the log sequence deviates from the normal log sequence;

[0010] A log category detection module is used to determine the preset log category corresponding to the operation log, and to count the occurrence frequency corresponding to each preset log category within the preset time period according to the operation log, to obtain the log category occurrence frequency distribution data corresponding to the current detection moment; to detect the degree of change between the log category occurrence frequency distribution data at the current detection moment and the log category occurrence frequency distribution data at the previous historical detection moment, to obtain a second abnormal value;

[0011] The log structure detection module is used to update the operation log to a log parsing tree; detect the degree of change between the log parsing tree at the current detection moment and the log parsing tree at the previous historical detection moment, and obtain a third abnormal value.

[0012] Optionally, the detection module includes:

[0013] A numerical histogram detection module is used to extract numerical values ​​from the operation log and convert the numerical values ​​into a histogram; detect the degree of change between the histogram at the current detection moment and the histogram at the previous historical detection moment to obtain a fourth abnormal value;

[0014] A numerical line graph detection module is used to extract numerical values ​​in the operation log and fit the numerical values ​​into a line graph; determine an outlier determination numerical interval according to the line graph; and obtain a fifth abnormal value according to the proportion of numerical values ​​deviating from the outlier determination numerical interval in all numerical values;

[0015] A numerical clustering detection module is used to extract numerical values ​​from the operation log; perform clustering processing on the numerical values ​​to obtain numerical clusters; calculate the deviation of the numerical values ​​not in the numerical cluster compared to the numerical cluster to obtain a sixth abnormal value;

[0016] A discrete value detection module is used to extract discrete variable values ​​from the operation log; count the occurrence frequencies of each value corresponding to the discrete variable value according to the discrete variable value, and obtain the discrete value occurrence frequency distribution data corresponding to the current detection moment; detect the degree of change between the discrete value occurrence frequency distribution data at the current detection moment and the discrete value occurrence frequency distribution data at the previous historical detection moment, and obtain the seventh abnormal value.

[0017] Optionally, the detection module is further used to:

[0018] Performing normalization processing on the outliers;

[0019] The normalized outliers are discretized to obtain discrete outliers.

[0020] Optionally, it also includes:

[0021] A log conversion module, used to convert the operation logs generated by the target system within a preset time period into log identification data;

[0022] The model detection module is further used to combine the detection identification data and the log identification data into an identification sequence.

[0023] Optionally, the model detection module is further used to:

[0024] Performing masking processing on part of the identification data in the identification sequence to obtain a to-be-processed identification sequence including masked identification data and uncovered identification data;

[0025] Inputting the identification sequence to be processed into the pre-trained neural network model, so that the pre-trained neural network model predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed, to obtain predicted identification data;

[0026] A preset loss function is used to calculate a loss value between the predicted identification data and the covered identification data in the identification sequence to be processed, and the loss value is used as the overall abnormal value.

[0027] Optionally, the model detection module is further used to:

[0028] Acquire the baseline identification sequence;

[0029] Performing masking processing on part of the identification data in the baseline identification sequence to obtain a training identification sequence including masked identification data and uncovered identification data;

[0030] Inputting the training identification sequence into an initial neural network model, so that the initial neural network model predicts the covered identification data in the training identification sequence according to the uncovered identification data in the training identification sequence, to obtain identification data to be compared;

[0031] The preset loss function is used to calculate the loss value between the identification data to be compared and the covered identification data in the training identification sequence, and the parameters of the initial neural network model are updated according to the loss value to obtain the pre-trained neural network model.

[0032] The present invention also provides an anomaly detection method, which is applied to the anomaly detection system as described above, and the method comprises:

[0033] The detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, converts the anomaly value into a discrete anomaly value, and adds an identifier corresponding to the anomaly detection operation to the discrete anomaly value to obtain detection identifier data; wherein different detection modules preset different anomaly detection operations;

[0034] The model detection module organizes the detection identification data into an identification sequence, and uses a pre-trained neural network model to perform anomaly detection on the identification sequence to obtain an overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal based on the overall abnormality value; wherein the pre-trained neural network model has learned a preset baseline identification sequence, and the overall abnormality value represents the degree to which the identification sequence deviates from the baseline identification sequence.

[0035] The present invention also provides an electronic device, comprising:

[0036] Memory for storing computer programs;

[0037] A processor is used to implement the above-mentioned anomaly detection method when executing the computer program.

[0038] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the above-mentioned anomaly detection method is implemented.

[0039] The present invention provides an anomaly detection system, comprising: a plurality of detection modules, wherein different detection modules are preset with different anomaly detection operations, and are used to execute the anomaly detection operation on the operation log generated by the target system within the preset time period to obtain an anomaly value, convert the anomaly value into a discrete anomaly value, and add an identifier corresponding to the anomaly detection operation to the discrete anomaly value to obtain detection identifier data; a model detection module, which is used to form an identifier sequence from the detection identifier data, and perform anomaly detection on the identifier sequence using a pre-trained neural network model to obtain an overall anomaly value corresponding to the target system in the preset time period; and determine whether the target system is abnormal according to the overall anomaly value; wherein the pre-trained neural network model has learned a preset baseline identifier sequence, and the overall anomaly value represents the degree to which the identifier sequence deviates from the baseline identifier sequence.

[0040] It can be seen that the anomaly detection system provided by the present invention first includes multiple detection modules, and different detection modules preset different anomaly detection operations. These detection modules are used to perform the anomaly detection operation on the operation log generated by the target system within the preset time period to obtain an anomaly value, convert the anomaly value into a discrete anomaly value, and add the identification corresponding to the anomaly detection operation to the discrete anomaly value to obtain the detection identification data. That is, these detection modules can be used to detect the operation log of the target system from different dimensions. In addition, the system also includes a model detection module, which is used to form the detection identification data into an identification sequence, and use a pre-trained neural network model to perform anomaly detection on the identification sequence to obtain the overall anomaly value corresponding to the target system in the preset time period; determine whether the target system is abnormal according to the overall anomaly value; wherein the pre-trained neural network model has learned a preset baseline identification sequence, and the overall anomaly value represents the degree to which the identification sequence deviates from the baseline identification sequence. That is, the pre-trained neural network model can be used to perform comprehensive detection on the detection results of different detection modules to obtain a more comprehensive and reliable comprehensive detection result, so as to be more adaptable to multi-dimensional anomalies in high-frequency and complex environments. The present invention also provides an anomaly detection method, an electronic device and a computer-readable storage medium, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 A structural block diagram of an anomaly detection system provided by an embodiment of the present invention;

[0043] Figure 2 A structural block diagram of another anomaly detection system provided by an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of an identification sequence processing provided by an embodiment of the present invention;

[0045] Figure 4 A flowchart of an anomaly detection method provided by an embodiment of the present invention;

[0046] Figure 5 The present invention provides a structural block diagram of an electronic device. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In modern large-scale computing environments, the large amount of logs generated by the system provide basic data for fault detection and system monitoring. However, traditional rule-based log analysis methods can only provide limited detection capabilities and cannot cope with multi-dimensional anomalies in high-frequency and complex environments.

[0049] In view of this, in order to solve the technical problem of how to provide a more efficient and accurate log anomaly detection system, the present invention can provide an anomaly detection system, which can take different anomaly detection operations to detect the operation log of the target system, and can further use the pre-trained neural network model to perform comprehensive detection on the detection results obtained by different anomaly detection operations, so as to obtain more comprehensive and reliable comprehensive detection results, thereby being more adaptable to multi-dimensional anomalies in high-frequency and complex environments.

[0050] For easier understanding, please refer to Figure 1 , Figure 1 This is a structural block diagram of an anomaly detection system provided by an embodiment of the present invention. The system may include multiple detection modules 10 and a single model detection module 20, wherein the detection module 10 is connected to the model detection module 20, and the model detection module 20 includes a pre-trained neural network model 21. The following are the specific uses of the detection module 10 and the model detection module 20:

[0051] A plurality of detection modules 10, different detection modules 10 are preset with different anomaly detection operations, and are used to perform anomaly detection operations on the operation logs generated by the target system within a preset time period to obtain anomaly values, convert the anomaly values ​​into discrete anomaly values, and add an identifier corresponding to the anomaly detection operation to the discrete anomaly values ​​to obtain detection identifier data;

[0052] The model detection module 20 is used to organize the detection identification data into an identification sequence, and use the pre-trained neural network model 21 to perform anomaly detection on the identification sequence to obtain the overall anomaly value corresponding to the target system in a preset time period, so as to determine whether the target system is abnormal based on the overall anomaly value; wherein the pre-trained neural network model has learned a preset baseline identification sequence, and the overall anomaly value represents the degree to which the identification sequence deviates from the baseline identification sequence.

[0053] It should be noted that the target system is the system monitored by the anomaly detection system, which can be any system that can generate logs, such as a network system or an Internet of Things system (IoT); the processed operation logs can be network traffic logs or IoT device logs. In addition, the specific form of using the anomaly detection system in this embodiment to detect the operation logs generated by the target system within a preset time period can be to use the anomaly detection system to detect the operation logs generated by the target system within a specified time period; it can also be to periodically use the anomaly detection system to detect the operation logs generated by the target system in each periodic time period, for example, every 10 minutes, the operation logs generated by the target system within the 10 minutes are detected, or every day, the operation logs generated by the target system today are detected.

[0054] It can be seen that in this embodiment, the operation log (Log Line) generated by the target system within a preset time period can first be input into different detection modules 10, and different detection modules 10 take different anomaly detection operations for detection, so as to perform preliminary anomaly detection on the target system from different angles and dimensions. The anomaly value here represents the abnormal situation of the target system in a single dimension. The larger the anomaly value, the higher the degree of anomaly; conversely, the smaller the anomaly value, the lower the degree of anomaly. After obtaining the anomaly score, in order to facilitate subsequent model processing, the detection module 10 can also convert the anomaly value into a discrete anomaly value, and add an identifier corresponding to the anomaly detection operation to the discrete anomaly value to obtain the detection identifier data (MarkerToken). It should be noted that this embodiment does not limit the specific anomaly detection operation, and reference may be made to the description in the subsequent embodiments. For example, according to the type of abnormal detection operation, the detection module 10 can be divided into seven types: log sequence detection module (Log SequenceMarker), log category detection module (Top Field Marker), log structure detection module (Log StructureMarker), numerical histogram detection module (Histogram Marker), numerical line graph detection module (Log rateMarker), numerical cluster detection module (Numeric Marker) and discrete value detection module (Discrete Marker). The specific uses of these seven detection modules can refer to the subsequent embodiments. This embodiment also does not limit the method of converting discrete abnormal values. For example, if the value range of the abnormal value is 0~10, multiple interval ranges can be set with 1 as the interval, that is, 0~1, 1~2, 2~3, etc., and then the abnormal value can be converted into a discrete abnormal value according to the interval range where the abnormal value is located, such as the abnormal value 1.1 can be converted into a discrete abnormal value 2. This embodiment also does not limit the identifier corresponding to each abnormal detection operation, but only needs to ensure that the identifier corresponding to each abnormal detection operation is different.

[0055] Furthermore, for the detection identification data generated by each detection module 10, the model detection module 20 can form a token sequence (Token Sequence), and use the pre-trained neural network model 21 to perform anomaly detection on the token sequence to obtain the overall abnormal value corresponding to the target system in the preset time period. It is worth noting that the pre-trained neural network model 21 has learned the preset baseline token sequence. For example, this embodiment can pre-collect the operation logs generated by the target system under normal conditions, and use these operation logs to generate several baseline token sequences. Subsequently, the pre-trained neural network model 21 can be controlled to learn these baseline token sequences. In addition, the above-mentioned overall abnormal value represents the degree to which the token sequence deviates from the baseline token sequence, that is, in subsequent detections, the pre-trained neural network model 21 can quantify the degree to which the target system deviates from the baseline state according to the learned baseline token sequence, thereby determining the overall condition of the target system.

[0056] It is worth noting that the detection module 10 focuses on the anomaly of the target system in a single dimension, while the model detection module 20 focuses on the anomaly of the target system in the global dimension. The combination of the two can avoid misjudgment of a single dimension. Specifically, if the target system is in normal operation, the abnormal values ​​of one or more single dimensions are always in a high state, and the state of these high abnormal values ​​has been recorded in the baseline identification sequence, then from the perspective of the detection module 10, this state of high abnormal values ​​belongs to an abnormal state; while from the perspective of the model detection module 20, this state of high abnormal values ​​belongs to the baseline state and is not abnormal. For example, for a network system, if the system often encounters a surge in access volume in a specific period of time, and the operation and maintenance personnel judge that the surge in access volume is a normal phenomenon, then for the detection module 10 that detects the surge in access volume, this is an abnormal state; while for the model detection module 20, this is a normal phenomenon. If only the detection module 10 is set, false alarms will be generated; and this embodiment further introduces the model detection module 20, so no false alarms will be generated. It can be seen that this embodiment can perform log anomaly detection more comprehensively and reliably, and thus can better adapt to multi-dimensional anomalies in high-frequency and complex environments.

[0057] Furthermore, since there may be a correlation between the abnormal values ​​output by each detection module, in order to facilitate the model to determine the correlation between the abnormal values ​​of various detection types, the detection identification data of each abnormal detection type can be sequentially combined into an identification sequence so that the model can learn and determine the correlation between the identification data. Among them, the abnormal detection type is the type of abnormal detection operation.

[0058] Furthermore, in order to facilitate the model analysis of the association between outliers, the detection module 10 may also perform normalization on the outliers before discretizing them.

[0059] Based on this, the detection module 10 can also be used for:

[0060] Normalize outliers;

[0061] The normalized outliers are discretized to obtain discrete outliers.

[0062] Furthermore, in addition to performing model detection on the detection identification data output by each detection module 10, the model detection module 20 can also perform model detection in combination with the operation log content. Figure 2 , Figure 2 This is a structural block diagram of another anomaly detection system provided by an embodiment of the present invention. The system may also include a log conversion module 30, which is used to convert the operation log generated by the target system within a preset time period into log identification data (Log Token). At this time, the model detection module 20 may also be used to form an identification sequence from the detection identification data and the log identification data, so as to input the identification sequence containing the detection identification data and the log identification data into the pre-trained neural network model for processing.

[0063] It should be noted that this embodiment does not limit how the log conversion module 30 converts the operation log into log identification data, as long as a small number of identifiers can be used to represent the complete operation log, and the identifier is used as the log identification data. For example, the DRAIN method can be used to parse the operation log and convert the operation log into a log Token identifier, so that the log Token identifier is used as the log identification data, wherein the DRAIN algorithm is an online log parsing method based on a fixed depth tree.

[0064] Based on the above embodiments, the anomaly detection system provided by the present invention first includes multiple detection modules, and different detection modules preset different anomaly detection operations. These detection modules are used to perform the anomaly detection operation on the operation log generated by the target system within the preset time period to obtain an anomaly value, convert the anomaly value into a discrete anomaly value, and add the identification corresponding to the anomaly detection operation to the discrete anomaly value to obtain the detection identification data. That is, these detection modules can be used to detect the operation log of the target system from different dimensions. In addition, the system also includes a model detection module, which is used to form an identification sequence with the detection identification data, and use a pre-trained neural network model to perform anomaly detection on the identification sequence to obtain the overall anomaly value corresponding to the target system in the preset time period; determine whether the target system is abnormal according to the overall anomaly value; wherein the pre-trained neural network model has learned a preset baseline identification sequence, and the overall anomaly value represents the degree to which the identification sequence deviates from the baseline identification sequence. That is, the pre-trained neural network model can be used to perform comprehensive detection on the detection results of different detection modules to obtain a more comprehensive and reliable comprehensive detection result, so as to better adapt to multi-dimensional anomalies in high-frequency and complex environments.

[0065] Based on the above embodiment, the specific processing process of the identification sequence by the model detection module 20 is introduced below. For easy understanding, please refer to Figure 3 , Figure 3 A schematic diagram of a marker sequence processing provided by an embodiment of the present invention. In a possible case, the model detection module 20 may also be used for:

[0066] Performing masking processing on part of the identification data in the identification sequence to obtain a to-be-processed identification sequence including masked identification data and uncovered identification data;

[0067] Inputting the identification sequence to be processed into the pre-trained neural network model 21, so that the pre-trained neural network model 21 predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed, and obtains the predicted identification data;

[0068] The preset loss function is used to calculate the loss value between the predicted identification data and the covered identification data in the identification sequence to be processed, and the loss value is used as the overall outlier.

[0069] like Figure 3 As shown, the overall outlier processing flow may include: 1. First, the operation log is input into each detection module 10 and the log conversion module 30 for preliminary anomaly detection and processing, and the detection identification data generated by the detection module 10 and the log identification data generated by the log conversion module 30 are orderly organized into an identification sequence.

[0070] 2. Mask some of the identification data in the identification sequence to obtain a to-be-processed identification sequence including masked identification data and unmasked identification data. Specifically, identification data can be randomly selected from the detection identification data and the log identification data for masking. Figure 3 The part covered by the slash is the covered identification data, and the part not covered by the slash is the uncovered identification data. It should be noted that for the convenience of representation, Figure 3 Only the detection identification data is masked, but in actual applications the log identification data in the log identification sequence may also be masked.

[0071] 3. The identification sequence to be processed can be input into the pre-trained neural network model 21 to obtain the predicted identification sequence output by the pre-trained neural network model 21. Figure 3 ST2' and STn' in the figure represent the predicted identification data corresponding to the covered identification data. The model needs to predict the covered identification data in the identification sequence to be processed based on the learned baseline identification sequence and the uncovered identification data in the identification sequence to be processed. In simple terms, the pre-trained neural network model 21 needs to complete the covered identification data in the identification sequence to be processed based on the baseline state.

[0072] 4. The preset loss function can be used to calculate the loss value (Loss) between the predicted identification data and the masked identification data in the identification sequence to be processed, and the loss value is used as the overall outlier. Figure 3 The f(x) in the expression indicates a preset loss function. This is because when obtaining the predicted identification data corresponding to each covered identification data, it is considered that the state corresponding to the identification sequence may deviate from the baseline state, that is, the covered identification data may be different from the predicted identification data. Therefore, this embodiment can use the preset loss function to calculate the loss value (Loss) between the predicted identification data and the covered identification data in the identification sequence to be processed, and use the loss value as the overall abnormal value. The loss value represents the degree of difference between the covered identification data and the predicted identification data, and further represents the degree of difference between the state corresponding to the identification sequence and the baseline state. In this way, this embodiment can complete the overall detection of the target system.

[0073] It should be noted that the present embodiment does not limit the specific pre-trained neural network model 21, as long as the model belongs to the MLM model (Masked Language Model), for example, the pre-trained language model can be a BERT model (Bidirectional Encoder Representations from Transformers). The present embodiment does not limit the processing process of the identification sequence by the pre-trained detection model, and reference may be made to the relevant technology of the BERT model. It is worth pointing out that the pre-trained detection model can first perform positional encoding (Positional Encoding) on ​​each identification data in the identification sequence to be processed to obtain a coding vector, and then determine the role of each identification data in the identification sequence to be processed and the association relationship between each identification data according to the coding vector, and predict the masked identification data.

[0074] It should also be noted that this embodiment does not limit the specific preset loss function. The loss function used in the model training process can be used as the preset loss function here. Please refer to the relevant technology of MLM model training.

[0075] The following is an introduction to the training process of the pre-trained neural network model 21. Based on this, the model detection module can also be used for:

[0076] Get the baseline identification sequence;

[0077] Performing masking processing on part of the identification data in the baseline identification sequence to obtain a training identification sequence including masked identification data and uncovered identification data;

[0078] Inputting the training identification sequence into the initial neural network model, so that the initial neural network model predicts the covered identification data in the training identification sequence according to the uncovered identification data in the training identification sequence, and obtains the identification data to be compared;

[0079] The preset loss function is used to calculate the loss value between the identification data to be compared and the masked identification data in the training identification sequence, and the parameters of the initial neural network model are updated according to the loss value to obtain the pre-trained neural network model 21.

[0080] Similar to the above-mentioned embodiment, the present embodiment needs to mask part of the identification data in the baseline identification sequence to obtain a training identification sequence including masked identification data and unmasked identification data. Subsequently, the training identification sequence can be input into the initial neural network model, and the initial neural network model predicts the masked identification data in the training identification sequence based on the unmasked identification data in the training identification sequence to obtain the identification data to be compared. Subsequently, when the identification data to be compared corresponding to each masked identification data is obtained, the present embodiment can use a preset loss function to calculate the loss value between the identification data to be compared and the masked identification data in the training identification sequence. Subsequently, unlike the above-mentioned embodiment, since the initial neural network model is trained here, the parameters of the initial neural network model need to be updated according to the loss value to obtain the pre-trained neural network model 21. The above training steps can be repeated multiple times until the performance of the pre-trained neural network model 21 meets the application requirements.

[0081] Based on the above embodiments, various detection models 10 are introduced below. In one possible case, the detection module 10 may include:

[0082] The log sequence detection module is used to convert the operation log into a log sequence; use a pre-trained log sequence detection model to perform anomaly detection on the log sequence to obtain a first anomaly value; wherein the pre-trained log sequence detection model has learned a preset normal log sequence, and the first anomaly value represents the degree to which the log sequence deviates from the normal log sequence.

[0083] In this embodiment, the log sequence detection module (Log Sequence Marker) is used to detect the log sequence (logsequence), and its core is a pre-trained log sequence detection model. The pre-trained log sequence detection model has learned a preset normal log sequence. Therefore, the first abnormal value obtained by the pre-trained log sequence detection model for abnormal detection of the log sequence can represent the degree to which the log sequence deviates from the normal log sequence. In this way, this embodiment can detect abnormal situations of the log sequence in a timely manner.

[0084] It should be noted that this embodiment does not limit the specific type of the pre-trained log sequence detection model, for example, it can be a logBERT model.

[0085] In a possible case, the detection module 10 may include:

[0086] The log category detection module is used to determine the preset log category corresponding to the operation log, and obtain the log category occurrence frequency distribution data corresponding to the current detection moment based on the occurrence frequency of each preset log category within the preset time period according to the operation log statistics; detect the degree of change between the log category occurrence frequency distribution data at the current detection moment and the log category occurrence frequency distribution data at the previous historical detection moment, and obtain the second abnormal value.

[0087] In this embodiment, the log category detection module (Top Field Marker) is used to detect the frequency change corresponding to each preset log category (Field), and quantify the frequency change as a second abnormal value. Specifically, the module can determine the preset log category corresponding to each operation log generated within the preset time period, and then count the occurrence frequency corresponding to each preset log category within the current preset time period to obtain the log category occurrence frequency distribution data corresponding to the current detection moment. Subsequently, the log category detection module can obtain the log category occurrence frequency distribution data at the previous historical detection moment, and determine the degree of change between the log category occurrence frequency distribution data at the two moments, so as to use the degree of change as the second abnormal value. In this way, the present embodiment can promptly discover the sudden change of the log category.

[0088] It should be noted that this embodiment does not limit specific preset log categories, which can be set according to actual application requirements. This embodiment also does not limit how to calculate the degree of change between the frequency distribution data of log categories at different times, for example, the JSD algorithm (Jensen-Shannon Drift Divergence, JS divergence) can be used for calculation.

[0089] In a possible case, the detection module 10 may include:

[0090] The log structure detection module is used to update the operation log to the log parsing tree; detect the transformation degree between the log parsing tree at the current detection time and the log parsing tree at the previous historical detection time, and obtain the third abnormal value.

[0091] In this embodiment, the log structure detection module (Log Structure Marker) is used to detect the corresponding changes of the log parsing tree at different times. Specifically, the module can update the operation log to the log parsing tree, and detect the degree of change between the log parsing tree at the current detection time and the log parsing tree at the previous historical detection time to obtain the third abnormal value. Among them, the log parsing tree is a multi-level hierarchical structure constructed based on the operation log, which is used to parse the log and can be generated using the DRAIN algorithm. The specific generation method can refer to the relevant technology of the DRAIN algorithm. In this way, this embodiment can promptly discover the mutation of the log parsing tree.

[0092] It should be noted that this embodiment does not limit how to calculate the transformation degree between log parse trees at different times. For example, the above-mentioned JSD algorithm can also be used for calculation.

[0093] The above three modules mainly detect the sequence, type and structure of the operation log. The detection module 10 for detecting the values ​​in the operation log will be introduced below. In one possible case, the detection module 10 may include:

[0094] A numerical histogram detection module is used to extract numerical values ​​from the operation log and convert the numerical values ​​into a histogram; detect the degree of change between the histogram at the current detection moment and the histogram at the previous historical detection moment to obtain a fourth abnormal value;

[0095] In this embodiment, for the cumulative type of numerical value, the numerical histogram detection module (Histogram Marker) can convert it into a histogram (Histogram), and determine the degree of change between the histogram at the current detection moment and the histogram at the previous historical detection moment, and obtain the fourth abnormal value. In this way, this embodiment can detect the sudden change of the histogram.

[0096] It should be noted that this embodiment does not limit how to calculate the degree of transformation between histograms. For example, the above-mentioned JSD algorithm can also be used for calculation.

[0097] In a possible case, the detection module 10 may include:

[0098] The numerical line graph detection module is used to extract numerical values ​​in the operation log and fit the numerical values ​​into a line graph; determine the outlier determination numerical interval based on the line graph; and obtain the fifth outlier value based on the proportion of the numerical values ​​that deviate from the outlier determination numerical interval in all the numerical values.

[0099] In this embodiment, for the numerical value of the fluctuation type, the numerical line graph detection module (Log rate Marker) can fit it into a line graph to reflect the fluctuation of the numerical value over time. Subsequently, the module can determine the outlier determination numerical interval based on the line graph. The outlier determination numerical interval is a numerical interval used to determine outlier values ​​(Outliers). For example, the numerical value in this interval is not an outlier value, and the numerical value not in this interval is an outlier value. Simply put, the outlier determination numerical interval is used to determine the numerical value with abnormal fluctuation amplitude. Finally, the module can obtain the fifth outlier value according to the proportion of the numerical value that deviates from the outlier determination numerical interval in all the numerical values. In this way, this embodiment can determine the abnormal fluctuation of the numerical value of the fluctuation type.

[0100] It should be pointed out that the outlier determination numerical interval can be constructed in a variety of ways. For example, it can be constructed based on a prediction interval algorithm (such as the Facebook Prophet algorithm); or the standard deviation of the values ​​​​composing the line chart can be calculated, and a preset multiple of the standard deviation can be used as the upper bound / lower bound of the interval to obtain the outlier determination numerical interval.

[0101] In a possible case, the detection module 10 may include:

[0102] The numerical clustering detection module is used to extract numerical values ​​in the operation log; cluster the numerical values ​​to obtain numerical clusters; calculate the deviation of the numerical values ​​that are not in the numerical cluster compared to the numerical cluster to obtain the sixth abnormal value.

[0103] In this embodiment, the numerical cluster detection module (Numeric Marker) can perform cluster detection on the numerical values, and detect the outliers of the numerical values ​​according to the data clusters obtained by clustering, such as calculating the deviation of the numerical values ​​not in the numerical cluster compared with the numerical cluster to obtain the sixth outlier. In this way, this embodiment can determine the abnormal outliers of the numerical values ​​in a timely manner.

[0104] It should be noted that this embodiment does not limit how to perform clustering processing on the values, and reference may be made to related technologies of clustering algorithms. For example, a DBSCAN algorithm (Density-Based Spatial Clustering of Applications with Noise) may be used for clustering processing.

[0105] In a possible case, the detection module 10 may include:

[0106] The discrete value detection module is used to extract the discrete variable values ​​in the operation log; according to the discrete variable values, the occurrence frequency of each value corresponding to the discrete variable value is counted to obtain the discrete value occurrence frequency distribution data corresponding to the current detection time; the degree of change between the discrete value occurrence frequency distribution data at the current detection time and the discrete value occurrence frequency distribution data at the previous historical detection time is detected to obtain the seventh abnormal value.

[0107] In this embodiment, a discrete variable value refers to a variable value with multiple preset values, and the target system will select the value of the discrete variable value from these multiple preset values. For such values, the discrete value detection module (Discrete Marker) can count the frequency of occurrence of each value corresponding to the discrete variable value, obtain the frequency distribution data of the discrete value corresponding to the current detection time, and detect the degree of change between the frequency distribution data of the discrete value at the current detection time and the frequency distribution data of the discrete value at the previous historical detection time, and obtain the seventh abnormal value. In this way, this embodiment can timely discover abnormal changes in the values ​​of discrete variable values.

[0108] It should be noted that this embodiment does not limit how to calculate the degree of change between the frequency distribution data of discrete value occurrences. For example, the above-mentioned JSD algorithm can also be used for calculation.

[0109] Finally, it should be noted that this embodiment does not limit the types of detection modules and the abnormality detection operations used by the detection modules. In addition to the detection modules described above, other detection modules may also be added as needed.

[0110] Based on the above embodiment, an abnormality detection method provided by an embodiment of the present invention is introduced below. Figure 4 , Figure 4 A flowchart of an anomaly detection method provided by an embodiment of the present invention, which is applied to the anomaly detection system as described above, may include:

[0111] S401. The detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, converts the anomaly value into a discrete anomaly value, and adds an identifier corresponding to the anomaly detection operation to the discrete anomaly value to obtain detection identifier data; wherein different detection modules preset different anomaly detection operations.

[0112] In this step, the operation logs generated by the target system within a preset time period can first be input into different detection modules 10, and different detection modules 10 take different abnormal detection operations for detection, so as to perform preliminary abnormality detection on the target system from different angles and dimensions. After obtaining the abnormal value, in order to facilitate subsequent model processing, the detection module 10 can also convert the abnormal value into a discrete abnormal value, and add an identification corresponding to the abnormal detection operation to the discrete abnormal value to obtain detection identification data.

[0113] Furthermore, in order to facilitate the model analysis of the association between outliers, the detection module 10 may also normalize the outliers before discretizing them.

[0114] Optionally, the method may further include:

[0115] Step 11: The detection module normalizes the outliers;

[0116] Step 12: Discretize the normalized outliers to obtain discrete outliers.

[0117] S402, the model detection module combines the detection identification data into an identification sequence, and uses the pre-trained neural network model to perform anomaly detection on the identification sequence to obtain the overall anomaly value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal based on the overall anomaly value; wherein the pre-trained neural network model has learned a preset baseline identification sequence, and the overall anomaly value represents the degree to which the identification sequence deviates from the baseline identification sequence.

[0118] In this step, for the detection identification data generated by each detection module 10, the model detection module 20 can form an identification sequence, and use the pre-trained neural network model 21 to perform anomaly detection on the identification sequence to obtain the overall abnormal value corresponding to the target system in the preset time period. It is worth noting that the pre-trained neural network model 21 has learned the preset baseline identification sequence. Therefore, the pre-trained neural network model 21 can be used to detect the input identification sequence to determine the degree to which the target system deviates from the baseline state.

[0119] Furthermore, in addition to performing model detection on the detection identification data output by each detection module 10, the model detection module 20 can also perform model detection in combination with the operation log content. Based on this, the method can also include:

[0120] Step 21: The log conversion module converts the operation log generated by the target system within a preset time period into log identification data;

[0121] Step 22: The model detection module combines the detection identification data and the log identification data into an identification sequence.

[0122] The following is an introduction to the way in which the model detection module detects the identification sequence. Based on this, the model detection module forms an identification sequence with the detection identification data, and uses the pre-trained neural network model to perform anomaly detection on the identification sequence to obtain the overall anomaly value corresponding to the target system in the preset time period, which may include:

[0123] S101: performing masking processing on part of the identification data in the identification sequence to obtain a to-be-processed identification sequence including masked identification data and uncovered identification data;

[0124] S102: inputting the identification sequence to be processed into a pre-trained neural network model, so that the pre-trained neural network model predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed, and obtains predicted identification data;

[0125] S103: Calculate the loss value between the predicted identification data and the covered identification data in the identification sequence to be processed using a preset loss function, and use the loss value as the overall abnormal value.

[0126] In this embodiment, the detection identification data generated by the detection module 10 and the log identification data generated by the log conversion module 30 can first be orderly organized into an identification sequence. Subsequently, this embodiment needs to mask part of the identification data in the identification sequence to obtain a pending identification sequence containing masked identification data and uncovered identification data. Subsequently, the pending identification sequence can be input into the pre-trained neural network model 21. The model needs to predict the masked identification data in the pending identification sequence based on the learned baseline identification sequence and the uncovered identification data in the pending identification sequence. Simply put, the pre-trained neural network model 21 needs to complete the masked identification data in the pending identification sequence based on the baseline state.

[0127] Subsequently, when the predicted identification data corresponding to each covered identification data is obtained, the loss value between the predicted identification data and the covered identification data in the identification sequence to be processed can be calculated using a preset loss function, and the loss value is used as the overall abnormal value, wherein the loss value characterizes the degree of difference between the covered identification data and the predicted identification data, and further characterizes the degree of difference between the state corresponding to the identification sequence and the baseline state. In this way, the present embodiment can complete the overall detection of the target system.

[0128] The following is an introduction to the training method of the pre-trained neural network model. Based on this, the method may also include:

[0129] S201: Acquire a baseline identification sequence;

[0130] S202: performing masking processing on part of the identification data in the baseline identification sequence to obtain a training identification sequence including masked identification data and uncovered identification data;

[0131] S203: inputting the training identification sequence into the initial neural network model, so that the initial neural network model predicts the covered identification data in the training identification sequence according to the uncovered identification data in the training identification sequence, and obtains identification data to be compared;

[0132] S204: Calculate the loss value between the identification data to be compared and the masked identification data in the training identification sequence using a preset loss function, and update the parameters of the initial neural network model according to the loss value to obtain a pre-trained neural network model.

[0133] Similar to the above-mentioned embodiment, the present embodiment needs to mask part of the identification data in the baseline identification sequence to obtain a training identification sequence including masked identification data and unmasked identification data. Subsequently, the training identification sequence can be input into the initial neural network model, and the initial neural network model predicts the masked identification data in the training identification sequence based on the unmasked identification data in the training identification sequence to obtain the identification data to be compared. Subsequently, when the identification data to be compared corresponding to each masked identification data is obtained, the present embodiment can use a preset loss function to calculate the loss value between the identification data to be compared and the masked identification data in the training identification sequence. Different from the above-mentioned embodiment, since the initial neural network model is trained here, the parameters of the initial neural network model need to be updated according to the loss value to obtain the pre-trained neural network model 21. The above training steps can be repeated multiple times until the performance of the pre-trained neural network model 21 meets the application requirements.

[0134] The following describes how the detection module performs anomaly detection operations:

[0135] Optionally, the detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, which may include:

[0136] Step 31: Convert the running log into a log sequence;

[0137] Step 32: Use a pre-trained log sequence detection model to perform anomaly detection on the log sequence to obtain a first anomaly value; wherein the pre-trained log sequence detection model has learned a preset normal log sequence, and the first anomaly value represents the degree to which the log sequence deviates from the normal log sequence.

[0138] In this embodiment, the log sequence detection module can detect the log sequence, so that the abnormal situation of the log sequence can be discovered in time.

[0139] Optionally, the detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, and may also include:

[0140] Step 41: Determine the preset log category corresponding to the operation log, and count the occurrence frequency of each preset log category within a preset time period according to the operation log to obtain the log category occurrence frequency distribution data corresponding to the current detection time;

[0141] Step 42: Detect the degree of change between the frequency distribution data of the log category at the current detection time and the frequency distribution data of the log category at the previous historical detection time, and obtain a second abnormal value.

[0142] In this embodiment, the log category detection module can detect the frequency changes corresponding to each preset log category (Field), so that the sudden change of the log category can be discovered in time.

[0143] Optionally, the detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, and may also include:

[0144] Step 51: Update the running log to the log parsing tree;

[0145] Step 52: Detect the degree of transformation between the log parsing tree at the current detection time and the log parsing tree at the previous historical detection time to obtain a third abnormal value.

[0146] In this embodiment, the log structure detection module can detect the corresponding changes of the log parsing tree at different times, so that the mutation of the log parsing tree can be discovered in time.

[0147] Optionally, the detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, and may also include:

[0148] Step 61: extracting the values ​​in the operation log and converting the values ​​into a histogram;

[0149] Step 62: Detect the degree of change between the histogram at the current detection moment and the histogram at the previous historical detection moment to obtain a fourth abnormal value.

[0150] In this embodiment, for cumulative values, a value histogram detection module may be used to convert the value into a histogram, and the degree of change of the histogram at different detection moments may be detected, so that a sudden change of the histogram may be detected.

[0151] Optionally, the detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, and may also include:

[0152] Step 71: extracting values ​​from the operation log and fitting the values ​​into a line graph;

[0153] Step 72: Determine the outlier determination value interval according to the line graph;

[0154] Step 73: Obtain the fifth outlier value according to the proportion of the values ​​that deviate from the outlier determination value interval in all the values.

[0155] In this embodiment, for the fluctuating type values, a numerical line graph detection module can be used to fit them into a line graph, and the outlier determination numerical interval can be determined in the line graph to timely determine the abnormal fluctuation of the fluctuating type values.

[0156] Optionally, the detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, and may also include:

[0157] Step 81: extracting the value in the operation log;

[0158] Step 82: clustering the values ​​to obtain value clusters;

[0159] Step 83: Calculate the deviation of the value that is not in the value cluster compared to the value cluster to obtain the sixth abnormal value.

[0160] In this embodiment, the numerical value cluster detection module can perform cluster detection on the numerical values, and detect the outliers of the numerical values ​​according to the data clusters obtained by clustering, so as to determine the abnormal outliers of the numerical values ​​in time.

[0161] Optionally, the detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, and may also include:

[0162] Step 91: extracting discrete variable values ​​in the operation log;

[0163] Step 92: Count the occurrence frequencies of each value corresponding to the discrete variable value according to the discrete variable value, and obtain the occurrence frequency distribution data of the discrete value corresponding to the current detection time;

[0164] Step 93: Detect the degree of change between the frequency distribution data of discrete value occurrences at the current detection moment and the frequency distribution data of discrete value occurrences at the previous historical detection moment, and obtain the seventh abnormal value.

[0165] In this embodiment, a discrete variable value refers to a variable value having multiple preset values. For such values, a discrete value detection module can be used to count the occurrence frequencies of each value corresponding to the discrete variable value, and detect the degree of change between the frequency distribution data of the discrete value at different detection times, so that abnormal changes in the discrete variable value can be discovered in time.

[0166] Please refer to Figure 5 , Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention provides an electronic device 50, including a processor 51 and a memory 52; wherein the memory 52 is used to store a computer program; the processor 51 is used to execute the anomaly detection method provided in the aforementioned embodiment when executing the computer program.

[0167] For the specific process of the above-mentioned abnormality detection method, reference may be made to the corresponding contents provided in the aforementioned embodiments, which will not be repeated here.

[0168] Furthermore, the memory 52, as a carrier for storing resources, may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0169] In addition, the electronic device 50 also includes a power supply 53, a communication interface 54, an input / output interface 55 and a communication bus 56; wherein the power supply 53 is used to provide working voltage for each hardware device on the electronic device 50; the communication interface 54 can create a data transmission channel between the electronic device 50 and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 55 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0170] In addition, multiple electronic devices mentioned above may form a cluster to jointly execute the abnormal cause detection method mentioned above, such as some electronic devices may be provided with a detection module to perform abnormal detection operations; and another part of the electronic devices may be provided with a model detection module to perform model detection on the identification sequence.

[0171] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the anomaly detection method of the above embodiment when the computer program / instruction is executed by a processor.

[0172] Since the embodiments of the computer program product part correspond to the embodiments of the anomaly detection method part, please refer to the description of the embodiments of the anomaly detection method part for the embodiments of the computer program product part, which will not be repeated here.

[0173] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the anomaly detection method of the above embodiment is implemented.

[0174] Since the embodiments of the computer-readable storage medium part correspond to the embodiments of the anomaly detection method part, the embodiments of the storage medium part refer to the description of the embodiments of the anomaly detection method part, which will not be repeated here.

[0175] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0176] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0177] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0178] The above is a detailed introduction to an abnormality detection system, method, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. An anomaly detection system, characterized in that: include: Multiple detection modules, different detection modules are preset with different anomaly detection operations, and are used to perform the anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, convert the anomaly value into a discrete anomaly value, and add an identifier corresponding to the anomaly detection operation to the discrete anomaly value to obtain detection identifier data; A model detection module is used to form an identification sequence by combining the detection identification data, and to perform anomaly detection on the identification sequence using a pre-trained neural network model to obtain an overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal based on the overall abnormality value; wherein the pre-trained neural network model has learned a preset baseline identification sequence, and the overall abnormality value represents the degree to which the identification sequence deviates from the baseline identification sequence.

2. The abnormality detection system according to claim 1, characterized in that: The detection module comprises: A log sequence detection module, configured to convert the operation log into a log sequence; perform anomaly detection on the log sequence using a pre-trained log sequence detection model to obtain a first anomaly value; wherein the pre-trained log sequence detection model has learned a preset normal log sequence, and the first anomaly value represents the degree to which the log sequence deviates from the normal log sequence; A log category detection module is used to determine the preset log category corresponding to the operation log, and to count the occurrence frequency corresponding to each preset log category within the preset time period according to the operation log, to obtain the log category occurrence frequency distribution data corresponding to the current detection moment; to detect the degree of change between the log category occurrence frequency distribution data at the current detection moment and the log category occurrence frequency distribution data at the previous historical detection moment, to obtain a second abnormal value; The log structure detection module is used to update the operation log to a log parsing tree; detect the degree of change between the log parsing tree at the current detection moment and the log parsing tree at the previous historical detection moment, and obtain a third abnormal value.

3. The abnormality detection system according to claim 1, characterized in that: The detection module comprises: A numerical histogram detection module is used to extract numerical values ​​from the operation log and convert the numerical values ​​into a histogram; detect the degree of change between the histogram at the current detection moment and the histogram at the previous historical detection moment to obtain a fourth abnormal value; A numerical line graph detection module is used to extract numerical values ​​in the operation log and fit the numerical values ​​into a line graph; determine an outlier determination numerical interval according to the line graph; and obtain a fifth abnormal value according to the proportion of numerical values ​​deviating from the outlier determination numerical interval in all numerical values; A numerical clustering detection module is used to extract numerical values ​​from the operation log; perform clustering processing on the numerical values ​​to obtain numerical clusters; calculate the deviation of the numerical values ​​not in the numerical cluster compared to the numerical cluster to obtain a sixth abnormal value; A discrete value detection module is used to extract discrete variable values ​​from the operation log; count the occurrence frequencies of each value corresponding to the discrete variable value according to the discrete variable value, and obtain the discrete value occurrence frequency distribution data corresponding to the current detection moment; detect the degree of change between the discrete value occurrence frequency distribution data at the current detection moment and the discrete value occurrence frequency distribution data at the previous historical detection moment, and obtain the seventh abnormal value.

4. The abnormality detection system according to claim 1, characterized in that: The detection module is further used for: Performing normalization processing on the outliers; The normalized outliers are discretized to obtain discrete outliers.

5. The abnormality detection system according to claim 1, characterized in that: Also includes: A log conversion module, used to convert the operation logs generated by the target system within a preset time period into log identification data; The model detection module is further used to combine the detection identification data and the log identification data into an identification sequence.

6. The abnormality detection system according to any one of claims 1 to 5, characterized in that: The model detection module is also used for: Performing masking processing on part of the identification data in the identification sequence to obtain a to-be-processed identification sequence including masked identification data and uncovered identification data; Inputting the identification sequence to be processed into the pre-trained neural network model, so that the pre-trained neural network model predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed, to obtain predicted identification data; A preset loss function is used to calculate a loss value between the predicted identification data and the covered identification data in the identification sequence to be processed, and the loss value is used as the overall abnormal value.

7. The abnormality detection system according to claim 6, characterized in that: The model detection module is also used for: Acquire the baseline identification sequence; Performing masking processing on part of the identification data in the baseline identification sequence to obtain a training identification sequence including masked identification data and uncovered identification data; Inputting the training identification sequence into an initial neural network model, so that the initial neural network model predicts the covered identification data in the training identification sequence according to the uncovered identification data in the training identification sequence, to obtain identification data to be compared; The preset loss function is used to calculate the loss value between the identification data to be compared and the covered identification data in the training identification sequence, and the parameters of the initial neural network model are updated according to the loss value to obtain the pre-trained neural network model.

8. An anomaly detection method, characterized in that: Applied to the anomaly detection system according to any one of claims 1 to 7, the method comprises: The detection module performs an anomaly detection operation on the operation log generated by the target system within a preset time period to obtain an anomaly value, converts the anomaly value into a discrete anomaly value, and adds an identifier corresponding to the anomaly detection operation to the discrete anomaly value to obtain detection identifier data; wherein different detection modules preset different anomaly detection operations; The model detection module organizes the detection identification data into an identification sequence, and uses a pre-trained neural network model to perform anomaly detection on the identification sequence to obtain an overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal based on the overall abnormality value; wherein the pre-trained neural network model has learned a preset baseline identification sequence, and the overall abnormality value represents the degree to which the identification sequence deviates from the baseline identification sequence.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the anomaly detection method as claimed in claim 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the anomaly detection method according to claim 8 is implemented.

Images