Abnormality detection method and device based on time information, electronic equipment and medium

Abstract

The invention provides an anomaly detection method and device based on time information, electronic equipment and a medium, relates to the field of artificial intelligence, and can divide a preset detection period into a plurality of preset time periods, and can use a pre-training detection model to learn preset baseline identification sequences corresponding to the preset time periods of a target system. And ensuring that the pre-training detection model masters the baseline operation state corresponding to each preset time period of the target system. Then, in a detection link, executing different anomaly detection operations on running logs generated by the target system in the preset time period, converting an obtained abnormal value into detection identification data, and converting the preset time period into timestamp identification data; therefore, the timestamp identification data and the detection identification data corresponding to the preset time period can form an identification sequence, and the pre-training detection model is utilized to perform anomaly detection on the identification sequence, so that whether the target system deviates from the baseline operation state corresponding to the preset time period is determined based on the model, and the detection effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an anomaly detection method, device, electronic equipment and medium based on time information. Background Art

[0002] In modern large-scale computing environments, the large amount of logs generated by the system provide basic data for fault detection and system monitoring. However, most existing log detection methods are based on sliding analysis based on fixed time windows, which not only causes their detection results to be affected by the reliability of the time window, but also leads to a lack of effective response strategies for complex time series log data, especially when the time series contains seasonal changes, traditional fixed window methods are difficult to capture the changes. Summary of the invention

[0003] The purpose of the present invention is to provide an anomaly detection method, device, electronic device and medium based on time information, which can divide a preset detection cycle into multiple preset time periods, and can use a pre-trained detection model to learn the baseline operating state of the target system corresponding to each preset time period. The model can then be used to determine the degree of deviation between the actual operating state of the target system corresponding to the preset time period and the baseline operating state, thereby determining whether the target system is abnormal, thereby improving the detection effect.

[0004] In order to solve the above technical problems, the present invention provides an anomaly detection method based on time information, comprising:

[0005] Dividing a preset detection cycle into a plurality of preset time periods;

[0006] Performing different anomaly detection operations on the operation logs generated by the target system during the preset time period to obtain anomaly values ​​of different detection types corresponding to the preset time period;

[0007] Converting the abnormal value into a discrete abnormal value, adding a preset detection type identifier to the discrete abnormal value to obtain detection identifier data, and adding a preset timestamp identifier to the timestamp corresponding to the preset time period to obtain timestamp identifier data;

[0008] The timestamp identification data and detection identification data corresponding to the preset time period are combined into an identification sequence, and the identification sequence is subjected to anomaly detection using a pre-trained detection model to obtain an overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal based on the overall abnormality value; wherein the pre-trained detection model has learned a preset baseline identification sequence corresponding to each of the preset time periods, and the overall abnormality value represents the degree to which the identification sequence deviates from the preset baseline identification sequence.

[0009] Optionally, it also includes:

[0010] Determine the number of logs generated by the target system in each of the preset time periods, and set a sampling rate corresponding to each of the preset time periods according to the number of logs;

[0011] Sampling the operation logs generated by the target system in each of the preset time periods according to the sampling rate corresponding to each of the preset time periods;

[0012] The performing different anomaly detection operations on the operation logs generated by the target system in each of the preset time periods includes:

[0013] Different anomaly detection operations are performed on the operation logs sampled in each of the preset time periods.

[0014] Optionally, it also includes:

[0015] Converting the operation logs generated by the target system in a preset time period into log identification data;

[0016] The step of forming an identification sequence from the timestamp identification data and the detection identification data corresponding to the preset time period includes:

[0017] The timestamp identification data, the detection identification data, and the log identification data corresponding to the preset time period are combined into the identification sequence.

[0018] Optionally, performing different anomaly detection operations on the operation logs generated by the target system during the preset time period includes:

[0019] Combining the operation logs into a log sequence, and inputting the log sequence into a pre-trained log sequence detection model to obtain an abnormal value of the log sequence detection type; wherein the pre-trained log sequence detection model has learned a preset normal log sequence, and the abnormal value of the log sequence detection type represents the degree to which the log sequence deviates from the normal log sequence;

[0020] and / or, updating the operation log to a log parsing tree, and determining a degree of change of the log parsing tree between adjacent preset time periods to obtain an abnormal value of a log structure detection type;

[0021] And / or, according to the preset log category to which each of the operation logs belongs, determining the degree of change between the occurrence frequency corresponding to each of the preset log categories in the current preset time period and the occurrence frequency corresponding to each of the preset log categories in the previous preset time period, and obtaining an abnormal value of the log category detection type;

[0022] and / or, extracting discrete variable values ​​from the operation log, and determining a degree of change between the frequency of occurrence of each preset value corresponding to the discrete variable value in the current preset time period and the frequency of occurrence in the previous preset time period, to obtain an abnormal value of the discrete value detection type;

[0023] and / or, extracting values ​​from the operation log, clustering the values, and determining the deviation of values ​​not in the value cluster compared to the value cluster to obtain abnormal values ​​of the value cluster detection type;

[0024] And / or, convert multiple numerical values ​​of the same type in the operation log into a line graph, determine the outlier judgment numerical interval based on the line graph, and determine the proportion of numerical values ​​outside the outlier judgment numerical interval in all numerical values ​​to obtain abnormal values ​​of the type detected by the numerical line graph.

[0025] Optionally, performing anomaly detection on the identification sequence using a pre-trained detection model to obtain an overall anomaly value of the target system corresponding to the preset time period includes:

[0026] Performing masking processing on part of the identification data in the identification sequence to obtain an identification sequence to be processed; wherein the masked identification data in the identification sequence to be processed is the detection identification data or a combination of the timestamp identification data and the detection identification data;

[0027] Inputting the identification sequence to be processed into the pre-trained detection model, so that the pre-trained detection model predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed, and obtains predicted identification data;

[0028] A preset loss function is used to calculate a loss value between the predicted identification data and the masked identification data, and the loss value is used as the overall abnormal value.

[0029] Optionally, the forming of an identification sequence with the timestamp identification data and the detection identification data corresponding to the preset time period includes:

[0030] Sequentially compose an identification sequence of the timestamp identification data and the detection identification data corresponding to the preset time period;

[0031] The step of inputting the identification sequence to be processed into the pre-trained detection model so that the pre-trained detection model predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed to obtain the predicted identification data comprises:

[0032] The identification sequence to be processed is input into the pre-trained detection model so that the pre-trained detection model performs position encoding on each identification data in the identification sequence to be processed, and the pre-trained detection model predicts the covered identification data in the identification sequence to be processed according to the position encoding and the uncovered identification data in the identification sequence to be processed to obtain the predicted identification data.

[0033] Optionally, there is overlap between adjacent preset time periods;

[0034] The method further comprises:

[0035] Inputting the identification sequence into the pre-trained detection model, so that the pre-trained detection model predicts the identification sequence of the next preset time period according to the identification sequence to obtain a predicted identification sequence;

[0036] A preset loss function is used to calculate a sequence loss value between the predicted identification sequence and an identification sequence of a next preset time period, so as to determine whether the target system is abnormal according to the sequence abnormality value.

[0037] Optionally, the training process of the pre-trained detection model includes:

[0038] Acquire a baseline identification sequence of each of the preset time periods; wherein the baseline identification sequence includes timestamp identification data of the preset time periods;

[0039] Performing masking processing on part of the identification data in the baseline identification sequence to obtain a training identification sequence;

[0040] Inputting the training identification sequence into an initial detection model, so that the initial detection model predicts the covered identification data in the training identification sequence according to the uncovered identification data in the training identification sequence, to obtain identification data to be compared;

[0041] The preset loss function is used to calculate the training loss value between the identification data to be compared and the masked identification data in the training identification sequence, and the parameters of the initial detection model are updated according to the training loss value to obtain the pre-trained detection model.

[0042] Optionally, it also includes:

[0043] Constructing a timestamp vocabulary table according to the timestamp identification data corresponding to each of the preset time periods;

[0044] The timestamp vocabulary is set for the initial detection model so that the initial detection model predicts the masked timestamp identification data according to the timestamp vocabulary.

[0045] The present invention also provides an abnormality detection device based on time information, comprising:

[0046] A time division module, used to divide a preset detection period into a plurality of preset time periods;

[0047] A detection module, used to perform different anomaly detection operations on the operation logs generated by the target system in the preset time period, and obtain anomaly values ​​of different detection types corresponding to the preset time period;

[0048] an identification generating module, configured to convert the abnormal value into a discrete abnormal value, add a preset detection type identification to the discrete abnormal value to obtain detection identification data, and add a preset timestamp identification to the timestamp corresponding to the preset time period to obtain timestamp identification data;

[0049] A model detection module is used to form an identification sequence with the timestamp identification data and the detection identification data corresponding to the preset time period, and use a pre-trained detection model to perform anomaly detection on the identification sequence to obtain an overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal based on the overall abnormality value; wherein the pre-trained detection model has learned the preset baseline identification sequence corresponding to each of the preset time periods, and the overall abnormality value represents the degree to which the identification sequence deviates from the preset baseline identification sequence.

[0050] The present invention also provides an electronic device, comprising:

[0051] Memory for storing computer programs;

[0052] A processor is used to implement the above-mentioned anomaly detection method based on time information when executing the computer program.

[0053] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the above-mentioned anomaly detection method based on time information is implemented.

[0054] The present invention provides an anomaly detection method based on time information, comprising: dividing a preset detection cycle into a plurality of preset time periods; performing different anomaly detection operations on an operation log generated by a target system in the preset time period to obtain anomaly values ​​of different detection types corresponding to the preset time period; converting the anomaly value into a discrete anomaly value, adding a preset detection type identifier to the discrete anomaly value to obtain detection identifier data, and adding a preset timestamp identifier to a timestamp corresponding to the preset time period to obtain timestamp identifier data; composing an identifier sequence with the timestamp identifier data and the detection identifier data corresponding to the preset time period, and performing anomaly detection on the identifier sequence using a pre-trained detection model to obtain an overall anomaly value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal according to the overall anomaly value; wherein the pre-trained detection model has learned a preset baseline identifier sequence corresponding to each of the preset time periods, and the overall anomaly value represents the degree to which the identifier sequence deviates from the preset baseline identifier sequence.

[0055] It can be seen that the present invention can first divide the preset detection cycle into multiple preset time periods, and can perform different abnormal detection operations on the operation logs generated by the target system in the preset time period to obtain abnormal values ​​of different detection types corresponding to the preset time period. Subsequently, the abnormal value can be converted into a discrete abnormal value, and a preset detection type identifier is added to the discrete abnormal value to obtain detection identification data, and a preset timestamp identifier is added to the timestamp corresponding to the preset time period to obtain timestamp identification data, and then the timestamp identification data and the detection identification data corresponding to the preset time period can be composed of an identification sequence, and the identification sequence can be abnormally detected using a pre-trained detection model to obtain the overall abnormal value corresponding to the preset time period of the target system, so as to determine whether the target system is abnormal according to the overall abnormal value. Among them, the pre-trained detection model has learned the preset baseline identification sequence corresponding to each preset time period, and the overall abnormal value represents the degree to which the identification sequence deviates from the preset baseline identification sequence. This is because in the preset detection cycle, the operating state of the target system can change periodically, and usually has a specific operating state in a specific time period. Therefore, the present invention can use the pre-trained detection model to learn the preset baseline identification sequence corresponding to the target system in each preset time period, that is, use the pre-trained detection model to learn the specific operating state corresponding to the target system in each preset time period. Furthermore, in the online detection link, the present invention only needs to use the pre-trained detection model to identify the identification sequence generated by the target system in each time period to determine whether the target system is abnormal, thereby improving the reliability of log sequence anomaly detection. The present invention provides an anomaly detection device based on time information, an electronic device and a computer-readable storage medium, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0057] Figure 1 A flowchart of an anomaly detection method based on time information provided by an embodiment of the present invention;

[0058] Figure 2 A structural block diagram of an anomaly detection system provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of an anomaly detection process provided by an embodiment of the present invention;

[0060] Figure 4 A structural block diagram of an abnormality detection device based on time information provided by an embodiment of the present invention;

[0061] Figure 5 The present invention provides a structural block diagram of an electronic device. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In modern large-scale computing environments, the large amount of logs generated by the system provide basic data for fault detection and system monitoring. However, most existing log detection methods are based on sliding analysis based on fixed time windows, which not only causes their detection results to be affected by the reliability of the time window, but also leads to a lack of effective response strategies for complex time series log data, especially when the time series contains seasonal changes, traditional fixed window methods are difficult to capture the changes.

[0064] In view of this, in order to solve the technical problem of how to better detect anomalies in a target system, the present invention can provide an anomaly detection method based on time information, which can divide a preset detection cycle into multiple preset time periods, and can use a pre-trained detection model to learn the baseline operating state of the target system corresponding to each preset time period. Subsequently, the model can be used to determine the degree of deviation between the actual operating state of the target system corresponding to the preset time period and the baseline operating state, thereby determining whether the target system is abnormal, thereby improving the detection effect.

[0065] It should be noted that the above-mentioned target system is the system monitored by the anomaly detection system that executes the present method, which can be any system that can generate logs, for example, a network system, an Internet of Things system (IoT); the logs involved can be network traffic logs, IoT device logs, etc.

[0066] For easier understanding, please refer to Figure 1 , Figure 1 A flowchart of an anomaly detection method based on time information provided by an embodiment of the present invention, the method may include:

[0067] S101. Divide a preset detection cycle into a plurality of preset time periods.

[0068] In this step, the preset detection cycle may first be divided into a plurality of preset time intervals, so as to determine the periodic change of the operating state of the target system in the preset detection cycle by detecting the operating state of the target system in each preset time interval.

[0069] It should be noted that the present embodiment does not limit the specific duration of the preset detection cycle, which can be set according to actual application requirements, such as one day, one week, one month, etc. The present embodiment also does not limit the specific duration of the preset time period, which can also be set according to actual application requirements, such as ten minutes, one hour, one day, etc.

[0070] In addition, adjacent preset time periods may not overlap or may overlap. The overlapping preset time periods may be, for example, 0:00-0:10, 0:05-0:15, 0:10-0:20, 0:15-0:25, that is, the first ten minutes and the last ten minutes overlap for five minutes.

[0071] S102: Perform different anomaly detection operations on the operation logs generated by the target system during the preset time period to obtain anomaly values ​​of different detection types corresponding to the preset time period.

[0072] In this step, different anomaly detection operations are performed on the operation logs (Log Line) generated by the target system in each preset time period, so as to perform preliminary anomaly detection on the target system from different angles and dimensions, and obtain anomaly scores (Anomaly Score) of different detection types in the preset time period. Among them, the anomaly score represents the degree of anomaly of the target system in a single detection dimension. The larger the anomaly score, the higher the degree of anomaly; conversely, the smaller the anomaly score, the lower the degree of anomaly. The detection type is the type of anomaly detection operation.

[0073] It should be noted that this embodiment does not limit the specific abnormality detection operation, which can be set according to actual application requirements, and can also refer to the description in subsequent embodiments.

[0074] S103: converting the abnormal value into a discrete abnormal value, adding a preset detection type identifier to the discrete abnormal value to obtain detection identifier data, and adding a preset timestamp identifier to the timestamp corresponding to the preset time period to obtain timestamp identifier data.

[0075] In this step, since the model cannot process continuous outliers, in order to facilitate subsequent model processing, the outliers can be converted into discrete outliers, and the preset detection type identifier corresponding to the detection type is added to the discrete outliers to obtain detection identification data (Marker Token). In addition, in order to provide the model with the time information of the detection, this step can also add a preset timestamp identifier to the timestamp corresponding to the preset time period to obtain timestamp identification data (Timestamp Token). The above detection identification data and timestamp identification data will be input into the model for processing at the same time.

[0076] It should be noted that this embodiment does not limit the method of converting discrete outliers. For example, if the range of outliers is 0-10, multiple intervals can be set with 1 as the interval, that is, 0-1, 1-2, 2-3, etc., and then the outliers can be converted into discrete outliers according to the interval range where the outliers are located, such as outlier 1.1 can be converted into discrete outlier 2. This embodiment also does not limit the preset detection category identifier corresponding to each detection category, and it only needs to ensure that the preset detection category identifier corresponding to each detection category is different.

[0077] Furthermore, since the present embodiment can preset a detection cycle, and can pre-divide the preset detection cycle into preset time periods, each preset time period can be distinguished based on the timestamp. At the same time, by converting the timestamp into timestamp identification data and inputting it into the model, the model can be clearly informed of which preset time period the target system operating status is currently being detected, and can then assist the model in detecting whether the actual operating status of the target system is abnormal based on the baseline operating status of the corresponding preset time period. It should be noted that the present embodiment also does not limit the specific preset timestamp identification, and only needs to ensure that the preset timestamp identification is different from the above-mentioned preset detection type identification.

[0078] S104. The timestamp identification data and the detection identification data corresponding to the preset time period are combined into an identification sequence, and the pre-trained detection model is used to perform anomaly detection on the identification sequence to obtain an overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal according to the overall abnormality value; wherein the pre-trained detection model has learned the preset baseline identification sequence corresponding to each of the preset time periods, and the overall abnormality value represents the degree to which the identification sequence deviates from the preset baseline identification sequence.

[0079] In this step, the timestamp identification data and detection identification data corresponding to the preset time period can be combined into an identification sequence (Token Sequence), and the identification sequence can be input into the pre-trained detection model for anomaly detection to obtain the overall anomaly value of the target system corresponding to the preset time period. The overall anomaly value represents the overall abnormality degree of the target system. The larger the overall anomaly value, the higher the overall abnormality degree of the target system; the smaller the overall anomaly value, the lower the overall abnormality degree of the target system.

[0080] It is worth pointing out that the pre-trained detection model has learned the baseline identification sequence corresponding to each preset time period. Therefore, when receiving the identification sequence, the model can extract the preset timestamp identification data (i.e., the preset time period information) in the identification sequence, and determine whether other identification data in the identification sequence is normal in combination with the baseline identification sequence corresponding to the preset time period, thereby determining whether the target system is normal; or, the model can also extract other identification data in the identification sequence, and predict the preset time period corresponding to the identification sequence in combination with the baseline identification sequence of each preset time period. If the predicted preset time period is the same as the preset time period actually corresponding to the identification sequence, then the target system can be determined to be normal. On the contrary, if the predicted preset time period is different from the preset time period actually corresponding to the identification sequence, then the target system can be determined to be abnormal.

[0081] In short, since the pre-trained detection model has learned the baseline operating status of the target system corresponding to each preset time period, it only needs to determine the degree of deviation of the actual operating status of the target system compared to the baseline operating status of the corresponding time period to determine whether the target system is abnormal.

[0082] It is also worth pointing out that the anomaly detection operation mainly focuses on the anomaly of the target system in a single dimension, while the pre-trained detection model mainly focuses on the anomaly of the target system in the global dimension. The combination of the two can avoid misjudgment of a single dimension. Specifically, if the target system has one or more single-dimensional anomaly values ​​that are always high in a preset time period, and the high state of these anomaly values ​​has been recorded in the baseline identification sequence corresponding to the preset time period, then executing a single anomaly detection operation can only determine that the target system is abnormal, while using the pre-trained detection model for processing can determine that the target system is normal. For example, for a network system, if the system often encounters a surge in access volume in a specific time period, and the operation and maintenance personnel determine that the surge in access volume is a normal phenomenon, then executing an anomaly detection operation to detect the surge in access volume can only determine that the target system is abnormal, which is prone to false alarms; while using the pre-trained detection model for processing can determine that the target system is normal, no false alarms will be generated, and the model processing result is more in line with the normal operation of the target system in the specific time period. It can be seen that this embodiment can perform anomaly detection more comprehensively and reliably, and can not only adapt to multi-dimensional anomalies in high-frequency and complex environments.

[0083] In addition, the traditional anomaly detection method based on a fixed time window can only detect and process the log sequence in a fixed time window, such as always detecting and processing the operation logs in the most recent period of time. This leads to: 1. The fixed time window ignores the operating status of the target system at each specific moment, resulting in it being unable to determine whether the target system is operating normally at a specific moment. 2. The fixed time window makes it difficult to detect the seasonal changes in the operating status of the target system. For example, when the seasonal change cycle of the operating status of the target system is very long, the traditional method can only extend the fixed time window, which directly leads to the need for the traditional method to process a huge amount of operation logs at the same time, which consumes a lot of resources and is difficult to achieve a good detection effect. In this embodiment, since the controllable model learns the operating status corresponding to each preset time period of the target system in the preset detection cycle, and the timestamp information is already included in the identification sequence, it is convenient for the model to determine the time node, and the duration of each preset time period is short, and the corresponding log volume is much smaller than the total log volume corresponding to the preset detection cycle, so the model can well learn the operating status of the target system at each specific moment, and can detect the system operating status at a specific moment, and the detection volume of each preset time period is small, which can achieve a higher detection efficiency. Based on this, the present invention can also achieve a good degree of adaptability to seasonal changes in the operating state of the target system.

[0084] Furthermore, in the normal operation state, the operation state of the target system is closely related to the time information, that is, the detection identification data, log identification data and timestamp identification data are closely related. In order to explicitly reflect this association in the identification sequence and highlight the role of timestamp identification data, the timestamp identification data, detection identification data and log identification data can be sequentially combined into an identification sequence so that the model can learn and determine the association between the identification data.

[0085] Based on this, the steps of forming an identification sequence from the timestamp identification data and the detection identification data corresponding to the preset time period may include:

[0086] Step 11: Sequentially compose an identification sequence by combining the timestamp identification data and the detection identification data corresponding to the preset time period.

[0087] Similarly, the detection identification data of each detection type may also be arranged in sequence so that the model can determine the association relationship between the detection identification data of various detection types.

[0088] Furthermore, in addition to processing the detection identification data and timestamp identification data, the pre-trained detection model can also perform model detection in combination with the operation log content. Specifically, the operation log generated by the target system within a preset time period can be converted into log identification data (Log Token), and the detection identification data, timestamp identification data, and log identification data can be combined into an identification sequence so that the pre-trained detection model can further analyze the log information.

[0089] Based on this, the method may further include:

[0090] Step 21: converting the operation log generated by the target system in a preset time period into log identification data;

[0091] The step of forming an identification sequence from the timestamp identification data and the detection identification data corresponding to the preset time period may include:

[0092] Step 31: The timestamp identification data, the detection identification data, and the log identification data corresponding to the preset time period are combined into the identification sequence.

[0093] Similarly, the timestamp identification data, detection identification data, and log identification data corresponding to the preset time period can be orderly combined into an identification sequence.

[0094] It should be noted that this embodiment does not limit how to convert the operation log into log identification data, as long as a small number of identifiers can be used to represent the complete operation log, and the identifier is used as the log identification data. For example, the DRAIN method can be used to parse the operation log and convert the operation log into a log Token identifier, so that the log Token identifier is used as the log identification data, wherein the DRAIN algorithm is an online log parsing method based on a fixed depth tree.

[0095] For easier understanding, please refer to Figure 2 , Figure 2 This is a structural block diagram of an anomaly detection system provided by an embodiment of the present invention. The system may include a timestamp conversion module, a detection module, a log conversion module, and a pre-trained detection model. The timestamp conversion module is used to generate timestamp identification data, the detection module is used to perform an anomaly detection operation to obtain detection identification data, and the log conversion module is used to convert the operation log into log identification data.

[0096] Based on the above embodiments, the present invention can first divide the preset detection cycle into multiple preset time periods, and can perform different abnormality detection operations on the operation logs generated by the target system in the preset time period to obtain abnormal values ​​of different detection types corresponding to the preset time period. Subsequently, the abnormal value can be converted into a discrete abnormal value, and a preset detection type identifier is added to the discrete abnormal value to obtain detection identification data, and a preset timestamp identifier is added to the timestamp corresponding to the preset time period to obtain timestamp identification data, and then the timestamp identification data and the detection identification data corresponding to the preset time period can be composed of an identification sequence, and the identification sequence can be abnormally detected using a pre-trained detection model to obtain the overall abnormal value corresponding to the preset time period of the target system, so as to determine whether the target system is abnormal according to the overall abnormal value. Among them, the pre-trained detection model has learned the preset baseline identification sequence corresponding to each preset time period, and the overall abnormal value represents the degree to which the identification sequence deviates from the preset baseline identification sequence. This is because in the preset detection cycle, the operating state of the target system can change periodically, and usually has a specific operating state in a specific time period. Therefore, the present invention can use the pre-trained detection model to learn the preset baseline identification sequence corresponding to the target system in each preset time period, that is, use the pre-trained detection model to learn the specific operating state corresponding to the target system in each preset time period. Furthermore, in the online detection link, the present invention only needs to use the pre-trained detection model to identify the identification sequence generated by the target system in each time period to determine whether the target system is abnormal, thereby improving the reliability of log sequence anomaly detection.

[0097] Based on the above embodiment, the specific process of using the pre-trained detection model to perform anomaly detection on the identification sequence is introduced below. Based on this, the use of the pre-trained detection model to perform anomaly detection on the identification sequence to obtain the overall anomaly value of the target system corresponding to the preset time period may include:

[0098] S201, performing masking processing on part of the identification data in the identification sequence to obtain an identification sequence to be processed; wherein the masked identification data in the identification sequence to be processed is the detection identification data or a combination of the timestamp identification data and the detection identification data;

[0099] S202, inputting the identification sequence to be processed into the pre-trained detection model, so that the pre-trained detection model predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed, to obtain predicted identification data;

[0100] S203: Calculate the loss value between the predicted identification data and the masked identification data using a preset loss function, and use the loss value as the overall abnormal value.

[0101] Please refer to Figure 3 , Figure 3 A schematic diagram of an anomaly detection process provided by an embodiment of the present invention. The process of generating an overall anomaly value may include:

[0102] 1. First, input the operation log into each detection module to obtain the detection identification data (ST1~STn) corresponding to each detection type, input the operation log into the log conversion module to obtain the log identification data, and input the timestamp corresponding to the preset time period into the timestamp conversion module to obtain the timestamp identification data (Time-Token, TT). Subsequently, the timestamp identification data, the detection identification data, and the log identification data are combined into an identification sequence. As mentioned above, in the normal operation state, the operation state of the target system is closely related to the time information, that is, the detection identification data, the log identification data and the timestamp identification data are closely related. In order to explicitly reflect this association in the identification sequence and highlight the role of the timestamp identification data, the timestamp identification data, the detection identification data, and the log identification data can be sequentially combined into an identification sequence. In the model training stage, the initial detection model can perform positional encoding for each identification data in the identification sequence, mark the order of each identification data in the sequence, and thus learn the association relationship between each identification data, especially the association relationship between the timestamp identification data and the detection identification data and the log identification data (that is, the association relationship between the time information and the baseline state of the target system). In the online reasoning stage, the pre-trained detection model can perform position encoding for each identification data in the identification sequence, and determine the role of each identification data and the relationship between the identification data according to the order of each identification data in the sequence, especially determine the role of the timestamp identification data and its relationship with other identification data, so as to better parse or infer and predict other identification data based on the timestamp identification data.

[0103] It should be noted that this embodiment does not limit the order of various identification data in the identification sequence. For example, the order may be: timestamp identification data, detection identification data, log identification data. Similarly, the detection identification data of each detection type may also be arranged in sequence so that the model can determine the association between the detection identification data of various detection types. This embodiment also does not limit the specific method of position encoding, and reference may be made to the relevant technology of the Transformer model.

[0104] 2. Mask part of the identification data in the identification sequence to obtain a to-be-processed identification sequence including masked identification data and unmasked identification data. Specifically, identification data may be randomly selected from the timestamp identification data and the detection identification data for masking, for example, the masked identification data may be the detection identification data, or may be a combination of the timestamp identification data and the detection identification data.

[0105] It is worth pointing out that when the timestamp identification data is not covered, the model can detect whether the target system is operating normally in the corresponding preset time period; when the timestamp identification data is covered, the model can detect whether the preset time period corresponding to the identification sequence to be processed is the preset time period corresponding to the timestamp identification data. Furthermore, when the identification sequence also contains log identification data, identification data can also be randomly selected in the log identification data for covering. For ease of understanding, Figure 3 The portion covered by the slash is the covered identification data, and the portion not covered by the slash is the uncovered identification data.

[0106] 3. Input the identification sequence to be processed into the pre-trained detection model to obtain the predicted identification sequence output by the pre-trained detection model. Figure 3 TT' and STn' in the figure represent the predicted identification data corresponding to the masked identification data. The model needs to predict the masked identification data in the identification sequence to be processed based on the learned baseline identification sequence and the unmasked identification data in the identification sequence to be processed. Simply put, the pre-trained detection model needs to complete the masked identification data in the identification sequence to be processed according to the baseline operating status corresponding to each preset time period.

[0107] 4. Use the preset loss function to calculate the loss value (Loss) between the predicted identification data and the masked identification data in the identification sequence to be processed, and use the loss value as the overall outlier. Figure 3 The f(x) in the identifier is a preset loss function. Specifically, since the model predicts the masked identification data in the identification sequence to be processed based on the baseline operating state, and the actual values ​​of these masked identification data may deviate from the baseline operating state, that is, they are inconsistent with the predicted identification data, this embodiment can use the preset loss function to calculate the loss value (Loss) between the predicted identification data and the masked identification data, and use the loss value as the overall abnormal value. The loss value characterizes the degree of difference between the masked identification data and the predicted identification data, and further characterizes the degree of difference between the actual operating state of the target system in the current time period and the baseline operating state of the time period. In this way, this embodiment can complete the overall detection of the target system.

[0108] It should be noted that the present embodiment does not limit the specific pre-trained detection model. Since the model needs to predict the masked identification sequence, the model can be an MLM model (Masked Language Model). For example, the pre-trained language model can be a BERT model (Bidirectional Encoder Representations from Transformers). The present embodiment does not limit the processing process of the identification sequence by the pre-trained detection model. The relevant technology of the BERT model can be referred to. It is worth pointing out that the pre-trained detection model can first position encode each identification data in the identification sequence to be processed to obtain a coding vector, and then determine the role of each identification data in the identification sequence to be processed and the association relationship between each identification data according to the coding vector, and predict the masked identification data.

[0109] Based on this, the step of inputting the identification sequence to be processed into the pre-trained detection model so that the pre-trained detection model predicts the masked identification data in the identification sequence to be processed according to the unmasked identification data in the identification sequence to be processed to obtain the predicted identification data may include:

[0110] Step 41: Input the identification sequence to be processed into the pre-trained detection model so that the pre-trained detection model performs position encoding on each identification data in the identification sequence to be processed, and enables the pre-trained detection model to predict the covered identification data in the identification sequence to be processed according to the position encoding and the uncovered identification data in the identification sequence to be processed, so as to obtain the predicted identification data.

[0111] It should also be noted that this embodiment does not limit the specific preset loss function. The loss function used in the model training process can be used as the preset loss function here. Please refer to the relevant technology of MLM model training.

[0112] Based on the above embodiment, when adjacent preset time periods overlap, the pre-trained detection model can also be used to generate a predicted identification sequence for the next time period based on the identification sequence of the current time period, and then the predicted identification sequence can be used to perform abnormal detection on the identification sequence of the next time period. The following is an introduction to the identification sequence prediction process. Based on this, when adjacent preset time periods overlap, the method can also include:

[0113] S301. Input the identification sequence into the pre-trained detection model, so that the pre-trained detection model predicts the identification sequence of the next preset time period according to the identification sequence to obtain a predicted identification sequence.

[0114] In this embodiment, since the preset time periods overlap, the operation log of the current time period includes part of the operation log of the next time period, that is, the operation status of the target system in the next time period is associated with the operation status of the current time period. Therefore, in this step, the identification sequence can be input into the pre-trained detection model, and the pre-trained detection model obtains the predicted identification sequence based on the identification sequence of the next preset time period. In this way, if the actual identification sequence of the next time period deviates greatly from the predicted identification sequence, it can be determined that the target system is abnormal.

[0115] It should be pointed out that for the MLM model, it can implement identification sequence prediction based on the NSP mechanism (Next Sentence Prediction). For details, please refer to the relevant technologies of NSP.

[0116] S302: Calculate a sequence loss value between the predicted identification sequence and an identification sequence of a next preset time period using a preset loss function, so as to determine whether the target system is abnormal according to the sequence abnormality value.

[0117] In this step, a preset loss function can be used to calculate the sequence loss value between the predicted identification sequence and the identification sequence of the next preset time period. The sequence loss value represents the degree of difference between the predicted identification sequence and the identification sequence of the next preset time period, and further represents the degree of difference between the expected operating state of the target system in the next time period and the actual operating state of the next time period. This also helps to improve the accuracy of anomaly detection.

[0118] Based on the above embodiment, the training process of the pre-trained detection model is introduced below. Of course, before training the model, considering that the timestamp identification data is the identification data set separately by this embodiment, the conventional MLM model cannot recognize the timestamp identification data. At the same time, considering that this embodiment has determined in advance which preset time periods are included in the preset detection cycle, that is, the specific value of the timestamp identification data can be clarified in advance, so a timestamp vocabulary table can be constructed according to the timestamp identification data corresponding to each preset time period, and the timestamp vocabulary table is set for the initial detection model. Based on this, the method can also include:

[0119] S401, constructing a timestamp vocabulary table according to the timestamp identification data corresponding to each of the preset time periods;

[0120] S402: Set the timestamp vocabulary for the initial detection model, so that the initial detection model predicts the masked timestamp identification data according to the timestamp vocabulary.

[0121] In this way, the preset time periods can be processed as independent identification data and form a vocabulary. For example, there are 144 10-minute time intervals in a day and 1008 intervals in a week. Time is the core input for prediction and is used together with other identification data signals for sequence processing in model training.

[0122] The following is a specific training process of the pre-trained detection model, which may include:

[0123] S501, obtaining a baseline identification sequence of each of the preset time periods; wherein the baseline identification sequence includes timestamp identification data of the preset time periods;

[0124] S502, performing masking processing on part of the identification data in the baseline identification sequence to obtain a training identification sequence;

[0125] S503, inputting the training identification sequence into an initial detection model, so that the initial detection model predicts the covered identification data in the training identification sequence according to the uncovered identification data in the training identification sequence, to obtain identification data to be compared;

[0126] S504, using the preset loss function to calculate the training loss value between the identification data to be compared and the masked identification data in the training identification sequence, and updating the parameters of the initial detection model according to the training loss value to obtain the pre-trained detection model.

[0127] Similar to the above embodiment, this embodiment needs to mask part of the identification data in the baseline identification sequence to obtain a training identification sequence containing masked identification data and unmasked identification data. Subsequently, the training identification sequence can be input into the initial detection model, and the initial detection model predicts the masked identification data in the training identification sequence according to the unmasked identification data in the training identification sequence to obtain the identification data to be compared. When the above baseline identification sequence is composed of timestamp identification data, detection identification data, and log identification data in sequence, the pre-trained detection model can first position encode each identification data in the training identification sequence, and determine the role of each identification data in the training identification sequence and the association relationship between each identification data according to the encoding situation, and then predict the masked identification data. Subsequently, when the identification data to be compared corresponding to each masked identification data is obtained, this embodiment can use a preset loss function to calculate the loss value between the identification data to be compared and the masked identification data in the training identification sequence. Subsequently, unlike the above embodiment, since the initial detection model is trained here, the parameters of the initial detection model need to be updated according to the loss value to obtain the pre-trained detection model. The above training steps can be repeated multiple times until the performance of the pre-trained detection model meets the application requirements.

[0128] Based on the above embodiments, considering that the target system will generate a large number of operation logs during operation, and it is not realistic to perform anomaly detection on all operation logs, it is possible to consider sampling the operation logs to reduce the actual number of operation logs processed. At the same time, it is also considered that the number of operation logs generated by the target system in different preset time periods is not the same. If the same sampling rate is used, it is still difficult to balance the number of operation logs in different preset time periods, thereby affecting the processing effect of the model. Therefore, this embodiment can also predetermine the number of logs generated by the target system in each preset time period, and set the corresponding sampling rate according to the number of logs, so as to balance the number of operation logs in different preset time periods. Based on this, the method can also include:

[0129] S601, determining the number of logs generated by the target system in each of the preset time periods, and setting a sampling rate corresponding to each of the preset time periods according to the number of logs;

[0130] S602: sampling the operation logs generated by the target system in each of the preset time periods according to the sampling rates corresponding to the preset time periods.

[0131] It should be noted that this embodiment does not limit how to set the sampling rate according to the number of logs, as long as the sampling rate matches the number of logs. For example, the sampling rate can be in a ratio relationship with the number of logs and can be set according to actual application requirements.

[0132] Furthermore, when different anomaly detection operations are performed on the operation logs generated by the target system in each preset time period, different anomaly detection operations may be performed on the operation logs sampled in each of the preset time periods.

[0133] Based on the above embodiment, the specific content of the abnormality detection operation is introduced below. In one possible case, performing different abnormality detection operations on the operation logs generated by the target system in the preset time period may include:

[0134] Step 51: Combine the operation logs into a log sequence, and input the log sequence into a pre-trained log sequence detection model to obtain an abnormal value of the log sequence detection type; wherein the pre-trained log sequence detection model has learned a preset normal log sequence, and the abnormal value of the log sequence detection type represents the degree to which the log sequence deviates from the normal log sequence.

[0135] In this embodiment, a pre-trained log sequence detection model can be used to detect the log sequence (Log Sequence), and the pre-trained log sequence detection model has learned a preset normal log sequence. Therefore, the pre-trained log sequence detection model will compare the input log sequence with the learned normal log sequence and generate an abnormal value of the log sequence detection type, which can characterize the degree to which the log sequence deviates from the normal log sequence. In this way, this embodiment can detect abnormal situations in a timely manner from the perspective of the log sequence.

[0136] It should be noted that the present embodiment does not limit the specific type of the pre-trained log sequence detection model, for example, it can be a logBERT model, and the specific log sequence detection process can refer to the related technology of logBERT. The present embodiment also does not limit how to convert and combine the operation log into a log sequence, for example, the DRAIN algorithm can be used to convert the operation log into a log Token identifier, and then the log Token identifier is combined into a log sequence. Among them, the DRAIN algorithm is an online log parsing method based on a fixed depth tree.

[0137] In a possible scenario, performing different anomaly detection operations on the operation logs generated by the target system during the preset time period may include:

[0138] Step 61: Update the operation log to a log parsing tree, and determine the degree of change of the log parsing tree between adjacent preset time periods to obtain an abnormal value of the log structure detection type.

[0139] In this embodiment, the corresponding changes of the log parsing tree at different times can be detected, wherein the log parsing tree is a multi-level hierarchical structure constructed based on the operation log, which is used to parse the log and can be generated by the DRAIN algorithm. The specific generation method can refer to the relevant technology of the DRAIN algorithm. Specifically, the operation log can be updated to the log parsing tree, and the degree of change between the log parsing trees between adjacent preset time periods can be detected to obtain the abnormal value of the log structure detection type. In this way, this embodiment can timely determine the mutation of the log parsing tree from the perspective of the log structure.

[0140] It should be noted that this embodiment does not limit how to calculate the transformation degree between log parse trees at different times. For example, the JSD algorithm (Jensen-Shannon Drift Divergence, JS divergence) can be used for calculation.

[0141] In a possible scenario, performing different anomaly detection operations on the operation logs generated by the target system during the preset time period may include:

[0142] Step 71: According to the preset log categories to which the operation logs belong, determine the degree of change between the occurrence frequency corresponding to each preset log category in the current preset time period and the occurrence frequency corresponding to each preset log category in the previous preset time period, and obtain an abnormal value of the log category detection type.

[0143] In this embodiment, the frequency change corresponding to each preset log category (Field) can be detected, and the frequency change can be quantified as an abnormal value. Specifically, according to the preset log category to which each operation log belongs, the degree of change between the frequency of occurrence corresponding to each preset log category in the current preset time period and the frequency of occurrence corresponding to each preset log category in the previous preset time period can be determined to obtain the abnormal value of the log category detection type. In this way, this embodiment can timely discover the sudden change of log category distribution from the perspective of log category (TopField).

[0144] It should be noted that this embodiment does not limit the specific preset log categories, which can be set according to actual application requirements. This embodiment also does not limit how to calculate the change degree of the frequency of occurrence of the preset log category, for example, the above-mentioned JSD algorithm can be used for calculation.

[0145] In a possible scenario, performing different anomaly detection operations on the operation logs generated by the target system during the preset time period may include:

[0146] Step 81: extract the discrete variable values ​​in the operation log, and determine the degree of change between the frequency of occurrence of each preset value corresponding to the discrete variable value in the current preset time period and the frequency of occurrence in the previous preset time period, to obtain an abnormal value of the discrete value detection type.

[0147] In this embodiment, a discrete variable value refers to a variable value having multiple preset values, and the target system will select the value of the discrete variable value from the multiple preset values. For such values, the degree of change between the frequency of occurrence of each preset value corresponding to the discrete variable value in the current preset time period and the frequency of occurrence in the previous preset time period can be determined to obtain an abnormal value of the discrete value detection type. In this way, this embodiment can timely discover the abnormal change of the discrete variable value from the perspective of the discrete variable value (Discrete).

[0148] It should be noted that this embodiment does not limit how to calculate the degree of change of the frequency of occurrence of discrete values. For example, the above-mentioned JSD algorithm can also be used for calculation.

[0149] In a possible scenario, performing different anomaly detection operations on the operation logs generated by the target system during the preset time period may include:

[0150] Step 91: extract the values ​​in the operation log, perform clustering processing on the values, and determine the deviation of the values ​​that are not in the value cluster compared to the value cluster to obtain abnormal values ​​of the value cluster detection type.

[0151] In this embodiment, cluster detection can be performed on the values, and outliers of the values ​​can be detected based on the data clusters obtained by clustering. For example, the deviation of the values ​​that are not in the value cluster compared to the value cluster can be calculated to obtain the abnormal values ​​of the value cluster detection type. In this way, this embodiment can timely determine the abnormal outliers of the values ​​from the perspective of the value cluster.

[0152] It should be noted that this embodiment does not limit how to perform clustering processing on the values, and reference may be made to related technologies of clustering algorithms. For example, a DBSCAN algorithm (Density-Based Spatial Clustering of Applications with Noise) may be used for clustering processing.

[0153] In a possible scenario, performing different anomaly detection operations on the operation logs generated by the target system during the preset time period may include:

[0154] Step 101: Convert multiple numerical values ​​of the same type in the operation log into a line graph, determine an outlier determination numerical interval based on the line graph, and determine the proportion of numerical values ​​outside the outlier determination numerical interval in all numerical values ​​to obtain an outlier value of the type detected by the numerical line graph.

[0155] In this embodiment, for the numerical value of the fluctuation type, it can be fitted into a line graph to reflect the fluctuation of the numerical value over time. Subsequently, the outlier determination numerical interval can be determined based on the line graph. The outlier determination numerical interval is a numerical interval used to determine outlier values ​​(Outliers). For example, the numerical value in this interval is not an outlier value, and the numerical value not in this interval is an outlier value. Simply put, the outlier determination numerical interval is used to determine the numerical value with abnormal fluctuation amplitude. In this way, this embodiment can timely determine the abnormal fluctuation of the numerical value of the fluctuation type from the perspective of the line graph (Log Rate).

[0156] It should be pointed out that the outlier determination numerical interval can be constructed in a variety of ways. For example, it can be constructed based on a prediction interval algorithm (such as the Facebook Prophet algorithm); or the standard deviation of the values ​​​​composing the line chart can be calculated, and a preset multiple of the standard deviation can be used as the upper bound / lower bound of the interval to obtain the outlier determination numerical interval.

[0157] It should be noted that the above anomaly detection operations can be selected according to actual needs. Of course, in addition to the anomaly detection operations introduced above, other anomaly detection operations can also be selected according to actual application needs.

[0158] The following is an introduction to the time information-based anomaly detection device, electronic device, computer program product, and computer-readable storage medium provided in an embodiment of the present invention. The time information-based anomaly detection device, electronic device, computer program product, and computer-readable storage medium described below can be referenced to the time information-based anomaly detection method described above.

[0159] Please refer to Figure 4 , Figure 4 A structural block diagram of an abnormality detection device based on time information provided by an embodiment of the present invention, the device may include:

[0160] The time division module 401 is used to divide the preset detection period into a plurality of preset time periods;

[0161] The detection module 402 is used to perform different anomaly detection operations on the operation logs generated by the target system in the preset time period to obtain anomaly values ​​of different detection types corresponding to the preset time period;

[0162] The identification generating module 403 is used to convert the abnormal value into a discrete abnormal value, add a preset detection type identification to the discrete abnormal value to obtain detection identification data, and add a preset timestamp identification to the timestamp corresponding to the preset time period to obtain timestamp identification data;

[0163] The model detection module 404 is used to form an identification sequence with the timestamp identification data and the detection identification data corresponding to the preset time period, and use the pre-trained detection model to perform anomaly detection on the identification sequence to obtain the overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal according to the overall abnormality value; wherein the pre-trained detection model has learned the preset baseline identification sequence corresponding to each of the preset time periods, and the overall abnormality value represents the degree to which the identification sequence deviates from the preset baseline identification sequence.

[0164] Optionally, the device may further include:

[0165] A sampling rate setting module, used to determine the number of logs generated by the target system in each of the preset time periods, and set the sampling rate corresponding to each of the preset time periods according to the number of logs;

[0166] A sampling rate module, used to sample the operation logs generated by the target system in each of the preset time periods according to the sampling rate corresponding to each of the preset time periods;

[0167] The detection module 402 may also be used to:

[0168] Different anomaly detection operations are performed on the operation logs sampled in each of the preset time periods.

[0169] Optionally, the device may further include:

[0170] A log conversion module, used to convert the operation log generated by the target system in a preset time period into log identification data;

[0171] The model detection module 404 may include:

[0172] The sequence generation module is used to form the identification sequence by combining the timestamp identification data, the detection identification data and the log identification data corresponding to the preset time period.

[0173] Optionally, the detection module 402 may include:

[0174] A log sequence detection submodule, used for combining the operation logs into a log sequence, and inputting the log sequence into a pre-trained log sequence detection model to obtain an abnormal value of a log sequence detection type; wherein the pre-trained log sequence detection model has learned a preset normal log sequence, and the abnormal value of the log sequence detection type represents the degree to which the log sequence deviates from the normal log sequence;

[0175] and / or, a log structure detection submodule, for updating the operation log to a log parsing tree, and determining a degree of change of the log parsing tree between adjacent preset time periods, to obtain an abnormal value of a log structure detection type;

[0176] And / or, a log category detection submodule, for determining, according to the preset log category to which each of the operation logs belongs, a degree of change between the frequency of occurrence corresponding to each of the preset log categories in the current preset time period and the frequency of occurrence corresponding to each of the preset log categories in the previous preset time period, and obtaining an abnormal value of the log category detection type;

[0177] and / or, a discrete variable value detection submodule, for extracting discrete variable values ​​from the operation log, and determining a degree of change between the frequency of occurrence of each preset value corresponding to the discrete variable value in the current preset time period and the frequency of occurrence in the previous preset time period, to obtain an abnormal value of the discrete value detection type;

[0178] and / or, a cluster detection submodule, for extracting values ​​from the operation log, performing clustering processing on the values, and determining the deviation of values ​​not in the value cluster compared to the value cluster to obtain abnormal values ​​of the value cluster detection type;

[0179] And / or, a line graph detection submodule is used to convert multiple numerical values ​​of the same type in the operation log into a line graph, determine an outlier judgment numerical interval based on the line graph, and determine the proportion of numerical values ​​outside the outlier judgment numerical interval in all numerical values, so as to obtain an abnormal value of the numerical line graph detection type.

[0180] Optionally, the model detection module 404 may include:

[0181] A masking module, used for masking part of the identification data in the identification sequence to obtain a to-be-processed identification sequence; wherein the masked identification data in the to-be-processed identification sequence is the detection identification data or a combination of the timestamp identification data and the detection identification data;

[0182] A detection submodule, used for inputting the identification sequence to be processed into the pre-trained detection model, so that the pre-trained detection model predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed, and obtains predicted identification data;

[0183] The outlier value calculation submodule is used to calculate the loss value between the predicted identification data and the masked identification data using a preset loss function, and use the loss value as the overall outlier value.

[0184] Optionally, the model detection module 404 may include:

[0185] A sequence combination submodule, used to sequentially combine the timestamp identification data and the detection identification data corresponding to the preset time period into an identification sequence;

[0186] The model processing submodule is used to input the identification sequence to be processed into the pre-trained detection model so that the pre-trained detection model performs position encoding on each identification data in the identification sequence to be processed, and the pre-trained detection model predicts the covered identification data in the identification sequence to be processed according to the position encoding and the uncovered identification data in the identification sequence to be processed to obtain the predicted identification data.

[0187] Optionally, there is overlap between adjacent preset time periods;

[0188] The model detection module 404 may include:

[0189] A prediction submodule, used for inputting the identification sequence into the pre-trained detection model, so that the pre-trained detection model predicts the identification sequence of the next preset time period according to the identification sequence to obtain a predicted identification sequence;

[0190] The sequence abnormality calculation submodule is used to calculate the sequence loss value between the predicted identification sequence and the identification sequence of the next preset time period using a preset loss function, so as to determine whether the target system is abnormal according to the sequence abnormality value.

[0191] Optionally, the device may further include:

[0192] An acquisition module, used to acquire a baseline identification sequence of each of the preset time periods; wherein the baseline identification sequence includes timestamp identification data of the preset time periods;

[0193] A training data setting module, used for performing masking processing on part of the identification data in the baseline identification sequence to obtain a training identification sequence;

[0194] A training module, used for inputting the training identification sequence into an initial detection model, so that the initial detection model predicts the covered identification data in the training identification sequence according to the uncovered identification data in the training identification sequence, and obtains identification data to be compared;

[0195] A parameter updating module is used to calculate the training loss value between the identification data to be compared and the masked identification data in the training identification sequence using the preset loss function, and to update the parameters of the initial detection model according to the training loss value to obtain the pre-trained detection model.

[0196] Optionally, the device may further include:

[0197] A vocabulary setting module, used to construct a timestamp vocabulary according to the timestamp identification data corresponding to each of the preset time periods;

[0198] The vocabulary adding module is used to set the timestamp vocabulary for the initial detection model so that the initial detection model predicts the masked timestamp identification data according to the timestamp vocabulary.

[0199] Please refer to Figure 5 , Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention provides an electronic device 10, including a processor 11 and a memory 12; wherein the memory 12 is used to store a computer program; and the processor 11 is used to execute the anomaly detection method based on time information provided in the aforementioned embodiment when executing the computer program.

[0200] For the specific process of the above-mentioned anomaly detection method based on time information, reference may be made to the corresponding contents provided in the aforementioned embodiments, which will not be described in detail here.

[0201] Furthermore, the memory 12 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0202] In addition, the electronic device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15 and a communication bus 16; wherein the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0203] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the anomaly detection method based on time information as described in the above embodiment.

[0204] Since the embodiments of the computer program product part correspond to the embodiments of the anomaly detection method part based on time information, please refer to the description of the above embodiments for the embodiments of the computer program product part, which will not be repeated here.

[0205] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the anomaly detection method based on time information as described in the above embodiment is implemented.

[0206] Since the embodiments of the computer-readable storage medium part correspond to the embodiments of the anomaly detection method part based on time information, the embodiments of the storage medium part refer to the description of the above embodiments and will not be repeated here.

[0207] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0208] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0209] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0210] The above is a detailed introduction to the anomaly detection method, device, electronic device and medium based on time information provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for anomaly detection based on time information, characterized in that: include: Dividing a preset detection cycle into a plurality of preset time periods; Performing different anomaly detection operations on the operation logs generated by the target system during the preset time period to obtain anomaly values ​​of different detection types corresponding to the preset time period; Converting the abnormal value into a discrete abnormal value, adding a preset detection type identifier to the discrete abnormal value to obtain detection identifier data, and adding a preset timestamp identifier to the timestamp corresponding to the preset time period to obtain timestamp identifier data; The timestamp identification data and detection identification data corresponding to the preset time period are combined into an identification sequence, and the identification sequence is subjected to anomaly detection using a pre-trained detection model to obtain an overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal based on the overall abnormality value; wherein the pre-trained detection model has learned a preset baseline identification sequence corresponding to each of the preset time periods, and the overall abnormality value represents the degree to which the identification sequence deviates from the preset baseline identification sequence.

2. The anomaly detection method according to claim 1, characterized in that: Also includes: Determine the number of logs generated by the target system in each of the preset time periods, and set a sampling rate corresponding to each of the preset time periods according to the number of logs; Sampling the operation logs generated by the target system in each of the preset time periods according to the sampling rate corresponding to each of the preset time periods; The performing different anomaly detection operations on the operation logs generated by the target system in each of the preset time periods includes: Different anomaly detection operations are performed on the operation logs sampled in each of the preset time periods.

3. The anomaly detection method according to claim 1, characterized in that: Also includes: Converting the operation logs generated by the target system in a preset time period into log identification data; The step of forming an identification sequence from the timestamp identification data and the detection identification data corresponding to the preset time period includes: The timestamp identification data, the detection identification data, and the log identification data corresponding to the preset time period are combined into the identification sequence.

4. The anomaly detection method according to claim 1, characterized in that: The performing different anomaly detection operations on the operation logs generated by the target system during the preset time period includes: Combining the operation logs into a log sequence, and inputting the log sequence into a pre-trained log sequence detection model to obtain an abnormal value of the log sequence detection type; wherein the pre-trained log sequence detection model has learned a preset normal log sequence, and the abnormal value of the log sequence detection type represents the degree to which the log sequence deviates from the normal log sequence; and / or, updating the operation log to a log parsing tree, and determining a degree of change of the log parsing tree between adjacent preset time periods to obtain an abnormal value of a log structure detection type; And / or, according to the preset log category to which each of the operation logs belongs, determining the degree of change between the occurrence frequency corresponding to each of the preset log categories in the current preset time period and the occurrence frequency corresponding to each of the preset log categories in the previous preset time period, and obtaining an abnormal value of the log category detection type; and / or, extracting discrete variable values ​​from the operation log, and determining a degree of change between the frequency of occurrence of each preset value corresponding to the discrete variable value in the current preset time period and the frequency of occurrence in the previous preset time period, to obtain an abnormal value of the discrete value detection type; and / or, extracting values ​​from the operation log, clustering the values, and determining the deviation of values ​​not in the value cluster compared to the value cluster to obtain abnormal values ​​of the value cluster detection type; And / or, convert multiple numerical values ​​of the same type in the operation log into a line graph, determine the outlier judgment numerical interval based on the line graph, and determine the proportion of numerical values ​​outside the outlier judgment numerical interval in all numerical values ​​to obtain abnormal values ​​of the type detected by the numerical line graph.

5. The abnormality detection method according to any one of claims 1 to 4, characterized in that: The performing anomaly detection on the identification sequence using the pre-trained detection model to obtain the overall anomaly value of the target system corresponding to the preset time period includes: Performing masking processing on part of the identification data in the identification sequence to obtain an identification sequence to be processed; wherein the masked identification data in the identification sequence to be processed is the detection identification data or a combination of the timestamp identification data and the detection identification data; Inputting the identification sequence to be processed into the pre-trained detection model, so that the pre-trained detection model predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed, and obtains predicted identification data; A preset loss function is used to calculate a loss value between the predicted identification data and the masked identification data, and the loss value is used as the overall abnormal value.

6. The anomaly detection method according to claim 5, characterized in that: The step of forming an identification sequence from the timestamp identification data and the detection identification data corresponding to the preset time period includes: Sequentially compose an identification sequence of the timestamp identification data and the detection identification data corresponding to the preset time period; The step of inputting the identification sequence to be processed into the pre-trained detection model so that the pre-trained detection model predicts the covered identification data in the identification sequence to be processed according to the uncovered identification data in the identification sequence to be processed to obtain the predicted identification data comprises: The identification sequence to be processed is input into the pre-trained detection model so that the pre-trained detection model performs position encoding on each identification data in the identification sequence to be processed, and the pre-trained detection model predicts the covered identification data in the identification sequence to be processed according to the position encoding and the uncovered identification data in the identification sequence to be processed to obtain the predicted identification data.

7. The abnormality detection method according to claim 5, characterized in that: There is overlap between adjacent preset time periods; The method further comprises: Inputting the identification sequence into the pre-trained detection model, so that the pre-trained detection model predicts the identification sequence of the next preset time period according to the identification sequence to obtain a predicted identification sequence; A preset loss function is used to calculate a sequence loss value between the predicted identification sequence and an identification sequence of a next preset time period, so as to determine whether the target system is abnormal according to the sequence abnormality value.

8. The abnormality detection method according to claim 5, characterized in that: The training process of the pre-trained detection model includes: Acquire a baseline identification sequence of each of the preset time periods; wherein the baseline identification sequence includes timestamp identification data of the preset time periods; Performing masking processing on part of the identification data in the baseline identification sequence to obtain a training identification sequence; Inputting the training identification sequence into an initial detection model, so that the initial detection model predicts the covered identification data in the training identification sequence according to the uncovered identification data in the training identification sequence, to obtain identification data to be compared; The preset loss function is used to calculate the training loss value between the identification data to be compared and the masked identification data in the training identification sequence, and the parameters of the initial detection model are updated according to the training loss value to obtain the pre-trained detection model.

9. The abnormality detection method according to claim 8, characterized in that: Also includes: Constructing a timestamp vocabulary table according to the timestamp identification data corresponding to each of the preset time periods; The timestamp vocabulary is set for the initial detection model so that the initial detection model predicts the masked timestamp identification data according to the timestamp vocabulary.

10. An abnormality detection device based on time information, characterized in that: include: A time division module, used to divide a preset detection period into a plurality of preset time periods; A detection module, used to perform different anomaly detection operations on the operation logs generated by the target system in the preset time period, and obtain anomaly values ​​of different detection types corresponding to the preset time period; an identification generating module, configured to convert the abnormal value into a discrete abnormal value, add a preset detection type identification to the discrete abnormal value to obtain detection identification data, and add a preset timestamp identification to the timestamp corresponding to the preset time period to obtain timestamp identification data; A model detection module is used to form an identification sequence with the timestamp identification data and the detection identification data corresponding to the preset time period, and use a pre-trained detection model to perform anomaly detection on the identification sequence to obtain an overall abnormality value corresponding to the target system in the preset time period, so as to determine whether the target system is abnormal based on the overall abnormality value; wherein the pre-trained detection model has learned the preset baseline identification sequence corresponding to each of the preset time periods, and the overall abnormality value represents the degree to which the identification sequence deviates from the preset baseline identification sequence.

11. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the anomaly detection method based on time information as described in any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the anomaly detection method based on time information as described in any one of claims 1 to 9 is implemented.

Images