Abnormal log detection method and device and nonvolatile storage medium

Abstract

The invention discloses an abnormal log detection method and device and a nonvolatile storage medium. The method comprises the following steps: determining a digital sequence corresponding to a to-be-processed log to obtain a plurality of digital sequences; for each digital sequence, selecting a plurality of subsequences with a preset length from the digital sequence, and performing shielding processing on elements at a target position in each subsequence to obtain a plurality of target subsequences corresponding to each digital sequence; for each digital sequence, determining the probability distribution of each target sub-sequence at the target position to obtain a plurality of probability distributions, and determining the average value of the plurality of probability distributions as the target probability distribution of each digital sequence at the target position; and according to the target probability distribution of each digital sequence at the target position, determining whether the to-be-processed log corresponding to the digital sequence is an abnormal log at the target position. According to the method and the device, the technical problem of relatively low detection accuracy of the abnormal logs caused by a quantity-based abnormal log detection method is solved.

Description

Technical Field

[0001] The present application relates to the field of communication network maintenance, and in particular, to a method and device for detecting abnormal logs, and a non-volatile storage medium. Background Art

[0002] Logs are files that record various information generated during system operation, and they record the operation process of the system. In recent years, as the scale of various systems and applications has become larger and larger, and the structure has become more and more complex, the probability of failure has gradually increased. Therefore, the importance of abnormal log detection in the actual operation of the system has gradually increased.

[0003] To a certain extent, log data itself is similar to natural language, that is, logs can be regarded as natural language that follows the rules set by a certain program. Compared with normal language, they may be more limited and structured in vocabulary expression. Based on this feature, some natural language-like processing can be performed on logs. First, based on the structured characteristics of logs, that is, logs are all generated by program design, which means that the type of logs generated by a specific device is fixed, and their "structure" is set in advance by the program, and only the parameter part changes. Therefore, for a certain type of device, its log categories are limited. Therefore, structured or semi-structured logs can be parsed and then summarized into templates.

[0004] Word embedding is a technology commonly used in natural language processing. It can represent natural text language as digital vectors that are easy for computers to process. Words with similar meanings often have similar word vectors, and they are adjacent when mapped to the word vector space. Since logs can be generated by some fixed templates, this type of template can be represented by the same word vector, that is, the number of templates finally parsed is the number of word vectors.

[0005] Since the total number of templates is fixed, and what needs to be predicted is the probability of each log template appearing at the current position, we consider using mask prediction technology for templates during training. To improve the generalization performance of the model, a set of fixed-length input log group data is randomly masked during training, and the model is asked to predict the probability of each log appearing at the masked position. Since the masked position is random, the data for each training is different, which improves the generalization performance of the model.

[0006] The current abnormal log detection scheme uses the method of taking the first K logs with the highest probability as normal logs in the final output probability processing. If the current log is outside this range, it is considered abnormal. Since the K value required for this judgment scheme is difficult to set and will be affected by the number of templates, it is difficult to use in practice.

[0007] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0008] The present application provides a method and device for detecting an abnormal log, and a non-volatile storage medium, so as to at least solve the technical problem of low detection accuracy of abnormal logs caused by an abnormal log detection method based on quantity.

[0009] According to one aspect of the present application, a method for detecting abnormal logs is provided, including: obtaining multiple logs to be processed, determining a digital sequence corresponding to each log to be processed, and obtaining multiple digital sequences; for each digital sequence, selecting multiple subsequences of preset lengths in the digital sequence, and masking the elements at a target position in each subsequence to obtain multiple target subsequences corresponding to each digital sequence, wherein the preset length is less than the length of the digital sequence; for each digital sequence, determining the probability distribution of each target subsequence at a target position to obtain multiple probability distributions, and determining the average value of the multiple probability distributions as a target probability distribution of each digital sequence at the target position; and determining whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position according to the target probability distribution of each digital sequence at the target position.

[0010] Optionally, according to the target probability distribution of each digital sequence at the target position, determining whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position includes: sorting the target probability distributions in order from large to small, and summing the first i target probability distributions in the sorting results in order from i from 1 to n to obtain multiple summation results, wherein n is the number of target probability distributions, n is a positive integer greater than 1, and i is a positive integer not greater than n; determining a target summation result that meets a preset condition among the multiple summation results, wherein the preset conditions include: the m-1th summation result is less than a preset threshold, the mth summation result is greater than the preset threshold, the target summation result is the mth summation result, and m is a positive integer not greater than n; determining that the first log is a normal log at the target position, and determining that the second log is an abnormal log at the target position, wherein the first log is a log to be processed corresponding to the nth to mth target probability distributions, and the second log is a log to be processed corresponding to the m-1th to 1st target probability distributions.

[0011] Optionally, after determining that the second log is an abnormal log at the target position, the method further includes: generating alarm information, wherein the alarm information includes: a target digital sequence corresponding to the second log and a target position in the target digital sequence.

[0012] Optionally, after determining the probability distribution of each target subsequence at the target position, the method further includes: modifying the probability distribution by the following formula: Among them, fi (x) is the probability distribution of the target subsequence at the target position, p i is the probability distribution of the target subsequence after correction at the target position, A is the first scaling parameter, and B is the second scaling parameter.

[0013] Optionally, the first scaling parameter and the second scaling parameter are obtained by training through the following loss function: ∑ i y i log(p i )+(1-y i )log(1-p i ) where y i is the true probability distribution of the target subsequence at the target position.

[0014] Optionally, determining the probability distribution of each target subsequence at the target position includes: inputting the target subsequence into multiple different deep learning models, obtaining a first probability distribution of the target subsequence at the target position output by each deep learning model, and obtaining multiple first probability distributions of each target subsequence at the target position, wherein the deep learning model is obtained by training through historical log digital sequences; and performing weighted summation on the multiple first probability distributions to obtain the probability distribution of each target subsequence at the target position.

[0015] Optionally, determining the digital sequence corresponding to each log to be processed includes: extracting the log template of each log to be processed by a fixed-depth tree parsing method to obtain multiple log templates; distributing the logs to be processed corresponding to the log templates into digital sequences according to the log templates.

[0016] According to another aspect of the present application, there is also provided an abnormal log detection device, including: an acquisition module, used to acquire multiple logs to be processed, determine the digital sequence corresponding to each log to be processed, and obtain multiple digital sequences; a processing module, used to select multiple subsequences of preset lengths in the digital sequence for each digital sequence, and mask the elements at the target position in each subsequence to obtain multiple target subsequences corresponding to each digital sequence, wherein the preset length is less than the length of the digital sequence; a first determination module, used to determine the probability distribution of each target subsequence at the target position for each digital sequence, obtain multiple probability distributions, and determine the average value of the multiple probability distributions as the target probability distribution of each digital sequence at the target position; a second determination module, used to determine whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position according to the target probability distribution of each digital sequence at the target position.

[0017] According to another aspect of the present application, a non-volatile storage medium is provided, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above abnormal log detection method.

[0018] According to another aspect of the present application, an electronic device is provided, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the above abnormal log detection method is executed when the program is running.

[0019] According to yet another aspect of the present application, a computer program is provided, wherein when the computer program is executed by a processor, the above abnormal log detection method is implemented.

[0020] According to another aspect of the present application, a computer program product is provided, which includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above abnormal log detection method is implemented.

[0021] In the present application, a plurality of logs to be processed are obtained, and a digital sequence corresponding to each log to be processed is determined to obtain a plurality of digital sequences; for each digital sequence, a plurality of subsequences of preset lengths are selected from the digital sequence, and the elements at the target position in each subsequence are masked to obtain a plurality of target subsequences corresponding to each digital sequence, wherein the preset length is less than the length of the digital sequence; for each digital sequence, a probability distribution of each target subsequence at the target position is determined to obtain a plurality of probability distributions, and an average value of the plurality of probability distributions is determined as a target probability distribution of each digital sequence at the target position; according to the target probability distribution of each digital sequence at the target position, a method of determining whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position is adopted, thereby achieving the purpose of detecting abnormal logs based on confidence and random windows, thereby realizing the technical effect of improving the detection accuracy of abnormal logs, and further solving the technical problem of low detection accuracy of abnormal logs caused by the abnormal log detection method based on quantity. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a flow chart of a method for detecting abnormal logs according to an embodiment of the present application;

[0024] Figure 2 is a structural diagram of an abnormal log detection device according to an embodiment of the present application;

[0025] Figure 3 It is a hardware structure block diagram of a computer terminal according to an abnormal log detection method of an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] According to an embodiment of the present application, a method embodiment of a method for detecting abnormal logs is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 is a flow chart of a method for detecting abnormal logs according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0030] Step S102, obtaining a plurality of logs to be processed, determining a digital sequence corresponding to each log to be processed, and obtaining a plurality of digital sequences.

[0031] Specifically, by using the Drain method or other template extraction techniques, the original log data (logs to be processed) is parsed to summarize the log templates. Next, the log data is converted into a digital sequence. A digital identifier can be assigned to each template, thereby converting the structured or semi-structured log text into a digital form that the model can process. For example, if 100 log templates are parsed, each log template can be assigned a digital identifier from 1 to 100.

[0032] Step S104: For each digital sequence, multiple subsequences of a preset length are selected from the digital sequence, and the elements at the target positions in each subsequence are masked to obtain multiple target subsequences corresponding to each digital sequence, where the preset length is less than the length of the digital sequence.

[0033] Specifically, after obtaining the digital sequence corresponding to each log, multiple subsequences are randomly selected from these sequences, and the length of the subsequence is less than the length of the original sequence. Assume that the length of each log sequence is N, and the length of the selected subsequence is M (M < N), and the elements at specific target positions in the subsequence are masked. For example, if M = 50, then for each log sequence, multiple consecutive subsequences of length 50 can be randomly selected, and the elements at the target positions in the subsequence are masked to create multiple target subsequences.

[0034] Step S106: For each digital sequence, determine the probability distribution of each target subsequence at the target position to obtain multiple probability distributions, and determine the average value of the multiple probability distributions as the target probability distribution of each digital sequence at the target position.

[0035] It should be noted that when the above sequence is input into the trained model, the model can output the probability of each log template appearing at the target position of each subsequence. For each digital sequence, collect the probability distributions of all target subsequences at the target position and calculate the average value of these probability distributions to obtain the target probability distribution of each digital sequence at the target position.

[0036] For example, for the digital sequence [x1, x2, x3......x 4n , sequences composed of any 2n consecutive logs can be selected from it. Assume the following three groups of log sequences are selected: [x2, x3, x4......x 2n+1 , [x n , x n+1 , x n+2 ......x 3n-1 , [x 2n-2 , x 2n-1 , x 2n ......x 4n-3 .

[0037] At the same time, the x in the above three sets of digital sequences 2n The position (target position) is masked, and then the three groups of logs are input into the model respectively to obtain three groups of predicted probability distributions of the position. Then the three groups of probability distributions are added and averaged to obtain the denoised log probability distribution, that is, the target probability distribution at the target position.

[0038] The above is an example of the case where the number of windows is 3. In actual use, the number of windows can be set according to the needs for denoising. The experimental results show that after random window processing, a more accurate probability distribution can be obtained, which effectively improves the accuracy of the prediction.

[0039] Step S108, determining whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position according to the target probability distribution of each digital sequence at the target position.

[0040] In step S108, the obtained target probability distribution is used to determine whether the log to be processed is an abnormal log at the target location. For example, a confidence threshold can be set, and the probability distribution is sorted and added according to the confidence judgment method until the total probability of the addition exceeds the confidence threshold. If the template of the target log is not within the range of templates whose total probability exceeds the threshold, the log is considered to be abnormal at the target location.

[0041] According to the above steps, a plurality of logs to be processed are obtained, and a digital sequence corresponding to each log to be processed is determined to obtain a plurality of digital sequences; for each digital sequence, a plurality of subsequences of preset lengths are selected from the digital sequence, and the elements at the target position in each subsequence are masked to obtain a plurality of target subsequences corresponding to each digital sequence, wherein the preset length is less than the length of the digital sequence; for each digital sequence, a probability distribution of each target subsequence at the target position is determined to obtain a plurality of probability distributions, and an average value of the plurality of probability distributions is determined as a target probability distribution of each digital sequence at the target position; according to the target probability distribution of each digital sequence at the target position, whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position is determined, thereby achieving the purpose of detecting abnormal logs based on confidence and random windows, thereby realizing the technical effect of improving the detection accuracy of abnormal logs.

[0042] The following Figure 1 The steps shown are exemplary and explanatory.

[0043] According to some optional embodiments of the present application, according to the target probability distribution of each digital sequence at the target position, determining whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position can be achieved by the following method: sorting the target probability distributions in order from large to small, and summing the first i target probability distributions in the sorting results in order from i from 1 to n to obtain multiple summation results, wherein n is the number of target probability distributions, n is a positive integer greater than 1, and i is a positive integer not greater than n; determining a target summation result that meets a preset condition among the multiple summation results, wherein the preset condition includes: the m-1th summation result is less than a preset threshold, the mth summation result is greater than the preset threshold, the target summation result is the mth summation result, and m is a positive integer not greater than n; determining that the first log is a normal log at the target position, and determining that the second log is an abnormal log at the target position, wherein the first log is a log to be processed corresponding to the nth to mth target probability distributions, and the second log is a log to be processed corresponding to the m-1th to 1st target probability distributions.

[0044] In the above embodiment, first, the target probability distribution of each log template at the target position is sorted from large to small. This means that the probability value of each template will be arranged from the largest to the smallest according to its probability of appearing at the target position. For the sorted probability distribution, starting from the first one (i.e. the one with the largest probability), the first i probability values ​​are summed in order from i from 1 to n. n is the number of all target probability distributions, which is the total number of log templates. The summation results in an array, where S i Represents the cumulative sum of the previous i probabilities.

[0045] Next, we need to find the element that meets the preset condition in the generated summation result array. The preset condition is: the m-1th summation result S m-1 Less than the preset threshold S t , and the mth summation result S m Greater than or equal to the preset threshold S t This means that S m-1 <St<=S m The target sum is S m , which is the first result whose cumulative probability exceeds the preset threshold.

[0046] Finally, sum the results S according to the found target m , the logs corresponding to the first m templates of the corresponding probability distribution are judged as normal logs (the first log), and the logs before the mth template (i.e. the logs corresponding to the templates from m-1 to 1) are judged as abnormal logs (the second log). This is because S mrepresents the point at which the sum of the probabilities of the first m templates reaches or exceeds the confidence threshold for the first time, so all templates contributing to this sum are considered to be normal log templates at the given confidence level.

[0047] It is worth noting that the selection of the threshold node is the mth log that is just greater than the set threshold, rather than the m-1th log that is just less than the set threshold. This is because in actual tests, it is found that the probability of the logs sorted by probability is very high. If the threshold St is set to a small value (such as 0.5), the probability of the first log template may be greater than the set threshold. At this time, m-1=0, that is, there is no correct log. At this time, any log at this position will be judged as an error log, which will cause the log judgment process to fail to run normally.

[0048] It should be explained that in the relevant confidence-based abnormal log judgment method, the mainstream method is to sort the probabilities after obtaining the probability distribution of a certain position, and then take the first K logs as the correct prediction results. If the actual log is the first K logs, it is considered correct, otherwise it is considered an abnormal log. This method has the following problems: 1. Different K values ​​are required for different types of logs, and the applicability in actual use is not enough. If the total number of log types generated by a machine is 100, and the total number of log types generated by another machine is 1000, then this method of taking the correct number of logs in the former case is completely inapplicable to the other case, and the K values ​​they need to take are very different. 2. It cannot be guaranteed that the selected logs are of low probability. Usually, the low probability situation predicted by the log will be regarded as an erroneous log. However, the method of taking a fixed number of logs cannot guarantee that the erroneous logs judged are of low probability.

[0049] The above process of the embodiment of the present application ensures that abnormal log detection is not only based on the probability size predicted by the model, but also takes into account the confidence threshold, thereby providing a more flexible and accurate abnormal log judgment mechanism. This method avoids the limitations of directly setting the K value, because the setting of the K value usually requires prior knowledge of the number of log templates, and the K value may need to be adjusted for different types of logs. In contrast, the preset confidence threshold can be more stable and applicable to different types and numbers of log templates. In addition, by using cumulative probability and comparison, logs with abnormally low probabilities of appearing in a specific context can be more accurately identified, thereby improving the accuracy and efficiency of anomaly detection.

[0050] On the other hand, according to the target probability distribution of each digital sequence at the target position, determining whether the to-be-processed log corresponding to the digital sequence is an abnormal log at the target position can also be achieved by the following method:

[0051] Assuming that the obtained log probabilities are sorted from large to small, the obtained probability distribution is P, and the corresponding log template set is X, then P(X=x i )=p i . Sum the probabilities one by one and create a new array, counted as S, then we have Set a threshold St for the logs to be judged, sort the log probabilities from large to small and then add them up, and compare the results with the thresholds in turn. Then there exists m such that S m-1 <S t , and S m ≥S t , then if the template at this position belongs to the log template x1, x2, ...x m , it is judged as a correct log, otherwise it is judged as an error.

[0052] According to other optional embodiments of the present application, after determining that the second log is an abnormal log at the target position, the following steps may also be performed: generating alarm information, wherein the alarm information includes: a target digital sequence corresponding to the second log and a target position in the target digital sequence.

[0053] When the second log is identified as an abnormal log at the target location, detailed information related to the log is extracted, including but not limited to: the specific content of the log: that is, the original text information of the second log; the target digital sequence: this is the digital representation form into which the second log is converted during the log processing and model prediction process; the target location: the specific location information of the abnormal log in the digital sequence.

[0054] Format the above information into an alert message. A structured data packet or message needs to be created that contains all relevant information. The format of the alert message may vary depending on the specific application scenario or system requirements, and may include the following: Timestamp: Record the time when the anomaly log is detected for tracking and time series analysis; Anomaly log ID / content: Directly identify or contain information about the anomaly log to facilitate quick problem location; Target digital sequence and location: Provide the specific digital representation and location of the anomaly log in the processing flow to help analysts understand the context in which the anomaly occurred. After the alert message is formatted, it is sent to the operation and maintenance team or integrated into the monitoring system. This can be achieved in a variety of ways, including but not limited to: Email or instant message notification: Send the alert message directly to the relevant responsible person or team. System log or monitoring panel: Record the alert message in the system log or display it on the monitoring panel for continuous tracking and analysis. API call: If the alert system is integrated with other maintenance or automation tools, the alert message can be sent to these tools through API calls to trigger the corresponding response or repair process.

[0055] In some optional embodiments of the present application, after determining the probability distribution of each target subsequence at the target position, the following steps may be performed: the probability distribution is corrected by the following formula: Among them, f i (x) is the probability distribution of the target subsequence at the target position, p i is the probability distribution of the target subsequence after correction at the target position, A is the first scaling parameter, and B is the second scaling parameter.

[0056] Preferably, the first scaling parameter and the second scaling parameter are obtained by training through the following loss function: ∑ i y i log(p i )+(1-y i )log(1-p i ) where y i is the true probability distribution of the target subsequence at the target position.

[0057] In some optional embodiments, determining the probability distribution of each target subsequence at the target position can be achieved by the following method: inputting the target subsequence into multiple different deep learning models, obtaining the first probability distribution of the target subsequence at the target position output by each deep learning model, and obtaining multiple first probability distributions of each target subsequence at the target position, wherein the deep learning model is obtained by training through historical log digital sequences; performing weighted summation on the multiple first probability distributions to obtain the probability distribution of each target subsequence at the target position.

[0058] In the above embodiment, first, a plurality of different deep learning models are constructed or selected. These models can be based on the same architecture (such as Transformer) but with different parameter initialization, different training data sets (implemented by data segmentation or enhancement), or models using different hyperparameter settings. Each model is trained on a historical log digital sequence to learn the probability of occurrence of log templates at different positions. Each target subsequence (i.e., the subsequence after masking) is input into these multiple deep learning models respectively. Each model outputs the first probability distribution of the target subsequence at the target position, that is, the probability of occurrence of each log template predicted by the model. In this way, for each target subsequence, we will get a set of different first probability distributions. In order to integrate the prediction results of these models, a weighted sum is performed on the multiple first probability distributions of each target subsequence at the target position. The setting of the weight can be based on the performance of the model (such as the accuracy on the validation set), the complexity of the model, or a predefined equal weight scheme, which depends on the specific application scenario and the evaluation of the model performance. The result of the weighted summation is the probability distribution of the target subsequence at the target position, which combines the prediction information of multiple models and is usually more accurate and stable than the prediction of a single model. Finally, the probability distribution obtained after integration is used to detect abnormal logs. This can be done by following the previous confidence-based method, that is, sorting the probability distribution, calculating the cumulative probability, and comparing it with the preset confidence threshold to determine the abnormal logs.

[0059] In some optional embodiments of the present application, determining the digital sequence corresponding to each log to be processed can be achieved by the following method: extracting the log template of each log to be processed by a fixed-depth tree parsing method to obtain multiple log templates; distributing the logs to be processed corresponding to the log templates according to the log templates, and converting them into digital sequences.

[0060] In the above embodiment, a parsing method (such as Drain) with a fixed depth tree is used to parse each log to be processed. This method is based on the structured characteristics of the log. By setting the depth of the parsing tree, the log data can be classified according to the length and the first depth-2 tokens, thereby extracting the template of the log. The log template is the basic structure of the log message, which contains a fixed format and changing parameters. For example, a template may be "[timestamp][level][source]:[message]", where timestamp, level, and source are fixed tokens and message is a changing parameter. Once the log template is extracted, the template and log data need to be converted into a digital sequence so that the model can process it. Specifically, the following steps are included: assign a unique digital identifier to each template. These identifiers can be integers used to indicate the type or ID of the template. For example, if 100 different templates are parsed, each template can be assigned a digital identifier from 1 to 100. Replace each template in the log with its corresponding digital identifier. This will convert the log text into a sequence of numbers, i.e., a digital sequence. The digital sequence can be regarded as the input of the model, each of which represents a specific template of the log.

[0061] Figure 2 is a structural diagram of an abnormal log detection device according to an embodiment of the present application, such as Figure 2 As shown, the device comprises:

[0062] The acquisition module 22 is used to acquire multiple logs to be processed, determine the digital sequence corresponding to each log to be processed, and obtain multiple digital sequences.

[0063] The processing module 24 is used to select a plurality of subsequences of preset lengths in each digital sequence, and perform masking processing on the elements at the target positions in each subsequence to obtain a plurality of target subsequences corresponding to each digital sequence, wherein the preset length is less than the length of the digital sequence.

[0064] The first determination module 26 is used to determine the probability distribution of each target subsequence at the target position for each digital sequence, obtain multiple probability distributions, and determine the average value of the multiple probability distributions as the target probability distribution of each digital sequence at the target position.

[0065] The second determination module 28 is used to determine whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position according to the target probability distribution of each digital sequence at the target position.

[0066] Optionally, the second determination module 28 is further used to perform the following steps: sort the target probability distributions in order from large to small, and sum the first i target probability distributions in the sorted results in order from i from 1 to n to obtain multiple summation results, wherein n is the number of target probability distributions, n is a positive integer greater than 1, and i is a positive integer not greater than n; determine the target summation result that meets the preset conditions among the multiple summation results, wherein the preset conditions include: the m-1th summation result is less than a preset threshold, the mth summation result is greater than a preset threshold, the target summation result is the mth summation result, and m is a positive integer not greater than n; determine that the first log is a normal log at the target location, and determine that the second log is an abnormal log at the target location, wherein the first log is a log to be processed corresponding to the nth to mth target probability distributions, and the second log is a log to be processed corresponding to the m-1th to 1st target probability distributions.

[0067] Optionally, the abnormal log detection device is further used to perform the following steps after determining that the second log is an abnormal log at the target position: generate alarm information, wherein the alarm information includes: a target digital sequence corresponding to the second log and a target position in the target digital sequence.

[0068] Optionally, the abnormal log detection device is further used to perform the following steps after determining the probability distribution of each target subsequence at the target position: modifying the probability distribution by the following formula: Among them, f i (x) is the probability distribution of the target subsequence at the target position, p i is the probability distribution of the target subsequence after correction at the target position, A is the first scaling parameter, and B is the second scaling parameter.

[0069] Optionally, the first scaling parameter and the second scaling parameter are obtained by training through the following loss function: ∑ i y i log(p i )+(1-y i )log(1-p i ) where y i is the true probability distribution of the target subsequence at the target position.

[0070] Optionally, the first determination module 26 is also used to perform the following steps: input the target subsequence into multiple different deep learning models, obtain the first probability distribution of the target subsequence at the target position output by each deep learning model, and obtain multiple first probability distributions of each target subsequence at the target position, wherein the deep learning model is obtained by training through historical log digital sequences; perform weighted summation on the multiple first probability distributions to obtain the probability distribution of each target subsequence at the target position.

[0071] Optionally, the acquisition module 22 is further used to perform the following steps: extracting the log template of each log to be processed by a fixed-depth tree parsing method to obtain multiple log templates; distributing the logs to be processed corresponding to the log templates into digital sequences according to the log templates.

[0072] It should be noted that the above Figure 2 The modules in the embodiment may be program modules (e.g., a set of program instructions for implementing a specific function) or hardware modules. For the latter, they may be expressed in the following forms, but are not limited thereto: the expression form of the above modules is a processor, or the functions of the above modules are implemented by a processor.

[0073] It should be noted that Figure 2 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0074] Figure 3 The hardware structure block diagram of a computer terminal for implementing a method for detecting abnormal logs is shown. Figure 3 As shown, the computer terminal 30 may include one or more (302a, 302b, ..., 302n are used to illustrate) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission module 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations shown.

[0075] It should be noted that the one or more processors 302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 30. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0076] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the detection method of the abnormal log in the embodiment of the present application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, the detection method of the abnormal log described above is realized. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the computer terminal 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0077] The transmission module 306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 30. In one example, the transmission module 306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0078] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 30 .

[0079] It should be noted that, in some optional embodiments, the above Figure 3 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 3 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0080] It should be noted that Figure 3 The computer terminal shown is used to execute Figure 1 The detection method of the abnormal log shown in the figure, therefore the relevant explanations in the execution method of the above command are also applicable to the electronic device and will not be repeated here.

[0081] An embodiment of the present application further provides a non-volatile storage medium, the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above abnormal log detection method.

[0082] A program for a non-volatile storage medium to perform the following functions: obtaining a plurality of logs to be processed, determining a digital sequence corresponding to each log to be processed, and obtaining a plurality of digital sequences; for each digital sequence, selecting a plurality of subsequences of preset lengths in the digital sequence, and performing masking processing on elements at a target position in each subsequence, and obtaining a plurality of target subsequences corresponding to each digital sequence, wherein the preset length is less than the length of the digital sequence; for each digital sequence, determining a probability distribution of each target subsequence at a target position, and obtaining a plurality of probability distributions, and determining an average value of the plurality of probability distributions as a target probability distribution of each digital sequence at a target position; and determining whether a log to be processed corresponding to the digital sequence is an abnormal log at the target position according to the target probability distribution of each digital sequence at the target position.

[0083] An embodiment of the present application further provides an electronic device, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the above abnormal log detection method is executed when the program is running.

[0084] The processor is used to run a program that performs the following functions: obtain multiple logs to be processed, determine the digital sequence corresponding to each log to be processed, and obtain multiple digital sequences; for each digital sequence, select multiple subsequences of preset lengths in the digital sequence, and mask the elements at the target position in each subsequence to obtain multiple target subsequences corresponding to each digital sequence, wherein the preset length is less than the length of the digital sequence; for each digital sequence, determine the probability distribution of each target subsequence at the target position to obtain multiple probability distributions, and determine the average value of the multiple probability distributions as the target probability distribution of each digital sequence at the target position; according to the target probability distribution of each digital sequence at the target position, determine whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position.

[0085] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0086] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0087] In the above-mentioned embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary protection measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0089] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0090] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0091] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0092] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for detecting abnormal logs, characterized in that: include: Acquire multiple logs to be processed, determine a digital sequence corresponding to each of the logs to be processed, and obtain multiple digital sequences; For each of the digital sequences, multiple subsequences of preset length are selected from the digital sequence, and elements at target positions in each of the subsequences are masked to obtain multiple target subsequences corresponding to each of the digital sequences, wherein the preset length is less than the length of the digital sequence; For each of the digital sequences, determine the probability distribution of each of the target subsequences at the target position to obtain a plurality of probability distributions, and determine the average of the plurality of probability distributions as the target probability distribution of each of the digital sequences at the target position; According to the target probability distribution of each of the digital sequences at the target position, it is determined whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position.

2. The method according to claim 1, characterized in that Determining whether a to-be-processed log corresponding to the digital sequence is an abnormal log at the target location according to the target probability distribution of each digital sequence at the target location includes: Sorting the target probability distributions in order from large to small, and summing the first i target probability distributions in the sorted results in order from i from 1 to n, to obtain multiple summation results, wherein n is the number of the target probability distributions, n is a positive integer greater than 1, and i is a positive integer not greater than n; Determine a target summation result that satisfies a preset condition from among the multiple summation results, wherein the preset condition includes: the m-1th summation result is less than a preset threshold, the mth summation result is greater than the preset threshold, the target summation result is the mth summation result, and m is a positive integer not greater than n; Determine that the first log is a normal log at the target location, and determine that the second log is an abnormal log at the target location, wherein the first log is a log to be processed corresponding to the nth to mth target probability distribution, and the second log is a log to be processed corresponding to the m-1th to 1st target probability distribution.

3. The method according to claim 1, characterized in that After determining that the second log is an abnormal log at the target position, the method further includes: generating alarm information, wherein the alarm information includes: a target digital sequence corresponding to the second log and a target position in the target digital sequence.

4. The method according to claim 1, characterized in that: After determining the probability distribution of each target subsequence at the target position, the method further includes: modifying the probability distribution by the following formula: Among them, f i (x) is the probability distribution of the target subsequence at the target position, p i is the probability distribution of the target subsequence after correction at the target position, A is the first scaling parameter, and B is the second scaling parameter.

5. The method according to claim 4, characterized in that The first scaling parameter and the second scaling parameter are obtained by training through the following loss function: Among them, y i is the true probability distribution of the target subsequence at the target position.

6. The method according to claim 1, characterized in that Determining the probability distribution of each of the target subsequences at the target position includes: Input the target subsequence into a plurality of different deep learning models, obtain a first probability distribution of the target subsequence at the target position output by each of the deep learning models, and obtain a plurality of first probability distributions of each of the target subsequences at the target position, wherein the deep learning model is obtained by training with a historical log digital sequence; A weighted sum is performed on the multiple first probability distributions to obtain a probability distribution of each target subsequence at the target position.

7. The method according to claim 1, characterized in that Determining the digital sequence corresponding to each of the logs to be processed includes: Extracting the log template of each of the logs to be processed by a fixed-depth tree parsing method to obtain multiple log templates; Distribution: According to the log template, the log to be processed corresponding to the log template is converted into a digital sequence.

8. A device for detecting abnormal logs, characterized in that: include: An acquisition module is used to acquire multiple logs to be processed, determine a digital sequence corresponding to each of the logs to be processed, and obtain multiple digital sequences; a processing module, configured to select, for each of the digital sequences, a plurality of subsequences of a preset length from the digital sequence, and perform masking processing on elements at target positions in each of the subsequences to obtain a plurality of target subsequences corresponding to each of the digital sequences, wherein the preset length is less than the length of the digital sequence; A first determination module is used to determine, for each of the digital sequences, a probability distribution of each of the target subsequences at the target position, obtain a plurality of probability distributions, and determine an average value of the plurality of probability distributions as a target probability distribution of each of the digital sequences at the target position; The second determination module is used to determine whether the log to be processed corresponding to the digital sequence is an abnormal log at the target position according to the target probability distribution of each digital sequence at the target position.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the abnormal log detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the abnormal log detection method according to any one of claims 1 to 7 when running.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the abnormal log detection method according to any one of claims 1 to 7 is implemented.

Images