A real-time abnormal log detection method, device, computer device and storage medium

Abstract

This application relates to the field of big data processing technology, and in particular, to a real-time abnormal log detection method, device, computer device, storage medium, and computer program product. The method includes: obtaining log data; inputting the log data into a lightweight log detection model to obtain an abnormal detection result corresponding to the log data; before inputting the log data into the lightweight log detection model to obtain an abnormal detection result corresponding to the log data, it further includes: obtaining historical log data, training an initial abnormal detection model based on the historical log data, where the initial abnormal detection model is used to implement abnormal log detection; removing target feature channels in the initial abnormal detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model. Using this method can reduce the redundant computation amount in the log detection model and improve the efficiency of abnormal log detection.

Description

Technical Field

[0001] This application relates to the field of big data processing technologies, and in particular, to a real-time abnormal log detection method, device, computer device, storage medium, and computer program product. Background Art

[0002] A log file is a record file or a set of files used to record system operation events, which can be divided into event logs and message logs. Log files play important roles such as processing historical data, tracing for diagnosing problems, and understanding system activities. Log files can record events occurring during system execution to provide a trace that can be used to understand system activities and diagnose problems.

[0003] With the rapid development of Internet technologies, during the daily operation of network systems, the number of generated log files has increased exponentially. Some information or related data in these log files can reflect the running state of the current network system. By performing log analysis, the occurrence of faults can be effectively predicted, thereby avoiding additional losses on the business side. From the dimension of data volume, it is statistically shown that a medium-sized enterprise can generate more than 100 GB of log files per day, with approximately 1,150 pieces of data being recorded per second on average, and even reaching 23,000 pieces of data per second during peak periods. In the traditional data operation and maintenance system, the manual processing method is obviously unable to handle such a large amount of data.

[0004] Currently, to solve the problem of a large workload for detecting log files, the method of training an artificial intelligence model is usually adopted, and the automated detection of log files is achieved by means of the trained model, thereby replacing the manual discrimination work.

[0005] However, the current log file detection methods have the following technical problems:

[0006] In the work of detecting log files using an artificial intelligence model, the artificial intelligence model needs to perform identification steps such as feature extraction on each log file, and the detection and identification efficiency is relatively low. Summary of the Invention

[0007] Based on this, in view of the above technical problems, it is necessary to provide an abnormal log detection method, device, computer device, computer-readable storage medium, and computer program product that can reduce the redundant computational amount in the log detection model and improve the abnormal log detection efficiency.

[0008] In a first aspect, this application provides a real-time abnormal log detection method. The method includes:

[0009] Obtain log data;

[0010] Input the log data into a lightweight log detection model to obtain an anomaly detection result corresponding to the log data;

[0011] Before inputting the log data into the lightweight log detection model to obtain an anomaly detection result corresponding to the log data, it further includes:

[0012] Obtain historical log data and train an initial anomaly detection model based on the historical log data. The initial anomaly detection model is used to implement anomaly log detection;

[0013] Remove the target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model.

[0014] In one embodiment, before removing the target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model, it further includes:

[0015] Randomly remove the feature channels in each convolutional layer of the initial anomaly detection model according to a preset proportional gradient;

[0016] Obtain the robustness verification parameters for testing the anomaly detection model after randomly removing the feature channels;

[0017] When the robustness verification parameters reach a preset robustness threshold range, record the current removal ratio of the feature channels and set the removal ratio as the pruning ratio of the current hidden layer.

[0018] In one embodiment, removing the target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model includes:

[0019] Input the historical log data into the initial anomaly detection model, and obtain the feature map set output by the target convolutional layer in the initial anomaly detection model. The feature map set includes the feature components corresponding to each feature channel in the target convolutional layer;

[0020] Perform downsampling processing on the feature map set to obtain the feature scalars corresponding to the feature components;

[0021] Determine the importance ranking of the feature channels based on the feature scalars, and select the target feature channels with the number satisfying the pruning ratio in the target convolutional layer in reverse order according to the importance ranking.

[0022] In one embodiment, removing the target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model includes:

[0023] Remove the target feature channels in the initial anomaly detection model layer by layer to obtain the first log detection model;

[0024] Construct a fine-tuning training set based on the historical log data, and use the fine-tuning training set to train the first log detection model until convergence to obtain the lightweight log detection model.

[0025] In one embodiment, the constructing a fine-tuning training set based on the historical log data, and using the fine-tuning training set to train the first log detection model until convergence to obtain the lightweight log detection model includes:

[0026] Input the historical log data into the initial anomaly detection model, and use the probability distribution output by the initial anomaly detection model as the auxiliary label of the historical log data;

[0027] Obtain the true label of the historical log data, where the true label is the manual annotation of the historical log data in advance;

[0028] Calculate the output probability distribution of the first log detection model based on the true label and the auxiliary label until the loss function converges to the global optimal solution.

[0029] In one embodiment, the calculating the output probability distribution of the first log detection model based on the true label and the auxiliary label until the loss function converges to the global optimal solution includes:

[0030] Set a first weight coefficient to adjust the influence degrees of the true label and the auxiliary label on the loss function by the first weight system.

[0031] In a second aspect, the present application further provides a real-time anomaly log detection device. The device includes:

[0032] A log data module, configured to obtain log data;

[0033] An anomaly detection module, configured to input the log data into the lightweight log detection model to obtain an anomaly detection result corresponding to the log data;

[0034] Before the anomaly detection module, it further includes:

[0035] An initial model module, configured to obtain historical log data, and train an initial anomaly detection model based on the historical log data, where the initial anomaly detection model is used to implement anomaly log detection;

[0036] A lightweight model module, configured to remove the target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model.

[0037] In one embodiment, before the lightweight model module, there is further included:

[0038] A random removal module, configured to randomly remove the feature channels in each convolutional layer of the initial anomaly detection model according to a preset proportional gradient;

[0039] A robustness verification module, configured to obtain the robustness verification parameters of the test anomaly detection model after randomly removing the feature channels;

[0040] A pruning ratio determination module, configured to record the current removal ratio of the feature channels and set the removal ratio as the pruning ratio of the current hidden layer when the robustness verification parameters reach a preset robustness threshold range.

[0041] In one embodiment, the lightweight model module includes:

[0042] A feature mapping module, configured to input the historical log data into the initial anomaly detection model to obtain a feature mapping set output by a target convolutional layer in the initial anomaly detection model, where the feature mapping set includes feature components corresponding to each feature channel in the target convolutional layer;

[0043] A feature scalar module, configured to perform downsampling processing on the feature mapping set to obtain feature scalars corresponding to the feature components;

[0044] An importance ranking module, configured to determine the importance ranking of the feature channels based on the feature scalars, and select, in reverse order according to the importance ranking, the target feature channels with a quantity that satisfies the pruning ratio in the target convolutional layer.

[0045] In one embodiment, the lightweight model module includes:

[0046] A first log model module, configured to layer - by - layer remove the target feature channels in the initial anomaly detection model to obtain the first log detection model;

[0047] A model fine - tuning module, configured to construct a fine - tuning training set based on the historical log data, and use the fine - tuning training set to train the first log detection model until convergence to obtain the lightweight log detection model.

[0048] In one embodiment, the model fine - tuning module includes:

[0049] An auxiliary label module, configured to input the historical log data into the initial anomaly detection model, and use the probability distribution output by the initial anomaly detection model as the auxiliary label of the historical log data;

[0050] A true label module for obtaining the true labels of the historical log data, where the true labels are manual annotations pre-made for the historical log data;

[0051] A loss function module for calculating the output probability distribution of the first log detection model based on the true labels and the auxiliary labels until the loss function converges to the global optimal solution.

[0052] In one embodiment, the loss function module includes:

[0053] A weight coefficient module for setting a first weight coefficient to adjust the influence degrees of the true labels and the auxiliary labels on the loss function with the first weight system.

[0054] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in a real-time abnormal log detection method as described in any one of the embodiments in the first aspect are implemented.

[0055] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the steps in a real-time abnormal log detection method as described in any one of the embodiments in the first aspect are implemented.

[0056] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in a real-time abnormal log detection method as described in any one of the embodiments in the first aspect are implemented.

[0057] For the above abnormal log detection method, device, computer device, storage medium, and computer program product, through derivation of the technical features in the independent claim, the following beneficial effects corresponding to the background technology can be achieved:

[0058] After obtaining the log data, a lightweight log detection model is used to automatically determine whether the log data is abnormal. The lightweight log detection model is a log detection model obtained by removing the target feature channels in each convolutional layer of the initial anomaly detection model, which is trained using historical log data. In implementation, due to the removal of the target feature channels, the lightweight log detection model can implement the final result calculation and output with relatively fewer floating-point operation amounts during operation, thereby helping to improve the efficiency of detecting a large amount of log data. On the other hand, during the process of processing from the initial anomaly detection model to the lightweight log detection model, the removed feature channels are the specified target feature channels, which helps technicians retain the feature channels that have too much influence on the accuracy of the log detection result, thereby helping to ensure the accuracy of the model output result on the basis of improving the model output efficiency. Description of the Drawings

[0059] Figure 1 It is a schematic diagram of the first process of a real-time anomaly log detection method in an embodiment;

[0060] Figure 2 It is a schematic diagram of the second process of a real-time anomaly log detection method in another embodiment;

[0061] Figure 3 It is a schematic diagram of the third process of a real-time anomaly log detection method in another embodiment;

[0062] Figure 4 It is a schematic diagram of the fourth process of a real-time anomaly log detection method in another embodiment;

[0063] Figure 5 It is a schematic diagram of the fifth process of a real-time anomaly log detection method in another embodiment;

[0064] Figure 6 It is a schematic diagram of the sixth process of a real-time anomaly log detection method in another embodiment;

[0065] Figure 7 It is a structural block diagram of a real-time anomaly log detection device in an embodiment;

[0066] Figure 8 It is an internal structure diagram of a computer device in an embodiment. Detailed Embodiments

[0067] In order to make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] Currently, in order to solve the problem of large workload in detecting log files, the method of training an artificial intelligence model is usually adopted, and the trained model is used to realize the automatic detection of log files, thereby replacing the manual discrimination work.

[0069] However, the current log file detection method has the following technical problems:

[0070] In the work of detecting log files using an artificial intelligence model, the artificial intelligence model needs to perform identification steps such as feature extraction on each log file, and the detection and identification efficiency is low.

[0071] Based on this, in one embodiment, as Figure 1 shown, the present application provides a real-time abnormal log detection method. In this embodiment, taking the application of this method to a terminal as an example, it can be understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is realized through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0072] Step 102: Obtain log data.

[0073] Among them, log data can refer to the data in a file used to record events occurring in the system. Log data can be in units of events, and log data can include business information, system hardware information, performance metrics, etc.

[0074] Exemplarily, the terminal can obtain the log data generated during the operation of the system on the premise of obtaining sufficient authorization from the system background.

[0075] Step 104: Input the log data into a lightweight log detection model to obtain an abnormal detection result corresponding to the log data.

[0076] Among them, the lightweight log detection model can refer to an intelligent model used to extract, identify, and output a discrimination result for the information in the log data.

[0077] Exemplarily, after obtaining the log data, the terminal can input the log data into the lightweight log detection model, and finally, what is output by the lightweight log detection model is the abnormal detection result.

[0078] Before the step 104, it further includes:

[0079] Step 106: Obtain historical log data, and train an initial abnormal detection model based on the historical log data. The initial abnormal detection model is used to realize the detection of abnormal logs.

[0080] Exemplarily, a conventional anomaly detection model can be trained based on historical log data. In implementation, the terminal can obtain historical log data on the premise of obtaining sufficient authorization. The historical log data can be data collected during the historical operation of the system, or data obtained from an external database.

[0081] In this way, after obtaining the historical log data and training, the terminal can obtain an initial anomaly detection model. At this time, the initial anomaly detection model can be used to implement anomaly log detection, that is, input the log data to be detected and output the discrimination result of the log data.

[0082] Step 108: Remove the target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model.

[0083] Among them, pruning can refer to the process of deleting or ignoring unnecessary calculation processes in the model through some judgment methods. A feature can refer to the data extracted from the log data for expressing the information contained in the log data, and a feature channel can refer to the feature component of the input information in the log data.

[0084] Exemplarily, after the terminal obtains the initial anomaly detection model through training, the initial anomaly detection model at this time includes all the traversal calculation processes for feature processing of the log data. Among them, a part has a strong correlation with the detection result of the anomaly log, while the other part has little impact on the detection result of the anomaly log. In order to improve the detection efficiency of the log data, the terminal can select a part of the target feature channels that have little impact on the detection result of the anomaly log and remove the target feature channels, so as to implement the pruning process of the initial anomaly detection model. In addition, since the model usually processes data in units of convolutional layers, the pruning process of the initial anomaly detection model can be implemented for each convolutional layer in the initial anomaly detection model. Further, in order to ensure that the output result remains accurate after the model reduces the calculation amount of traversal calculation, the terminal can gradually prune the model, alternating pruning and fine-tuning, so as to reduce the possibility of degradation of the final lightweight log detection model caused by removing too many feature channels at one time. Specifically, the target pruning rate can be set in advance by technicians, or determined by the terminal through pre-calculation steps, and the pruning ratios of each convolutional layer in the model can be set independently.

[0085] The above real-time anomaly log detection method can produce the following beneficial effects in solving the technical problems proposed in the background technology during implementation:

[0086] After obtaining the log data, a lightweight log detection model is used to automatically determine whether there are abnormalities in the log data. The lightweight log detection model is a log detection model obtained by removing the target feature channels in each convolutional layer of the initial anomaly detection model trained using historical log data. In implementation, due to the removal of the target feature channels, the lightweight log detection model can implement the inference process of the model with relatively less computational effort during operation, which helps to improve the real-time performance of log detection. On the other hand, during the evolution from the initial anomaly detection model to the lightweight log detection model, the removed feature channels are the designated target feature channels, which helps the technicians to retain the feature channels that have too much influence on the accuracy of the log detection results, thus helping to ensure the accuracy of the model output results on the basis of improving the model inference efficiency.

[0087] In one embodiment, as Figure 2 shown, before step 108, it further includes:

[0088] Step 202: Randomly remove the feature channels in each convolutional layer of the initial anomaly detection model according to a preset proportional gradient.

[0089] Among them, the convolutional layer can refer to the operation module in the neural network model for extracting the features of the input data.

[0090] Exemplarily, in order to determine the pruning ratio of the initial anomaly detection model, the terminal can randomly remove the feature channels in each convolutional layer of the initial anomaly detection model according to a certain ratio. Specifically, the terminal can remove the feature channels in each convolutional layer of the initial anomaly detection model according to a preset proportional gradient, so as to obtain the anomaly detection models of each convolutional layer under different pruning ratios.

[0091] Step 204: Obtain the robustness verification parameters of the test anomaly detection model after randomly removing the feature channels.

[0092] Among them, the robustness can refer to the characteristic of the model to maintain the accuracy of the log anomaly discrimination result of its output.

[0093] Exemplarily, after randomly pruning each layer of the initial anomaly detection model, the terminal can construct a set of test data, and determine the discrimination performance of the initial anomaly detection model after random pruning, that is, the robustness of the initial anomaly detection model, based on the robustness verification parameters output when the test data is input into the randomly pruned initial anomaly detection model. Specifically, the terminal can obtain the loss value of the initial anomaly detection model after random pruning, and use the loss value as a kind of robustness verification parameter. The greater the change range of the loss value, the greater the contribution of the feature channels included in this convolutional layer to the correct judgment of the model, and the smaller the robustness of this layer. It should be noted that the loss value is one kind of robustness verification parameter, but the robustness verification parameter is not limited to the loss value.

[0094] Step 206: When the robustness verification parameter reaches the preset robustness threshold range, record the current removal ratio of the feature channels, and set the removal ratio as the pruning ratio of the current convolutional layer.

[0095] Exemplarily, after obtaining the robustness verification parameter, the terminal can determine the robustness of a specific convolutional layer in the initial anomaly detection model under the current pruning ratio according to the robustness verification parameter. At this time, the stronger the robustness, the smaller the impact of the current pruning ratio on the output performance of the model. At this time, the terminal can increase the pruning ratio of the current convolutional layer and obtain the corresponding robustness verification parameter again. In this way, in the iteration, the terminal can preset the robustness threshold range, compare the robustness threshold range with the robustness verification parameters of each convolutional layer of the model, and finally obtain the limit value of the pruning ratio in each convolutional layer of the initial anomaly detection model. This limit value can be used as the pruning ratio for actually pruning the initial anomaly detection model.

[0096] In this embodiment, by removing a certain proportion of feature channels from each convolutional layer of the initial anomaly detection model and gradually iterating, the final pruning ratio is obtained. This helps to ensure that after finally pruning the initial anomaly detection model, the final lightweight log detection model can maintain a certain robustness, thereby reducing the possibility of accuracy degradation of the lightweight log detection model due to pruning, and ultimately helping to improve the accuracy of the log anomaly detection result.

[0097] In one embodiment, in order to determine the target feature channels to be removed, as Figure 3 shown, the step 108 includes:

[0098] Step 302: Input the historical log data into the initial anomaly detection model, and obtain the feature map set output by the target convolutional layer in the initial anomaly detection model. The feature map set includes the feature components corresponding to each feature channel in the target convolutional layer.

[0099] Among them, the feature map set may refer to the set of feature components corresponding to each feature channel in the convolutional layer.

[0100] Exemplarily, the terminal may input the historical log data into the initial anomaly detection model. At this time, the input historical log data may be The historical log data may be a two-dimensional or multi-dimensional vector. In the processing of the model, a convolutional layer or other feature abstraction layer layer may be used to process the historical log data, so that the terminal can obtain the set of feature components after dimensionality reduction of the historical log data, that is, the feature map set Among them represents the scale of the data, represents the number of feature channels.

[0101] Step 304: Perform downsampling processing on the feature map set to obtain a feature scalar corresponding to the feature component.

[0102] Among them, downsampling processing may refer to the process of reducing the number of sampling points in the matrix space and increasing the receptive field.

[0103] First, the downsampling operation is performed first. Downsampling processing may refer to the process of reducing the number of sampling points in the matrix space and increasing the receptive field.

[0104] Exemplarily, to avoid significant computational costs in the FMS, the terminal may perform downsampling processing on the feature map set. The downsampling function DS(·) may be used to convert the spatial features into a scalar As follows:

[0105]

[0106] Here can be passed through in the way of C TH feature components are converted into a scalar, and then the gating unit can be used to quantify channel saliency:

[0107]

[0108] Among them As the weight matrix can be further decomposed by the singular value decomposition algorithm (SVD) to reduce the computational amount, for this is usually initialized to 0. The gating unit finally outputs the feature scalar as the saliency index of its corresponding feature component. This quantitative indicator can be applied to its corresponding feature component through dot product, thereby stimulating or suppressing the activity of the feature component and adjusting it with the help of back propagation.

[0109] Step 306: Determine the importance ranking of the feature channels based on the feature scalar, and select the target feature channels whose number meets the pruning ratio in the target convolution layer in reverse order of the importance ranking.

[0110] For example, after obtaining the feature scalar corresponding to the feature component in the feature map set, the terminal can determine the importance of the feature channel based on the value of the feature scalar. The terminal can quantify the channel significance through a gating unit. In implementation, the terminal can use the SVD algorithm to decompose the weight matrix of the gating unit, thereby reducing the amount of computation while increasing nonlinearity.

[0111] In this way, the terminal can obtain the importance of each feature channel and sort the feature channels according to the obtained importance. Therefore, in the pruning process of removing feature channels, the feature channels of the initial anomaly detection model are removed in order of importance from small to large to a number that meets the pruning ratio. The removed feature channels are the target feature channels.

[0112] In this embodiment, by quantifying the importance of the feature channels of the model, it is helpful to preferentially remove a portion of the feature channels with the least importance to the model during the pruning process, thereby helping to improve the effectiveness of the retained feature channels and ultimately helping to improve the accuracy of the model's output results.

[0113] In one embodiment, the Figure 4 As shown, step 108 includes:

[0114] Step 402: Remove target feature channels in the initial anomaly detection model layer by layer to obtain the first log detection model.

[0115] Exemplarily, after pruning the initial anomaly detection model, the terminal may obtain a first log detection model.

[0116] Step 404: construct a fine-tuning training set based on the historical log data, and use the fine-tuning training set to train the first log detection model until convergence to obtain the lightweight log detection model.

[0117] The fine-tuning training set may include log data collected by the terminal, and may also include data with a probability distribution output by the training model as a label.

[0118] Exemplarily, compared with the initial anomaly detection model, the first log detection model obtained after pruning has the difference that some feature channels are removed, and other model parameters are the same. At this time, the terminal can construct a fine-tuning training set based on historical log data, and use the fine-tuning training set to train the first log detection model, and use the model after training convergence at this time as the lightweight log detection model.

[0119] In this embodiment, by performing secondary fine-tuning training on the first log detection model, it helps to make the finally applied lightweight log detection model further meet the requirements of log anomaly detection, thereby helping to further improve the accuracy of the output results of the lightweight log detection model.

[0120] In one embodiment, as Figure 5 shown, step 404 includes:

[0121] Step 502: Input the historical log data into the initial anomaly detection model, and use the output result of the initial anomaly detection model as the auxiliary label of the historical log data.

[0122] Exemplarily, during the fine-tuning process of the first log detection model, in order to verify the fine-tuning result of the first log detection model, the loss value of the first log detection model in the test can be used as the verification index. In order to obtain the loss value of the first log detection model, the terminal can obtain the output information of the initial anomaly detection model with historical log data as the input, and use it as the auxiliary label of the historical log data. The auxiliary label can include the information obtained after the initial log detection model performs feature processing and recognition on the historical log data.

[0123] Step 504: Obtain the true label of the historical log data, where the true label is the manual annotation performed on the historical log data in advance.

[0124] Exemplarily, the terminal can also obtain the true label of the historical log data, and the true label can be the manual annotation performed on the historical log by technicians in advance. Obviously, the true label only includes the information required by technicians, while in addition to the information required by technicians set in the auxiliary label, it also includes other prior information obtained by the model in feature extraction and processing.

[0125] Step 506: Calculate the loss function of the first log detection model based on the true label and the auxiliary label until the probability distribution of the first log detection model converges to the probability distribution of the initial anomaly detection model.

[0126] Among them, the loss value can refer to the cross-entropy loss generated under the joint action of the true label and the auxiliary label.

[0127] Exemplarily, after the terminal obtains the real label and the auxiliary label, it can calculate the loss value of the model based on the real label and the auxiliary label. The loss value can be a variable of the probability distribution output by the current model. The specific loss function can be as follows:

[0128]

[0129] Among them, x represents the input log data, y represents the label, and P(x) represents the probability distribution output by the model at the current stage.

[0130] In this embodiment, the terminal can calculate the loss value of the model based on the real label, the auxiliary label, and the probability distribution output by the model, and use the loss value of the model as the verification standard for the convergence of the first log detection model, which ultimately helps to improve the accuracy of the output of the lightweight log detection model.

[0131] In one embodiment, step 506 includes:

[0132] Step 602: Set a first weight coefficient to adjust the influence degrees of the real label and the auxiliary label on the loss function with the first weight system.

[0133] Exemplarily, since the unified log data includes both the real label and the auxiliary label, the terminal can set a first weight coefficient to adjust the influence degrees of the real label and the auxiliary label on the final lightweight log detection model. At this time, the loss function can be as follows:

[0134]

[0135] Among them, α ∈ [0, 1] is the weight coefficient, y t , y s are the real label and the auxiliary label respectively.

[0136] In this embodiment, by setting the weight coefficient in the loss function, it helps technicians adjust the influence degrees of the real label and the auxiliary label on the model according to actual needs, thereby helping to improve the flexibility of model application.

[0137] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0138] Based on the same inventive concept, an embodiment of the present application further provides an abnormal log detection device for implementing the abnormal log detection method involved above. The implementation solution provided by this device to solve problems is similar to the implementation solution described in the above method. Therefore, the specific limitations in one or more embodiments of the abnormal log detection device provided below can refer to the limitations on the abnormal log detection method in the above text, and will not be repeated here.

[0139] In one embodiment, as Figure 7 shown, a real-time abnormal log detection device is provided, including: a log data module, an abnormal detection module, an initial model module, and a lightweight model module, where:

[0140] The log data module is used to obtain log data;

[0141] The abnormal detection module is used to input the log data into a lightweight log detection model to obtain an abnormal detection result corresponding to the log data;

[0142] Before the abnormal detection module, there is also included:

[0143] The initial model module is used to obtain historical log data and train an initial abnormal detection model based on the historical log data. The initial abnormal detection model is used to implement abnormal log detection;

[0144] The lightweight model module is used to layer by layer remove target feature channels in the initial abnormal detection model according to a preset pruning ratio to obtain the lightweight log detection model.

[0145] In one of the embodiments, before the lightweight model module, there is also included:

[0146] The random removal module is used to randomly remove feature channels in each convolutional layer of the initial abnormal detection model according to a preset ratio gradient;

[0147] A robustness verification module, configured to obtain the robustness verification parameters of the test anomaly detection model after randomly removing the feature channels;

[0148] A pruning ratio determination module, configured to record the removal ratio of the current feature channels and set the removal ratio as the pruning ratio of the current convolutional layer when the robustness verification parameters reach a preset robustness threshold range.

[0149] In one embodiment, the lightweight model module includes:

[0150] A feature mapping module, configured to input the historical log data into the initial anomaly detection model, and obtain a set of feature maps output by a target convolutional layer in the initial anomaly detection model, where the set of feature maps includes feature components corresponding to each feature channel in the target convolutional layer;

[0151] A feature scalar module, configured to perform downsampling processing on the set of feature maps to obtain feature scalars corresponding to the feature components;

[0152] An importance ranking module, configured to determine the importance ranking of the feature channels based on the feature scalars, and select, in reverse order according to the importance ranking, the target feature channels with a quantity satisfying the pruning ratio in the target convolutional layer.

[0153] In one embodiment, the lightweight model module includes:

[0154] A first log model module, configured to layer - by - layer remove the target feature channels in the initial anomaly detection model to obtain the first log detection model;

[0155] A model fine - tuning module, configured to construct a fine - tuning training set based on the historical log data, and use the fine - tuning training set to train the first log detection model until convergence to obtain the lightweight log detection model.

[0156] In one embodiment, the model fine - tuning module includes:

[0157] An auxiliary label module, configured to input the historical log data into the initial anomaly detection model, and use the output result of the initial anomaly detection model as the auxiliary label of the historical log data;

[0158] A true label module, configured to obtain the true label of the historical log data, where the true label is the manual annotation of the historical log data in advance;

[0159] A loss function module, configured to calculate a loss function of the first log detection model based on the true label and the auxiliary label until the probability distribution of the first log detection model converges to the probability distribution of the initial anomaly detection model.

[0160] In one embodiment, the loss function module includes:

[0161] A weight coefficient module, configured to set a first weight coefficient to adjust the influence degrees of the true label and the auxiliary label on the loss function with the first weight system.

[0162] Each module in the above anomaly log detection device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0163] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structural diagram can be as Figure 8 shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. The computer program, when executed by the processor, implements a real-time anomaly log detection method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad set on the shell of the computer device, or an external keyboard, a touchpad, or a mouse, etc.

[0164] Those skilled in the art can understand that Figure 8 the structure shown in

[0165] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0167] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0169] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0170] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0171] The above-described embodiments merely represent several implementation manners of this application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application should be subject to the appended claims.

Claims

1. A real-time abnormal log detection method, characterized in that, The method includes: Obtain log data; Input the log data into a lightweight log detection model to obtain an anomaly detection result corresponding to the log data; Before inputting the log data into the lightweight log detection model to obtain an anomaly detection result corresponding to the log data, it further includes: Obtain historical log data, and train an initial anomaly detection model based on the historical log data. The initial anomaly detection model is used to implement anomaly log detection; Remove target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model; Before removing target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model, it further includes: Randomly remove feature channels in each convolutional layer of the initial anomaly detection model according to a preset proportional gradient; Obtain robustness verification parameters for testing the anomaly detection model after randomly removing the feature channels; When the robustness verification parameters reach a preset robustness threshold range, record the removal ratio of the current feature channels, and set the removal ratio as the pruning ratio of the current hidden layer.

2. The method according to claim 1, wherein Removing target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model includes: Input the historical log data into the initial anomaly detection model, and obtain a set of feature maps output by the target convolutional layer in the initial anomaly detection model. The set of feature maps includes feature components corresponding to each feature channel in the target convolutional layer; Perform downsampling processing on the set of feature maps to obtain feature scalars corresponding to the feature components; Determine the importance ranking of the feature channels based on the feature scalars, and select, in reverse order according to the importance ranking, the number of target feature channels that meet the pruning ratio in the target convolutional layer.

3. The method according to claim 1, characterized in that Removing target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model includes: Remove target feature channels in the initial anomaly detection model layer by layer to obtain a first log detection model; Construct a fine-tuning training set based on the historical log data, and use the fine-tuning training set to train the first log detection model until convergence to obtain the lightweight log detection model.

4. The method according to claim 3, characterized in that, Constructing a fine-tuning training set based on the historical log data and using the fine-tuning training set to train the first log detection model until convergence to obtain the lightweight log detection model includes: Input the historical log data into the initial anomaly detection model, and use the probability distribution output by the initial anomaly detection model as the auxiliary label of the historical log data; Obtain the true label of the historical log data. The true label is the manual annotation of the historical log data in advance; Calculate the output probability distribution of the first log detection model based on the true label and the auxiliary label until the loss function converges to the global optimal solution.

5. The method according to claim 4, wherein Calculating the output probability distribution of the first log detection model based on the true label and the auxiliary label until the loss function falls into the global optimal solution includes: Setting a first weight coefficient to adjust the influence degrees of the true label and the auxiliary label on the loss function by the first weight coefficient.

6. A real-time abnormal log detection device, characterized in that, The device includes: A log data module for obtaining log data; An anomaly detection module for inputting the log data into a lightweight log detection model to obtain an anomaly detection result corresponding to the log data; Before the anomaly detection module, there is also: An initial model module for obtaining historical log data and training an initial anomaly detection model based on the historical log data, where the initial anomaly detection model is used to implement anomaly log detection; A lightweight model module for removing target feature channels in the initial anomaly detection model layer by layer according to a preset pruning ratio to obtain the lightweight log detection model; Before the lightweight model module, there is also: A random removal module for randomly removing feature channels in each convolutional layer of the initial anomaly detection model according to a preset ratio gradient; A robustness verification module for obtaining robustness verification parameters of the test anomaly detection model after randomly removing the feature channels; A pruning ratio determination module for recording the current removal ratio of the feature channels and setting the removal ratio as the pruning ratio of the current hidden layer when the robustness verification parameters reach a preset robustness threshold range.

7. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 5.

Images