Abnormity detection method based on reinforcement learning and related device

Abstract

The invention provides an anomaly detection method based on reinforcement learning and a related device, relates to the technical field of data mining and artificial intelligence, and aims to train a deep learning network by a reinforcement learning method and a machine learning method in advance to obtain an anomaly detection model. Obtaining to-be-detected target data; the method comprises the steps of preprocessing target data to obtain target numerical value features corresponding to the target data; calling an anomaly detection model, and inputting the target numerical value features into the anomaly detection model, so that the anomaly detection model outputs a first expected probability of the target data to an anomaly detection algorithm; and performing anomaly detection on the target numerical value feature by using an algorithm with the maximum first expected probability in the anomaly detection algorithms to obtain an anomaly detection result of the target data. According to the invention, the anomaly detection strategy in the application scene can be adaptively adjusted, and the anomaly detection accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of data mining and artificial intelligence technology, and in particular to an anomaly detection method and related devices based on reinforcement learning. Background Art

[0002] Machine learning and deep learning methods are widely used in text analysis. Researchers often perform anomaly detection on data before text analysis to prevent abnormal points from interfering with subsequent analysis results. Therefore, the accuracy of anomaly detection is very critical.

[0003] However, currently, only one anomaly detection algorithm can be used for anomaly detection, such as Z-score, Sigma-3, MA, Boxplot, etc. This requires researchers to re-analyze each time when facing different data with large differences to determine the best anomaly detection algorithm. In addition, when facing time series data of different periods of the same data set, it is impossible to adaptively adjust the anomaly detection algorithm, which makes it inevitable that problems such as untimely detection or false alarms will occur when only one fixed anomaly detection algorithm is used. Summary of the invention

[0004] In view of the above problems, the present application provides an anomaly detection method and related devices based on reinforcement learning to achieve the purpose of adaptively adjusting the anomaly detection strategy in the application scenario. The specific scheme is as follows:

[0005] The first aspect of the present application provides an anomaly detection method based on reinforcement learning, and the anomaly detection method based on reinforcement learning includes:

[0006] Acquire target data to be detected;

[0007] By preprocessing the target data, a target numerical feature corresponding to the target data is obtained;

[0008] Retrieving a pre-trained anomaly detection model, and inputting the target numerical feature into the anomaly detection model, so that the anomaly detection model outputs a first expected probability of the target data to the anomaly detection algorithm; wherein the anomaly detection model is obtained by training a deep learning network using a reinforcement learning method and a machine learning method;

[0009] Anomaly detection is performed on the target numerical feature using the algorithm with the largest first expected probability in the anomaly detection algorithm to obtain an anomaly detection result of the target data.

[0010] In a possible implementation, the preprocessing of the target data to obtain the target numerical feature corresponding to the target data includes:

[0011] Calculate the first mean, maximum value, minimum value, variance, skewness, kurtosis and slope of the target data;

[0012] Normalizing the target data using the first mean and the variance;

[0013] Normalizing the maximum value, the minimum value, and the variance respectively using the first mean;

[0014] Normalizing the skewness, kurtosis and slope respectively by taking the inverse thereof;

[0015] The normalized result of the target data, the normalized result of the maximum value, the normalized result of the minimum value, the normalized result of the variance, the normalized result of the skewness, the normalized result of the kurtosis and the normalized result of the slope are integrated to obtain the target numerical feature.

[0016] In a possible implementation, the deep learning network includes a first deep learning network and a second deep learning network with the same network structure;

[0017] The process of training the deep learning network using the reinforcement learning method and the machine learning method to obtain the anomaly detection model includes:

[0018] Divide the original data to obtain sample data for this iterative training;

[0019] By preprocessing the sample data, a sample numerical feature corresponding to the sample data is obtained;

[0020] Inputting the sample numerical features into the first deep learning network and the second deep learning network respectively, so that the first deep learning network outputs a second expected probability of the sample data for the anomaly detection algorithm, and the second deep learning network outputs a third expected probability of the sample data for the anomaly detection algorithm;

[0021] Performing anomaly detection on the sample numerical features using the anomaly detection algorithm to obtain an anomaly detection result of the sample data;

[0022] Calculating a reward score of the anomaly detection algorithm according to an anomaly detection result of the sample data;

[0023] Converting the reward score into a pseudo label for the anomaly detection algorithm;

[0024] Penalizing the third expected probability with the pseudo label to obtain a fourth expected probability, and calculating a cross entropy loss function value of this iterative training according to the second expected probability and the fourth expected probability;

[0025] When the preset end condition is not met at present, the network parameters of the first deep learning network are updated based on the cross entropy loss function value, and the step of dividing the original data to obtain the sample data for this iterative training is returned to be executed. When the preset update condition is currently met, the network parameters of the first deep learning network are assigned to the second deep learning network;

[0026] When the preset end condition is currently met, the training is ended, and the trained second deep learning network is used as the anomaly detection model.

[0027] In a possible implementation, the step of dividing the original data to obtain sample data for this iterative training includes:

[0028] In the case where the original data is time series data, the original data is divided by a sliding window to obtain sample data in different time windows;

[0029] Select the sample data under the target time window for this iterative training.

[0030] In a possible implementation, the abnormality detection result of the sample data includes a set of abnormal points of the sample data;

[0031] The calculating the reward score of the anomaly detection algorithm according to the anomaly detection result of the sample data includes:

[0032] Taking the intersection of the outlier point sets to obtain an outlier point intersection set;

[0033] Taking the difference between the outlier point set and the outlier point intersection set as the outlier point difference set set;

[0034] Calculate the deviation score of the anomaly detection algorithm by adding points to the anomaly points in the anomaly point intersection set and deducting points from the anomaly points in the anomaly point difference set;

[0035] Calculating a second mean of the sample data and a Euclidean distance between an outlier in the outlier set and the second mean;

[0036] Calculating a third mean of the Euclidean distance, and calculating a sparsity score of the anomaly detection algorithm using the second mean and the third mean;

[0037] The reward score is calculated based on the deviation score and the sparsity score.

[0038] In a possible implementation, the deep learning network includes: an encoder and a decoder;

[0039] The encoder consists of a one-dimensional convolutional layer, a first multi-head self-attention mechanism layer, a first linear layer, a second multi-head self-attention mechanism layer and a second linear layer connected in sequence;

[0040] The decoder is composed of a third multi-head self-attention mechanism layer, a third linear layer, a fourth multi-head self-attention mechanism layer, a fourth linear layer, a fifth linear layer and a soft function connected in sequence;

[0041] Among them, the output of the first multi-head self-attention mechanism layer is used as the input of the first linear layer and the fourth linear layer respectively, the output of the third multi-head self-attention mechanism layer is used as the input of the second linear layer and the third linear layer respectively, a dimension conversion operation and a position encoding operation are arranged between the one-dimensional convolution layer and the first multi-head self-attention mechanism layer, and a random deactivation function is arranged after the first linear layer, the second linear layer and the third linear layer respectively.

[0042] A second aspect of the present application provides an anomaly detection device based on reinforcement learning, the anomaly detection device based on reinforcement learning comprising:

[0043] A model training module is used to train the deep learning network using reinforcement learning methods and machine learning methods to obtain an anomaly detection model;

[0044] The anomaly detection module is used to obtain the target data to be detected; obtain the target numerical features corresponding to the target data by preprocessing the target data; call the anomaly detection model and input the target numerical features into the anomaly detection model so that the anomaly detection model outputs the first expected probability of the target data for the anomaly detection algorithm; and perform anomaly detection on the target numerical features using the algorithm with the largest first expected probability in the anomaly detection algorithm to obtain the anomaly detection result of the target data.

[0045] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the reinforcement learning-based anomaly detection method of the above-mentioned first aspect or any implementation of the first aspect.

[0046] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0047] The memory is used to store computer programs;

[0048] The processor is used to execute the computer program so that the electronic device can implement the reinforcement learning-based anomaly detection method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0049] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the reinforcement learning-based anomaly detection method of the above-mentioned first aspect or any implementation method of the first aspect.

[0050] By means of the above technical scheme, the present application provides an anomaly detection method and related devices based on reinforcement learning, which pre-trains the deep learning network with reinforcement learning methods and machine learning methods to obtain an anomaly detection model. Obtain the target data to be detected; obtain the target numerical features corresponding to the target data by preprocessing the target data; call the anomaly detection model, and input the target numerical features into the anomaly detection model so that the anomaly detection model outputs the first expected probability of the target data for the anomaly detection algorithm; use the algorithm with the largest first expected probability in the anomaly detection algorithm to perform anomaly detection on the target numerical features to obtain the anomaly detection result of the target data. The present application combines reinforcement learning, machine learning and deep learning to construct an anomaly detection model, and uses the anomaly detection model to predict the expected probability of the target data for the anomaly detection algorithm, thereby selecting the anomaly detection algorithm with the highest expected probability for the target data for anomaly detection, which can realize adaptive adjustment of the anomaly detection strategy in the application scenario and improve the accuracy of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0052] Figure 1 A flowchart of an anomaly detection method based on reinforcement learning provided in an embodiment of the present application;

[0053] Figure 2 A partial flow chart of an anomaly detection method based on reinforcement learning provided in an embodiment of the present application;

[0054] Figure 3 Another partial flow chart of an anomaly detection method based on reinforcement learning provided in an embodiment of the present application;

[0055] Figure 4 Another partial flow chart of an anomaly detection method based on reinforcement learning provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the structure of a deep learning network provided in an embodiment of the present application;

[0057] Figure 6A structural diagram of an anomaly detection device for reinforcement learning provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0060] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0061] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] The embodiment of the present application provides an anomaly detection method based on reinforcement learning, which achieves a more flexible and more accurate anomaly detection effect by combining reinforcement learning, machine learning and deep learning. The anomaly detection method based on reinforcement learning in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0063] See also Figure 1 , Figure 1 The following is a flow chart of an anomaly detection method based on reinforcement learning provided in an embodiment of the present application. Figure 1 As shown, an anomaly detection method based on reinforcement learning provided in an embodiment of the present application may include steps S101 to S104, and these steps are described in detail below.

[0064] S101, obtaining target data to be detected.

[0065] In the embodiment of the present application, the target data is the data to be detected for anomalies, which may be time series data or non-time series data. The embodiment of the present application does not limit this. For example, the time series data may be call revenue data within a specified period of time with a time interval of days.

[0066] S102, obtaining target numerical features corresponding to the target data by preprocessing the target data.

[0067] In an embodiment of the present application, before the target data is sent to the anomaly detection model, the numerical features corresponding to the target data, that is, the target numerical features, can be obtained through preprocessing.

[0068] In a possible implementation, the maximum value, minimum value, variance, skewness, kurtosis, and slope can be used as additional feature inputs, and normalized to reduce the difficulty of model prediction and improve the accuracy of anomaly detection model prediction. Figure 2 , Figure 2 A partial flow chart of an anomaly detection method based on reinforcement learning provided in an embodiment of the present application. Figure 2 As shown, an embodiment of the present application provides an anomaly detection method based on reinforcement learning, wherein step S102 "obtaining target numerical features corresponding to the target data by preprocessing the target data" may include steps S1021 to S1025, and these steps are described in detail below.

[0069] S1021, calculate the first mean, maximum value, minimum value, variance, skewness, kurtosis and slope of the target data.

[0070] In the embodiment of the present application, the mean (i.e., the first mean), maximum value, minimum value, variance, skewness, kurtosis, and slope of the target data are calculated respectively. Among them, the first mean represents the center position of the target data, the maximum value represents the upper limit of the target data, the minimum value represents the lower limit of the target data, the variance represents the turbulence of the target data, the skewness represents the asymmetry of the target data (i.e., the degree of deviation from the normal distribution), the kurtosis represents the peak value and tail thickness of the target data, and the slope represents the rate of change with the actual index (representing the slope of the data).

[0071] S1022, normalize the target data using the first mean and variance.

[0072] In the embodiment of the present application, the target data includes multiple data points. The data points are , the first mean is , the variance is , then The normalized result of the data points is It should be noted that refer to function.

[0073] S1023: Use the first mean to normalize the maximum value, the minimum value, and the variance respectively.

[0074] In the present embodiment, it is assumed that the first mean is The maximum value is , the minimum value is , the variance is , then the maximum value The normalized result is , minimum The normalized result is ,variance The normalized result is .

[0075] S1024, normalizing the skewness, kurtosis and slope respectively by taking the inverse thereof.

[0076] In the present embodiment, it is assumed that the skewness is , the kurtosis is The slope is , then the skewness The normalized result is , kurtosis The normalized result is , slope The normalized result is .

[0077] S1025, integrating the normalized result of the target data, the normalized result of the maximum value, the normalized result of the minimum value, the normalized result of the variance, the normalized result of the skewness, the normalized result of the kurtosis and the normalized result of the slope to obtain the target numerical feature.

[0078] In the embodiment of the present application, the target numerical feature includes the normalized result of the target data, the normalized result of the maximum value, the normalized result of the minimum value, the normalized result of the variance, the normalized result of the skewness, the normalized result of the kurtosis, and the normalized result of the slope. For example, assuming that the length of the target data is 30, that is, the target data contains 30 data points, the scale of the target numerical feature is 36.

[0079] If the target data is time series data, normalization can reduce the scale impact between different features and facilitate model convergence. Since this application has the characteristics of unsupervised online learning, different data will not affect the overall numerical characteristics after normalization.

[0080] It should be noted that in the embodiment of the present application, the maximum value, minimum value, variance, skewness, kurtosis and slope are used as additional feature inputs. In practical applications, any value among the maximum value, minimum value, variance, skewness, kurtosis and slope can be selected as the feature input, and the embodiment of the present application does not limit this.

[0081] S103, calling a pre-trained anomaly detection model, and inputting the target numerical feature into the anomaly detection model, so that the anomaly detection model outputs a first expected probability of the target data to the anomaly detection algorithm; wherein the anomaly detection model is obtained by training a deep learning network using a reinforcement learning method and a machine learning method.

[0082] In the embodiment of the present application, reinforcement learning, machine learning and deep learning are combined to construct an anomaly detection model, making model training easier and more stable, and having a more flexible and accurate detection effect. The target numerical features are input into the anomaly detection model, and the anomaly detection model is used to predict the expected probability of the target data for different anomaly detection algorithms, that is, the first expected probability.

[0083] In a possible implementation, the deep learning network includes two networks with the same network structure, namely, a first deep learning network and a second deep learning network. Figure 3 , Figure 3 Another partial flow chart of an abnormality detection method based on reinforcement learning provided in an embodiment of the present application. Figure 3 As shown, an embodiment of the present application provides an anomaly detection method based on reinforcement learning, wherein the process of training a deep learning network with a reinforcement learning method and a machine learning method to obtain an anomaly detection model may include the following steps S201 to S209, and these steps are described in detail below.

[0084] S201, dividing the original data to obtain sample data for this iterative training.

[0085] In the embodiment of the present application, according to the data length that needs to be processed by the anomaly detection algorithm, the sample data for the current iterative training can be obtained by dividing the original data.

[0086] Specifically, when the original data is time series data, the original data can be divided into sample data in different time windows by using a sliding window; the sample data in the target time window is selected for this iterative training. For example, the time series data is divided into a plurality of sample data of length 30 by using a sliding window technique, and then the sample data in the target time window is selected for this iterative training in order from early to late.

[0087] S202, obtaining sample numerical features corresponding to the sample data by preprocessing the sample data.

[0088] In the embodiment of the present application, the sample data is preprocessed to obtain the corresponding numerical features, i.e., the sample numerical features. It should be noted that the process of preprocessing the sample data can refer to the process of preprocessing the target data in the above step S102, and the embodiment of the present application will not be repeated. Continuing with the example that the length of the sample data is 30, the scale of the sample numerical features is 36.

[0089] S203, inputting the sample numerical features into the first deep learning network and the second deep learning network respectively, so that the first deep learning network outputs the second expected probability of the sample data for the anomaly detection algorithm, and also enables the second deep learning network to output the third expected probability of the sample data for the anomaly detection algorithm.

[0090] In an embodiment of the present application, the sample numerical features are input into the first deep learning network and the second deep learning network respectively, and the first deep learning network and the second deep learning network predict the expected probability of the sample data for the anomaly detection algorithm in the form of action probability, so as to determine the matching degree of the sample data with different anomaly detection algorithms.

[0091] Specifically, the expected probability output by the first deep learning network is the second expected probability, and the expected probability output by the second deep learning network is the third expected probability. Taking the first deep learning network as an example, assuming that the anomaly detection algorithm includes Z-score, Sigma-3, MA, and Boxplot, the second expected probability output by the first deep learning network specifically includes the second expected probability of the sample data predicted by the first deep learning network for Z-score, the second expected probability of the sample data for Sigma-3, the second expected probability of the sample data for MA, and the second expected probability of the sample data for Boxplot.

[0092] S204, performing anomaly detection on the sample numerical features using an anomaly detection algorithm to obtain an anomaly detection result of the sample data.

[0093] In the embodiment of the present application, different anomaly detection algorithms are used to perform anomaly detection on sample numerical features respectively, so as to obtain the anomaly detection results of different anomaly detection algorithms on sample data. Continuing to assume that the anomaly detection algorithms include Z-score, Sigma-3, MA and Boxplot, Z-score is used as an example to illustrate that Z-score is used to perform anomaly detection on sample numerical features to obtain its anomaly detection results on sample data.

[0094] S205, calculating a reward score of the anomaly detection algorithm according to the anomaly detection result of the sample data.

[0095] In actual application scenarios, it is usually difficult to obtain raw data with labels, and manual labeling is extremely time-consuming and inefficient. To solve this problem, this application involves a reward scoring mechanism, which is used to match a pseudo label for the probability output of the model to achieve the effect of unsupervised training.

[0096] In the embodiment of the present application, the reward scores of the different anomaly detection algorithms can be calculated by combining the anomaly detection results of the sample data by the different anomaly detection algorithms.

[0097] In a possible implementation, the reward score is composed of a deviation score and a sparsity score. The anomaly detection result of the sample data contains a set of anomaly points of the sample data. Figure 4 , Figure 4 Another partial flow chart of an abnormality detection method based on reinforcement learning provided in an embodiment of the present application. Figure 4 As shown, an embodiment of the present application provides an anomaly detection method based on reinforcement learning, wherein step S205 "calculates the reward score of the anomaly detection algorithm according to the anomaly detection result of the sample data" may include the following steps S2051 to S2056, and these steps are described in detail below.

[0098] S2051, taking the intersection of the outlier point sets to obtain an outlier point intersection set.

[0099] In the embodiment of the present application, for ease of understanding, it is continued to be assumed that the anomaly detection algorithm includes Z-score, Sigma-3, MA and Boxplot. It is assumed that there are 3 anomalies in the anomaly set abnormal_set_0 detected by Z-score for sample data, 2 anomalies in the anomaly set abnormal_set_1 detected by Sigma-3 for sample data, 4 anomalies in the anomaly set abnormal_set_2 detected by MA for sample data, and 3 anomalies in the anomaly set abnormal_set_3 detected by Boxplot for sample data.

[0100] Take the intersection of abnormal_set_0, abnormal_set_1, abnormal_set_2 and abnormal_set_3 to obtain a common abnormal point set, that is, the abnormal point intersection set, denoted as abnormal_set_finally.

[0101] S2052: taking the difference set between the outlier point set and the outlier point intersection set as the outlier point difference set set.

[0102] In the embodiment of the present application, the reward score calculation of Z-score is used as an example to illustrate, and the process of calculating the reward score of other anomaly detection algorithms is the same. For Z-score, its abnormal point set abnormal_set_0 is compared with the abnormal point intersection set abnormal_set_finally, so as to determine the abnormal point set in the abnormal point set abnormal_set_0 that is not in the abnormal point intersection set abnormal_set_finally, so as to determine the difference set of the abnormal point set abnormal_set_0 and the abnormal point intersection set abnormal_set_finally, and use it as the abnormal point difference set set, recorded as diff_set.

[0103] S2053, calculating the deviation score of the anomaly detection algorithm by adding points to the anomaly points in the anomaly point intersection set and subtracting points from the anomaly points in the anomaly point difference set.

[0104] In the embodiment of the present application, the calculation of the reward score of the Z-score is continued as an example. For each abnormal point in its abnormal point set abnormal_set_0, if it is in the abnormal point intersection set abnormal_set_finally, then the abnormal point is added points, otherwise, if it is in the abnormal point difference set diff_set, then the abnormal point is subtracted points.

[0105] Assume that the abnormal point set abnormal_set_0 detected by Z-score for sample data includes abnormal point 1, abnormal point 2 and abnormal point 3. Among them, abnormal point 1 and abnormal point 2 are in the abnormal point intersection set abnormal_set_finally, then 1 point is added to abnormal point 1 and abnormal point 2 respectively, and abnormal point 3 is in the abnormal point difference set diff_set, then 0.5 point is subtracted from abnormal point 3. At this time, the deviation score of Z-score is It is 1+1-0.5=1.5.

[0106] S2054, calculating the second mean of the sample data and the Euclidean distance between the outliers in the outlier set and the second mean.

[0107] In the embodiment of the present application, the calculation of the reward score of the Z-score is continued as an example. On the one hand, the mean of the sample data, that is, the second mean, is calculated; on the other hand, the Euclidean distance between each abnormal point in the abnormal point set abnormal_set_0 and the second mean is calculated. At this time, abnormal point 1 corresponds to Euclidean distance 1, abnormal point 2 corresponds to Euclidean distance 2, and abnormal point 3 corresponds to Euclidean distance 3.

[0108] S2055, calculating the third mean of the Euclidean distance, and using the second mean and the third mean to calculate the sparsity score of the anomaly detection algorithm.

[0109] In the embodiment of the present application, the calculation of the reward score of the Z-score is continued as an example to illustrate that the mean of the Euclidean distance 1, the Euclidean distance 2 and the Euclidean distance 3, that is, the third mean, is calculated.

[0110] Assume that the second mean is The third mean is , the number of outliers in the outlier set is (at this time, ). Then the sparsity score of Z-score is .

[0111] S2056, calculate the reward score according to the deviation score and the sparsity score.

[0112] In the present embodiment, the calculation of the Z-score reward score is used as an example. .

[0113] S206, converting the reward score into a pseudo label for anomaly detection algorithm.

[0114] In the present embodiment, for ease of understanding, it is assumed that the anomaly detection algorithm includes Z-score, Sigma-3, MA and Boxplot. After the above steps S2051 to S2055, the reward score of Z-score can be obtained. , Sigma-3's bonus score , MA bonus score and bonus score for Boxplot . The reward score can be obtained through softmax , , and Convert them into the output probabilities of Z-score, Sigma-3, MA and Boxplot, i.e., pseudo labels. Taking the calculation of the pseudo labels of Z-score as an example, first calculate the power and , and then calculate the pseudo label of Z-score , which can be used to score the Z-score reward Normalized to the range of 0 to 1.

[0115] S207, penalizing the third expected probability with the pseudo label to obtain a fourth expected probability, and calculating the cross entropy loss function value of this iterative training according to the second expected probability and the fourth expected probability.

[0116] In the embodiment of the present application, for different anomaly detection algorithms, their pseudo labels can be used to penalize their third expected probability, so as to obtain their fourth expected probability. For ease of understanding, it is assumed that the anomaly detection algorithms include Z-score, Sigma-3, MA and Boxplot. Taking Z-score as an example, assuming that its third expected probability is out_target, its fourth expected probability out_finally is obtained after being penalized by its pseudo label out_label. , if the anomaly detection algorithm with the largest third expected probability among all anomaly detection algorithms is the same as the anomaly detection algorithm with the largest pseudo-label, the penalty coefficient is 1, otherwise, the penalty coefficient is 0.7.

[0117] It should be noted that the penalty coefficient Appropriate adjustments can be made according to actual task performance, and this embodiment of the present application does not limit this.

[0118] Furthermore, the cross entropy loss function value of this iterative training is comprehensively calculated based on the second expected probability and the fourth expected probability of each anomaly detection algorithm.

[0119] S208, when the preset end condition is not currently met, the network parameters of the first deep learning network are updated based on the cross entropy loss function value, and the process returns to step S201, and when the preset update condition is currently met, the network parameters of the first deep learning network are assigned to the second deep learning network.

[0120] In an embodiment of the present application, if the cross entropy loss function value is not in convergence and the current number of iterations has not reached the upper limit of the number of iterations, the network parameters of the first deep learning network are updated based on the cross entropy loss function value of this iterative training to achieve constrained update of the first deep learning network, and return to execute the above step S201 to enter the next iterative training. In addition, if the cross entropy loss function value decreases compared to the previous iterative training, or the current number of iterations meets the requirements (for example, the second deep learning network is updated every 5 iterations), the first deep learning network assigns its network parameters to the second deep learning network to achieve the update of the second deep learning network. This model update strategy based on reinforcement learning can ensure that the model training is more stable.

[0121] S209, when the preset end condition is currently met, the training is ended, and the trained second deep learning network is used as an anomaly detection model.

[0122] In an embodiment of the present application, if the cross entropy loss function value is in convergence, or the current number of iterations reaches the upper limit of the number of iterations, the training is terminated and the trained second deep learning network is used as an anomaly detection model.

[0123] After the above steps S201 to S209, an anomaly detection model adapted to the original data can be obtained. In practical applications, the anomaly detection model supports both offline data samples and real-time data streams. The corresponding numerical features can be obtained by preprocessing before each input, and then sent to the anomaly detection model to obtain the expected probability of different anomaly detection algorithms. The anomaly detection algorithm with the largest expected probability is selected as the best anomaly detection algorithm for this time. This can effectively solve the problem of inaccurate detection in practical application scenarios due to the inflexibility of previous algorithms.

[0124] S104, performing anomaly detection on the target numerical feature using the algorithm with the largest first expected probability in the anomaly detection algorithm to obtain an anomaly detection result of the target data.

[0125] In the embodiment of the present application, for the target data, an anomaly detection algorithm with the largest first expected probability can be selected to perform anomaly detection on the target data, thereby obtaining an anomaly detection result thereof. The anomaly points in the target data can be determined through the anomaly detection result of the target data.

[0126] In one possible implementation, the classification task can be implemented using only the fully connected layer based on the Unet concept. Figure 5 , Figure 5 A schematic diagram of the structure of a deep learning network provided in an embodiment of the present application. Figure 5 As shown, in an embodiment of the present application, the deep learning network includes an encoder A and a decoder B.

[0127] The encoder A consists of a one-dimensional convolutional layer A-1, a first multi-head self-attention mechanism layer A-2, a first linear layer A-3, a second multi-head self-attention mechanism layer A-4, and a second linear layer A-5 connected in sequence.

[0128] Decoder B consists of a third multi-head self-attention mechanism layer B-1, a third linear layer B-2, a fourth multi-head self-attention mechanism layer B-3, a fourth linear layer B-4, a fifth linear layer B-5 and a soft function B-6 connected in sequence.

[0129] Among them, the output of the first multi-head self-attention mechanism layer A-2 is used as the input of the first linear layer A-3 and the fourth linear layer B-4 respectively, the output of the third multi-head self-attention mechanism layer B-1 is used as the input of the second linear layer A-5 and the third linear layer B-2 respectively, a dimension conversion operation and a position encoding operation are set between the one-dimensional convolution layer A-1 and the first multi-head self-attention mechanism layer A-2, and a random deactivation function is set after the first linear layer A-3, the second linear layer A-5 and the third linear layer B-2 respectively.

[0130] In the process of training the anomaly detection model, the network structures of the first deep learning network and the second deep learning network are as follows: Figure 5 As shown. Taking one iteration training as an example, the sample data is preprocessed to obtain the sample numerical features. The scale of the sample numerical features is 36. The three-dimensional features with a shape of B * 1 * 36 are obtained by dimensional expansion of the sample numerical features. In order to extract more advanced information from a single feature, the one-dimensional convolution layer A-1 is used as the embedding layer to expand the shape = B * 1 * 36 to shape = B * 8 * 36. Then, the second and third dimensions of the shape = B * 8 * 36 are transformed into shape = B * 36 * 8 through the dimension conversion operation. In order to keep the position information of the 36 feature values ​​in the entire data, the position encoding information of shape = B * 36 * 1 is added to the three-dimensional features through the position encoding operation, so as to obtain the input features with position information. The input features with position information are input into the first multi-head self-attention mechanism layer A-2 to extract global semantic information, and a new three-dimensional feature shape = B * 36 * 8 is obtained, which is recorded as e1. Send e1 to the first linear layer A-3 to expand the scale of the third dimension to obtain a new three-dimensional feature shape = B * 36 * 16, and use a random deactivation function (Dropout) for deactivation to prevent model overfitting and improve the generalization ability of the model. Send the deactivated three-dimensional feature B * 36 * 16 to the second multi-head self-attention mechanism layer A-4 to obtain a new three-dimensional feature shape = B * 36 * 16, denoted as e2. Send e2 to the second linear layer A-5 to map the features of the third dimension to another domain to enhance the encoding ability of the encoder. The new three-dimensional feature shape = B * 36 * 16 is then deactivated using a random deactivation function (Dropout) to obtain the three-dimensional feature shape = B * 36 * 16 output by the encoder.

[0131] The three-dimensional features output by the encoder are used as the input features of the encoder with shape = B * 36 * 16. After the third multi-head self-attention mechanism layer B-1 extracts global semantic features, a new three-dimensional feature shape = B * 36 * 16 is obtained, denoted as: d1. Concatenate d1 and e2 according to the third dimension to obtain a new three-dimensional feature shape = B * 36 * 32, which is then sent to the third linear layer B-2 to reduce the scale of the third dimension to obtain a new three-dimensional feature shape = B * 36 * 8, and a random deactivation function (Dropout) is used for deactivation to prevent overfitting of the model and improve the generalization ability of the model. The deactivated three-dimensional feature shape = B * 36 * 8 is sent to the fourth multi-head self-attention mechanism layer B-3 to extract global semantic features, and a new three-dimensional feature shape = B * 36 * 8 is obtained, denoted as d2. Concatenate d2 and e1 along the third dimension to obtain a new three-dimensional feature shape = B * 36 * 16. Then flatten the second and third dimensions and send them to the fourth linear layer B-4 to reduce the scale of the second dimension to obtain a new two-dimensional feature shape = B * 144. Send the output features of the fourth linear layer B-4 to the fifth linear layer B-5 to further reduce the scale of the second dimension to obtain a new two-dimensional feature shape = B * 4. Finally, use the softmax function to output the expected probability shape = B * 4 of different anomaly detection algorithms.

[0132] To verify the effectiveness of this application, real-time streaming data is selected for experimental verification. The details are as follows:

[0133] 1) Use 70% of the original data as the training set to train the anomaly detection model, and 30% of the original data as the test set to evaluate the effectiveness of the anomaly detection model.

[0134] 2) The anomaly detection model is built and trained based on the pytorch deep learning framework. The Adam optimizer is used to update the model, with the initial learning rate set to 1e-3 and the upper limit of the number of iterations set to 100.

[0135] 3) The following four commonly used anomaly detection algorithms: Z-score, Sigma-3, MA, Boxplot are selected and compared with this application on the test set.

[0136] 4) In order to intuitively reflect the superiority of this application, the following indicators are used as a comparative evaluation method: accuracy, precision and recall. Specifically, accuracy , Accuracy , recall rate ,in, To identify the correct number of outliers, To identify erroneous outlier data, The number of normal points for identifying errors, is the number of normal points that are identified incorrectly. Based on the above experimental details, Table 1 shows the evaluation indicators of the anomaly detection method of this application (i.e., Ous) and other anomaly detection algorithms (including Method, Z-score, Sigma-3, MA, and Boxplot).

[0137] Table 1

[0138]

[0139] By looking at the evaluation indicators in the above table, it can be seen that the present application has a significant improvement in accuracy and precision, and the recall rate is only 0.01 lower than the best anomaly detection algorithm Boxplot. It can be seen that the present invention is more effective than other anomaly detection algorithms in this experimental data.

[0140] Through the above description, an anomaly detection method based on reinforcement learning provided in an embodiment of the present application combines reinforcement learning, machine learning and deep learning to construct an anomaly detection model, and uses the anomaly detection model to predict the expected probability of the target data for the anomaly detection algorithm, thereby selecting the anomaly detection algorithm with the highest expected probability for the target data for anomaly detection. This can realize adaptive adjustment of the anomaly detection strategy in the application scenario and improve the accuracy of anomaly detection.

[0141] The above introduces an anomaly detection method based on reinforcement learning provided in an embodiment of the present application. The following will introduce a device for executing the above anomaly detection method based on reinforcement learning.

[0142] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a reinforcement learning anomaly detection device provided in an embodiment of the present application. Figure 6 As shown, an abnormality detection device for reinforcement learning provided in an embodiment of the present application includes:

[0143] A model training module 10 is used to train a deep learning network using a reinforcement learning method and a machine learning method to obtain an anomaly detection model;

[0144] The anomaly detection module 20 is used to obtain the target data to be detected; obtain the target numerical features corresponding to the target data by preprocessing the target data; call the anomaly detection model, and input the target numerical features into the anomaly detection model so that the anomaly detection model outputs the first expected probability of the target data for the anomaly detection algorithm; and perform anomaly detection on the target numerical features using the algorithm with the largest first expected probability in the anomaly detection algorithm to obtain the anomaly detection result of the target data.

[0145] In a possible implementation, the anomaly detection module 20 for obtaining target numerical features corresponding to the target data by preprocessing the target data is specifically used to:

[0146] Calculate the first mean, maximum, minimum, variance, skewness, kurtosis and slope of the target data; normalize the target data using the first mean and variance; normalize the maximum, minimum and variance using the first mean; normalize the skewness, kurtosis and slope by taking the inverse; integrate the normalized results of the target data, the normalized results of the maximum value, the normalized results of the minimum value, the normalized results of the variance, the normalized results of the skewness, the normalized results of the kurtosis and the normalized results of the slope to obtain the target numerical features.

[0147] In a possible implementation, the deep learning network includes a first deep learning network and a second deep learning network having the same network structure;

[0148] The model training module 10 is specifically used for:

[0149] The sample data for this iterative training is obtained by dividing the original data; the sample numerical features corresponding to the sample data are obtained by preprocessing the sample data; the sample numerical features are respectively input into the first deep learning network and the second deep learning network, so that the first deep learning network outputs the second expected probability of the sample data for the anomaly detection algorithm, and the second deep learning network outputs the third expected probability of the sample data for the anomaly detection algorithm; the sample numerical features are detected for anomalies by using the anomaly detection algorithm to obtain an anomaly detection result of the sample data; the reward score of the anomaly detection algorithm is calculated according to the anomaly detection result of the sample data; the reward score is converted into the reward score of the anomaly detection algorithm The method uses a pseudo label; penalizes the third expected probability with the pseudo label to obtain a fourth expected probability, and calculates the cross entropy loss function value of this iterative training according to the second expected probability and the fourth expected probability; when the preset end condition is not met at present, the network parameters of the first deep learning network are updated based on the cross entropy loss function value, and the step of dividing the sample data for this iterative training from the original data is returned to execute. When the preset update condition is currently met, the network parameters of the first deep learning network are assigned to the second deep learning network; when the preset end condition is currently met, the training is terminated, and the trained second deep learning network is used as an anomaly detection model.

[0150] In a possible implementation, the model training module 10 for obtaining sample data for this iterative training from the original data is specifically used for:

[0151] When the original data is time series data, the original data is divided into sample data under different time windows by using a sliding window; the sample data under the target time window is selected for this iterative training.

[0152] In a possible implementation, the anomaly detection result of the sample data includes an abnormal point set of the sample data;

[0153] The model training module 10 is used to calculate the reward score of the anomaly detection algorithm according to the anomaly detection result of the sample data, and is specifically used to:

[0154] The intersection of the outlier points is obtained by taking the intersection set of the outlier points; the difference set between the outlier point set and the outlier point intersection set is taken as the outlier point difference set; the deviation score of the outlier detection algorithm is calculated by adding points to the outliers in the outlier point intersection set and subtracting points from the outliers in the outlier point difference set; the second mean of the sample data and the Euclidean distance between the outliers in the outlier point set and the second mean are calculated; the third mean of the Euclidean distance is calculated, and the sparsity score of the outlier detection algorithm is calculated using the second mean and the third mean; the bonus score is calculated based on the deviation score and the sparsity score.

[0155] In one possible implementation, the deep learning network includes: an encoder and a decoder;

[0156] The encoder consists of a one-dimensional convolutional layer, a first multi-head self-attention mechanism layer, a first linear layer, a second multi-head self-attention mechanism layer, and a second linear layer connected in sequence;

[0157] The decoder consists of a third multi-head self-attention mechanism layer, a third linear layer, a fourth multi-head self-attention mechanism layer, a fourth linear layer, a fifth linear layer and a soft function connected in sequence;

[0158] Among them, the output of the first multi-head self-attention mechanism layer is used as the input of the first linear layer and the fourth linear layer respectively, the output of the third multi-head self-attention mechanism layer is used as the input of the second linear layer and the third linear layer respectively, a dimension conversion operation and a position encoding operation are set between the one-dimensional convolution layer and the first multi-head self-attention mechanism layer, and a random deactivation function is set after the first linear layer, the second linear layer and the third linear layer respectively.

[0159] It should be noted that the detailed functions of each module in the embodiment of the present application can be found in the corresponding public part of the above-mentioned reinforcement learning-based anomaly detection method embodiment, which will not be repeated here.

[0160] The present application also provides an electronic device in an embodiment. Figure 7 , Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0161] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0162] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0163] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the reinforcement learning-based anomaly detection methods provided in the embodiments of the present application.

[0164] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the reinforcement learning-based anomaly detection methods provided in the embodiment of the present application.

[0165] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0166] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0167] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0168] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A reinforcement learning-based anomaly detection method, characterized in that: The anomaly detection method based on reinforcement learning includes: Acquire target data to be detected; By preprocessing the target data, a target numerical feature corresponding to the target data is obtained; Retrieving a pre-trained anomaly detection model, and inputting the target numerical feature into the anomaly detection model, so that the anomaly detection model outputs a first expected probability of the target data to the anomaly detection algorithm; wherein the anomaly detection model is obtained by training a deep learning network using a reinforcement learning method and a machine learning method; Anomaly detection is performed on the target numerical feature using the algorithm with the largest first expected probability in the anomaly detection algorithm to obtain an anomaly detection result of the target data.

2. The anomaly detection method based on reinforcement learning according to claim 1, characterized in that: The preprocessing of the target data to obtain the target numerical features corresponding to the target data includes: Calculate the first mean, maximum value, minimum value, variance, skewness, kurtosis and slope of the target data; Normalizing the target data using the first mean and the variance; Normalizing the maximum value, the minimum value, and the variance respectively using the first mean; Normalizing the skewness, kurtosis and slope respectively by taking the inverse thereof; The normalized result of the target data, the normalized result of the maximum value, the normalized result of the minimum value, the normalized result of the variance, the normalized result of the skewness, the normalized result of the kurtosis and the normalized result of the slope are integrated to obtain the target numerical feature.

3. The anomaly detection method based on reinforcement learning according to claim 1, characterized in that: The deep learning network includes a first deep learning network and a second deep learning network having the same network structure; The process of training the deep learning network using the reinforcement learning method and the machine learning method to obtain the anomaly detection model includes: Divide the original data to obtain sample data for this iterative training; By preprocessing the sample data, a sample numerical feature corresponding to the sample data is obtained; Inputting the sample numerical features into the first deep learning network and the second deep learning network respectively, so that the first deep learning network outputs a second expected probability of the sample data for the anomaly detection algorithm, and the second deep learning network outputs a third expected probability of the sample data for the anomaly detection algorithm; Performing anomaly detection on the sample numerical features using the anomaly detection algorithm to obtain an anomaly detection result of the sample data; Calculating a reward score of the anomaly detection algorithm according to an anomaly detection result of the sample data; Converting the reward score into a pseudo label for the anomaly detection algorithm; Penalizing the third expected probability with the pseudo label to obtain a fourth expected probability, and calculating a cross entropy loss function value of this iterative training according to the second expected probability and the fourth expected probability; When the preset end condition is not met at present, the network parameters of the first deep learning network are updated based on the cross entropy loss function value, and the step of dividing the original data to obtain the sample data for this iterative training is returned to be executed. When the preset update condition is currently met, the network parameters of the first deep learning network are assigned to the second deep learning network; When the preset end condition is currently met, the training is ended, and the trained second deep learning network is used as the anomaly detection model.

4. The anomaly detection method based on reinforcement learning according to claim 3, characterized in that: The sample data for this iterative training is obtained by dividing the original data, including: In the case where the original data is time series data, the original data is divided by a sliding window to obtain sample data in different time windows; Select the sample data under the target time window for this iterative training.

5. The anomaly detection method based on reinforcement learning according to claim 3, characterized in that: The anomaly detection result of the sample data includes an abnormal point set of the sample data; The calculating the reward score of the anomaly detection algorithm according to the anomaly detection result of the sample data includes: Taking the intersection of the outlier point sets to obtain an outlier point intersection set; Taking the difference between the outlier point set and the outlier point intersection set as the outlier point difference set set; Calculate the deviation score of the anomaly detection algorithm by adding points to the anomaly points in the anomaly point intersection set and deducting points from the anomaly points in the anomaly point difference set; Calculating a second mean of the sample data and a Euclidean distance between an outlier in the outlier set and the second mean; Calculating a third mean of the Euclidean distance, and calculating a sparsity score of the anomaly detection algorithm using the second mean and the third mean; The reward score is calculated based on the deviation score and the sparsity score.

6. The reinforcement learning-based anomaly detection method according to any one of claims 1 to 5, characterized in that: The deep learning network includes: an encoder and a decoder; The encoder consists of a one-dimensional convolutional layer, a first multi-head self-attention mechanism layer, a first linear layer, a second multi-head self-attention mechanism layer and a second linear layer connected in sequence; The decoder is composed of a third multi-head self-attention mechanism layer, a third linear layer, a fourth multi-head self-attention mechanism layer, a fourth linear layer, a fifth linear layer and a soft function connected in sequence; Among them, the output of the first multi-head self-attention mechanism layer is used as the input of the first linear layer and the fourth linear layer respectively, the output of the third multi-head self-attention mechanism layer is used as the input of the second linear layer and the third linear layer respectively, a dimension conversion operation and a position encoding operation are arranged between the one-dimensional convolution layer and the first multi-head self-attention mechanism layer, and a random deactivation function is arranged after the first linear layer, the second linear layer and the third linear layer respectively.

7. An anomaly detection device based on reinforcement learning, characterized in that: The abnormality detection device based on reinforcement learning includes: A model training module is used to train the deep learning network using reinforcement learning methods and machine learning methods to obtain an anomaly detection model; The anomaly detection module is used to obtain the target data to be detected; obtain the target numerical features corresponding to the target data by preprocessing the target data; call the anomaly detection model and input the target numerical features into the anomaly detection model so that the anomaly detection model outputs the first expected probability of the target data for the anomaly detection algorithm; and perform anomaly detection on the target numerical features using the algorithm with the largest first expected probability in the anomaly detection algorithm to obtain the anomaly detection result of the target data.

8. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the reinforcement learning-based anomaly detection method as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the reinforcement learning-based anomaly detection method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the reinforcement learning-based anomaly detection method as described in any one of claims 1 to 6.

Images