Anomaly detection method fusing active learning and deep Q network

Abstract

The application relates to an anomaly detection method combining active learning and a deep Q network, which comprises the following steps: acquiring a single-variable time sequence, adopting an isolated forest as an unsupervised anomaly score generator to calculate an anomaly score; modeling a time sequence dynamic threshold adjustment problem into a Markov decision process, defining a state space, an action space reward function and constructing a double-network DQN architecture; inputting a current state into the double-network DQN architecture to obtain an action Q value and calculate decision uncertainty; constructing a multi-strategy label propagation mechanism to obtain a training label set; performing active learning iterative training, taking a state vector as input, using the trained double-network DQN architecture to obtain an action Q value, selecting the action value with the maximum Q value as a final detection result, and realizing anomaly detection combining active learning and the deep Q network. Compared with the prior art, the application has the advantages of reducing calculation complexity, expanding anomaly samples and the like.

Description

Technical Field

[0001] This invention relates to the field of industrial monitoring, and in particular to an anomaly detection method that integrates active learning and deep Q-networks. Background Technology

[0002] Time series anomaly detection is a core technology in fields such as industrial monitoring, intelligent operation and maintenance, and aerospace equipment monitoring. It is used to identify abnormal events that deviate from normal operating patterns from continuous time series data. In practical applications, anomaly samples are naturally scarce and the cost of annotation by domain experts is high. Anomaly detection in scenarios with extreme annotation budgets (requiring annotation of less than 0.1% of the total sample size) has become a core problem that the industry urgently needs to solve.

[0003] Currently, temporal anomaly detection techniques for low-label scenarios can be mainly divided into three categories: The first category is unsupervised methods, such as Isolation Forest (IF), Autoencoder (AE), and TranAD, which do not require labeled data but have low detection accuracy and high false alarm rate; the second category is traditional active learning methods, which use greedy query strategies and have large model bias in the cold start phase; the third category is dynamic thresholding methods based on reinforcement learning, such as the ADT framework, which models threshold adjustment as a Markov decision process, but depends on initial labeled samples and does not solve the problem of insufficient training samples under extreme labeling conditions.

[0004] Chinese invention patent CN119759797A discloses an embedded software temporal anomaly detection method based on value function reinforcement learning. The method includes: acquiring temporal data and dividing it into temporal segments; constructing a state space matrix; extracting feature vectors using a long short-term memory network; generating a Q-value function network based on a deep neural network; constructing a reward function based on Mahalanobis distance; training the network and optimizing parameters using a deep Q-learning algorithm; and finally achieving anomaly detection. This invention effectively captures long-term dependencies in temporal data, improves the accuracy of anomaly detection, reduces the false alarm rate, and has strong generalization ability. However, it still suffers from the cold-start problem faced by traditional methods in extreme annotation scenarios, insufficient annotation budgets to support effective training, and overfitting to a small number of labeled samples, leading to a rapid decline in detection accuracy in scenarios with distribution drift.

[0005] In summary, there is currently a lack of an anomaly detection method that integrates active learning and deep Q-networks to solve or partially solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an anomaly detection method that integrates active learning and deep Q-networks. This method aims to solve or partially solve the problems faced by traditional methods in extreme annotation scenarios, such as cold start, insufficient annotation budget to support effective training, overfitting to a small number of labeled samples, and rapid decline in detection accuracy in distribution drift scenarios.

[0007] The objective of this invention can be achieved through the following technical solutions: This invention provides an anomaly detection method that integrates active learning and deep Q-networks, the method comprising: S1. Obtain univariate time series data, and use an isolated forest as an unsupervised anomaly score generator to calculate the anomaly score for each time series window in the univariate time series data. S2. The time series dynamic threshold adjustment problem is modeled as a Markov decision process. Based on the anomaly score, the state space and action space are defined, the reward function is set, and a dual-network DQN architecture is constructed. S3. Input the current state into the dual-network DQN architecture to obtain the action Q value, and calculate the decision uncertainty based on the action Q value; S4. Based on the anomaly score and temporal proximity, a multi-strategy label propagation mechanism is constructed to obtain the training label set. The multi-strategy label propagation mechanism includes embedding space Gaussian kernel similarity propagation, temporal proximity propagation, and K-nearest neighbor voting propagation. S5. Based on the aforementioned decision uncertainty, active learning iterative training is performed using the training annotation set. The state vector is used as input, and the trained dual-network DQN architecture is used to obtain the action Q value. The action value with the largest Q value is selected as the final detection result, thereby realizing the anomaly detection that integrates active learning and deep Q network.

[0008] As a preferred technical solution, the univariate time series is divided using an overlapping sliding window.

[0009] As a preferred technical solution, if the unsupervised anomaly score generator is covered by multiple windows at a certain time point, the arithmetic mean of the scores of all windows is taken as the anomaly score at that time point.

[0010] As a preferred technical solution, the state space uses a six-dimensional vector to represent the current state, including the mean and variance of the anomaly scores at the most recent time step, as well as the true positive rate, false positive rate, true negative rate, and false negative rate within the recent window.

[0011] As a preferred technical solution, the action space adopts a discrete binary action set {0,1}, where 0 indicates that it is judged as normal and 1 indicates that it is judged as abnormal.

[0012] As a preferred technical solution, the reward function is constructed based on the anomaly detection results. A positive reward is given when the prediction is consistent with the actual labeled normal or abnormal label, and a negative reward is given when they are inconsistent.

[0013] As a preferred technical solution, the dual-network DQN architecture adopts a structure that separates the online network from the target network.

[0014] As a preferred technical solution, the online network includes an input layer, a hidden layer, and an output layer, and the target network synchronizes the weights of the online network during the training process.

[0015] As a preferred technical solution, the embedding space Gaussian kernel similarity propagation uses the anomaly score as the embedding vector, calculates the standardized Euclidean distance between unlabeled samples and labeled samples, converts it into similarity through a Gaussian kernel, and determines the label based on the similarity of the nearest normal or anomaly sample. The temporal proximity propagation utilizes temporal proximity to statistically analyze the label distribution of labeled samples within a window. If the proportion of a certain type of label exceeds a preset threshold, the corresponding label is propagated. The K-nearest neighbor voting propagation uses nearest neighbor voting to determine the label when the number of labeled samples reaches a threshold.

[0016] As a preferred technical solution, the active learning iterative training process includes: The dual-network DQN architecture is trained using an initial seed annotation set. Based on the action Q-values ​​obtained during training and the decision uncertainty, a query sample is selected. After expert annotation, the annotation set is expanded using the multi-strategy label propagation mechanism. The dual-network DQN architecture is then retrained based on the expanded annotation set. The training process is repeated until the preset model performance evaluation conditions are met, resulting in a trained dual-network DQN architecture.

[0017] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) The present invention directly measures decision uncertainty based on the difference between the Q values ​​of two actions output by the dual-network DQN architecture, without the need to introduce an additional independent uncertainty estimation model. Compared with the scheme that requires the introduction of an additional model for uncertainty estimation, the computational complexity is greatly reduced. Since the action Q value itself reflects the impact of the current decision on the future cumulative return, the query samples selected based on the difference in Q value can preferentially cover the samples that contribute the most to the long-term detection performance improvement of the model. Thus, under the extreme labeling condition that the initial seed set accounts for only a very small number of total samples, the model F1-score can reach more than 0.7, which effectively solves the cold start problem faced by traditional methods in extreme labeling scenarios.

[0018] (2) The multi-strategy label propagation mechanism constructed in this invention integrates three complementary strategies: embedding space Gaussian kernel similarity propagation, temporal proximity propagation and K-nearest neighbor voting propagation. It also sets a differentiated confidence threshold to take into account the natural scarcity of abnormal samples. Under the premise of ensuring the quality of pseudo-labels and avoiding false propagation, it maximizes the effective diffusion range of labels, solves the problem of insufficient labeling budget that cannot support effective training, and significantly reduces the cost of manual labeling in practical applications.

[0019] (3) This invention uses the decision uncertainty output by the dual-network DQN architecture as the basis for active learning query sample selection. The real labels obtained by expert annotation and the pseudo labels obtained by multi-strategy label propagation are used together for updating and iterating the dual-network DQN architecture. A seed initialization strategy based on class balance and a performance-driven early stopping mechanism are introduced to improve the training stability and generalization ability in extreme annotation scenarios. This solves the problem of overfitting a small number of labeled samples and the rapid decline in detection accuracy in distribution drift scenarios, and achieves the technical effect of significantly improving anomaly detection and overall model performance. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the main steps of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] Example 1 To address the problems existing in the aforementioned prior art, this embodiment provides an anomaly detection method that integrates active learning and deep Q-networks, such as... Figure 1 As shown, it specifically includes: S1. Obtain univariate time series data, and use an isolated forest as an unsupervised anomaly score generator to calculate the anomaly score for each time series window in the univariate time series.

[0023] Specifically, the input univariate time series is divided using an overlapping sliding window with a window length of 12 and a step size of 1. An Isolation Forest (IF) is used as an unsupervised anomaly score generator, which can quickly calculate the anomaly score for each window without training. If a time point is covered by multiple windows, the arithmetic mean of all window scores is taken as the final anomaly score for that point, eliminating window boundary effects.

[0024] S2. The problem of dynamic threshold adjustment in time series is modeled as a Markov decision process. The state space and action space are defined based on the anomaly score, the reward function is set based on the instantaneous classification result, and a dual-network DQN architecture is constructed.

[0025] The state space uses a 6-dimensional vector to represent the current state, including the mean and variance of the anomaly scores of the most recent k time steps, as well as the true positive rate, false positive rate, true negative rate, and false negative rate within the recent window.

[0026] The action space uses a discrete binary action set {0,1}, where 0 indicates a normal action and 1 indicates an abnormal action.

[0027] The reward function is designed based on the real-time classification results. A +1 reward is given when the prediction matches the true label, and a -1 reward is given when they do not match.

[0028] The dual-network DQN architecture adopts a structure that separates the online network and the target network. The online network includes an input layer (6-dimensional), two hidden layers (32 neurons and 16 neurons) and an output layer (2-dimensional). The target network synchronizes the weights of the online network every 100 training rounds.

[0029] The Adam optimizer is used with a learning rate of 1×10⁻⁶. -4 Discount factor 0.99, experience replay buffer capacity 20000.

[0030] S3. Input the current state into the dual-network DQN architecture to obtain the action Q value, and calculate the decision uncertainty based on the action Q value.

[0031] The decision uncertainty can be calculated directly using the Q-values ​​of the two actions output by DQN, without the need for an additional model. The smaller the absolute value of the Q-difference, the lower the model's confidence in the current decision, the closer the sample is to the decision boundary, and the greater the value of labeling in improving the long-term performance of the model.

[0032] S4. Based on anomaly scores and temporal proximity, a multi-strategy label propagation mechanism is constructed to obtain the training label set. The multi-strategy label propagation mechanism includes embedding space Gaussian kernel similarity propagation, temporal proximity propagation, and K-nearest neighbor voting propagation.

[0033] To expand the training set without consuming the labeling budget, three label propagation strategies are combined. Embedding space Gaussian kernel similarity propagation uses anomaly scores as embedding vectors, calculates the standardized Euclidean distance between unlabeled and labeled samples, converts it to similarity using a Gaussian kernel, and determines the label based on the similarity to the nearest normal / anomaly sample. Temporal proximity propagation utilizes temporal proximity, statistically analyzing the label distribution of labeled samples within a 5-step window; if the proportion of a certain label exceeds 0.5, the corresponding label is propagated. K-nearest neighbor voting propagation uses K=10 nearest neighbor voting to determine the label when the number of labeled samples reaches a threshold.

[0034] The average confidence scores of the three strategies were taken, with the confidence threshold for the normal class set at 0.85 and the threshold for the abnormal class set at 0.60. This promotes the diffusion of scarce abnormal labels, and only high-confidence samples are labeled as pseudo-labels.

[0035] Based on decision uncertainty, active learning iterative training is performed using the training annotation set. The state vector is used as input, and the trained dual-network DQN architecture is used to obtain the action Q value. The action value with the largest Q value is selected as the final detection result, thus realizing the anomaly detection that integrates active learning and deep Q network.

[0036] The annotation window is defined centered on the first true anomaly. Samples representing 50% of the total budget are sampled according to category balance as the initial seed set. A dual-network DQN is trained based on the current annotation set. Q-differences are calculated for the entire sequence, and anomaly detection is performed. Samples for the current round of queries are selected in ascending order of uncertainty. After expert annotation, multi-strategy label propagation is used to expand the annotation set. Finally, the model performance is evaluated. 20 samples are queried in each round. Training is terminated early if the total budget is exhausted or the F1-score improvement is less than 0.001 for 3 consecutive rounds.

[0037] Example 2 This embodiment applies the anomaly detection method RADT, which integrates active learning and deep Q-networks, provided by this invention, and implements it on the UCR Anomaly Archive dataset. The UCR Anomaly Archive is a comprehensive archive specifically built for time series anomaly detection tasks, covering multiple application fields such as finance, transportation, healthcare, and industry, and containing more than 250 real-world time series datasets.

[0038] This embodiment is applied to the scenario of abnormal power generation detection in photovoltaic power plants. A centralized inverter in a photovoltaic power plant collects output power data at fixed intervals, forming a univariate time series. Due to various factors such as shading, component aging, inverter failure, and grid-side power scheduling, abnormal fluctuations may occur in the power series. However, the abnormal patterns are complex and diverse, and normal fluctuations are difficult to distinguish from real faults using fixed thresholds. Furthermore, manual investigation of each abnormal point by on-site maintenance personnel is very costly. This is precisely the typical scenario of anomaly detection under extreme labeling that this invention addresses. To comprehensively quantify the algorithm's performance in time series anomaly detection tasks, F1 score and UCR score (UCR score, a specific evaluation metric for UCR time series anomaly datasets) are used. For each sequence, the time point with the highest model output anomaly score is extracted, and it is determined whether this point falls within the tolerance window of 100 points before and after the actual anomaly interval; a hit is considered correct. The final score is the percentage of hits in all sequences, with a value of 0-1 as the main evaluation metric.

[0039] To verify the detection performance of RADT, the following representative anomaly detection methods were selected as baselines for comparison. The parameter configurations of the baseline methods are shown in Table 1. These include: Isolation Forest, an ensemble unsupervised detection method based on random splitting trees; Autoencoder (AE), an unsupervised deep learning method based on reconstruction error; TranAD, a multivariate time series anomaly detection method based on Transformer adversarial training; and MDI (Maximum Divergence Interval), a maximum divergence interval anomaly score generator. RADT is a dynamic threshold anomaly detection framework based on deep reinforcement learning, modeling threshold adjustment as a Markov decision process and learning an adaptive threshold policy through DQN. The parameter configurations of the RADT method are shown in Table 2.

[0040] Table 1 Baseline Method Parameter Configuration Table 2 RADT Method Parameter Configuration Table 3 shows a comparison between RADT and unsupervised methods. The results in Table 3 demonstrate that RADT significantly outperforms traditional unsupervised baseline methods such as IF, AE, TranAD, and MDI in F1 score. While unsupervised methods do not require labeled data, their anomaly detection performance is limited by prior assumptions about the data distribution. IF, based on the assumption that outliers are more easily isolated in random segmentation, is unstable in detecting group anomalies or fine-grained structural differences. AE and TranAD, based on reconstruction or prediction error detection paradigms, struggle to adaptively adjust reconstruction error thresholds when faced with scarce labels and domain-specific anomaly definitions, leading to high false positives or false negatives. In contrast, RADT, through a combination of reinforcement learning and active learning, can learn decision strategies more adaptable to data distribution evolution in a dynamic environment than static thresholds, even with minimal labeled information, validating RADT's significant advantages in semi-supervised time series anomaly detection.

[0041] Table 3. Overall Performance Comparison of Anomaly Detection Methods The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An anomaly detection method integrating active learning and deep Q-networks, characterized in that, The method includes: S1. Obtain univariate time series data, and use an isolated forest as an unsupervised anomaly score generator to calculate the anomaly score for each time series window in the univariate time series data. S2. The time series dynamic threshold adjustment problem is modeled as a Markov decision process. Based on the anomaly score, the state space and action space are defined, the reward function is set, and a dual-network DQN architecture is constructed. S3. Input the current state into the dual-network DQN architecture to obtain the action Q value, and calculate the decision uncertainty based on the action Q value; S4. Based on the anomaly score and temporal proximity, a multi-strategy label propagation mechanism is constructed to obtain the training label set. The multi-strategy label propagation mechanism includes embedding space Gaussian kernel similarity propagation, temporal proximity propagation, and K-nearest neighbor voting propagation. S5. Based on the aforementioned decision uncertainty, active learning iterative training is performed using the training annotation set. The state vector is used as input, and the trained dual-network DQN architecture is used to obtain the action Q value. The action value with the largest Q value is selected as the final detection result, thereby realizing the anomaly detection that integrates active learning and deep Q network.

2. The anomaly detection method integrating active learning and deep Q-networks according to claim 1, characterized in that, The univariate time series is divided using an overlapping sliding window.

3. The anomaly detection method integrating active learning and deep Q-networks according to claim 1, characterized in that, If the unsupervised anomaly score generator is covered by multiple windows at a certain time point, the arithmetic mean of the scores of all windows is taken as the anomaly score at that time point.

4. The anomaly detection method integrating active learning and deep Q-networks according to claim 1, characterized in that, The state space uses a six-dimensional vector to represent the current state, including the mean and variance of the anomaly scores at the most recent time step, as well as the true positive rate, false positive rate, true negative rate, and false negative rate within the recent window.

5. The anomaly detection method integrating active learning and deep Q-networks according to claim 1, characterized in that, The action space adopts a discrete binary action set {0,1}, where 0 indicates that it is judged as normal and 1 indicates that it is judged as abnormal.

6. The anomaly detection method integrating active learning and deep Q-networks according to claim 1, characterized in that, The reward function is constructed based on the anomaly detection results. A positive reward is given when the prediction matches the actual labeled normal or abnormal label, and a negative reward is given when they do not match.

7. The anomaly detection method integrating active learning and deep Q-networks according to claim 1, characterized in that, The dual-network DQN architecture adopts a structure that separates the online network from the target network.

8. The anomaly detection method integrating active learning and deep Q-networks according to claim 7, characterized in that, The online network includes an input layer, a hidden layer, and an output layer, and the target network synchronizes the weights of the online network during training.

9. The anomaly detection method integrating active learning and deep Q-networks according to claim 1, characterized in that, The embedding space Gaussian kernel similarity propagation uses the anomaly score as the embedding vector, calculates the standardized Euclidean distance between unlabeled and labeled samples, converts it into similarity through a Gaussian kernel, and determines the label based on the similarity of the nearest normal or anomaly sample. The temporal proximity propagation utilizes temporal proximity to statistically analyze the label distribution of labeled samples within a window. If the proportion of a certain type of label exceeds a preset threshold, the corresponding label is propagated. The K-nearest neighbor voting propagation uses nearest neighbor voting to determine the label when the number of labeled samples reaches a threshold.

10. The anomaly detection method integrating active learning and deep Q-networks according to claim 1, characterized in that, The active learning iterative training process includes: The dual-network DQN architecture is trained using an initial seed annotation set. Based on the action Q-values ​​obtained during training and the decision uncertainty, a query sample is selected. After expert annotation, the annotation set is expanded using the multi-strategy label propagation mechanism. The dual-network DQN architecture is then retrained based on the expanded annotation set. The training process is repeated until the preset model performance evaluation conditions are met, resulting in a trained dual-network DQN architecture.

Images