An online ecological observation data anomaly detection method and system

Abstract

The application discloses an online ecological observation data anomaly detection method and system, and comprises the following detection process: learning a data drift detection method from historical ecological observation data, establishing a historical data anomaly detection model and a data drift segmentation list required by data drift detection; based on the historical data anomaly detection model, fine-tuning is performed to obtain an online anomaly detection model of a current drift segmentation, and the online anomaly detection model is used for online anomaly detection; when the training data is insufficient, similar data is matched from the drift segmentation list of the historical ecological observation data to enhance the fine-tuning data training amount. The model learned by the historical data and the historical data enhanced training sample are used for drift detection on online observation data, and an online drift online data anomaly detection model is obtained through fine-tuning training, so that the online data anomaly detection accuracy is improved.

Description

Technical Field

[0001] This invention relates to a data anomaly detection method and system, and more particularly to an online ecological observation data anomaly detection method and system. Background Technology

[0002] The ecological observation data received by the National Ecological Data Center is characterized by a large number of observation indicators and data reporting stations. Therefore, different ecological observation indicators require data quality control to ensure that the data aggregated into the data center is usable. Outlier detection is an important part of quality data control, defined as identifying abnormal data from normal time series.

[0003] Anomaly detection in time series data is a classic problem with numerous methods, including statistical (such as the 3σ rule), prediction-based (such as ARIMA and exponential smoothing), machine learning-based (such as LOF, Isolation Forest, and One-class SVM), and deep learning-based (such as autoencoders, GANs, and variational autoencoders). Based on runtime efficiency requirements, anomaly detection can be categorized into online and offline methods, with online anomaly detection placing higher demands on the algorithm's adaptability.

[0004] In ecological observation data, some observation indicators, such as soil moisture, surface water quality, and air pollutant concentration, are affected by the environment and change the distribution pattern of the data. For example, precipitation and irrigation can change the distribution of soil moisture observation data, resulting in data drift.

[0005] While many methods exist for anomaly detection, traditional anomaly detection algorithms generally assume a stable data distribution pattern. When applied to observations with shifted data distributions, their performance typically degrades, leading to decreased accuracy in anomaly detection. Therefore, when data distribution changes, algorithm adjustments are necessary. Some high-precision anomaly detection algorithms, such as deep learning algorithms, often require specific training data for fine-tuning. Insufficient online data can affect model convergence, easily causing fine-tuning failures and reducing the accuracy of the adjusted model.

[0006] Furthermore, data drift (concept drift) detection is a technique used to detect changes in the distribution of input data. There are three main types of data drift detection methods:

[0007] 1) Error rate-based detection algorithms are the most widely used algorithms in data classification, and their main focus is on tracking changes in the online error rate of the base classifier;

[0008] 2) Drift detection based on data distribution: This type of algorithm uses a distance function / metric to quantify the difference between the distribution of historical data and the distribution of new data;

[0009] 3) The multiple hypothesis testing drift detection algorithm uses multiple hypothesis testing to detect different concept drifts.

[0010] One of the most frequently cited concept drift detection algorithms is the Data Drift Detection (DDM) method. However, existing data drift methods are mainly applied to data classification scenarios and have difficulty detecting drift in data sequences. Furthermore, ecological observation data is subject to environmental changes, and there is currently a lack of stable online methods for detecting drift in ecological observation data. Summary of the Invention

[0011] To address the shortcomings of the aforementioned technologies, this invention provides a method and system for detecting anomalies in online ecological observation data.

[0012] To solve the above technical problems, the technical solution adopted by the present invention is: an online ecological observation data anomaly detection method, the detection method including the following detection process:

[0013] We learn data drift detection methods from historical ecological observation data, establish a historical data anomaly detection model required for data drift detection, and build a list of drift segments in historical ecological observation data.

[0014] Online detection of anomalies in ecological observation data: Detecting drift segments in online data, based on historical data anomaly detection models, fine-tuning the online anomaly detection model for the current drift segment, and then using the online anomaly detection model to perform anomaly detection online;

[0015] When there is insufficient training data for the current drift segments, similar data are matched from the list of drift segments in historical ecological observation data to enhance the amount of training data for fine-tuning.

[0016] Furthermore, a historical data anomaly detection model is trained to obtain historical anomaly marker sequences. The specific process is as follows:

[0017] a) Train the LSTM-AE model using time series data (H) from historical ecological observation data to obtain the historical data anomaly detection model needed for detection. Then through the model Historical ecological observation data are reconstructed to obtain the difference sequence R{r1,r2,...r} between the original sequence and the reconstructed sequence. n};

[0018] b) Calculate the standard deviation of the difference sequence R, and use 3 times the standard deviation as the anomaly detection threshold σ;

[0019] c) In the difference sequence R, observations not exceeding the threshold σ are considered normal observations and are marked as 0, while observations exceeding the threshold σ are considered abnormal observations and are marked as 1. Thus, the difference sequence R is transformed into an anomaly label sequence U{u1, u2, ... u... i ...u n}, where u i ∈{0,1}.

[0020] Furthermore, the process for detecting the drift segmentation list of historical ecological observation data is as follows:

[0021] Using the acquired anomaly detection marker sequence U, and based on the DDM algorithm, data drift in historical ecological observation data sequences is detected. The original sequence data is segmented by data drift, transforming the time series H into a segmented data sequence Hs{{H s1},{H s2},{H si},...,{H sm}, where {H si} represents a continuous time segment of the original sequence data {h j h j+1 ,...h j+k Set the drift start position D. start If the value is 0, perform drift detection;

[0022] For the data segment sequence Hs{{H s1},{H s2},{H si},...,{H sm For each segment, calculate its feature vector, and use the feature vector as an index to build a list of historical data drift segments.

[0023] Furthermore, the specific process of drift detection is as follows:

[0024] a) Based on the obtained historical anomaly marker sequence U, sequentially take the anomaly value u at index i. i According to the DDM algorithm, calculate the variable p i and s i ;

[0025] p i The calculation method is as follows:

[0026] i. Initialize m = 0;

[0027] ii. Calculate sequentially

[0028] iii.p i =m;

[0029] s i The calculation method is as follows:

[0030]

[0031] b) If p i and s i If Formula 1 is satisfied, then data drift is considered to have occurred at index i, denoted as D. end For i-1, from the time series H according to the index interval (D) start D end Take data to form a segmented sequence Add to the segmented sequence Hs and set D. start For i, repeat steps a) and b) above until all data in time series H have been detected;

[0032] p i +s i ≥p min +3*s min (1)

[0033] Where, p min s min p, representing the interval (j, i) i and s i The minimum value.

[0034] Furthermore, extract segments {H} from the data segment sequence Hs. si}, establish segmentation {H si} eigenvector V si By combining segmented data and their feature vectors, a historical data drift segment list D is established. L {V si , {H si}}, where si∈{s1, s2, ... s m}

[0035] Furthermore, the feature vectors selected include mean, median, minimum, maximum, standard deviation, skewness, kurtosis, first-order difference mean, and first-order difference median.

[0036] Furthermore, online detection of anomalies in ecological observation data includes the following steps:

[0037] (2.1) Obtain the anomaly detection threshold σ of historical data, and the historical data anomaly detection model. Historical data drift segment list D L {V si , {H si}};Initial online observation data drift point D start The value is 0, the algorithm parameter is γ, and L min Wherein, parameter γ is the learning rate during online model fine-tuning, and parameter L... minThis represents the minimum amount of training data required for fine-tuning the online model.

[0038] (2.2) Detection of online data drift points: Create an online data buffer window W to receive online data blocks {h} t1 ,h t2 ,...,h tk The data is added to the end of window W in chronological order, using a historical data anomaly detection model. Perform anomaly calculations and generate an online anomaly identifier sequence U with the same width as window W. w Using formula (1) in the DDM method, abnormal drift points are detected; if in window W, D end If location data drift is detected, proceed to the next step (2.3);

[0039] (2.3) Learning the online observation data anomaly detection model: using historical data anomaly detection models Fine-tuning is performed on the basic model to obtain the current online anomaly detection model;

[0040] (2.4) Online detection of anomalies in online observation data: using an online anomaly detection model And the threshold σ, for the data sequence H D Perform anomaly detection, report the anomaly detection results, and send the data sequence H D Move out of window W and put D start Set to D end Repeat steps (2.2)-(2.4) to continuously perform online detection of data anomalies.

[0041] Furthermore, the specific process of model fine-tuning is as follows:

[0042] a) According to the index range (D) in (2.2) start D end Retrieve data sequence H from window W D Added to the model fine-tuning training dataset T D {H D In}, set D start D end If the data sequence H D The length is less than parameter L min Then calculate H D Feature vector V D Using the feature vector V D Based on the method of minimizing the characteristic geometric distance, the historical data drift segment list D is divided into segments. L Matching segments H sk And add it to the dataset to form dataset T. D {H D Hsk};

[0043] b) Transfer the data set T D The data is divided into training and testing sets, with a learning rate of γ. This is used to develop an anomaly detection model for historical data. Perform training and fine-tuning to obtain an online anomaly detection model.

[0044] A data anomaly detection system, applied to the online ecological observation data anomaly detection method of claim 1, comprises the following modules:

[0045] Historical data drift detection and drift segment list management module: used for drift segment detection, drift segment storage, addition, and deletion, and provides drift segment matching service for online data anomaly monitoring;

[0046] Online Data Anomaly Detection Module: This module is used to receive ecological observation data online, detect anomalies in the received ecological observation data, mark abnormal data, and report the online anomaly detection results.

[0047] This invention discloses a method and system for detecting anomalies in online ecological observation data, which has the following technical advantages:

[0048] 1) This invention addresses the current instability of online ecological observation data drift detection. It learns a data drift detection method from long-term historical observation data and uses this method to detect online data drift. This allows the online data anomaly detection model to be fine-tuned based on the data drift, improving the consistency and accuracy of online ecological observation data drift detection. Furthermore, it creates conditions for subsequent data augmentation and online anomaly detection model learning.

[0049] 2) This invention addresses the reliability of online observation data training. Addressing the issue that insufficient sample size during deep learning model fine-tuning can lead to convergence difficulties and reduced model accuracy, this invention segments historical ecological observation data over a long period to create a historical data drift block list. When fine-tuning the model across online data partitions, the most similar data segments are matched from this list, thereby increasing the amount of training data, improving the reliability of model fine-tuning, and enhancing the accuracy of the anomaly detection model. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the technical solution of the present invention.

[0051] Figure 2 This is a visualization of the segmented soil moisture data in the example. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0053] This invention discloses an online ecological observation data anomaly detection method, which is generally divided into two parts. The first part learns the data drift detection method from historical ecological observation data, establishes the historical data anomaly detection model required for data drift detection, and establishes a drift segment list of historical ecological observation data. The second part is to perform online anomaly detection of ecological observation data.

[0054] Ecological observation data is acquired through automated observation instruments, including monitoring data on groundwater levels, water quality, and air pollutant concentrations, with the time series denoted as H{h1,h2,...h}. n The specific testing methods are as follows:

[0055] I. Learning methods for detecting historical ecological observation data drift and establishing a data drift segment list

[0056] (1.1) First, a historical data anomaly detection model is learned, and a historical anomaly label sequence is obtained; the commonly used LSTM-AE (Long Short-Term Memory Network-Autoencoder) deep learning model is adopted, and its basic process is as follows:

[0057] a) Train the LSTM-AE model using time series data (H) from historical ecological observation data to obtain the historical data anomaly detection model needed for detection. Then through the model Historical ecological observation data are reconstructed to obtain the difference sequence R{r1,r2,...r} between the original sequence and the reconstructed sequence. n};

[0058] b) Calculate the standard deviation of the difference sequence R, and use 3 times the standard deviation as the anomaly detection threshold σ;

[0059] c) In the difference sequence R, observations not exceeding the threshold σ are considered normal observations and are marked as 0, while observations exceeding the threshold σ are considered abnormal observations and are marked as 1. Thus, the difference sequence R is transformed into an anomaly label sequence U{u1, u2, ... u... i ...u n}, where u i ∈{0,1};

[0060] (1.2) Historical ecological observation data drift segmentation;

[0061] Using the anomaly detection marker sequence U obtained in (1.1), and referring to the DDM (Drift Detection Method) algorithm, data drift in the historical ecological observation data sequence is detected. The original sequence data is segmented by data drift, and the time series H is transformed into a data segment sequence Hs{{H s1},{H s2},{H si},...,{H sm}, where {H si} represents a continuous time segment of the original sequence data {h j h j+1 ,...h j+k Set the drift start position D. start The value is 0. The drift detection process is as follows:

[0062] a) Based on the historical anomaly marker sequence U obtained in (1.1), take the anomaly value u at index i in sequence. i According to the DDM algorithm, calculate the variable p i and s i ;

[0063] p i The calculation method is as follows:

[0064] i. Initialize m = 0;

[0065] ii. Calculate sequentially

[0066] iii.p i =m;

[0067] s i The calculation method is as follows:

[0068]

[0069] b) If p i and s i If Formula 1 is satisfied, then data drift is considered to have occurred at index i, denoted as D. end For i-1, from the time series H according to the index interval (D) start D end Take data to form a segmented sequence Add to segmented sequence H s and set D start For i, repeat steps a) and b) in 1.2 above until all data in time series H have been detected;

[0070] Formula 1 is shown below:

[0071] p i +s i≥p min +3*s min (1)

[0072] Where, p min s min p, representing the interval (j, i) i and s i The minimum value.

[0073] (1.3) Establish a historical data drift segment list; for the data segment sequence Hs{{H in (1.2) s1},{H s2},{H si},...,{H sm For each segment of the dataset, its eigenvector is calculated. Using the eigenvector as an index, a list of historical data drift segments is created. The process is as follows:

[0074] a) Calculate the characteristics of each time series segment: Extract the segment {H} from the data segment sequence Hs. si}, calculate multiple features of the sequence, and establish segmentation {H}. si} eigenvector V si In the implementation of the invention, the feature vector uses nine features: mean, median, minimum, maximum, standard deviation, skewness, kurtosis, first-order difference mean, and first-order difference median.

[0075] b) Combine the segmented data and their feature vectors to create a historical data drift segment list D. L {V si , {H si}}, where si∈{s1, s2, ... s m}

[0076] II. Online Detection of Anomalies in Ecological Observation Data

[0077] (2.1) Obtain the anomaly detection threshold σ of historical data, and the historical data anomaly detection model. Historical data drift segment list D L {V si , {H si}};Initial online observation data drift point D start The value is 0, the algorithm parameter is γ, and L min Wherein, parameter γ is the learning rate during online model fine-tuning, and parameter L... min This represents the minimum amount of training data required for fine-tuning the online model.

[0078] (2.2) Detection of online data drift points: Create an online data buffer window W to receive online data blocks {h} t1 ,h t2 ,...,htk The data is added to the end of window W in chronological order, using a historical data anomaly detection model. Perform anomaly calculations and generate an online anomaly identifier sequence U with the same width as window W. w Using formula (1) in the DDM method, abnormal drift points are detected; if in window W, D end If location data drift is detected, proceed to the next step (2.3);

[0079] (2.3) Learning the online observation data anomaly detection model: using historical data anomaly detection models Fine-tuning is performed on the basic model to obtain the current online anomaly detection model;

[0080] The specific process is as follows:

[0081] a) According to the index range (D) in (2.2) start D end Retrieve data sequence H from window W D Added to the model fine-tuning training dataset T D {H D In}, set D start D end If the data sequence H D The length is less than parameter L min Then calculate H D Feature vector V D (Mean, median, minimum, maximum, standard deviation, skewness, kurtosis, first-difference mean, first-difference median), using the feature vector V D Based on the method of minimizing the characteristic geometric distance, the historical data drift segment list D is divided into segments. L Matching segments H sk And add it to the dataset to form dataset T. D {H D H sk};

[0082] b) Transfer the data set T D The data is divided into training and testing sets, with a learning rate of γ. This is used to develop an anomaly detection model for historical data. Perform training and fine-tuning to obtain an online anomaly detection model.

[0083] (2.4) Online detection of anomalies in online observation data: using an online anomaly detection model And the threshold σ, for the data sequence H D Perform anomaly detection, report the anomaly detection results, and send the data sequence H D Move out of window W and put D startSet to D end Repeat steps (2.2)-(2.4) to continuously perform online detection of data anomalies.

[0084] This invention also discloses a data anomaly detection system, which is an implementation system of the online ecological observation data anomaly detection method of this invention, and includes the following modules:

[0085] (1) Historical data drift detection and drift segment list management module: used for drift segment detection, drift segment storage, addition, and deletion, and provides drift segment matching service for online data anomaly monitoring;

[0086] The historical data drift detection and drift segment list management module stores historical ecological observation data drift detection and segment list management for multiple ecological observation indicators. The data drift detection model and drift segment list for each ecological observation indicator are independent of each other.

[0087] (2) Data anomaly online detection module: used to receive ecological observation data online, perform anomaly detection on the received ecological observation data, mark the abnormal data, and report the online anomaly detection results.

[0088] The online data anomaly detection module can receive data from multiple ecological observation indicators online and independently monitor for anomalies in each indicator. This module utilizes a drift detection method derived from historical data to detect drift in the online data. Before detecting anomalies in drift blocks, it extracts features from the data blocks to be detected and then interacts with the historical data drift segment list management module through these features to match data drift segments, thereby enhancing the training data and fine-tuning the historical anomaly detection model. After obtaining the online data anomaly detection results, the module can report the findings.

[0089] Therefore, the online ecological observation data anomaly detection method and system disclosed in this invention, by performing data drift segmentation on long-term historical ecological observation data and obtaining a list of data drift segments, when fine-tuning the model for online data partitions, the most similar data segments are matched from the historical data drift partition list, thereby increasing the amount of training data during model fine-tuning, improving the reliability of model fine-tuning training and the accuracy of model anomaly detection.

[0090] The online ecological observation data anomaly detection method of the present invention will be further explained below with reference to the accompanying drawings and specific application examples.

[0091] This embodiment provides a method for detecting anomalies in online ecological observation data by combining historical data with a learning model. For example... Figure 1As shown, the first part involves learning a data drift detection method from historical ecological observation data, outputting a historical data anomaly detection model and a list of historical data drift segments required for data drift detection. The specific steps are as follows:

[0092] S1. Historical ecological observation data were selected as training data. As a specific implementation case, long-term soil moisture content observation data from a certain station was selected, with an observation frequency of half an hour. The historical observation data constituted a time data sequence H{h1,h2,...h... n};

[0093] S2. Learning the historical observation data reconstruction model. A historical data anomaly detection model is built based on time series reconstruction methods.

[0094] S201 uses the commonly used LSTM-AE model, a representative deep learning model for time series reconstruction. Historical data is divided into training and testing sets, and the model is reconstructed by training on historical time series data.

[0095] S202, using a reconstruction model Reconstructing the data sequence H yields the reconstructed sequence. Original sequence H and reconstructed sequence The differences between them form a difference sequence R{r1, r2, ... r n}

[0096] S3. Obtaining the anomaly detection threshold and anomaly detection marker sequence;

[0097] S301. Calculate the standard deviation of the difference sequence R, and use three times the standard deviation as the anomaly detection threshold σ. Observed values ​​and reconstructed values ​​exceeding the threshold σ are considered anomalous data. Reconstruct the model. Together with the anomaly detection threshold σ, they constitute the anomaly detection model for historical time observation data;

[0098] S302. In the difference sequence, observations not exceeding the threshold σ are considered normal observations and are marked as 0; observations exceeding the threshold σ are considered abnormal observations and are marked as 1. Thus, the difference sequence R is transformed into an anomaly label sequence U{u1, u2, ... u... i ...u n}, where u i ∈{0,1};

[0099] S4. Historical observation data sequence data drift segmentation. In this embodiment, the data drift detection uses the representative DDM algorithm, which obtains the change pattern of the marker values ​​in the anomaly marker sequence U, and segments the time series H into a data segment sequence Hs{{H s1},{H s2},{H si},...,{H sm}, where {H si} represents a continuous time segment of the original sequence data {h j h j+1 ,...h j+k Set the drift start position D. start The value is 0, and the drift detection process is shown in S401 and S402;

[0100] S401. Using the anomaly marker sequence U, sequentially take the outlier value u at index i. i According to the DDM algorithm, calculate the variable p i and s i ;

[0101] Where, p i The calculation method is as follows:

[0102] i. Initialize m = 0;

[0103] ii. Calculate sequentially

[0104] iii.p i =m;

[0105] s i The calculation method is as follows:

[0106]

[0107] S402, if p i and s i If Formula 1 is satisfied, then data drift is considered to have occurred at index i.

[0108] Formula 1 is shown below:

[0109] p i +s i ≥p min +3*s min (1)

[0110] Where, p min s min p, representing the interval (j, i) i and s i The minimum value;

[0111] Remember D end For i-1, from the time series H according to the index interval (D) start D end Take data to form a segmented sequence Add to segmented sequence H s and set Dstart For i, repeat steps S401 and S402 above until all data in time series H has been detected. Figure 2 The image shows the visualization results of the soil moisture data segment in this example.

[0112] S5. Establishing a historical data drift segment list; for the data segment sequence Hs{{H obtained in S3... s1},{H s2},{H si},...,{H sm For each segment, calculate its feature vector, and use the feature vector as an index to build a list of historical data drift segments;

[0113] S501. Calculate the characteristics of each time series segment: Extract the segment {H} from the data segment sequence Hs. si}, calculate multiple features of the sequence, and establish segmentation {H}. si} eigenvector V si The feature vector uses nine features: mean, median, minimum, maximum, standard deviation, skewness, kurtosis, first-order difference mean, and first-order difference median.

[0114] S502. Combine the segmented data and their feature vectors to establish a historical data drift segment list D. L {V si , {H si}}, where si∈{s1, s2, ... s m}

[0115] The second part of the data anomaly detection method in this embodiment is based on the sequence reconstruction model learned from historical data, the anomaly detection threshold, and the data drift segment list. The online prediction method monitors the drift segments of online data, and obtains the online anomaly detection model for the current drift segment by fine-tuning based on the historical anomaly detection model. When the amount of training data for the current drift segment is insufficient, the amount of training data is enhanced by matching similar data from the historical data drift segment list.

[0116] The online detection method first obtains the anomaly detection threshold σ from historical data, and then uses a historical data anomaly detection model. Historical data drift segment list D L {V si , {H si}};Initial online observation data drift point D start The value is 0, and the algorithm parameters are Y (learning rate during online model fine-tuning) and L. min (Minimum amount of training data required for online model fine-tuning); Create an online data buffer window W, the specific process is as follows:

[0117] S6. Online data drift point detection, the process is as follows:

[0118] S601, Receive online data block {h t1 ,h t2 ,...,h tk The data is added to the end of window W in chronological order, using a historical data anomaly detection model. Perform anomaly calculations and generate an online anomaly identifier sequence U with the same width as window W. w Perform abnormal drift point detection according to steps S401 and S402;

[0119] S602. If no anomaly is detected, receive a new data block and repeat steps S401 and S402 to detect the drift point; if the anomaly is detected in window W's D... end If location data drift is detected, proceed to step S7;

[0120] S7. Construction of the online anomaly detection model fine-tuning dataset, the process is as follows:

[0121] S701, according to the drift index interval (D) obtained in S6. start D end Retrieve data sequence H from window W D Added to the model fine-tuning training dataset T D {H D In}, set D start D end ;

[0122] S702, If the data sequence H D The length is less than parameter L min Then calculate H D eigenvector V D (Mean, median, minimum, maximum, standard deviation, skewness, kurtosis, first-difference mean, first-difference median), using the feature vector V D Based on the method of minimizing the characteristic geometric distance, the historical data drift segment list D is divided into segments. L Matching segments H sk And add it to the dataset to form dataset T. D {H D H sk};

[0123] S8. Online observation data anomaly detection model learning, fine-tuning the model training data set T D The data is divided into training and testing sets, with a learning rate of γ. This is used to develop an anomaly detection model for historical data. Perform training and fine-tuning to obtain an online anomaly detection model.

[0124] S9. Online detection of anomalies in observation data and reporting of detection results, the process is as follows:

[0125] S901, Utilizing an online anomaly detection model And the threshold σ, for the data sequence H D Perform anomaly detection and report the anomaly detection results;

[0126] S902, Transfer the data sequence H D Move out of window W and put D start Set to D end Repeat steps S6-S9 to continuously perform online detection of data anomalies.

[0127] Therefore, this invention designs an online ecological observation data anomaly detection method based on current data drift and data anomaly detection technologies. It utilizes a model learned from historical data and augmented training samples to detect drift in online observation data, and obtains an online data anomaly detection model through fine-tuning training, thereby improving the accuracy of online data anomaly detection.

[0128] Since the online data anomaly detection model of this invention is based on deep learning of long-term historical data, it learns more knowledge from the historical data through deep learning methods, thereby making the detection results more accurate and stable.

[0129] This invention enhances the amount of training data during online model fine-tuning by matching data from the historical data drift block list, thereby better ensuring the convergence of online model fine-tuning and the reliability of the trained model.

[0130] The above embodiments are not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the technical solution of the present invention are also within the protection scope of the present invention.

Claims

1. A method for detecting anomalies in online ecological observation data, characterized in that: The testing method includes the following testing process: We learn data drift detection methods from historical ecological observation data, establish a historical data anomaly detection model required for data drift detection, and build a list of drift segments in historical ecological observation data. Online detection of anomalies in ecological observation data: Detecting drift segments in online data, based on historical data anomaly detection models, fine-tuning the online anomaly detection model for the current drift segment, and then using the online anomaly detection model to perform anomaly detection online; When there is insufficient training data for the current drift segments, similar data are matched from the list of drift segments in historical ecological observation data to enhance the amount of training data for fine-tuning. The specific process of drift detection is as follows: a) Based on the obtained historical anomaly marker sequence U, retrieve the index sequentially. i Location outlier u i According to the DDM algorithm, calculate the variables. p i and s i ; p i The calculation method is as follows: i. Initialize m = 0; ii. Calculate sequentially ; ⅲ. ； s i The calculation method is as follows: ； b) If p i and s i If Formula 1 is satisfied, then the index is considered to be... i The location has shifted due to data drift, denoted as D. end for i -1, from time series H according to index interval (D start D end The data is used to construct a segmented sequence H. sl { , ,… }, add to the segmented sequence Hs, and set D start for i Repeat steps a) and b) above until all data in time series H has been detected; (1) in, , They are intervals ( j , i )between p i and s i The minimum value; Online detection of anomalies in ecological observation data includes the following steps: (2.1) Obtain the anomaly detection threshold σ of historical data, and the historical data anomaly detection model. Historical data drift segment list Initial online observation data drift point D start The value is 0, the algorithm parameter is γ, and L min Wherein, parameter γ is the learning rate during online model fine-tuning, and parameter L... min This represents the minimum amount of training data required for fine-tuning the online model. (2.2) Detection of online data drift points: Create an online data buffer window W to receive online data blocks {h t1 ,h t2 ,…,h tk The data is added to the end of window W in chronological order, using a historical data anomaly detection model. Perform anomaly calculations and generate an online anomaly identifier sequence U with the same width as window W. w The abnormal drift point detection is performed using formula (1) in the DDM method; if the D in window W is... end If location data drift is detected, proceed to the next step (2.3). (2.3) Learning the online observation data anomaly detection model: using historical data anomaly detection models Fine-tuning is performed on the basic model to obtain the current online anomaly detection model; (2.4) Online detection of anomalies in online observation data: using an online anomaly detection model And the threshold σ, for the data sequence H D Perform anomaly detection, report the anomaly detection results, and send the data sequence H D Move out of window W and put D start Set to D end Repeat steps (2.2)-(2.4) to continuously perform online detection of data anomalies; The specific process of model fine-tuning is as follows: a) According to the index range (D) in (2.2) start D end Retrieve data sequence H from window W D Added to the model fine-tuning training dataset T D {H D In}, set D start D end If the data sequence H D The length is less than parameter L min Then calculate H D Feature vector V D Using the feature vector V D The historical data drift segmentation list is segmented according to the method of minimizing the characteristic geometric distance. Middle matching segmentation And add it to the dataset to form dataset T. D {H D H sk }; b) Transfer the data set T D The data is divided into training and testing sets, with a learning rate of γ. This is used to develop an anomaly detection model for historical data. Perform training and fine-tuning to obtain an online anomaly detection model. .

2. The online ecological observation data anomaly detection method according to claim 1, characterized in that: The historical data anomaly detection model is trained to obtain historical anomaly marker sequences. The specific process is as follows: a) Train the LSTM-AE model using time series data (H) from historical ecological observation data to obtain the historical data anomaly detection model needed for detection. Then through the model Historical ecological observation data are reconstructed to obtain the difference sequence R{r1,r2,…r} between the original sequence and the reconstructed sequence. n }; b) Calculate the standard deviation of the difference sequence R, and use 3 times the standard deviation as the anomaly detection threshold σ; c) In the difference sequence R, observations not exceeding the threshold σ are considered normal observations and are marked as 0, while observations exceeding the threshold σ are considered abnormal observations and are marked as 1. Thus, the difference sequence R is transformed into an anomaly label sequence U{u1,u...} 2, …u i …u n },in .

3. The online ecological observation data anomaly detection method according to claim 2, characterized in that: The process for detecting drift segmentation in historical ecological observation data is as follows: Using the acquired anomaly detection marker sequence U, and based on the DDM algorithm, data drift in historical ecological observation data sequences is detected. The original sequence data is segmented by data drift, transforming the time series H into a segmented data sequence Hs{{H s1 },{H s2 },{H si },…,{H sm }, where {H si } represents a continuous time segment of the original sequence data {h j ,h j+1 ,…h j+k Set the drift start position D. start If the value is 0, perform drift detection; For the data segment sequence Hs{{H s1 },{H s2 },{H si },…,{H sm For each segment, calculate its feature vector, and use the feature vector as an index to build a list of historical data drift segments.

4. The online ecological observation data anomaly detection method according to claim 3, characterized in that: Extract segments {Hsi} from the segmented data sequence Hs and construct the feature vector V of each segment {Hsi}. si By combining segmented data and their feature vectors, a list of historical data drift segments is established. ,in, .

5. The online ecological observation data anomaly detection method according to claim 4, characterized in that: The eigenvectors selected are mean, median, minimum, maximum, standard deviation, skewness, kurtosis, first-order difference mean, and first-order difference median.

6. A data anomaly detection system, characterized in that: This anomaly detection system is applied to the online ecological observation data anomaly detection method described in claim 1, and includes the following modules: Historical data drift detection and drift segment list management module: used for drift segment detection, drift segment storage, addition, and deletion, and provides drift segment matching service for online data anomaly monitoring; Online Data Anomaly Detection Module: This module is used to receive ecological observation data online, detect anomalies in the received ecological observation data, mark abnormal data, and report the online anomaly detection results.

Images