A method for detecting abnormal points in time-series marine ecological environment monitoring data

Abstract

This invention provides a method for anomaly detection in time-series marine ecological environment monitoring data, belonging to the field of marine data monitoring technology. The method includes the following steps: Step 1, Data Acquisition: Standardizing and normalizing the content data arriving at the current timestamp to obtain preprocessed data; Step 2, Anomaly Detection: Training a DSC model to obtain a trained DSC model; using partial training, generating an online similarity matrix for preprocessed data arriving at different timestamps, and detecting three types of anomalies in the current data to obtain the three types of anomalies in the current data; Step 3, Change Point Detection Mechanism: Using a change point detection mechanism to detect the data distribution of the current data; Step 4, Dynamic Model Update: Based on the detection results in Step 3, selecting or retraining the DSC model. This invention solves the problem that existing anomaly detection methods cannot adapt to high-dimensional dynamic time-series data.

Description

Technical Field

[0001] This invention relates to the field of marine data monitoring technology, and more particularly to a method for detecting anomalies in time-series marine ecological environment monitoring data. Background Technology

[0002] Real-time acquisition of marine ecological environment and water quality information via wireless sensor networks to form marine data streams is a crucial method for obtaining marine remote sensing data. Marine ecological environment monitoring primarily involves the detection and investigation of the ecological health of marine water bodies, sediments, marine organisms, marine atmosphere, meteorology, hydrology, and sea ice. It generally includes monitoring and investigating mariculture environments, fisheries environments, nearshore marine pollution, coastline ecology, and the ecological environment of bathing beaches. As can be seen from the content of marine ecological environment monitoring, the data covers many aspects. The data delivered by wireless sensor networks is collected from multiple sensors, resulting in high-dimensional data. Furthermore, marine ecological environment monitoring data is time-series, and the marine environment is dynamic; therefore, the distribution of data delivered by the wireless sensor network at the current timestamp may differ from previously received data.

[0003] However, water quality sensors and marine environmental monitoring stations are mostly located in natural environments such as seawater and remote islands, making sensor networks susceptible to environmental anomalies. Furthermore, network fluctuations during data transmission can lead to data loss or the introduction of anomalies. Even during the final data analysis, the large volume of data covering various remote measurements can easily introduce numerous anomalies. Considering these factors, anomaly detection in marine ecological environment monitoring is crucial for accurately assessing the ecological environment of a given marine area. Unlike traditional static data, marine time-series monitoring data has the following four characteristics: 1) Data typically arrives continuously in time series form, requiring real-time processing; 2) The data is multivariate, meaning each observation has multiple variables that change over time; 3) In addition to normal observations, various types of anomalous observations may coexist in the data, such as isolated anomalies and aggregate anomalies; 4) The data distribution, i.e., the correlations between data points, changes dynamically over time.

[0004] Currently, existing anomaly detection methods for processing marine environmental data time series data have the following limitations: 1) High-dimensional (or multivariate) data typically have more complex structures and potential correlations between attributes, requiring more robust models and computational costs. Most existing high-dimensional data solutions first reduce the dimensionality of the data and then perform anomaly detection in the compressed data space. However, dimensionality reduction inevitably results in the loss of some information, which may affect the accuracy of distance calculations used for clustering, thus reducing the effectiveness of anomaly detection; 2) Most existing anomaly detection methods, such as PCA, LOF, and OCSVM, are designed for batch processing and are not suitable for time series data; 3) Due to various triggering factors, anomalies can be classified into three categories: point anomalies, contextual anomalies, and ensemble anomalies. The type of anomaly will influence the choice of method used, as most existing methods are only applicable to a certain type of anomaly; 4) For marine environmental data time series data, the statistical distribution and autocorrelation of the data are expected to evolve over time; however, existing methods lack adaptability to dynamically changing data. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose an anomaly detection method in time-series marine ecological environment monitoring data. This method fully utilizes the advantages of clustering algorithms in anomaly detection, considering both the extraction of high-dimensional data subspace features and the dynamic characteristics of time-series data. It designs an anomaly detection method for high-dimensional time-series data, enabling the detection of anomalies in marine environmental monitoring data delivered via wireless sensor networks. This provides a sound data foundation for marine environmental application analysis and solves the problem that existing anomaly detection methods cannot adapt to high-dimensional time-series data.

[0006] The technical means employed in this invention are as follows:

[0007] A method for detecting outliers in time-series marine ecological environment monitoring data, comprising the following steps:

[0008] Step 1, Data Acquisition: Acquire the content data of n substances in seawater within the monitoring area of ​​the sensor network. The content data is sent through the network for data processing according to a unified time window. The content data arriving at the current timestamp is standardized and normalized to obtain preprocessed data. The preprocessed data includes normal data and abnormal data. The abnormal data includes context anomalies, set anomalies, and point anomalies.

[0009] Step 2, Anomaly Detection: Train the DSC model to obtain the trained DSC model; based on the trained DSC model, use part of the training to generate an online similarity matrix for preprocessed data arriving at different timestamps; based on the generated similarity matrix, detect three types of anomalies in the current data to obtain the three types of anomalies in the current data.

[0010] Step 3, Change Point Detection Mechanism: The change point detection mechanism is used to detect the data distribution of the current data and determine whether the distribution of the current data has changed significantly compared with the data at previous times;

[0011] Step 4, Dynamic Model Update: Based on the results of the probe in Step 3, select or retrain the DSC model.

[0012] Furthermore, step 1 specifically includes:

[0013] Step 1-1: At timestamp t, a batch of data is sent by the sensor network. The data size is the size of the time window set in the sensor network, and the dimension of the data is the types of substances in the seawater counted by each sensor. The data includes normal data and three types of abnormal data. Some of the data arriving at some timestamps does not contain contextual anomalies.

[0014] Step 1-2: Standardize and normalize the data from Step 1-1 to obtain preprocessed data.

[0015] Furthermore, step 2 specifically includes:

[0016] Step 2-1: Initialize the DSC model offline and determine the relevant parameters; the DSC model includes a stacked encoder, a self-expression layer, and a stacked decoder; train the initialized DSC model using normal data that does not contain anomalies to obtain the trained DSC model;

[0017] Step 2-2: Based on the trained DSC model, perform partial online training on the preprocessed data, only updating the self-expression layer in the DSC model, and retaining the encoding and decoding layers in the DSC model to obtain the feature mapping of high-dimensional data in the latent subspace;

[0018] Steps 2-3: Based on the feature mapping of high-dimensional data in the latent subspace, minimize the norm of the parameter C of the self-expression layer so that C has a block diagonal structure. On this basis, construct the similarity matrix of the data using matrix C, and obtain the similarity between the data based on the similarity matrix.

[0019] Steps 2-4: Based on the similarity between data points, the SLC clustering algorithm is used to effectively cluster the high-dimensional data; during the clustering process, a similarity threshold δ is used. s To determine the cluster to which each record in the data belongs, the micro-cluster threshold δ is used. c Determine whether the obtained clusters belong to large clusters, microclusters, or isolated point data;

[0020] Steps 2-5: Detect outlier data based on the clusters obtained in Step 2-4; Initial screening: Utilize the large cluster threshold δ d Contextual anomalies exist within large clusters during the filtering process;

[0021] Steps 2-6: Coarse screening: using the threshold δ r From the micro-clusters and outliers obtained in steps 2-4, aggregate anomalies and outliers are filtered out, and non-anomaly micro-clusters and outliers are classified into normal data.

[0022] Steps 2-7: After initial screening and coarse screening, three types of abnormal data are detected from the incoming high-dimensional data.

[0023] Furthermore, in steps 2-3, the similarity matrix is ​​as follows:

[0024]

[0025] The similarity matrix is ​​a symmetric matrix, s ij =s ji , and s ii = 1, 1≤i, j≤n; where n represents the number of records contained in a batch of data arriving at each timestamp, s ij Representing data x i and data x j The similarity.

[0026] Furthermore, step 3 specifically includes:

[0027] Step 3-1: Set the threshold δ cd The RMSE generated after processing the newly arrived data through the current DSC model is compared with the threshold δ. cd Compare the results to detect whether a change point has occurred;

[0028] Step 3-2: If a change point is detected, retrain the entire DSC model using new normal data, and update the threshold δ based on the newly trained model. r ;

[0029] Step 3-3: For data arriving at subsequent timestamps, if no change point data arrives, perform anomaly detection based on the model determined in Step 3-2.

[0030] The present invention also provides a storage medium comprising a stored program, wherein, when the program is executed, it performs any of the above-described methods for detecting anomalies in time-series marine ecological environment monitoring data.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described anomaly detection method in time-series marine ecological environment monitoring data through the computer program.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] This invention provides an anomaly detection method for time-series marine ecological environment monitoring data, solving the problem that existing anomaly detection methods cannot be fully applied to high-dimensional time-series data. While utilizing deep learning methods to extract features of high-dimensional data in the latent subspace, it proposes an SLC clustering method to effectively handle different types of anomalies. The use of a combined initial and fine screening strategy in anomaly monitoring significantly improves the accuracy of anomaly detection. Addressing the non-static nature of time-series data, especially the potential for significant changes in the distribution of data delivered via wireless sensor networks for certain timestamps, this invention introduces a change point detection mechanism and performs partial online training on the incoming data, while the entire model update operation is performed offline. This ensures that the proposed model can adapt well to dynamically changing time-series data without compromising real-time performance.

[0034] In summary, the technical solution of this invention, while considering the data sparsity problem in high-dimensional data space, also takes into account the dynamic changes of time-series data. Utilizing the characteristics of clustering algorithms, it proposes an efficient algorithm for detecting outliers in high-dimensional time-series data, completing the detection of outliers in marine ecological environment data and providing a solid data foundation for higher-level data analysis. Therefore, the technical solution of this invention solves the problem that existing data anomaly detection methods cannot adapt to high-dimensional time-series data.

[0035] Based on the above reasons, this invention can be widely applied in fields such as marine data monitoring. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the process of the present invention.

[0038] Figure 2 This is a schematic diagram of the anomaly detection structure of the present invention.

[0039] Figure 3 This is a schematic diagram of the change point detection mechanism and the overall structure of the present invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] like Figure 1 As shown, the present invention provides a method for detecting outliers in time-series marine ecological environment monitoring data, comprising the following steps:

[0043] Step 1: Data Acquisition: Given data S = {s1, s2, ..., Sn} on the content of n substances in seawater within the monitoring area of ​​the sensor network, the data is delivered through the network for data processing according to a unified time window. Data arriving at the current timestamp is standardized and normalized. This data includes normal data and three types of anomalies: context anomalies, set anomalies, and point anomalies.

[0044] Step 1-1: At timestamp t, a batch of data is sent by the sensor network. The data size is the size of the time window set in the sensor network, and the dimension of the data is the types of substances in the seawater counted by each sensor.

[0045] Steps 1-2: The incoming data contains normal data as well as three types of anomalies (context anomalies, set anomalies, and outlier anomalies). Some timestamps of incoming data may not contain context anomalies themselves.

[0046] Steps 1-3: Standardize and normalize the data to transform it into decimals between (0, 1). This is to make data processing more convenient and faster, and to transform dimensional expressions into dimensionless expressions, so that indicators of different units or magnitudes can be compared and weighted.

[0047] Step 2: Anomaly detection: Based on the offline trained model, generate an online similarity matrix using a small amount of training on data arriving at different timestamps. Based on the generated similarity matrix, detect three types of anomalies in the current data.

[0048] Step 2-1: Initialize the DSC model offline and determine relevant parameters. The DSC network consists of a stacked encoder, a self-expression layer, and a stacked decoder. Furthermore, the model is trained using normal data that does not contain anomalies, the distribution of which is roughly similar to the distribution of normal data in data arriving at one or more subsequent time stamps. Because anomalies are ultimately rare in the data collected by the sensor network, it is feasible to extract relevant normal data from previous data processing based on experience; this task is not difficult.

[0049] Step 2-2: Based on the DSC model trained offline, perform partial online training on the data processed in Step 1, such as... Figure 3 As shown, in the online part, only the self-expression layer in the DSC model is updated, while its encoding and decoding layers are retained;

[0050] Steps 2-3: (e.g.) Figure 1 As shown, a snapshot X of a given time series data X at timestamp t. t As shown in Table 1, X t The entire X is composed of high-dimensional data detected by multiple sensors. t Each record is input into the DSC network as a single batch. Mapped to a potential low-dimensional vector via a stacked encoder. Then, all the latent vectors are fully connected to learn the affinity between all observations in the subspace through self-expression;

[0051] Table 1 shows an example of a batch of data arriving at a certain timestamp using the anomaly detection method.

[0052]

[0053] Steps 2-4: Following steps 2-3, the feature mappings of high-dimensional data in the latent subspace can be obtained. Regarding the parameter C of the self-expression layer, existing papers have demonstrated that, under the assumption of subspace independence, minimizing certain norms of C ensures that C has a block diagonal structure. Based on this, the similarity matrix of the data (i.e., ...) can be constructed using matrix C. Figure 1 (Affinity mayrix in the text).

[0054]

[0055] The similarity matrix is ​​shown above. Here, the similarity matrix is ​​a symmetric matrix, s ij =s ji , and s ii = 1 (1≤i,j≤n) where n represents the number of records contained in a batch of data arriving at each timestamp, s ij Representing data x i and data x j The similarity. Based on the above theory, X obtained after model processing. t The corresponding similarity matrix is ​​shown in Table 2.

[0056] Table 2 shows the similarity of data for a batch of data arriving at a certain timestamp.

[0057]

[0058] Steps 2-5: Based on the similarity between the obtained data, propose an efficient SLC (sequential leader clustering) clustering algorithm. Figure 2 Part ② of the document describes effective clustering of high-dimensional data. For a snapshot X... t Assuming its clustering result is The corresponding cluster center is The main steps of SLC are as follows: ① First, for X t A record in If the similar matrix S contains Then x o Considered a cluster The cluster center. Calculations show that the data in Table 2... It is X t The first cluster ① Cluster center; ② For each remaining data beg With cluster center similarity s 1j ③ Given a threshold δ s =0.65, as shown in Table 2, s was found to be 0.65. 12 and s 13 All are greater than δs Therefore and Add to cluster Then, repeat steps similar to step ①, from X. t Identify new cluster centers from the remaining unmatched data. ④ Repeat steps ②-③ until all data in Xt has been processed. The time complexity of SLC is O(mn), where m is the number of clusters.

[0059] Steps 2-6: Based on the clusters obtained in Step 2-5, perform outlier detection. Initial screening (e.g.) Figure 2 Process I in the clustering operation: For the clusters obtained through clustering operations If the amount of data within the cluster (δ c (This is a threshold for measuring the size of a cluster), then the cluster is considered... These are microclusters, where clusters with a data size of 1 are considered outliers. Correspondingly, if but It is a large cluster. Furthermore, if the cluster... Data in With cluster center similarity s jk Less than the threshold δ d Then it is believed It is a large cluster The context is abnormal. After initial screening, such as... Figure 2 The result shown is that X was filtered out. t The context anomaly was identified, and set anomalies and outlier anomalies were preliminarily determined.

[0060] Steps 2-7: Fine sieving (e.g.) Figure 2 Part II of the text is for microclusters. This is the root mean square error of the data in the cluster after DSC-Net reconstruction. If Then it is considered that the cluster The data contains set anomalies or outliers (i.e., The data in the cluster is either a set anomaly or an outlier anomaly. Otherwise, the cluster... Data considered normal was excluded from the outlier cluster and reclassified as normal data. The threshold δ... r It is determined based on the RMSE between input and output after offline model training and data detection experience. After fine screening, such as Figure 2 As shown, this method precisely identifies which data points in the initial screening results are true set anomalies or outlier anomalies.

[0061] Steps 2-8: After the above steps, three types of anomalous data can be detected from the incoming high-dimensional data, namely context anomalies, set anomalies, and outlier anomalies.

[0062] Step 3: Change Point Detection Mechanism: Dynamic changes in data distribution are a key characteristic of time series data. If concept drift occurs, the previously trained DSC model may become unsuitable for subsequent data. Therefore, introducing change point detection into the model is essential. When data with a changed distribution arrives, the root mean square error reconstructed from historical data by the DSC-Net model will inevitably change significantly. Based on this, we set a corresponding drift threshold δ. cd Based on step 2, determine whether the distribution of the currently arriving data has changed significantly compared to the data from previous time points. For example... Figure 3 As shown, based on the detection results, if the current data is not detected as change point data, partial online training is performed, and outlier detection is performed according to step 2. If a change point is detected, the model is retrained offline, and the relevant parameters are updated.

[0063] Step 3-1: Change point detection: Set a threshold δ based on expert experience in data processing. cd The RMSE generated after processing the newly arrived data through the current DSC model is compared with the threshold δ. cd Compare the results to detect whether a change point has occurred;

[0064] Step 3-2: After step 3-1, if a change point is detected, the entire DSC model is retrained offline using new normal data, and the threshold δ is updated based on the newly trained model. r ;

[0065] Step 3-3: For the continuously arriving data, we only need to repeat the above operation, determine whether the ChangePoint has arrived, and perform the corresponding operation to achieve the requirement of accurately updating the DSC-Net model and adapting it to the dynamically changing data flow of anomaly detection.

[0066] The present invention also provides a storage medium comprising a stored program, wherein, when the program is executed, an anomaly detection method is performed in time-series marine ecological environment monitoring data.

[0067] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes an anomaly detection method in time-series marine ecological environment monitoring data through the computer program.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting outliers in time-series marine ecological environment monitoring data, characterized in that, Includes the following steps: Step 1, Data Acquisition: Acquire the content data of n substances in the seawater monitored within the sensor network monitoring area. The content data is transmitted through the network for data processing according to a unified time window. The content data arriving at the current timestamp is standardized and normalized to obtain preprocessed data; the preprocessed data includes normal data and abnormal data, and the abnormal data includes context abnormalities, set abnormalities and point abnormalities. Step 2, Anomaly Detection: Train the DSC model to obtain the trained DSC model; based on the trained DSC model, use part of the training to generate an online similarity matrix for preprocessed data arriving at different timestamps; based on the generated similarity matrix, detect three types of anomalies in the current data to obtain the three types of anomalies in the current data. The training process involves updating only the self-expression layer in the DSC model while retaining the encoding and decoding layers, thus obtaining the feature mapping of high-dimensional data in the latent subspace. The method for generating the online similarity matrix is ​​based on the feature mapping of high-dimensional data in the latent subspace, minimizing the norm of the parameter C of the self-expression layer so that C has a block diagonal structure, and then using matrix C to construct the similarity matrix of the data, and obtaining the similarity between the data based on the similarity matrix. Step 3, Change Point Detection Mechanism: The change point detection mechanism is used to detect the data distribution of the current data and determine whether the distribution of the current data has changed significantly compared with the data at previous times; Step 3-1: Set the threshold The RMSE and threshold generated after processing the newly arrived data through the current DSC model. A comparison is made to detect whether a change point has occurred; Step 3-2: If a change point is detected, retrain the entire DSC model using the new normal data, and update the threshold based on the newly trained model. ; Step 3-3: For data arriving at subsequent timestamps, if no change point data arrives, anomaly detection will be performed based on the model determined in Step 3-2. Step 4, Dynamic Model Update: Based on the results of the probe in Step 3, select or retrain the DSC model.

2. The method for detecting outliers in time-series marine ecological environment monitoring data according to claim 1, characterized in that, Step 1 specifically includes: Step 1-1: At timestamp t, a batch of data is sent by the sensor network. The data size is the size of the time window set in the sensor network, and the dimension of the data is the types of substances in the seawater counted by each sensor. The data includes normal data and three types of abnormal data. Some of the data arriving at some timestamps does not contain contextual anomalies. Step 1-2: Standardize and normalize the data from Step 1-1 to obtain preprocessed data.

3. The method for detecting outliers in time-series marine ecological environment monitoring data according to claim 1, characterized in that, Step 2 specifically includes: Step 2-1: Initialize the DSC model offline and determine the relevant parameters; the DSC model includes a stacked encoder, a self-expression layer, and a stacked decoder; train the initialized DSC model using normal data that does not contain anomalies to obtain the trained DSC model; Step 2-2: Based on the trained DSC model, perform partial online training on the preprocessed data, only updating the self-expression layer in the DSC model, and retaining the encoding and decoding layers in the DSC model to obtain the feature mapping of high-dimensional data in the latent subspace; Steps 2-3: Based on the feature mapping of high-dimensional data in the latent subspace, minimize the norm of the parameter C of the self-expression layer so that C has a block diagonal structure. On this basis, construct the similarity matrix of the data using matrix C, and obtain the similarity between the data based on the similarity matrix. Steps 2-4: Based on the similarity between data points, the SLC clustering algorithm is used to effectively cluster the high-dimensional data; during the clustering process, a similarity threshold is used. Determine the cluster to which each record in the data belongs, using a micro-cluster threshold. Determine whether the obtained clusters belong to large clusters, microclusters, or isolated point data; Steps 2-5: Detect outlier data based on the clusters obtained in Step 2-4; Initial screening: Utilize large cluster thresholds. Contextual anomalies exist within large clusters during the filtering process; Steps 2-6: Coarse screening: using threshold values From the micro-clusters and outliers obtained in steps 2-4, aggregate anomalies and outliers are filtered out, and non-anomaly micro-clusters and outliers are classified into normal data. Steps 2-7: After initial screening and coarse screening, three types of abnormal data are detected from the incoming high-dimensional data.

4. The method for detecting outliers in time-series marine ecological environment monitoring data according to claim 3, characterized in that, In steps 2-3, the similarity matrix is ​​shown below: The similarity matrix is ​​a symmetric matrix. , and in This indicates the number of records contained in a batch of data arriving at each timestamp. Representing data and data The similarity.

5. A storage medium, characterized in that, The storage medium includes a stored program, wherein when the program is executed, it performs the anomaly detection method in the time-series marine ecological environment monitoring data as described in any one of claims 1 to 4.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the anomaly detection method in the time-series marine ecological environment monitoring data as described in any one of claims 1 to 4 through the computer program.

Images