Temperature-salinity-depth combined ocean data anomaly detection method and system, electronic equipment and program product

Abstract

The invention discloses a temperature-salinity-depth combined ocean data anomaly detection method, which comprises the following steps: step 1, preprocessing a data set D, step 2, respectively separating salinity and temperature data in the data set D * to obtain a first salinity data set # imgabs0 # and a first temperature data set # imgabs1 #, and performing polynomial regression analysis according to the two data sets to obtain a first salinity data set # imgabs0 # and a first temperature data set # imgabs1 #; the method comprises the following steps: step 1, obtaining a data set D, training to obtain a polynomial regression model Rs and a polynomial regression model Rt, respectively separating salinity and temperature data in the data set D to obtain a second salinity data set D-s and a second temperature data set D-t, step 2, respectively inputting the data sets DsP and DtP into an auto-encoder model AEM to correspondingly obtain reconstructed data sets DsR and DtR, and step 3, respectively inputting the reconstructed data sets DsR and DtR into an auto-encoder model AEM to correspondingly obtain reconstructed data sets DsR and DtR. And respectively calculating errors Es and Et between the data set DsR and the data set DtR and the data set D, and determining that the data is noise data according to error values as temperature abnormal data and salinity abnormal data.

Description

Technical Field

[0001] The present invention belongs to the field of ocean measurement technology, and in particular relates to a temperature-salinity-depth combined ocean data anomaly detection method, system, electronic equipment and program product. Background Art

[0002] Early machine learning methods played a fundamental role in anomaly detection. The local outlier factor (LOF) proposed by Breunig et al. [1] is one of the classic methods in the field of anomaly detection. This method detects outliers by evaluating the local density difference between a data point and its neighborhood, and is particularly suitable for processing data sets with complex distributions. Similarly, the Isolation Forest proposed by Liu et al. [2] is also a commonly used unsupervised anomaly detection method. It isolates outliers through recursive partitioning, has high computational efficiency, and is particularly suitable for processing high-dimensional and large-scale data. Another common algorithm is the one-class support vector machine (One-Class SVM) [3]. This method learns the hyperplane of normal data and identifies outliers that deviate from the boundary. With the rapid development of deep learning technology, autoencoders have become one of the important tools for anomaly detection. Kramer [4] first proposed the concept of autoencoders. This method detects anomalies that are difficult to reconstruct by compressing and reconstructing data. Kingma and Welling [5] further developed the variational autoencoder (VAE) by introducing probabilistic modeling into the autoencoder, making it more robust when processing complex distributions. In addition, recurrent neural networks (RNNs) and their variants, long short-term memory networks (LSTMs) [6], have performed well in anomaly detection tasks that capture temporal dependencies and can effectively handle anomalies in sequence data. In recent years, models based on attention mechanisms, such as Transformer [7], have performed well in the field of anomaly detection. Transformer captures long-range dependencies in data through a self-attention mechanism and is particularly suitable for processing high-dimensional data with complex temporal relationships. In addition, generative adversarial networks (GANs) [8] are also important tools for anomaly detection in recent years. GAN generates new data points through a game process between the generator and the discriminator, identifies anomalies that are difficult to generate, has extremely high detection sensitivity, and is particularly suitable for scenarios with insufficient data annotation.

[0003] In the ocean field, Kang et al. [9] first proposed a method combining autoencoder (AE) and synthetic minority oversampling technique (SMOTE) to deal with the imbalance problem of ocean temperature data. Autoencoders perform well in unsupervised learning and can reconstruct data by learning the distribution characteristics of normal data, while SMOTE enhances the diversity of data by generating additional minority class samples, thereby improving the model's ability to detect anomalies. This method is particularly effective in detecting anomalies in ocean temperature data and can improve the accuracy of detection under unbalanced data conditions. In addition to traditional autoencoder models, variational autoencoders (VAEs) have gradually emerged in ocean data processing in recent years. Jiang et al.

[10] proposed a method based on combining VAE with K-means clustering, which further enhances the adaptability to complex multivariate data. VAE introduces a probabilistic model to make the data reconstruction process uncertain, thereby better capturing the potential patterns of complex data. When applied to Argo float data, this method can effectively identify anomalies, especially when faced with complex distributions and multidimensional variables, and shows good generalization ability. K-means clustering can further divide the data to help better distinguish between abnormal and normal behaviors. Liu et al.

[11] constructed a deep learning multilayer perceptron (MLP) and convolutional neural network (DNN) classification model to quickly identify the quality of sea temperature data in the western Pacific Ocean, and the recall rate of erroneous and problematic data reached 87% to 92%. Sui et al.

[12] developed a new anomaly detection method for processing Argo data by combining the Transformer model with the fast Fourier transform (FFT). The Transformer can effectively capture long-range dependencies through its self-attention mechanism and is suitable for time series analysis of multivariate data in ocean observations. FFT enhances the model's predictive ability by capturing the periodic changes of the data. In experiments, this method significantly improves the accuracy of anomaly detection in complex ocean data, especially in multidimensional data. At the same time, Wang et al.

[13] developed a method based on the Wavelet Neural Network (WNN) to focus on anomaly detection in ocean time series data. By combining the advantages of wavelet analysis and neural networks, WNN can decompose and analyze ocean observation data at multiple scales, especially simulating non-abnormal behavior without labels. This method can effectively detect anomalies caused by equipment failure or natural factors by constructing classification thresholds generated by historical residual distribution. Compared with traditional neural networks, WNN shows stronger robustness and sensitivity to noisy data, and is particularly suitable for real-time data monitoring in marine engineering.In addition, in the application of clustering methods, Jiang et al.

[14] proposed an improved DBSCAN algorithm that was optimized for complex anomalies in Argo data. By adjusting the parameters of DBSCAN, this method can better adapt to data changes in the ocean environment, especially when processing large-scale and complex buoy data, and has demonstrated strong performance. DBSCAN, a density-based clustering method, can effectively distinguish between normal data clusters and abnormal data clusters, thereby identifying outliers in the data. In response to climate phenomena in the ocean, Houghton et al.

[15] proposed a method of using K-means clustering to process Argo temperature data to detect El. Phenomenon. This method effectively identifies abnormal patterns by clustering a large amount of temperature data, demonstrating the potential of unsupervised clustering algorithms in large-scale climate data. This method provides a new idea for climate prediction based on ocean observation data. Since Argo observation profiles are also a kind of sequence data, Jiang et al.

[16] proposed a deep learning model based on long short-term memory network (LSTM) for temperature and salinity anomaly detection in Argo data. LSTM can capture long-term dependencies in time series data through its memory gate mechanism, which is particularly suitable for processing dynamic ocean data. The model shows good adaptability in large-scale ocean observation data and can effectively detect anomalies in time series. Zhang et al.

[17] further combined VAE and K-means clustering to propose an anomaly detection method for Argo temperature data quality control. By reducing the dimension of data through VAE and using K-means clustering analysis, anomalies in the data can be effectively identified, improving the quality control level of temperature data. Zhang et al.

[18] proposed a twin convolutional LSTM network (VAS-CLSTM) based on vertical attention for error detection of Argo data. The method solves the data imbalance problem through deep clustering and oversampling, and uses a twin network to learn the spatiotemporal characteristics of temperature and salinity. The vertical attention mechanism enhances the feature extraction of different depth layers, and experiments show that the method performs well in detecting anomalies in complex ocean data.

[0004] Although the existing ocean data anomaly detection technology has made significant progress, it still faces some problems and shortcomings in practical applications. First, the current technology is not capable of processing complex relationships in multivariate data. Traditional methods such as local outlier factor (LOF), isolation forest (IForest) and one-class support vector machine (One-Class SVM) can identify some abnormal data, but often cannot accurately capture the complex relationships between variables in multidimensional data detection. Even deep learning models such as autoencoders and long short-term memory networks (LSTM) perform well on multidimensional and time series data, but still have great limitations for highly dynamic spatiotemporal relationships in the marine environment. Secondly, noise and data imbalance problems are relatively common in marine observation data, which brings challenges to existing detection methods. Traditional autoencoders are prone to misjudgment in noisy data, and oversampling technologies such as SMOTE can partially alleviate the problem of data imbalance, but at the same time increase the complexity of data processing and the computational burden, especially when processing large-scale data. It is difficult to balance efficiency and accuracy.

[0005] In addition, the generalization and adaptability of existing models are limited, and it is difficult to cope with regional and seasonal changes in the marine environment. Due to the differences in the marine environment, the stability of the model in different regions and conditions may not be guaranteed, making it difficult for the model to maintain consistent detection performance in new regional applications. Especially in different sea areas and observation conditions, machine learning-based detection methods usually need to be retrained, which increases the complexity of practical applications. At the same time, many deep learning models (such as generative adversarial networks GAN and Transformer) perform well in processing complex data, but have high computational complexity and are heavily dependent on hardware resources, which is not conducive to widespread promotion and application in large-scale data and real-time monitoring scenarios.

[0006] Finally, existing technologies also have problems in relying on labeled data and expert experience. Some methods (such as LSTM and convolutional neural networks (CNN)) require a large amount of labeled data or rely on expert annotations, especially in delayed mode quality control (DMQC). This dependence will significantly reduce the detection efficiency and make it difficult to meet the real-time processing requirements of ocean data. In summary, although existing technologies have made great progress in ocean data anomaly detection, there is still much room for improvement in complex data relationship processing, noise and unbalanced data handling, model generalization improvement, and computational efficiency and real-time performance. Future research needs to develop more intelligent, robust and efficient detection methods to meet the challenges brought by the diversity and complexity of ocean data. Summary of the invention

[0007] One of the embodiments of the present disclosure is a method for detecting anomalies in temperature, salinity and depth combined ocean data, which detects abnormal data in the temperature, salinity and depth combined ocean data through a trained detection model.

[0008] Assume that the temperature, salinity and depth joint ocean data set input by the detection model is D = (d, t, s, LON, LAT) × m, where depth d, temperature t, salinity s, longitude LON and latitude LAT are m-dimensional data with five features.

[0009] The method comprises the following steps:

[0010] Step 1: preprocess the data set D to extract the data set D with normal or approximately normal values. * ;

[0011] Step 2: Dataset D * The salinity and temperature data in the image are separated to obtain the first salinity data set. With the first temperature data set Based on these two data sets, polynomial regression analysis is performed to train the polynomial regression model R s with polynomial regression model in R t ,

[0012] Separate the salinity and temperature data in data set D to obtain the second salinity data set D -s With the second temperature data set D -t , input other factors into the multinomial regression model R s with polynomial regression model in R t In the dataset D, the salinity data and other elements are predicted separately. sP And temperature data and other elements form data set D tP ,

[0013] Step 3: Dataset D sP With D tP Input them into the autoencoder model AEM respectively, and the corresponding reconstructed dataset D is obtained sR With D tR ,

[0014] Step 4: Dataset D sR With D tR Calculate the error E with the data set D respectively s With E t , the data that are judged as noise data according to the error value are temperature anomaly data and salinity anomaly data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0016] Figure 1A schematic flow chart of a TSD-AD method according to one embodiment of the present invention.

[0017] Figure 2 Temperature, depth and salt data distribution diagram according to one embodiment of the present invention.

[0018] Figure 3 A schematic diagram of the results of TSD-AD anomaly detection according to one embodiment of the present invention.

[0019] Figure 4 A schematic diagram of the results of TSD-AD anomaly detection according to one embodiment of the present invention.

[0020] Figure 5 A schematic diagram of comparative analysis of F1 Score and training time according to one embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present disclosure proposes a method for anomaly detection in oceanographic data combining temperature, salinity, and depth (TSD-AD). In the present disclosure, a clustering algorithm is introduced to cooperate with the autoencoder to further improve the detection accuracy. By grouping the data, the clustering algorithm can identify outliers in the data, thereby further distinguishing the boundaries between normal data and abnormal data. The multivariate anomaly detection method based on autoencoders and clustering not only improves the accuracy of detection, but also reduces the dependence on a large amount of labeled data, which helps to cope with large-scale and complex ocean observation data scenarios.

[0022] By combining the multivariate anomaly detection method of autoencoders and clustering, it is possible to efficiently process massive multidimensional data under unsupervised conditions and significantly reduce human intervention. This has important application value for real-time monitoring of ocean data, long-term trend analysis, and marine disaster warning, and provides an intelligent and efficient solution for current ocean data quality control, which can cope with complex changes in the marine environment.

[0023] According to one or more embodiments, a method for detecting anomalies in temperature, salinity and depth combined ocean data. The method first includes the selection of a data set. For the selection of multivariate ocean observation data, anomaly detection of ocean temperature and salinity is focused on. Among them, the experimental data comes from the Argo data management website (https: / / argo.ucsd.edu / data / data-from-gdacs / ). These experimental data include depth, temperature, salinity, longitude and latitude, as well as real-time automatic quality control tags (RTQC) and delayed quality control tags (DMQC) for temperature and salinity.

[0024] The disclosed embodiment introduces the constraint of spatial correlation, further considers the correlation between various ocean elements such as temperature, salinity, density, etc., and proposes a temperature-salinity-depth combined ocean data anomaly detection method to realize anomaly detection of multivariate ocean data. It has wider global applicability while ensuring accurate data quality control.

[0025] Assume that the input data set is D = (d, t, s, LON, LAT) × m, which is m-dimensional data containing five features: depth d, temperature t, salinity s, longitude LON and latitude LAT. The TSD-AD method framework is as follows: Figure 1 As shown, it mainly includes four steps:

[0026] Step 1: Data preprocessing: Use DBSCAN clustering to extract a set of approximately normal data D from the dataset D * .

[0027] Step 2: Temperature-salinity interaction polynomial regression prediction: Use the polynomial regression model to mine the correlation between factors, which consists of the following two modules:

[0028] (1) Data D * The salinity and temperature elements in the dataset are separated to obtain the data set and As target data, train a polynomial regression model R s With R t .

[0029] (2) Separate the salinity and temperature elements in the original data set D to obtain data set D -s With D -t , input other factors into the multinomial regression model R s With R t In the dataset D, the salinity data and other elements are predicted separately. sP And temperature data and other elements form data set D tP .

[0030] Step 3: Autoencoder model: Dataset D sP The anomalies of the medium salinity data are strictly controlled by the temperature anomalies. tP The abnormal temperature data is strictly controlled by the abnormal salinity data, and both show a positive correlation. sP With D tP The reconstructed dataset D is obtained by inputting the autoencoder model AEM designed by the method. sR With D tR .

[0031] Step 4: Label the quality of abnormal data:sR With D tR Calculate the error E with the original data set D respectively s With E t , DBSCAN clustering technology is used to detect noise points for errors. Any detected noise points are temperature anomaly data or salinity anomaly data. Finally, the detected abnormal data sets are labeled for quality.

[0032] Among them, data preprocessing in step one is the first step of data analysis, which aims to optimize data quality and lay the foundation for subsequent analysis and modeling.

[0033] First, StandardScaler is used to standardize the data to obtain the data set D as input to eliminate the dimensional influence between different features, so that its mean is 0 and the standard deviation is 1.

[0034] If the outliers or abnormal clusters in the data set deviate significantly from the normal values, it will have a negative impact on the accuracy and generalization ability of subsequent model training. Therefore, the DBSCAN algorithm is used to perform cluster analysis on the data, divide the data into core points and noise points, and select data that strictly belongs to the normal value for subsequent model training.

[0035] DBSCAN is a density-based clustering algorithm that identifies clusters by observing the density around data points. It has the characteristics of discovering clusters of arbitrary shapes and effectively processing noise points. The algorithm first determines the core object, and determines whether it is a core object based on the number of data points within a certain radius of a data point. Then, clustering is established through density reachability, and finally different clusters are formed, while marking noise points.

[0036] The core idea of ​​DBI is to calculate the similarity between each cluster and its most similar cluster, and then measure the quality of the entire clustering result by taking the average of all similarities. If the similarity between clusters is higher, the DBI value is larger, which means the distance between clusters is smaller and the clustering result is worse; conversely, the smaller the DBI value is, the better the clustering result is. Define cluster C i With cluster C j (C i ≠C j ) ij for:

[0037]

[0038] in represents the i-th cluster C i The diameter, L ij For two clusters C i and C j The distance between centers.

[0039] Furthermore, the calculation formula of DBI is:

[0040]

[0041] Where K represents the total number of clusters, and the value range of DBI is between [0, +∞). Based on the DBSCAN clustering process under the optimal parameters selected by DBI, a set of clusters can be obtained, among which the largest cluster C max That is, it is an approximately normal data set, and the unstable values ​​will be scattered in the remaining small clusters. Thus, an approximately normal data subset D is extracted. * =C max ∈D, which is used for model training and evaluation to improve the robustness and prediction performance of the model.

[0042] For step 2, temperature-salinity interaction polynomial regression prediction. Among them,

[0043] The purpose of TSD-AD is to predict salinity by combining temperature data with spatial variables, and temperature by combining salinity data with spatial variables, so as to deeply explore the association between temperature and salinity and spatial information. We introduce a polynomial regression model to explore the association between various factors. The polynomial regression model is a key statistical method used to model the nonlinear relationship between independent variables and dependent variables. By increasing the power terms of the independent variables, polynomial regression can fit the data more accurately and thus capture more complex patterns. Compared with simple linear regression, polynomial regression provides greater flexibility by introducing higher-order terms of the independent variables, and can better capture complex data patterns. The basic form of the polynomial regression model is as follows:

[0044] y=β0+β1x+β2x 2 +β3x 3 +…+β n x n +ε (3)

[0045] Among them, y is the dependent variable, x is the independent variable, β0, β1, β2, …, β n is the parameter to be estimated, and ε is the error term, which is assumed to have a mean of zero and a constant variance. 2 、x 3 etc.), polynomial regression can model nonlinear relationships that linear regression cannot capture.

[0046] The parameter estimation of the polynomial regression model usually adopts the least squares method (Ordinary Least Squares, OLS). The goal of OLS is to minimize the residual sum of squares between the predicted value and the actual value, that is:

[0047]

[0048] The core idea of ​​the least squares method is to find the parameter value that minimizes the sum of squares of the residuals, thereby making the model have the strongest predictive ability.

[0049] like Figure 2 As shown in Figure 2, the relationship between the elements in dataset D is nonlinear. Therefore, it is particularly important to use a polynomial regression model to mine the relationship between the elements. * Divided into two subsets and in That is, m-dimensional data containing four features: depth d, salinity s, longitude LON, and latitude LAT. That is, m-dimensional data containing four features: depth d, salinity s, longitude LON, and latitude LAT. as the feature set, temperature as the target, and As the feature set and salinity as the target, two 5th-order polynomial regression models R were trained respectively. t , R s .

[0050] After the model training is completed, the subset D without temperature data in the dataset D is -t and the subset D without salinity data -s As features, they are respectively used to input the trained polynomial regression model R t , R s , used to predict D -t The temperature and D -s The salinity of tP and D sP . Since the model is trained using approximately normal data, if the input data is a normal value, the predicted target is also a normal value; if the input data is abnormal, the predicted target must be abnormal. In the present disclosure, it is considered that only the temperature and salinity data in the data set used have anomalies, so it can be deduced that the abnormal salinity state is the same as the predicted abnormal temperature state, and the abnormal temperature state is the same as the predicted abnormal salinity state.

[0051] For step three, the autoencoder model, where

[0052] An autoencoder is a neural network used for unsupervised learning. Its main function is to learn the effective compression representation of data, i.e. encoding, and then reconstruct the approximate representation of the data through these encodings, i.e. decoding. In the TSD-AD method, the autoencoder is used to identify abnormal data points of temperature and salinity, and the training is completed by analyzing the difference between the input data and the reconstructed data (i.e. reconstruction error).

[0053] The specific implementation of the encoder is to convert the high-dimensional input D tP and D sP Encoded into low-dimensional latent variables Z1 and Z2, we can get a high-dimensional input D TP and D SP The most informative data feature is:

[0054] Z1=σ(W1·D tP +b1) (5)

[0055] Z2=σ(W2·D sP +b2) (6)

[0056] Among them, W1 and W2 are the weight matrices of the encoding layer, b1 and b2 are bias vectors, and σ is the nonlinear activation function ReLU.

[0057] The decoder is used to reconstruct the latent variables Z1 and Z2 of the hidden layer into Y1 and Y2 with the same dimensions as the input, that is, AE(D tR )、AE(D sR ), the decoding process can be described as:

[0058]

[0059] Among them, W3 and W4 are the weight matrices of the decoding layer, b3 and b4 are bias vectors, It is the linear activation function Linear.

[0060] The role of the loss function is to measure the degree of information loss caused by data dimensionality reduction, ensuring that AE can effectively extract features and reconstruct the input more accurately. The ideal state of AE is that the output of the decoder can perfectly reconstruct the input of the encoder. The loss function used by the autoencoder in this method is MSE, and its formula is:

[0061]

[0062] Where n is the number of samples, X i and X′ i are the i-th sample of the original data and the reconstructed data respectively.

[0063] The autoencoder model AEM of the TSD-AD method adopts a typical three-layer neural network structure: input layer, hidden layer (encoding layer) and output layer (decoding layer). The input layer receives a five-dimensional data vector corresponding to the preprocessed features. The encoding layer compresses the input data into a low-dimensional encoding representation. In this layer, the ReLU activation function is used to increase the nonlinearity of the network, and L1 regularization is introduced to extract and compress important features in the data. By adding a regularization term to the loss function, the model is pushed to learn a more sparse feature representation to promote the generation of sparse coding, which helps to improve the generalization ability of the model and reduce the risk of overfitting. The decoding layer attempts to reconstruct the output of the encoding layer. The output size of the decoding layer is the same as that of the input layer, and the Linear activation function is used to reconstruct the original input data as accurately as possible. The autoencoder is compiled using the adam optimizer and the MSE loss function, which can effectively minimize the mean square error between the input and the reconstructed output, that is, the reconstruction error.

[0064] Autoencoder model AEM uses dataset D * During training, the model adjusts the network weights and biases by minimizing the mean squared error between the input data and the reconstructed data. By setting appropriate training cycles and batch sizes, the model can learn the essential features of the data without overfitting.

[0065] For step 4, abnormal data quality labeling, where:

[0066] After the autoencoder model AEM is trained, TSD-AD is used to train the dataset D tP With D sP Reconstruction to get D tR With D sR , respectively, and calculate the reconstruction error E of the temperature and salinity data points with the original data set D t With E s :

[0067] E t =|D(t)-D tR (t)|∈[0,1] (10)

[0068] E s =|D(s)-D sR (s)|∈[0,1] (11)

[0069] The obtained error (the error between input and output) is used as the basis for anomaly detection. t 、E s The DBSCAN clustering is performed again with depth, and the selection of parameters is also based on the DBI of the clustering as the criterion to detect noise and obtain abnormal data points. tR The error E tThe detected anomaly is the salinity anomaly value, D sR The error E s The detected anomaly is the temperature anomaly value.

[0070] In order to ensure data quality, strict monitoring and anomaly detection of sensor outputs are required. Therefore, TSD-AD will perform statistical analysis on the outliers in the temperature and salinity data and attribute these outliers to the corresponding sensors. If the proportion of abnormal data points of a sensor exceeds a certain proportion ∝, the sensor is considered to be faulty as a whole and is marked as an abnormal sensor. In this case, all data collected by the sensor need to be further verified and processed. On the contrary, if the proportion of abnormal data points of a sensor does not exceed 30%, only the detected abnormal data points are marked as abnormal without affecting the overall status of the sensor. In this way, the faulty sensor can be effectively identified and isolated, ensuring data quality and avoiding the misjudgment of the sensor as a whole as a fault due to a small number of abnormal data points.

[0071] Therefore, the beneficial effect of the embodiment of the present disclosure is to propose a multivariate ocean data anomaly detection method based on the combination of autoencoder and clustering. The method uses an unsupervised autoencoder combined with a clustering method to associate multi-factor Argo data, reducing the reliance on manually labeled data.

[0072] According to one or more embodiments, a method for constructing a temperature-salinity-depth combined ocean data anomaly detection model includes:

[0073] S101, data input and preprocessing, data set D is processed by DBSCAN algorithm to extract the approximate normal data set D * .

[0074] S102, polynomial regression model training, polynomial regression model training is performed on the separated salinity and temperature data to obtain R s With R t .

[0075] S103, using an autoencoder model AEM to reconstruct predicted data to identify anomalies.

[0076] S104, by calculating the error between the reconstructed data and the original data and using DBSCAN to detect noise points, abnormal data is identified. Finally, the abnormal data is labeled for further analysis and processing.

[0077] Therefore, the disclosed method improves the accuracy of anomaly detection by combining clustering and autoencoders, reduces the dependence on large amounts of labeled data, and is suitable for large-scale and complex ocean observation data scenarios.

[0078] Further, the method framework of TSD-AD disclosed in the present invention is as follows Figure 1 As shown, it mainly includes four steps:

[0079] Step 1 (data preprocessing): Use DBSCAN clustering to extract a set of approximately normal data D from data set D * .

[0080] Step 2 (temperature-salinity interaction polynomial regression prediction): Use the polynomial regression model to mine the association between factors, which consists of the following two modules:

[0081] (a) Data D * The salinity and temperature elements in the dataset are separated to obtain the data set and As target data, train a polynomial regression model R s With R t .

[0082] (b) Separate the salinity and temperature elements in the original data set D to obtain data set D -s With D -t , input other factors into the multinomial regression model R s With R t In the dataset D, the salinity data and other elements are predicted separately. sP And temperature data and other elements form data set D tP .

[0083] Step 3 (Autoencoder Model): Dataset D sP The anomalies of the medium salinity data are strictly controlled by the temperature anomalies. tP The abnormal temperature data is strictly controlled by the abnormal salinity data, and both show a positive correlation. sP With D tP are respectively input into the autoencoder model AEM designed by the disclosed method to obtain the reconstructed data set D sR With D tR .

[0084] Step 4 (labeling of abnormal data quality): sR With D tR Calculate the error E with the original data set D respectively s With E t , DBSCAN clustering technology is used to detect noise points for errors. Any detected noise points are temperature anomaly data or salinity anomaly data. Finally, the detected abnormal data sets are labeled for quality.

[0085] Through these four steps, the TSD-AD method disclosed in the present invention can effectively identify and process anomalies in ocean data and improve the accuracy and reliability of the data.

[0086] An experimental example is given below to verify the temperature-salinity-depth combined ocean data anomaly detection method disclosed in the present invention.

[0087] Observational data sets from multiple ocean areas were selected for analysis and verification. The experimental results show three representative sea areas: Area 1, with a time span from January to March 2021; Area 2, with a time span from June 2020 to November 2020; and Area 3, with a time span from April 2022 to November 2022. In the quality control label, quality control symbol 1 represents the real-time automatic quality control label, and quality control symbol 2 represents the delayed quality control label. A value of 1 indicates that the data is normal, and a value of 0 indicates that the data is abnormal. Since the delayed quality control label is manually proofread, we use the delayed quality control label as the benchmark when evaluating the experimental results; at the same time, the real-time automatic quality control results represent the existing business operation results, which are compared with the real-time quality control results to evaluate the performance improvement effect. For temperature outlier detection, compared with the real-time automatic quality labeling (RTQC) results, the precision of this method in the three regions is improved by about 41.5%, the recall is improved by about 31.0%, and the F1 Score is improved by about 37.2%. For salinity outlier detection, compared with the real-time automatic quality labeling results, the precision of this method in the three regions is improved by about 1.0%, the recall is improved by about 1.2%, and the F1 Score is improved by about 1.1%, which effectively improves the effect of data quality inspection.

[0088] In the evaluation process, the present disclosure uses Precision, Recall and F1 Score as evaluation indicators, and the calculation formulas are shown in (12), (13) and (14) respectively. Among them, TP means that the actual positive example is also a positive example, FP means that the actual negative example is a positive example, and FN means that the actual positive example is a negative example.

[0089]

[0090] Precision reflects the accuracy of positive sample detection. The lower its value, the higher the false positive rate. Recall reflects the ratio of detected positive samples to total positive samples. The lower its value, the higher the missed detection rate. The F1 score reflects the harmonic average of the two indicators, Precision and Recall.

[0091] In the subsequent evaluation, the evaluation indexes between the output results of various methods and the benchmark were calculated based on manual identification. At the same time, in order to avoid the evaluation bias caused by a large amount of normal data relative to a very small number of abnormal data, the evaluation indexes of the detection results were calculated for 0-abnormal and 1-normal data respectively.

[0092] The following is an analysis and discussion of the experimental results.

[0093] First, the test performance is compared. In order to verify the effectiveness of the proposed method TSD-AD, it is compared with a variety of methods, including Isolation Forest (IForest), Local Outlier Factor (LOF), Convolutional Neural Network (CNN), Autoencoder (AE), VAE_Kmeans and VAS-CLSTM. The details are as follows:

[0094] 1) Isolation Forest IForest: By constructing a random partitioning tree, the data points are gradually separated and the outliers are determined based on the unique path length of the outliers in the tree.

[0095] 2) Local Outlier Factor (LOF): Outliers are identified by calculating the ratio of the local density of each data point to the average local density of its neighboring points.

[0096] 3) Convolutional Neural Network (CNN): Liu et al.

[11] found that CNN was more outstanding in performance when comparing the performance of MLP and CNN. In view of this, this paper uses CNN for comparison. The CNN model uses Softmax as the activation function, uses one-hot encoding to convert the classification target into a vector, uses cross entropy as the classification loss function, and adds weights to the losses corresponding to different classes to balance this data bias.

[0097] 4) Autoencoder AE: detects anomalies by compressing and reconstructing data. It mainly judges anomalies by calculating the difference between input data and reconstructed data. When the difference is large, the data is considered to be an anomaly.

[0098] 5) VAE_Kmeans: The anomaly detection method that combines VAE and K-means uses VAE to reduce dimension and enhance data, and then uses K-means for clustering detection, which effectively improves the accuracy of detection.

[0099] 6) VAS-CLSTM: This method solves the data imbalance problem through deep clustering and oversampling techniques, and combines spatiotemporal features and vertical attention mechanism to improve the accuracy of anomaly detection in Argo data.

[0100] 7) TSD-AD: Combining density clustering, cross polynomial regression and autoencoder, first filter normal data through density clustering, then use regression modeling to calculate residuals, and finally use autoencoder to detect residual abnormalities and dynamically adjust parameters to improve accuracy.

[0101] Table 1 Comparative experimental results

[0102]

[0103]

[0104] By comparing the experimental results, it can be found that the proposed method TSD-AD performs well in the task of temperature and salinity data anomaly detection in multiple regions. Its performance is more comprehensive and balanced in all aspects than other existing methods, thus highlighting its advantages in practical application scenarios. For temperature anomaly detection, compared with the real-time automatic quality labeling results, the precision of the proposed method in the three regions is improved by an average of about 41.5%, the recall is improved by about 31.0%, and the F1 Score is improved by about 37.2%. These results show that the proposed method has greatly improved various performance indicators in temperature anomaly detection, significantly reduced the number of false detections and missed detections, and thus ensured the reliability and accuracy of the detection results. Compared with other methods (such as IF, LOF, CNN, and autoencoder), the proposed method performs well in temperature detection in all regions. The other methods have lower precision and recall in different regions, and the F1 value is generally poor, indicating that the proposed method has obvious advantages in temperature detection. Figure 3 This is a visualization of the temperature detection results of the disclosed method in three areas.

[0105] In terms of outlier detection of salinity, compared with the results of real-time automatic quality labeling, the precision of this method in the three regions has been improved by about 1.0% on average, the recall has been improved by about 1.2%, and the F1 Score has been improved by about 1.1%. Although the improvement is relatively small, compared with other existing methods (such as IF, LOF, CNN, and autoencoder), the performance of this method is significantly better in all regions. The performance of other methods fluctuates greatly in different regions, especially in terms of detection accuracy and recall. For example, although the recall of the IF method in salinity detection is higher in some areas, the precision is generally lower; the precision and recall of the LOF method are both low in all areas; the detection performance of the CNN method in most areas is also not ideal; although the autoencoder performs better in some areas, the overall performance is still not as stable as this method. However, this method maintains high precision and high recall in all areas, showing its stability and superiority in processing complex data. These results further verify the effectiveness and advantages of this method in practical applications. Figure 4 Visualization of salinity detection results of our method in three regions.

[0106] Next, let’s talk about ablation experiments. In order to explore the impact of each component module of the model on performance, this paper conducts ablation experiments, removes the clustering (DBSCAN), polynomial regression (PR), sensor (Sensor) and multi-element (ME) modules in the model, and evaluates their specific impact on model performance. The ablation experiments of each module are conducted on three areas, and the experimental results are shown in Table 2.

[0107] Table 2 Comparison of ablation experiment results

[0108]

[0109] As can be seen from Table 2, our method TSD-AD shows excellent performance in all areas and all indicators, and almost all Precision, Recall and F1 values ​​are close to or reach 1, which shows that our model performs best when all modules are considered comprehensively.

[0110] The ablation experiment results show that the DBSCAN, PR, Sensor and ME modules all play an important role in improving the performance of the model. After removing these modules, the model's false positives and missed positives increased significantly, especially in salinity detection. This shows that each module plays an important role in reducing false positives and missed positives, further verifying the necessity and effectiveness of each module in our model design, and ensuring the excellent performance of the overall model.

[0111] Again, about efficiency analysis. The present disclosure further compares the anomaly detection efficiency of this method, CNN, AE, VAE_Kmeans and VAS-CLSTM on three different regional data sets. The clustering, polynomial regression and autoencoder in the disclosed method are regarded as a training process, and the process of using the trained autoencoder to predict the reconstruction error is regarded as a testing process. Both the training data and the test data are all the data used, of which the data volume of region 1 is 29,897 samples, that of region 2 is 12,809 samples, and that of region 3 is 94,884 samples. The training and testing time required for each model is shown in Table 3.

[0112] Table 3 Comparison of time efficiency results

[0113]

[0114]

[0115] As can be seen from the table, the differences in test time among the methods are small, and the test time is generally short, so it has little impact on the model. In view of this, no separate performance chart is made for the test time, but the focus of the analysis is on the training time. Figure 5 shown.

[0116] In terms of training time, the methods show significant differences between regions. The training time of TSD-AD and AE is relatively short and remains stable in all regions. The training time of CNN and VAS-CLSTM is significantly longer, especially the training time of VAS-CLSTM in region 3 reaches 340.22 seconds. Therefore, TSD-AD and AE have a greater advantage in time cost.

[0117] Secondly, from the perspective of model accuracy, TSD-AD showed extremely high accuracy in both temperature and salinity detection, especially in temperature detection, where the accuracy was close to 1, showing excellent classification ability. VAS-CLSTM also had high accuracy in salinity detection, but slightly lower accuracy in temperature detection. CNN had the worst accuracy among all models, especially in temperature detection, where it was almost impossible to effectively distinguish data, indicating that it may not be suitable for the dataset of the current task. In addition, the accuracy of AE was slightly lower than that of TSD-AD and VAS-CLSTM in all indicators, but considering its short training time, it still had certain practicality. In terms of recall rate, both TSD-AD and VAS-CLSTM performed well, especially TSD-AD, whose recall rate in temperature and salinity was close to 1, indicating that it could effectively capture target data in the detection task. VAS-CLSTM had a high recall rate in salinity, but was slightly insufficient in temperature detection. AE had a moderate recall rate, while CNN had a low recall rate, which was consistent with its accuracy results, indicating that this method may have a large missed detection problem.

[0118] Finally, the F1 score, as a balance indicator between precision and recall, further verifies the above analysis. TSD-AD has high F1 scores in both temperature and salinity detection, showing a good balance between precision and recall. VAS-CLSTM has a high F1 score in salinity detection and is suitable for salinity-related detection tasks, but performs poorly in temperature detection. AE has a certain advantage in the F1 score of salinity detection, while the F1 scores of CNN and VAE_Kmeans are generally low, which verifies their disadvantages in the current task.

[0119] In summary, the disclosed TSD-AD is the model with the best comprehensive performance in the current task. It can provide high-precision and high-recall detection results while ensuring a short training time. It is suitable for application scenarios that need to comprehensively consider efficiency and accuracy. VAS-CLSTM has a high recall rate and F1 score in salinity detection, which is suitable for high-accuracy requirements related to salinity, but the training time is long. CNN is not recommended for such tasks due to its poor performance in accuracy, recall rate and F1 score. Although AE is slightly lower than TSD-AD in accuracy and recall rate, its shorter training time makes it suitable for time-sensitive application scenarios.

[0120] References related to the present disclosure:

[0121] 1. Breunig, M.M., et al. LOF: identifying density - based local outliers. in Proceedings of the 2000 ACM SIGMOD international conference on Management of data. 2000.

[0122] 2. Liu, F.T., K.M. Ting, and Z.-H. Zhou, Isolation - based anomaly detection. ACM Transactions on Knowledge Discovery from Data (TKDD), 2012. 6(1): p. 1 - 39.

[0123] 3. B., et al., Estimating the support of a high - dimensional distribution. Neural computation, 2001. 13(7): p. 1443 - 1471.

[0124] 4. Kramer, M.A., Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 1991. 37(2): p. 233 - 243.

[0125] 5. Kingma, D.P., Auto - encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.

[0126] 6. Graves, A. and A. Graves, Long short - term memory. Supervised sequence labelling with recurrent neural networks, 2012: p. 37 - 45.

[0127] 7.Vaswani,A.,et al.,Attention is all you need.Advances in neuralinformation processing systems,2017.30.

[0128] 8.Goodfellow,I.,et al.,Generative adversarial nets(Advances in neuralinformation processing systems)(pp.2672–2680).Red Hook,NY Curran,2014.

[0129] 9.Kang,H.,D.Kim,and S.Lim,Machine Learning-Based Anomaly Detection onSeawater Temperature Data with Oversampling.Journal of Marine Science andEngineering,2024.12(5):p.807.

[0130] 10.Jiang,Y.,et al.Anomaly detection of Argo data using variationalautoencoder and k-means clustering.in 2022IEEE 5th Advanced InformationManagement,Communicates,Electronic and Automation Control Conference(IMCEC).2022.IEEE.

[0131] 11.Yulong,L.,et al.,Quality control of sea temperature observationdata using deeplearning neural networks.Marine Science Bulletin,2021.40(03):p.283-291.

[0132] 12.Sui,L.and Y.Jiang,Argo data anomaly detection based on transformerand Fourier transform.Journal of Sea Research,2024.198:p.102483.

[0133] 13.Wang,Y.,et al.,Study on wavelet neural network based anomalydetection in ocean observing data series.Ocean Engineering,2019.186:p.106129.

[0134] 14.Jiang,Y.,et al.Research on Argo Data Anomaly Detection Based onImproved DBSCAN Algorithm.in China Conference on Wireless SensorNetworks.2022.Springer.

[0135] 15.Houghton,I.A.and J.D.Wilson,El detection via unsupervisedclustering of Argo temperature profiles.Journal of Geophysical Research:Oceans,2020.125(9):p.e2019JC015947.

[0136] 16.Jiang,F.,et al.,Ocean Observation Data Prediction for Argo DataQuality Control Using Deep Bidirectional LSTM Network.Security andCommunication Networks,2021.2021(1):p.5665386.

[0137] 17. Zhang, Q., C. Qian, and C. Dong, A machine learning approach to quality-control Argo temperature data. Atmospheric and Oceanic Science Letters, 2023.16(4):p.100292.

[0138] 18. Zhang, S., et al., Vertical Attention-Based Siamese ConvLSTM Network for Argo Data Error Detection. IEEE Transactions on Geoscience and RemoteSensing, 2023.

[0139] It should be understood that in the embodiments of the present invention, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0140] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0143] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting anomalies in temperature, salinity and depth combined ocean data, characterized in that: Abnormal data is detected in the temperature-salinity-depth combined ocean data through the trained detection model.

2. The method according to claim 1, characterized in that: The construction method of the detection model includes the definition of temperature, salinity and depth combined ocean data, Assume that the temperature, salinity and depth joint ocean data set input by the detection model is D = (d, t, s, LON, LAT) × m, where depth d, temperature t, salinity s, longitude LON and latitude LAT are m-dimensional data with five features.

3. The method according to claim 2, characterized in that The method comprises the following steps: Step 1: preprocess the data set D to extract the data set D with normal or approximately normal values. * ; Step 2: Dataset D * The salinity and temperature data in the image are separated to obtain the first salinity data set. With the first temperature data set Based on these two data sets, polynomial regression analysis is performed to train the polynomial regression model R s with polynomial regression model in R t , Separate the salinity and temperature data in data set D to obtain the second salinity data set D -s With the second temperature data set D -t , input other data in the corresponding data set into the polynomial regression model R s with polynomial regression model in R t In the dataset D, the salinity data and other data are predicted respectively. sP And the data set D consisting of temperature data and other data tP , Step 3: Dataset D sP With D tP Input them into the autoencoder model AEM respectively, and the corresponding reconstructed dataset D is obtained sR With D tR , Step 4: Dataset D sR With D tR Calculate the error E with the data set D respectively s With E t , the data that are judged as noise data according to the error value are temperature anomaly data and salinity anomaly data.

4. The method according to claim 3, characterized in that In step 1, the DBSCAN clustering algorithm is used to extract an approximate normal data set D from the data set D. * .

5. The method according to claim 3, characterized in that: In step 4, the DBSCAN clustering algorithm is used to detect noise data.

6. The method according to claim 5, characterized in that The detected anomaly dataset is labeled for quality.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 6.

8. A temperature, salinity and depth combined ocean data anomaly detection system, characterized in that: The system includes a server having a memory; and A processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images