Quality control method and system based on buoy observation data

Abstract

The invention relates to the technical field of data processing, and discloses a quality control method and system based on buoy observation data. The method comprises the following steps: performing multi-dimensional anomaly detection processing on preprocessed standard format data to obtain an anomaly detection report marked with abnormal and suspicious data points; performing multi-source data fusion and cross validation processing on the data in the anomaly detection report to obtain an initial data quality score; performing intelligent correction processing on the anomaly detection report to obtain a corrected data set and uncertainty estimation data corresponding to the corrected data set; performing consistency analysis processing on the corrected data set in a space-time dimension to obtain a space-time consistency score and abnormal region marking data; and carrying out comprehensive evaluation processing on the abnormal region mark data to obtain a target data quality grade identifier and a target quality report, and carrying out quality control strategy analysis to obtain a target quality control strategy. According to the invention, the accuracy of quality control based on buoy observation data is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a quality control method and system based on buoy observation data. Background Art

[0002] The ocean buoy observation system is an important means of obtaining ocean environmental parameters and can provide long-term continuous ocean data. However, due to the complex and changeable ocean environment, the long-term exposure of instruments and equipment to harsh environments, and the interference that may occur during data transmission, buoy observation data often have quality problems such as noise, outliers, and missing values. Traditional data quality control methods mainly rely on fixed thresholds and simple statistical models, which are difficult to adapt to the dynamic changes of the ocean environment.

[0003] Existing quality control technologies usually treat time and space dimensions separately, lacking comprehensive consideration of the spatiotemporal coupling characteristics of data. At the same time, when faced with multi-source heterogeneous data, existing methods often find it difficult to effectively integrate information from different sources, resulting in insufficient accuracy and reliability in data quality assessment. In addition, existing technologies often lack the quantification and transmission of uncertainty during anomaly detection and data correction, making it difficult to provide reliable quality assurance for subsequent data applications. Summary of the invention

[0004] The present application provides a quality control method and system based on buoy observation data, which are used to improve the accuracy of quality control based on buoy observation data.

[0005] In a first aspect, the present application provides a quality control method based on buoy observation data, the quality control method based on buoy observation data comprising: performing format unification and basic numerical check processing on the original ocean environment parameter data collected by the buoy to obtain pre-processed standard format data; performing multi-dimensional anomaly detection processing on the pre-processed standard format data to obtain an anomaly detection report marked with abnormal and suspicious data points; performing multi-source data fusion and cross-validation processing on the data in the anomaly detection report to obtain an initial data quality score; based on the initial data quality score, performing intelligent correction processing on the anomaly detection report through a gradient boosting tree model to obtain a corrected data set and uncertainty estimation data corresponding to the corrected data set; based on the uncertainty estimation data, performing consistency analysis processing on the corrected data set in the spatiotemporal dimension to obtain a spatiotemporal consistency score and abnormal area marking data; based on the spatiotemporal consistency score, performing comprehensive evaluation processing on the abnormal area marking data to obtain a target data quality level identifier and a target quality report, and performing quality control strategy analysis through the target data quality level identifier and the target quality report to obtain a target quality control strategy.

[0006] In combination with the first aspect, in a first implementation method of the first aspect of the present application, the format unification and basic numerical value check processing of the original ocean environment parameter data collected by the buoy to obtain pre-processed standard format data includes: performing data type identification processing on the original ocean environment parameter data to obtain parameter type identification data; based on the parameter type identification data, performing data format conversion processing on the original ocean environment parameter data to obtain unified format data; performing physical range threshold check processing on the unified format data to obtain preliminary screening data; performing high-frequency noise removal processing on the preliminary screening data through a wavelet transform algorithm to obtain denoised data; performing time series alignment and interpolation processing on the denoised data to obtain time-aligned data; performing probability distribution analysis processing on the time-aligned data through an adaptive kernel density estimation algorithm to obtain the pre-processed standard format data.

[0007] In combination with the first aspect, in a second implementation of the first aspect of the present application, the pre-processed standard format data is subjected to multi-dimensional anomaly detection processing to obtain an anomaly detection report marked with abnormal and suspicious data points, including: performing sliding window segmentation processing on the pre-processed standard format data to obtain a time window data set; performing dynamic threshold calculation processing on the time window data set through an adaptive threshold algorithm to obtain parameter threshold range data; based on the parameter threshold range data, performing an outlier marking processing on the time window data set to obtain preliminary anomaly marking data; performing time series decomposition processing on the preliminary anomaly marking data through a seasonal decomposition algorithm to obtain trend component data, seasonal component data and residual component data; performing local anomaly factor calculation processing on the residual component data to obtain local anomaly score data; based on the trend component data and the seasonal component data, performing probability density estimation processing on the local anomaly score data through a multivariate Gaussian mixture model to obtain abnormal probability distribution data; based on the abnormal probability distribution data, performing cluster analysis processing on the local anomaly score data to obtain an abnormal type classification result; performing abnormal cause analysis processing on the abnormal type classification result to obtain the anomaly detection report marked with abnormal and suspicious data points.

[0008] In combination with the first aspect, in a third implementation of the first aspect of the present application, the data in the anomaly detection report is subjected to multi-source data fusion and cross-validation processing to obtain an initial data quality score, including: performing spatiotemporal matching processing on the data in the anomaly detection report to obtain a multi-source data correspondence table; performing time series alignment processing on the multi-source data correspondence table to obtain an aligned multi-source data set; performing data type standardization processing on the aligned multi-source data set to obtain a standardized multi-source data set; performing weight calculation processing on the standardized multi-source data set to obtain a data source weight coefficient; performing weighted fusion processing on the standardized multi-source data set based on the data source weight coefficient to obtain a fused data set; performing data completion processing on the fused data set to obtain a complete fused data set; performing cross-validation grouping processing on the complete fused data set to obtain a validation data subset; performing error estimation processing on the validation data subset to obtain a validation error estimate value; performing consistency scoring processing on the complete fused data set based on the validation error estimate value to obtain a data consistency score; performing multi-factor comprehensive evaluation processing on the data consistency score to obtain the initial data quality score.

[0009] In combination with the first aspect, in a fourth implementation method of the first aspect of the present application, based on the initial data quality score, the abnormal detection report is intelligently corrected by a gradient boosting tree model to obtain a corrected data set and uncertainty estimation data corresponding to the corrected data set, including: feature extraction processing is performed on the abnormal detection report to obtain an abnormal data feature vector; normalization processing is performed on the initial data quality score to obtain a standardized quality score; based on the standardized quality score, weighting processing is performed on the abnormal data feature vector to obtain a weighted feature vector; data enhancement processing is performed on the weighted feature vector to obtain an expanded training data set; partitioning processing is performed on the expanded training data set to obtain a training subset and a validation subset; model training processing is performed on the training subset by a gradient boosting tree model to obtain an initial corrected model; cross-validation processing is performed on the validation subset to obtain a model performance evaluation index; based on the model performance evaluation index, parameter optimization processing is performed on the initial corrected model to obtain an optimized corrected model; data correction processing is performed on the abnormal detection report by the optimized corrected model to obtain the corrected data set; Bootstrap resampling processing is performed on the corrected data set to obtain uncertainty estimation data corresponding to the corrected data set.

[0010] In combination with the first aspect, in a fifth implementation of the first aspect of the present application, based on the uncertainty estimation data, the revised data set is subjected to consistency analysis processing in the spatiotemporal dimension to obtain a spatiotemporal consistency score and abnormal area marking data, including: spatially gridding the revised data set to obtain a gridded data matrix; performing principal component analysis processing on the gridded data matrix to obtain a spatial main mode; performing wavelet continuity analysis processing on the revised data set to obtain a time scale characteristic spectrum; based on the spatial main mode and the time scale characteristic spectrum, performing spatiotemporal modal decomposition processing on the revised data set to obtain a spatiotemporal coupling mode; performing singular spectrum analysis processing on the spatiotemporal coupling mode to obtain key spatiotemporal variation characteristics; based on the key spatiotemporal variation characteristics, performing abnormal pattern recognition processing on the revised data set to obtain the abnormal area marking data; performing spatial adaptive kernel density estimation processing on the abnormal area marking data to obtain an abnormal area probability distribution map; performing time evolution analysis processing on the abnormal area probability distribution map through empirical mode decomposition to obtain the spatiotemporal consistency score.

[0011] In combination with the first aspect, in a sixth implementation of the first aspect of the present application, based on the spatiotemporal consistency score, the abnormal area marking data is comprehensively evaluated and processed to obtain a target data quality level identifier and a target quality report, and a quality control strategy analysis is performed through the target data quality level identifier and the target quality report to obtain a target quality control strategy, including: performing piecewise linear fitting processing on the spatiotemporal consistency score to obtain a scoring trend function; based on the scoring trend function, performing classification threshold calculation processing on the abnormal area marking data to obtain a multi-level quality threshold; performing fuzzy clustering processing on the abnormal area marking data to obtain a data quality level cluster; based on the multi-level The quality threshold and the data quality level cluster are used to perform level mapping processing on the abnormal area marking data to obtain the target data quality level identifier; the target data quality level identifier is subjected to statistical distribution analysis processing to obtain quality distribution characteristics; based on the quality distribution characteristics, the abnormal area marking data is subjected to abnormal pattern extraction processing to obtain a typical abnormal pattern library; the typical abnormal pattern library is subjected to causal relationship analysis processing to obtain abnormal cause inference results; based on the abnormal cause inference results, the abnormal area marking data is subjected to quality improvement suggestion generation processing to obtain the target quality report; the target quality report is subjected to strategy optimization processing to obtain the target quality control strategy.

[0012] In a second aspect, the present application provides a quality control system based on buoy observation data, the quality control system based on buoy observation data comprising:

[0013] The inspection module is used to unify the format and perform basic numerical inspection on the original ocean environment parameter data collected by the buoy to obtain the pre-processed standard format data;

[0014] A detection module, used to perform multi-dimensional anomaly detection processing on the pre-processed standard format data to obtain an anomaly detection report marked with abnormal and suspicious data points;

[0015] A verification module, used to perform multi-source data fusion and cross-validation processing on the data in the anomaly detection report to obtain an initial data quality score;

[0016] A correction module, used to perform intelligent correction processing on the anomaly detection report through a gradient boosting tree model based on the initial data quality score, to obtain a corrected data set and uncertainty estimation data corresponding to the corrected data set;

[0017] A processing module, configured to perform consistency analysis processing on the modified data set in the spatiotemporal dimension based on the uncertainty estimation data, to obtain a spatiotemporal consistency score and abnormal region marking data;

[0018] An evaluation module is used to perform comprehensive evaluation processing on the abnormal area marking data based on the spatiotemporal consistency score to obtain a target data quality level identifier and a target quality report, and to perform quality control strategy analysis through the target data quality level identifier and the target quality report to obtain a target quality control strategy.

[0019] In the technical solution provided by this application, the standardization and preliminary cleaning of data are achieved by unifying the format and performing basic numerical check processing on the original marine environmental parameter data; secondly, a multi-dimensional anomaly detection method is adopted, combined with sliding window, adaptive threshold and time series decomposition and other technologies, which can more comprehensively and accurately identify anomalies and suspicious points in the data, significantly improving the accuracy and robustness of anomaly detection; through multi-source data fusion and cross-validation processing, data information from different sources is fully utilized, effectively reducing the deviation and uncertainty that may be caused by a single data source, and improving the reliability of data quality assessment; the gradient boosting tree model is used for intelligent correction processing, which can not only adaptively learn the inherent laws of the data, but also It can generate corrected data sets and their corresponding uncertainty estimates, providing reliable quality assurance for subsequent data applications; at the same time, through consistency analysis in the spatiotemporal dimensions, combined with advanced technologies such as principal component analysis, wavelet continuity analysis and singular spectrum analysis, it realizes in-depth mining of the spatiotemporal coupling characteristics of the data, and effectively identifies abnormal areas and spatiotemporal change patterns; finally, through comprehensive evaluation and processing, the target data quality level identification and target quality report are obtained, and quality control strategy analysis is carried out based on this, which significantly improves the reliability and availability of buoy observation data, provides high-quality data support for marine scientific research, environmental monitoring, climate change research and other fields, and improves the accuracy of quality control of buoy observation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 This is a schematic diagram of an embodiment of a quality control method based on buoy observation data in an embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of an embodiment of a quality control system based on buoy observation data in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The embodiment of the present application provides a quality control method and system based on buoy observation data. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In one embodiment of the present application, a quality control method based on buoy observation data includes:

[0025] Step S101, performing format unification and basic value check processing on the original ocean environment parameter data collected by the buoy to obtain pre-processed standard format data;

[0026] Step S102: Perform multi-dimensional anomaly detection on the pre-processed standard format data to obtain an anomaly detection report marked with abnormal and suspicious data points;

[0027] Step S103, performing multi-source data fusion and cross-validation processing on the data in the anomaly detection report to obtain an initial data quality score;

[0028] Step S104: Based on the initial data quality score, the abnormal detection report is intelligently corrected by using a gradient boosting tree model to obtain a corrected data set and uncertainty estimation data corresponding to the corrected data set;

[0029] Step S105: Based on the uncertainty estimation data, the modified data set is subjected to consistency analysis in the spatiotemporal dimension to obtain a spatiotemporal consistency score and abnormal region marking data;

[0030] Step S106: Based on the spatiotemporal consistency score, the abnormal area marked data is comprehensively evaluated and processed to obtain the target data quality level identification and the target quality report, and the quality control strategy is analyzed through the target data quality level identification and the target quality report to obtain the target quality control strategy.

[0031] It is understandable that the execution subject of the present application may be a quality control system based on buoy observation data, or a terminal or a server, which is not limited here. The present application embodiment is described by taking a server as the execution subject as an example.

[0032] Specifically, the data types of these raw data are identified. The specific process is to classify the data collected by different buoys, such as temperature, salinity, current speed, etc., and identify them according to the different characteristics of these parameters. Then, the data format is converted according to these identifiers, and the data of different buoys are converted into a unified format, which can ensure the consistency of the data in the subsequent processing steps. The physical range threshold of these unified format data is checked to ensure that the data is within a reasonable range. For example, salinity data should be between 0 and 40 PSU (Practical Salinity Units). If the data exceeds this range, it needs to be screened. Then, the high-frequency noise in the data is removed by the wavelet transform algorithm to obtain more accurate denoised data. Wavelet transform is a signal processing technology that can decompose the signal into different frequency components, thereby identifying and removing high-frequency noise. Subsequently, the denoised data is time-series aligned and interpolated to ensure the continuity and consistency of the data in time. Finally, the probability distribution analysis of the time-aligned data is performed by the adaptive kernel density estimation algorithm to obtain the pre-processed standard format data.

[0033] After obtaining the preprocessed standard format data, the next step is to perform multi-dimensional anomaly detection processing. This step first divides the data into multiple time windows by the sliding window segmentation method so that the data can be analyzed locally. Then, the adaptive threshold algorithm is used to calculate the dynamic threshold of the data in each time window to obtain the threshold range of the parameter. Based on these threshold ranges, the time window data is marked as outliers, and the abnormal data points are preliminarily identified. Next, the time series data is decomposed into trend components, seasonal components, and residual components by the seasonal decomposition algorithm, and the local anomaly factor is calculated for the residual component to obtain the local anomaly score. The probability density of the local anomaly score is estimated using the multivariate Gaussian mixture model to obtain the abnormal probability distribution data. Based on these data, the local anomaly scores are clustered and analyzed to classify different types of anomalies, and the causes of anomalies are analyzed, and finally an anomaly detection report marked with abnormal and suspicious data points is generated. For the data in the anomaly detection report, multi-source data fusion and cross-validation processing are performed next. First, the data in the anomaly detection report is matched in time and space to establish a corresponding relationship table for multi-source data. Then, these multi-source data are aligned in time series to generate an aligned multi-source data set. The aligned data are type standardized to ensure that data from different sources have consistent formats and standards. Next, the weight coefficients of the data sources are calculated, and the standardized data sets are weighted and fused to generate a fused data set. For the fused data set, data completion is performed to ensure data integrity. Then, the fused data set is validated and grouped by the cross-validation method to evaluate the consistency of the data and obtain the validation error estimate. Based on these error estimates, the data is scored for consistency, and the initial data quality score is obtained through a multi-factor comprehensive evaluation. Based on the initial data quality score, the anomaly detection report is intelligently corrected by the gradient boosting tree model. First, the feature vectors of the anomaly data are extracted from the anomaly detection report, and the initial data quality score is normalized. Then, these feature vectors are weighted to generate a weighted feature vector. Next, the training data set is expanded by the data augmentation method and divided into a training subset and a validation subset. The training subset is trained using the gradient boosting tree model to generate an initial correction model. The validation subset is evaluated by cross-validation to obtain the performance indicators of the model. Based on these indicators, the parameters of the initial correction model are optimized to generate an optimized correction model. Finally, the optimization model is used to correct the data of the anomaly detection report to obtain a corrected data set, and the uncertainty estimation data of the corrected data set is obtained through the Bootstrap resampling method.

[0034] Based on the uncertainty estimation data, the consistency analysis of the modified data set in the spatiotemporal dimension is performed. First, the modified data set is spatially gridded to generate a gridded data matrix. Then, principal component analysis is performed to obtain the spatial main mode. The time scale characteristic spectrum is obtained through wavelet continuity analysis, and spatiotemporal modal decomposition is performed to identify the key spatiotemporal variation characteristics. Based on these characteristics, abnormal pattern recognition is performed on the modified data, abnormal areas are marked, and the probability distribution map of abnormal areas is generated through spatial adaptive kernel density estimation. Finally, the time evolution analysis of abnormal areas is performed through empirical mode decomposition to obtain the spatiotemporal consistency score. Based on the spatiotemporal consistency score, the abnormal area marked data is comprehensively evaluated to obtain the target data quality level identification and target quality report. First, the spatiotemporal consistency score is piecewise linearly fitted to generate a scoring trend function. Based on the scoring trend function, the classification threshold is calculated, and the abnormal area data is fuzzy clustered to generate data quality level clusters. Then, the data quality level clusters are mapped to the target data quality level identification. By statistically analyzing the distribution characteristics of the target data quality level identification, typical abnormal patterns are extracted, and causal relationship analysis is performed to generate quality improvement suggestions, and finally a target quality report is compiled. Based on the target quality report, strategy optimization is performed to obtain the final target quality control strategy.

[0035] For example, suppose there is a salinity drift problem in the raw data collected by the buoy. First, the format of these raw data is unified, and the salinity data is converted from different formats to a unified format through data type identification. Then, the physical range threshold check is performed on the salinity data to filter out the data that exceeds the reasonable range. After removing high-frequency noise through wavelet transform, time series alignment and interpolation processing are performed to ensure the continuity of the data. Finally, the probability distribution of the data is analyzed by the adaptive kernel density estimation algorithm to obtain the preprocessed standard format data. Multi-dimensional anomaly detection is performed on the preprocessed salinity data. The data is segmented by sliding window, and the adaptive threshold algorithm is applied to calculate the threshold range of each time window, marking the abnormal salinity data points. Then, the time series data is decomposed by the seasonal decomposition algorithm, the local anomaly score of the residual component is calculated, and the anomaly probability distribution is estimated by the Gaussian mixture model. Finally, cluster analysis is performed to classify and analyze the causes of anomalies and generate an anomaly detection report. Multi-source data fusion and cross-validation processing are performed on the salinity data in the anomaly detection report. First, the data is matched in time and space, a corresponding relationship table is generated, and time series alignment is performed. Then, data type standardization and weight calculation are performed, and the data is weightedly fused to generate a fused data set. The consistency of the data is evaluated by cross-validation to obtain the initial data quality score. Based on the initial data quality score, the anomaly detection report is intelligently corrected by the gradient boosting tree model. First, the feature vector of the abnormal data is extracted and normalized. Then, the training data set is expanded by data enhancement, and model training and parameter optimization are performed. Finally, the anomaly detection report is corrected using the optimized model to obtain a corrected data set and its uncertainty estimation data. Based on the uncertainty estimation data, the corrected data set is analyzed for spatiotemporal consistency. Through spatial gridding, principal component analysis and wavelet continuity analysis, key spatiotemporal variation features are identified, and abnormal pattern recognition is performed to mark abnormal areas and generate probability distribution maps of abnormal areas. Finally, the spatiotemporal consistency score is obtained through time evolution analysis. Based on the spatiotemporal consistency score, the abnormal area marked data is comprehensively evaluated to generate the target data quality level identification and target quality report. The classification threshold is calculated by piecewise linear fitting, and fuzzy clustering is performed to generate data quality level clusters. Finally, the data quality level is mapped to the target data quality level identifier, and quality improvement suggestions and target quality control strategies are generated.

[0036] In the embodiment of the present application, the standardization and preliminary cleaning of the data are achieved by unifying the format and performing basic numerical check processing on the original marine environmental parameter data; secondly, a multi-dimensional anomaly detection method is adopted, combined with sliding window, adaptive threshold and time series decomposition and other technologies, which can more comprehensively and accurately identify anomalies and suspicious points in the data, significantly improving the accuracy and robustness of anomaly detection; through multi-source data fusion and cross-validation processing, data information from different sources is fully utilized, effectively reducing the deviation and uncertainty that may be caused by a single data source, and improving the reliability of data quality assessment; the gradient boosting tree model is used for intelligent correction processing, which can not only adaptively learn the inherent laws of the data, but also The generation of a corrected data set and its corresponding uncertainty estimate provides a reliable quality assurance for subsequent data applications. At the same time, through consistency analysis in the spatiotemporal dimensions, combined with advanced technologies such as principal component analysis, wavelet continuity analysis, and singular spectrum analysis, an in-depth exploration of the spatiotemporal coupling characteristics of the data is achieved, and abnormal areas and spatiotemporal change patterns are effectively identified. Finally, through comprehensive evaluation and processing, the target data quality level identification and target quality report are obtained, and quality control strategy analysis is carried out based on this, which significantly improves the reliability and availability of buoy observation data, provides high-quality data support for marine scientific research, environmental monitoring, climate change research and other fields, and improves the accuracy of quality control of buoy observation data.

[0037] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0038] (1) Perform data type identification processing on the original marine environment parameter data to obtain parameter type identification data;

[0039] (2) Based on the parameter type identification data, the original marine environment parameter data is converted into a unified format;

[0040] (3) Performing physical range threshold check processing on the unified format data to obtain preliminary screening data;

[0041] (4) removing high-frequency noise from the preliminary screening data using a wavelet transform algorithm to obtain noise-reduced data;

[0042] (5) Performing time series alignment and interpolation processing on the denoised data to obtain time-aligned data;

[0043] (6) The time-aligned data is analyzed and processed for probability distribution using an adaptive kernel density estimation algorithm to obtain preprocessed data in a standard format.

[0044] Specifically, during the buoy observation process, the collected marine environmental parameter data are diverse and complex, and common parameters include temperature, salinity, pH value, etc. These raw data directly used for subsequent analysis may affect the accuracy of the analysis results due to problems such as inconsistent formats and noise. Therefore, it is necessary to convert the raw marine environmental parameter data into standard format data through a series of processing steps for further application and analysis. The raw marine environmental parameter data is processed for data type identification to obtain parameter type identification data. This process includes classifying and marking the collected data, such as identifying temperature data as T, salinity data as S, and pH value data as pH. The key to data type identification is to accurately distinguish different types of data so that corresponding processing methods can be used according to different parameter types in the subsequent processing process. A rule-based classification algorithm can be used, such as identification based on data units, ranges and other characteristics.

[0045] Based on the parameter type identification data, the original marine environment parameter data is converted to obtain unified format data. Since the data formats collected by different types of sensors and equipment are different, unified format data helps to simplify subsequent processing steps. Common format conversions include unit conversion (for example, converting temperature from Fahrenheit to Celsius), timestamp format unification (for example, converting timestamps in different formats to ISO 8601 format), etc. Through format conversion, all data is guaranteed to have consistent formats and units so that they can be processed and analyzed on the same platform.

[0046] Perform physical range threshold check on the uniform format data to obtain preliminary screening data. The physical range threshold check is to set a reasonable range based on the physical characteristics of each parameter. For example, the temperature in the marine environment is usually between -2℃ and 30℃, and the salinity is between 30‰ and 40‰. Any data outside this range may be an outlier or erroneous data and needs to be eliminated in the preliminary screening. In this way, the rationality and credibility of the data can be ensured.

[0047] The high-frequency noise of the preliminary screened data is removed by wavelet transform algorithm to obtain denoised data. Wavelet transform is a commonly used signal processing method that can effectively separate the high-frequency and low-frequency components in the signal. The specific steps include wavelet decomposition of the signal, decomposing the signal into wavelet coefficients of different scales, then thresholding the wavelet coefficients of the high-frequency part, and finally performing wavelet reconstruction to obtain the denoised data. This method can retain important information in the signal while removing high-frequency noise and improving the quality of the data.

[0048] Perform time series alignment and interpolation processing on the denoised data to obtain time-aligned data. Time series alignment refers to aligning data collected at different times to the same time axis for synchronous analysis. In actual operation, there may be problems such as different sampling frequencies and missing data, and the data needs to be supplemented by interpolation methods. Commonly used interpolation methods include linear interpolation and spline interpolation. Through interpolation, incomplete data sequences can be supplemented to have a uniform time interval.

[0049] The time-aligned data is analyzed and processed by the adaptive kernel density estimation algorithm to obtain the preprocessed standard format data. Kernel density estimation is a non-parametric statistical method that can be used to estimate the probability density function of data. Adaptive kernel density estimation is based on traditional kernel density estimation and adjusts the bandwidth of the kernel function according to the local characteristics of the data distribution to improve the accuracy of the estimation. In this way, the probability distribution of time-aligned data can be obtained, which helps to discover abnormal patterns or trends in the data.

[0050] During the buoy observation process, the buoy in a certain sea area collected the following raw data: temperature data (20.5℃, 21.0℃, 22.1℃), salinity data (34.5‰, 35.0‰, 34.8‰) and pH data (8.1, 8.2, 8.0). First, through data type identification, the temperature data is identified as T, the salinity data is identified as S, and the pH data is identified as pH. Then, the temperature data is uniformly converted to Celsius format, and the timestamp is unified to ISO 8601 format. Then, according to the physical range threshold check, the abnormal data is eliminated and the data within a reasonable range is retained. The data after preliminary screening is subjected to wavelet transformation to remove high-frequency noise and obtain denoised data. Through time series alignment and interpolation processing, the incomplete data sequence is completed to have a uniform time interval. Finally, the time-aligned data is subjected to probability distribution analysis through adaptive kernel density estimation to obtain the preprocessed standard format data.

[0051] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0052] (1) Perform sliding window segmentation on the preprocessed standard format data to obtain a time window data set;

[0053] (2) Dynamic threshold calculation is performed on the time window data set through an adaptive threshold algorithm to obtain parameter threshold range data;

[0054] (3) Based on the parameter threshold range data, outlier marking is performed on the time window data set to obtain preliminary outlier marking data;

[0055] (4) Performing time series decomposition processing on the preliminary abnormality marked data through seasonal decomposition algorithm to obtain trend component data, seasonal component data and residual component data;

[0056] (5) Calculate the local anomaly factor of the residual component data to obtain the local anomaly score data;

[0057] (6) Based on the trend component data and the seasonal component data, the probability density estimation of the local anomaly score data is performed through a multivariate Gaussian mixture model to obtain the anomaly probability distribution data;

[0058] (7) Based on the abnormal probability distribution data, cluster analysis is performed on the local abnormal score data to obtain the abnormal type classification result;

[0059] (8) Perform an anomaly cause analysis on the anomaly type classification results to obtain an anomaly detection report marked with abnormal and suspicious data points.

[0060] Specifically, the original marine environmental parameter data collected by the buoy is formatted and basic numerical check is performed to obtain the preprocessed standard format data. This step includes data type identification of the original data, identification of various parameters such as temperature, salinity, etc., and data format conversion based on the identification results to unify the data formats of different data sources. In addition, the physical range threshold check is performed on the unified format data to remove abnormal data that exceeds the reasonable range. Subsequently, the preliminary screening data is processed for high-frequency noise removal by the wavelet transform algorithm to retain important information in the signal and remove noise. Finally, the denoised data is aligned and interpolated for time series to ensure the consistency of the data in time, and the probability distribution analysis is performed by the adaptive kernel density estimation algorithm to obtain the preprocessed standard format data. The preprocessed standard format data needs to be processed for multi-dimensional anomaly detection to obtain an anomaly detection report marked with abnormal and suspicious data points. This includes sliding window segmentation of the preprocessed data to generate a time window data set. The data parameter threshold range of each time window is calculated by the adaptive threshold algorithm, and the abnormal values ​​are marked based on the range. Then, the seasonal decomposition algorithm is applied to decompose the time series data into trend, seasonal and residual components, and the anomaly score of the residual component is further calculated by the local anomaly factor. The probability density of the anomaly score data is estimated based on the multivariate Gaussian mixture model, and the anomaly type is determined by cluster analysis, the cause of the anomaly is analyzed, and an anomaly detection report is generated. The data in the anomaly detection report is subjected to multi-source data fusion and cross-validation processing to obtain the initial data quality score. This includes time-space matching processing, generating a multi-source data correspondence table, and performing time series alignment and data type standardization. Next, the data source weight coefficient is calculated, and the standardized data set is subjected to weighted fusion and data completion processing to generate a complete fused data set. The validation data subset is obtained through cross-validation grouping processing, and error estimation and consistency scoring are performed, and the initial data quality score is obtained through comprehensive evaluation.

[0061] Based on the initial data quality score, the anomaly detection report is intelligently corrected through the gradient boosting tree model to obtain the corrected data set and its corresponding uncertainty estimation data. First, the abnormal data feature vector is extracted, and the initial quality score is normalized to generate a standardized quality score. The abnormal data feature vector is weighted and data enhanced to generate an expanded training data set. The gradient boosting tree model is used for model training and cross-validation to optimize the model parameters, and finally the anomaly detection report is corrected. Bootstrap resampling is performed on the corrected data set to generate uncertainty estimation data.

[0062] Based on the uncertainty estimation data, the revised data set is analyzed and processed for consistency in the spatiotemporal dimension to obtain the spatiotemporal consistency score and abnormal area marking data. This includes spatial gridding and principal component analysis of the revised data set to obtain the spatial main mode. The time scale characteristic spectrum is obtained through wavelet continuity analysis, and spatiotemporal modal decomposition is performed to obtain the spatiotemporal coupling mode and key spatiotemporal variation characteristics. The revised data set is subjected to abnormal pattern recognition to generate abnormal area marking data, and spatial adaptive kernel density estimation is performed to generate the abnormal area probability distribution map. The time evolution of the abnormal area probability distribution map is analyzed through empirical mode decomposition, and the spatiotemporal consistency score is finally obtained.

[0063] Based on the spatiotemporal consistency score, the abnormal area marked data is comprehensively evaluated and processed to obtain the target data quality level identification and target quality report, and the quality control strategy is analyzed through these data to obtain the target quality control strategy. This includes piecewise linear fitting of the spatiotemporal consistency score, calculating the scoring trend function, and calculating the classification threshold of the abnormal area marked data based on this function to generate multi-level quality thresholds. The data quality level cluster is obtained through fuzzy clustering processing, and the level mapping is performed to generate the target data quality level identification. The identification is statistically analyzed, the typical abnormal pattern library is extracted, and the causal relationship analysis is performed to generate the abnormal cause inference results and quality improvement suggestions, and finally the quality control strategy is optimized.

[0064] For example, during a buoy observation, the following raw data were collected: temperature (20.5℃, 21.0℃, 22.1℃), salinity (34.5‰, 35.0‰, 34.8‰) and pH value (8.1, 8.2, 8.0). First, through data type recognition, temperature, salinity and pH value are marked as T, S and pH respectively, and the timestamp format is unified. After physical range check, abnormal data with a temperature of 50℃ are eliminated. After removing noise by wavelet transform, missing data are supplemented by interpolation. Then, the processed data is subjected to sliding window segmentation and adaptive threshold calculation, and abnormal data with a salinity of 40‰ is marked. After applying the seasonal decomposition algorithm and local anomaly factor calculation, it is found that the temperature data in a certain period of time is abnormal, and an anomaly detection report is generated. Then, through multi-source data fusion and weighted calculation, the initial data quality score is generated, and the gradient boosting tree model is used for data correction and uncertainty estimation. Finally, the revised data were analyzed for spatiotemporal consistency, and abnormal fluctuations in a certain area were found within a specific time period. After comprehensive evaluation, a target quality report and control strategy were generated.

[0065] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0066] (1) Performing spatiotemporal matching on the data in the anomaly detection report to obtain a multi-source data correspondence table;

[0067] (2) Perform time series alignment on the multi-source data correspondence table to obtain an aligned multi-source data set;

[0068] (3) performing data type standardization on the aligned multi-source dataset to obtain a standardized multi-source dataset;

[0069] (4) Perform weight calculation on the standardized multi-source data set to obtain the data source weight coefficient;

[0070] (5) Based on the data source weight coefficient, the standardized multi-source data set is weighted fused to obtain a fused data set;

[0071] (6) Performing data completion processing on the fused data set to obtain a complete fused data set;

[0072] (7) Perform cross-validation grouping processing on the complete fusion data set to obtain a validation data subset;

[0073] (8) performing error estimation processing on the validation data subset to obtain a validation error estimate;

[0074] (9) Based on the verification error estimate, the complete fusion data set is scored for consistency to obtain a data consistency score;

[0075] (10) The data consistency score is evaluated comprehensively by multiple factors to obtain the initial data quality score.

[0076] Specifically, a time-space matching process is performed to obtain a multi-source data correspondence table. Time-space matching process refers to matching data from different sources according to time and space positions. The key to this step is to ensure that the observation data from different data sources are compared and analyzed at the same time and space positions. Through time-space matching, a correspondence table of multi-source data can be established to record the observation values ​​of each data source at the same time and space positions. This process usually requires high-precision timestamps and location information to ensure the accuracy of the match. Time series alignment is performed on the multi-source data correspondence table to obtain an aligned multi-source data set. Time series alignment refers to aligning data from different data sources to a unified time axis for synchronous analysis. Since the sampling frequency and timestamp format of different data sources may be different, interpolation and completion are required to ensure that there are complete data records at each time point. Common methods for time series alignment include linear interpolation, spline interpolation, etc., through which observation data at different times can be aligned to a unified time axis.

[0077] Data type standardization is performed on the aligned multi-source datasets to obtain standardized multi-source datasets. Data type standardization refers to converting data from different data sources into consistent units and formats for comparison and fusion. For example, temperature data in different units (Celsius and Fahrenheit) are uniformly converted to Celsius, and timestamps in different formats are unified into ISO 8601 format. Through standardization, it can be ensured that all data has consistent measurement units and formats for further analysis and processing.

[0078] The standardized multi-source data set is weighted and processed to obtain the data source weight coefficient. The weight calculation is based on the reliability and accuracy of each data source, and different weight coefficients are assigned to each data source. The calculation of the weight coefficient can be based on the accuracy evaluation of historical data, the quality report of the data source, etc., through which the importance of each data source in the fusion process can be determined. For example, for sensors with high historical data quality, the weight coefficient can be larger, while for data sources with poor quality, the weight coefficient is smaller. Based on the data source weight coefficient, the standardized multi-source data set is weighted fused to obtain a fused data set. Weighted fusion refers to weighted averaging of data from different data sources according to the calculated weight coefficient to obtain fused data. The key to this process is to reasonably weight the data of each data source according to the weight coefficient to improve the accuracy and consistency of the fused data. Common methods of weighted fusion include weighted average, weighted median, etc., through which data from different data sources can be fused into a comprehensive observation value.

[0079] Perform data completion on the fused data set to obtain a complete fused data set. Data completion refers to filling in missing data to ensure data integrity. Common data completion methods include interpolation and regression, which can be used to infer missing data values ​​based on existing data, thereby ensuring data integrity and continuity. For example, through the linear interpolation method, the data value of the missing time point can be inferred based on the data of adjacent time points, thereby completing the data.

[0080] The complete fusion data set is processed by cross-validation grouping to obtain validation data subsets. Cross-validation refers to dividing the data set into multiple subsets, which are used for model training and validation respectively to evaluate the performance of the model. Common cross-validation methods include k-fold cross-validation and leave-one-out method, which can effectively evaluate the generalization ability and prediction accuracy of the model. Through cross-validation grouping, the complete fusion data set can be divided into multiple validation data subsets for subsequent error estimation and consistency scoring.

[0081] Perform error estimation on the validation data subset to obtain the validation error estimate. Error estimation refers to calculating the difference between the model prediction value and the actual observation value to evaluate the accuracy of the model. Common error estimation methods include mean square error, absolute error, etc., which can quantify the error of model prediction and thus evaluate the performance of the model. For example, by calculating the mean square error, the average square error between the model prediction value and the actual observation value can be obtained to evaluate the accuracy of the model.

[0082] Based on the validation error estimate, the complete fusion data set is processed for consistency scoring to obtain a data consistency score. Consistency scoring refers to scoring based on the consistency and accuracy of the data to evaluate the quality of the data. Common consistency scoring methods include consistency index, Pearson correlation coefficient, etc., which can quantify the consistency of the data and thus evaluate the quality of the data. For example, by calculating the Pearson correlation coefficient, the linear correlation between observations from different data sources can be quantified to evaluate the consistency of the data.

[0083] The data consistency score is subjected to a multi-factor comprehensive evaluation to obtain an initial data quality score. Comprehensive evaluation refers to evaluating data quality based on multiple factors to obtain a comprehensive quality score. Commonly used comprehensive evaluation methods include weighted average and multi-criteria decision making, which can combine multiple evaluation factors to obtain a comprehensive quality score. For example, through the weighted average method, multiple evaluation factors such as consistency score and error estimate can be weighted averaged to obtain a comprehensive initial data quality score.

[0084] For example, suppose that during a buoy observation, multi-source ocean environment parameter data, including temperature, salinity, and pH value, are collected. First, these data are processed in time and space, and a multi-source data correspondence table is established to match the data at the same time and space position. Then, the data are aligned in time series by linear interpolation to ensure that all data have complete observation records on a unified time axis. Then, the data type of the aligned data is standardized, all temperature data are uniformly converted to degrees Celsius, and the timestamp format is unified to ISO 8601 format. Next, the weight coefficient of the data source is calculated based on the quality assessment results of the historical data, and the standardized data is weighted fused to obtain a fused data set. Then, the interpolation method is used to complete the fused data set to fill in the missing data points and obtain a complete fused data set. The complete fused data set is divided into multiple validation data subsets by the k-fold cross-validation method, and the error estimate of each subset is calculated. Based on these error estimates, the data is scored for consistency to quantify the consistency and accuracy of the data. Finally, the data consistency score is evaluated comprehensively by multiple factors, and the initial data quality score is obtained by combining multiple factors such as consistency score and error estimate. For example, assuming that in a period of time, the temperature data observed by the buoy is processed through the above steps, the initial data quality score is 0.95, indicating that the data accuracy and consistency are high.

[0085] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0086] (1) Perform feature extraction on the anomaly detection report to obtain an abnormal data feature vector;

[0087] (2) Normalizing the initial data quality scores to obtain standardized quality scores;

[0088] (3) Based on the standardized quality score, the abnormal data feature vector is weighted to obtain a weighted feature vector;

[0089] (4) Performing data enhancement processing on the weighted feature vector to obtain an expanded training data set;

[0090] (5) Divide the expanded training data set into a training subset and a validation subset;

[0091] (6) Performing model training on the training subset using a gradient boosting tree model to obtain an initial revised model;

[0092] (7) Perform cross-validation on the validation subset to obtain model performance evaluation indicators;

[0093] (8) Based on the model performance evaluation index, the parameters of the initial correction model are optimized to obtain the optimized correction model;

[0094] (9) Performing data correction processing on the anomaly detection report by optimizing the correction model to obtain a corrected data set;

[0095] (10) Bootstrap resampling is performed on the corrected data set to obtain the uncertainty estimation data corresponding to the corrected data set.

[0096] Specifically, feature extraction is performed on the anomaly detection report. This step involves extracting key features such as salinity drift rate, temperature change, depth information, etc. Dimensionality reduction techniques such as principal component analysis (PCA) or autoencoders are used to compress high-dimensional features into low-dimensional representations to obtain abnormal data feature vectors. The initial data quality scores are normalized. Here, min-max normalization or z-score normalization methods are used to unify quality scores of different scales into the same range for subsequent processing. For example, the quality score is mapped to the 0-1 interval to obtain a standardized quality score.

[0097] Based on the standardized quality score, the feature vectors of abnormal data are weighted. This step aims to highlight the importance of high-quality data while reducing the impact of low-quality data. The weights are calculated using the softmax function, and then the weights are multiplied by the feature vector to obtain the weighted feature vector. Data augmentation of the weighted feature vector is a key step in expanding the training set. New samples are generated using techniques such as random noise addition, data rotation, and scale transformation. This not only increases the amount of training data, but also improves the generalization ability of the model. Through these enhancement techniques, an expanded training data set is obtained.

[0098] The expanded training data set is divided into a training subset and a validation subset, usually using random sampling or stratified sampling. The common division ratio is 8:2 or 7:3 to balance the needs of model training and validation.

[0099] The training subset is processed by model training using the gradient boosting tree model. Gradient boosting tree is an ensemble learning method that builds a strong learner by iteratively training multiple weak learners (decision trees). Using implementations such as XGBoost or LightGBM, the model parameters are optimized by minimizing the loss function (such as mean square error) to obtain the initial revised model. The validation subset is cross-validated to evaluate the generalization ability of the model. A common method is k-fold cross-validation, such as 5-fold or 10-fold. Various performance indicators are calculated, such as root mean square error (RMSE), coefficient of determination (R 2 ) and so on, and obtain the model performance evaluation index.

[0100] Based on the model performance evaluation indicators, the parameters of the initial correction model are optimized. The hyperparameters of the model, such as the depth of the tree, the learning rate, etc., are adjusted by grid search, random search, or Bayesian optimization. Through iterative optimization, the best parameter combination is found to obtain the optimized correction model. The anomaly detection report is corrected by the optimized correction model. The new anomaly data is input into the model, the correction value is predicted, and the original value is replaced by the correction value. This process may require multiple iterations until the preset convergence condition is reached, and the corrected data set is finally obtained.

[0101] Finally, the corrected data set is subjected to Bootstrap resampling to estimate the uncertainty of the corrected results. The Bootstrap method generates multiple resampled data sets through sampling with replacement. The correction model is applied to each resampled data set, and statistics such as mean and confidence interval are calculated to obtain the uncertainty estimation data corresponding to the corrected data set.

[0102] For example, in the correction of salinity drift of buoy observation data in a certain sea area, 10 key features were first extracted from the anomaly detection report, including salinity drift rate, temperature, pressure, etc. These 10 features were compressed into 3 principal components through PCA, explaining 95% of the variance. The initial data quality score range was 0-100, and was mapped to the 0-1 interval after min-max normalization. The weight was calculated using the softmax function, and the weight of high-quality data (score>0.8) was about 4 times that of low-quality data (score<0.2). Data enhancement processing expanded the original 1000 records to 5000, including methods such as adding Gaussian noise and random rotation. The data set was divided into training set and validation set in a ratio of 8:2. The initial model was trained using the XGBoost algorithm, with the maximum tree depth set to 6 and the learning rate set to 0.1. The 5-fold cross validation results showed that the RMSE of the initial model was 0.05PSU (practical salinity unit). By optimizing the parameters through grid search, the tree depth was adjusted to 8, the learning rate was reduced to 0.05, and the optimized RMSE was reduced to 0.03PSU. The optimized model was applied to correct 100 abnormal data points, with an average correction of 0.08PSU. Finally, through 1000 Bootstrap resampling, the 95% confidence interval of the correction value was estimated to be ±0.02PSU.

[0103] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0104] (1) Perform spatial gridding processing on the corrected data set to obtain a gridded data matrix;

[0105] (2) Perform principal component analysis on the gridded data matrix to obtain the spatial main modes;

[0106] (3) Perform wavelet continuity analysis on the corrected data set to obtain the time scale characteristic spectrum;

[0107] (4) Based on the spatial main mode and time scale characteristic spectrum, the modified data set is subjected to spatiotemporal modal decomposition to obtain the spatiotemporal coupling mode;

[0108] (5) Perform singular spectrum analysis on the spatiotemporal coupling pattern to obtain key spatiotemporal variation characteristics;

[0109] (6) Based on the key spatiotemporal variation characteristics, the modified data set is processed for abnormal pattern recognition to obtain abnormal area marking data;

[0110] (7) Performing spatial adaptive kernel density estimation on the abnormal region marking data to obtain a probability distribution map of the abnormal region;

[0111] (8) The temporal evolution analysis of the probability distribution map of the abnormal area is performed through empirical mode decomposition to obtain the spatiotemporal consistency score.

[0112] Specifically, in-depth spatiotemporal analysis of the corrected dataset is a key step in evaluating the correction effect and identifying abnormal patterns. First, the corrected dataset is spatially gridded, a process that interpolates irregularly distributed buoy data onto a regular grid. The observation point data are converted into two-dimensional or three-dimensional grid data using Kriging interpolation or inverse distance weighted method to obtain a gridded data matrix. The gridded data matrix is ​​processed by principal component analysis. Principal component analysis (PCA) is a dimensionality reduction technique used to extract the main variation patterns of data. By calculating the eigenvalues ​​and eigenvectors of the covariance matrix, the data are projected into the principal component space, and several principal components with the largest explained variance are selected as the spatial main modes. These main modes reflect the main spatial variation characteristics of the salinity field.

[0113] Wavelet continuity analysis of the modified data set is an effective method to identify time scale characteristics. Wavelet analysis can provide information in both time and frequency domains. Select appropriate wavelet basis functions (such as Morlet wavelets) to transform the time series and obtain a coefficient matrix on the time-scale plane. By analyzing the distribution of these coefficients, significant periods and variation characteristics on different time scales are identified to form a time scale characteristic spectrum. Based on the spatial main mode and the time scale characteristic spectrum, the modified data set is subjected to spatiotemporal modal decomposition. This step combines information from the spatial and temporal dimensions, using methods such as empirical orthogonal function (EOF) analysis or composite principal component analysis (CPCA). Through this decomposition, a series of spatiotemporal coupling modes are obtained, each of which contains a spatial field and a corresponding time evolution sequence.

[0114] The spatiotemporal coupling patterns are processed by singular spectrum analysis to further extract key spatiotemporal variation features. Singular spectrum analysis (SSA) is a non-parametric time series analysis method that can separate trends, cycles, and noise components. SSA is applied to the time series of each spatiotemporal coupling pattern to identify the main oscillation modes and long-term trends, and obtain key spatiotemporal variation features. Based on the key spatiotemporal variation features, the modified data set is processed for abnormal pattern recognition. A threshold (such as 3 times the standard deviation) is set to mark data that exceeds the normal range as abnormal. This process takes into account the continuity of space and time to avoid misjudging isolated abnormal points as systematic anomalies. In this way, abnormal area labeled data is obtained.

[0115] The abnormal area marked data is processed by spatial adaptive kernel density estimation, which aims to quantify the spatial distribution characteristics of the abnormality. Kernel density estimation is a non-parametric method that can smoothly describe the spatial distribution of point patterns. Adaptive bandwidth is used to use a larger bandwidth in data sparse areas and a smaller bandwidth in data dense areas, so as to obtain a more accurate probability distribution map of the abnormal area.

[0116] Finally, the time evolution analysis of the probability distribution map of the abnormal area is performed through empirical mode decomposition. Empirical mode decomposition (EMD) is an adaptive signal decomposition method that can handle nonlinear and non-stationary time series. The change of the probability distribution of the abnormal area over time is regarded as a time series, which is decomposed into multiple intrinsic mode functions (IMFs) and a residual term through EMD. The characteristics of these IMFs are analyzed, the temporal consistency of the abnormal distribution is evaluated, and the spatiotemporal consistency score is obtained.

[0117] For example, in the correction of salinity drift of buoy observation data in a certain sea area, the corrected data of 100 buoys were first interpolated onto a regular grid of 0.1°×0.1° to obtain a 360×180 gridded data matrix. Through PCA analysis, it was found that the first three principal components explained 85% of the total variance, and these principal components represented the main spatial variation patterns. Wavelet analysis used Morlet wavelet to identify significant annual and semi-annual cycle signals. Spatiotemporal modal decomposition obtained five main spatiotemporal coupling modes, of which the first mode explained 40% of the variance, showing large-scale seasonal changes. Singular spectrum analysis extracted long-term trends and short-term oscillations of 2-7 days from these modes. Abnormal pattern recognition set 3.5 times the standard deviation as the threshold, marking about 2% of the data points as abnormal. Spatial kernel density estimation used adaptive bandwidth, with the bandwidth set to 10km in the nearshore area and increased to 50km in the open sea, generating an abnormal probability distribution map. EMD decomposition decomposes the time evolution of the abnormal distribution into four IMFs and a residual trend term. By analyzing the energy distribution and phase consistency of these IMFs, the spatiotemporal consistency score was calculated to be 0.85 (out of a full score of 1), indicating that the corrected data have good consistency in space and time.

[0118] In a specific embodiment, the process of executing step S106 may specifically include the following steps:

[0119] (1) Perform piecewise linear fitting on the spatiotemporal consistency score to obtain the score trend function;

[0120] (2) Based on the scoring trend function, the classification threshold calculation is performed on the abnormal area marked data to obtain a multi-level quality threshold;

[0121] (3) Perform fuzzy clustering on the abnormal area marked data to obtain data quality level clusters;

[0122] (4) Based on the multi-level quality threshold and data quality level cluster, the abnormal area marked data is graded and mapped to obtain the target data quality level identification;

[0123] (5) Performing statistical distribution analysis on the target data quality level identifier to obtain quality distribution characteristics;

[0124] (6) Based on the quality distribution characteristics, the abnormal pattern extraction processing is performed on the abnormal area marking data to obtain a typical abnormal pattern library;

[0125] (7) Perform causal relationship analysis on the typical abnormal pattern library to obtain abnormal cause inference results;

[0126] (8) Based on the abnormal cause inference results, quality improvement suggestions are generated for the abnormal area marked data to obtain a target quality report;

[0127] (9) Perform strategy optimization processing on the target quality report to obtain the target quality control strategy.

[0128] Specifically, the spatiotemporal consistency score is processed by piecewise linear fitting. This process uses a piecewise linear regression algorithm to divide the score sequence into multiple time periods, and each period is fitted with a linear function. The fitting parameters are optimized by the least squares method to obtain a score trend function, which reflects the trend of data quality over time. Based on the score trend function, the classification threshold calculation is performed on the abnormal area marking data. Using the k-means clustering algorithm, the score trend function is divided into multiple levels, the statistical characteristics of each level (such as mean, standard deviation) are calculated, and multi-level quality thresholds are set. These thresholds are used for subsequent data classification and quality assessment.

[0129] Fuzzy clustering of abnormal area marked data is an important step to consider data uncertainty. The fuzzy c-means (FCM) algorithm is used to assign membership to each data point instead of hard division. This method is more suitable for processing fuzzy boundaries in salinity data and obtaining data quality grade clusters. Based on multi-level quality thresholds and data quality grade clusters, the abnormal area marked data is grade mapped. This step maps each data point to the corresponding quality grade based on its score and membership. The weighted average method is used to comprehensively consider the threshold and cluster membership to obtain the target data quality grade identification.

[0130] Perform statistical distribution analysis on the target data quality level identification, including calculating the proportion of each level, spatial distribution characteristics, etc. Use the kernel density estimation method to draw the probability density function of the quality distribution, identify the quality distribution pattern and abnormal points, and obtain the quality distribution characteristics. Based on the quality distribution characteristics, perform abnormal pattern extraction on the abnormal area marking data. Use dimensionality reduction techniques such as principal component analysis (PCA) or autoencoders to extract key features from high-dimensional abnormal data. Use clustering algorithms (such as DBSCAN) to identify typical abnormal patterns and build a typical abnormal pattern library.

[0131] Perform causal analysis on the typical abnormal pattern library. This step aims to understand the potential causes of the abnormalities. Use Bayesian networks or structural equation models to build a causal relationship diagram between variables. Through conditional probability reasoning or path analysis, the abnormal cause inference results are obtained.

[0132] Based on the inference results of the abnormal cause, the quality improvement suggestions are generated for the abnormal area marked data. Using decision trees or rule-based systems, corresponding improvement suggestions are generated according to different abnormal types and causes. These suggestions include sensor calibration, data processing algorithm adjustment, etc., to form a target quality report.

[0133] Finally, the target quality report is optimized. A reinforcement learning algorithm (such as Q-learning) is used to simulate the quality improvement process as a Markov decision process. Through repeated iterations and optimization, the optimal quality control strategy is found and the target quality control strategy is obtained.

[0134] For example, in the correction of salinity drift of buoy observation data in a certain sea area, a piecewise linear fit was performed on the spatiotemporal consistency score of one year, and the data was divided into four seasonal segments. The fitting function of the summer segment was y = -0.002x + 0.95, indicating a slight downward trend in quality. The score was divided into 5 levels using the k-means algorithm, and the quality thresholds obtained were 0.9, 0.85, 0.8, and 0.75, respectively. Fuzzy c-means clustering divided the data into 3 quality clusters, and each data point had a membership to these three clusters. During the level mapping process, a data point with a score of 0.87 had a membership of 0.7 in the high-quality cluster and was finally mapped to the "good" level. Statistical analysis showed that 75% of the data points were of "good" or above quality, but there were clusters of lower quality in the nearshore area. Anomaly pattern extraction identified three typical patterns: sensor drift, biofouling, and sudden pollution. Causal analysis showed that biofouling was 87% correlated with increased water temperature. Quality improvement recommendations included increasing the frequency of sensor cleaning and optimizing the data smoothing algorithm. The final quality control strategy recommended increasing the frequency of buoy maintenance during the summer and implementing a real-time anomaly detection algorithm.

[0135] The above describes the quality control method based on buoy observation data in the embodiment of the present application. The following describes the quality control system based on buoy observation data in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of the quality control system based on buoy observation data includes:

[0136] The inspection module 201 is used to perform format unification and basic value inspection processing on the original ocean environment parameter data collected by the buoy to obtain pre-processed standard format data;

[0137] A detection module 202 is used to perform multi-dimensional anomaly detection processing on the pre-processed standard format data to obtain an anomaly detection report marked with abnormal and suspicious data points;

[0138] Verification module 203, used for performing multi-source data fusion and cross-validation processing on the data in the anomaly detection report to obtain an initial data quality score;

[0139] A correction module 204 is used to perform intelligent correction processing on the anomaly detection report through a gradient boosting tree model based on the initial data quality score to obtain a corrected data set and uncertainty estimation data corresponding to the corrected data set;

[0140] A processing module 205 is used to perform consistency analysis processing on the modified data set in the spatiotemporal dimension based on the uncertainty estimation data to obtain a spatiotemporal consistency score and abnormal area marking data;

[0141] The evaluation module 206 is used to perform comprehensive evaluation processing on the abnormal area marking data based on the spatiotemporal consistency score to obtain a target data quality level identifier and a target quality report, and perform quality control strategy analysis through the target data quality level identifier and the target quality report to obtain a target quality control strategy.

[0142] Through the collaborative cooperation of the above-mentioned components, the standardization and preliminary cleaning of the data are achieved by unifying the format and checking the basic values ​​of the original marine environmental parameter data; secondly, the multi-dimensional anomaly detection method, combined with sliding window, adaptive threshold and time series decomposition and other technologies, can more comprehensively and accurately identify anomalies and suspicious points in the data, significantly improving the accuracy and robustness of anomaly detection; through multi-source data fusion and cross-validation processing, full use is made of data information from different sources, effectively reducing the deviation and uncertainty that may be caused by a single data source, and improving the reliability of data quality assessment; the gradient boosting tree model is used for intelligent correction processing, which can not only adaptively learn the inherent laws of the data It can also generate corrected data sets and their corresponding uncertainty estimates, providing reliable quality assurance for subsequent data applications; at the same time, through consistency analysis in the spatiotemporal dimensions, combined with advanced technologies such as principal component analysis, wavelet continuity analysis and singular spectrum analysis, it realizes in-depth mining of the spatiotemporal coupling characteristics of the data, and effectively identifies abnormal areas and spatiotemporal change patterns; finally, through comprehensive evaluation and processing, the target data quality level identification and target quality report are obtained, and quality control strategy analysis is carried out based on this, which significantly improves the reliability and availability of buoy observation data, provides high-quality data support for marine scientific research, environmental monitoring, climate change research and other fields, and improves the accuracy of quality control of buoy observation data.

[0143] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A quality control method based on buoy observation data, characterized in that: The quality control method based on buoy observation data includes: The original ocean environment parameter data collected by the buoy is formatted and basic numerical inspection is performed to obtain pre-processed standard format data; Performing multi-dimensional anomaly detection processing on the pre-processed standard format data to obtain an anomaly detection report marked with abnormal and suspicious data points; Perform multi-source data fusion and cross-validation processing on the data in the anomaly detection report to obtain an initial data quality score; Based on the initial data quality score, the anomaly detection report is intelligently corrected by a gradient boosting tree model to obtain a corrected data set and uncertainty estimation data corresponding to the corrected data set; Based on the uncertainty estimation data, performing consistency analysis processing on the modified data set in the spatiotemporal dimension to obtain a spatiotemporal consistency score and abnormal area marking data; Based on the spatiotemporal consistency score, the abnormal area marking data is comprehensively evaluated and processed to obtain a target data quality level identifier and a target quality report, and a quality control strategy analysis is performed through the target data quality level identifier and the target quality report to obtain a target quality control strategy.

2. The quality control method based on buoy observation data according to claim 1 is characterized in that: The raw ocean environment parameter data collected by the buoy is subjected to format unification and basic numerical value inspection processing to obtain pre-processed standard format data, including: Performing data type identification processing on the original ocean environment parameter data to obtain parameter type identification data; Based on the parameter type identification data, the original ocean environment parameter data is converted into a data format to obtain unified format data; Performing physical range threshold check processing on the unified format data to obtain preliminary screening data; Performing high-frequency noise removal processing on the preliminary screening data by using a wavelet transform algorithm to obtain noise-reduced data; Performing time series alignment and interpolation processing on the noise reduction data to obtain time-aligned data; The time-aligned data is subjected to probability distribution analysis processing by an adaptive kernel density estimation algorithm to obtain the pre-processed standard format data.

3. The quality control method based on buoy observation data according to claim 1 is characterized in that: The performing multi-dimensional anomaly detection processing on the pre-processed standard format data to obtain an anomaly detection report marked with abnormal and suspicious data points includes: Performing sliding window segmentation processing on the preprocessed standard format data to obtain a time window data set; Performing dynamic threshold calculation processing on the time window data set by an adaptive threshold algorithm to obtain parameter threshold range data; Based on the parameter threshold range data, performing outlier marking processing on the time window data set to obtain preliminary outlier marking data; Performing time series decomposition processing on the preliminary abnormality mark data by using a seasonal decomposition algorithm to obtain trend component data, seasonal component data and residual component data; Performing local anomaly factor calculation processing on the residual component data to obtain local anomaly score data; Based on the trend component data and the seasonal component data, a probability density estimation process is performed on the local abnormal score data by a multivariate Gaussian mixture model to obtain abnormal probability distribution data; Based on the abnormal probability distribution data, cluster analysis is performed on the local abnormal score data to obtain an abnormal type classification result; The abnormality type classification result is subjected to abnormality cause analysis processing to obtain the abnormality detection report marked with abnormal and suspicious data points.

4. The quality control method based on buoy observation data according to claim 1 is characterized in that: The data in the anomaly detection report is subjected to multi-source data fusion and cross-validation processing to obtain an initial data quality score, including: Performing time-space matching processing on the data in the anomaly detection report to obtain a multi-source data correspondence table; Performing time series alignment processing on the multi-source data correspondence table to obtain an aligned multi-source data set; Performing data type standardization processing on the aligned multi-source data set to obtain a standardized multi-source data set; Performing weight calculation processing on the standardized multi-source data set to obtain a data source weight coefficient; Based on the data source weight coefficients, weighted fusion processing is performed on the standardized multi-source data set to obtain a fused data set; Performing data completion processing on the fused data set to obtain a complete fused data set; Performing cross-validation grouping processing on the complete fusion data set to obtain a validation data subset; Performing error estimation processing on the verification data subset to obtain a verification error estimate value; Based on the verification error estimate, performing consistency scoring processing on the complete fusion data set to obtain a data consistency score; The data consistency score is subjected to a multi-factor comprehensive evaluation process to obtain the initial data quality score.

5. The quality control method based on buoy observation data according to claim 1 is characterized in that: Based on the initial data quality score, the abnormal detection report is intelligently corrected by using a gradient boosting tree model to obtain a corrected data set and uncertainty estimation data corresponding to the corrected data set, including: Performing feature extraction processing on the anomaly detection report to obtain an abnormal data feature vector; Normalizing the initial data quality score to obtain a standardized quality score; Based on the standardized quality score, weighting the abnormal data feature vector to obtain a weighted feature vector; Performing data enhancement processing on the weighted feature vector to obtain an expanded training data set; The expanded training data set is divided into a training subset and a validation subset; Performing model training processing on the training subset by using a gradient boosting tree model to obtain an initial revised model; Performing cross-validation processing on the validation subset to obtain a model performance evaluation index; Based on the model performance evaluation index, the initial revised model is subjected to parameter optimization processing to obtain an optimized revised model; Performing data correction processing on the abnormal detection report by using the optimization correction model to obtain the corrected data set; Bootstrap resampling is performed on the corrected data set to obtain uncertainty estimation data corresponding to the corrected data set.

6. The quality control method based on buoy observation data according to claim 1 is characterized in that: The step of performing consistency analysis on the modified data set in the spatiotemporal dimension based on the uncertainty estimation data to obtain a spatiotemporal consistency score and abnormal region marking data includes: Performing spatial gridding processing on the corrected data set to obtain a gridded data matrix; Performing principal component analysis on the gridded data matrix to obtain spatial main modes; Performing wavelet continuity analysis on the corrected data set to obtain a time scale characteristic spectrum; Based on the spatial main mode and the time scale characteristic spectrum, performing space-time modal decomposition processing on the corrected data set to obtain a space-time coupling mode; Performing singular spectrum analysis on the spatiotemporal coupling mode to obtain key spatiotemporal variation characteristics; Based on the key spatiotemporal variation characteristics, performing abnormal pattern recognition processing on the corrected data set to obtain the abnormal region marking data; Performing spatial adaptive kernel density estimation processing on the abnormal region marking data to obtain a probability distribution map of the abnormal region; The time evolution analysis and processing of the probability distribution map of the abnormal area is performed through empirical mode decomposition to obtain the time-space consistency score.

7. The quality control method based on buoy observation data according to claim 1 is characterized in that: Based on the spatiotemporal consistency score, the abnormal region marked data is comprehensively evaluated and processed to obtain a target data quality level identifier and a target quality report, and a quality control strategy is analyzed through the target data quality level identifier and the target quality report to obtain a target quality control strategy, including: Performing piecewise linear fitting processing on the spatiotemporal consistency score to obtain a score trend function; Based on the scoring trend function, the abnormal area marking data is subjected to classification threshold calculation processing to obtain a multi-level quality threshold; Performing fuzzy clustering processing on the abnormal area marking data to obtain data quality level clusters; Based on the multi-level quality threshold and the data quality level cluster, the abnormal area marking data is subjected to level mapping processing to obtain the target data quality level identifier; Performing statistical distribution analysis on the target data quality level identifier to obtain quality distribution characteristics; Based on the quality distribution characteristics, performing abnormal pattern extraction processing on the abnormal region marking data to obtain a typical abnormal pattern library; Performing causal relationship analysis on the typical abnormal pattern library to obtain abnormal cause inference results; Based on the abnormal cause inference result, the abnormal area marking data is processed to generate quality improvement suggestions to obtain the target quality report; The target quality report is subjected to strategy optimization processing to obtain the target quality control strategy.

8. A quality control system based on buoy observation data, used to execute the quality control method based on buoy observation data as claimed in any one of claims 1 to 7, characterized in that: The quality control system based on buoy observation data includes: The inspection module is used to unify the format and perform basic numerical inspection on the original ocean environment parameter data collected by the buoy to obtain the pre-processed standard format data; A detection module, used to perform multi-dimensional anomaly detection processing on the pre-processed standard format data to obtain an anomaly detection report marked with abnormal and suspicious data points; A verification module, used to perform multi-source data fusion and cross-validation processing on the data in the anomaly detection report to obtain an initial data quality score; A correction module, used to perform intelligent correction processing on the anomaly detection report through a gradient boosting tree model based on the initial data quality score, to obtain a corrected data set and uncertainty estimation data corresponding to the corrected data set; A processing module, configured to perform consistency analysis processing on the modified data set in the spatiotemporal dimension based on the uncertainty estimation data, to obtain a spatiotemporal consistency score and abnormal region marking data; An evaluation module is used to perform comprehensive evaluation processing on the abnormal area marking data based on the spatiotemporal consistency score to obtain a target data quality level identifier and a target quality report, and to perform quality control strategy analysis through the target data quality level identifier and the target quality report to obtain a target quality control strategy.

Images