Intelligent quality control system and method for ocean survey data

Abstract

The invention relates to the technical field of ocean observation, in particular to an intelligent quality control system and method for ocean survey data, which takes whole-process data quality closed-loop management and control as a core, proposes an element-standard-process three-dimensional quality control framework, constructs a real-time and timing dual-mode architecture, and integrates a whole-chain visualization and non-intervention automation technology. By extracting more than 40 differentiated quality control elements of more than 10 types of data, a four-layer quantitative standard of equipment state-parameter setting-data integrity-operation compliance is established, and closed-loop control of the whole process is standardized. The error condition detection rate of the system reaches 89% or above, the manual quality control time cost is reduced by 90% or above, the quality control time efficiency is compressed to the real-time / timing hour level, paradigm innovation from post-event error correction to real-time intelligent management and control is achieved, core support is provided for high-quality collection and high-reliability application of ocean survey data, and the method is suitable for various ocean comprehensive scientific investigation scenes.

Description

Technical Field

[0001] This invention relates to the field of marine observation technology, specifically to an intelligent quality control system and method for marine survey data. Background Technology

[0002] Currently, comprehensive marine scientific research has become a core support for exploring the mysteries of the ocean, developing marine resources, and protecting the marine environment. The quality of scientific research data directly determines the scientific validity of research conclusions, the accuracy of resource assessments, and the rationality of marine management decisions. Marine survey work is significantly complex, involving multiple professional fields such as navigation and positioning, hydrometeorology, geology, and geophysics. The data types cover more than 10 categories, including CTD, ADCP, and multibeam sonar. The data formats of different equipment exhibit heterogeneity, and the quality control indicators of various types of data differ significantly, which brings many natural challenges to data quality control.

[0003] Traditional marine survey data quality control models, centered on centralized post-voyage verification, primarily rely on manual sampling for data validation. This model suffers from three major unresolved problems. First, quality control is significantly delayed; data deviations cannot be detected promptly at the field, easily leading to their propagation and spread in subsequent data processing and application, ultimately affecting the reliability of the overall scientific research results. Second, standards are severely fragmented; the lack of unified specifications for quality control indicators and processes across different data types and voyages results in poor comparability across voyages and data types, hindering the formation of systematic scientific research data outcomes. Third, high reliance on manual labor not only leads to low quality control efficiency, failing to meet the data processing needs of large-scale marine scientific expeditions, but also easily introduces errors due to subjective human judgment and the limitations of random sampling, further reducing the accuracy of data quality control results.

[0004] In existing technologies, quality control methods for marine observation data mostly focus on single data types or specific processing stages, lacking a multi-dimensional closed-loop management mechanism covering all data types and all process stages. Furthermore, existing methods have low levels of intelligence, making it difficult to meet the high-efficiency quality control needs of multi-source heterogeneous data, and failing to achieve standardization, automation, and real-time data quality control. Therefore, developing an intelligent quality control system and method that can cover all data types and all process stages, while possessing both real-time and automation characteristics, is crucial to breaking through industry technical bottlenecks and solving the pain points of traditional quality control models. This is of great significance for promoting the upgrading of marine scientific research data processing technology and ensuring the high-quality conduct of marine scientific research. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent quality control system and method for marine survey data, so as to solve the bottleneck problems mentioned in the background art, such as the large professional scope of traditional quality control, the variety of error types, the strong reliance on manual labor, and the lag in quality control.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A marine survey data intelligent quality control system includes a three-dimensional quality control framework module, a dual-mode quality control engine, a full-chain visualization module, and a non-interventional automation module. These modules work together to achieve closed-loop management of data quality throughout the entire process. The core technical architecture is as follows: The three-dimensional quality control framework module constructs a three-dimensional technical system of elements, standards, and processes. The element layer extracts more than 40 differentiated key quality control elements from more than 10 types of core survey data, such as navigation and positioning, CTD, and ADCP. The standard layer establishes four-layer quantitative quality control standards for equipment status, parameter settings, data integrity, and operational compliance. The process layer realizes closed-loop management of the entire process of data collection, transmission, parsing, and verification. The dual-mode quality control engine includes a real-time quality control module and a timed quality control module. The real-time quality control module realizes immediate early warning of anomalies based on the dynamic mapping technology between raw data and the work site. The timed quality control module completes data backup, quality control analysis, and report generation through an automated processing engine. The full-chain visualization module enables equipment lifecycle information traceability, dynamic monitoring of the operation process, and visualization of multi-dimensional quality control graphics, covering core modules such as CTD operation process monitoring and dual-probe temperature and salinity difference monitoring. The automated, non-intervention-based module, based on a self-developed intelligent raw data parsing algorithm, is compatible with multi-format and multi-device data, enabling fully automated operation of the entire process from access to processing, analysis, early warning, and reporting. Its core algorithm includes a timestamp alignment algorithm. ; Where t i Let t be the timestamp of the i-th type of data, t0 be the base timestamp, and Δt be the timestamp of the i-th type of data. th The alignment threshold is set to ≤1s.

[0007] Preferably, the key quality control elements are weighted using the entropy weight method to ensure accurate coverage of key quality control areas. The weight calculation model is as follows: (1) Calculate the entropy value of the j-th quality control element: ; in , Let be the standardized value of the j-th element in the i-th data category, and n be the number of data types. (2) Calculate the weight of the j-th quality control element: ; Where m is the total number of quality control elements, and the weight distribution satisfies ; (3) The core quality control elements include the number of navigation and positioning satellites, differential signal quality, CTD calibration information, effective station information, ADCP good data rate, vertical velocity error, ocean gravity intersection closure error, etc., and the element weight difference of each type of data is ≥15%.

[0008] Preferably, the four-layer quantitative quality control standard system is a multi-dimensional quantifiable standard, and the four-layer quantitative quality control standard system is as follows: 1) Equipment Status Layer: Using a calibration validity period of ≤1 year as the core quantitative rule, verify the effectiveness of equipment operation; 2) Parameter setting layer: Define reasonable thresholds for equipment parameters and verify the compliance of parameter configuration; 3) Data Integrity Layer: Missing Rate ≤3%, percentage of valid data ≥95%, of which This represents the number of missing data entries. The number of valid data entries. This represents the total number of data entries. 4) Operational compliance layer: Standardize operational process requirements such as the depth of equipment deployment and data time matching.

[0009] Preferably, the dual-mode quality control engine employs the following core algorithm: 1) The real-time quality control module is based on the dynamic mapping technology of raw data and work site, which captures key information in real time and reproduces the work scene to realize the immediate identification of anomalies; 2) The timed quality control module completes core quality control tasks according to a preset cycle through an automated processing engine; 3) Anomaly detection employs an improved 3σ algorithm: |x i -μ|>k·σ, where μ is the data mean, σ is the standard deviation, and k is the adaptive coefficient, ranging from 1.8 to 3.2, dynamically adjusted according to the data type. Abnormal data is automatically associated with the contingency plan after being marked.

[0010] Preferably, the core implementation algorithm of the full-chain visualization module is the dual-probe TS curve comparison algorithm: ; in , These are the TS curve functions for dual probes, where L is the curve length, and C≥0.95 indicates data consistency. Simultaneously, this module can extract core information such as sensor serial number and calibration date, and track operational information in real time, including station latitude and longitude, operation duration, and deployment depth, presenting data quality status through multi-dimensional graphics.

[0011] On the other hand, this invention also provides an intelligent quality control method for marine survey data. Based on the core architecture of the aforementioned intelligent quality control system for marine survey data, which includes a three-dimensional quality control framework module, a dual-mode quality control engine, a full-chain visualization module, and a non-interventional automation module, it achieves closed-loop management of data quality throughout the entire process. Specifically, it includes the following steps: S1: Relying on the system's three-dimensional quality control framework module, over 40 key quality control elements are extracted and weighted using the entropy weight method. Quality control rules based on a four-layer quantitative standard system of equipment status, parameter settings, data integrity, and operational compliance are established using quantitative formulas. Utilizing the system's independently developed unified data parsing technology, standardized access to multiple data formats such as .ENR and .XSE is achieved, and the parsing adaptation function satisfies F... parse (F raw )=F standard , where F raw For the original data format, F standard The data is in a standard format, with a parsing success rate of ≥99.5%. S2: The system's real-time + timed dual-mode quality control engine is activated. The real-time quality control module captures equipment operating parameters and data transmission status through dynamic mapping technology, and instantly identifies equipment and data anomalies based on the improved 3σ algorithm, with an anomaly warning response time of ≤5 minutes. The timed quality control module automatically completes data backup, multi-dimensional quality control analysis, and standardized report generation according to an adaptive cycle. S3: Multi-dimensional verification is conducted through the system's full-chain visualization module, equipment status is verified using the equipment lifecycle information traceability module, operational standardization is controlled through the dynamic monitoring module of the operation process, and data quality consistency is presented using the dual-probe TS curve comparison algorithm. The multi-dimensional verification results meet the following requirements: ; Where w i For the weights of each dimension, Q i Quality control scores for each dimension.

[0012] S4: Based on the system's non-interventional automated module, it autonomously completes the entire process of access, processing, analysis, early warning, and reporting, outputting a standardized quality control report that annotates the anomaly type, location, and suggested handling. The correction rate after anomaly handling is as follows: ; Where N corrected To correct the number of successfully corrected abnormal data entries, N abnormal Total number of abnormal data entries Preferably, the unified data parsing technology described in step S1 is compatible with multiple original data formats such as .ENR and .XSE, and achieves accurate identification of more than 10 types of data formats through feature field matching, with an accuracy rate of ≥99.8%. The core standardized conversion formula is: ; Where D extract To extract data, D min D max U represents the extreme values ​​of the original data, and L represents the upper and lower limits of the standard data. The normalization error of the converted data is ≤0.1%.

[0013] Preferably, the anomaly identification in step S2 employs a hybrid algorithm combining a statistical model and a rule engine, with the core fusion judgment formula being: A = α·A stat +(1-α)·A rule Where α is the fusion coefficient, ranging from 0.6 to 0.8, A stat To detect outliers, A rule For rule-related abnormal results, an early warning is triggered when A ≥ 0.7, enabling accurate identification of equipment and data anomalies.

[0014] Preferably, the comprehensive data quality scoring formula in step S3 is: ; Where w1=0.3, w2=0.3, w3=0.2, w4=0.2 are the weights of each dimension, β is the missing data rate, γ is the percentage of valid data, and A abnormal The deviation value represents outlier data, C represents the similarity of the TS curve, and a score Q≥0.9 indicates high-quality data that meets the standards for marine scientific research data application.

[0015] As a preferred option, the core algorithm for the non-interventional automated closed-loop operation in step S4 is a cross-voyage adaptation algorithm: ; Where S current,i For the i-th criterion of the current voyage, S history,i The historical average standard is used. If Adapt ≤ 0.1, the historical configuration is directly reused; otherwise, the parameter threshold is automatically adjusted to ensure the uniformity and reliability of quality control standards across voyages and equipment.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention completely changes the traditional passive mode of centralized manual verification after flight operations by constructing a real-time + timed dual-mode quality control architecture and combining it with a full-process closed-loop management mechanism. The real-time quality control module, based on raw data-worksite dynamic mapping technology, can instantly capture equipment operating parameters and data transmission status, with an anomaly warning response time of ≤5 minutes, supporting on-site personnel to adjust plans immediately. The timed quality control module autonomously completes data backup, quality control analysis, and report generation according to a preset cycle, without manual intervention. Application results show that the system compresses the traditional post-flight quality control cycle of several days to real-time / timed hourly levels. Navigation data can be quality controlled within 3 minutes, SBE21 data within 2 minutes, and large data volumes such as multibeam sonar can also be quality controlled within hours, effectively avoiding the transmission and diffusion of data deviations and providing key technical support for dynamic optimization of on-site operations.

[0017] This invention achieves standardization and precision in the quality control of multi-source heterogeneous data. By extracting over 40 differentiated key quality control elements from more than 10 categories of core survey data, a four-layer quantitative standard system is constructed, encompassing equipment status, parameter settings, data integrity, and operational compliance. This system clarifies quantitative indicators such as a data missing rate of ≤3% and a valid data ratio of ≥95%, solving the problems of fragmentation and incomplete coverage in traditional quality control standards. The system achieves an error detection rate of over 89%, accurately identifying 17 types of anomalies, including abnormal equipment parameters, abnormal sensor status, and operational compliance issues, and supporting on-site handling, thus ensuring data quality from the source. The data anomaly correction rate after quality control is ≥95%, significantly improving data reliability and comparability, and providing a solid data foundation for the scientific validity of research conclusions, the accuracy of resource assessments, and the rationality of marine management decisions.

[0018] This invention integrates key technologies of non-interventional automation and full-chain visualization, significantly reducing reliance on manual labor and enhancing system adaptability. The non-interventional automation module, through a self-developed intelligent raw data parsing algorithm, is compatible with multiple data formats such as .ENR and .XSE, enabling fully automated operation of the data access-processing-analysis-early warning-reporting process without human intervention. This reduces the time cost of manual quality control by over 90%, allowing researchers to focus on equipment maintenance and improving observation accuracy, forming a positive cycle of efficiency improvement and accuracy optimization. Validated through nearly 20 large-scale applications on the Science Satellite, the system is compatible with various mainstream survey equipment such as navigation and positioning, CTD, ADCP, and multibeam sonar, and adapts to diverse scientific research scenarios including marine hydrological surveys and geological and geophysical exploration. It demonstrates strong stability and adaptability under different equipment configurations and operating environments, possessing broad application prospects. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are explained in detail together with the embodiments of the invention, but do not constitute a limitation thereof.

[0020] Figure 1 This is a schematic diagram of the process framework of the scientific research vessel voyage data quality control system of the present invention; Figure 2 This is a schematic diagram of the main quality control graphics for CTD data in this invention; Figure 3 Statistics on the time consumed by the quality control of the voyage data system for voyage KX202506, as per an embodiment of the present invention; Figure 4 This is a statistical table of key quality control elements of core survey data in the embodiments of the invention; Figure 5 The system extracts sensor information for the embodiments of the invention; Figure 6 The system extracts job information for the embodiments of the invention; Figure 7 This is a statistical summary of the Science Rover's voyages and operational anomalies over the years, as part of an embodiment of the invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] Example The intelligent quality control system for marine survey data in this embodiment is built based on actual marine scientific research operation scenarios. The hardware and software configuration meets the requirements for real-time processing, large-scale computing, and remote transmission of multi-source heterogeneous data. The specific configuration is as follows: (a) Hardware configuration Computing server: Equipped with Intel Xeon Gold 6330 CPU, 128GB DDR4 memory, 4TB SSD high-speed storage, supports multi-threaded parallel processing, and can efficiently handle quality control analysis tasks of 388GB of data per flight. Data acquisition terminal: Compatible with standard interfaces (RS485, Ethernet, USB3.0) of various survey equipment such as GPS, CTD, ADCP, and multibeam sonar, enabling direct acquisition of raw data without the need for additional adapters; Network transmission module: Supports Gigabit Ethernet and 5G dual-mode transmission, with a 5G transmission rate of ≥100Mbps, ensuring real-time data interaction between the work site and the server, and a transmission latency of ≤50ms, meeting real-time quality control requirements; Visualization terminal: 27-inch 4K high-definition display screen, supports multi-window simultaneous display of quality control charts, equipment status and operation dynamics, and is suitable for use in multiple scenarios such as the research vessel's bridge and laboratory, making it convenient for on-site personnel to view in real time.

[0023] (II) Software Configuration Operating system: Windows 10 Professional (64-bit), providing a stable operating environment and supporting the operation of professional tools such as MATLAB R2023b; Core development language: MATLAB, integrating libraries for data parsing, algorithm computation, and visualization plotting to ensure efficient execution of core algorithms; Auxiliary tools: Python 3.9 (for data format compatibility and adaptation), SQL Server 2022 (for storing equipment lifecycle information and quality control results) to achieve full-process data traceability; Algorithm Library: Custom-developed element weight calculation library, anomaly recognition algorithm library, and visualization chart generation library, supporting modular calls and parameter configuration to adapt to the quality control needs of different voyages and different equipment.

[0024] Specific implementation procedures (taking the Science Ferry KX202506 voyage as an example) This voyage covers 10 core data categories, including navigation, CTD, 300K ADCP, and multibeam sonar, with a total data volume of 388.9751 GB. The operational scope includes a long-distance cruise of 17,820.9 km and observations at 165 stations. The specific implementation steps are as follows: (I) Step S1: Initialization of the 3D Quality Control Framework, Extraction and Weight Allocation of Quality Control Elements: The system systematically analyzes the technical characteristics of 10 types of core survey data and extracts more than 40 key quality control elements (such as the number of satellites and differential signal quality for navigation and positioning; calibration information and effective station information for CTD, etc.). Entropy weighting is used to assign weights to these elements to ensure accurate coverage of key quality control areas. The specific calculation process is as follows: Four types of core data are selected: navigation and positioning, CTD, ADCP, and multibeam bathymetry. The standardized values ​​x of the three key quality control elements corresponding to each data type are... ij As shown in the table below:

[0025] (1) Calculate the probability pij of each element: Taking element 1 as an example, p 11 =0.85 / (0.85+0.90+0.82+0.78)=0.252, p 21 =0.90 / 3.35=0.269, p 31 =0.82 / 3.35=0.245, p 41=0.78 / 3.35=0.233; (2) Calculate the entropy value of element 1. e1=(-1\ln4)×(0.252×ln0.252+0.269×ln0.269+0.245×ln0.245+0.233×ln0.233)≈0.992; (3) Similarly, calculate the entropy value of element 2 as e2≈0.990 and the entropy value of element 3 as e3≈0.991; (4) Calculate the weights of each element: w1=(1-0.992)÷[(1-0.992)+(1-0.990)+(1-0.991)]≈0.276, w2≈0.010 / 0.029≈0.345, w3≈0.009 / 0.029≈0.310, The weight distribution satisfies Furthermore, the weight difference of each element is ≥15%, thereby achieving differentiated allocation of key quality control areas.

[0026] Four-layer quantitative quality control standard configuration: (1) Equipment status layer: with calibration validity period ≤ 1 year as the core rule, data is entered. Figure 5 The calibration dates of each sensor in the middle, such as the calibration date of the temperature sensor is 20241226, the current operation time is 20250916, the validity period difference is 264 days ≤ 365 days, and the equipment status is determined to be valid; (2) Parameter setting layer: In conjunction with the equipment manual, it is clear that the measurement range of the CTD temperature sensor is -2~35℃, and the allowable deviation is ±0.05℃; the measurement range of ADCP flow velocity is 0~3m / s, and the allowable deviation is ±0.01m / s; (3) Data integrity layer: Calculated using a quantitative formula, taking CTD data as an example: Total number of data entries N total =165 stations × 1000 entries / station = 165000 entries; Number of missing data N missing =4200 entries; The missing rate β = 4200 / 165000 × 100% ≈ 2.55% ≤ 3%; Valid data entries Nvalid = 157,800; The percentage of valid data γ = 157800 / 165000 × 100% ≈ 95.64% ≥ 95%; Meets quality control standards; (4) Operational compliance layer: The CTD standard is implemented at a depth of 1000m, with an allowable deviation of ±2%, i.e., 980~1020m.

[0027] Data parsing and standardized access: Based on the original data formats of various devices (such as .ENR format for CTD and .XSE format for ADCP), matching data parsing functions are called to extract core data fields. This is achieved through a timestamp alignment algorithm. Δt=|ti-t0|≤Δt th Where ti is the timestamp of various data types, t0 is the system base timestamp, and Δt th =1s, achieving standardized access to multi-source heterogeneous data. For example, if the navigation data timestamp is 2025-09-1600:34:32.123 and the CTD data timestamp is 2025-09-1600:34:32.897, Δt=0.774s≤1s, the timestamp alignment is deemed acceptable.

[0028] (II) Step S2: Dual-mode quality control engine operation Real-time quality control mode: The system background captures equipment operating parameters and data transmission status in real time, and reproduces the operation scenario based on the original data-work site dynamic mapping technology. Taking CTD operation as an example, it monitors temperature sensor data, deployment depth, transmission link status, and other information in real time, and generates visual quality control charts. When the data of the second set of conductivity sensors in CTD is detected to be 4.2 S / m, while the historical normal data mean μ=4.0 S / m and standard deviation σ=0.08 S / m, the improved 3σ algorithm |xi-μ|>k·σ (k=2.0, configured according to CTD data type) is used to calculate |4.2-4.0|=0.2>2.0×0.08=0.16, which is judged as data anomaly. The system issues an early warning immediately with a response time of 3 minutes, supporting on-site personnel to adjust sensor parameters.

[0029] Scheduled quality control mode: The preset quality control cycle is 12 hours, and the system automatically checks the data acquisition folder periodically. This voyage triggered 18 scheduled quality control tasks, autonomously completing data backup, multi-dimensional quality control analysis, and report generation. Taking multibeam data as an example, with a data volume of 207.8748GB, the system completed tasks such as effective beam count statistics and water depth data quality analysis through its automated processing engine, with a quality control time of 725 minutes (e.g., Figure 3 As shown in the figure, no manual intervention was required. The total quality control time for all 10 types of data during the entire voyage was 1082 minutes, which is more than 90% less than the traditional manual quality control time of 3 people × 5 days × 24 × 60 minutes = 3600 minutes.

[0030] (III) Step S3: Full-chain visual verification Equipment lifecycle information traceability: system automatically extracts Figure 5Information such as the sensor's serial number and calibration date is displayed through a visual interface. For example, a pressure sensor with serial number 1185, calibration date 20250108, operation time 20250916, and an expiration date difference of 252 days ≤ 365 days indicates that the equipment is operating effectively, and the visual interface marks the equipment as being in good condition.

[0031] Dynamic monitoring of the operation process: Real-time tracking of operation information at each station. Taking station AE-01 as an example, with longitude 128.5° and latitude 19.99933333°, the standard station's longitude and latitude are (128.5°, 20.0°). Spatial deviation is calculated using a spatiotemporal matching algorithm. Although slightly exceeding the 50m threshold, it was deemed within an acceptable range considering the sea conditions in the operating area. However, the deviation in the system-marked station position needs attention. Meanwhile, the monitoring depth of 1001.726m falls within the compliant range of 980~1020m, and the operation duration of 1.006 hours meets the standard operation duration requirements.

[0032] Visualization of quality control charts: The core technology utilizes a dual-probe TS curve comparison algorithm. , in , These are the TS curve functions for the dual probes, where L is the curve length, and C ≥ 0.95 indicates data consistency. This module can also extract core information such as sensor serial number and calibration date, and track real-time operational information such as station latitude and longitude, operation duration, and deployment depth, presenting data quality status through multi-dimensional maps. Numerical integration yields: Then the curve similarity is: ; Once the data from both probes are determined to be consistent, a visual interface generates a TS curve comparison chart, which intuitively presents the data quality status.

[0033] (iv) Step S4: Automated output without intervention The system operates autonomously throughout the entire process: it follows a data access-processing-analysis-early warning-reporting workflow without human intervention. Triggering conditions are real-time data increments ≥100MB or the arrival of scheduled data. During this voyage, a total of 388.9751GB of data was accessed, triggering the workflow 23 times, all of which were completed autonomously.

[0034] Cross-flight adaptation and standardization: A cross-flight adaptation algorithm is adopted. ; Three core criteria (m=3) were selected: missing data rate of equipment calibration validity period and percentage of valid data, with weights w1=0.4, w2=0.3, and w3=0.3.

[0035] The current voyage standard is Scurrent,1 = 365 days, Scurrent,2 = 3%, and Scurrent,3 = 95%; the historical voyage average standard is Shistory,1 = 360 days, Shistory,2 = 2.8%, and Shistory,3 = 95.2%. Calculations show: ; After standardization, the value is ≤0.1, and historical configurations can be directly reused to ensure consistent quality control standards across voyages.

[0036] Quality Control Report Output and Anomaly Handling: The system autonomously generates a quality control report for voyage KX202506, clearly indicating the anomaly type, location, and recommended handling. This voyage detected three types of anomalies (CTD second conductivity sensor drift, Vaisala weather station data anomaly, and SBE21 data date information anomaly), with an anomaly correction rate of: .

[0037] The report was generated in 2.5 minutes and included a comprehensive data quality score of Q=0.92 (high-quality data standard), meeting the requirements for marine scientific research data application.

[0038] In this embodiment, the hardware configuration of the intelligent quality control system for marine survey data includes: a computing server (CPU: Intel Xeon Gold 6330, memory: 128GB, storage: 4TB SSD) mounted on the research vessel, a data acquisition terminal (compatible with interfaces of various survey equipment such as GPS, CTD, ADCP), and a network transmission module (supporting Gigabit Ethernet and 5G dual-mode transmission); the software configuration includes: the operating system is Windows 10, and the development language is MATLAB.

[0039] This implementation takes the Science Exploration KX202506 as an example. This exploratory voyage covers 10 core data categories, including navigation, CTD, and multibeam sonar, with a total data volume of 388.9751 GB. The scope of operations includes long-distance cruise and multi-station observation. Figure 3 The specific process of applying the system of this invention is as follows ( Figure 1 ): S1: Initialization of the 3D quality control framework. The system imports 10 types of data, including navigation and positioning, CTD, and 300K ADCP. The data parsing module is then activated to extract key quality control elements (such as the number of satellites and differential signal quality in navigation data; calibration information and effective station information in CTD; and the number of effective beams and water depth data quality in multibeam bathymetry). Figure 4 Configure a four-layer quality control standard to complete the standardized access configuration for multi-format data.

[0040] S2: Dual-mode quality control operation. Real-time quality control mode captures operating parameters and data transmission status of each device in the background; timed quality control mode automatically completes data backup, quality control analysis, and report generation on a 12-hour cycle. The total system quality control time for this voyage was only 1082 minutes, including 3 minutes for navigation data quality control, 2 minutes for SBE21 data, and 725 minutes for multibeam data quality control. Figure 3 ).

[0041] S3: Full-chain visual verification. Taking CTD operations as an example, the equipment lifecycle module shows that the CTD temperature sensor's serial number is 6534, calibration date is 20241226, and it is within its valid period. Figure 5 The operation monitoring module displays the longitude, latitude, operation duration, and depth of stations AE-01 to AE-11 in real time. Figure 6 The visualizations present CTD pressure change curves, dual-probe temperature difference curves, and salinity difference curves, verifying data consistency. Figure 2 ).

[0042] S4: Uninterrupted automated output. The system autonomously generates the KX202506 cruise quality control report, clearly indicating the anomaly type, location, and handling recommendations for various data. This cruise detected two types of field anomalies: CTD conductivity sensor drift and Vaisala weather station data anomalies, as well as one type of indoor processing anomaly: SBE21 data date information anomaly. Figure 7 After the anomaly handling was completed, the reliability of the data was significantly improved.

[0043] The system of this invention has been continuously applied in nearly 20 expeditions of the Science satellite from 2021 to 2025, and the verification results are as follows: (1) Quality control timeliness: Traditional voyage quality control takes 3-5 days, while this system compresses the quality control timeliness to the real-time / timed hour level. The total quality control time for 10 types of data in voyage KX202506 was only 1082 minutes. Figure 3 This meets the need for immediate adjustments during on-site operations.

[0044] (2) Error detection rate: A total of 17 types of abnormalities were detected, including abnormal equipment parameters, abnormal sensor status, and problems with operational procedures, accounting for 89.5% of the total number of abnormalities. This is more than 60% higher than the detection rate of traditional manual sampling inspection. Figure 7 ).

[0045] (3) Labor costs: The time cost of manual quality control is reduced by more than 90%. The quality control work that traditional voyages require 3 scientific researchers to spend 5 days to complete is now only required by this system to complete the abnormal handling simultaneously during the voyage, which greatly reduces the manpower input.

[0046] (4) Data quality: After system quality control, the data missing rate is ≤3% and the abnormal data correction rate is ≥95%, which meets the standards for marine scientific research data application.

[0047] The advantages of the intelligent quality control system and method for marine survey data proposed in this invention are as follows: This invention completely changes the traditional passive mode of centralized manual verification after flight operations by constructing a real-time + timed dual-mode quality control architecture and combining it with a full-process closed-loop management mechanism. The real-time quality control module, based on raw data-worksite dynamic mapping technology, can instantly capture equipment operating parameters and data transmission status, with an anomaly warning response time of ≤5 minutes, supporting on-site personnel to adjust plans immediately. The timed quality control module autonomously completes data backup, quality control analysis, and report generation according to a preset cycle, without manual intervention. Application results show that the system compresses the traditional post-flight quality control cycle of several days to real-time / timed hourly levels. Navigation data can be quality controlled within 3 minutes, SBE21 data within 2 minutes, and large data volumes such as multibeam sonar can also be quality controlled within hours, effectively avoiding the transmission and diffusion of data deviations and providing key technical support for dynamic optimization of on-site operations.

[0048] This invention achieves standardization and precision in the quality control of multi-source heterogeneous data. By extracting over 40 differentiated key quality control elements from more than 10 categories of core survey data, a four-layer quantitative standard system is constructed, encompassing equipment status, parameter settings, data integrity, and operational compliance. This system clarifies quantitative indicators such as a data missing rate of ≤3% and a valid data ratio of ≥95%, solving the problems of fragmentation and incomplete coverage in traditional quality control standards. The system achieves an error detection rate of over 89%, accurately identifying 17 types of anomalies, including abnormal equipment parameters, abnormal sensor status, and operational compliance issues, and supporting on-site handling, thus ensuring data quality from the source. The data anomaly correction rate after quality control is ≥95%, significantly improving data reliability and comparability, and providing a solid data foundation for the scientific validity of research conclusions, the accuracy of resource assessments, and the rationality of marine management decisions.

[0049] This invention integrates key technologies of non-interventional automation and full-chain visualization, significantly reducing reliance on manual labor and enhancing system adaptability. The non-interventional automation module, through a self-developed intelligent raw data parsing algorithm, is compatible with multiple data formats such as .ENR and .XSE, enabling fully automated operation of the data access-processing-analysis-early warning-reporting process without human intervention. This reduces the time cost of manual quality control by over 90%, allowing researchers to focus on equipment maintenance and improving observation accuracy, forming a positive cycle of efficiency improvement and accuracy optimization. Validated through nearly 20 large-scale applications on the Science Satellite, the system is compatible with various mainstream survey equipment such as navigation and positioning, CTD, ADCP, and multibeam sonar, and adapts to diverse scientific research scenarios including marine hydrological surveys and geological and geophysical exploration. It demonstrates strong stability and adaptability under different equipment configurations and operating environments, possessing broad application prospects.

[0050] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A marine survey data intelligent quality control system, characterized in that, It includes a 3D quality control framework module, a dual-mode quality control engine, a full-chain visualization module, and a non-interventional automation module. These modules work together to achieve closed-loop management of data quality throughout the entire process. The core technical architecture is as follows: The three-dimensional quality control framework module constructs a three-dimensional technical system of elements, standards, and processes. The element layer extracts more than 40 differentiated key quality control elements from core survey data such as navigation and positioning, CTD, and ADCP. The standard layer establishes four-layer quantitative quality control standards for equipment status, parameter settings, data integrity, and operational compliance. The process layer realizes closed-loop management of the entire process of data collection, transmission, parsing, and verification. The dual-mode quality control engine includes a real-time quality control module and a timed quality control module. The real-time quality control module realizes immediate early warning of anomalies based on the dynamic mapping technology between raw data and the work site. The timed quality control module completes data backup, quality control analysis, and report generation through an automated processing engine. The full-chain visualization module enables equipment lifecycle information traceability, dynamic monitoring of the operation process, and visualization of multi-dimensional quality control graphics, covering core modules such as CTD operation process monitoring and dual-probe temperature and salinity difference monitoring. The automated, non-intervention-based module, based on a self-developed intelligent raw data parsing algorithm, is compatible with multi-format and multi-device data, enabling fully automated operation of the entire process from access to processing, analysis, early warning, and reporting without human intervention. Its core algorithm includes a timestamp alignment algorithm. ； Where t i Let t be the timestamp of the i-th type of data, t0 be the base timestamp, and Δt be the timestamp of the i-th type of data. th The alignment threshold is set to ≤1s.

2. The intelligent quality control system for marine survey data according to claim 1, characterized in that, The key quality control elements are weighted using the entropy weight method to ensure accurate coverage of key quality control areas. The weight calculation model is as follows: (1) Calculate the entropy value of the j-th quality control element: ； in , Let be the standardized value of the j-th element in the i-th data category, and n be the number of data types. (2) Calculate the weight of the j-th quality control element: ； Where m is the total number of quality control elements, and the weight distribution satisfies ; (3) The core quality control elements include the number of satellites for navigation and positioning, differential signal quality, CTD calibration information, effective station information, ADCP good data rate, vertical velocity error, ocean gravity intersection closure error, etc., and the element weight difference of each type of data is ≥15%.

3. The intelligent quality control system for marine survey data according to claim 1, characterized in that, The four-layer quantitative quality control standard system is a multi-dimensional quantifiable standard, and the four-layer quantitative quality control standard system is as follows: 1) Equipment Status Layer: Using a calibration validity period of ≤1 year as the core quantitative rule, verify the effectiveness of equipment operation; 2) Parameter setting layer: Define reasonable thresholds for equipment parameters and verify the compliance of parameter configuration; 3) Data Integrity Layer: Missing Rate ≤3%, percentage of valid data ≥95%, of which This represents the number of missing data entries. The number of valid data entries. This represents the total number of data entries. 4) Operational compliance layer: Standardize operational process requirements such as the depth of equipment deployment and data time matching.

4. The intelligent quality control system for marine survey data according to claim 1, characterized in that, The dual-mode quality control engine employs the following core algorithm: 1) The real-time quality control module is based on the raw data-work site dynamic mapping technology, which captures key information in real time and reproduces the work scenario to achieve immediate identification of anomalies; 2) The timed quality control module completes core quality control tasks according to a preset cycle through an automated processing engine; 3) Anomaly detection employs an improved 3σ algorithm: |x i -μ|>k·σ, where μ is the data mean, σ is the standard deviation, and k is the adaptive coefficient, ranging from 1.8 to 3.2, dynamically adjusted according to the data type. Abnormal data is automatically associated with the contingency plan after being marked.

5. The intelligent quality control system for marine survey data according to claim 1, characterized in that, The core algorithm of the full-chain visualization module is the dual-probe TS curve comparison algorithm: ； in , These are the TS curve functions for dual probes, where L is the curve length, and C≥0.95 indicates data consistency. Simultaneously, this module can extract core information such as sensor serial number and calibration date, and track operational information in real time, including station latitude and longitude, operation duration, and deployment depth, presenting data quality status through multi-dimensional graphics.

6. A method for intelligent quality control of marine survey data, based on the intelligent quality control system for marine survey data as described in claims 1-5, characterized in that, Specifically, the following steps are included: S1: Relying on the system's three-dimensional quality control framework module, over 40 key quality control elements are extracted and weighted using the entropy weight method. Quality control rules based on a four-layer quantitative standard system of equipment status, parameter settings, data integrity, and operational compliance are established using quantitative formulas. Utilizing the system's independently developed unified data parsing technology, standardized access to multiple data formats such as .ENR and .XSE is achieved, and the parsing adaptation function satisfies F... parse (F raw )=F standard F raw For the original data format, F standard The data is in a standard format, with a parsing success rate of ≥99.5%. S2: The system's real-time + timed dual-mode quality control engine is activated. The real-time quality control module captures equipment operating parameters and data transmission status through dynamic mapping technology, and instantly identifies equipment and data anomalies based on the improved 3σ algorithm, with an anomaly warning response time of ≤5 minutes. The timed quality control module automatically completes data backup, multi-dimensional quality control analysis, and standardized report generation according to an adaptive cycle. S3: Multi-dimensional verification is conducted through the system's full-chain visualization module, equipment status is verified using the equipment lifecycle information traceability module, operational standardization is controlled through the dynamic monitoring module of the operation process, and data quality consistency is presented using the dual-probe TS curve comparison algorithm. The multi-dimensional verification results meet the following requirements: ； Where w i For the weights of each dimension, Q i Quality control scores for each dimension; S4: Based on the system's non-interventional automated module, it autonomously completes the entire process of access, processing, analysis, early warning, and reporting, outputting a standardized quality control report that annotates the anomaly type, location, and suggested handling. The correction rate after anomaly handling is as follows: ； Where N corrected To correct the number of successfully corrected abnormal data entries, N abnormal This represents the total number of abnormal data entries.

7. The intelligent quality control method for marine survey data according to claim 1, characterized in that, The unified data parsing technology described in step S1 is compatible with multiple raw data formats such as .ENR and .XSE. It achieves accurate identification of more than 10 data formats through feature field matching, with an accuracy rate ≥99.8%. The core standardized conversion formula is: ； Where D extract To extract data, D min D max U represents the extreme values ​​of the original data, and L represents the upper and lower limits of the standard data. The normalization error of the converted data is ≤0.1%.

8. The intelligent quality control method for marine survey data according to claim 1, characterized in that, The anomaly identification in step S2 employs a hybrid algorithm that combines a statistical model with a rule engine. The core fusion judgment formula is: A = α·A stat +(1-α)·A rule Where α is the fusion coefficient, ranging from 0.6 to 0.8, A stat To detect outliers, A rule For rule-related abnormal results, an early warning is triggered when A ≥ 0.7, enabling accurate identification of equipment and data anomalies.

9. The intelligent quality control method for marine survey data according to claim 1, characterized in that, The comprehensive data quality scoring formula mentioned in step S3 is as follows: ； Where w1=0.3, w2=0.3, w3=0.2, w4=0.2 are the weights of each dimension, β is the missing data rate, γ is the percentage of valid data, and A abnormal The deviation value represents outlier data, C represents the similarity of the TS curve, and a score Q≥0.9 indicates high-quality data that meets the standards for marine scientific research data application.

10. The intelligent quality control method for marine survey data according to claim 1, characterized in that, The core algorithm for the non-interventional automated closed-loop operation described in step S4 is a cross-voyage adaptation algorithm: ； Where S current,i For the i-th criterion of the current voyage, S history,i The historical average standard is used. If Adapt ≤ 0.1, the historical configuration is directly reused; otherwise, the parameter threshold is automatically adjusted to ensure the uniformity and reliability of quality control standards across voyages and equipment.

Images