Land-based data automatic receiving and preprocessing management system and method

Abstract

The invention discloses a land-based data automatic receiving and preprocessing management system and method, and relates to the technical field of ocean observation data processing, and the system comprises a communication service subsystem, a data preprocessing subsystem, an intelligent perception linkage subsystem, and a data management subsystem. And full-process automatic processing of underwater acoustic real-time communication subsurface buoy observation data is realized through modular integration. Automatic receiving, safe storage and efficient preprocessing of deep sea power and biogeochemical element data are achieved, reliable data support is provided for marine scientific research and environment monitoring, and the long-term observation requirement of a 6000-meter water depth subsurface buoy system is met.

Description

Technical Field

[0001] This invention relates to the field of marine observation data processing technology, and in particular to a land-based automatic data reception and preprocessing management system and method. Background Technology

[0002] With the development of deep-sea observation technology, underwater acoustic real-time communication buoys have become core equipment for acquiring ocean dynamic (temperature, salinity, current velocity, depth) and biogeochemical (chlorophyll, nitrate, dissolved oxygen, pH) data. Buoy observation data needs to be transmitted to a land-based platform for processing and analysis, but existing land-based data processing methods have the following key problems: Data reception is untimely and lacks integrity: Existing land-based systems mostly rely on a single satellite (such as only BeiDou satellites) or a single link (such as only underwater acoustic links) to receive data. When satellite signal coverage is insufficient (such as the success rate of BeiDou signals in some areas of the western Pacific Ocean being as low as 1.63%) or the link is interrupted, data loss is likely to occur. Furthermore, there is a lack of effective breakpoint resume mechanism, making it impossible to retransmit missing data. The data reception success rate is difficult to meet the needs of scientific research (the average success rate of existing technologies is <80%).

[0003] Data preprocessing is inefficient and lacks accuracy: The raw data from underwater moorings contains invalid values ​​(NaN), non-integer data, and data with transmission errors. Existing processing methods mostly rely on manual screening and verification, which is inefficient and prone to errors. In particular, ADCP current velocity data has not been specially processed for its sampling characteristics (integer data has high reliability), resulting in large deviations in subsequent marine phenomenon identification (such as near-inertial internal waves).

[0004] Lack of intelligent linkage with underwater moorings: Existing land-based systems only have data receiving and storage functions, and cannot identify marine phenomena (such as near-inertial internal waves) based on data characteristics and adjust the underwater mooring observation mode accordingly. As a result, the underwater moorings still maintain a fixed observation frequency (such as ADCP 1 time / hour), making it difficult to meet the high-frequency capture of sudden phenomena and the low power consumption requirements during normal periods.

[0005] Weak data security and management capabilities: Some land-based systems use the Windows operating system, which has security vulnerabilities; data storage lacks partitioning strategies and dual backup mechanisms, which can easily lead to data chaos or loss; and the retrieval efficiency is low, making it impossible to quickly locate observation data of specific elements or specific time periods, which affects the efficiency of scientific research and analysis.

[0006] To address the aforementioned issues, there is an urgent need for a land-based data system with automatic reception, efficient preprocessing, intelligent linkage, and security management capabilities to meet the long-term, multi-element observation needs of deep-sea moorings. Summary of the Invention

[0007] To address the shortcomings of existing technologies, a land-based automatic data reception and preprocessing management system and method are provided. Through modular design, the system enables automatic reception, precise preprocessing, intelligent linkage control, and safety management of underwater buoy observation data, thereby improving data reception success rate, preprocessing efficiency, and scientific research applicability, and overcoming the deficiencies of existing technologies.

[0008] The land-based automatic data reception and preprocessing management system includes: The communication service subsystem is used to establish a data transmission link with the underwater acoustic real-time communication buoy, enabling automatic reception and breakpoint resumption of observation data. The data preprocessing subsystem is used to clean, verify, compress, and convert the format of the received raw observation data, and output standardized data. The intelligent sensing and linkage subsystem is used to identify marine phenomena based on preprocessed observation data and generate control commands for the mooring observation mode. The data management subsystem is used to store, retrieve, and back up preprocessed standardized data, and supports data visualization and report generation.

[0009] The aforementioned land-based automatic data reception and preprocessing management system includes a communication service subsystem comprising a satellite communication receiving unit, an underwater acoustic link adaptation unit, and a breakpoint resume control unit. The satellite communication receiving unit supports dual-mode switching between domestic Beidou satellites and Tiantong satellites. It adopts a single-module multi-card integrated circuit (integrating ≥12 satellite communication cards) and receives observation data sent by the surface float of the underwater buoy through a dynamic packet transmission mechanism. The receiving rate is adapted to the MFSK mode (300bps) and QPSK mode (6000bps) of the underwater acoustic communication unit. The underwater acoustic link adapter unit: establishes data interaction with the underwater acoustic communication device via RS-232 serial port (baud rate 9600), parses the data verification information (cyclic redundancy code) in the underwater acoustic communication protocol, and judges the integrity of data reception; The breakpoint resume control unit records the timestamp and data block number of the received data. When the satellite signal is interrupted or the underwater acoustic link is abnormal, it automatically sends a request for unreceived data to the underwater buoy after the link is restored, and retransmits the missing data to ensure that the data reception success rate is ≥90% (when the system is completed).

[0010] The aforementioned land-based automatic data reception and preprocessing management system also employs a Linux kernel operating system in its communication service subsystem to avoid Microsoft system security vulnerabilities; it is configured with a dual-server hot standby mechanism, so that when the primary server fails, the backup server can switch over and take over the data reception task within ≤5 minutes to ensure continuous reception.

[0011] The aforementioned land-based automatic data reception and preprocessing management system includes a data preprocessing subsystem comprising a data cleaning unit, a multi-dimensional verification unit, an adaptive compression unit, and a format conversion unit. The data cleaning unit identifies and marks invalid values ​​(NaN) and outlier values ​​(such as values ​​that exceed the normal temperature and salinity range of the ocean) in the original data, removes the observation data layer with NaN accounting for more than 1 / 3 (for ADCP current velocity data), and retains the valid data segments. The multi-dimensional verification unit: Logical verification: Based on the correlation of ocean dynamic parameters (such as the negative correlation between temperature and dissolved oxygen concentration) and the balance of biogeochemical elements (such as the coupling relationship between pH value and nitrate concentration), the logical consistency of the data is judged. Transmission verification: Compare the data checksum in the underwater acoustic communication protocol with the locally calculated value, filter out erroneous data, and trigger retransmission; The adaptive compression unit uses a compression algorithm that combines arithmetic encoding and dictionary encoding to compress standardized data with a compression ratio of ≥1:3, while preserving data precision during the compression process (e.g., temperature data retains 2 decimal places and flow rate data retains 3 decimal places). The format conversion unit converts the raw binary data of the underwater mooring into a common data format (NetCDF, CSV) to be compatible with commonly used data processing software in marine science research (such as MATLAB, OceanDataView).

[0012] The aforementioned land-based automatic data receiving and preprocessing management system includes an intelligent sensing and linkage subsystem comprising a data feature extraction unit, a marine phenomenon identification unit, and a control command generation unit. The data feature extraction unit extracts the horizontal velocity components (u, v) from the preprocessed ADCP velocity data, selects the latest 128-hour hourly data (2 to the power of n data volume, adapted to Fourier transform), and calculates the power spectral density S(k). The marine phenomenon identification unit integrates the power spectral density S, with integration bands of 0.8-1.2f and 0.8f to 0.5ω, where f is the local Coriolis frequency and ω is the sampling frequency; it then calculates the near-inertial energy proportion through integration. Set a dynamic threshold: the initial threshold is 0.42, and it is lowered to 0.40 when no near-inertial signal is detected for two consecutive months; when r > threshold, it is determined that there is a near-inertial internal wave, and when r < threshold, it is determined that the near-inertial internal wave has ended. The control command generation unit generates control commands for the moored observation mode based on the marine phenomenon identification results (such as generating a command to "switch the ADCP observation interval from 1 time / hour to 1 time / 10 minutes" when near-inertial internal waves occur), and transmits the commands in reverse to the surface float of the moored moored moored moored moored moored moored moored moored moored moored moored moored moored mooring ...ing moored mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring

[0013] The aforementioned land-based automatic data reception and preprocessing management system includes a data management subsystem comprising a distributed storage unit, an intelligent retrieval unit, a data backup unit, and a visualization unit. The distributed storage unit adopts a partitioned storage strategy, establishing a data storage catalog based on the type of observed elements (marine dynamic elements: temperature, salinity, current velocity, depth; biogeochemical elements: chlorophyll, nitrate, dissolved oxygen, pH value) and time dimension (year / month / day), supporting the storage of continuous observation data for ≥6 months (average daily data volume of a single mooring ≥100MB). The intelligent retrieval unit supports multi-condition combined retrieval (element type, time range, water depth interval), with a retrieval response time of ≤10 seconds. The data backup unit adopts a dual backup mechanism of local and off-site backups. The local backup cycle is 1 hour / time, the off-site backup cycle is 1 day / time, and the backup data retention time is ≥3 years. The visualization unit automatically generates time series plots (such as temperature change curves at a certain water depth) and profile distribution maps (such as salinity profiles at different water depths) of the observed data, and supports the marking and export of data anomalies.

[0014] The aforementioned land-based automatic data reception and preprocessing management system further includes a satellite communication receiving unit that includes a signal quality monitoring module. This module monitors satellite signal strength in real time (BeiDou satellite regional success rate ≥ 94.59%, TianTong satellite coverage of the western Pacific Ocean 110°E-155°E, 0°-35°N region). When the signal strength is lower than a preset threshold (e.g., BeiDou signal success rate < 85%), it automatically switches to a backup satellite (e.g., switching from TianTong satellite to BeiDou satellite) and records the signal switching log.

[0015] The aforementioned land-based automatic data reception and preprocessing management system also includes a data cleaning unit that performs special processing on ADCP data: removing non-integer data with minute digits of 10, 20, 30, 40, and 50, and retaining integer sampled data; performing layer screening on ADCP velocity data (u and v components), removing depth layers with poor data quality near the surface (close to the main buoy of the underwater mooring), and retaining data from effective observation depth layers.

[0016] The above-mentioned method for the automatic reception and preprocessing management system of land-based data includes the following steps: S1: System initialization, the communication service subsystem starts the satellite communication receiving unit and the underwater acoustic link adapter unit, establishes a data transmission link with the underwater buoy, and the breakpoint resume control unit initializes the data receiving record list; S2: Automatic data reception. The satellite communication receiving unit receives observation data transmitted by the surface buoy of the underwater glider via Beidou / Tiantong satellites. The underwater acoustic link adapter unit receives deep data transmitted by the underwater acoustic communication device. The breakpoint resume control unit records the timestamp and data block number of the received data in real time. S3: Data preprocessing, data cleaning unit removes invalid values ​​and non-integer data, multi-dimensional verification unit completes logical verification and transmission verification, adaptive compression unit compresses the verified data, and format conversion unit converts the data into NetCDF / CSV format; S4: Intelligent sensing and linkage. The intelligent sensing and linkage subsystem performs power spectrum analysis on the preprocessed ADCP data, calculates the near-inertial energy ratio r, generates a mooring control command based on the comparison between r and the dynamic threshold, and sends it back to the mooring via the satellite communication link. S5: Data Management. The data management subsystem stores the pre-processed standardized data into a distributed storage unit, activates intelligent retrieval, dual backup and visualization functions, and generates data reception and pre-processing reports.

[0017] The above-mentioned method for the automatic reception and preprocessing management system of land-based data further includes an anomaly handling mechanism in step S2: when the communication service subsystem fails to receive data for 5 consecutive minutes, it automatically triggers link diagnosis (checking satellite signal and underwater acoustic link connection status). If the problem is with the satellite signal, it switches to a backup satellite; if the problem is with the underwater acoustic link, it records the missing data period and resumes transmission after the link is restored. In step S3, if the verification finds a data error (such as a mismatch in the transmission checksum), it sends a retransmission request for the data block to the underwater buoy until the correct data is received.

[0018] The beneficial effects of the land-based data automatic reception and preprocessing management system and method of the present invention are that the data reception is more stable and complete: the dual-mode satellite switching and breakpoint resume mechanism of the communication service subsystem solves the problem of insufficient satellite signal of a single link, improves the data reception success rate to ≥90% (when completed), and can make up for the missing data during the link interruption period, ensuring the continuity of observation data.

[0019] Double improvement in preprocessing efficiency and accuracy: The automated cleaning, multi-dimensional verification and ADCP-specific processing of the data preprocessing subsystem improve preprocessing efficiency by more than 5 times (compared to manual processing), while eliminating invalid and erroneous data, providing high-quality data for marine phenomenon identification, with a near-inertial internal wave identification accuracy of ≥95%.

[0020] Realize intelligent linkage between land-based and underwater mooring: The intelligent sensing and linkage subsystem adjusts the underwater mooring observation mode based on data characteristics, which not only meets the high-frequency capture requirements of sudden phenomena such as near-inertial internal waves, but also avoids power consumption waste during normal periods, and extends the continuous working time of underwater mooring at sea to ≥6 months.

[0021] Data management is safer and more efficient: The Linux kernel operating system and dual backup mechanism enhance data security; distributed storage and intelligent retrieval shorten data location time to ≤10 seconds; visualization functions and report generation provide researchers with intuitive data results and lower the threshold for data analysis. Attached Figure Description

[0022] Figure 1 : A schematic diagram of the workflow of this invention; Figure 2 : Flowchart of the marine phenomenon recognition unit. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1-2 As shown, the land-based automatic data reception and preprocessing management system includes: The communication service subsystem is used to establish a data transmission link with the underwater acoustic real-time communication buoy, enabling automatic reception and breakpoint resumption of observation data. The data preprocessing subsystem is used to clean, verify, compress, and convert the format of the received raw observation data, and output standardized data. The intelligent sensing and linkage subsystem is used to identify marine phenomena based on preprocessed observation data and generate control commands for the mooring observation mode. The data management subsystem is used to store, retrieve, and back up preprocessed standardized data, and supports data visualization and report generation.

[0025] The aforementioned land-based automatic data reception and preprocessing management system includes a communication service subsystem comprising a satellite communication receiving unit, an underwater acoustic link adaptation unit, and a breakpoint resume control unit. This communication service subsystem is the core connecting the land-based platform and the deep-sea mooring, and is responsible for stable data reception and link assurance.

[0026] The satellite communication receiving unit supports dual-mode switching between domestic Beidou satellites and Tiantong satellites. It adopts a single-module multi-card integrated circuit (integrating ≥12 satellite communication cards) and receives observation data sent by the surface float of the underwater buoy through a dynamic packet transmission mechanism. The receiving rate is adapted to the MFSK mode (300bps) and QPSK mode (6000bps) of the underwater acoustic communication unit. Adapted to dual-mode transmission from domestic satellites, this system integrates BeiDou and Tiantong satellite communication modules, employing a single-module multi-card design (integrating 12-16 satellite communication cards) and supporting dynamic packet-splitting transmission. For observation areas such as the western Pacific Ocean (110°E-155°E, 0°-35°N), the satellite communication receiving unit monitors signal quality in real time (e.g., BeiDou satellite regional success rate, Tiantong satellite coverage area). When the primary satellite signal success rate is <85%, it automatically switches to the backup satellite to ensure continuous data reception. For example, during observations in the eastern Luzon Strait, if the Tiantong satellite signal is interrupted, the system can switch to BeiDou satellites within 30 seconds to maintain data transmission.

[0027] The underwater acoustic link adapter unit: establishes data interaction with the underwater acoustic communication device via RS-232 serial port (baud rate 9600), parses the data verification information (cyclic redundancy code) in the underwater acoustic communication protocol, and judges the integrity of data reception; A connection is established with the underwater acoustic communication device via an RS-232 serial port (baud rate 9600). The cyclic redundancy check (CRC) information in the underwater acoustic communication protocol is parsed to determine the integrity of the received deep-sea (over 500 meters) observation data. When the check detects a data error, a retransmission request is automatically sent to the underwater acoustic communication device to ensure the reliability of deep-sea data (such as CTD data at a depth of 5000 meters).

[0028] The breakpoint resume control unit records the timestamp and data block number of the received data. When the satellite signal is interrupted or the underwater acoustic link is abnormal, it automatically sends a request for unreceived data to the underwater buoy after the link is restored, and retransmits the missing data to ensure that the data reception success rate is ≥90% (when the system is completed).

[0029] A data reception record list is constructed to store the timestamps (accurate to the second), data block numbers, and feature types of the received data in real time. When the link is interrupted (e.g., the satellite signal is temporarily lost), the unit compares the received data with the underwater glider data transmission logs after the link is restored, identifies the period of missing data, and sends a retransmission request to the underwater glider to ensure a data reception success rate of ≥90% (when the system is completed), thus avoiding the loss of critical data due to link fluctuations.

[0030] The communication service subsystem also uses a Linux kernel operating system to avoid security vulnerabilities in Microsoft systems; it is configured with a dual-server hot standby mechanism, so that when the primary server fails, the backup server will switch over and take over the data receiving task within ≤5 minutes to ensure continuous reception.

[0031] The data preprocessing subsystem includes a data cleaning unit, a multi-dimensional verification unit, an adaptive compression unit, and a format conversion unit; the data preprocessing subsystem achieves automated and high-precision preprocessing tailored to the characteristics of the original underwater glider data.

[0032] The data cleaning unit identifies and marks invalid values ​​(NaN) and outlier values ​​(such as values ​​exceeding the normal temperature and salinity range of the ocean) in the raw data, removes observation data layers with NaN accounting for more than 1 / 3 (for ADCP current velocity data), and retains valid data segments. The data cleaning unit also performs special processing on ADCP data: removes non-integer data with minute digits of 10, 20, 30, 40, and 50, and retains integer sampled data; it performs layer screening on ADCP current velocity data (u and v components), removes depth layers with poor data quality near the surface (close to the main buoy of the mooring), and retains valid observation depth layer data.

[0033] A dual strategy of "general cleaning and specialized cleaning" was adopted. General cleaning removed NaN values ​​from the raw data (data from depth layers accounting for more than 1 / 3 of the total) and abnormal extreme values ​​that exceeded the normal range of the ocean (such as temperature <-2℃ or >35℃, salinity <30 or >38). Specialized cleaning optimized ADCP data: it removed non-integer data with minute digits of 10, 20, 30, 40, and 50, and retained integer sampling data; it removed near-surface data close to the main float of the mooring vessel (500 meters deep) (data from this layer is easily affected by the float and has poor quality), and retained effective data from the middle layer (500-5000 meters) and the deep layer (5000-6000 meters), providing a high-quality data source for subsequent ocean phenomenon identification.

[0034] The multi-dimensional verification unit: Logical verification: Based on the correlation of ocean dynamic parameters (such as the negative correlation between temperature and dissolved oxygen concentration) and the balance of biogeochemical elements (such as the coupling relationship between pH value and nitrate concentration), the logical consistency of the data is judged. Transmission verification: Compare the data checksum in the underwater acoustic communication protocol with the locally calculated value, filter out erroneous data, and trigger retransmission; Data accuracy is ensured through both "transmission and logic" dimensions. Transmission verification compares the checksum in the underwater acoustic communication protocol with the locally calculated value to filter out erroneous data and trigger retransmission. Logic verification is based on the inherent correlations between ocean elements, such as the negative correlation between temperature and dissolved oxygen concentration (as temperature increases, dissolved oxygen concentration decreases) and the positive correlation between salinity and depth (as depth increases, salinity gradually stabilizes at 34-35). If data violates these logical relationships (e.g., both temperature and dissolved oxygen concentration increase), it is marked as abnormal data and recorded for subsequent review by researchers.

[0035] The adaptive compression unit uses a compression algorithm that combines arithmetic encoding and dictionary encoding to compress standardized data with a compression ratio of ≥1:3, while preserving data precision during the compression process (e.g., temperature data retains 2 decimal places and flow rate data retains 3 decimal places). A combined algorithm of "arithmetic coding and dictionary coding" is used to compress the verified data. Arithmetic coding optimizes the code length based on the probability distribution of the data, while dictionary coding establishes a dictionary mapping for recurring data segments (such as fixed-format feature identifiers and unit information). This compresses the data volume to 1 / 3 of the original size, reducing storage usage without sacrificing data precision (temperature is retained to 2 decimal places, and flow rate is retained to 3 decimal places), thus meeting the storage requirements of long-term observation data.

[0036] The format conversion unit converts the raw binary data of the underwater mooring into a common data format (NetCDF, CSV) to be compatible with commonly used data processing software in marine science research (such as MATLAB, OceanDataView).

[0037] The compressed raw binary data (output format of underwater mooring equipment) is converted into NetCDF format (which supports multidimensional data storage and is compatible with software such as OceanDataView) and CSV format (which supports reading by Excel and MATLAB) commonly used in marine scientific research, avoiding analysis obstacles caused by data format incompatibility and improving data applicability.

[0038] The intelligent sensing and linkage subsystem includes a data feature extraction unit, a marine phenomenon identification unit, and a control command generation unit; the intelligent sensing and linkage subsystem is the core of the linkage between the land-based system and the underwater mooring, and realizes marine phenomenon identification and observation mode control based on data features.

[0039] The data feature extraction unit extracts the horizontal velocity components (u, v) from the preprocessed ADCP velocity data, selects the latest 128-hour hourly data (2 to the power of n data volume, adapted to Fourier transform), and calculates the power spectral density S(k). Key features were extracted from the preprocessed ADCP data: hourly velocity data (u and v components) from the latest 128 hours were selected, with a data volume of 2 to the power of 7, which is suitable for the spectral analysis requirements of Fourier transform; time series of velocity data at each depth layer were calculated to provide basic data for subsequent power spectrum analysis.

[0040] The marine phenomenon identification unit integrates the power spectral density S, with integration bands of 0.8-1.2f and 0.8f to 0.5ω, where f is the local Coriolis frequency and ω is the sampling frequency; it then calculates the near-inertial energy proportion through integration. Set a dynamic threshold: the initial threshold is 0.42, and it is lowered to 0.40 when no near-inertial signal is detected for two consecutive months; when r > threshold, it is determined that there is a near-inertial internal wave, and when r < threshold, it is determined that the near-inertial internal wave has ended. The marine phenomenon identification unit focuses on near-inertial internal wave identification, achieved through the following steps: ① Calculate the Coriolis frequency f based on the latitude of the mooring deployment (e.g., 20.5°N); ② Perform Fourier transform on the ADCP current velocity data to obtain the power spectral density S(k); ③ Integrate the near-inertial energy (0.8f-1.2f frequency band) and the total internal wave energy (0.8f-0.5ω frequency band, where ω is the sampling frequency of 24 times / day), and calculate the ratio r; ④ Introduce a dynamic threshold: the initial threshold is 0.42. If no near-inertial signal is detected for two consecutive months (r always < 0.42), the threshold is lowered to 0.40 to avoid missed identification due to differences in sea area characteristics. When r > threshold, a significant near-inertial internal wave is determined to exist; when r < threshold for all depth layers, the near-inertial internal wave identification ends.

[0041] The control command generation unit generates control commands for the moored observation mode based on the marine phenomenon identification results (such as generating a command to "switch the ADCP observation interval from 1 time / hour to 1 time / 10 minutes" when near-inertial internal waves occur), and transmits the commands in reverse to the surface float of the moored moored moored moored moored moored moored moored moored moored moored moored moored moored mooring ...ing moored mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring mooring

[0042] Based on the phenomenon identification results, precise control commands are generated: when near-inertial internal waves occur, a command is generated to "switch the ADCP observation interval from 1 time / hour to 1 time / 10 minutes, and simultaneously shorten the satellite transmission interval to 1 time / 10 minutes"; when the near-inertial internal waves end, a command is generated to "restore ADCP observation to 1 time / hour, and restore the satellite transmission interval to 1 time / hour". The commands are transmitted back to the surface buoy of the mooring via the satellite communication link, and the central control device of the buoy executes the observation mode switch, realizing a closed-loop linkage of "land-based analysis - mooring response".

[0043] The data management subsystem includes a distributed storage unit, an intelligent retrieval unit, a data backup unit, and a visualization unit; the system enables secure storage, efficient retrieval, and visualization of preprocessed data.

[0044] The distributed storage unit adopts a partitioned storage strategy, establishing a data storage catalog based on the type of observed elements (marine dynamic elements: temperature, salinity, current velocity, depth; biogeochemical elements: chlorophyll, nitrate, dissolved oxygen, pH value) and time dimension (year / month / day), supporting the storage of continuous observation data for ≥6 months (average daily data volume of a single mooring ≥100MB). A dual-dimensional partitioning strategy based on "elements and time" is adopted. A primary directory is created according to the type of observed element (dynamic elements, biochemical elements), and a secondary directory is created according to "year / month / day," for example, "Dynamic Elements / 2024 / 06 / 15 / Temperature Data.nc" and "Biochemical Elements / 2024 / 06 / 15 / Nitrate Data.csv." It supports data storage for 6 months of continuous observations from a single underwater mooring system (average daily data volume of 100MB, total data volume of approximately 18GB for 6 months), using industrial-grade hard drives to ensure data read / write stability.

[0045] The intelligent retrieval unit supports multi-condition combined retrieval (element type, time range, water depth interval), with a retrieval response time of ≤10 seconds. It supports multi-condition combined search. Users can enter "feature type (such as flow velocity) + time range (such as 2024.06.01-2024.06.10) and water depth range (such as 1000-2000 meters)", and the system will locate the target data and return the search results within 10 seconds. The search results include key information such as data storage path, data volume, and completeness (percentage of time periods without missing data), which makes it convenient for researchers to quickly obtain the data they need.

[0046] The data backup unit adopts a dual backup mechanism of local and off-site backups. The local backup cycle is 1 hour / time, the off-site backup cycle is 1 day / time, and the backup data retention time is ≥3 years. It adopts a dual backup mechanism of "local and off-site". Local backup is performed once every hour (backing up to the system's built-in backup hard drive), and off-site backup is performed once a day (backing up to a remote server and transmitting via an encrypted network). The backup data retention period is ≥3 years, and the integrity of the backup data is checked regularly (once a month) to avoid permanent data loss due to hardware failure.

[0047] The visualization unit automatically generates time series plots (such as temperature change curves at a certain water depth) and profile distribution maps (such as salinity profiles at different water depths) of the observed data, and supports the marking and export of data anomalies.

[0048] The system automatically generates two types of visualization charts: time series charts (such as temperature change curves over time at a fixed water depth (1000 meters), with outlier data points marked) and profile distribution charts (such as salinity distribution curves at different water depths at a certain time point (June 15, 2024, 12:00). The charts can be exported in PNG and PDF formats. At the same time, it generates data reception and preprocessing reports (including the amount of data received on the day, preprocessing pass rate, proportion of outlier data, and number of satellite link switching times), providing a reference for system operation and maintenance and scientific research analysis.

[0049] The satellite communication receiving unit also includes a signal quality monitoring module, which monitors the satellite signal strength in real time (the success rate of BeiDou satellites in the region is ≥94.59%, and TianTong satellites cover the western Pacific Ocean region of 110°E-155°E and 0°-35°N). When the signal strength is lower than a preset threshold (such as the success rate of BeiDou signals <85%), it automatically switches to a backup satellite (such as switching from TianTong satellites to BeiDou satellites) and records the signal switching log.

[0050] The above-mentioned method for the automatic reception and preprocessing management system of land-based data includes the following steps: S1: System initialization, the communication service subsystem starts the satellite communication receiving unit and the underwater acoustic link adapter unit, establishes a data transmission link with the underwater buoy, and the breakpoint resume control unit initializes the data receiving record list; S2: Automatic data reception. The satellite communication receiving unit receives observation data transmitted by the surface buoy of the underwater glider via Beidou / Tiantong satellites. The underwater acoustic link adapter unit receives deep data transmitted by the underwater acoustic communication device. The breakpoint resume control unit records the timestamp and data block number of the received data in real time. S3: Data preprocessing, data cleaning unit removes invalid values ​​and non-integer data, multi-dimensional verification unit completes logical verification and transmission verification, adaptive compression unit compresses the verified data, and format conversion unit converts the data into NetCDF / CSV format; S4: Intelligent sensing and linkage. The intelligent sensing and linkage subsystem performs power spectrum analysis on the preprocessed ADCP data, calculates the near-inertial energy ratio r, generates a mooring control command based on the comparison between r and the dynamic threshold, and sends it back to the mooring via the satellite communication link. S5: Data Management. The data management subsystem stores the pre-processed standardized data into a distributed storage unit, activates intelligent retrieval, dual backup and visualization functions, and generates data reception and pre-processing reports.

[0051] The above-mentioned method for the automatic reception and preprocessing management system of land-based data further includes an anomaly handling mechanism in step S2: when the communication service subsystem fails to receive data for 5 consecutive minutes, it automatically triggers link diagnosis (checking satellite signal and underwater acoustic link connection status). If the problem is with the satellite signal, it switches to a backup satellite; if the problem is with the underwater acoustic link, it records the missing data period and resumes transmission after the link is restored. In step S3, if the verification finds a data error (such as a mismatch in the transmission checksum), it sends a retransmission request for the data block to the underwater buoy until the correct data is received.

[0052] The following examples are based on the observation data processing scenario of the "underwater acoustic real-time communication mooring system" (deployed in the western Pacific Ocean at a depth of 6000 meters): Example

[0053] Routine observation data reception and preprocessing 1. System Configuration: Communication service subsystem: Enables BeiDou-TianTong dual-mode satellites, integrates 16 communication cards in a single module, and sets the baud rate of the underwater acoustic link adapter unit to 9600; Data preprocessing subsystem: ADCP data cleaning retains hourly data, and outlier thresholds are set for temperature (-2℃-35℃) and salinity (30-38). The compression algorithm uses arithmetic encoding + dictionary encoding. Data Management Subsystem: The storage directory is divided into "Power Elements / Year / Month / Day" and "Biochemical Elements / Year / Month / Day". Local backup is performed once every hour, and off-site backup is performed once every day.

[0054] 2. Implementation process: S1: The system starts up, the communication service subsystem establishes the BeiDou satellite link and the underwater acoustic link, and the breakpoint resume control unit initializes the data reception record. S2: Data reception. The Beidou satellite receives CTD and chlorophyll data from the upper layer (0-500 meters) of the underwater buoy (1 time / hour), and the underwater acoustic link receives ADCP and nitrate data from the deep layer (500-6000 meters) (1 time / hour). The timestamp is recorded as "2024-06-15 00:00-23:00". S3: Preprocessing, the cleaning unit removes non-hourly ADCP data from 10:10 and 14:30 on 2024-06-15, the verification unit finds logical anomalies in the nitrate data at 16:00 (temperature rises and dissolved oxygen rises), triggers retransmission and obtains the correct data, the compression unit compresses the data to 32% of the original size and converts it to NetCDF format; S4: Data management. The storage unit stores the data in the directories “Power Elements / 2024 / 06 / 15” and “Biochemical Elements / 2024 / 06 / 15”. The backup unit performs local and off-site backups. The visualization unit generates a time series graph of water temperature at a depth of 1000 meters. The report shows that the data preprocessing pass rate for the day is 98.5%.

[0055] 3. Implementation Results: The data reception success rate was 99.2%, with no missing data; the preprocessed data had no invalid or erroneous values, and the NetCDF format could be directly imported into OceanDataView for analysis, meeting the needs of routine marine dynamics and biochemical element research. Example

[0056] Near-inertial internal wave identification and underwater mooring linkage control 1. System Configuration: Intelligent sensing and linkage subsystem: The latitude of the buoy is set to 20.5°N (Coriolis frequency f≈0.00012Hz), and the initial dynamic threshold is 0.42; The configurations of other subsystems are the same as in Example 1.

[0057] 2. Implementation process: S1: Receive ADCP data from 00:00 on June 16, 2024 to 23:00 on June 20, 2024, and obtain 128 hours of hourly flow velocity data (u and v components) after preprocessing. S2: The intelligent sensing and linkage subsystem calculates the power spectral density S(k), integrates it to obtain the near-inertial energy ratio r=0.47 (>0.42), and determines that there is a near-inertial internal wave; S3: Generate control command "ADCP observation interval changed from 1 time / hour to 1 time / 10 minutes, satellite transmission interval 1 time / 10 minutes", and send it back to the mooring via Tiantong satellite; S4: After the underwater buoy executes the command, the land-based system receives high-frequency data (1 time / 10 minutes), and after preprocessing, identifies the process of near-inertial internal wave energy propagating downwards from the upper 100 meters to 500 meters; S5: On June 28, 2024, r = 0.39 (< 0.42) was calculated, and the command "Restore Low-Frequency Observation" was generated, restoring the moored buoy to 1 observation per hour.

[0058] 3. Implementation results: Successfully captured the entire process of near-inertial internal waves, and acquired complete high-frequency data; the average delay of control command transmission was 0.8 days, the response of underwater glider mode switching was timely, there was no power consumption waste, and the overall system operation was stable.

[0059] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A land-based automatic data receiving and preprocessing management system, characterized in that, include: The communication service subsystem is used to establish a data transmission link with the underwater acoustic real-time communication buoy, enabling automatic reception and breakpoint resumption of observation data. The data preprocessing subsystem is used to clean, verify, compress, and convert the format of the received raw observation data, and output standardized data. The intelligent sensing and linkage subsystem is used to identify marine phenomena based on preprocessed observation data and generate control commands for the mooring observation mode. The data management subsystem is used to store, retrieve, and back up preprocessed standardized data, and supports data visualization and report generation.

2. The land-based data automatic reception and preprocessing management system according to claim 1, characterized in that, The communication service subsystem includes a satellite communication receiving unit, an underwater acoustic link adaptation unit, and a breakpoint resume control unit. The satellite communication receiving unit supports dual-mode switching between domestic Beidou satellites and Tiantong satellites, adopts a single-module multi-card integrated circuit, and receives observation data sent by the surface float of the underwater buoy through a dynamic packet transmission mechanism. The receiving rate is adapted to the MFSK mode and QPSK mode of the underwater acoustic communication unit. The underwater acoustic link adapter unit: establishes data interaction with the underwater acoustic communication device of the buoy through RS-232 serial port, parses the data verification information in the underwater acoustic communication protocol, and judges the integrity of data reception; The interruption resume control unit records the timestamp and data block number of the received data. When the satellite signal is interrupted or the underwater acoustic link is abnormal, it automatically sends a request for the unreceived data to the underwater buoy after the link is restored, and retransmits the missing data to ensure that the data reception success rate is ≥90%.

3. The land-based automatic data reception and preprocessing management system according to claim 2, characterized in that, The communication service subsystem also uses a Linux kernel operating system to avoid security vulnerabilities in Microsoft systems; it is configured with a dual-server hot standby mechanism, so that when the primary server fails, the backup server will switch over and take over the data receiving task within ≤5 minutes to ensure continuous reception.

4. The land-based automatic data reception and preprocessing management system according to claim 3, characterized in that, The data preprocessing subsystem includes a data cleaning unit, a multi-dimensional verification unit, an adaptive compression unit, and a format conversion unit; The data cleaning unit identifies and marks invalid values ​​and outliers in the original data, removes observation data layers with NaN ratios exceeding 1 / 3, and retains valid data segments. The multi-dimensional verification unit: Logical verification: Based on the correlation of ocean dynamic parameters and the balance of biogeochemical elements, the logical consistency of the data is judged; Transmission verification: Compare the data checksum in the underwater acoustic communication protocol with the locally calculated value, filter out erroneous data, and trigger retransmission; The adaptive compression unit uses a compression algorithm that combines arithmetic encoding and dictionary encoding to compress standardized data with a compression ratio of ≥1:3, while preserving data precision during the compression process. The format conversion unit converts the raw binary data of the underwater mooring into a common data format, which is compatible with data processing software commonly used in marine scientific research.

5. The land-based automatic data reception and preprocessing management system according to claim 4, characterized in that, The intelligent sensing and linkage subsystem includes a data feature extraction unit, a marine phenomenon recognition unit, and a control command generation unit. The data feature extraction unit extracts the horizontal velocity component from the preprocessed ADCP velocity data, selects the latest 128-hour hourly data, and calculates the power spectral density S(k). The marine phenomenon identification unit integrates the power spectral density S, with integration bands of 0.8-1.2f and 0.8f to 0.5ω, where f is the local Coriolis frequency and ω is the sampling frequency; it then calculates the near-inertial energy proportion through integration. Set a dynamic threshold: the initial threshold is 0.42, and it is lowered to 0.40 when no near-inertial signal is detected for two consecutive months; when r > threshold, it is determined that there is a near-inertial internal wave, and when r < threshold, it is determined that the near-inertial internal wave has ended. The control command generation unit generates control commands for the moored observation mode based on the marine phenomenon identification results, and transmits them in reverse to the surface float of the moored moored moored moored moored moored moored moored moored moored moored moored moored moored moored moored moored moored mooring ...

6. The land-based automatic data reception and preprocessing management system according to claim 5, characterized in that, The data management subsystem includes a distributed storage unit, an intelligent retrieval unit, a data backup unit, and a visualization unit; The distributed storage unit adopts a partitioned storage strategy, establishes a data storage catalog according to the type of observation element, and supports the storage of continuous observation data for ≥6 months. The intelligent retrieval unit supports multi-condition combined retrieval with a retrieval response time of ≤10 seconds. The data backup unit adopts a dual backup mechanism of local and off-site backups. The local backup cycle is 1 hour / time, the off-site backup cycle is 1 day / time, and the backup data retention time is ≥3 years. The visualization unit automatically generates time series plots and profile distribution maps of the observation data, and supports the marking and export of data outliers.

7. The land-based automatic data reception and preprocessing management system according to claim 6, characterized in that, The satellite communication receiving unit also includes a signal quality monitoring module, which monitors the satellite signal strength in real time. When the signal strength is lower than a preset threshold, it automatically switches to a backup satellite and records the signal switching log.

8. The land-based automatic data reception and preprocessing management system according to claim 7, characterized in that, The data cleaning unit also performs special processing on ADCP data: it removes non-integer data with minute digits of 10, 20, 30, 40, and 50, and retains integer sampled data; it performs layer filtering on ADCP flow velocity data, removing depth layers with poor near-surface data quality, and retaining data from effective observation depth layers.

9. A method for an automatic land-based data reception and preprocessing management system according to any one of claims 1-8, characterized in that, Includes the following steps: S1: System initialization, the communication service subsystem starts the satellite communication receiving unit and the underwater acoustic link adapter unit, establishes a data transmission link with the underwater buoy, and the breakpoint resume control unit initializes the data receiving record list; S2: Automatic data reception. The satellite communication receiving unit receives observation data transmitted by the surface buoy of the underwater glider via Beidou / Tiantong satellites. The underwater acoustic link adapter unit receives deep data transmitted by the underwater acoustic communication device. The breakpoint resume control unit records the timestamp and data block number of the received data in real time. S3: Data preprocessing, data cleaning unit removes invalid values ​​and non-integer data, multi-dimensional verification unit completes logical verification and transmission verification, adaptive compression unit compresses the verified data, and format conversion unit converts the data into NetCDF / CSV format; S4: Intelligent sensing and linkage. The intelligent sensing and linkage subsystem performs power spectrum analysis on the preprocessed ADCP data, calculates the near-inertial energy ratio r, generates a mooring control command based on the comparison between r and the dynamic threshold, and sends it back to the mooring via the satellite communication link. S5: Data Management. The data management subsystem stores the pre-processed standardized data into a distributed storage unit, activates intelligent retrieval, dual backup and visualization functions, and generates data reception and pre-processing reports.

10. The method for the automatic reception and preprocessing management system for land-based data according to claim 9, characterized in that, In step S2, an anomaly handling mechanism is also included: when the communication service subsystem does not receive data for 5 consecutive minutes, link diagnosis is automatically triggered. If it is a satellite signal problem, a backup satellite is switched. If it is an underwater acoustic link problem, the missing data period is recorded. After the link is restored, the interrupted transmission is resumed. In step S3, if the verification finds that the data is incorrect, a retransmission request for the data block is sent to the underwater buoy until the correct data is received.

Images