A lightweight ocean data acquisition and processing system and method based on edge computing

Abstract

The present invention provides a lightweight ocean data acquisition and processing system and method based on edge computing. The system includes: an adaptive data compression module for adaptively compressing the raw ocean data collected by the buoy terminal in real time, obtaining compressed ocean monitoring data and sending it to the edge computing node; a data processing module for the edge computing node to decompress and preprocess the compressed ocean monitoring data, and input the preprocessed ocean monitoring data into a lightweight ocean anomaly diagnosis model for real-time anomaly detection; and a data transmission module for transmitting the ocean monitoring data processed by the edge node and the anomaly detection results to an onshore data center using an incremental transmission method. By integrating sensor data compression technology with the lightweight ocean anomaly diagnosis model, the present invention achieves real-time acquisition, efficient compression, rapid processing, and anomaly detection of ocean data in open sea environments.

Description

Technical Field

[0001] The present invention relates to the technical field of marine environment monitoring data transmission, and in particular to a lightweight marine data acquisition and processing system and method based on edge computing. Background Art

[0002] With the continuous deepening of ocean exploration and development activities, the demand for real-time and accurate monitoring of marine environmental data is becoming increasingly urgent. However, the marine environment is complex and changeable, and traditional data collection and processing methods face many challenges: on the one hand, communication conditions in offshore areas are poor, network bandwidth is limited, and the transmission of large amounts of raw data is extremely difficult, which can easily cause data transmission delays or even interruptions, and cannot meet real-time requirements; on the other hand, if massive amounts of marine data are all transmitted to the cloud or land data center for processing, it will not only consume huge communication and storage resources, but also increase the time cost of data processing, making it difficult to achieve timely response to abnormal conditions in the marine environment.

[0003] At present, marine data acquisition terminals usually have high power consumption and are difficult to operate stably for a long time in the offshore environment. The data collected by the sensors are not effectively compressed and processed, and the data volume is huge, which puts great pressure on transmission and subsequent processing. At the same time, in terms of marine environmental anomaly detection, the existing detection models are often highly complex and cannot run efficiently on resource-limited edge devices, resulting in the inability to timely and accurately identify abnormal conditions in the marine environment. Summary of the Invention

[0004] The present invention provides a lightweight ocean data acquisition and processing system and method based on edge computing. By integrating sensor data compression technology with a lightweight ocean anomaly diagnosis module, it realizes real-time acquisition, efficient compression, rapid processing and anomaly detection of ocean data in offshore environments, thereby solving the above-mentioned problems existing in the prior art.

[0005] The present invention provides a lightweight ocean data acquisition and processing system based on edge computing, comprising:

[0006] Adaptive data compression module, used to adaptively compress the raw ocean data collected by the buoy terminal in real time, obtain compressed ocean monitoring data and send it to the edge computing node;

[0007] The data processing module is used by the edge computing node to decompress and preprocess the compressed ocean monitoring data, and input the preprocessed ocean monitoring data into the lightweight ocean anomaly diagnosis model for real-time anomaly detection;

[0008] The data transmission module is used to send the ocean monitoring data and anomaly detection results processed by the edge node to the onshore data center using an incremental transmission method.

[0009] Preferably, in a lightweight ocean data acquisition and processing system based on edge computing, the adaptive data compression module includes:

[0010] A data acquisition unit, used to acquire the original ocean monitoring data collected by the buoy terminal;

[0011] A data feature extraction unit is used to extract data change features from the original ocean monitoring data corresponding to the current buoy terminal to obtain data change features and data performance features;

[0012] a data compression unit, configured to determine a compression ratio of the original ocean monitoring data based on data variation characteristics and data presentation characteristics, in combination with a mapping relationship between the data characteristics and compression parameters, and compress the original ocean data according to the compression ratio to obtain compressed ocean monitoring data;

[0013] The internal transmission unit is used to send the compressed ocean monitoring data to the edge node of the buoy terminal.

[0014] Preferably, in a lightweight ocean data acquisition and processing system based on edge computing, the data compression unit further includes:

[0015] The data analysis subunit is used to obtain the original ocean data collected by multiple sensors within a preset collection period based on the edge node, generate multiple time series according to the time axis, and compare adjacent values of each time series to obtain the data error between adjacent data of the same original ocean data;

[0016] Obtaining data errors corresponding to the same time series, generating a data floating sequence, randomly selecting the data floating sequence based on a preset byte window, and calculating the data information entropy of the selected byte segment based on the selected byte segment;

[0017] a first mapping relationship determination subunit, configured to, when the data information entropy is less than a preset value, obtain data representation characteristics of the original ocean data of the data floating sequence corresponding to the data information entropy, obtain a plurality of compressed similar data based on the data representation characteristics and in combination with the data information entropy using big data acquisition technology, calculate a hash value corresponding to each type of compressed similar data, and determine, based on the hash value, a Hamming distance between the hash value and its corresponding standard hash value;

[0018] Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter;

[0019] Classify data floating sequences based on data information entropy and data performance characteristics, and determine the target data type of the original ocean data corresponding to the data floating sequences of the same category;

[0020] Extract and compare the data change characteristics of the target data types respectively to obtain the public data characteristics. Based on the public data characteristics and data performance characteristics corresponding to the target data types, obtain the data characteristics of the target data types;

[0021] Based on the data characteristics and the optimal compression parameters, a mapping relationship between the data characteristics and the compression parameters is established.

[0022] Preferably, in a lightweight ocean data acquisition and processing system based on edge computing, the data compression unit further includes:

[0023] The second mapping relationship determination subunit is configured to, when the data information entropy is greater than or equal to a preset value, determine the mutation node of the corresponding original ocean data based on the data error between adjacent data in the data floating sequence, segment the original ocean data in the time series based on the mutation node, obtain time subsequences corresponding to multiple time periods, and obtain each time subsequence separately:

[0024] Based on the data length of all time subsequences, the preset window is adaptively adjusted to obtain the full sequence adaptation window. Based on the full sequence adaptation window, the data sub-information entropy corresponding to each time subsequence is calculated respectively, and the data sub-information entropy of each time subsequence is compared to obtain the information entropy error.

[0025] When the information entropy errors are all within a preset range, the data realization characteristics of the original ocean data of the current type are obtained. Based on the data realization characteristics and the average information entropy corresponding to the data sub-information entropy, a large data collection technology is used to obtain a variety of compressed similar data, and a hash value corresponding to each type of compressed similar data is calculated. Based on the hash value, the Hamming distance between the hash value and its corresponding standard hash value is determined;

[0026] Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter;

[0027] The data change features of the original ocean data corresponding to each time subsequence are extracted and compared to obtain the public data sub-features. Based on the public data sub-features, data performance characteristics and the time points corresponding to the mutation nodes, the data features of the current type of ocean data are obtained.

[0028] Based on the data characteristics and the optimal compression parameters, establishing a mapping relationship between the data characteristics and the compression parameters;

[0029] When the information entropy errors are all outside the preset range, determining the data mutation distribution characteristics within the preset collection period based on the time point corresponding to the mutation point, and based on the data mutation distribution characteristics and combined with the performance characteristics of the current data, using big data collection technology to obtain a variety of compressed similar data, and calculating the hash value corresponding to each type of compressed similar data, and based on the hash value, determining the Hamming distance between the hash value and its corresponding standard hash value;

[0030] Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter;

[0031] Based on the data mutation distribution characteristics and data performance characteristics, the data characteristics of the current type of ocean data are obtained;

[0032] Based on the data characteristics and the optimal compression parameters, a mapping relationship between the data characteristics and the compression parameters is established.

[0033] Preferably, in a lightweight ocean data acquisition and processing system based on edge computing, the data processing module includes:

[0034] A training set generation unit is used to obtain massive historical ocean data as ocean sample data, and mark abnormal data on the ocean sample data to obtain an abnormal ocean data set;

[0035] The model training and deployment module is used to train the lightweight model for ocean anomaly diagnosis based on the abnormal ocean dataset, and deploy the trained lightweight model for ocean anomaly diagnosis to the edge computing node of each sensor.

[0036] Preferably, in a lightweight ocean data acquisition and processing system based on edge computing, the data processing module includes:

[0037] The data decompression processing unit is used to decompress the compressed ocean monitoring data received by the edge node to obtain decompressed data, and compare the decompressed data with the last received data of the corresponding type to determine whether the data has been sent or not;

[0038] If a change occurs, the decompressed data is pre-processed accordingly based on the type of the decompressed data, and the decompressed data is stored and recorded;

[0039] Otherwise, skip the decompressed data, control the edge node to keep the last anomaly detection result, and only receive the time point for storage and recording;

[0040] The anomaly monitoring unit is used to input the pre-processed decompressed data into the ocean anomaly diagnosis lightweight model to determine whether the current ocean data is abnormal.

[0041] Preferably, in a lightweight ocean data acquisition and processing system based on edge computing, the data transmission module includes:

[0042] A transmission processing unit, configured to generate an early warning report based on abnormal ocean data, and generate unchanged data instructions based on the type of changed data;

[0043] Generate transmission data packets based on warning reports, unchanged data instructions, and pre-processed changed ocean data;

[0044] An onshore data updating unit is configured to parse the transmission data packet received by the onshore data center and determine the type of unchanged ocean data based on the unchanged data instruction;

[0045] Obtaining receiving time points corresponding to unchanged ocean data and changed ocean data respectively, and updating the unchanged ocean data and changed ocean data respectively based on the receiving time points;

[0046] The onshore automatic early warning unit is used to generate corresponding abnormal early warning signals based on abnormal ocean data and issue early warnings when the onshore data center receives an early warning report.

[0047] The present invention provides a lightweight ocean data acquisition and processing method based on edge computing, comprising:

[0048] Step 1: Adaptively compress the raw ocean data collected by the buoy terminal in real time to obtain compressed ocean monitoring data and send it to the edge computing node;

[0049] Step 2: The edge computing node decompresses and preprocesses the compressed ocean monitoring data, and inputs the preprocessed ocean monitoring data into the ocean anomaly diagnosis lightweight model for real-time anomaly detection;

[0050] Step 3: Use incremental transmission to send the ocean monitoring data and anomaly detection results processed by the edge node to the onshore data center.

[0051] Preferably, in a lightweight ocean data acquisition and processing method based on edge computing, step 1 includes:

[0052] Obtain the original ocean monitoring data collected by the buoy terminal;

[0053] Extract data change features from the original ocean monitoring data corresponding to the current buoy terminal to obtain data change features and data performance features;

[0054] Based on the data change characteristics and data presentation characteristics, combined with the mapping relationship between the data characteristics and the compression parameters, the compression ratio of the original ocean monitoring data is determined, and the original ocean data is compressed according to the compression ratio to obtain compressed ocean monitoring data;

[0055] The compressed ocean monitoring data is sent to the edge node of the buoy terminal.

[0056] Compared with the prior art, the present invention has at least the following beneficial effects:

[0057] The present invention performs adaptive real-time compression on raw ocean data, which can significantly reduce the data volume and can be transmitted to edge computing nodes more quickly through communication links, effectively shortening the time interval from data acquisition to subsequent processing, improving the response speed of the entire system to changes in the ocean environment, and providing strong support for real-time monitoring and early warning. It also adjusts the compression strategy according to the dynamic characteristics of the data, ensuring that various types of ocean data can effectively reduce the data volume while maintaining the integrity and availability of the data during the compression process, thereby improving the reliability of data processing. Subsequently, the compressed ocean monitoring data is decompressed at the edge computing node to ensure that the data is restored to a state close to the original state, which can ensure that the data input into the lightweight model for ocean anomaly diagnosis is accurate and complete, thereby Improve the reliability of anomaly detection results, effectively avoid misjudgment or missed judgment due to data quality issues, and at the same time, the lightweight processing of the model also reduces the storage requirements during the operation of the edge node, and ensures the timely discovery of marine anomalies. The incremental transmission method is adopted, and only the changed data and anomaly detection results after processing by the edge node are transmitted compared with the last transmission. This can avoid repeated transmission of a large amount of unchanged data, significantly reduce the amount of data transmitted, thereby greatly shortening the data transmission time, improving data transmission efficiency, and further enhancing the real-time performance of the system. At the same time, the smaller amount of data transmission reduces the probability of data loss or error during transmission, and improves the stability and reliability of data transmission in the case of complex offshore communication environment and susceptible signals to interference.

[0058] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be achieved and obtained through the structures specifically pointed out in this application document.

[0059] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0061] Figure 1 This is a structural diagram of a lightweight ocean data acquisition and processing system based on edge computing in the present invention;

[0062] Figure 2 This is a structural diagram of an adaptive data compression module of a lightweight ocean data acquisition and processing system based on edge computing in the present invention;

[0063] Figure 3 This is a structural diagram of a data processing module of a lightweight ocean data acquisition and processing system based on edge computing in the present invention;

[0064] Figure 4 This is a structural diagram of the data transmission module of a lightweight ocean data acquisition and processing system based on edge computing in the present invention. DETAILED DESCRIPTION

[0065] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0066] Example 1:

[0067] The present invention provides a lightweight ocean data acquisition and processing system based on edge computing, such as Figure 1 Shown, including:

[0068] Adaptive data compression module, used to adaptively compress the raw ocean data collected by the buoy terminal in real time, obtain compressed ocean monitoring data and send it to the edge computing node;

[0069] The data processing module is used by the edge computing node to decompress and preprocess the compressed ocean monitoring data, and input the preprocessed ocean monitoring data into the lightweight ocean anomaly diagnosis model for real-time anomaly detection;

[0070] The data transmission module is used to send the ocean monitoring data and anomaly detection results processed by the edge node to the onshore data center using an incremental transmission method.

[0071] The beneficial effects of the above technical solution: The present invention performs adaptive real-time compression on the original ocean data, which can greatly reduce the data volume and can be transmitted to the edge computing node more quickly through the communication link, effectively shortening the time interval from data collection to subsequent processing, improving the response speed of the entire system to changes in the ocean environment, providing strong support for real-time monitoring and early warning, and adjusting the compression strategy according to the dynamic characteristics of the data, ensuring that all types of ocean data can effectively reduce the data volume while maintaining the integrity and availability of the data during the compression process, thereby improving the reliability of data processing. Subsequently, the compressed ocean monitoring data is decompressed at the edge computing node to ensure that the data is restored to a state close to the original state, which can ensure the accuracy of the data input into the lightweight model for ocean anomaly diagnosis , complete, thereby improving the reliability of anomaly detection results, effectively avoiding misjudgment or missed judgment due to data quality problems, and at the same time, the lightweight processing of the model also reduces the storage requirements during the operation of the edge node, and ensures the timely discovery of marine anomalies. The incremental transmission method is adopted, and only the changed data and anomaly detection results after processing by the edge node are transmitted compared with the last transmission, which can avoid repeated transmission of a large amount of unchanged data and significantly reduce the amount of data transmitted, thereby greatly shortening the data transmission time, improving data transmission efficiency, and further enhancing the real-time performance of the system. At the same time, the smaller amount of data transmission reduces the probability of data loss or error during transmission, and improves the stability and reliability of data transmission in the case of complex offshore communication environment and susceptible signals to interference. Example 2:

[0072] Based on Example 1, the adaptive data compression module, such as Figure 2 Shown, including:

[0073] A data acquisition unit, used to acquire the original ocean monitoring data collected by the buoy terminal;

[0074] A data feature extraction unit is used to extract data change features from the original ocean monitoring data corresponding to the current buoy terminal to obtain data change features and data performance features;

[0075] A data compression unit is configured to determine a compression ratio of the original ocean monitoring data based on the data variation characteristics and the data expression characteristics, in combination with a mapping relationship between the data characteristics and the compression parameters, and compress the original ocean data according to the compression ratio to obtain compressed ocean monitoring data;

[0076] The internal transmission unit is used to send the compressed ocean monitoring data to the edge node of the buoy terminal.

[0077] The beneficial effects of the above technical solution are as follows: the present invention directly collects raw ocean monitoring data from the buoy terminal, which can timely capture the dynamic changes of the ocean environment. By extracting data change features from the raw ocean monitoring data, it can deeply understand the inherent laws and change trends of the data. For example, for seawater temperature data, the temperature change rate over time and periodic fluctuation characteristics can be extracted; for wave height data, the change amplitude and frequency of wave crests and troughs can be obtained. At the same time, the extraction of data performance features, such as data distribution and discreteness, further enriches the understanding of the data. Different types of ocean monitoring data have different characteristics. The data feature extraction unit can analyze the uniqueness of each type of data, so that the entire adaptive data compression module can better adapt to various data situations, improve the flexibility and reliability of data processing, and determine the compression ratio of the raw ocean monitoring data based on the data change characteristics and data performance characteristics, combined with the mapping relationship between data features and compression parameters, to achieve data compression optimization, and quickly send the compressed ocean monitoring data to the edge node of the buoy terminal, ensuring the timeliness of ocean abnormality data. Example 3:

[0078] Based on embodiment 2, the data compression unit includes:

[0079] The data analysis subunit is used to obtain the original ocean data collected by multiple sensors within a preset collection period based on the edge node, generate multiple time series according to the time axis, and compare adjacent values of each time series to obtain the data error between adjacent data of the same original ocean data;

[0080] Obtaining data errors corresponding to the same time series, generating a data floating sequence, randomly selecting the data floating sequence based on a preset byte window, and calculating the data information entropy of the selected byte segment based on the selected byte segment;

[0081] a first mapping relationship determination subunit, configured to, when the data information entropy is less than a preset value, obtain data representation characteristics of the original ocean data of the data floating sequence corresponding to the data information entropy, obtain a plurality of compressed similar data based on the data representation characteristics and in combination with the data information entropy using big data acquisition technology, calculate a hash value corresponding to each type of compressed similar data, and determine, based on the hash value, a Hamming distance between the hash value and its corresponding standard hash value;

[0082] Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter;

[0083] Classify data floating sequences based on data information entropy and data performance characteristics, and determine the target data type of the original ocean data corresponding to the data floating sequences of the same category;

[0084] Extract and compare the data change characteristics of the target data types respectively to obtain the public data characteristics. Based on the public data characteristics and data performance characteristics corresponding to the target data types, obtain the data characteristics of the target data types;

[0085] Based on the data characteristics and the optimal compression parameters, establishing a mapping relationship between the data characteristics and the compression parameters;

[0086] The second mapping relationship determination subunit is configured to, when the data information entropy is greater than or equal to a preset value, determine the mutation node of the corresponding original ocean data based on the data error between adjacent data in the data floating sequence, segment the original ocean data in the time series based on the mutation node, obtain time subsequences corresponding to multiple time periods, and obtain each time subsequence separately:

[0087] Based on the data length of all time subsequences, the preset window is adaptively adjusted to obtain the full sequence adaptation window. Based on the full sequence adaptation window, the data sub-information entropy corresponding to each time subsequence is calculated respectively, and the data sub-information entropy of each time subsequence is compared to obtain the information entropy error.

[0088] When the information entropy errors are all within a preset range, the data realization characteristics of the original ocean data of the current type are obtained. Based on the data realization characteristics and the average information entropy corresponding to the data sub-information entropy, a large data collection technology is used to obtain a variety of compressed similar data, and a hash value corresponding to each type of compressed similar data is calculated. Based on the hash value, the Hamming distance between the hash value and its corresponding standard hash value is determined;

[0089] Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter;

[0090] The data change features of the original ocean data corresponding to each time subsequence are extracted and compared to obtain the public data sub-features. Based on the public data sub-features, data performance characteristics and the time points corresponding to the mutation nodes, the data features of the current type of ocean data are obtained.

[0091] Based on the data characteristics and the optimal compression parameters, establishing a mapping relationship between the data characteristics and the compression parameters;

[0092] When the information entropy errors are all outside the preset range, determining the data mutation distribution characteristics within the preset collection period based on the time point corresponding to the mutation point, and based on the data mutation distribution characteristics and combined with the performance characteristics of the current data, using big data collection technology to obtain a variety of compressed similar data, and calculating the hash value corresponding to each type of compressed similar data, and based on the hash value, determining the Hamming distance between the hash value and its corresponding standard hash value;

[0093] Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter;

[0094] Based on the data mutation distribution characteristics and data performance characteristics, the data characteristics of the current type of ocean data are obtained;

[0095] Based on the data characteristics and the optimal compression parameters, a mapping relationship between the data characteristics and the compression parameters is established.

[0096] In this embodiment, the selected byte segment refers to a data segment randomly selected by the element and the serial port in the actual sequence.

[0097] In this embodiment, a mutation node refers to a point where data suddenly increases or decreases rapidly.

[0098] In this embodiment, the standard hash value refers to a preset ideal hash value corresponding to each type of compressed similar data. Compressed similar data refers to data that has the same data characteristics (including data format and monitoring time, such as image, number, discrete, continuous, etc.) as the original ocean data and has the same information entropy within the length corresponding to the preset window.

[0099] The beneficial effects of the above technical solution are as follows: the present invention efficiently starts from the original ocean data collected by multiple sensors, generates a time series according to the time axis, and obtains data error by comparing adjacent values, and then generates a data floating sequence, preliminarily sorting out the change pattern of the data, laying the foundation for subsequent in-depth analysis. For example, when analyzing seawater temperature data, abnormal temperature fluctuations can be quickly discovered by comparing adjacent values, providing clues for subsequent judgment whether there is a mutation in the data. At the same time, the data information entropy of the data floating sequence is calculated, and different strategies are used for processing according to the size of the information entropy to further explore the potential characteristics of the data, such as the randomness and regularity of the data, to provide a basis for subsequent data classification and compression; in terms of determining the compression parameters, different methods are adopted for different situations: when the data information entropy is less than the preset value, by obtaining the data performance characteristics, combined with big data collection technology to obtain compressed similar data, and using the Hamming distance between the hash value and the standard hash value to screen out the compressed similar data with the minimum loss, thereby determining the optimal compression parameters, which can achieve efficient data compression while ensuring minimal data information loss. For example, for relatively stable ocean data, such as long-term, stable salinity data for a specific ocean area, this method can identify the most appropriate compression method, reducing data storage space while preserving the data's key features to the greatest extent possible. When the data entropy is greater than or equal to a preset value, the method further considers data mutation nodes and segments the time series. By calculating the data sub-entropy and entropy error for each time subsequence, optimal compression parameters are determined for each case. This approach addresses data mutations, enabling more precise adaptation to complex and volatile ocean data, ensuring high-quality compression even when data mutations occur, and avoiding information loss or poor compression performance caused by these mutations. Furthermore, based on data entropy and data performance characteristics, data floating series are classified to identify target data types and extract public and data features. This helps rationally categorize complex and diverse ocean data according to their inherent characteristics, facilitating subsequent targeted processing and analysis of different types of data. For example, marine biomass data can be distinguished from marine hydrological data, and their respective data variation characteristics can be extracted, providing more targeted data support for marine ecological research and marine environmental monitoring. At the same time, by extracting and comparing the data features of different time subsequences, we can understand the changes in the data in different time periods in more detail. In particular, when there are mutations in the data, we can accurately grasp the mutation distribution characteristics of the data, providing strong data support for in-depth research on the dynamic changes of the marine environment. The present invention establishes an effective mapping relationship: by establishing a mapping relationship between data features and compression parameters, it greatly facilitates the storage, transmission and processing of subsequent marine data. In practical applications, when new raw marine data arrives, the system can quickly find the corresponding optimal compression parameters from the established mapping relationship based on its data features, thereby achieving rapid data compression and improving data processing efficiency.This mapping also helps accurately restore the original data characteristics based on the compression parameters during data decompression, ensuring data accuracy and integrity. Furthermore, using different compression parameters for different data types based on their characteristics can achieve a better balance between storage space and data quality, meeting the needs of marine environmental monitoring systems for massive data storage and efficient processing.

[0100] Establishing a mapping relationship between data features and compression parameters is conducive to shortening the time interval from data collection to subsequent processing, improving the response speed of the entire system to changes in the marine environment while ensuring that the data is restored to a near-original state. It can ensure that the data input into the lightweight model of marine anomaly diagnosis is accurate and complete, thereby improving the reliability of anomaly detection results and effectively avoiding misjudgments or missed judgments due to data quality issues. Example 4:

[0101] Based on Example 1, the data processing module, such as Figure 3 Shown, including:

[0102] A training set generation unit is used to obtain massive historical ocean data as ocean sample data, and mark abnormal data on the ocean sample data to obtain an abnormal ocean data set;

[0103] The model training and deployment module is used to train the lightweight model for ocean anomaly diagnosis based on the abnormal ocean dataset, and deploy the trained lightweight model for ocean anomaly diagnosis to the edge computing node of each sensor.

[0104] The beneficial effects of the above technical solution are as follows: the present invention realizes the training and deployment of a lightweight model for marine anomaly diagnosis, and integrates sensor data compression technology with the lightweight model for marine anomaly diagnosis, providing a basis for the rapid processing and timely discovery of marine anomalies. Example 5:

[0105] Based on Example 1, the data processing module, such as Figure 3 Shown, including:

[0106] The data decompression processing unit is used to decompress the compressed ocean monitoring data received by the edge node to obtain decompressed data, and compare the decompressed data with the last received data of the corresponding type to determine whether the data has been sent or not;

[0107] If there is a change, the decompressed data is preprocessed accordingly based on the type of the decompressed data, and the decompressed data is stored and recorded.

[0108] Otherwise, skip the decompressed data, control the edge node to keep the last anomaly detection result, and only receive the time point for storage and recording;

[0109] The anomaly monitoring unit is used to input the pre-processed decompressed data into the ocean anomaly diagnosis lightweight model to determine whether the current ocean data is abnormal.

[0110] The beneficial effects of the above technical solution are as follows: the present invention decompresses the compressed ocean monitoring data after receiving it at the edge node, thereby ensuring the integrity of the data used for abnormal monitoring judgment. At the same time, the decompressed data is compared with the previous received data of the corresponding type to determine whether the data sent has changed, providing a basis for the subsequent incremental transmission selection and abnormal monitoring judgment. The pre-processed decompressed data is input into the lightweight model of ocean anomaly diagnosis to determine whether the current ocean data is abnormal, thereby realizing the judgment of automatic monitoring results before the data is sent to the onshore data center, ensuring the timely discovery of ocean anomalies. Example 6:

[0111] Based on Example 1, the data transmission module, such as Figure 4 Shown, including:

[0112] A transmission processing unit, configured to generate an early warning report based on abnormal ocean data, and generate unchanged data instructions based on the type of changed data;

[0113] Generate transmission data packets based on warning reports, unchanged data instructions, and pre-processed changed ocean data;

[0114] An onshore data updating unit is configured to parse the transmission data packet received by the onshore data center and determine the type of unchanged ocean data based on the unchanged data instruction;

[0115] Obtaining receiving time points corresponding to unchanged ocean data and changed ocean data respectively, and updating the unchanged ocean data and changed ocean data respectively based on the receiving time points;

[0116] The onshore automatic early warning unit is used to generate corresponding abnormal early warning signals based on abnormal ocean data and issue early warnings when the onshore data center receives an early warning report.

[0117] The beneficial effects of the above technical solution: the present invention adopts an incremental transmission method to effectively reduce the amount of data transmission, shorten the data transmission time, ensure the real-time and timeliness of the data received by the onshore data center, which is conducive to improving the response speed of sudden ocean conditions, and parse the transmission data packet after receiving it at the onshore data center, and determine the type of unchanged ocean data based on the unchanged data instruction; obtain the corresponding receiving time points of the unchanged ocean data and the changed ocean data respectively, and update the unchanged ocean data and the changed ocean data respectively based on the receiving time points, ensuring that even if only the changed data is transmitted, the unchanged data is recorded in time, providing comprehensive and complete data support for the subsequent analysis of the laws of ocean activities by relevant departments, and at the same time, when the onshore data center receives the early warning report, it generates a corresponding abnormal early warning signal based on the abnormal ocean data, and issues an early warning, ensuring that the relevant departments can discover abnormal ocean conditions in time. Example 7:

[0118] The present invention provides a lightweight ocean data acquisition and processing method based on edge computing, comprising:

[0119] Step 1: Adaptively compress the raw ocean data collected by the buoy terminal in real time to obtain compressed ocean monitoring data and send it to the edge computing node;

[0120] Step 2: The edge computing node decompresses and preprocesses the compressed ocean monitoring data, and inputs the preprocessed ocean monitoring data into the ocean anomaly diagnosis lightweight model for real-time anomaly detection;

[0121] Step 3: Use incremental transmission to send the ocean monitoring data and anomaly detection results processed by the edge node to the onshore data center.

[0122] The beneficial effects of the above technical solution: The present invention performs adaptive real-time compression on the original ocean data, which can greatly reduce the data volume and can be transmitted to the edge computing node more quickly through the communication link, effectively shortening the time interval from data collection to subsequent processing, improving the response speed of the entire system to changes in the ocean environment, providing strong support for real-time monitoring and early warning, and adjusting the compression strategy according to the dynamic characteristics of the data, ensuring that all types of ocean data can effectively reduce the data volume while maintaining the integrity and availability of the data during the compression process, thereby improving the reliability of data processing. Subsequently, the compressed ocean monitoring data is decompressed at the edge computing node to ensure that the data is restored to a state close to the original state, which can ensure the accuracy of the data input into the lightweight model for ocean anomaly diagnosis , complete, thereby improving the reliability of anomaly detection results, effectively avoiding misjudgment or missed judgment due to data quality problems, and at the same time, the lightweight processing of the model also reduces the storage requirements during the operation of the edge node, and ensures the timely discovery of marine anomalies. The incremental transmission method is adopted, and only the changed data and anomaly detection results after processing by the edge node are transmitted compared with the last transmission, which can avoid repeated transmission of a large amount of unchanged data and significantly reduce the amount of data transmitted, thereby greatly shortening the data transmission time, improving data transmission efficiency, and further enhancing the real-time performance of the system. At the same time, the smaller amount of data transmission reduces the probability of data loss or error during transmission, and improves the stability and reliability of data transmission in the case of complex offshore communication environment and susceptible signals to interference. Example 8:

[0123] Based on Example 7, a lightweight ocean data collection and processing method based on edge computing, step 1, includes:

[0124] Obtain the original ocean monitoring data collected by the buoy terminal;

[0125] Extract data change features from the original ocean monitoring data corresponding to the current buoy terminal to obtain data change features and data performance features;

[0126] Based on the data change characteristics and data presentation characteristics, combined with the mapping relationship between the data characteristics and the compression parameters, the compression ratio of the original ocean monitoring data is determined, and the original ocean data is compressed according to the compression ratio to obtain compressed ocean monitoring data;

[0127] The compressed ocean monitoring data is sent to the edge node of the buoy terminal.

[0128] The beneficial effects of the above technical solution are as follows: the present invention directly collects raw ocean monitoring data from the buoy terminal, which can timely capture the dynamic changes of the ocean environment. By extracting data change features from the raw ocean monitoring data, it can deeply understand the inherent laws and change trends of the data. For example, for seawater temperature data, the temperature change rate over time and periodic fluctuation characteristics can be extracted; for wave height data, the change amplitude and frequency of wave crests and troughs can be obtained. At the same time, the extraction of data performance features, such as data distribution and discreteness, further enriches the understanding of the data. Different types of ocean monitoring data have different characteristics. The data feature extraction unit can analyze the uniqueness of each type of data, so that the entire adaptive data compression module can better adapt to various data situations, improve the flexibility and reliability of data processing, and determine the compression ratio of the raw ocean monitoring data based on the data change characteristics and data performance characteristics, combined with the mapping relationship between data features and compression parameters, to achieve data compression optimization, and quickly send the compressed ocean monitoring data to the edge node of the buoy terminal, ensuring the timeliness of ocean abnormality data.

[0129] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A lightweight ocean data acquisition and processing system based on edge computing, characterized in that: include: Adaptive data compression module, used to adaptively compress the raw ocean data collected by the buoy terminal in real time, obtain compressed ocean monitoring data and send it to the edge computing node; The data processing module is used by the edge computing node to decompress and preprocess the compressed ocean monitoring data, and input the preprocessed ocean monitoring data into the lightweight ocean anomaly diagnosis model for real-time anomaly detection; The data transmission module is used to send the ocean monitoring data and anomaly detection results processed by the edge node to the onshore data center using an incremental transmission method; The adaptive data compression module includes: A data acquisition unit, used to acquire the original ocean monitoring data collected by the buoy terminal; A data feature extraction unit is used to extract data change features from the original ocean monitoring data corresponding to the current buoy terminal to obtain data change features and data performance features; a data compression unit, configured to determine a compression ratio of the original ocean monitoring data based on data variation characteristics and data presentation characteristics, in combination with a mapping relationship between the data characteristics and compression parameters, and compress the original ocean data according to the compression ratio to obtain compressed ocean monitoring data; an internal transmission unit for transmitting compressed ocean monitoring data to an edge node of the buoy terminal; Wherein, the data compression unit includes: The data analysis subunit is used to obtain the original ocean data collected by multiple sensors within a preset collection period based on the edge node, generate multiple time series according to the time axis, and compare adjacent values of each time series to obtain the data error between adjacent data of the same original ocean data; Obtaining data errors corresponding to the same time series, generating a data floating sequence, randomly selecting the data floating sequence based on a preset byte window, and calculating the data information entropy of the selected byte segment based on the selected byte segment; a first mapping relationship determination subunit, configured to, when the data information entropy is less than a preset value, obtain data representation characteristics of the original ocean data of the data floating sequence corresponding to the data information entropy, obtain a plurality of compressed similar data based on the data representation characteristics and in combination with the data information entropy using a big data acquisition technology, calculate a hash value corresponding to each type of compressed similar data, and determine, based on the hash value, a Hamming distance between the hash value and its corresponding standard hash value; Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter; Classify data floating sequences based on data information entropy and data performance characteristics, and determine the target data type of the original ocean data corresponding to the data floating sequences of the same category; Extract and compare the data change characteristics of the target data types respectively to obtain the public data characteristics. Based on the public data characteristics and data performance characteristics corresponding to the target data types, obtain the data characteristics of the target data types; Based on the data characteristics and the optimal compression parameters, a mapping relationship between the data characteristics and the compression parameters is established.

2. A lightweight ocean data acquisition and processing system based on edge computing according to claim 1, characterized in that: The data compression unit further comprises: The second mapping relationship determination subunit is configured to, when the data information entropy is greater than or equal to a preset value, determine the mutation node of the corresponding original ocean data based on the data error between adjacent data in the data floating sequence, segment the original ocean data in the time series based on the mutation node, obtain time subsequences corresponding to multiple time periods, and obtain each time subsequence separately: Based on the data length of all time subsequences, the preset window is adaptively adjusted to obtain the full sequence adaptation window. Based on the full sequence adaptation window, the data sub-information entropy corresponding to each time subsequence is calculated respectively, and the data sub-information entropy of each time subsequence is compared to obtain the information entropy error. When the information entropy errors are all within a preset range, the data realization characteristics of the original ocean data of the current type are obtained. Based on the data realization characteristics and the average information entropy corresponding to the data sub-information entropy, a large data collection technology is used to obtain a variety of compressed similar data, and a hash value corresponding to each type of compressed similar data is calculated. Based on the hash value, the Hamming distance between the hash value and its corresponding standard hash value is determined; Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter; Extract and compare the data change features of the original ocean data corresponding to each time subsequence to obtain the public data sub-features. Based on the public data sub-features, data performance characteristics and the time points corresponding to the mutation nodes, obtain the data features of the current type of ocean data; Based on the data characteristics and the optimal compression parameters, a mapping relationship between the data characteristics and the compression parameters is established.

3. A lightweight ocean data acquisition and processing system based on edge computing according to claim 2, characterized in that: The second mapping relationship determination subunit is further configured to: When the information entropy errors are all outside the preset range, determining the data mutation distribution characteristics within the preset collection period based on the time point corresponding to the mutation point, and based on the data mutation distribution characteristics and combined with the performance characteristics of the current data, using big data collection technology to obtain a variety of compressed similar data, and calculating the hash value corresponding to each type of compressed similar data, and based on the hash value, determining the Hamming distance between the hash value and its corresponding standard hash value; Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter; Based on the data mutation distribution characteristics and data performance characteristics, the data characteristics of the current type of ocean data are obtained; Based on the data characteristics and the optimal compression parameters, a mapping relationship between the data characteristics and the compression parameters is established.

4. The lightweight ocean data acquisition and processing system based on edge computing according to claim 1 is characterized in that: Data processing module, including: A training set generation unit is used to obtain massive historical ocean data as ocean sample data, and mark abnormal data on the ocean sample data to obtain an abnormal ocean data set; The model training and deployment module is used to train the lightweight model for ocean anomaly diagnosis based on the abnormal ocean dataset, and deploy the trained lightweight model for ocean anomaly diagnosis to the edge computing node of each sensor.

5. The lightweight ocean data acquisition and processing system based on edge computing according to claim 1 is characterized in that: Data processing module, including: The data decompression processing unit is used to decompress the compressed ocean monitoring data received by the edge node to obtain decompressed data, and compare the decompressed data with the last received data of the corresponding type to determine whether the data has been sent or not; If a change occurs, the decompressed data is pre-processed accordingly based on the type of the decompressed data, and the decompressed data is stored and recorded; Otherwise, skip the decompressed data, control the edge node to keep the last anomaly detection result, and only receive the time point for storage and recording; The anomaly monitoring unit is used to input the pre-processed decompressed data into the ocean anomaly diagnosis lightweight model to determine whether the current ocean data is abnormal.

6. A lightweight ocean data acquisition and processing system based on edge computing according to claim 1, characterized in that: Data transmission module, including: A transmission processing unit, configured to generate an early warning report based on abnormal ocean data and generate unchanged data instructions based on unchanged data types; Generate transmission data packets based on warning reports, unchanged data instructions, and pre-processed changed ocean data; An onshore data updating unit is configured to parse the transmission data packet received by the onshore data center and determine the type of unchanged ocean data based on the unchanged data instruction; Obtaining receiving time points corresponding to unchanged ocean data and changed ocean data respectively, and updating the unchanged ocean data and changed ocean data respectively based on the receiving time points; The onshore automatic early warning unit is used to generate corresponding abnormal early warning signals based on abnormal ocean data and issue early warnings when the onshore data center receives an early warning report.

7. A lightweight ocean data acquisition and processing method based on edge computing, characterized in that: include: Step 1: Adaptively compress the raw ocean data collected by the buoy terminal in real time to obtain compressed ocean monitoring data and send it to the edge computing node; Step 2: The edge computing node decompresses and preprocesses the compressed ocean monitoring data, and inputs the preprocessed ocean monitoring data into the ocean anomaly diagnosis lightweight model for real-time anomaly detection; Step 3: Using incremental transmission, the ocean monitoring data and anomaly detection results processed by the edge node are sent to the onshore data center; Wherein, step 1 includes: obtaining original ocean monitoring data collected by the buoy terminal; Extract data change features from the original ocean monitoring data corresponding to the current buoy terminal to obtain data change features and data performance features; Based on the data change characteristics and data presentation characteristics, combined with the mapping relationship between the data characteristics and the compression parameters, the compression ratio of the original ocean monitoring data is determined, and the original ocean data is compressed according to the compression ratio to obtain compressed ocean monitoring data; Sending compressed ocean monitoring data to the edge node of the buoy terminal; The establishment of the mapping relationship between data features and compression parameters includes: Based on edge nodes, we obtain the raw ocean data collected by various sensors within a preset collection period, generate multiple time series according to the time axis, and compare adjacent values of each time series to obtain the data error between adjacent data of the same raw ocean data. Obtaining data errors corresponding to the same time series, generating a data floating sequence, randomly selecting the data floating sequence based on a preset byte window, and calculating the data information entropy of the selected byte segment based on the selected byte segment; When the data information entropy is less than a preset value, obtaining data performance characteristics of the original ocean data of the data floating sequence corresponding to the data information entropy, based on the data performance characteristics and in combination with the data information entropy, adopting big data acquisition technology to obtain a plurality of compressed similar data, and calculating a hash value corresponding to each compressed similar data, and based on the hash value, determining the Hamming distance between the hash value and its corresponding standard hash value; Screening based on the Hamming distance to obtain minimum loss compressed similar data, and obtaining the compression parameter corresponding to the minimum loss compressed similar data as the optimal compression parameter; Classify data floating sequences based on data information entropy and data performance characteristics, and determine the target data type of the original ocean data corresponding to the data floating sequences of the same category; Extract and compare the data change characteristics of the target data types respectively to obtain the public data characteristics. Based on the public data characteristics and data performance characteristics corresponding to the target data types, obtain the data characteristics of the target data types; Based on the data characteristics and the optimal compression parameters, a mapping relationship between the data characteristics and the compression parameters is established.

Images