A land-sea long-distance data transmission system

Abstract

The application discloses a kind of sea-land remote data transmission systems, it is related to information communication technical field, including: configuration module, offshore data sending module, land data receiving module, land data import module and sea-land communication module;Offshore data sending module is used to collect offshore data, according to corresponding time span according to time sequence split and form compression file and check information, time stamp and id information;With fixed time interval to land data receiving module sends signal and judges whether the current network state is normal, in normal time to land data receiving module sends compression file and check information, generates data transmission progress table;In abnormal time, continue to send file and check information to land data receiving module until network state is normal;The system can still work continuously in the state of network and no network or unstable network, improve file transmission success rate, reduce network consumption, greatly shorten processing time.

Description

Technical Field

[0001] This invention relates to the field of information and communication technology, and in particular to a long-distance data transmission system for land and sea. Background Technology

[0002] Offshore platforms have numerous data acquisition devices and involve a massive amount of data transmission, posing a severe challenge to data transmission security, stability, and synchronization. The network for sea-to-land data transmission is highly unstable, experiencing intermittent failures. Currently, existing solutions for long-distance sea-to-land data transmission have the following problems:

[0003] 1. Common sea-land data transmission is based on synchronous processing, that is: waiting for network access -> data export -> data direct transmission -> receiving and importing. If data is only processed and sent when there is network access, the data processing efficiency is low.

[0004] 2. During the transmission process, the transmitted files usually need to be completely transmitted before they can be used. Splitting a large file into smaller files for transmission also requires complete transmission and reassembly before it can be used. If the network is interrupted during the transmission process and the file is not completely transmitted, it is necessary to wait until the network is reconnected before retransmitting. This is inefficient and will repeatedly send files, consuming memory.

[0005] 3. The amount of data collected by the sensors is large. A single ".csv" file usually contains tens of millions of data entries. However, the data transmission process is usually handled by a single file and a single thread. The network on the marine platform is unstable and only available at certain times. Therefore, multiple untransmitted files may accumulate. When multiple files accumulate, using single-file single-thread processing will result in extremely low efficiency of land-sea data transmission and waste of network resources.

[0006] Therefore, the existing sea-land data transmission system network is unstable. During data transmission, it is necessary to establish network connections in real time to determine the network status. This results in a large amount of resources being used for network connection establishment, leading to low data processing efficiency. Furthermore, due to network issues, the data files transmitted in a single transmission are too large when there is a network connection. Using single-threaded data processing is too slow, resulting in extremely low transmission efficiency and wasted network resources. If the network is interrupted during transmission and the file is not completely transmitted, it is necessary to wait for the network to be reconnected before retransmitting. This will result in repeated file transmission, consuming memory and affecting practical applications. Summary of the Invention

[0007] This invention provides a land-sea remote data transmission system to overcome the above-mentioned technical problems.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A sea-land long-distance data transmission system includes: a configuration module, a sea data transmission module, a land data receiving module, a land data import module, and a sea-land communication module;

[0010] The configuration module is used to configure the corresponding data acquisition path, data storage path and time span according to the data type of the marine data, and to calculate the number of corresponding processing threads according to the number of files of different data types of marine data, and to calculate the number of processing threads to be allocated to each file according to the time span of different data types.

[0011] The maritime data transmission module is used to collect maritime data, form corresponding raw data files according to the data type of the maritime data, and split the maritime data in chronological order according to the number of raw data files, the corresponding time span and the corresponding processing thread, and form compressed files and corresponding verification information, timestamps and ID information.

[0012] The maritime data transmission module is also used to send signals to the land data receiving module at fixed time intervals, and to determine whether the current network status is normal based on whether the feedback signal sent by the land data receiving module can be received. When the network status is normal, it sends a compressed file and corresponding verification information to the land data receiving module, and generates a data transmission progress table that represents the information of the compressed file sent by the maritime data transmission module and the real-time reception status of the compressed file. When the network status is abnormal, it performs the aforementioned operation process of collecting maritime data and generating a compressed file until the network status is normal, and then continues to send the generated compressed file and corresponding verification information to the land data receiving module.

[0013] The land data receiving module is used to receive the transmitted compressed file and the corresponding verification information, perform verification according to the verification information, determine whether the transmitted compressed file has been received, whether the compressed file is correct and complete, and save the correct and complete compressed files to form a data record table; the data record table is used to represent the information of the received compressed file and the real-time saving status of the corresponding compressed file.

[0014] The land data import module is used to decompress the compressed file saved by the land data receiving module according to the data record table, store the decompressed data in the database for retrieval, and update the storage status of the corresponding compressed file in the data record table.

[0015] The sea-land communication module is used to monitor the data transmission status between the sea data transmission module and the land data receiving module in real time. After receiving the data reception result and verification result from the land data receiving module, it sends feedback to the sea data transmission module.

[0016] After receiving the data storage status feedback from the land data receiving module, it sends feedback to the marine data sending module.

[0017] Furthermore, the configuration module calculates the number of processing threads corresponding to the number of files of different data types in the maritime data, and calculates the number of processing threads to be allocated to each file based on the time span of different data types, including:

[0018] The configuration field sets the number of files. If the number of files of a certain data type is greater than the configuration field, a basic thread is allocated to each file of that data type for processing. If the number of files of a certain data type is less than the configuration field, a basic thread is allocated to all files of that data type for processing.

[0019] A configuration field sets an upper limit for the time interval. If the time interval of data in a certain data file of a certain data type is greater than the set configuration field, the data in the file is split according to the time span, and the number of threads allocated after calculating the time span is obtained, and the split data is processed in parallel. If the time interval of data in a certain data file is less than the set configuration field, one thread is allocated to process the data in the file.

[0020] Furthermore, the formula for calculating the time span is shown in formula (1).

[0021] (1)

[0022] in, Indicates the time span, This indicates network bandwidth.

[0023] Furthermore, the maritime data is split chronologically according to the corresponding time span and formed into compressed files containing corresponding verification information, timestamps, and ID information, including:

[0024] S11. The maritime data is split into data packets according to the corresponding time span in chronological order, and the data packets are compressed twice to generate a compressed file;

[0025] S12. Generate verification information based on the filename of the compressed file. The verification information includes the total file length, MD5 checksum, filename, file type, file date, and file path. Add timestamp and ID information to the compressed file.

[0026] S13. Encapsulate the verification information into header information in JSON format and temporarily store the compressed file.

[0027] Furthermore, timestamp and ID information are added to the compressed file, including:

[0028] For each compressed file that needs to be transmitted, perform a hash calculation and encrypt the calculation result with MD5. The encrypted result is used as the ID information of the compressed file.

[0029] Add a timestamp to each compressed file that needs to be transmitted. The timestamp represents the time span during which the data in the compressed file was collected.

[0030] Furthermore, the land data receiving module is used to receive the transmitted compressed file and corresponding verification information, perform verification according to the verification information, determine whether the transmitted compressed file has been received, whether the compressed file is correct and complete, and save the correct and complete compressed file to form a data record table, including:

[0031] S21. Receive the compressed file and corresponding verification information transmitted by the maritime data transmission module, perform verification according to the verification information, and if the verification result is correct, receive the compressed file transmitted by the maritime data transmission module and save the data compressed package according to the header to form a data record table.

[0032] S22. If the verification result is incorrect, feedback is sent through the maritime data transmission module, and the maritime data transmission module resends the compressed file and the corresponding verification information.

[0033] S23. Compare the header data with the data record table to determine whether the verification information and corresponding compressed file transmitted by the maritime data transmission module have been received. If they have been received, terminate the current reception. If they have not been received, repeat steps S21-S22 to verify and save the verification information and update the data record table.

[0034] Furthermore, the land data import module is used to decompress the compressed file saved by the land data receiving module according to the data record table, store the decompressed data in the database for retrieval, and update the storage status of the corresponding compressed file in the data record table, including:

[0035] S31. Decompress the compressed file according to the data record table;

[0036] S32. According to the data record table, import the decompressed file data into the database and update the save status of the corresponding compressed file in the data record table.

[0037] Beneficial effects: This invention provides a sea-land remote data transmission system that can continue to operate even with and without network connectivity, or in unstable network conditions, solving the problem that traditional systems can only operate with network connectivity. By configuring the time span to reduce data granularity, smaller granular data has a higher success rate in transmitting data under unstable network conditions. Synchronously updating the data transmission progress table of the sea data transmission module and the data record table of the land data receiving module avoids duplicate data transmission by the sea data transmission module, reducing network consumption. Based on the number of sea and land data files, multiple threads are allocated for processing the sea and land data, allowing different threads to be assigned to different cores to run simultaneously, making full use of computer hardware resources and processing multiple files in parallel, greatly shortening processing time. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A system block diagram of a land-sea remote data transmission system provided by the present invention;

[0040] Figure 2 This invention provides a system data flow diagram for a land-sea remote data transmission system. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] This embodiment provides a sea-land long-distance data transmission system, such as Figure 1 As shown, it includes: a configuration module, a maritime data transmission module, a land data reception module, a land data import module, and a maritime-land communication module;

[0043] The configuration module is used to configure the corresponding data acquisition path, data storage path and time span according to the data type of the marine data, and to calculate the number of corresponding processing threads according to the number of files of different data types of marine data, and to calculate the number of processing threads to be allocated to each file according to the time span of different data types.

[0044] The maritime data transmission module is used to collect maritime data, form corresponding raw data files according to the data type of the maritime data, and split the maritime data in chronological order according to the number of raw data files, the corresponding time span and the corresponding processing thread, and form compressed files and corresponding verification information, timestamps and ID information.

[0045] The maritime data transmission module is also used to send signals to the land data receiving module at fixed time intervals, and to determine whether the current network status is normal based on whether the feedback signal sent by the land data receiving module can be received. When the network status is normal, it sends a compressed file and corresponding verification information to the land data receiving module, and generates a data transmission progress table that represents the information of the compressed file sent by the maritime data transmission module and the real-time reception status of the compressed file. When the network status is abnormal, it performs the aforementioned operation process of collecting maritime data and generating a compressed file until the network status is normal, and then continues to send the generated compressed file and corresponding verification information to the land data receiving module.

[0046] The land data receiving module is used to receive the transmitted compressed file and the corresponding verification information, perform verification according to the verification information, determine whether the transmitted compressed file has been received, whether the compressed file is correct and complete, and save the correct and complete compressed files to form a data record table; the data record table is used to represent the information of the received compressed file and the real-time saving status of the corresponding compressed file.

[0047] The land data import module is used to decompress the compressed file saved by the land data receiving module according to the data record table, store the decompressed data in the database for retrieval, and update the storage status of the corresponding compressed file in the data record table.

[0048] The sea-land communication module is used to monitor the data transmission status between the sea data transmission module and the land data receiving module in real time. After receiving the data reception result and verification result from the land data receiving module, it sends feedback to the sea data transmission module.

[0049] After receiving the data storage status feedback from the land data receiving module, it sends feedback to the marine data sending module.

[0050] Specifically, such as Figure 1 and Figure 2As shown, the configuration module includes a marine configuration module and a land configuration module. The marine configuration module is used to configure the data acquisition path and time span according to the data type of the marine data, and to calculate the number of corresponding processing threads according to the number of files of different data types. It also calculates the number of processing threads to be allocated to each file according to the time span of different data types. The land configuration module is used to configure the data storage path. The configuration content includes, but is not limited to, the start and end time of transmission, the allowed file formats for transmission, and the file transmission rest interval.

[0051] In this embodiment, the configuration module configures different time spans for maritime data, as shown in Table 1.

[0052] Table 1 Time Span Configuration Table

[0053]

[0054] Specifically, the marine data transmission module is used to collect marine data and form corresponding raw data files according to the data type. Each raw data file contains data from a sensor over a period of time. The module manages data transmission and constructs a data transmission schedule. The data transmission schedule records information about compressed files sent by the marine data transmission module and the real-time reception status of the compressed files. Based on the current schedule data and time span, the module calculates the time period for collecting new marine data and calls the configuration module to manage, call, and control the land data receiving module and the land data import module.

[0055] like Figure 2 As shown, the time span specified in the schedule data and configuration module includes the configured time type and span field. The types are 1: seconds, 2: minutes, 3: hours, and 4: days. For example, if the type is set to 2 and the span field is set to 10, it means that the file will be split into smaller files every 10 minutes at the minute level. Or, if the type is set to 1 and the span is set to 30, it means that the file will be split into smaller files every 30 seconds at the second level. By reading the start and end times of the file, the overall span of the data in the file is calculated, and the number of file segments that can be split into is calculated based on the custom time span.

[0056] The maritime data is saved into multiple raw data files according to the sensor type. Two different processing threads are allocated to the maritime data based on the number of raw data files, the corresponding time span, and the threads calculated based on the number of files and the time span, respectively. The allocated processing threads then split the maritime data files into several data packets in chronological order according to the time span. Each data packet contains maritime data for a specific time period, which can be directly accessed for subsequent processing after transmission. The data packets are first compressed to generate a ".csv" file, and then a second compression is performed using the Univocity method to compress the ".csv" file into a ".zip" file. The file size after these two compressions is only one-tenth of the original data file size. Verification information, timestamps, and ID information are generated from the compressed file.

[0057] The marine data transmission module sends signals to the land data receiving module at fixed time intervals, and determines whether the current network status is normal based on whether the feedback signal sent by the land data receiving module can be received. If the current network status is normal, the module sends verification information and the corresponding compressed file to the land data receiving module. If the current network status is abnormal, the module continues to perform data acquisition, export, and compression operations, temporarily stores the data, and sends it together when communication is normal.

[0058] Specifically, the sea-land communication module is used to monitor the data transmission status between the sea data sending module and the land data receiving module in real time. The sea-land communication module can realize the transmission progress confirmation of the sea data sending end. First, the sea data sending module sends the compressed file and the corresponding verification information to the land data receiving module, and records the names of the sent files in the data transmission progress table.

[0059] The land data receiving module receives and verifies the data. If the verification information is correct, it saves the data and sends feedback to the land-sea communication module that the file verification is correct and the data has been saved. After receiving the information, the land-sea communication module sends the information to the maritime data sending module. The maritime data sending module records the file as saved in the data transmission progress table. If there is an error, it sends feedback to the land-sea communication module, which then informs the maritime data sending module that the file verification failed and the data was not saved, and the file needs to be resent.

[0060] The land-sea communication module also functions as follows: after the land data import module saves the data into the database, it updates the data record table to confirm that the file has been saved. The land data receiving module then retrieves the data record table and sends the saved result to the maritime data sending module via the land-sea transmission module. If the saving is successful, both the maritime data sending module and the land data receiving module delete the successfully saved data. If the saving fails, it first determines whether the failure is due to data reception failure. If so, the maritime data sending module resends the data. If not, it further determines the cause of the failure by retrying the saving process. If the issue cannot be resolved by retrying (e.g., file corruption, human deletion, unknown garbled characters), a resend command is sent. If the issue can be resolved by retrying (e.g., system crash, unstable network conditions), it continuously retryes without sending failure information to the maritime data sending module, thus reducing network overhead and data transmission costs.

[0061] The overall communication of this system is based on the Netty TCP communication framework. This framework can ensure the integrity of the data packets generated by the compression system under extremely high network fluctuations, and can achieve more than 100,000 TCP connections on a single machine, which fully guarantees the high load availability of the system.

[0062] By setting a time span, large data packets are broken down into several smaller data packets. Even when network conditions are poor and large data packets cannot be transmitted, some smaller data packets can still be sent, improving the transmission rate. This system segments files into independent small files. Taking a 100MB file as an example, by calculating the start and end times to determine a 100-minute span, and dividing the file into 100 smaller files according to a custom 1-minute span, the file will be sent. When the network is poor, even if only one small file is successfully transmitted, the receiving end can still process and use that file without having to wait for the file to be complete. This greatly improves efficiency compared to traditional full-data transmission and power-off resume transmission methods.

[0063] In a specific embodiment, the formula for calculating the configuration time span is shown in formula (2).

[0064] (2)

[0065] in, Indicates the time span, This indicates network bandwidth.

[0066] Configure time spans to facilitate data splitting and transmission based on time spans.

[0067] In a specific embodiment, the configuration module calculates the number of processing threads corresponding to the number of files of different data types in the maritime data. The scheme for calculating the number of processing threads to be allocated to each file based on the time span of different data types is as follows:

[0068] The configuration field sets the number of files. If the number of files of a certain data type is greater than the configuration field, a basic thread is allocated to each file of that data type for processing. If the number of files of a certain data type is less than the configuration field, a basic thread is allocated to all files of that data type for processing.

[0069] A configuration field sets an upper limit for the time interval. If the time interval of data in a certain data file of a certain data type is greater than the set configuration field, the data in the file is split according to the time span, and the number of threads allocated after calculating the time span is obtained, and the split data is processed in parallel. If the time interval of data in a certain data file is less than the set configuration field, one thread is allocated to process the data in the file.

[0070] Specifically, a configuration field for the number of files is set. If the number of files of a certain data type is greater than the configuration field, then a basic thread is allocated to process each file of that data type. If the number of files of a certain data type is less than the configuration field, then a basic thread is allocated to process all files of that data type.

[0071] Taking a file count of 5 as an example, if there are 6 data files in the scanned directory, the number of files is greater than the configured number of files, i.e., 5. The system will automatically allocate a thread to process each of these 6 data files. If there are 4 data files in the scanned directory, the number of files is less than the configured number of files. In this case, a thread will be allocated to process each of the 4 data files.

[0072] The data is stored in a ".csv" format file, with one data entry per line, and each data entry contains time information;

[0073] A configuration field sets an upper limit for the time interval. If the time interval of data in a data file of a certain data type exceeds the set configuration field, the data in the file is split according to the time span, and the number of threads allocated after calculating the time span is obtained. The split data is then processed in parallel. If the time interval of data in a data file of a certain data type is less than the set configuration field, then one thread is allocated to process the data in that file.

[0074] Taking a configuration field with an upper limit of 60 minutes for the time interval and a time span of 5 minutes as an example, if a data file contains more than 60 minutes of data, then the time interval of the data in this data file exceeds the set configuration field. The data is split into 12 files according to the time span, that is, 12 threads are allocated to process these 12 data files at the same time, that is, the 12 data files are compressed at the same time.

[0075] In this embodiment, multiple threads are allocated to process the files, enabling parallel task execution and full utilization of multi-core processor resources. When multiple files are waiting to be transmitted, multi-threaded processing can split multiple data files simultaneously, greatly shortening processing time. Maritime data transmission modules are often equipped with powerful multi-core computers. While a single thread can only utilize the computing power of one core, multi-threading can allocate different threads to different cores to run simultaneously, thus making full use of the computer's hardware resources.

[0076] In a specific embodiment, the scheme for splitting the maritime data according to the corresponding time span in chronological order and forming compressed files along with corresponding verification information, timestamps, and ID information is as follows:

[0077] S11. The maritime data is split into data packets according to the corresponding time span and in chronological order. The data packets are then compressed twice to generate compressed files. In this embodiment, the data packets are first compressed into ".csv" format files, and then compressed into ".zip" format files using the Univocity method.

[0078] S12. Generate verification information based on the filename of the compressed file. The verification information includes the total file length, MD5 checksum, filename, file type, file date, and file path. Add timestamp and ID information to the compressed file.

[0079] S13. Encapsulate the verification information into header information in JSON format and temporarily store the compressed file.

[0080] Specifically, such as Figure 2 As shown, the collected sensor data is converted into ".csv" format, compressed again into ".zip" format archive and named in a specific format, the naming rule is "data type + start and end time", for example: 1_2023111812000020231118130000_.zip;

[0081] Based on the compressed filename, generate verification information, including the total file length, MD5 checksum, filename, file type, file date, and file path. Add a timestamp and ID information to the compressed file and store them in memory.

[0082] For each compressed file that needs to be transmitted, a hash calculation is performed, and the calculation result is encrypted with MD5. The encrypted result is used as the ID information of the compressed file, which is the digital signature of the compressed file. At the same time, a timestamp is added to each compressed file that needs to be transmitted. The timestamp is the time span of the data collection time in the compressed file.

[0083] The verification information is encapsulated into header information in JSON format, stored in memory, and awaits transmission.

[0084] Temporarily store compressed data files in ".zip" format;

[0085] Check if the network status is normal. If the network status is normal, send data by sending the encapsulated header information and the corresponding temporarily stored compressed file to the land data receiving module.

[0086] If the network status is abnormal, continue to perform data collection, export, compression, and encapsulation operations, and store the data in a temporary directory. Wait until communication is normal before sending it together. The temporary directory is a folder for storing temporary files.

[0087] Due to network instability in the maritime data transmission module, there is a possibility that the maritime data transmission module may be unaware that the land-based data receiving module has already received the data, resulting in duplicate transmissions. To address this issue, this system adds timestamps and ID information to files. Each time a file is sent, a header containing the ID and timestamp is sent. After sending the file, the sliced ​​file is then sent. At this point, the land-based data receiving module can determine whether the file has already been received based on the ID, timestamp, and data. If it has been received, the receiving of that file is abandoned, saving transmission resources and improving efficiency.

[0088] In a specific embodiment, such as Figure 1 and Figure 2 As shown, the land data receiving module receives the compressed file and corresponding verification information transmitted by the maritime data sending module, forms a data record table, verifies the verification information, determines whether the transmitted compressed file has been received, whether the compressed file is correct and complete, and saves the correct compressed file.

[0089] S21. Receive the compressed file and corresponding verification information transmitted by the maritime data transmission module. Verify the verification information. If the verification result is correct, receive the compressed file transmitted by the maritime data transmission module and save the data compressed package according to the header to form a data record table.

[0090] S22. If the verification result is incorrect, feedback is sent to the maritime data transmission module, which needs to resend the compressed file and the corresponding verification information.

[0091] S23. Compare the header data with the data record table to determine whether the verification information and corresponding compressed file transmitted by the maritime data transmission module have been received. If they have been received, terminate the current reception. If they have not been received, repeat steps S21-S22 to verify and save the verification information and update the data record table.

[0092] The land data receiving module separates data reception, data verification, and data decompression, and these processes can be performed synchronously. This means that even if there is no network between land and sea and data transmission is impossible, the system can still verify the files previously transmitted by the maritime data sending module, determine whether the file has been received, and perform operations such as decompression, data storage, and record table updates on the transmitted data, regardless of whether data has been received from the maritime data sending module. Compared to the sequential reception, verification, and decompression processes, this greatly improves data processing efficiency.

[0093] This module can also verify the integrity of data files. If the verification passes, the subsequent decompression and import will proceed. If the verification fails, the subsequent decompression operation will not be performed, and the maritime data sender will be notified to resend this part of the data. This improves data transmission efficiency while ensuring data integrity.

[0094] In a specific embodiment, such as Figure 1 and Figure 2 As shown, the land data import module decompresses the compressed file saved by the land data receiving module according to the data record table, stores the decompressed data in the database for retrieval, and updates the saving status of the corresponding compressed file in the data record table.

[0095] S31. Decompress the compressed file according to the data record table;

[0096] Based on the data transmission record table, the ".zip" format compressed package is decompressed into a ".csv" format file, and then the data is entered into the database;

[0097] S32. According to the data record table, import the decompressed file data into the database and update the save status of the corresponding compressed file in the data record table.

[0098] This module can update the data record table based on the imported data. By updating the data record table, the land data receiving module can call the updated data record table to delete the saved file data in a timely manner. The land data receiving module, through the sea-land communication module, can then feed back to the sea data sending module to delete the data files that have been transmitted and stored in the database in a timely manner, thereby reducing disk usage and avoiding the risk of system crash.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A land-sea remote data transmission system, characterized in that, The application relates to a marine-land data transmission system. The configuration module is used for configuring corresponding data acquisition paths, data storage paths and time spans according to the data types of marine data, calculating the number of corresponding processing threads according to the file numbers of different data types of the marine data, and calculating the number of processing threads required by each file according to the time spans of different data types. The marine data sending module is used for collecting marine data, forming corresponding original data files according to the data types of the marine data, splitting the marine data according to corresponding time spans in time sequence and forming compressed files, corresponding check information, time stamps and id information according to the number of the original data files, corresponding time spans and corresponding processing threads. The marine data sending module is also used for sending signals to the land data receiving module at fixed time intervals, judging whether the current network state is normal according to whether the feedback signals sent by the land data receiving module can be received, sending compressed files and corresponding check information to the land data receiving module when the network state is normal, and generating information representing the compressed files sent by the marine data sending module and a data transmission progress table representing the real-time receiving state of the compressed files. When the network state is abnormal, the operation process of collecting marine data and generating compressed files is executed until the network state is normal, and then the generated compressed files and corresponding check information are continuously sent to the land data receiving module. The land data receiving module is used for receiving transmitted compressed files and corresponding check information, performing check according to the check information, judging whether the transmitted compressed files have been received, whether the compressed files are correct and complete, saving correct and complete compressed files, and forming a data record table. The land data importing module is used for decompressing the compressed files saved by the land data receiving module according to the data record table, storing the decompressed data into a database for calling and updating the saving state of the corresponding compressed files in the data record table. The marine-land communication module is used for monitoring the data transmission state of the marine data sending module and the land data receiving module in real time, feeding back to the marine data sending module after receiving the data receiving result and the check result fed back by the land data receiving module. The marine-land communication module is used for monitoring the data transmission state of the marine data sending module and the land data receiving module in real time, feeding back to the marine data sending module after receiving the data receiving result and the check result fed back by the land data receiving module. The marine-land communication module is used for monitoring the data transmission state of the marine data sending module and the land data receiving module in real time, feeding back to the marine data sending module after receiving the data receiving result and the check result fed back by the land data receiving module.

2. A land-sea remote data transmission system according to claim 1, characterized in that, ​ The configuration field of file quantity is set. If the file quantity of a data type is greater than the configuration field of file quantity, one base thread is allocated for each file in the data type. If the file quantity of a data type is less than the configuration field of file quantity, one base thread is allocated for all files in the data type. The configuration field of upper limit of time interval is set. If the time interval of data in a data file of a data type is greater than the set configuration field, the data in the file is split according to time span, the number of allocated threads after time span calculation is obtained, and the split data is processed in parallel. If the time interval of data in a data file of a data type is less than the set configuration field, one thread is allocated for the data in the file.

3. The system of claim 1, wherein, The calculation formula of time span is shown in formula (1), （1） wherein denotes a time span, denotes a network bandwidth.

4. The system of claim 1, wherein, The offshore data is split according to corresponding time span in time sequence and compressed to form a compressed file and corresponding check information, a time stamp and id information, including: S11, the offshore data is split according to corresponding time span in time sequence to form a data packet, and the data packet is compressed twice to generate a compressed file; S12, check information is generated according to the file name of the compressed file, the check information includes file total length, md5 check code, file name, file type, file date and file path, a time stamp and id information are added to the compressed file; S13, the check information is encapsulated into header information, the information adopts Json format, and the compressed file is temporarily stored.

5. A land-sea remote data transmission system according to claim 4, wherein, The compressed file is added with a time stamp and id information, including: Hash calculation is performed on each compressed file to be transmitted, and the calculation result is encrypted by md5 to obtain the encryption result as the id information of the compressed file; A time stamp is added to each compressed file to be transmitted, and the time stamp is the time span of the data acquisition time in the compressed file.

6. A land-sea remote data transmission system according to claim 5, wherein, The land data receiving module receives the compressed file and corresponding check information transmitted by the offshore data sending module, forms a data record table, checks the check information, judges whether the transmitted compressed file has been received, whether the compressed file is correct and complete, and saves the correct and complete compressed file, including: S21, receiving the compressed file and corresponding check information transmitted by the offshore data sending module, checking according to the check information, if the check result is correct, receiving the compressed file transmitted by the offshore data sending module, and saving the data compression package according to the header to form a data record table; S22, if the check result is incorrect, feedback is given to the offshore data transmission module to inform the offshore data transmission module to resend the compressed file and corresponding check information; S23, comparing the header data with the data record table to determine whether the check information and the corresponding compressed file transmitted by the marine data transmission module have been received, if yes, terminating the current receiving, if not, repeating the steps of S21-S22 to check and save the check information, and updating the data record table.

7. A land-sea remote data transmission system according to claim 6, wherein, The land data import module is used for decompressing the compressed file saved by the land data receiving module according to the data record table, storing the decompressed data into a database for calling and updating the saving state of the corresponding compressed file in the data record table, and comprising: S31, decompressing the compressed file according to the data record table; S32, importing the decompressed file data into the database according to the data record table and updating the saving state of the corresponding compressed file in the data record table.

Images