Ship-shore weak network data synchronization method and device, electronic equipment and storage medium

Abstract

The invention discloses a ship-shore weak network data synchronization method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the grouping compression of to-be-synchronized first data in a target ship end, and obtaining second data; sending the compression information to a target shore end, and comparing the compression information based on the transmitted file through the target shore end to obtain a transmission mode and a transmission progress of each compressed packet in the second data; based on the transmission mode and the transmission progress, metadata initialization is carried out on the target ship end and the target shore end, and transmission preparation is completed; transmitting the compressed packets of the target ship end to a target shore end in sequence in a blocking manner based on a sliding window; the sliding window comprises a preset number of data blocks which are arranged in sequence, and when the target shore end completes synchronization of any number of front data blocks in the sliding window, moving the sliding window backwards by a corresponding number of data blocks until the target shore end completes synchronization of all the data blocks. The method can efficiently realize data synchronization, and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a ship-shore weak network data synchronization method, device, electronic equipment and storage medium. Background Art

[0002] At present, data synchronization between the ship and the shore is mainly based on TCP, which is consistent with the idea of ​​data synchronization between two points on land, that is, a TCP communication channel is established between the two points, and then the synchronization data is sent. However, the existing technology has the following disadvantages:

[0003] 1) Slow speed: Currently, data from ship to shore is mainly transmitted via satellite. The satellite bandwidth is relatively low, resulting in slow data synchronization.

[0004] 2) Poor experience: Considering the low satellite bandwidth and the fact that data synchronization and onboard technicians need to be connected to the Internet, there will be bandwidth competition, resulting in a poor Internet experience for users.

[0005] 3) Low synchronization success rate: Currently, data synchronization is mainly based on TCP transmission. The integrity of data transmission is theoretically guaranteed, but considering the low bandwidth of satellites, data transmission often fails halfway through. Summary of the invention

[0006] The present invention aims to solve the problem of related technical limitations at least to a certain extent. To this end, the present invention proposes a ship-shore weak network data synchronization method, device, electronic device and storage medium, which can achieve data synchronization efficiently.

[0007] On the one hand, an embodiment of the present invention provides a ship-shore weak network data synchronization method, including:

[0008] The first data to be synchronized in the target ship end is grouped and compressed to obtain second data; the second data includes at least one compressed package and compression information corresponding to the compressed package;

[0009] The compressed information is sent to the target shore end, and the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed packet in the second data; the transmission mode includes transmission from the beginning and transmission at a breakpoint, and the transmission progress of transmission from the beginning is 0;

[0010] Based on the transmission mode and transmission progress, metadata is initialized at the target ship end and the target shore end respectively to complete the transmission preparation;

[0011] Based on the sliding window, the compressed packets at the target ship end are sequentially transmitted to the target shore end in blocks;

[0012] Among them, the sliding window includes a preset number of data blocks arranged in sequence. When the target shore completes the synchronization of any number of data blocks in front of the sliding window, the sliding window is moved back by the corresponding number of data blocks until the target shore completes the synchronization of all data blocks of the compressed package.

[0013] Optionally, the compressed information includes a file name, a file size, a file MD5, and a last modification time; and grouping and compressing the first data to be synchronized in the target ship end to obtain the second data comprises the following steps:

[0014] Sort the data files in the first data according to modification time to obtain a file sequence;

[0015] Based on a preset size benchmark, sequentially traverse the file sequence and group all data files in the file sequence into multiple data groups;

[0016] Perform file compression on each data group using a preset compression algorithm to obtain a corresponding compressed package;

[0017] The file name is obtained based on the combination of the names of all data files in the data group, the file size and file MD5 are calculated based on a preset algorithm, and the last modification time is determined based on the modification time of the last data file in the data group.

[0018] Optionally, the compressed information includes a file name, a file size, a file MD5, and a last modification time; and obtaining a transmission mode and a transmission progress of each compressed package in the second data by comparing the compressed information based on the transmitted file by the target shore, comprises the following steps:

[0019] The target shore end compares the file name based on the name in the file information of each file in the transmitted file; the file information is determined based on the compression information of the compressed package historically transmitted by the target ship end;

[0020] When the file name of the compressed package does not exist in the transmitted files on the target shore, the transmission mode of the compressed package is determined to be retransmission; otherwise, the MD5 of the file information of the target file with the same file name is compared by the target shore;

[0021] When the MD5 in the file information of the target file is inconsistent with the file MD5, the transmission mode of the compressed package is determined to be retransmission; otherwise, the file size is compared based on the size in the file information of the target file by the target shore;

[0022] When the size in the file information of the target file is inconsistent with the file size, the transmission mode of the compressed package is determined to be retransmission; otherwise, the transmission mode of the compressed package is determined to be breakpoint transmission, and the transmission progress is determined based on the actual file content of the target file;

[0023] The target shore end compares the last modification time based on the target file. When the modification time of the target file is inconsistent with the last modification time, an early warning reminder is sent to the target ship end, and then the transmission method and progress of the compressed package are determined in response to the selection instruction of the target ship end.

[0024] Optionally, sequentially transmitting the compressed packets of the target ship end to the target shore end in blocks based on the sliding window includes the following steps:

[0025] The compressed packet of the target ship is processed into blocks based on a preset data size to obtain a plurality of sequentially arranged data blocks;

[0026] Select a preset number of data blocks after the transmission progress as synchronization data through a sliding window;

[0027] Sending each data block in the synchronization data to the target shore end in sequence, so that the target shore end synchronizes the data blocks in the synchronization data one by one; obtaining the synchronization progress of the data blocks in the synchronization data by the target shore end;

[0028] When the synchronization progress does not change compared to the transmission progress, the synchronization data is retransmitted; otherwise, the transmission progress is updated based on the synchronization progress, and the step of selecting a preset number of data blocks after the transmission progress as the synchronization data through the sliding window is returned to execute until the target shore completes the synchronization of all data blocks of the compressed package.

[0029] Optionally, the method further comprises the following steps:

[0030] When the target shore end completes synchronization of the target number of data blocks, or when the target shore end receives a heartbeat request from the target ship end, the synchronization progress of the data blocks of the compressed package is sent to the target ship end through the target shore end.

[0031] Optionally, the method further comprises the following steps:

[0032] The target ship periodically sends a heartbeat request to the target shore;

[0033] Determine the network quality of the target ship end and the target shore end according to the timeliness of the heartbeat reply from the target shore end to the target ship end based on the heartbeat request;

[0034] When the delay of network quality is greater than the preset threshold, the data transmission from the target ship to the target shore is stopped.

[0035] Optionally, the method further comprises the following steps:

[0036] When the target shore completes the synchronization of all data blocks of the compressed package, MD5 calculation is performed on the synchronized data to obtain the verification MD5;

[0037] Compare the verification MD5 with the file MD5 in the compressed information corresponding to the compressed package;

[0038] When the comparison results are inconsistent, retransmission feedback is sent to the target ship.

[0039] On the other hand, an embodiment of the present invention provides a ship-shore weak network data synchronization device, including:

[0040] The first module is used to group and compress the first data to be synchronized in the target ship end to obtain second data; the second data includes at least one compressed package and compression information corresponding to the compressed package;

[0041] The second module is used to send the compressed information to the target shore end, and the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed packet in the second data; the transmission mode includes transmission from the beginning and breakpoint transmission, and the transmission progress of transmission from the beginning is 0;

[0042] The third module is used to initialize metadata at the target ship end and the target shore end respectively based on the transmission mode and transmission progress, and complete the transmission preparation;

[0043] The fourth module is used to sequentially transmit the compressed packets of the target ship end to the target shore end in blocks based on the sliding window;

[0044] Among them, the sliding window includes a preset number of data blocks arranged in sequence. When the target shore completes the synchronization of any number of data blocks in front of the sliding window, the sliding window is moved back by the corresponding number of data blocks until the target shore completes the synchronization of all data blocks of the compressed package.

[0045] Optionally, the device further comprises:

[0046] The fifth module is used to send the synchronization progress of the compressed data blocks to the target ship through the target shore end when the target shore end completes the synchronization of the target number of data blocks, or when the target shore end receives the heartbeat request from the target ship end.

[0047] Optionally, the device further comprises:

[0048] The sixth module is used to periodically send a heartbeat request to the target shore end through the target ship end;

[0049] The seventh module is used to determine the network quality of the target ship end and the target shore end according to the timeliness of the heartbeat reply from the target shore end to the target ship end based on the heartbeat request;

[0050] The eighth module is used to stop the data transmission from the target ship end to the target shore end when the delay of the network quality is greater than a preset threshold.

[0051] Optionally, the device further comprises:

[0052] The ninth module is used for calculating MD5 of the synchronized data to obtain the verification MD5 when the target shore completes the synchronization of all data blocks of the compressed package;

[0053] The tenth module is used to compare the verification MD5 with the file MD5 in the compressed information corresponding to the compressed package;

[0054] The eleventh module is used to retransmit feedback to the target ship when the comparison results are inconsistent.

[0055] On the other hand, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used to store programs; the processor executes the program to implement the above-mentioned ship-shore weak network data synchronization method.

[0056] On the other hand, an embodiment of the present invention provides a computer storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned ship-shore weak network data synchronization method.

[0057] The embodiment of the present invention obtains the second data by grouping and compressing the first data to be synchronized in the target ship end; the second data includes at least one compressed package and compression information corresponding to the compressed package; the compressed information is sent to the target shore end, and the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed package in the second data; the transmission mode includes transmission from the beginning and breakpoint transmission, and the transmission progress of transmission from the beginning is 0; based on the transmission mode and transmission progress, metadata is initialized at the target ship end and the target shore end respectively to complete the transmission preparation; based on the sliding window, the compressed package of the target ship end is sequentially transmitted to the target shore end in blocks; wherein the sliding window includes a preset number of data blocks arranged in sequence, and when the target shore end completes the synchronization of any number of data blocks in front of the sliding window, the sliding window is moved back by a corresponding number of data blocks until the target shore end completes the synchronization of all data blocks of the compressed package. The embodiment of the present invention realizes breakpoint continuation based on breakpoint transmission, improves the success rate of ship-shore data synchronization, and the present invention provides a sequential transmission strategy, which not only ensures data integrity, but also improves data synchronization speed. The present invention can efficiently realize data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0059] Figure 1 A schematic diagram of an implementation environment for data synchronization provided by an embodiment of the present invention;

[0060] Figure 2 A schematic flow chart of a ship-shore weak network data synchronization method provided by an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of the communication process between the ship end and the shore end provided in an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of the expansion process of group compression provided by an embodiment of the present invention;

[0063] Figure 5 A schematic diagram of an expanded flow chart for determining a transmission mode and transmission progress of a compressed package provided in an embodiment of the present invention;

[0064] Figure 6 A schematic diagram of the principle of data synchronization interaction provided by an embodiment of the present invention;

[0065] Figure 7 A schematic diagram of the principle of data retransmission provided by an embodiment of the present invention;

[0066] Figure 8 A schematic diagram of the overall business process principle of the ship-shore weak network data synchronization method provided by an embodiment of the present invention;

[0067] Fig. 9 A schematic diagram of the structure of a ship-shore weak network data synchronization device provided by an embodiment of the present invention;

[0068] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0070] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0071] Reference to "embodiments" in the present invention means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present invention may be combined with other embodiments.

[0072] It is understandable that the ship-shore weak network data synchronization method provided in the embodiment of the present invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, tablet computer, laptop computer, desktop computer, etc., but is not limited to this.

[0073] To facilitate the understanding of the technical solution of the present invention, the specific terms of the technical features that may appear in the embodiments of the present invention are first explained:

[0074] VSAT satellite communication (Very Small Aperture Terminal): Since VSAT is derived from the traditional satellite communication system, it is also called a small satellite data station or a personal earth station. The "small" here refers to the small antenna aperture of the small station equipment in the VSAT system, which is usually 0.3m to 1.4m. The equipment has a compact structure, is solid, intelligent, cheap, easy to install, has low requirements on the use environment, is not restricted by the ground network, and has flexible networking.

[0075] Drilling vessels are ships used to drill seabed geological structures, mainly for marine geological exploration. With the in-depth implementation of the marine strategy, drilling vessels appear in international oceans such as the Pacific Ocean. At this time, the only means of communication between the ship and the shore base is satellite. The communication process is referenced in Figure 3 .

[0076] TCP Transmission Control Protocol (TCP) is a connection-oriented, reliable, byte-stream-based transport layer communication protocol. TCP is designed to adapt to a layered protocol hierarchy that supports multi-network applications. Pairs of processes in host computers connected to different but interconnected computer communication networks rely on TCP to provide reliable communication services. TCP assumes that it can obtain simple, possibly unreliable datagram services from lower-level protocols. In principle, TCP should be able to operate on a variety of communication systems from hard-wired connections to packet switching or circuit switching networks.

[0077] The UDP transmission control protocol (User Datagram Protocol) is based on a connectionless, unreliable datagram transmission mechanism, and only provides source port, destination port, data length, checksum and encapsulation to achieve fast and concise end-to-end communication. It does not guarantee the orderly arrival of data, reliable transmission or flow control, and is suitable for application scenarios that are sensitive to delay, can tolerate packet loss, and have self-correction capabilities, such as real-time audio and video streaming, online games, network broadcasting, DNS queries, etc.

[0078] LZMA compression algorithm: Among the actual compression algorithms, LZMA (Lempel-Ziv-Markov chain Algorithm) is generally considered to provide extremely high compression rates. It was proposed in 1998 and is an improved version of the LZ77 algorithm, used to implement the 7-Zip file format (.7z). LZMA uses a chain compression method and applies a modified LZ77 algorithm at the bit level to achieve excellent compression efficiency, especially when processing highly repetitive data such as text and source code.

[0079] LZMA2 compression algorithm: As a successor to LZMA, LZMA2 further improves compression performance and multi-core processor support while maintaining a high compression rate. In practical applications, such as using 7-Zip software, choosing the LZMA2 compression method can often achieve a very high compression rate.

[0080] MD5 (Message-Digest Algorithm 5): It is a widely used cryptographic hash function designed to convert information of any length into a fixed-length hash value. This process is one-way, which means that it is difficult to infer the original information from the hash value. The core significance of MD5 is to provide a data verification mechanism. By calculating the MD5 value of the data, it is possible to efficiently compare whether the data is consistent without directly comparing the data itself. It is achieved through complex mathematical operations, and even a small change in the data will result in a huge difference in the calculated MD5 value.

[0081] like Figure 1FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0082] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0083] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0084] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention.

[0085] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a ship-shore weak network data synchronization method. The following is explained using the application of the ship-shore weak network data synchronization method in the server 101 as an example. It can be understood that the ship-shore weak network data synchronization method can also be applied to the terminal 102.

[0086] Reference Figure 2 , Figure 2 The flowchart of the ship-shore weak network data synchronization method applied to the server provided by the embodiment of the present invention, the execution subject of the ship-shore weak network data synchronization method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method comprises the following steps:

[0087] S100, grouping and compressing first data to be synchronized in the target ship end to obtain second data;

[0088] The second data includes at least one compressed package and compression information corresponding to the compressed package;

[0089] It should be noted that the compressed information includes the file name, file size, file MD5 and last modification time; in some embodiments, such as Figure 4 As shown, step S100 may include the following steps: S101, sorting the data files in the first data according to the modification time to obtain a file sequence; S102, based on a preset size benchmark, sequentially traversing the file sequence, and grouping all the data files in the file sequence into multiple data groups; S103, performing file compression on each data group through a preset compression algorithm to obtain a corresponding compressed package; wherein the file name is obtained based on a combination of the names of all the data files in the data group, the file size and the file MD5 are calculated based on a preset algorithm, and the last modification time is determined based on the modification time of the last data file in the data group.

[0090] For example, in some specific implementations, local compression of data on the ship side can be implemented as follows:

[0091] There are generally many files to be transmitted on the ship side, mainly the test results of technical experiments, in formats including tables, texts, pictures and other files, and fewer high-definition videos. These data files need to be compressed before transmission. The present invention adopts the open source LZMA2 compression algorithm, which has a very high compression rate. Compared with other compression algorithms, the compression speed will be slightly slower, but considering the satellite bandwidth, a higher compression rate is still necessary. In the data transmission management regulations on board, it is not recommended to transmit high-definition video. This is because current videos are usually in compressed formats. Even if LZMA2 is used for recompression, the file compression rate will not be too high, and the video will take up a lot of bandwidth and affect the transmission of other files, so high-definition video files are usually synchronized by direct copying after the ship docks.

[0092] Under normal circumstances, there will be multiple files to be synchronized, and these files are sorted according to the last modification time, and the files will be divided into multiple groups for compression. The present invention is grouped according to 100M as the benchmark, that is, the sum of the file sizes of each group is about 100M. If a single file exceeds 100M, then this file is grouped separately. Then compression is performed for each group, and the compressed package name is the MD5 of all file names added together, so that a series of compressed packages will be obtained. This grouping standard can be customized and adjusted, but it is not recommended to exceed 200M, because the larger the compressed file, the higher the file transfer failure rate. These compressed packages are transmitted synchronously one by one, and the present invention is for data synchronization of a single compressed package.

[0093] In most cases, the compression rate of the document file can reach about 60%, that is, the size of the compressed document file is about 40% of the original size. According to the 100M grouping, a single compressed package is about 40M. In actual scenarios, the compression rate can even reach 70%.

[0094] S200, sending the compressed information to the target shore end, and obtaining the transmission mode and transmission progress of each compressed packet in the second data by comparing the compressed information with the transmitted file through the target shore end;

[0095] The transmission modes include transmission from the beginning and transmission from a breakpoint. The transmission progress of transmission from the beginning is 0.

[0096] It should be noted that the compressed information includes the file name, file size, file MD5 and last modification time; in some embodiments, such as Figure 5 As shown, the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed package in the second data, which may include the following steps: S201, the target shore end compares the file name based on the name in the file information of each file in the transmitted file; the file information is determined according to the compression information of the compressed package historically transmitted by the target ship end; S202, when the file name of the compressed package does not exist in the transmitted file of the target shore end, determine that the transmission mode of the compressed package is to be transmitted from the beginning; otherwise, the target shore end compares the MD5 of the file information of the target file with the same file name; S203, when the MD5 in the file information of the target file is the same as the file name, determine that the compressed package is transmitted from the beginning; If the MD5 of the target file is inconsistent, the transmission mode of the compressed package is determined to be transmission from the beginning; otherwise, the target shore end compares the file size based on the size in the file information of the target file; S204, when the size in the file information of the target file is inconsistent with the file size, the transmission mode of the compressed package is determined to be transmission from the beginning; otherwise, the transmission mode of the compressed package is determined to be breakpoint transmission, and the transmission progress is determined based on the actual file content of the target file; S205, the target shore end compares the last modification time based on the target file, and when the modification time of the target file is inconsistent with the last modification time, an early warning reminder is sent to the target ship end, and then the transmission mode and transmission progress of the compressed package are determined in response to the selection instruction of the target ship end.

[0097] For example, in some specific implementations, a ship-side data synchronization request is first initiated, which can be specifically implemented as follows:

[0098] The ship sends a data synchronization request, which is forwarded to the shore via satellite. It mainly includes detailed information about the compressed package of transmitted data, including the file name, file size, file MD5, file creation time, last modification time and other information of the compressed package. This information is mainly used to determine whether breakpoint resumption is needed.

[0099] Then, check the data synchronization status, which can be achieved as follows:

[0100] 1) Check the file name to see if the file exists on the shore. If it does not exist, it means that the file is being transmitted for the first time. If it does exist, continue to compare the file's MD5;

[0101] 2) Check the file MD5 to see if the MD5 of the file on the shore is consistent with the MD5 of the target file. If they are inconsistent, it means that the target file has changed and needs to be retransmitted. If they are consistent, continue to compare the file sizes.

[0102] 3) Check the file size. If they are the same, it means the file has been transferred before and you can choose to resume the transfer. If they are not the same, it means the target file has changed and you need to transfer it again. This step is necessary because the file is stored in the cache before it is successfully transferred, and the cache may be tampered with or cleared.

[0103] 4) Check the last modification time. If they are consistent, it means that the file is being retransmitted from the last compression. Otherwise, it should be a new compression package, and the ship side will be actively reminded.

[0104] Then, the shore data synchronization status is restored. After the inspection is completed, the monitoring situation will be directly fed back to the ship, mainly the existing file information on the shore (file name, MD5, file size, last modified time, transmission progress, etc.), as well as transmission suggestions (transmission from the beginning or breakpoint resumption) and other information.

[0105] S300, based on the transmission mode and transmission progress, initializing metadata at the target ship end and the target shore end respectively, and completing the transmission preparation;

[0106] For example, in some specific implementations, first, the synchronization method is clarified and metadata is initialized, which can be specifically implemented as follows:

[0107] After receiving feedback from the shore, the user will be reminded of the transfer method. According to the judgment of the previous steps, if the files are inconsistent, you can only choose to transfer from the beginning. If there are consistent files on the shore, you can choose to transfer from the beginning or resume the transfer. In most cases, in order to improve the transfer efficiency, if you can choose to resume the transfer, choose to resume the transfer, otherwise, transfer from the beginning.

[0108] Then, initialize the metadata, which can be implemented as follows:

[0109] After explicitly selecting to transfer from the beginning or to resume the transfer, metadata will be initialized on the ship side, including information about the currently transferred file (file name, file ID, MD5, file size, last modification time, etc.), as well as the current transfer progress.

[0110] A single UDP data packet is generally 1KB (1024 bytes). The present invention selects 1KB blocks for sequential transmission. The transmission progress here saves the size of the file that has been transmitted. If the transmission is from the beginning, the transmission progress is 0; if the transmission is resumed from a breakpoint, the transmission progress is the progress of the last transmission.

[0111] Finally, the metadata initialization reply is carried out. After receiving the metadata from the ship, the shore also initializes a set of metadata. After the initialization is successful, the reply is sent to the ship. After receiving the reply, the ship can start transmitting data.

[0112] S400, sequentially transmitting the compressed packets at the target ship end to the target shore end in blocks based on the sliding window;

[0113] Among them, the sliding window includes a preset number of data blocks arranged in sequence. When the target shore completes the synchronization of any number of data blocks in front of the sliding window, the sliding window is moved back by the corresponding number of data blocks until the target shore completes the synchronization of all data blocks of the compressed package.

[0114] It should be noted that, in some embodiments, step S400 may include the following steps: based on a preset data size, the compressed package of the target ship end is divided into blocks to obtain a plurality of data blocks arranged in sequence; a preset number of data blocks after the transmission progress are selected as synchronization data through a sliding window; each data block in the synchronization data is sent to the target shore end in turn, so that the target shore end synchronizes the data blocks in the synchronization data one by one; the synchronization progress of the target shore end for the data blocks in the synchronization data is obtained; when the synchronization progress is unchanged compared to the transmission progress, the synchronization data is retransmitted; otherwise, the transmission progress is updated based on the synchronization progress, and the step of selecting a preset number of data blocks after the transmission progress as synchronization data through a sliding window is returned to execute until the target shore end completes the synchronization of all data blocks of the compressed package.

[0115] For example, in some specific implementations, sequential block transmission can be implemented as follows:

[0116] The present invention selects 1KB UDP data block transmission, so the file will be divided into several blocks according to the size of 1KB, and each UDP data packet includes three parts of information: file ID, block ID, and data block. Figure 6 Each block has three states: completed synchronization, synchronizing, and waiting for synchronization. The data synchronization strategy proposed in the present invention is sequential synchronization, that is, all data blocks are sent one by one for synchronization, strictly in the order of blocks.

[0117] The sending strategy is fixed window sending, sending at most WindowSize data blocks each time (the window size can be customized, at least 1, the recommended range is 1-6), and the maximum number of packets in synchronization is WindowSize. Whenever a transmission progress reply is received, based on the data reception on the shore side, the status of the data block is marked (some are changed from synchronizing to completed synchronization, and some are changed from waiting for synchronization to synchronizing), then the sending window is moved and new data blocks are prepared to be sent.

[0118] The retransmission strategy is to sequentially retransmit all missing data, that is, when it is found that the shore has not received a block, all data after the block will be automatically retransmitted. When it is found that a block is not received by the shore, it means that the network may be congested. At this time, the subsequent blocks of the block may not be received. Therefore, we choose to retransmit the block and subsequent data here. The advantage of doing so is to try our best to ensure that all data is stored in sequence, thereby improving the transmission speed.

[0119] Among them, the shore end receiving data blocks can be implemented as follows: the shore end will receive multiple UDP data packets, which include three parts of information such as file ID, block ID, and data block. The file ID determines which file the data belongs to, and the block position is determined according to the block ID. Finally, the shore end stores them in order according to the block ID. At the same time, the shore end may receive data packets with the same file ID and block ID multiple times. If they have been saved, the duplicate data packets are directly discarded. These packets may be retransmitted data packets. Whenever a data packet is received, it will try to update the transmission progress and synchronize the latest transmission progress according to the rules. This part of the rules will be explained later.

[0120] refer to Figure 7 In the data sending process, the normal data synchronization situation is the repeated interaction of data sending and data receiving. If data loss is found, the retransmission state will be entered immediately. At this time, the current sending progress will be stopped, and the lost data block will be retransmitted, and then wait for confirmation whether the data has been received. If it is still not received, it will continue to retransmit until the data is received.

[0121] Among them, in some embodiments, the method may also include the following steps: when the target shore end completes the synchronization of the target number of data blocks, or when the target shore end receives the heartbeat request from the target ship end, the synchronization progress of the compressed data blocks is sent to the target ship end through the target shore end.

[0122] For example, in some specific implementations, the shore end can periodically reply to the reception status of data blocks. The shore end will receive a large number of data packets one after another and store them in sequence, but the shore end does not send a reply every time a data block is received, which will reduce the data transmission efficiency. The shore end will immediately reply to the reception status of data blocks if the following two conditions are met:

[0123] Case 1: When receiving (If it cannot be divided evenly, round up to an integer) valid data blocks, the data block reception status will be replied immediately. The valid data blocks here do not include duplicate packets. For example, if WindowsSize is 3, the data reception order is data packet 1 (block 5), data packet 2 (block 5), data packet 3 (block 6), and data packet 4 (block 6). When data packet 3 is received, it means that 2 valid data blocks (block 5 and block 6) are received, and the data block reception status will be replied at this time. Data packets 2 and 4 are duplicate data packets and will be discarded. It can be seen from here that under normal circumstances, the number of data retransmissions on the ship side is mostly indivual.

[0124] Case 2: When a heartbeat request is received, the data block reception status is immediately replied.

[0125] The progress of the reply includes information such as heartbeat SeqID, file ID, and the latest ID of continuous storage. For example, if blocks 1-6 and blocks 8-9 have been received, the latest ID of continuous storage is block 6.

[0126] Among them, in some embodiments, the method may also include the following steps: periodically sending a heartbeat request to the target shore end through the target ship end; determining the network quality of the target ship end and the target shore end according to the timeliness of the heartbeat reply of the target shore end to the target ship end based on the heartbeat request; when the delay of the network quality is greater than a preset threshold, stopping the data transmission from the target ship end to the target shore end.

[0127] For example, in some specific implementations, the ship will periodically send a heartbeat request, the request content includes the heartbeat SeqID, file ID, the latest sending progress and other information, each time a heartbeat is sent, SeqID = SeaID + 1. The sending frequency SendFrequency can be customized and is generally sent once every 1s (recommended between 500ms and 1s). The main reasons for setting the heartbeat request are as follows:

[0128] 1) Determine the network status. The ship can determine the network status through the heartbeat request reply. If the heartbeat reply is not received for many times, it means that the network quality is particularly poor. At this time, the sending progress will be adjusted or even stopped.

[0129] 2) Improve the transmission speed. After receiving the heartbeat, the shore will immediately synchronize the data transmission progress, and after receiving the latest progress, the ship will immediately adjust the sending strategy. Because the heartbeat is periodic, this mechanism can effectively avoid the extreme situation where the ship can never obtain the data progress due to the loss of other data packets, speed up the movement of the ship's sending window, and improve the transmission speed.

[0130] Among them, after the shore end receives the heartbeat request, it will process and reply to the ship end with information such as the heartbeat SeqID, file ID, and the latest ID of continuous storage. The ship end needs to perform two operations of network quality determination and data loss determination based on the heartbeat reply.

[0131] Network quality determination: Since the heartbeat packet is very concise, the transmission by the shore end and the reception and processing by the ship end hardly consume time. The round-trip of the heartbeat can directly reflect the network quality. If no heartbeat reply packet is received for a continuous time of M * SendFrequency (M can be customized, generally 3), that is, M heartbeat packets have no reply, at this time, it can be determined that the network quality is extremely poor, and all data transmissions will be immediately stopped, but the sending frequency of the heartbeat packet remains unchanged.

[0132] If a heartbeat reply is received, assuming the received heartbeat reply is SeqID1 and the latest sent heartbeat is SeqID2, if SeqID2 - SeqID1 >= M, it means the network quality is extremely poor, resulting in a long round-trip time of the heartbeat. At this time, data transmission is still stopped. If SeqID2 - SeqID1 < M, it means the network quality is acceptable, and at this time, data packets can be resumed or continued to be sent. Generally speaking, the heartbeat is a fallback means for the ship end to determine the network. It can not only improve the transmission speed but also resume data transmission from the breakpoint in a timely manner after the network returns to normal.

[0133] Among them, in some embodiments, the method may further include the following steps: when the target shore end completes the synchronization of all data chunks of the compressed package, calculate the MD5 of the synchronized data to obtain the verification MD5; compare the verification MD5 with the file MD5 in the compression information corresponding to the compressed package; when the comparison result is inconsistent, send a retransmission feedback to the target ship end.

[0134] Exemplarily, in some specific embodiments, based on the latest data transmission progress, when the ship end determines that all data has been received, it indicates that the data synchronization is successful. At this time, a verification request will be sent, and the request content is mainly information about the currently transmitted file (file name, file ID, MD5, file size, last modification time, etc.).

[0135] After the ship end receives the verification request, it will recalculate the MD5 of the file and then compare it with the requested MD5. If they are the same, it indicates that the file transmission is successful. If they are different, it means that some data packets may have been tampered with. In most cases, the MD5s are the same.

[0136] The ship responds with the MD5 verification status, which mainly includes the information of the currently transmitted file (file name, file ID, MD5, file size, last modified time, etc.). After receiving the response, the shore side records the status of this transmission, marks the file as synchronized, and notifies the user. If the MD5 is consistent, the user is notified that the file transfer is successful. Otherwise, the user is notified that the intermediate data has been tampered with and needs to be resynchronized (the data will not be reset and cleared here, and the user needs to actively retransmit).

[0137] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0138] Reference Figure 8 The present invention proposes a data (sequence) synchronization method from a ship end to a shore end, and the specific steps can be implemented as follows:

[0139] Step 1: Local data compression. There are generally many files to be transmitted on the ship side, mainly the test results of technical experiments, in formats including tables, texts, pictures and other files, and fewer high-definition videos. These data files need to be compressed before transmission. The present invention adopts the open source LZMA2 compression algorithm, which has a very high compression rate. Compared with other compression algorithms, the compression speed will be slightly slower, but considering the satellite bandwidth, a higher compression rate is still necessary. In the data transmission management regulations on board, it is not recommended to transmit high-definition video. This is because current videos are usually in compressed formats. Even if LZMA2 is used for re-compression, the file compression rate will not be too high, and the video will take up a lot of bandwidth and affect the transmission of other files, so high-definition video files are usually synchronized by direct copying after the ship docks.

[0140] Under normal circumstances, there will be multiple files to be synchronized, and these files are sorted according to the last modification time, and the files will be divided into multiple groups for compression. The present invention is grouped according to 100M as the benchmark, that is, the sum of the file sizes of each group is about 100M. If a single file exceeds 100M, then this file is grouped separately. Then compression is performed for each group, and the compressed package name is the MD5 of all file names added together, so that a series of compressed packages will be obtained. This grouping standard can be customized and adjusted, but it is not recommended to exceed 200M, because the larger the compressed file, the higher the file transfer failure rate. These compressed packages are transmitted synchronously one by one, and the present invention is for data synchronization of a single compressed package.

[0141] In most cases, the compression rate of the document file can reach about 60%, that is, the size of the compressed document file is about 40% of the original size. According to the 100M grouping, a single compressed package is about 40M. In actual scenarios, the compression rate can even reach 70%.

[0142] Regarding grouping, for example, if 7 files A (size 80M), B (size 17M), C (size 60M), D (size 60M), E (size 70M), F (size 80M), G (170M) have been sorted by the last modification time, then the grouping is as follows:

[0143] 1) A (size 80M) and B (size 17M) are a group, because A+B=97M, which is around 100M, the compressed package name is MD5(AB);

[0144] 2) C (size 60M) and D (size 60M) are a group. C is 60M, which is 40M away from 100M, but C+D=120M, which is only 20M more than 100M. Therefore, C and D are a group, and the compressed package name is MD5(CD);

[0145] 3) E (size 70M) is a separate group. E is 70M, 30M short of 100M, but E+F=150M, 50M more than 100M, so E is a separate group, and the compressed package name is MD5(E);

[0146] 4) F (size 80M) is a separate group for the same reason as E;

[0147] 5) G (size 170M) is a separate group;

[0148] In summary, it is divided into 5 groups, each of which is compressed into a package using LZMA2.

[0149] Step 2: Initiate a ship-side data synchronization request. The ship sends a data synchronization request, which is forwarded to the shore via satellite. It mainly includes detailed information about the compressed package of transmitted data, including the file name, file size, file MD5, file creation time, last modification time, etc. This information is mainly used to determine whether breakpoint resuming is required.

[0150] Step 3: Check data synchronization. When the shore receives a data synchronization request, it will check the transferred files. The judgment process is mainly as follows:

[0151] 1) Check the file name to see if the file exists on the shore. If it does not exist, it means that the file is being transmitted for the first time. If it does exist, continue to compare the file's MD5;

[0152] 2) Check the file MD5 to see if the MD5 of the file on the shore is consistent with the MD5 of the target file. If they are inconsistent, it means that the target file has changed and needs to be retransmitted. If they are consistent, continue to compare the file sizes.

[0153] 3) Check the file size. If they are the same, it means the file has been transferred before and you can choose to resume the transfer. If they are not the same, it means the target file has changed and you need to transfer it again. This step is necessary because the file is stored in the cache before it is successfully transferred, and the cache may be tampered with or cleared.

[0154] 4) Check the last modification time. If they are consistent, it means that the file is being retransmitted from the last compression. Otherwise, it should be a new compression package, and the ship side will be actively reminded.

[0155] Step 4: Data synchronization status reply on the shore side. After the inspection is completed, the monitoring status will be directly fed back to the ship side, mainly the existing file information on the shore side (file name, MD5, file size, last modified time, transfer progress, etc.), as well as transfer suggestions (transfer from the beginning or resume transfer from a breakpoint) and other information.

[0156] Step 5: Specify the synchronization method and initialize metadata. After receiving feedback from the shore, the user will be reminded of the transfer method. According to the judgment in step 3, if the files are inconsistent, you can only choose to transfer from the beginning. If there are consistent files on the shore, you can choose to transfer from the beginning or resume the transfer. In most cases, in order to improve the transfer efficiency, if you can choose to resume the transfer, choose to resume the transfer. Otherwise, transfer from the beginning.

[0157] Step 6: Initialize metadata. After explicitly selecting to transfer from the beginning or to resume transfer from a breakpoint, metadata will be initialized on the ship side, including information about the currently transferred file (file name, file ID, MD5, file size, last modified time, etc.), as well as the current transfer progress.

[0158] A single UDP data packet is generally 1KB (1024 bytes). The present invention selects 1KB blocks for sequential transmission. The transmission progress here saves the size of the file that has been transmitted. If the transmission is from the beginning, the transmission progress is 0; if the transmission is resumed from a breakpoint, the transmission progress is the progress of the last transmission.

[0159] Step 7: Metadata initialization reply. After receiving the metadata from the ship, the shore also initializes a set of metadata. After the initialization is successful, reply to the ship. After receiving the reply, the ship can start transmitting data.

[0160] Step 8: Send data blocks sequentially. The present invention selects 1KB UDP data block transmission, so the file will be divided into several blocks according to the size of 1KB. Each UDP data packet includes three parts of information: file ID, block ID, and data block. Figure 6 Each block has three states: completed synchronization, synchronizing, and waiting for synchronization. The data synchronization strategy proposed in the present invention is sequential synchronization, that is, all data blocks are sent one by one for synchronization, strictly in the order of blocks.

[0161] The sending strategy is fixed window sending, sending at most WindowSize data blocks each time (the window size can be customized, at least 1, the recommended range is 1-6), and the maximum number of packets in synchronization is WindowSize. Whenever a transmission progress reply is received, based on the data reception on the shore side, the status of the data block is marked (some are changed from synchronizing to completed synchronization, and some are changed from waiting for synchronization to synchronizing), then the sending window is moved and new data blocks are prepared to be sent.

[0162] The retransmission strategy is to retransmit all missing data in sequence, that is, when it is found that the shore has not received a block, all data after the block will be automatically retransmitted. When it is found that a block is not received by the shore, it means that the network may be congested. At this time, the subsequent blocks of the block may not be received. Therefore, the block and subsequent data are retransmitted here. The advantage of doing so is to try to ensure that all data are stored in sequence and improve the transmission speed. As for how to judge whether the shore has received the data block, the judgment rules will be given in step 12 later, which will not be expanded here.

[0163] For example, refer to Figure 6 , assuming that WindowSize is 3, after block 5 is synchronized, block 6, block 7 and block 8 are automatically sent. At this time, there are at most 3 data blocks in synchronization. At this time, it is found that block 6 has completed synchronization, and blocks 7 and 8 are still in synchronization, so block 9 is sent immediately. The sending strategy is to do its best to ensure that at most 3 data blocks are in synchronization. If it is found that the shore end has not received block 6 at this time, then retransmit blocks 6, block 7 and block 8 immediately; if it is found that the shore end has not received block 7, then retransmit blocks 7 and block 8 immediately; if it is found that the shore end has not received block 8, then retransmit block 8 immediately.

[0164] Step 9: Receive data blocks. The shore will receive multiple UDP data packets, which include three parts of information: file ID, block ID, and data block. Determine which file the data belongs to based on the file ID, and determine the block location based on the block ID. Finally, the shore stores them in order based on the block ID. At the same time, the shore may receive data packets with the same file ID and block ID multiple times. If they have been saved, the duplicate data packets are directly discarded. These packets may be retransmitted data packets. Whenever a data packet is received, it will try to update the transmission progress and synchronize the latest transmission progress according to the rules. This part of the rules is explained in step 10.

[0165] refer to Figure 7 In the data sending process, the normal data synchronization situation is the repeated interaction of data sending and data receiving. If data loss is found, the retransmission state will be entered immediately. At this time, the current sending progress will be stopped, and the lost data block will be retransmitted, and then wait for confirmation whether the data has been received. If it is still not received, it will continue to retransmit until the data is received.

[0166] Step 10: Regularly reply to the data block reception status. The shore end will receive a large number of data packets one after another and store them in sequence, but the shore end does not send a reply every time it receives a data block, which will reduce the data transmission efficiency. The shore end will immediately reply to the data block reception status if the following two conditions are met:

[0167] Case 1: When receiving (If it cannot be divided evenly, round up to an integer) valid data blocks, the data block reception status will be replied immediately. The valid data blocks here do not include duplicate packets. For example, if WindowsSize is 3, the data reception order is data packet 1 (block 5), data packet 2 (block 5), data packet 3 (block 6), and data packet 4 (block 6). When data packet 3 is received, it means that 2 valid data blocks (block 5 and block 6) are received, and the data block reception status will be replied at this time. Data packets 2 and 4 are duplicate data packets and will be discarded. It can be seen from here that under normal circumstances, the number of data retransmissions on the ship side is mostly indivual.

[0168] Case 2: When a heartbeat request is received, the data block reception status is immediately replied.

[0169] The progress of the reply includes information such as heartbeat SeqID, file ID, and the latest ID of continuous storage. For example, if blocks 1-6 and blocks 8-9 have been received, the latest ID of continuous storage is block 6.

[0170] Regarding the data retransmission strategy in step 8, there is one thing that needs to be discussed here. The shore side will send the data block reception status according to the fixed rules of Case 1 and Case 2. When the ship side receives the transmission progress, it will first suspend sending data, prioritize the data end progress, and take the maximum value as the latest progress. Based on the latest progress, choose to retransmit data or send data normally. For example, if the received data progress is block 6, block 7, and block 8 respectively, then the data reception progress at this time is block 8, and then it will start retransmission or normal transmission from block 9.

[0171] Step 11: Send heartbeat request. The ship will periodically send heartbeat request, the request content includes heartbeat SeqID, file ID, latest sending progress and other information, each time a heartbeat is sent, SeqID = SeaID + 1. The sending frequency SendFrequency can be customized and is generally sent once every 1s (recommended between 500ms and 1s). The main reasons for setting the heartbeat request are as follows:

[0172] 1) Determine the network status. The ship can determine the network status through the heartbeat request reply. If the heartbeat reply is not received for many times, it means that the network quality is particularly poor. At this time, the sending progress will be adjusted or even stopped.

[0173] 2) Improve the transmission speed. After receiving the heartbeat, the shore will immediately synchronize the data transmission progress, and after receiving the latest progress, the ship will immediately adjust the sending strategy. Because the heartbeat is periodic, this mechanism can effectively avoid the extreme situation where the ship can never obtain the data progress due to the loss of other data packets, speed up the movement of the ship's sending window, and improve the transmission speed.

[0174] Step 12: Heartbeat reply. After receiving the heartbeat request, the shore side will process it and reply to the ship side with the heartbeat SeqID, file ID, the latest ID of continuous storage, etc. The ship side needs to perform two operations based on the heartbeat reply: network quality judgment and data loss judgment.

[0175] Network quality determination: Because the heartbeat packet is very concise, it takes almost no time for the shore end to send it and the ship end to receive it. The heartbeat can directly feedback the network quality. If there is no heartbeat reply packet for M*SendFrequency time (M can be customized, usually 3), that is, there is no reply to M heartbeat packets, then it can be determined that the network quality is particularly poor, and all data transmission will be stopped immediately, but the sending frequency of the heartbeat packet remains unchanged.

[0176] If a heartbeat reply is received, assume the received heartbeat reply is SeqID1 and the most recently sent heartbeat is SeqID2. If SeqID2 - SeqID1 >= M, it indicates that the network quality is extremely poor, resulting in a very long heartbeat round-trip time. In this case, data transmission is still stopped. If SeqID2 - SeqID1 < M, it means the network quality is acceptable, and at this time, data packets can be resumed or continued to be sent. Generally speaking, heartbeat is a fallback method for the ship end to judge the network. It can not only improve the transmission speed but also resume data transmission from the breakpoint in a timely manner after the network returns to normal.

[0177] Here is an example. Assume the ship end's heartbeat sending and receiving situation is as follows: sending heartbeat 1, receiving heartbeat reply 1, sending heartbeat 2, receiving heartbeat reply 2, sending heartbeat 3, receiving heartbeat reply 3. In this case of normal data network, the general network delay on land is generally 30ms to 50ms. Considering satellite transmission, the ship-shore network delay is estimated to be 200ms, that is, the reply of heartbeat 1 can be received after 200ms. So the above belongs to the normal network situation.

[0178] Assume the ship end's heartbeat sending and receiving situation is as follows: sending heartbeat 1, sending heartbeat 2, receiving heartbeat reply 1, receiving heartbeat reply 2, sending heartbeat 3, receiving heartbeat reply 3. This situation belongs to slightly worse network, but data can still be sent continuously.

[0179] Assume the ship end's heartbeat sending and receiving situation is as follows: sending heartbeat 1, sending heartbeat 2, sending heartbeat 3, sending heartbeat 4. At this time, it belongs to the situation where no heartbeat reply is received for M consecutive times, indicating that the network quality is very poor, and all data transmission will be immediately stopped.

[0180] Assume the ship end's heartbeat sending and receiving situation is as follows: sending heartbeat 4, receiving heartbeat reply 1. At this time, although a heartbeat reply is received, this reply is from a long time ago, and all data transmission will still be stopped.

[0181] Assume the ship end's heartbeat sending and receiving situation is as follows: sending heartbeat 4, receiving heartbeat reply 1, receiving heartbeat reply 4. At this time, it indicates that the network situation has improved, and data transmission will be resumed.

[0182] Data loss determination: Under normal circumstances, the data transmission progress is continuously increasing, so the ship end will continuously move the sending window and continue to send data. However, when the network is extremely poor, data packet loss may occur. There are two situations for determining data loss

[0183] 1) If no heartbeat reply packet is received for M*SendFrequency time (M can be customized, usually 3), it is determined that all data in the sending window is lost and the data packet sending is stopped. Wait for the network to recover and then send again. This means that the network is very poor and data sending needs to be stopped. At this time, it will be determined that data is lost.

[0184] 2) If the data transmission progress reply is not received within the LostInterval time after sending (which can be customized, generally 200ms-1000ms, generally set to 400ms), the data will be considered lost. This situation belongs to the poor network and the data is likely to be lost. At this time, the data is considered lost and the lost data packet is retransmitted immediately.

[0185] Please note here, Figure 1 The following is a data synchronization interaction diagram to describe the basic process, where steps 8 to 12 belong to the data synchronization phase and will be executed repeatedly until all data blocks are successfully synchronized.

[0186] Step 13: Data synchronization is successful, and a verification request is sent. Based on the latest data transmission progress, the ship determines that all data has been received, indicating that data synchronization is successful. At this time, a verification request will be sent, and the request content is mainly the information of the current transmission file (file name, file ID, MD5, file size, last modification time, etc.).

[0187] Step 14: Local data verification. After receiving the verification request, the ship will recalculate the MD5 of the file and compare it with the requested MD5. If they are consistent, it means that the file transfer is successful. If they are inconsistent, it means that some data packets may have been tampered with. In most cases, the MD5 is consistent.

[0188] Step 15: Reply with data verification status. The ship responds with MD5 verification status, which mainly includes the information of the currently transmitted file (file name, file ID, MD5, file size, last modified time, etc.). After receiving the reply, the shore side records the status of this transmission, marks the file as synchronized, and notifies the user. If the MD5 is consistent, the user is notified that the file transfer is successful. Otherwise, the user is notified that the intermediate data has been tampered with and needs to be resynchronized (the data will not be reset and cleared here, and the user needs to actively retransmit).

[0189] When steps 13 to 15 are completed, the file will be marked as synchronized, and then another file will be transferred. There is an extreme case that needs to be explained. If the file has just been successfully transferred, and the network is suddenly disconnected, the ship side cannot receive the data verification reply for a long time. At this time, the file is still in the synchronization state. When the network is restored to normal, the ship side will resend the verification request until the verification reply is received. The transmission process of the file is truly completed.

[0190] In summary, the entire transmission is divided into three stages:

[0191] 1) Preparation phase, including steps 1 to 7. This phase mainly clarifies the information of the file to be transmitted on the shore side and determines whether to transmit from the beginning or to resume transmission from a breakpoint. It belongs to the data preparation phase before transmission.

[0192] 2) The synchronization phase includes steps 8 to 12. This phase is mainly responsible for data synchronization and retransmission, and dynamically adjusts the sending rhythm through heartbeats to improve file transfer efficiency.

[0193] 3) Verification phase, including steps 13 to 15, this phase is mainly the final data verification.

[0194] Compared with the prior art, the present invention has at least the following beneficial effects:

[0195] 1) Make full use of idle bandwidth to increase data transmission speed, avoid competing for bandwidth with technical personnel, and improve user experience.

[0196] 2) Support breakpoint resume transmission to improve the success rate of ship-shore data synchronization.

[0197] 3) Data is sent based on the UDP protocol, which provides a sequential sending strategy, which not only ensures data integrity but also improves data synchronization speed.

[0198] On the other hand, Fig. 9 As shown, an embodiment of the present invention provides a ship-shore weak network data synchronization device 900, which may include:

[0199] The first module 901 is used to group and compress the first data to be synchronized in the target ship end to obtain second data; the second data includes at least one compressed package and compression information corresponding to the compressed package;

[0200] The second module 902 is used to send the compressed information to the target shore end, and the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed packet in the second data; the transmission mode includes transmission from the beginning and transmission at a breakpoint, and the transmission progress of transmission from the beginning is 0;

[0201] The third module 903 is used to initialize metadata at the target ship end and the target shore end respectively based on the transmission mode and transmission progress, and complete the transmission preparation;

[0202] The fourth module 904 is used to sequentially transmit the compressed packets of the target ship end to the target shore end in blocks based on the sliding window;

[0203] Among them, the sliding window includes a preset number of data blocks arranged in sequence. When the target shore completes the synchronization of any number of data blocks in front of the sliding window, the sliding window is moved back by the corresponding number of data blocks until the target shore completes the synchronization of all data blocks of the compressed package.

[0204] In some embodiments, the apparatus may further include:

[0205] The fifth module is used to send the synchronization progress of the compressed data blocks to the target ship through the target shore end when the target shore end completes the synchronization of the target number of data blocks, or when the target shore end receives the heartbeat request from the target ship end.

[0206] In some embodiments, the apparatus may further include:

[0207] The sixth module is used to periodically send a heartbeat request to the target shore end through the target ship end;

[0208] The seventh module is used to determine the network quality of the target ship end and the target shore end according to the timeliness of the heartbeat reply from the target shore end to the target ship end based on the heartbeat request;

[0209] The eighth module is used to stop the data transmission from the target ship end to the target shore end when the delay of the network quality is greater than a preset threshold.

[0210] In some embodiments, the apparatus may further include:

[0211] The ninth module is used for calculating MD5 of the synchronized data to obtain the verification MD5 when the target shore completes the synchronization of all data blocks of the compressed package;

[0212] The tenth module is used to compare the verification MD5 with the file MD5 in the compressed information corresponding to the compressed package;

[0213] The eleventh module is used to retransmit feedback to the target ship when the comparison results are inconsistent.

[0214] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0215] On the other hand, an embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned hydrate stability region bottom boundary prediction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0216] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0217] like Fig.10 As shown, Fig.10 The hardware structure of an electronic device 1000 of another embodiment is illustrated. The electronic device 1000 includes:

[0218] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0219] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the network node population optimization method of the embodiment of the present invention;

[0220] Input / output interface 1003, used to implement information input and output;

[0221] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0222] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0223] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0224] The electronic device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0225] The contents of the method embodiments of the present invention are all applicable to the electronic device embodiments. The functions specifically implemented by the electronic device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0226] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0227] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read to Only Memory, CD to ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0228] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0229] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0230] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0231] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0232] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD to ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.

[0233] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented by the present invention. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0234] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of various functional modules in the device disclosed in the present invention, the actual implementation of the module will be understood within the conventional skills of engineers. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0235] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0236] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution device, apparatus or device (such as a computer-based device, a device including a processor, or other device that can fetch instructions from an instruction execution device, apparatus or device and execute the instructions), or in conjunction with such instruction execution device, apparatus or device. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution device, apparatus or device, or in conjunction with such instruction execution device, apparatus or device.

[0237] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0238] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0239] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0240] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0241] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A ship-shore weak network data synchronization method, characterized in that: The following steps are involved: Grouping and compressing the first data to be synchronized in the target ship end to obtain second data; the second data includes at least one compressed package and compression information corresponding to the compressed package; The compressed information is sent to the target shore end, and the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed packet in the second data; the transmission mode includes transmission from the beginning and transmission at a breakpoint, and the transmission progress of the transmission from the beginning is 0; Based on the transmission mode and the transmission progress, initializing metadata at the target ship end and the target shore end respectively to complete transmission preparation; Transmitting the compressed packets of the target ship end to the target shore end in blocks in sequence based on a sliding window; Wherein, the sliding window includes a preset number of data blocks arranged in sequence. When the target shore completes the synchronization of any number of the data blocks in front of the sliding window, the sliding window is moved back by a corresponding number of the data blocks until the target shore completes the synchronization of all the data blocks of the compressed package.

2. The ship-shore weak network data synchronization method according to claim 1, characterized in that: The compressed information includes the file name, file size, file MD5 and last modification time; the first data to be synchronized in the target ship end is grouped and compressed to obtain the second data, including the following steps: Sort the data files in the first data according to modification time to obtain a file sequence; Based on a preset size benchmark, sequentially traverse the file sequence and group all the data files in the file sequence into a plurality of data groups; Perform file compression on each of the data groups using a preset compression algorithm to obtain the corresponding compressed package; Among them, the file name is obtained based on the combination of the names of all the data files in the data group, the file size and the file MD5 are calculated based on a preset algorithm, and the last modification time is determined based on the modification time of the last data file in the data group.

3. The ship-shore weak network data synchronization method according to claim 1, characterized in that: The compressed information includes the file name, file size, file MD5 and last modification time; the target shore compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed package in the second data, including the following steps: The target shore end compares the file name based on the name in the file information of each file in the transmitted file; the file information is determined according to the compression information of the compressed package historically transmitted by the target ship end; When the file name of the compressed package does not exist in the transmitted file of the target shore, determine that the transmission mode of the compressed package is the retransmission; otherwise, compare the file MD5 with the MD5 in the file information of the target file with the same file name by the target shore; When the MD5 in the file information of the target file is inconsistent with the MD5 of the file, determining that the transmission mode of the compressed package is the retransmission; otherwise, comparing the file size based on the size in the file information of the target file by the target shore; When the size in the file information of the target file is inconsistent with the file size, determining that the transmission mode of the compressed package is the re-scratch transmission; otherwise, determining that the transmission mode of the compressed package is the breakpoint transmission, and determining the transmission progress based on the actual file content of the target file; The target shore end compares the last modification time based on the target file. When the modification time of the target file is inconsistent with the last modification time, an early warning reminder is sent to the target ship end, and then the transmission mode and the transmission progress of the compressed package are determined in response to the selection instruction of the target ship end.

4. The ship-shore weak network data synchronization method according to claim 1, characterized in that: The method of sequentially transmitting the compressed packets of the target ship end to the target shore end in blocks based on the sliding window comprises the following steps: The compressed packet of the target ship is processed into blocks based on a preset data size to obtain a plurality of sequentially arranged data blocks; Selecting the preset number of data blocks after the transmission progress as synchronization data through the sliding window; Sending each of the data blocks in the synchronization data to the target shore end in sequence, so that the target shore end synchronizes the data blocks in the synchronization data one by one; obtaining the synchronization progress of the data blocks in the synchronization data by the target shore end; When the synchronization progress does not change compared to the transmission progress, the synchronization data is retransmitted; otherwise, the transmission progress is updated based on the synchronization progress, and the step of selecting the preset number of data blocks after the transmission progress as the synchronization data through the sliding window is returned to execute until the target shore completes the synchronization of all the data blocks of the compressed package.

5. The ship-shore weak network data synchronization method according to claim 1, characterized in that: The method further comprises the following steps: When the target shore end completes the synchronization of the target number of data blocks, or when the target shore end receives the heartbeat request of the target ship end, the synchronization progress of the data blocks of the compressed package is sent to the target ship end through the target shore end.

6. The ship-shore weak network data synchronization method according to claim 1, characterized in that: The method further comprises the following steps: Periodically sending a heartbeat request to the target shore end through the target ship end; Determine the network quality of the target ship end and the target shore end according to the timeliness of the heartbeat reply of the target shore end to the target ship end based on the heartbeat request; When the delay of the network quality is greater than a preset threshold, the data transmission from the target ship end to the target shore end is stopped.

7. The ship-shore weak network data synchronization method according to claim 1, characterized in that: The method further comprises the following steps: When the target shore end completes synchronization of all the data blocks of the compressed package, MD5 calculation is performed on the synchronized data to obtain verification MD5; Compare the verification MD5 with the file MD5 in the compressed information corresponding to the compressed package; When the comparison results are inconsistent, retransmission feedback is performed to the target ship end.

8. A ship-shore weak network data synchronization device, characterized in that: include: The first module is used to group and compress the first data to be synchronized in the target ship end to obtain the second data; The second data includes at least one compressed package and compression information corresponding to the compressed package; The second module is used to send the compressed information to the target shore end, and the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed packet in the second data; the transmission mode includes transmission from the beginning and breakpoint transmission, and the transmission progress of the transmission from the beginning is 0; The third module is used to initialize metadata at the target ship end and the target shore end respectively based on the transmission mode and the transmission progress to complete the transmission preparation; A fourth module is used to sequentially transmit the compressed packets of the target ship end to the target shore end in blocks based on a sliding window; Wherein, the sliding window includes a preset number of data blocks arranged in sequence. When the target shore completes the synchronization of any number of the data blocks in front of the sliding window, the sliding window is moved back by a corresponding number of the data blocks until the target shore completes the synchronization of all the data blocks of the compressed package.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images