Large file one-way gatekeeper passing method based on UDP (User Datagram Protocol)

Abstract

The invention discloses a UDP-based large file one-way gatekeeper passing method, and relates to the technical field of data processing. The problems of high packet loss rate, low bandwidth utilization rate and insufficient verification efficiency in large file transmission in a one-way non-feedback gatekeeper environment are solved. The method comprises the following steps: segmenting a file into small fragments based on configuration, generating metadata with verification, and compressing the metadata; a fragment transmission sequence is dynamically scheduled through a priority queue, a uniform retransmission strategy is adopted to guarantee reliability, and overtime tasks are automatically cleared; a metadata verification and deduplication mechanism is utilized to ensure that the fragments are complete and unique, and redundant storage is avoided; and recombining the fragments into a complete file according to a metadata sequence, updating a database state to be'received ', and synchronously cleaning temporary fragments and redundant logs. The method has a good application prospect in the fields of distributed storage, cloud computing, CDN (Content Delivery Network), big data transmission and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly relates to a method for one-way network isolation of large files based on UDP. Background Art

[0002] A one-way network isolation device is a one-way non-feedback data transmission device used to ensure the security of data exchange. In enterprise production activities, a network is required for efficient data exchange. When files or data need to be exchanged between networks with different security levels due to business reasons, the one-way network isolation device, as a network device adopting physical isolation technology, has higher security and reliability compared with the firewall technology based on logical isolation. In recent years, with the development of intelligent devices, more and more power plants have started to introduce intelligent terminals to complete the collection, storage, and analysis of field-level data. In order to meet the overall scheme of the national secondary security protection for power systems, one-way network isolation devices should be used for security isolation between the security zones I, II, and III in the device system. With the development of intelligent terminals, it has become normal to collect and monitor data from power generation equipment. The data of power generation equipment mainly comes from: Supervisory Control And Data Acquisition (SCADA), Distributed Control System (DCS), Fieldbus Control System (FCS), Condition Monitoring System (CMS), and a large number of newly added intelligent sensor systems. A large amount of data is collected at the local end and saved as files. When transferring large files from security zones I and II to security zone III, a method is needed to ensure the reliability of file transfer. For this reason, the present invention proposes a method for passing network isolation of large files based on the UDP protocol. This method can slice files according to specified parameters and construct metadata information of the files. Then, the sliced small file blocks and the file metadata are together transmitted using a redundant strategy with time sharing to avoid packet loss problems caused by temporary unavailability of the network isolation device. Compared with other methods for passing network isolation, the method in this paper is applicable to any one-way network isolation device and has higher hardware resource utilization and fault tolerance.

[0003] The invention with the application number CN202210127473.2 provides a file transmission method and system based on a one-way network gateway. The method includes: disassembling the file to be transmitted based on a preset disassembly rule to obtain a plurality of data packets sorted in the transmission order, where the data packet at least includes a sub-file and the sequence number information corresponding to the sub-file; sending the data packets to the server in the transmission order, and the sequence number information carried in the data packet is used for the server to judge the data packet missing status, where the data packet missing status includes whether the data packet is missing and the sequence number of the missing data packet. When using the one-way network gateway to transmit a large number of files, the method of the invention can effectively determine whether the transmitted file is lost in packets and determine the position of the lost packet, improving the reliability of the file transmission process.

[0004] The invention with the application number CN202010657656.6 provides a data transmission method based on an internal and external network isolation network gateway. The advantages of this algorithm are that before data transmission, large files are compressed to reduce the amount of data transmission. The file is sliced to reduce the impact of the uncontrollable transmission rate and success rate caused by the direct transmission of large files, increase the number of control times, and ensure the efficient transmission of the file. The dynamic slicing algorithm is used to adjust the size of the transmitted file in real time according to the actual network situation, reducing the number of failures. During the transmission process, the management of the failure of slice transmission is carried out, and different failure times are processed in different ways, reasonably reducing the pressure on the isolation network gateway and increasing the probability of transmission success. After the slice transmission is completed, the dynamic feedback algorithm is used to calculate the transmission deviation value in real time, and the sleep waiting method is used to prevent data transmission overload and ensure the stability of file transmission and the isolation network gateway.

[0005] The existing technologies have the following deficiencies: The invention with the application number CN202210127473.2 can only judge whether the file is lost in packets during the file transmission process and determine the position of the lost file. Under the condition of using a non-feedback one-way network gateway, efficient error correction of data cannot be carried out. For the method proposed in the invention with the application number CN202010657656.6, whether it is the dynamic slicing of the file or the failure replacement, its dynamic feedback algorithm requires the reverse feedback of data. If the isolation network gateway is strictly one-way and there is no reverse feedback function of data, this method cannot be used for file transmission. Summary of the Invention

[0006] To solve the problems of high packet loss rate, low bandwidth utilization rate and insufficient verification efficiency in the transmission of large files in a one-way non-feedback network gateway environment, the present invention provides a method for one-way passing of large files based on UDP, including the following steps:

[0007] S1. The slicing strategy of file slicing, according to the configuration information, the list of files to be sliced is sliced into small shards to obtain sharded files, and a metadata file is generated and compression and verification are completed;

[0008] S2. Transmission strategy for file slices. Based on the metadata file, the expiration time configured by the user, and the number of retransmissions, schedule the transmission order of the sharded files through a priority queue, combine the uniform retransmission strategy to ensure reliable file transmission, and automatically clean up expired tasks;

[0009] S3. Verification strategy for file slices. Through the metadata checksum and non-duplicate reception mechanism, ensure the integrity and uniqueness of the sharded files, avoid redundant storage, and obtain a list of verified sharded files;

[0010] S4. Merging strategy for file slices. According to the metadata file, reorganize the sharded files into a complete file in order, update the database and mark it as "received", and clean up temporary shards, invalid logs, and redundant storage files.

[0011] Furthermore, S1 specifically includes the following steps:

[0012] S11. Read the list of files to be sliced;

[0013] S12. Determine whether there are new files to be sliced in the list of files to be sliced;

[0014] S13. If there are new files that need to be sliced, read in the configuration information belonging to this file; otherwise, enter S18 after sleeping for N seconds;

[0015] S14. According to the shard size in the configuration information, slice the file to be sliced into x small shards to obtain the sharded files, where x is greater than or equal to 3. Except for the last shard, all other shards have the same byte size;

[0016] S15. According to the compression configuration in the configuration information, determine whether it is necessary to perform individual independent compression on the sharded files. When compressing the sharded files, if only a single compression algorithm is configured, or the number of configured compression algorithms is greater than or equal to the number of file shards, both belong to the use of a fixed compression strategy. The so-called dynamic compression strategy means using the compression algorithms listed in the configuration information to compress different shards respectively. Since the sizes of different shards are exactly the same, at this time, select the algorithm with the smallest compression ratio after compression as the only compression algorithm for future transmission of this file;

[0017] S16. According to the verification configuration in the configuration information, determine whether it is necessary to verify after the sharded files are transmitted. If verification is required, calculate the verification value of each sharded file according to the specified verification algorithm, and write the verification value into the metadata file;

[0018] S17. Based on the established redundant multiple retransmission strategy, transmit the sharded files and the metadata file through the UDP protocol;

[0019] S18. If the task end flag is False, continue to execute and jump to S111; otherwise, terminate the operation.

[0020] Furthermore, S2 specifically includes the following steps:

[0021] S21. Read the sharded file to be transmitted.

[0022] S22. Obtain the file shard information at the head of the priority queue, and check whether the timestamp of the shard information is less than the timestamp of the current moment; if it is less, it means there is a file to be transmitted, enter S23, otherwise it means there is no sharded file to be transmitted, sleep for N seconds, and then enter S26.

[0023] S23. Transmit the sharded file based on the UDP protocol.

[0024] S24. Check whether the transmission plan of the sharded file has been completed.

[0025] S25. If the transmission plan of the sharded file has been completed, delete the sharded file and the corresponding metadata file, and clean up the relevant information; otherwise, enter step S26.

[0026] S26. If the task end flag is False, continue to execute S21, otherwise terminate the operation.

[0027] Furthermore, in S22, the timestamp is obtained through:

[0028]

[0029] where n represents the number of the current shard.

[0030] Furthermore, S3 specifically includes the following steps:

[0031] S31. The server based on the UDP protocol receives the sharded file.

[0032] S32. Check whether the file information of the sharded file has been saved in the file information library.

[0033] S33. If it has been saved, directly enter step S34; otherwise, save the file information of the sharded file to the specified location and update the file information to the list of files to be merged.

[0034] S34. If the task end flag is False, continue to execute and jump to S31, otherwise terminate the operation.

[0035] Furthermore, S4 specifically includes the following steps:

[0036] S41. Read the shard files to be merged;

[0037] S42. Determine whether there are shard files to be merged. If so, go to S43; if not, after sleeping for N seconds, go to S45;

[0038] S43. Determine the integrity of the current shard file to be merged according to the metadata file;

[0039] S44. If all shard files of the file to be segmented have been completely saved, merge the shard files and store the corresponding metadata file in the saved file information library as a subsequent verification basis; if the saving is not completed, return to S41;

[0040] S45. If the task end flag is False, continue to execute and jump to S41; otherwise, terminate the process.

[0041] The beneficial effects of the present invention are as follows:

[0042] 1. A multiple retransmission strategy based on transmission priority is proposed to avoid file transmission failures caused by transient hardware failures and efficiently utilize the bandwidth resources of the hardware;

[0043] 2. A verification strategy based on file metadata is proposed, which can more accurately ensure the integrity of the transmitted file;

[0044] 3. A dynamic file shard compression strategy is proposed, which can effectively save transmission bandwidth. Description of the Drawings

[0045] Figure 1 It is a flowchart of a method for a large file one-way network isolation based on UDP;

[0046] Figure 2 It is a flowchart of the segmentation strategy of file slicing;

[0047] Figure 3 It is a flowchart of the transmission strategy of file slicing;

[0048] Figure 4 It is a flowchart of the verification strategy of file slicing;

[0049] Figure 5 It is a flowchart of the merging strategy of file slicing. Detailed Embodiments

[0050] To make the technical solutions and advantages in the embodiments of the present invention clearer and more understandable, the following further details the exemplary embodiments of the present invention with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than an exhaustive list of all embodiments. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0051] Embodiment 1, in combination with Figure 1 To illustrate this embodiment, the present invention provides a method for one-way network isolation of large files based on UDP, including the following steps:

[0052] S1. The splitting strategy of file slicing. According to the configuration information, the list of files to be sliced is sliced into small shards to obtain sharded files, and a metadata file is generated and compression and verification are completed;

[0053] S2. The transmission strategy of file slicing. Based on the metadata file, the expiration time configured by the user, and the number of retransmissions, the transmission order of the sharded files is scheduled through a priority queue, and the reliable transmission of the file is ensured by combining the uniform retransmission strategy, and expired tasks are automatically cleared at the same time;

[0054] S3. The verification strategy of file slicing. Through the metadata checksum and non-duplicate reception mechanism, the integrity and uniqueness of the sharded files are ensured, redundant storage is avoided, and a list of sharded files after verification is obtained;

[0055] S4. The merging strategy of file slicing. According to the metadata file, the sharded files are reorganized in order into a complete file, the database is updated and marked as "received", and temporary shards, expired logs, and redundant storage files are cleared.

[0056] Specifically, the present invention splits large files into small shards through dynamic slicing and compresses them, generates metadata to record shard information; uses a priority queue to schedule the transmission order, and combines the uniform retransmission strategy to cope with the non-feedback characteristic of the one-way network isolation, ensuring the reliability of transmission; ensures the integrity and uniqueness of the shards through the metadata checksum and non-duplicate reception mechanism, avoiding redundant storage; finally, merges the shards in order and clears temporary resources, realizing safe, efficient, and low-consumption one-way transmission of large files, and solving the deficiencies of traditional methods in terms of packet loss rate, bandwidth utilization, and verification efficiency.

[0057] In combination with Figure 2 It can be seen that S1 specifically includes the following steps:

[0058] S11. Read the list of files to be sliced;

[0059] S12. Determine whether there are new files to be sliced in the list of files to be sliced;

[0060] S13. If there is a new file to be split, the configuration information of the file shall be read; otherwise, after sleeping for N seconds, proceed to S18;

[0061] S14. According to the split size in the configuration information, split the file to be split into x small slices to obtain the sliced files, where x is greater than or equal to 3. Except for the last slice, all other slices have the same byte size;

[0062] S15. According to the compression configuration in the configuration information, determine whether to perform individual independent compression on the sliced files. When compressing the sliced files, if only a single compression algorithm is configured, or the number of configured compression algorithms is greater than or equal to the number of file slices, both belong to the use of a fixed compression strategy. The so-called dynamic compression strategy means using the compression algorithms listed in the configuration information to compress different slices separately. Since the sizes of different slices are exactly the same, at this time, select the algorithm with the smallest compression ratio after compression as the only compression algorithm for future transmission of the file;

[0063] S16. According to the verification configuration in the configuration information, determine whether verification is required after the transmission of the sliced files is completed. If verification is required, calculate the verification value of each sliced file according to the specified verification algorithm and write the verification value into the metadata file;

[0064] S17. Based on the established redundant multiple retransmission strategy, transmit the sliced files and the metadata file through the UDP protocol;

[0065] S18. If the task end flag is False, continue to execute and jump to S111; otherwise, terminate the operation.

[0066] Combined with Figure 3 it can be seen that S2 specifically includes the following steps:

[0067] S21. Read the sliced files to be transmitted;

[0068] S22. Obtain the file slice information at the head of the priority queue and check whether the timestamp of the slice information is less than the timestamp of the current moment; if it is less, it means there is a file to be transmitted, proceed to S23, otherwise it means there is no sliced file to be transmitted. After sleeping for N seconds, proceed to S26;

[0069] S23. Transmit the sliced files based on the UDP protocol;

[0070] S24. Check whether the transmission plan of the sliced files has been completed;

[0071] S25. If the transfer plan of the sharded file has been completed, delete the sharded file and the corresponding metadata file, and clean up the relevant information; otherwise, go to step S26;

[0072] S26. If the task end flag is False, continue to execute S21; otherwise, terminate the operation.

[0073] Furthermore, in S22, the timestamp is obtained through:

[0074]

[0075] where n represents the number of the current shard.

[0076] Combined with Figure 4 it can be seen that S3 specifically includes the following steps:

[0077] S31. The server based on the UDP protocol receives the sharded file;

[0078] S32. Check whether the file information of the sharded file has been saved in the file information library;

[0079] S33. If it has been saved, directly go to step S34; otherwise, save the file information of the sharded file to the specified location and update the file information to the list of files to be merged;

[0080] S34. If the task end flag is False, continue to execute and jump to S31; otherwise, terminate the operation.

[0081] Combined with Figure 5 it can be seen that S4 specifically includes the following steps:

[0082] S41. Read the sharded file to be merged;

[0083] S42. Determine whether there is a sharded file to be merged. If there is, go to S43; if not, after sleeping for N seconds, go to S45;

[0084] S43. Judge the integrity of the currently sharded file to be merged according to the metadata file;

[0085] S44. If all the sharded files of the file to be split have been completely saved, merge the sharded file and the corresponding metadata file and store them in the saved file information library as the subsequent verification basis; if the saving is not completed, return to S41;

[0086] S45. If the task end flag is False, continue to execute and jump to S41; otherwise, terminate the process.

Claims

1. A UDP-based large file one-way network gate method, characterized in that: The following steps are involved: S1. The file slicing strategy is to divide the file list to be sliced ​​into small slices according to the configuration information, obtain the slice files, generate the metadata files and complete the compression and verification; S2. The transmission strategy of file slices is based on the metadata file, the expiration time and the number of retransmissions configured by the user, and the transmission order of the sliced ​​files is scheduled through the priority queue, combined with the uniform retransmission strategy to ensure reliable file transmission, and automatically clean up expired tasks; S3. File slice verification strategy, through the metadata verification and no duplication receiving mechanism, ensure the integrity and uniqueness of the slice file, avoid redundant storage, and obtain the verified slice list; S4. The merge strategy of file slices is to reorganize the slice files into complete files in order according to the metadata file, update the database mark as "received", and clean up temporary slices, invalid logs and redundant storage files.

2. According to the UDP-based large file one-way network gate method of claim 1, it is characterized in that: S1 specifically includes the following steps: S11. Read the list of files to be segmented; S12. Determine whether there is a new file to be segmented in the list of files to be segmented; S13. If there is a new file that needs to be split, the configuration information of the file needs to be read in; otherwise, the program goes into S18 after sleeping for N seconds; S14. According to the fragment size in the configuration information, the file to be fragmented is divided into x small fragments to obtain the fragment file, where x is greater than or equal to 3, and all fragments except the last fragment have the same byte size; S15. According to the compression configuration in the configuration information, it is determined whether the segmented files need to be compressed individually. When compressing the segmented files, if only a single compression algorithm is configured, or the number of configured compression algorithms is greater than or equal to the number of file segments, a fixed compression strategy is used. The so-called dynamic compression strategy refers to using the compression algorithms listed in the configuration information to compress different segments separately. Since the sizes of different segments are exactly the same, the algorithm with the smallest compression ratio is selected after compression as the only compression algorithm for future transmission of the file. S16. According to the checksum configuration in the configuration information, determine whether the slice file needs to be checked after the transfer is completed. If a checksum is required, calculate the checksum value of each slice file according to the specified checksum algorithm and write the checksum value into the metadata file; S17. Based on the established redundant multiple retransmission strategy, the segmented file and the metadata file are transmitted via the UDP protocol; S18. If the task end flag is False, continue execution and jump to S111; otherwise terminate the operation.

3. According to the UDP-based large file one-way network gate method of claim 1, it is characterized in that: S2 specifically includes the following steps: S21. Read the fragment file to be transmitted; S22. Get the file fragment information at the head of the priority queue, and check whether the timestamp of the fragment information is less than the timestamp of the current time; if it is less than, it means that there is a file to be transmitted, and enter S23; otherwise, it means that there is no fragment file to be transmitted, and sleep for N seconds before entering S26; S23. Transmitting the fragmented file based on UDP protocol; S24. Check whether the transmission plan of the fragmented file has been completed; S25. If the transmission plan of the slice file has been completed, the slice file and the corresponding metadata file are deleted, and the relevant information is cleaned up, otherwise, the process proceeds to step S26; S26. If the task end flag is False, continue to execute S21, otherwise terminate the operation.

4. According to the UDP-based large file one-way network gate method of claim 3, it is characterized in that: In S22, the timestamp is obtained by: Get, where n represents the number of the current fragment.

5. The method for one-way large file passing through a network gate based on UDP according to claim 1, characterized in that: S3 specifically includes the following steps: S31. The server receives the fragment file based on the UDP protocol; S32. Check whether the file information of the fragment file has been saved in the file information library; S33. If it has been saved, directly proceed to step S34, otherwise save the file information of the segmented file to the specified location, and update the file information to the list of files to be merged; S34. If the task end flag is False, continue execution and jump to S31, otherwise terminate the operation.

6. The method for one-way large file passing through a network gate based on UDP according to claim 1, characterized in that: S4 specifically includes the following steps: S41. Read the fragment files to be merged; S42. Determine whether the fragment file to be merged exists. If so, proceed to S43. If not, sleep for N seconds and then proceed to S45. S43. Determine the integrity of the fragment file to be merged according to the metadata file; S44. If all the fragment files of the file to be segmented have been completely saved, the fragment files and the corresponding metadata files are merged and stored in the saved file information library as a basis for subsequent verification; if the saving is not completed, return to S41; S45. If the task end flag is False, continue execution and jump to S41, otherwise terminate the process.

Images