Distributed cross-network segment file transmission method

Abstract

The invention discloses a distributed cross-network segment file transmission method. The method comprises the following steps of S1, deploying a file transmission service in a local area network; s2, carrying out transmission node registration among the local area networks through a file transmission service; s3, determining a transmission path according to whether a target node is contained in the local area network; s4, after a transmission path is obtained, encrypting the file fragments according to path information, and sending the encrypted file fragments to a next transmission node; s5, the receiving node decrypts the data packet according to the source node information and continues to transmit the data packet according to the routing information until the data packet arrives at the destination; s6, after receiving all the fragmented data, the receiver returns confirmation information of successful receiving to the sender; and S7, if the sender does not receive the receiving confirmation for a long time, re-sending the unconfirmed fragmented data packet. According to the invention, the file transmission efficiency and security are improved, and the reliability and stability of the system are enhanced.

Description

Technical Field

[0001] The present application relates to the field of file transmission, and in particular to a distributed cross-network segment file transmission method. Background Art

[0002] With the development of informatization, cross-segment file transfer plays a vital role in enterprises and data centers. Common file transfer protocols include FTP, SFTP, HTTP, etc., which are mainly used for file transfer in the same network segment or close-range network environment. However, with the increasing demand for cross-regional and cross-segment transmission, traditional transmission protocols have exposed many problems in cross-segment transmission.

[0003] Regarding the above-mentioned related technologies, the inventors believe that the following defects exist: the network configuration is complex. The traditional file transfer protocol realizes file transfer from one node to another by establishing a point-to-point connection. When the number of nodes in the network increases, it is necessary to establish a complex network configuration, which increases the operation and maintenance costs. The transmission reliability is insufficient. Due to network delays and packet loss, it is difficult to ensure the stability of transmission. The triggering time and mechanism of the breakpoint resume function after transmission failure are not flexible enough, resulting in low transmission efficiency, and therefore needs to be improved. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present application provides a distributed cross-segment file transfer method, aiming to solve the above problems.

[0005] A distributed cross-segment file transmission method includes the following steps: Step S1: deploy a file transfer service in a local area network, ensure that the service can communicate with all nodes in the local area network, and ensure that the file transfer service network of other nodes is connected; Step S2: Before performing file transfer authentication configuration, each local area network registers a transfer node through a file transfer service; Step S3: When a file needs to be transferred from one node to another, a transfer path is determined based on whether the local area network contains the target node; Step S4: After obtaining the transmission path, encrypt the file fragments according to the path information and send them to the next transmission node; Step S5: The receiving node decrypts the data packet according to the source node information and continues to transmit according to the routing information until it reaches the destination; Step S6: After receiving all the fragmented data, the receiver combines them into a complete file and returns a confirmation message of successful reception to the sender; Step S7: If the sender does not receive a reception confirmation for a long time, it resends the unconfirmed fragmented data packets.

[0006] By adopting the above technical solution, a file transfer service is deployed in a local area network, and it is ensured that the service can communicate with all node networks in the local area network, and at the same time, it is ensured that the file transfer service network of other nodes is connected. Before the file transfer authentication configuration is performed, the transfer nodes are registered between the local area networks through the file transfer service, which simplifies the complex network configuration required for cross-segment file transfer, reduces the operation and maintenance costs, and enhances the security and trust of the system. When a file needs to be transferred from one node to another, the optimal transmission path is determined according to whether the target node is included in the local area network, and then the file is fragmented and encrypted, and then sent to the next transmission node. The receiving node decrypts the data packet according to the source node information and continues to transmit according to the routing information until it reaches the destination. After receiving all the fragmented data, the receiver combines them into a complete file and returns a confirmation message of successful reception to the sender; if the confirmation is not received for a long time, the unconfirmed fragmented data packet is resent. This process not only improves the efficiency and security of file transmission, but also enhances the reliability and stability of the system, supports large file transmission, simplifies error handling, and optimizes resource utilization, providing a solid foundation for achieving efficient, secure and easy-to-manage large-scale data transmission.

[0007] Optionally, the file transfer service includes a file transfer module, an encryption module, a message queue module and a transmission node management module. The file transfer module is used to segment the file and write it into the message queue module. The encryption component encrypts the content of the file transfer. The message queue module is used as a hub for transmitting the segmented file content. The transmission node management module is used to save a list of mutually authenticated nodes and obtain the transmission path before file transfer.

[0008] By adopting the above technical solution, the file transfer module is responsible for slicing the file and writing it to the message queue module, which not only improves the transmission efficiency, but also reduces memory usage and supports reading while transmitting; the encryption component encrypts the file content to ensure the confidentiality and integrity of the data during transmission, and prevents unauthorized access and data tampering; the message queue module serves as the hub for the transmission of slicing file content, effectively managing and scheduling the data slicing to be transmitted, optimizing the system throughput, and supporting the breakpoint resume function to improve the reliability and stability of the system; the transmission node management module saves the list of mutually authenticated nodes and obtains the optimal path before file transmission, which simplifies the complex network configuration required for cross-segment file transmission and reduces operation and maintenance costs.

[0009] Optionally, the file transfer module supports registering file data sources of Minio, FTP, and SFTP types.

[0010] Optionally, the specific steps of encrypting the file in step S4 are as follows: Step S4.1: registering each other between the transmission nodes, and saving each other's public key and private key, as well as the access method of the message queue module; Step S4.2: When the file transfer module needs to send a file, the file segment is encrypted using the public key of the recipient; Step S4.3: Write the encrypted file segments into the message queue module and send them to the next transmission node; Step S4.4: The receiving node reads the data packet from the message queue module and decrypts the data packet using the corresponding private key according to the source node information.

[0011] By adopting the above technical solution, this process enhances the security of file content and ensures that even if the data is intercepted, unauthorized third parties cannot read the file content; at the same time, it provides a means of verifying data integrity and can detect potential data damage or tampering; in addition, this mechanism realizes security authentication between nodes, reduces the risk of unauthorized access, and simplifies key management. Throughout the entire transmission link, file fragments always remain encrypted until they reach the final destination, thereby ensuring end-to-end security. This encryption method also supports breakpoint resume function, which means that files can continue to be securely transmitted based on existing keys even after network interruption without renegotiating keys; in summary, the above encryption steps not only enhance the security and data integrity of file transmission and improve the overall security of the system, but also simplify key management and support flexible and reliable data transmission.

[0012] Optionally, the access method of the message queue module in step S4.1 includes IP, port, user name and password.

[0013] Optionally, the fragment size is customized according to the file size to be transferred, specifically: files with a size less than 10MB are set as small files, and the fragment size is set to 512KB-1MB; files with a size of 10MB-1GB are set as medium files, and the fragment size is set to 4MB-16MB; files with a size greater than 1GB are set as large files, and the fragment size is set to 16MB-32MB.

[0014] By adopting the above technical solutions, this approach can optimize network bandwidth utilization, improve transmission efficiency, and enhance fault tolerance. For small files, smaller fragments can be quickly transmitted; for large files, larger fragments reduce the total number of fragments and speed up the overall transmission speed; in addition, when a transmission error or network interruption occurs, only the specific fragment that was not successfully transmitted needs to be retransmitted instead of the entire file, thereby speeding up the recovery process. This strategy also simplifies the file transfer management process, facilitates configuration and maintenance, and improves the user experience, especially in scenarios that process large amounts of data or are time-sensitive. In summary, adjusting the fragment size based on the file size not only improves the efficiency and reliability of file transfer, but also enhances the flexibility of the system and user satisfaction.

[0015] Optionally, when reading file data from a file data source, read the specified file through the client provided by the file data source, convert it into a file stream, and convert it into a byte array when reading in slices.

[0016] By adopting the above technical solutions, the efficiency and reliability of file transfer are significantly improved. This method supports reading while transferring, reduces memory usage, and enables the system to process files of any size without causing memory overflow. At the same time, it enhances the system's fault tolerance and recovery efficiency, because even when the network is interrupted or an error occurs, only the specific fragment that was not successfully delivered needs to be retransmitted instead of the entire file. In addition, by dynamically adjusting the fragment size to optimize network bandwidth utilization and reduce congestion or packet loss caused by large data packets, each fragment is processed as an independent data unit, simplifying the error detection and correction mechanism and focusing on solving problems for a single fragment. Finally, using the client provided by the file data source (such as Minio SDK, Apache Commons Net's FtpClient, etc.) to access different types of storage services, it ensures support for multiple file protocols and improves the flexibility and compatibility of the solution.

[0017] Optionally, the content transmitted by the file transfer module to the message queue module is a byte stream, and the byte stream includes a routing length segment, a routing content segment, a source node segment, a fragment sequence number segment, a file length segment, a file content segment, and a file hash value segment.

[0018] By adopting the above technical solution, when the file transfer component reads a data packet, it obtains the length of the routing content through the routing length segment, reads the routing content segment based on the length, then reads the source node segment to obtain the source node number, then reads the fragment sequence number segment to obtain the index of the file fragment, then reads the file length segment, reads the file content segment based on the file length, and finally reads the file hash value segment to determine whether the file is complete after the file is merged.

[0019] Optionally, the file transfer module monitors the data in the message queue module, and once a new file segment is detected, it is immediately consumed, parsed, and transferred to the next node.

[0020] By adopting the above technical solution, the file transfer component monitors the data in the message queue module, and immediately consumes, parses and transmits it to the next node when a new file segment is detected. This mechanism significantly improves the efficiency and reliability of file transfer. By processing the newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, speeding up the speed of the entire file transfer process. At the same time, this instant processing method helps to quickly discover and respond to possible transmission errors or failures, and supports the breakpoint resume function, that is, after the network is interrupted, it can continue to transmit from the last successfully transmitted segment without retransmitting the entire file, thereby improving the success rate of large file transmission across network segments. In addition, dividing the file into multiple small blocks for processing not only optimizes the use of network bandwidth and other computing resources, but also simplifies the error handling logic, so that most of the data can be correctly transmitted even in poor network conditions. The design based on the message queue module also supports asynchronous operation, allowing the file transfer service to work in parallel with other services, improving the concurrent processing capability of the system.

[0021] In summary, the present application includes at least one of the following beneficial technical effects: 1. Deploy the file transfer service in a local area network and ensure that the service can communicate with all nodes in the local area network, and ensure the network connectivity with the file transfer service of other nodes. Before configuring the file transfer authentication, the transfer nodes are registered between the local area networks through the file transfer service. This simplifies the complex network configuration required for cross-segment file transfer, reduces the operation and maintenance costs, and enhances the security and trust of the system. When a file needs to be transferred from one node to another, the optimal transmission path is determined based on whether the target node is included in the local area network, and then the file is fragmented and encrypted, and then sent to the next transmission node. The receiving node decrypts the data packet according to the source node information and continues to transmit according to the routing information until it reaches the destination. After receiving all the fragmented data, the receiver combines them into a complete file and returns the confirmation information of successful reception to the sender; if the confirmation is not received for a long time, the unconfirmed fragmented data packet is resent. This process not only improves the efficiency and security of file transfer, but also enhances the reliability and stability of the system, supports large file transfer, simplifies error handling, and optimizes resource utilization, providing a solid foundation for achieving efficient, secure and easy-to-manage large-scale data transmission.

[0022] 2. This process enhances the security of file contents, ensuring that even if the data is intercepted, unauthorized third parties cannot read the file contents; at the same time, it provides a means of verifying data integrity and can detect potential data corruption or tampering; in addition, this mechanism implements security authentication between nodes, reduces the risk of unauthorized access, and simplifies key management. Throughout the entire transmission link, file segments remain encrypted until they reach the final destination, thereby ensuring end-to-end security. This encryption method also supports breakpoint resume function, which means that files can continue to be securely transmitted based on existing keys even after a network interruption, without the need to renegotiate keys; in summary, the above encryption steps not only enhance the security and data integrity of file transmission and improve the overall security of the system, but also simplify key management and support flexible and reliable data transmission.

[0023] 3. This approach can optimize network bandwidth utilization, improve transmission efficiency, and enhance fault tolerance. For small files, smaller fragments can be quickly transmitted; for large files, larger fragments reduce the total number of fragments and speed up the overall transmission speed; in addition, when a transmission error or network interruption occurs, only the specific fragment that was not successfully transmitted needs to be retransmitted instead of the entire file, thereby speeding up the recovery process. This strategy also simplifies the file transfer management process, facilitates configuration and maintenance, and improves the user experience, especially in scenarios that process large amounts of data or are time-sensitive. In summary, adjusting the fragment size based on the file size not only improves the efficiency and reliability of file transfer, but also enhances the flexibility of the system and user satisfaction.

[0024] 4. Significantly improve the efficiency and reliability of file transfer. This method supports reading while transferring, reduces memory usage, and enables the system to process files of any size without causing memory overflow. At the same time, it enhances the system's fault tolerance and recovery efficiency, because even when the network is interrupted or an error occurs, only the specific fragment that was not successfully delivered needs to be retransmitted instead of the entire file. In addition, by dynamically adjusting the fragment size to optimize network bandwidth utilization, reduce congestion or packet loss caused by large data packets, and process each fragment as an independent data unit, simplifying the error detection and correction mechanism, focusing on solving problems for a single fragment. Finally, using the client provided by the file data source (such as Minio SDK, Apache Commons Net's FtpClient, etc.) to access different types of storage services, it ensures support for multiple file protocols and improves the flexibility and compatibility of the solution.

[0025] 5. The file transfer component monitors the data in the message queue module, and immediately consumes, parses and transmits it to the next node when a new file segment is detected. This mechanism significantly improves the efficiency and reliability of file transfer. By processing the newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, speeding up the entire file transfer process. At the same time, this instant processing method helps to quickly discover and respond to possible transmission errors or failures, and supports breakpoint resume function, that is, after the network is interrupted, it can continue to transmit from the last successfully transmitted segment without retransmitting the entire file, thereby improving the success rate of large file transmission across network segments. In addition, dividing the file into multiple small blocks for processing not only optimizes the use of network bandwidth and other computing resources, but also simplifies the error handling logic, so that even in poor network conditions, most of the data can be correctly transmitted. The design based on the message queue module also supports asynchronous operation, allowing the file transfer service to work in parallel with other services, improving the system's concurrent processing capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of an embodiment of the present application.

[0027] Figure 2 It is a flowchart of the file transfer service in the embodiment of the present application.

[0028] Figure 3 It is a schematic diagram of the format of the content transmitted from the file transfer component module to the message queue module in the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to facilitate the understanding of the present application, the present application is described in more detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that when an element is described as "fixed to" another element, it can be directly on the other element, or there can be one or more centered elements therebetween. When an element is described as "connected to" another element, it can be directly connected to the other element, or there can be one or more centered elements therebetween. The terms "vertical", "horizontal", "left", "right" and similar expressions used in this specification are for illustrative purposes only.

[0030] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0031] The present application embodiment discloses a distributed cross-segment file transmission method. Figure 1 and Figure 2 , including the following steps: Step S1: deploy a file transfer service in a local area network, ensure that the service can communicate with all node networks in the local area network, and ensure that the file transfer service network of other nodes is connected; Step S2: before performing file transfer authentication configuration, each local area network registers the transfer node through the file transfer service; Step S3: when it is necessary to transfer a file from one node to another, determine the transmission path according to whether the local area network contains the target node; Step S4: after obtaining the transmission path, encrypt the file fragments according to the path information and send them to the next transmission node; Step S5: the receiving node decrypts the data packet according to the source node information, and continues to transmit according to the routing information until it reaches the destination; Step S6: after receiving all the fragment data, the receiver combines them into a complete file and returns the confirmation information of successful reception to the sender; Step S7: if the sender does not receive the reception confirmation for a long time, it resends the unconfirmed fragment data packet. Deploy a file transfer service in a local area network and ensure that the service can communicate with all node networks in the local area network, and ensure network connectivity with file transfer services of other nodes. Before configuring file transfer authentication, each local area network registers the transfer node through the file transfer service, which simplifies the complex network configuration required for cross-segment file transfer, reduces operation and maintenance costs, and enhances the security and trust of the system. When a file needs to be transferred from one node to another, the optimal transmission path is determined based on whether the target node is included in the local area network, and then the file is fragmented and encrypted, and then sent to the next transmission node. The receiving node decrypts the data packet according to the source node information and continues to transmit according to the routing information until it reaches the destination. After receiving all the fragmented data, the receiver combines them into a complete file and returns a confirmation message of successful reception to the sender; if the confirmation is not received for a long time, the unconfirmed fragmented data packet is resent. This process not only improves the efficiency and security of file transfer, but also enhances the reliability and stability of the system, supports large file transfer, simplifies error handling, and optimizes resource utilization, providing a solid foundation for achieving efficient, secure and easy-to-manage large-scale data transmission.

[0032] Reference Figure 1 and Figure 2, the file transfer service includes a file transfer module, an encryption module, a message queue module, and a transmission node management module. The file transfer module is used to slice and process the registrable file data source and write it to the message queue module. The file transfer module supports the registration of file data sources of Minio, FTP, and SFTP types. The encryption component encrypts the content of the file transfer, the message queue module is used as the hub for the transmission of the sliced ​​file content, and the transmission node management module is used to save the list of mutually authenticated nodes and obtain the transmission path before the file is transferred. The file transfer module is responsible for slicing the file and writing it to the message queue module, which not only improves the transmission efficiency, but also reduces the memory usage and supports reading while transferring; the encryption component encrypts the file content to ensure the confidentiality and integrity of the data during the transmission process, and prevents unauthorized access and data tampering; the message queue module, as the hub of the sliced ​​file content transmission, effectively manages and schedules the data slices to be transmitted, optimizes the system throughput, and supports the breakpoint resume function to improve the reliability and stability of the system; the transmission node management module saves the list of mutually authenticated nodes and obtains the optimal path before the file is transferred, which simplifies the complex network configuration required for cross-segment file transmission and reduces the operation and maintenance cost.

[0033] Reference Figure 1 and Figure 2 , the file transfer module listens to the data in the message queue module. Once a new file segment is detected, it will be consumed, parsed, and transmitted to the next node immediately. The file transfer component listens to the data in the message queue module, and consumes, parses, and transmits it to the next node immediately when a new file segment is detected. This mechanism significantly improves the efficiency and reliability of file transfer. By processing the newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, speeding up the entire file transfer process. At the same time, this instant processing method helps to quickly discover and respond to possible transmission errors or failures, and supports breakpoint resume function, that is, after the network is interrupted, it can continue to transmit from the last successfully transmitted segment without retransmitting the entire file, thereby improving the success rate of large file transmission across network segments. In addition, dividing the file into multiple small blocks for processing not only optimizes the use of network bandwidth and other computing resources, but also simplifies the error handling logic, so that most of the data can be correctly transmitted even in poor network conditions. The design based on the message queue module also supports asynchronous operation, allowing the file transfer service to work in parallel with other services, improving the concurrent processing capability of the system.

[0034] Reference Figure 1 and Figure 2, the specific steps of encrypting the file in step S4 are as follows: Step S4.1: register with each other between transmission nodes, and save the other party's public key and private key, as well as the access method of the message queue module; Step S4.2: when the file transmission module needs to send a file, use the public key of the recipient to encrypt the file fragment; Step S4.3: write the encrypted file fragment into the message queue module and send it to the next transmission node; Step S4.4: the receiving node reads the data packet from the message queue module and uses the corresponding private key to decrypt the data packet according to the source node information. Before configuring the file transmission authentication, each LAN needs to register the transmission node through the file transmission module.

[0035] Reference Figure 1 and Figure 2 , file transfer component A needs to initiate an authentication request to file transfer component B. File transfer component A generates public key A and private key A, file transfer component B generates public key B and private key B, file transfer component A saves private key A and public key B, file transfer component B saves public key A and private key B, and saves the connection information of the other party's transmission node and the access method of the message queue module (IP, port, user name, password). File transfer component B and file transfer component C, file transfer component B and file transfer component D, file transfer component C and file transfer component D also need to do this authentication. Among them, the public-private key pair is generated by the RSA algorithm. In the subsequent data transmission process, it is encrypted by the public key and decrypted by the private key by the recipient. This process enhances the security of file contents, ensuring that even if the data is intercepted, unauthorized third parties cannot read the file contents; at the same time, it provides a means of verifying data integrity and can detect potential data corruption or tampering; in addition, this mechanism implements security authentication between nodes, reduces the risk of unauthorized access, and simplifies key management. Throughout the entire transmission link, file fragments remain encrypted until they reach the final destination, thereby ensuring end-to-end security. This encryption method also supports breakpoint resume function, which means that files can continue to be securely transmitted based on existing keys even after a network interruption, without the need to renegotiate keys; in summary, the above encryption steps not only enhance the security and data integrity of file transmission and improve the overall security of the system, but also simplify key management and support flexible and reliable data transmission.

[0036] Reference Figure 1 and Figure 2When it is necessary to transfer a file on node A to node F, the file transfer service first confirms whether the local area network contains node F. If so, the returned transmission path is [node A->file transfer component A->node F]. If the local area network does not contain node F, the request is sent to the only registered node device (in this example, file transfer component B is registered, and a request to obtain the path is sent to the file transfer component). Similarly, file transfer component B returns path 1 [file transfer component B->file transfer component C->node F] and path 2 [file transfer component B->file transfer component D->file transfer component C->node F]. According to the length of the transmission path, the final selected transmission path is [node A->file transfer component A->file transfer component B->file transfer component C->node F].

[0037] Reference Figure 1 , Figure 2 and Figure 3After obtaining the transmission path, the file transfer component A reads the file on node A in slices. The slice size is customized according to the size of the file to be transferred. Files with a size less than 10MB are set as small files, and the slice size is set to 512KB-1MB; files with a size of 10MB-1GB are set as medium files, and the slice size is set to 4MB-16MB; files with a size greater than 1GB are set as large files, and the slice size is set to 16MB-32MB. When reading file data from the file data source, the client provided by the file data source (such as Minio SDK, apache common net's FtpClient) reads the specified file and converts it into a file stream, and converts it into a byte array when reading the slices. This approach can optimize network bandwidth utilization, improve transmission efficiency, and enhance fault tolerance. For small files, smaller slices can complete the transmission quickly; for large files, larger slices reduce the total number of slices and speed up the overall transmission speed; in addition, when a transmission error or network interruption occurs, only the specific slice that was not successfully transmitted needs to be retransmitted instead of the entire file, thereby speeding up the recovery process. This strategy also simplifies the file transfer management process, facilitates configuration and maintenance, and improves the user experience, especially in scenarios that process large amounts of data or are time-sensitive. The efficiency and reliability of file transfer are significantly improved. This method supports reading while transferring, reduces memory usage, and enables the system to process files of any size without causing memory overflow. At the same time, it enhances the system's fault tolerance and recovery efficiency, because even when a network interruption or error occurs, only the specific fragment that was not successfully delivered needs to be retransmitted instead of the entire file. In addition, by dynamically adjusting the fragment size to optimize network bandwidth utilization, congestion or packet loss caused by large data packets can be reduced. Each fragment is processed as an independent data unit, which simplifies the error detection and correction mechanism and focuses on solving problems with a single fragment. Finally, using the client provided by the file data source (such as MinioSDK, Apache Commons Net's FtpClient, etc.) to access different types of storage services ensures support for multiple file protocols and improves the flexibility and compatibility of the solution.

[0038] Reference Figure 1 , Figure 2 and Figure 3, the content transmitted from the file transfer module to the message queue module is a byte stream, and the byte stream includes a route length segment, a route content segment, a source node segment, a fragment sequence number segment, a file length segment, a file content segment, and a file hash value segment. When the file transfer component reads a data packet, it obtains the length of the route content through the route length segment, reads the route content segment according to the length, then reads the source node segment to obtain the source node number, then reads the fragment sequence number segment to obtain the index of the file fragment, then reads the file length segment, reads the file content segment according to the file length, and finally reads the file hash value segment to determine whether the file is complete after the file is merged. For example, when the file transfer component reads a data packet, it obtains the first 8 bits of the byte stream, obtains the length of the route content, reads the route content according to the length, then reads the last 32 bytes of data, indicating the source node number, the next 8 bits indicate the fragment sequence number, which is used to indicate the index of the file fragment, the next 8 bits indicate the file length, obtains the file content according to the file length, and the last 256 bits indicate the hash value of the file, which is used to determine whether the file is complete after the file is merged.

[0039] The implementation principle of a distributed cross-segment file transfer method in the embodiment of the present application is as follows: deploy a file transfer service in a local area network, and ensure that the service can communicate with all node networks in the local area network, and at the same time ensure that the file transfer service network of other nodes is connected. Before performing file transfer authentication configuration, each local area network registers the transfer node through the file transfer service, which simplifies the complex network configuration required for cross-segment file transfer, reduces operation and maintenance costs, and enhances the security and trust of the system. When a file needs to be transferred from one node to another, the optimal transmission path is determined based on whether the local area network contains the target node, and then the file is fragmented and encrypted, and then sent to the next transmission node. The receiving node decrypts the data packet according to the source node information, and continues to transmit according to the routing information until it reaches the destination. After receiving all the fragmented data, the receiver combines them into a complete file and returns a confirmation message of successful reception to the sender; if no confirmation is received for a long time, the unconfirmed fragmented data packet is resent. This process not only improves the efficiency and security of file transfer, but also enhances the reliability and stability of the system, supports large file transfer, simplifies error handling, and optimizes resource utilization, providing a solid foundation for achieving efficient, secure and easy-to-manage large-scale data transfer.

[0040] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A distributed cross-segment file transmission method, characterized in that: The following steps are involved: Step S1: deploy a file transfer service in a local area network, ensure that the service can communicate with all nodes in the local area network, and ensure that the file transfer service network of other nodes is connected; Step S2: Before performing file transfer authentication configuration, each local area network registers a transfer node through a file transfer service; Step S3: When a file needs to be transferred from one node to another, a transfer path is determined based on whether the local area network contains the target node; Step S4: After obtaining the transmission path, encrypt the file fragments according to the path information and send them to the next transmission node; Step S5: The receiving node decrypts the data packet according to the source node information and continues to transmit according to the routing information until it reaches the destination; Step S6: After receiving all the fragmented data, the receiver combines them into a complete file and returns a confirmation message of successful reception to the sender; Step S7: If the sender does not receive a reception confirmation for a long time, it resends the unconfirmed fragmented data packets.

2. A distributed cross-segment file transmission method according to claim 1, characterized in that: The file transfer service includes a file transfer module, an encryption module, a message queue module and a transmission node management module. The file transfer module is used to segment the file and write it into the message queue module. The encryption component encrypts the content of the file transfer. The message queue module is used as a hub for the transmission of segmented file content. The transmission node management module is used to save a list of mutually authenticated nodes and obtain the transmission path before file transfer.

3. A distributed cross-segment file transmission method according to claim 2, characterized in that: The file transfer module supports registering file data sources of Minio, FTP, and SFTP types.

4. A distributed cross-segment file transmission method according to claim 3, characterized in that: The specific steps of encrypting the file in step S4 are as follows: Step S4.1: registering each other between the transmission nodes, and saving each other's public key and private key, as well as the access method of the message queue module; Step S4.2: When the file transfer module needs to send a file, the file segment is encrypted using the public key of the recipient; Step S4.3: Write the encrypted file segments into the message queue module and send them to the next transmission node; Step S4.4: The receiving node reads the data packet from the message queue module and decrypts the data packet using the corresponding private key according to the source node information.

5. A distributed cross-segment file transmission method according to claim 4, characterized in that: The access method of the message queue module in step S4.1 includes IP, port, user name and password.

6. The distributed cross-segment file transmission method according to claim 2 is characterized in that: The fragment size is customized according to the size of the file to be transferred. Specifically, files with a size less than 10MB are set as small files, and the fragment size is set to 512KB-1MB; files with a size of 10MB-1GB are set as medium files, and the fragment size is set to 4MB-16MB; files with a size greater than 1GB are set as large files, and the fragment size is set to 16MB-32MB.

7. A distributed cross-segment file transmission method according to claim 6, characterized in that: When reading file data from a file data source, the client provided by the file data source is used to read the specified file and convert it into a file stream. When reading in slices, it is converted into a byte array.

8. A distributed cross-segment file transmission method according to claim 7, characterized in that: The content transmitted to the message queue module by the file transfer module is a byte stream, which includes a route length segment, a route content segment, a source node segment, a fragment sequence number segment, a file length segment, a file content segment, and a file hash value segment.

9. A distributed cross-segment file transmission method according to claim 8, characterized in that: The file transfer module monitors the data in the message queue module, and once a new file segment is detected, it is immediately consumed, parsed, and transferred to the next node.

Images