A distributed cross-network segment file transfer method

Abstract

The present application discloses a distributed cross-network segment file transmission method, which includes the following steps: Step S1: Deploy a file transmission service within a local area network; Step S2: Conduct transmission node registration among local area networks through the file transmission service; Step S3: Determine the transmission path according to whether the target node is included in the local area network; Step S4: After obtaining the transmission path, encrypt the file fragments according to the path information and send them to the next transmission node; Step S5: The receiving node decrypts the data packet according to the source node information and continues to transmit according to the routing information until it reaches the destination; Step S6: After the receiving party receives all the fragmented data, it returns a confirmation message of successful reception to the sending party; Step S7: If the sending party does not receive the reception confirmation for a long time, resend the unconfirmed fragmented data packets. The present application not only improves the efficiency and security of file transmission, but also enhances the reliability and stability of the system.

Description

Technical Field

[0001] The present application relates to the field of file transfer, and in particular to a distributed cross-network segment file transfer method. Background Art

[0002] With the development of informatization, cross-network segment file transfer plays a crucial role in enterprises and data centers. Common file transfer protocols include FTP, SFTP, HTTP, etc., which are mainly used for file transfer in the same network segment or short-distance network environments. However, with the increasing demand for cross-region and cross-network segment, traditional transfer protocols have exposed many problems in cross-network segment transfer.

[0003] Regarding the above related technologies, the inventors believe that the following defects exist: the network configuration is complex. Traditional file transfer protocols achieve file transfer from one node to another by establishing point-to-point connections. When the number of nodes in the network increases, complex network configurations need to be established, increasing the operation and maintenance costs. The transmission reliability is insufficient. Due to network latency and packet loss, it is difficult to ensure the stability of the transmission. The triggering time and mechanism of the breakpoint resume function after transmission failure are not flexible enough, resulting in low transmission efficiency. Therefore, it needs to be improved. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the present application provides a distributed cross-network segment file transfer method to solve the above problems.

[0005] A distributed cross-network segment file transfer method includes the following steps:

[0006] Step S1: Deploy a file transfer service within a local area network to ensure that the service can communicate with all nodes within the local area network and ensure network connectivity with the file transfer services of other nodes;

[0007] Step S2: Before performing file transfer authentication configuration, register transfer nodes between local area networks through the file transfer service;

[0008] Step S3: When it is necessary to transfer a file from one node to another node, determine the transfer path according to whether the target node is included in the local area network;

[0009] Step S4: After obtaining the transfer path, encrypt the file in slices according to the path information and send it to the next transfer node;

[0010] Step S5: The receiving node decrypts the data packet according to the source node information and continues to transfer according to the routing information until it reaches the destination;

[0011] Step S6: After receiving all the sharded data, the receiver combines them into a complete file and returns a confirmation message of successful reception to the sender;

[0012] Step S7: If the sender does not receive the reception confirmation for a long time, it resends the unconfirmed sharded data packets.

[0013] By adopting the above technical solution, a file transfer service is deployed within a local area network, ensuring that the service can communicate with all node networks within the local area network, and at the same time ensuring network connectivity with the file transfer services of other nodes. Before performing file transfer authentication configuration, each local area network registers its transfer nodes through the file transfer service, which simplifies the complex network configuration required for cross-segment file transfer, reduces the operation and maintenance costs, and enhances the security and trust of the system. When it is necessary to transfer a file from one node to another node, the optimal transfer path is determined according to whether the target node is included in the local area network, and then the file is sharded and encrypted, and then sent to the next transfer node. The receiving node decrypts the data packet according to the source node information and continues to transfer according to the routing information until it reaches the destination. After receiving all the sharded data, the receiver combines them into a complete file and returns a confirmation message of successful reception to the sender; if the confirmation is not received for a long time, the unconfirmed sharded data packets are resent. This process not only improves the efficiency and security of file transfer, but also enhances the reliability and stability of the system, supports large file transfer, simplifies error handling, and optimizes resource utilization, providing a solid foundation for realizing efficient, secure and easy-to-manage large-scale data transfer.

[0014] Optionally, the file transfer service includes a file transfer module, an encryption module, a message queue module, and a transfer node management module. The file transfer module is used to shard the file and write it into the message queue module. The encryption component encrypts the content of the file transfer. The message queue module is used as a hub for the transfer of sharded file content. The transfer node management module is used to save the list of mutually authenticated nodes and obtain the transfer path before file transfer.

[0015] By adopting the above technical solution, the file transfer module is responsible for sharding the file and writing it into the message queue module, which not only improves the transfer efficiency, but also reduces the memory occupancy and supports transfer while reading; the encryption component encrypts the file content to ensure the confidentiality and integrity of the data during transmission, preventing unauthorized access and data tampering; the message queue module, as a hub for the transfer of sharded file content, effectively manages and schedules the data shards to be transferred, optimizes the system throughput, and supports the resume function, improving the reliability and stability of the system; the transfer node management module saves the list of mutually authenticated nodes and obtains the optimal path before file transfer, simplifies the complex network configuration required for cross-segment file transfer, and reduces the operation and maintenance costs.

[0016] Optionally, the file transfer module supports registering file data sources of Minio, FTP, and SFTP types.

[0017] Optionally, the specific steps for encrypting the file in step S4 are as follows:

[0018] Step S4.1: Register with each other between transmission nodes, and save the public key and private key of the other party, as well as the access method of the message queue module.

[0019] Step S4.2: When the file transfer module needs to send a file, encrypt the file shards using the public key of the receiving party.

[0020] Step S4.3: Write the encrypted file shards into the message queue module and send them to the next transmission node.

[0021] Step S4.4: The receiving node reads the data packet from the message queue module and decrypts the data packet using the corresponding private key according to the source node information.

[0022] By adopting the above technical solution, this process enhances the security of the file content, ensuring that even if the data is intercepted, an unauthorized third party cannot read the file content; at the same time, it provides a means of verifying data integrity and can detect potential data corruption or tampering; in addition, this mechanism realizes secure authentication between nodes, reduces the risk of unauthorized access, and simplifies key management. Throughout the transmission link, the file shards remain encrypted until they reach the final destination, thus ensuring end-to-end security. Such an encryption method also supports the resume function, enabling secure transmission of files based on the existing keys even after a network interruption without the need to renegotiate keys. In summary, through the above encryption steps, not only the security and data integrity of file transmission are enhanced, the overall security of the system is improved, but also key management is simplified, and flexible and reliable data transmission is supported.

[0023] Optionally, the access method of the message queue module in step S4.1 includes IP, port, username, and password.

[0024] Optionally, the shard size is customized according to the size of the transmitted file. Specifically: files with a size less than 10MB are set as small files, and the shard size is set to 512KB - 1MB; files with a size of 10MB - 1GB are set as medium files, and the shard size is set to 4MB - 16MB; files with a size greater than 1GB are set as large files, and the shard size is set to 16MB - 32MB.

[0025] By adopting the above technical solutions, such an approach can optimize network bandwidth utilization, improve transmission efficiency, and enhance fault tolerance. For small files, smaller shards can be quickly transmitted; for large files, larger shards reduce the total number of shards and accelerate the overall transmission speed. In addition, when transmission errors or network interruptions occur, only the specific shards that have not been successfully transmitted need to be retransmitted instead of the entire file, thus accelerating the recovery process. This strategy also simplifies the file transmission management process, facilitates configuration and maintenance, and improves the user experience, especially in scenarios involving large amounts of data or time-sensitive applications. In summary, adjusting the shard size based on file size not only improves the efficiency and reliability of file transmission but also enhances the flexibility of the system and user satisfaction.

[0026] Optionally, when reading file data from a file data source, a client provided by the file data source is used to read the specified file, convert it into a file stream, and convert it into a byte array when reading in shards.

[0027] By adopting the above technical solutions, the efficiency and reliability of file transmission have been significantly improved. This method supports transmission while reading, reduces memory occupancy, enables the system to handle files of any size without causing memory overflow. At the same time, it enhances the fault tolerance and recovery efficiency of the system because even when network interruptions or errors occur, only the specific shards that have not been successfully delivered need to be retransmitted instead of the entire file. In addition, network bandwidth utilization is optimized by dynamically adjusting the shard size, reducing congestion or packet loss problems caused by large data packets. Each shard is processed as an independent data unit, simplifying the error detection and correction mechanism and focusing on solving problems for individual shards. Finally, using clients provided by the file data source (such as Minio SDK, FtpClient of Apache Commons Net, etc.) to access different types of storage services ensures support for multiple file protocols and enhances the flexibility and compatibility of the solution.

[0028] Optionally, the content transmitted by the file transmission module to the message queue module is a byte stream, which includes a routing length segment, a routing content segment, a source node segment, a shard sequence number segment, a file length segment, a file content segment, and a file hash value segment.

[0029] By adopting the above technical solutions, when the file transmission component reads a data packet, it obtains the length of the routing content through the routing length segment, reads the routing content segment according to this length, then reads the source node segment to obtain the source node number, then reads the shard sequence number segment to obtain the index of the file shard, then reads the file length segment, reads the file content segment according to the file length, and finally reads the file hash value segment to determine whether the file is complete after merging.

[0030] Optionally, the file transfer module listens for data in the message queue module. Once a new file fragment is detected, it immediately consumes, parses, and transfers it to the next node.

[0031] By adopting the above technical solution, the file transfer component listens for data in the message queue module and immediately consumes, parses, and transfers it to the next node when a new file fragment is detected. This mechanism significantly improves the efficiency and reliability of file transfer. By processing newly arrived data in real time, the waiting time of data in the message queue module can be reduced, and the speed of the entire file transfer process can be accelerated. At the same time, this immediate processing method helps to quickly discover and respond to possible transfer errors or failures, supports the resume function, that is, it can continue to transfer from the last successfully transferred fragment after a network interruption without having to re - transfer the entire file, thus increasing the success rate of cross - network segment large file transfer. In addition, dividing the file into multiple small pieces for processing not only optimizes the use of network bandwidth and other computing resources, but also simplifies the error handling logic, ensuring that most data can be correctly transferred even in poor network conditions. The design based on the message queue module also supports asynchronous operations, allowing the file transfer service to work in parallel with other services, enhancing the concurrent processing ability of the system.

[0032] In summary, the present application includes at least one of the following beneficial technical effects:

[0033] 1. Deploy a file transfer service within a local area network and ensure that the service can communicate with all nodes in the local area network and also ensure network connectivity with the file transfer services of other nodes. Before file transfer authentication configuration, each local area network registers transfer nodes through the file transfer service, which simplifies the complex network configuration required for cross - network segment file transfer, reduces operation and maintenance costs, and enhances the security and trust of the system. When a file needs to be transferred from one node to another, the optimal transfer path is determined based on whether the target node is within the local area network, then the file is fragmented and encrypted, and then sent to the next transfer node. The receiving node decrypts the data packet according to the source node information and continues to transfer according to the routing information until it reaches the destination. After the receiving party receives all the fragmented data, it combines them into a complete file and returns a confirmation message of successful reception to the sending party; if the confirmation is not received for a long time, the unconfirmed fragmented data packets are re - sent. This process not only improves the efficiency and security of file transfer, but also enhances the reliability and stability of the system, supports large file transfer, simplifies error handling, and optimizes resource utilization, providing a solid foundation for achieving efficient, secure, and easy - to - manage large - scale data transfer.

[0034] 2. This process enhances the security of the file content, ensuring that even if the data is intercepted, unauthorized third parties cannot read the file content. At the same time, it provides a means to verify data integrity, capable of detecting potential data corruption or tampering. In addition, this mechanism realizes secure authentication between nodes, reduces the risk of unauthorized access, and simplifies key management. Throughout the transmission link, the file shards remain encrypted until they reach the final destination, thus ensuring end-to-end security. Such an encryption method also supports the resume function, enabling the secure transmission of files based on the existing keys even after a network interruption without the need to renegotiate keys. In summary, through the above encryption steps, not only the security and data integrity of file transmission are enhanced, the overall security of the system is improved, but also key management is simplified, and flexible and reliable data transmission is supported.

[0035] 3. Such an approach can optimize network bandwidth utilization, improve transmission efficiency, and enhance fault tolerance. For small files, smaller shards can be transmitted quickly; for large files, larger shards reduce the total number of shards and speed up the overall transmission speed. In addition, when transmission errors or network interruptions occur, only the specific shards that have not been successfully transmitted need to be retransmitted instead of the entire file, thus accelerating the recovery process. This strategy also simplifies the file transmission management process, facilitating configuration and maintenance, while improving the user experience, especially in application scenarios involving large amounts of data or time-sensitive ones. In summary, adjusting the shard size based on the file size not only improves the efficiency and reliability of file transmission, but also enhances the flexibility and user satisfaction of the system.

[0036] 4. It significantly improves the efficiency and reliability of file transmission. This method supports transmission while reading, reducing memory occupancy, enabling the system to handle files of any size without causing memory overflow. At the same time, it enhances the fault tolerance and recovery efficiency of the system because even when network interruptions or errors occur, only the specific shards that have not been successfully delivered need to be retransmitted instead of the entire file. In addition, by dynamically adjusting the shard size to optimize network bandwidth utilization, congestion or packet loss problems caused by large data packets are reduced. Each shard is processed as an independent data unit, simplifying the error detection and correction mechanism and focusing on solving problems of individual shards. Finally, using clients provided by the file data source (such as Minio SDK, FtpClient of Apache Commons Net, etc.) to access different types of storage services ensures support for multiple file protocols and enhances the flexibility and compatibility of the solution.

[0037] 5. The file transfer component listens to the data in the message queue module and immediately consumes, parses, and transfers it to the next node when a new file fragment is detected. This mechanism significantly improves the efficiency and reliability of file transfer. By processing newly arrived data in real-time, the waiting time of data in the message queue module can be reduced, and the speed of the entire file transfer process can be accelerated. At the same time, this immediate processing method helps to quickly discover and respond to possible transfer errors or failures, supports the resume function, that is, it can continue to transfer from the last successfully transferred fragment after a network interruption without having to re-transfer the entire file, thus increasing the success rate of cross-network segment large file transfer. In addition, dividing the file into multiple small pieces for processing not only optimizes the use of network bandwidth and other computing resources, but also simplifies the error handling logic, ensuring that most data can be correctly transferred even in poor network conditions. The design based on the message queue module also supports asynchronous operations, allowing the file transfer service to work in parallel with other services and enhancing the concurrent processing ability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flowchart of an embodiment of the present application.

[0039] Figure 2 is a flowchart of the file transfer service in an embodiment of the present application.

[0040] Figure 3 is a schematic diagram of the format of the content transmitted from the file transfer component module to the message queue module in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] To facilitate the understanding of the present application, the present application will be described in more detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that when an element is expressed as "fixed to" another element, it can be directly on the other element, or there can be one or more intermediate elements therebetween. When an element is expressed as "connected to" another element, it can be directly connected to the other element, or there can be one or more intermediate elements therebetween. The terms "vertical", "horizontal", "left", "right" and similar expressions used in this specification are only for the purpose of illustration.

[0042] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not used to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0043] An embodiment of the present application discloses a distributed cross-network segment file transfer method. Refer toFigure 1 and Figure 2 , including the following steps: Step S1: Deploy a file transfer service within a local area network to ensure that the service can communicate with all nodes within the local area network and that the file transfer service network connectivity with other nodes is guaranteed; Step S2: Before file transfer authentication configuration, each local area network conducts transfer node registration through the file transfer service; Step S3: When a file needs to be transferred from one node to another, determine the transfer path based on whether the target node is included within the local area network; Step S4: After obtaining the transfer path, encrypt the file in fragments according to the path information and send it to the next transfer node; Step S5: The receiving node decrypts the data packet based on the source node information and continues to transfer according to the routing information until it reaches the destination; Step S6: After the receiving party receives all the fragmented data, combine it into a complete file and return a confirmation message of successful reception to the sending party; Step S7: If the sending party does not receive a reception confirmation for a long time, resend the unconfirmed fragmented data packets. Deploy a file transfer service within a local area network and ensure that the service can communicate with all nodes within the local area network, while also ensuring network connectivity with the file transfer services of other nodes. Before file transfer authentication configuration, each local area network conducts transfer node registration through the file transfer service. This simplifies the complex network configuration required for cross-segment file transfer, reduces operation and maintenance costs, and enhances the security and trust of the system. When a file needs to be transferred from one node to another, determine the optimal transfer path based on whether the target node is included within the local area network, then fragment and encrypt the file and send it to the next transfer node. The receiving node decrypts the data packet based on the source node information and continues to transfer according to the routing information until it reaches the destination. After the receiving party receives all the fragmented data, combine it into a complete file and return a confirmation message of successful reception to the sending party; if no confirmation is received for a long time, resend the unconfirmed fragmented data packets. This process not only improves the efficiency and security of file transfer, but also enhances the reliability and stability of the system, supports large file transfer, simplifies error handling, and optimizes resource utilization, providing a solid foundation for achieving efficient, secure, and easily managed large-scale data transfer.

[0044] Refer to Figure 1 and Figure 2, the file transfer service includes a file transfer module, an encryption module, a message queue module, and a transfer node management module. The file transfer module is used to slice the registerable file data source and write it into the message queue module. The file transfer module supports registering file data sources of Minio, FTP, and SFTP types. The encryption component encrypts the content of the file transfer. The message queue module is used as the hub for the transfer of sliced file content. The transfer node management module is used to save the list of mutually authenticated nodes and obtain the transfer path before file transfer. The file transfer module is responsible for slicing the file and writing it into the message queue module, which not only improves the transfer efficiency but also reduces the memory occupancy and supports reading and transferring simultaneously. The encryption component encrypts the file content to ensure the confidentiality and integrity of the data during the transfer process, preventing unauthorized access and data tampering. The message queue module, as the hub for the transfer of sliced file content, effectively manages and schedules the data slices to be transferred, optimizes the system throughput, and supports the resume function, improving the reliability and stability of the system. The transfer node management module saves the list of mutually authenticated nodes and obtains the optimal path before file transfer, simplifies the complex network configuration required for cross-network segment file transfer, and reduces the operation and maintenance costs.

[0045] Refer to Figure 1 and Figure 2 , the file transfer module listens to the data in the message queue module. Once a new file slice is detected, it immediately consumes, parses, and transfers it to the next node. The file transfer component listens to the data in the message queue module and immediately consumes, parses, and transfers it to the next node when a new file slice is detected. This mechanism significantly improves the efficiency and reliability of file transfer. By processing the newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, and the speed of the entire file transfer process can be accelerated. At the same time, this immediate processing method helps to quickly discover and respond to possible transfer errors or failures, supports the resume function, that is, it can continue to transfer from the last successfully transferred slice after the network is interrupted without retransferring the entire file, thus increasing the success rate of cross-network segment large file transfer. In addition, dividing the file into multiple small pieces for processing not only optimizes the use of network bandwidth and other computing resources but also simplifies the error handling logic, ensuring that most of the data can be correctly transferred even in a poor network condition. The design based on the message queue module also supports asynchronous operations, allowing the file transfer service to work in parallel with other services, improving the concurrent processing ability of the system.

[0046] Refer to Figure 1 and Figure 2, the specific steps for encrypting the file in step S4 are as follows: Step S4.1: Register with each other between transmission nodes, and save the public key and private key of the other party, as well as the access method of the message queue module; Step S4.2: When the file transmission module needs to send a file, encrypt the file shards using the public key of the receiving party; Step S4.3: Write the encrypted file shards into the message queue module and send them to the next transmission node; Step S4.4: The receiving node reads the data packet from the message queue module and decrypts the data packet using the corresponding private key according to the source node information. Before configuring the file transmission authentication, the transmission nodes between local area networks need to be registered through the file transmission module.

[0047] Refer to Figure 1 and Figure 2 , file transfer component A needs to initiate an authentication request to file transfer component B. File transfer component A generates public key A and private key A, file transfer component B generates public key B and private key B. File transfer component A saves private key A and public key B, and file transfer component B saves public key A and private key B, and also saves the connection information of the other party's transmission node and the access method of the message queue module (IP, port, username, password). File transfer component B and file transfer component C, file transfer component B and file transfer component D, file transfer component C and file transfer component D also need to perform this authentication. Among them, the public-private key pair is generated by the RSA algorithm. During the subsequent data transmission process, it is encrypted with the public key and decrypted by the receiving party with the private key. This process enhances the security of the file content, ensuring that even if the data is intercepted, an unauthorized third party cannot read the file content; at the same time, it provides a means to verify data integrity and can detect potential data corruption or tampering; in addition, this mechanism realizes secure authentication between nodes, reduces the risk of unauthorized access, and simplifies key management. Throughout the transmission link, the file shards remain encrypted until they reach the final destination, thus ensuring end-to-end security. Such an encryption method also supports the resume function. Even after a network interruption, the file can continue to be transmitted securely based on the existing key without having to renegotiate the key; In summary, through the above encryption steps, not only the security and data integrity of file transmission are enhanced, the overall security of the system is improved, but also key management is simplified, and flexible and reliable data transmission is supported.

[0048] Refer to Figure 1 and Figure 2, when it is necessary to transfer the file on node A to node F, the file transfer service first confirms whether node F is included in the local area network. If it is included, the return transfer path is [Node A -> File Transfer Component A -> Node F]. If node F is not included in the local area network, the request will be sent to the registered node device (in this example, File Transfer Component B is registered, so a request for obtaining the path is sent to the file transfer component), and so on. File Transfer Component B returns Path 1 [File Transfer Component B -> File Transfer Component C -> Node F] and Path 2 [File Transfer Component B -> File Transfer Component D -> File Transfer Component C -> Node F]. According to the length of the transfer path, the finally selected transfer path is [Node A -> File Transfer Component A -> File Transfer Component B -> File Transfer Component C -> Node F].

[0049] Refer to Figure 1 , Figure 2 and Figure 3, after obtaining the transmission path, the file transfer component A reads the file on node A in slices. The slice size is customized according to the size of the file to be transmitted. Files smaller than 10MB are set as small files, and the slice size is set to 512KB - 1MB; files with a size of 10MB - 1GB are set as medium files, and the slice size is set to 4MB - 16MB; files larger than 1GB are set as large files, and the slice size is set to 16MB - 32MB. When reading file data from the file data source, through the client provided by the file data source (such as Minio SDK, FtpClient of apache common net), the specified file is read, converted into a file stream, and converted into a byte array when reading in slices. This approach can optimize the utilization of network bandwidth, improve the transmission efficiency, and enhance the fault tolerance. For small files, smaller slices can be transmitted quickly; for large files, larger slices reduce the total number of slices and speed up the overall transmission speed. In addition, when a transmission error or network interruption occurs, only the specific slices that have not been successfully transmitted need to be retransmitted instead of the entire file, thus accelerating the recovery process. This strategy also simplifies the file transfer management process, facilitates configuration and maintenance, and improves the user experience, especially in application scenarios dealing with large amounts of data or time-sensitive ones. It significantly improves the efficiency and reliability of file transfer. This method supports reading and transmitting simultaneously, reduces memory occupancy, enabling the system to handle files of any size without causing memory overflow. At the same time, it enhances the fault tolerance and recovery efficiency of the system because even when a network interruption or error occurs, only the specific slices that have not been successfully delivered need to be retransmitted instead of the entire file. Moreover, by dynamically adjusting the slice size to optimize the utilization of network bandwidth, congestion or packet loss problems caused by large data packets are reduced. Each slice is processed as an independent data unit, simplifying the error detection and correction mechanism and focusing on solving problems of individual slices. Finally, using the client provided by the file data source (such as Minio SDK, FtpClient of Apache Commons Net, etc.) to access different types of storage services ensures support for multiple file protocols, enhancing the flexibility and compatibility of the solution.

[0050] Refer to Figure 1 , Figure 2 and Figure 3, the content transmitted by the file transfer module to the message queue module is a byte stream, which includes a routing length segment, a routing content segment, a source node segment, a shard sequence number segment, a file length segment, a file content segment, and a file hash value segment. When the file transfer component reads a data packet, it obtains the length of the routing content through the routing length segment, reads the routing content segment according to this length, then reads the source node segment to obtain the source node number, then reads the shard sequence number segment to obtain the index of the file shard, then reads the file length segment, reads the file content segment according to the file length, and finally reads the file hash value segment to determine whether the file is complete after merging. For example, when the file transfer component reads a data packet, it obtains the first 8 bits of the byte stream to get the length of the routing content, reads the routing content according to this length, then reads the data of the next 32 bits of bytes, which represents the source node number, and the next 8 bits represent the shard sequence number, used to represent the index of the file shard, the next 8 bits represent the file length, obtains the file content according to the file length, and the last 256 bits represent the file hash value, used to determine whether the file is complete after merging.

[0051] The implementation principle of a distributed cross-network segment file transfer method in an embodiment of this application is as follows: Deploy a file transfer service within a local area network and ensure that the service can communicate with all nodes within the local area network, and at the same time ensure network connectivity with the file transfer services of other nodes. Before performing file transfer authentication configuration, each local area network registers transfer nodes through the file transfer service, which simplifies the complex network configuration required for cross-network segment file transfer, reduces the operation and maintenance cost, and enhances the security and trust of the system. When it is necessary to transfer a file from one node to another node, determine the optimal transfer path according to whether the target node is included in the local area network, then perform sharding and encryption on the file, and then send it to the next transfer node. The receiving node decrypts the data packet according to the source node information and continues to transfer according to the routing information until it reaches the destination. After the receiving party receives all shard data, it combines them into a complete file and returns a confirmation message of successful reception to the sending party; if the confirmation is not received for a long time, resend the unconfirmed shard data packets. This process not only improves the efficiency and security of file transfer, but also enhances the reliability and stability of the system, supports large file transfer, simplifies error handling, and optimizes resource utilization, providing a solid foundation for realizing efficient, secure, and easy-to-manage large-scale data transfer.

[0052] The above are all preferred embodiments of this application, and the protection scope of this application is not limited by this. Therefore, all equivalent changes made according to the structure, shape, and principle of this application should be covered within the protection scope of this application.

Claims

1. A distributed cross-segment file transmission method, characterized in that: The following steps are involved: Step S1: deploy a file transfer service in a local area network, ensure that the service can communicate with all nodes in the local area network, and ensure that the file transfer service network of other nodes is connected; Step S2: Before configuring the file transfer authentication, each LAN registers with the transfer nodes through the file transfer service; Step S3: When a file needs to be transferred from one node to another, the transmission path is determined according to whether the local area network contains the target node. If the local area network does not contain the target node, the request is sent to the registered node, and the file transfer component in the registered node returns the transmission path; Step S4: After obtaining the transmission path, encrypt the file fragments according to the path information and send them to the next transmission node; Step S5: The receiving node decrypts the data packet according to the source node information and continues to transmit according to the routing information until it reaches the destination; Step S6: After receiving all the fragmented data, the receiver combines them into a complete file and returns a confirmation message of successful reception to the sender; Step S7: If the sender does not receive a reception confirmation for a long time, it resends the unconfirmed fragmented data packets.

2. A distributed cross-segment file transmission method according to claim 1, characterized in that: The file transfer service includes a file transfer module, an encryption module, a message queue module and a transmission node management module. The file transfer module is used to segment the file and write it into the message queue module. The encryption module encrypts the content of the file transfer. The message queue module is used as a hub for the transmission of segmented file content. The transmission node management module is used to save a list of mutually authenticated nodes and obtain the transmission path before file transfer.

3. A distributed cross-segment file transmission method according to claim 2, characterized in that: The file transfer module supports registering file data sources of Minio, FTP, and SFTP types.

4. A distributed cross-segment file transmission method according to claim 3, characterized in that: The specific steps of encrypting the file in step S4 are as follows: Step S4.1: registering each other between the transmission nodes, and saving each other's public key and private key, as well as the access method of the message queue module; Step S4.2: When the file transfer module needs to send a file, the file segment is encrypted using the public key of the recipient; Step S4.3: Write the encrypted file segments into the message queue module and send them to the next transmission node; Step S4.4: The receiving node reads the data packet from the message queue module and decrypts the data packet using the corresponding private key according to the source node information.

5. A distributed cross-segment file transmission method according to claim 4, characterized in that: The access method of the message queue module in step S4.1 includes IP, port, user name and password.

6. The distributed cross-segment file transmission method according to claim 2 is characterized in that: The fragment size is customized according to the size of the file to be transferred. Specifically, files with a size less than 10MB are set as small files, and the fragment size is set to 512KB-1MB; files with a size of 10MB-1GB are set as medium files, and the fragment size is set to 4MB-16MB; files with a size greater than 1GB are set as large files, and the fragment size is set to 16MB-32MB.

7. A distributed cross-segment file transmission method according to claim 6, characterized in that: When reading file data from a file data source, the client provided by the file data source is used to read the specified file and convert it into a file stream. When reading in slices, it is converted into a byte array.

8. A distributed cross-segment file transmission method according to claim 7, characterized in that: The content transmitted to the message queue module by the file transfer module is a byte stream, which includes a route length segment, a route content segment, a source node segment, a fragment sequence number segment, a file length segment, a file content segment, and a file hash value segment.

9. A distributed cross-segment file transmission method according to claim 8, characterized in that: The file transfer module monitors the data in the message queue module, and once a new file segment is detected, it is immediately consumed, parsed, and transferred to the next node.

Images