File transmission method, electronic equipment and storage medium

Abstract

The invention discloses a file transmission method. The method comprises the steps that a sending node and a receiving node subscribe file transmission task information configured by a configuration center; when the sending node determines that the sending node information is matched with own node information, the sending node obtains a target file according to target file information, splits the target file, generates a plurality of sub-files, constructs file metadata information and sub-file metadata information corresponding to the target file and the plurality of sub-files, and sends the file metadata information and the sub-file metadata information to the sending node; sending the plurality of sub-files, the file metadata information and the sub-file metadata information to a receiving node in parallel in an RPC (Remote Procedure Call) mode; the receiving node verifies the integrity according to the file metadata and the sub-file metadata and then feeds back a verification result; and after the verification result is that the file is complete, the sending node notifies the receiving node to perform file merging. The subfiles can be transmitted in parallel, the transmission efficiency is high, the file integrity is verified, and the file consistency is guaranteed. The invention further provides a corresponding device, electronic equipment and a computer readable storage medium.

Description

Technical Field

[0001] The present application relates to the technical field of computer networks, and in particular to a method for file transmission. In addition, the present application also relates to a related device, an electronic device and a computer-readable storage medium. Background Art

[0002] With the rapid development of enterprise business, file transfer technology has also undergone continuous innovation and development to meet the growing demand for data transmission. In the context of the expansion of enterprise scale and the complexity of business, file transfer has become more important, but also faces more challenges and demands.

[0003] In the initial stage, simple file transfer methods such as FTP or HTTP may be used to meet basic data transmission needs. However, as business expands and data volume increases, this traditional method gradually shows some limitations, such as slow transmission speed, low reliability, and difficulty in meeting real-time transmission needs.

[0004] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the invention

[0005] Therefore, an embodiment of the present invention provides a method for file transmission, which is intended to solve the problems of slow transmission speed and low reliability. It can perform concurrent transmission of split files after checking the file transmission task, and perform file integrity check and merge the split files after the transmission is completed. It can quickly transmit files and ensure the integrity of the files. Specifically, an embodiment of the present invention provides a method for file transmission, which is applied to a sending node, and the method includes the following steps:

[0006] Subscribe to the file transfer task information configured by the configuration center, wherein the file transfer task information includes target file information, file unpacking size information, sending node information and receiving node information;

[0007] When it is determined that the sending node information matches the own node information, obtaining the target file according to the target file information;

[0008] Splitting the target file according to the file unpacking size information to generate multiple sub-files;

[0009] Constructing file metadata information and sub-file metadata information corresponding to the target file and the plurality of sub-files, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value;

[0010] According to the receiving node information, the plurality of sub-files, the file metadata information and the sub-file metadata information are sent to the receiving node in an RPC manner, so that the receiving node verifies the integrity according to the file metadata and the sub-file metadata and then feeds back the verification result;

[0011] After determining that the verification result sent by the receiving node is that the file is complete, sending a file merging notification to the receiving node, so that the receiving node merges the multiple sub-files.

[0012] In some embodiments of the present invention, before the file transfer task information configured by the subscription configuration center is subscribed, the following steps are included:

[0013] Send the sending node information, the file directory and the file name information of the sending node to the configuration center for registration, so that the configuration center configures the file transmission task information.

[0014] In some embodiments of the present invention, the target file information includes a directory where the target file is located and a target file name; and obtaining the target file according to the target file information includes:

[0015] A file having the same name as the target file is obtained from the directory where the target file is located as the target file.

[0016] In some embodiments of the present invention, obtaining the target file according to the target file information further includes:

[0017] When the sending node does not have the directory where the target file is located or a file with the same name as the target file, the configuration center is fed back that the task has failed to be sent, so that the configuration center regenerates the file transfer task information.

[0018] In some embodiments of the present invention, the step of constructing the target file and the file metadata information and the sub-file metadata information corresponding to the plurality of sub-files includes:

[0019] Generate multiple sub-file hash values ​​according to the contents of the multiple sub-files;

[0020] Setting the splitting sequence numbers of the plurality of sub-files to the plurality of sub-file names;

[0021] The sub-file Hash cumulative value is generated according to the multiple sub-file Hash values.

[0022] In some embodiments of the present invention, the receiving node information includes a receiving node name, a receiving node address, and a receiving node port; and the sending of the plurality of sub-files, the file metadata information, and the sub-file metadata information to the receiving node in parallel includes:

[0023] According to the receiving node name, the receiving node address and the receiving node port, the multiple sub-files, the file metadata information and the sub-file metadata information are sent to the receiving node concurrently, and the file sending progress is recorded.

[0024] In some embodiments of the present invention, before sending the file to the receiving node, the method further includes:

[0025] A sending notification is sent to the receiving node so that the receiving node prepares for receiving.

[0026] In some embodiments of the present invention, it further includes:

[0027] After determining that the verification result sent by the receiving node is that the file is incomplete, the multiple sub-files, file metadata information and sub-file metadata information are re-sent to the receiving node in parallel, so that the receiving node verifies the integrity according to the file metadata and sub-file metadata and then feeds back the verification result.

[0028] In another aspect of the present invention, an embodiment of the present invention further provides a method for file transmission, which is applied to a receiving node, and the method comprises the following steps:

[0029] Receiving in parallel through the RPC mode a plurality of sub-files, file metadata information and sub-file metadata information sent by a sending node, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value;

[0030] Verify integrity information of the plurality of sub-files according to the file metadata and the sub-file metadata, and send the integrity information to the sending node, so that the sending node sends a file merging notification to the receiving node after determining that the plurality of sub-files are complete;

[0031] A file merging notification is received, and the multiple sub-files are merged.

[0032] In some embodiments of the present invention, before receiving the file sent by the sending node, the method further includes:

[0033] Receive sending notification and prepare for receiving.

[0034] In some embodiments of the present invention, verifying the integrity information of the plurality of sub-files according to the file metadata and the sub-file metadata includes:

[0035] Calculating multiple received sub-file hash values ​​according to the contents of the received multiple sub-files;

[0036] When it is determined that the received sub-file hash values ​​corresponding to the received sub-files are consistent with the sub-file hash values ​​in the sub-file metadata, generating a received sub-file hash cumulative value according to the multiple received sub-file hash values;

[0037] When it is determined that the received sub-file hash cumulative value is consistent with the sub-file hash cumulative value in the file metadata, the integrity information is set to file integrity.

[0038] In some embodiments of the present invention, the sub-file name is a split sequence number; and merging the multiple sub-files includes:

[0039] The sub-files are sorted according to their sub-file names, and the sorted sub-files are merged.

[0040] In some embodiments of the present invention, before receiving the file sent by the sending node, the method further includes:

[0041] The receiving node information and the file directory of the receiving node are sent to the configuration center for registration, so that the configuration center configures the file transfer task information, wherein the file transfer task information includes the target file information, the file unpacking size information, the sending node information and the receiving node information.

[0042] In some embodiments of the present invention, the receiving node information includes a file receiving directory, and before receiving the file sent by the sending node, it also includes:

[0043] Subscribe to the file transfer task configured by the configuration center, and when it is determined that the file receiving directory does not belong to the receiving node, feedback the file transfer task error status to the configuration center so that the configuration center reconfigures the file transfer task.

[0044] In some embodiments of the present invention, after the file merging is completed, the file transfer status is reported to the configuration center.

[0045] In another aspect, the present invention further provides a device for file transmission, including a subscription module, a target file acquisition module, a splitting module, a metadata building module, a parallel sending module and a notification merging module, wherein:

[0046] The subscription module is configured to subscribe to the file transfer task information configured by the configuration center, wherein the file transfer task information includes target file information, file unpacking size information, sending node information and receiving node information;

[0047] The target file acquisition module is configured to acquire the target file according to the target file information when determining that the sending node information matches the own node information;

[0048] The splitting module is configured to split the target file according to the file unpacking size information to generate a plurality of sub-files;

[0049] The metadata construction module is configured to construct file metadata information and sub-file metadata information corresponding to the target file and the plurality of sub-files, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value;

[0050] The parallel sending module is configured to send the plurality of sub-files, file metadata information and sub-file metadata information to the receiving node in parallel according to the receiving node information, so that the receiving node verifies the integrity according to the file metadata and the sub-file metadata and then feeds back the verification result;

[0051] The notification merging module is configured to send a file merging notification to the receiving node after determining that the verification result sent by the receiving node is that the file is complete, so that the receiving node merges the multiple sub-files.

[0052] In another aspect of the present invention, an embodiment of the present invention further provides a device for file transmission, including a receiving module, a checking module and a merging module, wherein:

[0053] The receiving module is configured to receive multiple sub-files, file metadata information and sub-file metadata information sent in parallel by the sending node, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value;

[0054] The verification module is configured to verify the integrity information of the plurality of sub-files according to the file metadata and the sub-file metadata, and send the integrity information to the sending node, so that the sending node sends a file merging notification to the receiving node after determining that the plurality of sub-files are complete;

[0055] The merging module is configured to receive a file merging notification and merge the multiple sub-files.

[0056] In an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, the method for file transmission of any embodiment of the present invention is implemented.

[0057] In an embodiment of the present invention, an electronic device is provided, including: a processor and a memory storing a computer program, wherein the processor is configured to execute any file transmission method of the embodiments of the present invention when running the computer program.

[0058] The embodiment of the present invention proposes a method and device for file transmission, wherein a sending node subscribes to file transmission task information of a configuration center, verifies that the node information in the file transmission task matches itself, obtains the corresponding target file from the local, splits the target file, generates multiple sub-files, generates corresponding sub-file metadata information and file metadata information, and then sends the sub-file metadata information, file metadata information and sub-file to a receiving node in parallel. After receiving the sub-file, the receiving node verifies the file integrity of the received sub-file according to the received sub-file metadata information and file metadata information. After verifying that the file is complete, the verification result is fed back to the sending node, and the sending node sends a merge notification to the receiving node. The receiving node merges the files and feeds back the file reception status to the configuration center. By subscribing to the file transmission task and performing parallel transmission, the file transmission can be performed quickly with high transmission efficiency, and integrity verification is performed to ensure file consistency.

[0059] Other optional features and technical effects of the embodiments of the present invention are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The elements shown are not limited to the proportions shown in the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements, wherein:

[0061] Figure 1 Schematic diagram showing the interaction of components of a system for file transmission according to an embodiment of the present invention

[0062] Figure 2 A schematic diagram showing the interaction of components of a file interaction system according to an embodiment of the present invention is shown;

[0063] Figure 3 An exemplary flow chart of a method for file transmission according to an embodiment of the present invention is shown;

[0064] Figure 4 An exemplary flow chart of constructing metadata according to a method for file transmission according to an embodiment of the present invention is shown;

[0065] Figure 5 An exemplary flow chart showing a method for file transmission according to other embodiments of the present invention is shown;

[0066] Figure 6 An exemplary flow chart showing a method for file transmission according to other embodiments of the present invention is shown;

[0067] Figure 7 An exemplary flow chart of a method for file transmission according to some embodiments of the present invention is shown;

[0068] Figure 8An exemplary flow chart of verifying file integrity of a file transmission method according to other embodiments of the present invention is shown;

[0069] Fig. 9 An exemplary flow chart showing a method for file transmission according to other embodiments of the present invention is shown;

[0070] Fig.10 An exemplary structural diagram of a device for file transmission according to an embodiment of the present invention is shown;

[0071] Fig.11 An exemplary structural diagram of a file transmission device according to some other embodiments of the present invention is shown;

[0072] Fig.12 An exemplary structural diagram of an electronic device capable of implementing the method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and drawings. Here, the exemplary implementation methods of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0074] In the embodiment of the present invention, "FTP" is a set of standard protocols for transferring files on the network. It works at the seventh layer of the OSI model and the fourth layer of the TCP model, i.e., the application layer. Before the client establishes a connection with the server, it must go through a "three-way handshake" process. "HTTP" refers to the Hypertext Transfer Protocol, which is a simple request-response protocol. It usually runs on top of TCP. Before the client establishes a connection with the server, it must go through a "three-way handshake" process. "RPC" is a remote procedure call protocol, i.e., a program on one machine can call a program on another machine, just like calling a local method. It spans the transport layer and the application layer in the OSI network model. Compared with HTTP, it reduces header information and does not require a three-way handshake like HTTP, which reduces network overhead. Moreover, since it is a private protocol, it has higher security. "MD5" refers to a hash algorithm, which is a widely used cryptographic hash function.

[0075] The invention discovered that there are many problems with existing FTP or HTTP file transfer protocols, such as high maintenance costs, cumbersome FTP file directory and permission management; low transmission efficiency, inability to ensure file transfer integrity; limitations in large file transmission; lack of data protection during transmission; and excessive server resource usage. In order to solve the problem of low efficiency of existing file transfer methods, the embodiments of the present invention provide an efficient and reliable two-way file transfer method and system based on RPC parallel transmission technology. The system in the embodiments of the present invention integrates RPC technology to achieve automation, efficiency, and consistency of remote procedure calls, making file transfer easier and faster. At the same time, the stability and reliability of the transmission process are ensured.

[0076] like Figure 1 As shown, an embodiment of the present invention provides a system 100 for file transmission, including a configuration center 110, a sending node 110 and a receiving node 120. In the embodiment of the present invention, the sending node 110 and the receiving node 120 can exchange roles for different file transmission tasks, that is, the sending node can be transformed into a receiving node, and the receiving node can be transformed into a sending node.

[0077] After the configuration center 110, the sending node 120 and the receiving node 130 are deployed and started, the sending node 120 sends the sending node information, the file directory and the file name information of the sending node to the configuration center for registration, and the receiving node 130 sends the receiving node information and the file directory of the receiving node to the configuration center for registration, and the configuration center 120 configures the file transmission task according to the registered information.

[0078] The sending node 120 and the receiving node 130 subscribe to the file transfer task information configured by the configuration center 110, wherein the file transfer task information includes target file information, file unpacking size information, sending node information and receiving node information. The user can configure the file transfer task from the configuration center 110 through the visual operation interface. For example, the user selects to send the File1 file from the data / store directory in the A node to the data / receive directory in the B node, and receive it through the 8801 port of the B node. The file unpacking size is 20M, then the file transfer task information is {data / store / File1, 20M, A node, B node: 8801, data / receive}.

[0079] When the sending node 120 determines that the sending node information matches its own node information, the sending node 120 obtains the target file according to the target file information. In some embodiments of the present invention, the target file information includes the directory where the target file is located and the target file name, and the sending node 120 obtains a file with the same name as the target file from the directory where the target file is located as the target file.

[0080] For example, for node A, if the sending node of the above file transfer task verifies that the information matches its own, then the target file File1 is obtained according to the target file information data / store / File1.

[0081] In some embodiments of the present invention, a monitoring alarm mechanism is also provided. When the sending node 120 does not have the directory where the target file is located or a file with the same name as the target file, the configuration center 110 is fed back that the task sending failed, so that the configuration center 110 regenerates the file transfer task information.

[0082] For example, for node A, what is stored in the data / store / directory is File*, which is inconsistent with the target file name in the file transfer task. Then node A reports to the configuration center that the task sending failed, and the configuration center reconfigures and generates file transfer task information.

[0083] In some embodiments of the present invention, the receiving node information includes a file receiving directory; before receiving the file sent by the sending node, the receiving node 130 subscribes to the file transfer task configured by the configuration center 110, and when it is determined that the file receiving directory does not belong to the receiving node, it feeds back the file transfer task error status to the configuration center 110, so that the configuration center 110 reconfigures the file transfer task.

[0084] For example, the file directory of node B is data / rec, and the receiving file directory in the file transfer task is data / receive. If the directories do not match, the receiving node will feedback the error status of the file transfer task to the configuration center 110.

[0085] In some embodiments of the present invention, after the sending node 120 and the receiving node 130 subscribe to obtain the file transfer task of the configuration center, they will also obtain the address of the other node. For example, when the sending node A determines that the file transfer task matches its own node, it obtains the address of the receiving node B from the file transfer task, and can send a handshake packet to the node B and receive a response to verify the real existence of the node B. Similarly, the receiving node B can also send a handshake packet to the node A and receive a response to verify the real existence of the node A.

[0086] After obtaining the target file, the sending node 120 splits the target file according to the file unpacking size information to generate multiple sub-files. For example, if the file size of File1 is 55M, then according to the file unpacking size information of 20M, it is split into three sub-files, namely File1-1 (20M), File1-2 (20M) and File1-3 (15M). The three sub-files can be stored in a specific directory, for example, in data / temp / File1 / .

[0087] After splitting, the sending node 120 constructs the file metadata information and sub-file metadata information corresponding to the target file and the multiple sub-files, the sub-file metadata information includes the sub-file hash value and the sub-file name, and the file metadata information includes the file name, file size, and the sub-file hash cumulative value.

[0088] In some embodiments of the present invention, the sending node 120 constructs metadata according to the following steps:

[0089] Generate multiple sub-file hash values ​​according to the contents of the multiple sub-files;

[0090] Setting the splitting sequence numbers of the plurality of sub-files to the plurality of sub-file names;

[0091] The sub-file Hash cumulative value is generated according to the multiple sub-file Hash values.

[0092] For example, sub-file hash values ​​1fa4, 1b4a, 2cd3 are generated based on the contents of sub-files File1-1, File1-2, and File1-3. The sub-file name of sub-file File1-1 is "1", the sub-file name of sub-file File1-2 is "2", and the sub-file name of sub-file File1-3 is "3". Then, the sub-file hash cumulative value is calculated based on the sub-file hash values ​​1fa4, 1b4a, 2cd3 to obtain 67c1.

[0093] In some embodiments of the present invention, the sending node uses the MD5 algorithm to generate a hash value.

[0094] The sending node 120 sends the multiple sub-files, file metadata information and sub-file metadata information to the receiving node 130 in parallel through the RPC method according to the receiving node information.

[0095] In some embodiments of the present invention, the receiving node information includes a receiving node name, a receiving node address and a receiving node port; the sending node 120 sends the multiple sub-files, file metadata information and sub-file metadata information to the receiving node 130 concurrently according to the receiving node name, the receiving node address and the receiving node port, and records the file sending progress.

[0096] For example, the receiving node name is node B, the receiving node address is 172.168.12.21, the receiving node port is 8801, and the sending node sends files in parallel to 172.168.12.21:8801.

[0097] The system in the embodiment of the present invention performs file transmission by RPC parallel transmission, does not require a three-way handshake, reduces network overhead, and because it is a private protocol, has higher security and improved file transmission efficiency.

[0098] In some embodiments of the present invention, the sending node 120 feeds back the sending progress information to the configuration center 110, and the configuration center 110 displays the file sending progress through a visual interface.

[0099] In some embodiments of the present invention, before sending a file, the sending node 120 may send a sending notification to the receiving node 130 so that the receiving node 130 can prepare for receiving. For example, the receiving node arranges resources in advance for receiving.

[0100] The receiving node 130 receives multiple sub-files, file metadata information and sub-file metadata information sent by the sending node 120 in parallel through RPC, verifies the integrity information of the multiple sub-files according to the file metadata and sub-file metadata, and sends the integrity information to the sending node 120.

[0101] In some embodiments of the present invention, the receiving node 130 verifies the file integrity according to the following steps:

[0102] Calculating multiple received sub-file hash values ​​according to the contents of the received multiple sub-files;

[0103] When it is determined that the received sub-file hash values ​​corresponding to the received sub-files are consistent with the sub-file hash values ​​in the sub-file metadata, generating a received sub-file hash cumulative value according to the multiple received sub-file hash values;

[0104] When it is determined that the received sub-file hash cumulative value is consistent with the sub-file hash cumulative value in the file metadata, the integrity information is set to file integrity.

[0105] For example, the receiving node receives subfiles "1", "2", and "3", calculates and obtains the received subfile hash values ​​as 1fa4, 1b4a, and 2cd3, and compares them with the received subfile metadata. If the comparison result is consistent, and the cumulative value is also consistent, the integrity information is set to file integrity. The receiving node 130 feeds back the integrity information to the sending node 120.

[0106] After the sending node 120 determines that the verification result sent by the receiving node 130 is that the file is complete, it sends a file merge notification to the receiving node 130. The system in the embodiment of the present invention ensures the consistency of file transmission by verifying the integrity of the file and avoids the failure of the file transmission task.

[0107] The receiving node 130 receives the file merging notification and merges the multiple sub-files.

[0108] In some embodiments of the present invention, the sub-file names are split sequence numbers, and the receiving node 130 sorts the sub-files according to the sub-file names and merges the sorted sub-files.

[0109] For example, if the received sub-files are "3", "1", and "2", they are sorted according to the names, and the sub-files are "1", "2", and "3", and then merged.

[0110] In some embodiments of the present invention, after the sending node 120 determines that the verification result sent by the receiving node 130 is that the file is incomplete, the sending node 120 re-sends the multiple sub-files, file metadata information and sub-file metadata information to the receiving node 120 in parallel, so that the receiving node 130 verifies the integrity according to the file metadata and sub-file metadata and then feeds back the verification result, thereby ensuring the success rate of the file transfer task execution.

[0111] In some embodiments of the present invention, in order to allow the user to perceive the file transfer status, the receiving node 130 reports the file transfer status to the configuration center 110 after the file merging is completed.

[0112] In the system of the embodiment of the present invention, the sending node subscribes to the file transfer task information of the configuration center, verifies that the node information in the file transfer task matches itself, obtains the corresponding target file from the local, splits the target file, generates multiple sub-files, generates corresponding sub-file metadata information and file metadata information, and then sends the sub-file metadata information, file metadata information and sub-file to the receiving node in parallel. After receiving, the receiving node verifies the file integrity of the received sub-file according to the received sub-file metadata information and file metadata information. After verifying that the file is complete, the verification result is fed back to the sending node, and the sending node sends a merge notification to the receiving node. The receiving node merges the file and feeds back the file reception status to the configuration center. By subscribing to the file transfer task and sending in parallel, the file can be quickly transferred, the transmission efficiency is high, and the integrity check is performed to ensure the consistency of the file. It can also provide a visual management interface, the operation is simpler and more convenient, the large file transmission efficiency is high, the resource occupation is small, the file transmission is reliable, the consistency of the file transmission is guaranteed, and the file transmission process is safe. Since there is no need to parse the head header information like HTTP, it saves labor costs and does not need to write too many scripts to transfer files.

[0113] like Figure 2 As shown, the present invention provides a file exchange system for nodes implemented by RPC, including a configuration center and a file transfer node. Each file transfer node acts as both a sender and a receiver. The transfer node implements efficient bidirectional P2P file transfer based on RPC by subscribing to the file transfer task of the configuration center. Therefore, the system is divided into three parts, namely, a configuration center, a sender, and a receiver.

[0114] The specific functions of the configuration center are as follows:

[0115] 1. Configure the file transfer task information: file split package size, transfer node name, transfer node IP and port;

[0116] 2. Publish file transfer task message;

[0117] 3. Receive the registration of the file transfer node

[0118] 4. Receive the task report of the transmission node

[0119] The specific functions of the sender are as follows:

[0120] 1. Register the sender transmission node information to the configuration center;

[0121] 2. The transmission node at the sending end subscribes to the configuration center to enable transmission nodes to discover each other;

[0122] 3. Transfer files, split files into specified directories according to packet size, calculate the MD5 of a single file and the cumulative MD5 of all files;

[0123] 4. The sending node constructs the file metadata information and notifies the receiving end to prepare to receive the file;

[0124] 5. Build metadata information of the file content, including file content, MD5, serial number and other information, use RPC to concurrently send the file to the receiving end, and record the progress of file sending;

[0125] 6. Notify the receiving end to merge the files

[0126] The specific functions of the receiving end are as follows:

[0127] 1. Register the receiving end transmission node information to the configuration center;

[0128] 2. The receiving end receives the file from the sending end, writes the file to a local temporary directory, parses the file metadata information, and uses the MD5 information in the metadata to verify the integrity of the file, and returns an RPC response message indicating whether the reception is successful;

[0129] Merge the received files and report that the file transfer is complete.

[0130] The system is deployed as follows:

[0131] 1. Build configuration center service.

[0132] 2. Deploy the sender and start it.

[0133] 3. Deploy the receiving end and start it.

[0134] The system is implemented as follows:

[0135] The file exchange system between nodes is implemented based on RPC. It completes efficient file transfer between nodes through the configuration center, sender, and receiver.

[0136] 1. Configure the file transfer task information in the configuration center: file split package size, transfer node name, transfer node IP and port;

[0137] 2. Enable the transmission node and register the transmission node information to the configuration center;

[0138] 3. The transmission nodes subscribe to the configuration center to enable mutual discovery of transmission nodes;

[0139] 4. Transfer files, split files into specified directories according to packet size, calculate the MD5 of a single file and the cumulative MD5 of all files;

[0140] 5. The sending node constructs file metadata information, including file name, file size, file cumulative MD5 and other information, and notifies the receiving node to prepare to receive the file;

[0141] 6. Construct file metadata information, including file content, MD5, serial number and other information, and use RPC to send the file to the receiving end, while recording the progress of file sending;

[0142] 7. The receiving end writes the file to a local temporary directory, parses the file metadata information, and uses the MD5 information in the original data to verify the file integrity, and returns an RPC response message indicating whether the reception is successful;

[0143] 8. After receiving the file response information, the sender analyzes whether the file is received successfully. If it is received successfully, it marks the file as sent successfully and records it in the memory. If it fails to receive, it triggers retransmission.

[0144] 9. After sending, notify the receiving end to merge the files according to the sequence number;

[0145] 10. After completing the file merging, the client reports the task completion status.

[0146] After the sending end and the receiving end of the system in the embodiment of the present invention are deployed, the information of the file task can be configured on the page, thereby realizing efficient file transfer. Based on RPC implementation, the concurrent file transmission efficiency is high; the system architecture is clear, with fewer dependent components and higher stability; the large file transmission efficiency is high and the resource occupation is small; the deployment process is simple and easier to use; the file transmission is reliable and the consistency of the file transmission is guaranteed.

[0147] The solution of the present invention is described below from the perspectives of a sending node and a receiving node respectively.

[0148] like Figure 3 As shown, an embodiment of the present invention provides a method for file transmission, which is applied to a sending node, and the method includes the following steps:

[0149] S210: Subscribe to the file transfer task information configured by the configuration center, wherein the file transfer task information includes target file information, file unpacking size information, sending node information and receiving node information. The user can configure the file transfer task from the configuration center through the visual operation interface. For example, the user selects to send the File1 file from the data / store directory in the A node to the data / receive directory in the B node, and receive it through the 8801 port of the B node. The file unpacking size is 20M, then the file transfer task information is {data / store / File1, 20M, A node, B node: 8801, data / receive}.

[0150] S220: When it is determined that the sending node information matches the own node information, the target file is acquired according to the target file information.

[0151] In some embodiments of the present invention, the target file information includes a directory where the target file is located and a target file name, and the sending node obtains a file with the same name as the target file from the directory where the target file is located as the target file.

[0152] For example, for node A, if the sending node of the above file transfer task verifies that the information matches its own, then the target file File1 is obtained according to the target file information data / store / File1.

[0153] In some embodiments of the present invention, a monitoring alarm mechanism is also provided. When the sending node does not have the directory where the target file is located or a file with the same name as the target file, the configuration center is fed back that the task sending failed, so that the configuration center regenerates the file transfer task information.

[0154] For example, for node A, what is stored in the data / store / directory is File*, which is inconsistent with the target file name in the file transfer task. Then node A reports to the configuration center that the task sending failed, and the configuration center reconfigures and generates file transfer task information.

[0155] In some embodiments of the present invention, after the sending node subscribes to obtain the file transfer task of the configuration center, it will also obtain the address of the other node. For example, when the sending node A determines that the file transfer task matches its own node, it obtains the address of the receiving node B from the file transfer task, and can send a handshake packet to node B and then receive a response to verify that node B actually exists.

[0156] S230: Split the target file according to the file unpacking size information to generate multiple sub-files.

[0157] For example, if the size of File1 is 55M, then according to the file unpacking size information, it is 20M, and it is split into three sub-files, namely File1-1 (20M), File1-2 (20M) and File1-3 (15M). The three sub-files can be stored in a specific directory, such as data / temp / File1 / .

[0158] S240: Construct file metadata information and sub-file metadata information corresponding to the target file and the multiple sub-files, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value.

[0159] In some embodiments of the invention, Figure 4 As shown, the step of constructing the target file and the file metadata information and sub-file metadata information corresponding to the plurality of sub-files includes:

[0160] S241: Generate multiple sub-file hash values ​​according to the contents of the multiple sub-files;

[0161] S242: setting the splitting sequence numbers of the plurality of sub-files to the names of the plurality of sub-files;

[0162] S243: Generate the sub-file hash cumulative value according to the multiple sub-file hash values.

[0163] For example, sub-file hash values ​​1fa4, 1b4a, 2cd3 are generated based on the contents of sub-files File1-1, File1-2, and File1-3. The sub-file name of sub-file File1-1 is "1", the sub-file name of sub-file File1-2 is "2", and the sub-file name of sub-file File1-3 is "3". Then, the sub-file hash cumulative value is calculated based on the sub-file hash values ​​1fa4, 1b4a, 2cd3 to obtain 67c1.

[0164] In some embodiments of the present invention, the sending node uses the MD5 algorithm to generate a hash value.

[0165] S250: According to the receiving node information, the multiple sub-files, file metadata information and sub-file metadata information are sent to the receiving node in parallel via RPC, so that the receiving node verifies integrity according to the file metadata and sub-file metadata and then feeds back the verification result.

[0166] In some embodiments of the present invention, the receiving node information includes a receiving node name, a receiving node address and a receiving node port; the sending node concurrently sends the multiple sub-files, file metadata information and sub-file metadata information to the receiving node according to the receiving node name, the receiving node address and the receiving node port, and records the file sending progress.

[0167] For example, the receiving node name is node B, the receiving node address is 172.168.12.21, the receiving node port is 8801, and the sending node sends files in parallel to 172.168.12.21:8801.

[0168] The method in the embodiment of the present invention performs file transmission by RPC parallel transmission, does not require a three-way handshake, reduces network overhead, and because it is a private protocol, has higher security and improves file transmission efficiency.

[0169] In some embodiments of the present invention, the sending node feeds back the sending progress information to the configuration center, and the configuration center displays the file sending progress through a visual interface.

[0170] In some embodiments of the present invention, before sending a file, the sending node may send a sending notification to the receiving node so that the receiving node can prepare for receiving. For example, the receiving node arranges resources in advance for receiving.

[0171] S260: After determining that the verification result sent by the receiving node is that the file is complete, sending a file merging notification to the receiving node, so that the receiving node merges the multiple sub-files.

[0172] In order to ensure the success rate of file transmission, in some embodiments of the present invention, such as Figure 5 As shown, it also includes:

[0173] S270: After determining that the verification result sent by the receiving node is that the file is incomplete, the multiple sub-files, file metadata information and sub-file metadata information are sent to the receiving node in parallel, so that the receiving node verifies the integrity according to the file metadata and sub-file metadata and then feeds back the verification result, thereby ensuring the success rate of the file transfer task.

[0174] In order to provide the configuration center with sending node information, such as Figure 6 As shown, before the file transfer task information configured in the subscription configuration center is subscribed, it includes:

[0175] S201: Send the sending node information, the file directory and file name information of the sending node to the configuration center for registration, so that the configuration center configures the file transfer task information. The configuration center configures the file transfer task according to the information of the sending node. In some embodiments, the file directory and file name information of the sending interface can also be displayed through a visual interface.

[0176] In the method of the embodiment of the present invention, the sending node subscribes to the file transfer task information of the configuration center, verifies that the node information in the file transfer task matches itself, obtains the corresponding target file from the local, splits the target file, generates multiple sub-files, generates corresponding sub-file metadata information and file metadata information, and then sends the sub-file metadata information, file metadata information and sub-file to the receiving node in parallel. After receiving, the receiving node verifies the file integrity of the received sub-file according to the received sub-file metadata information and file metadata information. After verifying that the file is complete, the verification result is fed back to the sending node, and the sending node sends a merge notification to the receiving node. The receiving node merges the files and feeds back the file reception status to the configuration center. By subscribing to the file transfer task and sending in parallel, the file can be quickly transferred, the transmission efficiency is high, and the integrity check is performed to ensure the consistency of the file. It can also provide a visual management interface, the operation is simpler and more convenient, the large file transmission efficiency is high, the resource occupation is small, the file transmission is reliable, the consistency of the file transmission is guaranteed, and the file transmission process is safe. Since there is no need to parse the head header information like HTTP, it saves labor costs and does not need to write too many scripts to transfer files.

[0177] like Figure 7 As shown, an embodiment of the present invention also provides a method for file transmission, which is applied to a receiving node and includes the following steps:

[0178] S310: multiple sub-files, file metadata information and sub-file metadata information sent by the sending node are received in parallel via RPC, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size and a sub-file hash cumulative value. In the embodiment of the present invention, the file is transmitted via RPC, with low network overhead, high efficiency, small code volume and high system construction efficiency.

[0179] S320: Verify integrity information of the multiple sub-files according to the file metadata and the sub-file metadata, and send the integrity information to the sending node, so that the sending node sends a file merging notification to the receiving node after determining that the multiple sub-files are complete.

[0180] In some embodiments of the present invention, Figure 8 As shown, the checking the integrity information of the plurality of sub-files according to the file metadata and the sub-file metadata includes:

[0181] S321: Calculating multiple received sub-file hash values ​​according to the contents of the received multiple sub-files;

[0182] S322: When it is determined that the received sub-file hash values ​​corresponding to the received sub-files are consistent with the sub-file hash values ​​in the sub-file metadata, generating a received sub-file hash cumulative value according to the multiple received sub-file hash values;

[0183] S323: When it is determined that the received sub-file hash cumulative value is consistent with the sub-file hash cumulative value in the file metadata, the integrity information is set to file integrity.

[0184] For example, the receiving node receives sub-files "1", "2", and "3", calculates the hash values ​​of the received sub-files to be 1fa4, 1b4a, and 2cd3, and compares them with the received sub-file metadata. If the comparison result is consistent and the cumulative value is also consistent, the integrity information is set to file integrity. The receiving node feeds back the integrity information to the sending node. By verifying the integrity of the file, the consistency of file transmission can be guaranteed.

[0185] In some embodiments of the present invention, the file metadata information also includes file size information, and the receiving node can also verify whether the sub-file size cumulative value is consistent with the file size information, and generate file integrity information if they are consistent.

[0186] The embodiment of the present invention verifies the hash value of each sub-file and detects whether each sub-file has been tampered with, thereby improving the security of file transmission.

[0187] S330: Receive a file merging notification, and merge the multiple sub-files.

[0188] In some embodiments of the present invention, the sub-file name is a split sequence number; and merging the multiple sub-files includes:

[0189] The sub-files are sorted according to their sub-file names, and the sorted sub-files are merged.

[0190] For example, if the received sub-files are "3", "1", and "2", they are sorted according to the names, and the sub-files are "1", "2", and "3", and then merged.

[0191] In some embodiments of the present invention, in order to allow the user to perceive the file transfer status, the receiving node reports the file transfer status to the configuration center after the file merging is completed. The configuration center may use a visual interface for display.

[0192] In some embodiments of the present invention, before receiving the file sent by the sending node, the method further includes:

[0193] Receive the sending notification and prepare for receiving. The receiving node can prepare relevant resources in advance and can receive quickly.

[0194] In some embodiments of the present invention, Fig. 9 As shown, before receiving the file sent by the sending node, it also includes:

[0195] S301: Send receiving node information and the file directory of the receiving node to the configuration center for registration, so that the configuration center configures file transfer task information, wherein the file transfer task information includes target file information, file unpacking size information, sending node information and receiving node information.

[0196] In some embodiments of the present invention, the receiving node information includes a file receiving directory; before receiving the file sent by the sending node, it also includes:

[0197] S302: Subscribe to the file transfer task configured by the configuration center, and when it is determined that the file receiving directory does not belong to the receiving node, feedback the file transfer task error status to the configuration center so that the configuration center reconfigures the file transfer task.

[0198] The receiving node can know in advance whether the received information in the file transfer task is wrong, avoiding errors during the transmission process and wasting the time of the sending node to prepare sub-files and metadata. Knowing in advance whether the received information is wrong can quickly trigger the regeneration of the file transfer task.

[0199] In the method of the embodiment of the present invention, the receiving node receives sub-file metadata information, file metadata information and sub-file in parallel, verifies the file integrity of the received sub-file based on the received sub-file metadata information and file metadata information, and after verifying that the file is complete, feeds back the verification result to the sending node, and the sending node sends a merge notification to the receiving node, the receiving node merges the files, and feeds back the file reception status to the configuration center. By subscribing to file transfer tasks and performing parallel transmission, files can be transferred quickly with high transmission efficiency, and integrity verification is performed to ensure file consistency. It can also provide a visual management interface, which is simpler and more convenient to operate, has high efficiency in large file transmission, occupies less resources, is reliable in file transmission, ensures consistency in file transmission, and ensures a safe file transmission process. Since there is no need to parse the head information like HTTP, labor costs are saved, and there is no need to write too many scripts to transfer files.

[0200] like Fig.10 As shown, the embodiment of the present invention also provides a file transmission device 400, including a subscription module 410, a target file acquisition module 420, a split module 430, a metadata construction module 440, a parallel sending module 450 and a notification merging module 460, wherein:

[0201] The subscription module 410 is configured to subscribe to the file transfer task information configured by the configuration center, wherein the file transfer task information includes target file information, file unpacking size information, sending node information and receiving node information;

[0202] The target file acquisition module 420 is configured to acquire the target file according to the target file information when it is determined that the sending node information matches the own node information;

[0203] The splitting module 430 is configured to split the target file according to the file unpacking size information to generate multiple sub-files;

[0204] The metadata construction module 440 is configured to construct file metadata information and sub-file metadata information corresponding to the target file and the plurality of sub-files, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value;

[0205] The parallel sending module 450 is configured to send the plurality of sub-files, file metadata information and sub-file metadata information to the receiving node in parallel according to the receiving node information, so that the receiving node verifies the integrity according to the file metadata and the sub-file metadata and then feeds back the verification result;

[0206] The notification merging module 460 is configured to send a file merging notification to the receiving node after determining that the verification result sent by the receiving node is that the file is complete, so that the receiving node merges the multiple sub-files.

[0207] In some embodiments of the present invention, a registration module 401 is further included, and the registration module 401 is configured to:

[0208] Send the sending node information, the file directory and the file name information of the sending node to the configuration center for registration, so that the configuration center configures the file transmission task information.

[0209] In some embodiments of the present invention, the target file acquisition module 420 is further configured to:

[0210] A file having the same name as the target file is obtained from the directory where the target file is located as the target file.

[0211] In some embodiments of the present invention, the target file acquisition module 420 is further configured to:

[0212] When the sending node does not have the directory where the target file is located or a file with the same name as the target file, the configuration center is fed back that the task has failed to be sent, so that the configuration center regenerates the file transfer task information.

[0213] In some embodiments of the present invention, the metadata construction module 440 is configured to:

[0214] Generate multiple sub-file hash values ​​according to the contents of the multiple sub-files;

[0215] Setting the splitting sequence numbers of the plurality of sub-files to the plurality of sub-file names;

[0216] The sub-file Hash cumulative value is generated according to the multiple sub-file Hash values.

[0217] In some embodiments of the present invention, the receiving node information includes a receiving node name, a receiving node address, and a receiving node port; the parallel sending module 450 is configured as follows:

[0218] According to the receiving node name, the receiving node address and the receiving node port, the multiple sub-files, the file metadata information and the sub-file metadata information are sent to the receiving node concurrently, and the file sending progress is recorded.

[0219] In some embodiments of the present invention, the parallel sending module 450 is further configured to:

[0220] A sending notification is sent to the receiving node so that the receiving node prepares for receiving.

[0221] In some embodiments of the present invention, a resending module 470 is further included, and the resending module is configured to:

[0222] After determining that the verification result sent by the receiving node is that the file is incomplete, the multiple sub-files, file metadata information and sub-file metadata information are re-sent to the receiving node in parallel, so that the receiving node verifies the integrity according to the file metadata and sub-file metadata and then feeds back the verification result.

[0223] like Fig.11 As shown, the embodiment of the present invention also provides a file transmission device 500, including a receiving module 510, a checking module 520 and a merging module 530, wherein:

[0224] The receiving module 510 is configured to receive multiple sub-files, file metadata information and sub-file metadata information sent in parallel by the sending node, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value;

[0225] The verification module 520 is configured to verify the integrity information of the multiple sub-files according to the file metadata and the sub-file metadata, and send the integrity information to the sending node, so that the sending node sends a file merging notification to the receiving node after determining that the multiple sub-files are complete;

[0226] The merging module 530 is configured to receive a file merging notification and merge the multiple sub-files.

[0227] In some embodiments of the present invention, the receiving module 510 is further configured to:

[0228] Before receiving the file sent by the sending node, receive the sending notification and make receiving preparations.

[0229] In some embodiments of the present invention, the verification module 520 is configured to:

[0230] Calculating multiple received sub-file hash values ​​according to the contents of the received multiple sub-files;

[0231] When it is determined that the received sub-file hash values ​​corresponding to the received sub-files are consistent with the sub-file hash values ​​in the sub-file metadata, generating a received sub-file hash cumulative value according to the multiple received sub-file hash values;

[0232] When it is determined that the received sub-file hash cumulative value is consistent with the sub-file hash cumulative value in the file metadata, the integrity information is set to file integrity.

[0233] In some embodiments of the present invention, the sub-file name is a split sequence number; the merging module 530 is configured to:

[0234] The sub-files are sorted according to their sub-file names, and the sorted sub-files are merged.

[0235] In some embodiments of the present invention, a registration module 501 is further included, and the registration module 501 is configured to:

[0236] The receiving node information and the file directory of the receiving node are sent to the configuration center for registration, so that the configuration center configures the file transfer task information, wherein the file transfer task information includes the target file information, the file unpacking size information, the sending node information and the receiving node information.

[0237] In some embodiments of the present invention, the receiving node information includes a file receiving directory; the apparatus 500 further includes a subscription module 502, and the subscription module 502 is configured to:

[0238] Subscribe to the file transfer task configured by the configuration center, and when it is determined that the file receiving directory does not belong to the receiving node, feedback the file transfer task error status to the configuration center so that the configuration center reconfigures the file transfer task.

[0239] In some embodiments of the present invention, the merging module 530 is further configured to: report the file transfer status to the configuration center after the file merging is completed.

[0240] The file transmission device proposed in the embodiment of the present invention is that the file transmission device at the sending end subscribes to the file transmission task information of the configuration center, verifies that the node information in the file transmission task matches itself, obtains the corresponding target file from the local, splits the target file to generate multiple sub-files, generates corresponding sub-file metadata information and file metadata information, and then sends the sub-file metadata information, file metadata information and sub-file to the file transmission device at the receiving end in parallel. The file transmission device at the receiving end verifies the file integrity of the received sub-file according to the received sub-file metadata information and file metadata information. After verifying that the file is complete, the verification result is fed back to the file transmission device at the sending end. The file transmission device at the sending end sends a merge notification to the file transmission device at the receiving end. The file transmission device at the receiving end merges the files and feeds back the file reception status to the configuration center. By subscribing to the file transmission task and sending in parallel, the file can be quickly transmitted with high transmission efficiency, and integrity verification is performed to ensure file consistency.

[0241] In some embodiments, the device may combine the features of the method for file transfer of any embodiment, and vice versa, which will not be elaborated here.

[0242] In an embodiment of the present invention, an electronic device is provided, including: a processor and a memory storing a computer program, wherein the processor is configured to execute any file transmission method of the embodiments of the present invention when running the computer program.

[0243] Fig.12 A schematic diagram of an electronic device 1200 that can implement a method or implement an embodiment of the present invention is shown, and in some embodiments, more or fewer electronic devices than shown may be included. In some embodiments, a single or multiple electronic devices may be used for implementation. In some embodiments, a cloud or distributed electronic device may be used for implementation.

[0244] like Fig.12 As shown, the electronic device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) 1202 or the programs and / or data loaded from the storage part 1208 to the random access memory (RAM) 1203. CPU 1201 can be a multi-core processor, or it can include multiple processors. In some embodiments, CPU 1201 can include a general main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1203, various programs and data required for the operation of the electronic device 1200 are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0245] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the steps or functions of the file transmission method described in the above embodiments can be implemented.

[0246] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage section 1208 as needed. Fig.12Only some components are schematically shown in the figure, which does not mean that the computer system 1200 only includes Fig.12 Components shown.

[0247] In some embodiments, the electronic device 1000 refers to a mobile terminal or a computer, including a mobile phone, a car terminal, a smart TV, etc. Taking a mobile phone as an example, the electronic device 1200 also includes a display screen with a touch function, an external speaker, a gyroscope, a camera, a 4G / 5G antenna and other device modules.

[0248] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0249] Although not shown, in an embodiment of the present invention, a storage medium is provided, the storage medium storing a computer program, the computer program being configured to execute any file transmission method of the embodiments of the present invention when executed.

[0250] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0251] The methods, programs, systems, devices, etc. of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0252] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.

[0253] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0254] In this article, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" are meant to be applicable to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0255] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for file transmission, characterized in that: Applied to a sending node, the method comprises the following steps: Subscribe to the file transfer task information configured by the configuration center, wherein the file transfer task information includes target file information, file unpacking size information, sending node information and receiving node information; When it is determined that the sending node information matches the own node information, obtaining the target file according to the target file information; Splitting the target file according to the file unpacking size information to generate multiple sub-files; Constructing file metadata information and sub-file metadata information corresponding to the target file and the plurality of sub-files, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value; According to the receiving node information, the plurality of sub-files, the file metadata information and the sub-file metadata information are sent to the receiving node in parallel through the RPC mode, so that the receiving node verifies the integrity according to the file metadata and the sub-file metadata and then feeds back the verification result; After determining that the verification result sent by the receiving node is that the file is complete, sending a file merging notification to the receiving node, so that the receiving node merges the multiple sub-files.

2. The method according to claim 1, characterized in that Before the file transfer task information configured in the subscription configuration center is configured, it includes: Send the sending node information, the file directory and the file name information of the sending node to the configuration center for registration, so that the configuration center configures the file transmission task information.

3. The method according to claim 1, characterized in that The target file information includes the directory where the target file is located and the target file name; and obtaining the target file according to the target file information includes: A file having the same name as the target file is obtained from the directory where the target file is located as the target file.

4. The method according to claim 3, characterized in that: The step of obtaining the target file according to the target file information further includes: When the sending node does not have the directory where the target file is located or a file with the same name as the target file, the configuration center is fed back that the task has failed to be sent, so that the configuration center regenerates the file transfer task information.

5. The method according to claim 1, characterized in that The constructing of the target file and the file metadata information and sub-file metadata information corresponding to the plurality of sub-files includes: Generate multiple sub-file hash values ​​according to the contents of the multiple sub-files; Setting the splitting sequence numbers of the plurality of sub-files to the plurality of sub-file names; The sub-file Hash cumulative value is generated according to the multiple sub-file Hash values.

6. The method according to claim 1, characterized in that: The receiving node information includes a receiving node name, a receiving node address, and a receiving node port; the sending of the plurality of sub-files, file metadata information, and sub-file metadata information to the receiving node in parallel includes: According to the receiving node name, the receiving node address and the receiving node port, the multiple sub-files, the file metadata information and the sub-file metadata information are sent to the receiving node concurrently, and the file sending progress is recorded.

7. The method according to claim 6, characterized in that Before sending the file to the receiving node, it also includes: A sending notification is sent to the receiving node so that the receiving node prepares for receiving.

8. The method according to claim 1, characterized in that Also includes: After determining that the verification result sent by the receiving node is that the file is incomplete, the multiple sub-files, file metadata information and sub-file metadata information are re-sent to the receiving node in parallel, so that the receiving node verifies the integrity according to the file metadata and sub-file metadata and then feeds back the verification result.

9. A method for file transmission, characterized in that: Applied to a receiving node, the method comprises the following steps: Receiving in parallel through the RPC mode a plurality of sub-files, file metadata information and sub-file metadata information sent by a sending node, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value; Verify integrity information of the plurality of sub-files according to the file metadata and the sub-file metadata, and send the integrity information to the sending node, so that the sending node sends a file merging notification to the receiving node after determining that the plurality of sub-files are complete; A file merging notification is received, and the multiple sub-files are merged.

10. The method according to claim 9, characterized in that Before receiving the file sent by the sending node, it also includes: Receive sending notification and prepare for receiving.

11. The method according to claim 9, characterized in that The verifying the integrity information of the plurality of sub-files according to the file metadata and the sub-file metadata includes: Calculating multiple received sub-file hash values ​​according to the contents of the received multiple sub-files; When it is determined that the received sub-file hash values ​​corresponding to the received sub-files are consistent with the sub-file hash values ​​in the sub-file metadata, generating a received sub-file hash cumulative value according to the multiple received sub-file hash values; When it is determined that the received sub-file hash cumulative value is consistent with the sub-file hash cumulative value in the file metadata, the integrity information is set to file integrity.

12. The method according to claim 9, characterized in that The subfile name is a split sequence number; and the merging of the multiple subfiles comprises: The sub-files are sorted according to their sub-file names, and the sorted sub-files are merged.

13. The method according to claim 9, characterized in that Before receiving the file sent by the sending node, it also includes: The receiving node information and the file directory of the receiving node are sent to the configuration center for registration, so that the configuration center configures the file transfer task information, wherein the file transfer task information includes the target file information, the file unpacking size information, the sending node information and the receiving node information.

14. The method according to claim 13, characterized in that The receiving node information includes a file receiving directory; before receiving the file sent by the sending node, it also includes: Subscribe to the file transfer task configured by the configuration center, and when it is determined that the file receiving directory does not belong to the receiving node, feedback the file transfer task error status to the configuration center so that the configuration center reconfigures the file transfer task.

15. The method according to claim 9, characterized in that After the file merge is completed, report the file transfer status to the configuration center.

16. A file transmission device, characterized in that: It includes a subscription module, a target file acquisition module, a splitting module, a metadata building module, a parallel sending module and a notification merging module, among which, The subscription module is configured to subscribe to the file transfer task information configured by the configuration center, wherein the file transfer task information includes target file information, file unpacking size information, sending node information and receiving node information; The target file acquisition module is configured to acquire the target file according to the target file information when determining that the sending node information matches the own node information; The splitting module is configured to split the target file according to the file unpacking size information to generate a plurality of sub-files; The metadata construction module is configured to construct file metadata information and sub-file metadata information corresponding to the target file and the plurality of sub-files, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value; The parallel sending module is configured to send the plurality of sub-files, the file metadata information and the sub-file metadata information to the receiving node in parallel according to the receiving node information, so that the receiving node verifies the integrity according to the file metadata and the sub-file metadata and then feeds back the verification result; The notification merging module is configured to send a file merging notification to the receiving node after determining that the verification result sent by the receiving node is that the file is complete, so that the receiving node merges the multiple sub-files.

17. A file transmission device, characterized in that: It includes a receiving module, a checking module and a merging module, wherein: The receiving module is configured to receive multiple sub-files, file metadata information and sub-file metadata information sent in parallel by the sending node, wherein the sub-file metadata information includes a sub-file hash value and a sub-file name, and the file metadata information includes a file name, a file size, and a sub-file hash cumulative value; The verification module is configured to verify the integrity information of the plurality of sub-files according to the file metadata and the sub-file metadata, and send the integrity information to the sending node, so that the sending node sends a file merging notification to the receiving node after determining that the plurality of sub-files are complete; The merging module is configured to receive a file merging notification and merge the multiple sub-files.

18. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

19. An electronic device, characterized in that: include: A processor and a memory storing a computer program, wherein the processor is configured to execute the method according to any one of claims 1 to 15 when running the computer program.

Images