Method for increasing transmission speed of large number of small files

Abstract

The invention discloses a method for improving the transmission speed of a large number of small files. When a client transmits a large number of small files, the small files are assembled to generate a large file not larger than 2G, then the large file is transmitted, and after a transmission server receives data, the large file is split, and the corresponding file is written into a corresponding path. Therefore, the transmission speed is improved, the number of disk I / O times is reduced, the storage load is reduced, and the efficiency of transmitting the transmission server is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of file transmission, and in particular to a method for improving the transmission speed of a large number of small files. Background Art

[0002] In the use scenario of cloud rendering, a user's scene may contain tens of thousands or even hundreds of thousands of asset files. Before rendering begins, these files need to be transferred to the platform storage through the client. In mainstream applications, the transfer engine uses multi-threading for transmission, and each file will perform I / O operations on the transfer server. As we all know, the I / O of the disk is limited, and the disk will read and write small files very slowly, which is often the bottleneck of disk performance. It often happens that there are tens of thousands of asset files less than 10M in the user's assets. At this time, the efficiency of the transfer engine will be very low, and the transmission speed cannot reach the optimal state. If the user's bandwidth is 100M, the speed of small file transmission may only reach a dozen or a few M, and the efficiency of the transmission server is also very low, because it has to wait for the file to fall to the disk and write files frequently, which will also cause a high storage load, often resulting in the need to queue up to write files during transmission.

[0003] The technical background of existing transmission engines is as follows: Figure 1 shown. Technical background:

[0005] 1. When the transfer client transfers files, it will establish a connection for each file.

[0006] 2. After the transmission server establishes a connection with the client, it starts receiving data, creates a new file on the storage, and starts writing the file.

[0007] 3. After the file data is sent by the transfer client and the file is saved on the transfer server, the file transfer is completed.

[0008] 4. When there are multiple files to be transferred, the transfer client will start multiple threads to perform the process 1-3.

[0009] Technical issues:

[0010] 1. One link for one file. If the file is too small, the transmission efficiency cannot be improved.

[0011] 2. Each file takes up one disk I / O.

[0012] 3. Small files are a test for disk I / O. If there are too many small files, the disk I / O will be occupied in a short time. Then other files need to queue when they want to be written to the disk, affecting the efficiency of the transmission server.

[0013] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0014] The technical problem to be solved by the present invention is to provide a method for improving the transmission speed of a large number of small files, thereby increasing the transmission speed, reducing the number of disk I / 0 operations, reducing the storage load, and improving the efficiency of the transmission server.

[0015] The technical solution of the present invention is as follows: a method for improving the transmission speed of a large number of small files is provided, comprising the following steps.

[0016] S1: Before the transmission client starts the transmission, it first scans the transmission file list to see if there are files smaller than 20M, and defines the files smaller than 20M as small files. If there are, the local path of the file and the mapping path of the transmission server are recorded in a temporary database. In this step, the files to be transmitted are divided into two parts, one part is the small files that need to be assembled into a large file, and the other part is the files that need to be transmitted individually. Preferably, the size of the small file is defined as less than 10M.

[0017] S2: After the transmission client scans the file list, it assembles the small files into several large files and generates a corresponding number of temporary files locally on the user; the size of the assembled large files does not exceed 2GB. Preferably, the size of the assembled large files does not exceed 1GB.

[0018] S3: After completing S2, the transmission client establishes a connection with the transmission server and starts the transmission, and sets a special identifier in the packet header of the data transmission of the assembled large file to indicate that the current file is a combined file.

[0019] S4: The transmission server receives the transmission request sent by the transmission client, starts to receive data, and confirms whether the current file is a single file or a combined file through the identifier in the packet header.

[0020] S5: through the judgment of S4, if the current file is a combined file, it is necessary to split the received data into different formats and parse the data packets to obtain the data information of each file; if it is a single file, the data information of the single file is obtained.

[0021] S6: According to the data information of the file received in S5, a file is created on the disk of the transmission server through the transmission server mapping path, and then the file data is written to the disk.

[0022] S7: Repeat S4-S6 until all files are transferred. After the transfer is complete, check each file to ensure that the file is correct.

[0023] As the cornerstone of TCP / IP, Socket can assemble data packets according to its own needs during transmission, and then parse the correct data at the receiving end.

[0024] Disks are limited by the upper limit of I / O and cannot read and write files without limit. At the same time, under the existing technical background, there is no I / O multiplexing technology or file system. At the same time, the file descriptor, as a unique identifier of a file, is basically equivalent to a disk I / O, and the file descriptor cannot be reused. The current solution is that one file can correspond to multiple file descriptors, but it is rare to implement one file descriptor corresponding to multiple files. Therefore, the only solution is to limit the program to use descriptors, that is, one transmission task only opens one file descriptor, thus avoiding multiple I / O operations for one transmission task. So-called fast is slow, slow is fast.

[0025] One problem with small file transmission is that it consumes a lot of socket links and disk I / O. Often, the file is transmitted just as the transmission speed starts to improve, but the file has to wait for the disk to be saved. The present invention can greatly improve the transmission efficiency by merging small files and limiting disk I / O.

[0026] Furthermore, in step S7, the verification is to compare the file list of the transmission client with the file list of the transmission server to check whether there are any files that are missed.

[0027] Furthermore, in step S1, it also includes: the transmission client calculates the hash value of each small file and records the hash value of each file in the file data; in step S7, the transmission server calculates the hash value of each small file and compares it with the hash value of the small file before transmission; if they are consistent, the small file is successfully transmitted; if they are inconsistent, the small file is unsuccessful, and the transmission server feeds back the information of the unsuccessful transmission of the small file to the transmission client, and the client re-uploads the small file. The hash values ​​before and after the transmission of the small file are calculated and compared to determine whether the file is damaged, so as to avoid the damaged small file being stored in the transmission server.

[0028] Furthermore, the assembled large file includes: storage of small file data, local path, transmission server mapping path, file size, file modification time, and file MD5 data.

[0029] Furthermore, the data information of the file includes: local path, transmission server mapping path, file data, file size, file modification time, and file MD5 data.

[0030] Furthermore, in step S2, the transmission client calculates the hash value of the file to be transmitted, and in step S4, the transmission server calculates the hash value of the received file and compares it with the hash value before transmission. If they are consistent, it proceeds to step S5. If they are inconsistent, it means that data is damaged during the transmission process. The transmission server feeds back the information of the file damage to the transmission client, and the transmission client re-uploads the file. The files to be transmitted include: assembled large files and single files.

[0031] In this solution, a hash value is calculated for the file transmitted in step S2, and the transmission server calculates a hash value for the received file and compares them to determine whether the file is damaged during the transmission process, thereby avoiding damaged files from being stored in the transmission server.

[0032] During the entire transmission process, the transmission client only calculates the hash value once for the files to be transmitted and once for the assembled large files, and the transmission server only calculates the hash value once for the files to be received and once for the assembled large files, avoiding repeated calculations and reducing the amount of calculations. The files to be transmitted and the files to be received include: small files that need to be assembled into large files and files that are transmitted individually.

[0033] By adopting the above scheme, the present invention provides a method for improving the transmission speed of a large number of small files. When a large number of small files are transmitted on the client, the small files are assembled to generate a large file of 2G, and then transmitted. After the transmission server receives the data, the large file is split and the corresponding files are written to the corresponding paths. Thereby, the transmission speed is improved, the number of disk I / 0 is reduced, the storage load is reduced, and the efficiency of the transmission server is improved. By adopting the method of the present invention, the utilization rate of the network bandwidth can be greatly improved, the storage load can be reduced, and the I / O queuing time can be reduced. The transmission efficiency is improved by 20-40%, and the storage load is reduced by 20%. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a transmission schematic diagram of the prior art;

[0035] Figure 2 An embodiment of the data structure for merging small files into a large file according to the present invention;

[0036] Figure 3 It is a transmission schematic diagram of the present invention. DETAILED DESCRIPTION

[0037] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0038] See also Figure 2 and Figure 3The present invention provides a method for improving the transmission speed of a large number of small files, comprising the following steps.

[0039] S1: Before the transmission client starts the transmission, it first scans the transmission file list to see if there is a file smaller than 20M. If so, the local path of the file and the mapping path of the transmission server are recorded in a temporary database.

[0040] S2: After the transmission client finishes scanning the file list, it assembles the small files into several large files and generates a corresponding number of temporary files locally on the user; the size of the assembled large files does not exceed 2GB.

[0041] S3: After completing S2, the transmission client establishes a connection with the transmission server and starts the transmission, and sets a special identifier in the packet header of the data transmission of the assembled large file to indicate that the current file is a combined file.

[0042] S4: The transmission server receives the transmission request sent by the transmission client, starts to receive data, and confirms whether the current file is a single file or a combined file through the identifier in the packet header.

[0043] S5: through the judgment of S4, if the current file is a combined file, it is necessary to split the received data into different formats and parse the data packets to obtain the data information of each file; if it is a single file, the data information of the single file is obtained.

[0044] S6: According to the data information of the file received in S5, a file is created on the disk of the transmission server through the transmission server mapping path, and then the file data is written to the disk.

[0045] S7: Repeat S4-S6 until all files are transferred. After the transfer is complete, check each file to ensure that the file is correct.

[0046] In this embodiment, in step S7, the verification is to compare the file list of the transmission client with the file list of the transmission server to check whether there are any files that are missed.

[0047] In this embodiment, step S1 also includes: the transmission client calculates the hash value of each small file, and records the hash value of each file in the file data; in step S7, the transmission server calculates the hash value of each small file, and compares it with the hash value of the small file before transmission; if they are consistent, the small file is transmitted successfully; if they are inconsistent, the small file is transmitted unsuccessfully, and the transmission server feeds back the information of the unsuccessful transmission of the small file to the transmission client, and the client re-uploads the small file.

[0048] In this embodiment, the assembled large file includes: storage of small file data, local path, transmission server mapping path, file size, file modification time, and file MD5 data.

[0049] In this embodiment, the data information of the file includes: local path, transmission server mapping path, file data, file size, file modification time, and file MD5 data.

[0050] In this embodiment, in step S2, the transmission client calculates the hash value of the file to be transmitted, and in step S4, the transmission server calculates the hash value of the received file and compares it with the hash value before transmission. If they are consistent, the process proceeds to step S5. If they are inconsistent, it means that data is damaged during the transmission process. The transmission server feeds back the information about the file damage to the transmission client, and the transmission client re-uploads the file. The files to be transmitted include: assembled large files and single files.

[0051] In summary, the present invention provides a method for improving the transmission speed of a large number of small files. When a client transmits a large number of small files, the small files are assembled to generate a 2G large file, and then transmitted. After the transmission server receives the data, the large file is split and the corresponding file is written to the corresponding path. Thereby improving the transmission speed, reducing the number of disk I / 0, reducing the storage load, and improving the efficiency of the transmission server. The method of the present invention can greatly improve the utilization rate of network bandwidth, reduce storage load, and reduce I / O queuing time. The transmission efficiency is improved by 20-40%, and the storage load is reduced by 20%.

[0052] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for improving the transmission speed of a large number of small files, characterized in that: The following steps are involved: S1: Before the transmission client starts the transmission, it first scans whether there is a file smaller than 20M in the transmission file list. If there is, the local path of the file and the mapping path of the transmission server are recorded in a temporary database; S2: After the transmission client scans the file list, it assembles the small files into several large files and generates a corresponding number of temporary files locally on the user; the size of the assembled large files does not exceed 2GB; S3: After completing S2, the transmission client establishes a connection with the transmission server and starts the transmission. A special identifier is set in the packet header of the data transmission of the assembled large file to indicate that the current file is a combined file. S4: The transmission server receives the transmission request sent by the transmission client, starts receiving data, and confirms whether the current file is a single file or a combined file through the identifier in the packet header; S5: through the judgment of S4, if the current file is a combined file, it is necessary to split the received data into different formats and parse the data packets to obtain the data information of each file; If it is a single file, obtain the data information of the single file; S6: according to the data information of the file received in S5, a file is created on the disk of the transmission server through the mapping path of the transmission server, and then the file data is written to the disk; S7: Repeat S4-S6 until all files are transferred. After the transfer is completed, check each file to ensure that the file is correct.

2. A method for improving the transmission speed of a large number of small files according to claim 1, characterized in that: In step S7, the verification is to compare the file list of the transmission client with the file list of the transmission server to check whether there are any files that are missed.

3. A method for improving the transmission speed of a large number of small files according to claim 1, characterized in that: In step S1, it also includes: the transmission client calculates the hash value of each small file and records the hash value of each file in the file data; in step S7, the transmission server calculates the hash value of each small file and compares it with the hash value of the small file before transmission; if they are consistent, the small file is transmitted successfully, if they are inconsistent, the small file is transmitted unsuccessfully, and the transmission server feeds back the information of the unsuccessful transmission of the small file to the transmission client, and the client re-uploads the small file.

4. A method for improving the transmission speed of a large number of small files according to claim 1, characterized in that: The assembled large file includes: storage of small file data, local path, transmission server mapping path, file size, file modification time, and file MD5 data.

5. A method for improving the transmission speed of a large number of small files according to claim 1, characterized in that: The file data information includes: local path, transmission server mapping path, file data, file size, file modification time, and file MD5 data.

6. A method for improving the transmission speed of a large number of small files according to claim 1, characterized in that: In step S2, the transmission client calculates the hash value of the file to be transmitted. In step S4, the transmission server calculates the hash value of the received file and compares it with the hash value before transmission. If they are consistent, it goes to step S5. If they are inconsistent, it means that data is damaged during the transmission process. The transmission server feeds back the information of file damage to the transmission client, and the transmission client re-uploads the file. The files to be transferred include: assembled large files and single files.

Images