Method and system for optimizing file transmission delay

Abstract

The invention relates to a file transmission delay optimization method and system, and the method comprises the steps: S1, carrying out the real-time fragmentation processing of a file when the file is uploaded, and carrying out the transmission and uploading of the fragmentation according to a network condition; s2, when the file is transmitted to the last two fragments, inquiring and verifying the file size condition of the last two fragments, and judging the current network condition; s3, when the size of the last fragment exceeds 50% of the size of the last second fragment or the current network condition is not smooth, independently uploading the last two fragments; and S4, when the size of the last fragment does not exceed 50% of the size of the last second fragment and the current network condition is relatively stable or smooth, combining the last fragment and the last second fragment into one fragment and uploading the fragment. According to the method, the long-tail effect can be effectively reduced, and the overall large file fragment uploading efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer communications, and in particular to a method and system for optimizing file transmission delay. Background Art

[0002] Many systems currently have the function of uploading files. Usually, small files can be successfully uploaded with one request. If a large file is to be uploaded, the function of uploading in pieces is usually used. That is, a large file is split into multiple small files, and then these small files are uploaded separately. Finally, after all small files are successfully uploaded, the server performs a merge operation to merge all small files into a large file, and finally the upload operation of the large file is realized. Usually, large files are processed in pieces, and finally each piece is uploaded. However, for the processing of the end of the uploaded file, it can be concluded through a large number of comparative experiments that when the last piece is a piece of a few bytes, the delay time for its upload is almost the same as the upload time of the larger piece. In fact, this is a long-tail effect of file piece upload. Summary of the invention

[0003] In order to overcome the above problems, the purpose of the present invention is to provide a method and system for optimizing file transmission delay, which can effectively reduce the long tail effect and improve the efficiency of uploading large files in segments.

[0004] The present invention is implemented by the following scheme: a method for optimizing file transmission delay, the method comprising the following steps:

[0005] Step S1: When uploading a file, the file is segmented in real time and the segments are transmitted and uploaded according to the network conditions;

[0006] Step S2: When the last two fragments are transmitted, the file sizes of the last two fragments are checked and the current network conditions are determined;

[0007] Step S3: when the size of the last fragment exceeds 50% of the size of the second last fragment or the current network condition is not smooth, the last two fragments are uploaded separately;

[0008] Step S4: When the size of the last fragment does not exceed 50% of the size of the second last fragment, and the current network condition is stable or unblocked, the last fragment and the second last fragment are merged into one fragment and uploaded.

[0009] Furthermore, the real-time slicing of the file in step S1 is further as follows: slicing the file multiple times, recording which data point each slicing is processed to, and slicing from this point to subsequent data points each time.

[0010] Furthermore, the real-time segmentation processing of files in the segmentation processing module is further as follows: the current network condition is further judged as follows: when the network speed is less than 1m / s, it is not smooth; when the network speed is greater than 1m / s and less than 5m / s, it is smooth; when the network speed is greater than 5m / s and less than 10m / s, it is stable.

[0011] The present invention also provides a file transmission delay optimization system, the system comprising a fragmentation processing module, a judgment module, a first uploading module, and a second uploading module;

[0012] The fragment processing module: when uploading a file, the file is fragmented in real time and the fragments are transmitted and uploaded according to the network conditions;

[0013] The judgment module: when transmitting to the last two fragments, query and verify the file sizes of the last two fragments, and judge the current network conditions;

[0014] The first uploading module: when the size of the last fragment exceeds 50% of the size of the second last fragment or the current network condition is not smooth, the last two fragments are uploaded separately;

[0015] The second uploading module: when the size of the last fragment does not exceed 50% of the size of the second last fragment and the current network condition is stable or unblocked, the last fragment and the second last fragment are merged into one fragment and uploaded.

[0016] Furthermore, the real-time slicing processing of the file in the slicing processing module is further as follows: slicing the file multiple times, recording which data point each slicing process is to, and slicing from this point to subsequent data points each time.

[0017] Furthermore, the real-time segmentation processing of files in the segmentation processing module is further as follows: the current network condition is further judged as follows: when the network speed is less than 1m / s, it is not smooth; when the network speed is greater than 1m / s and less than 5m / s, it is smooth; when the network speed is greater than 5m / s and less than 10m / s, it is stable.

[0018] The beneficial effect of the present invention is that: in the prior art, a large file is processed by fragmentation, and finally each fragment is uploaded. However, for the processing of the end of the uploaded file, it can be concluded through a large number of comparative experiments that when the size of the last fragment is even a fragment of a few bytes, the delay time for its upload is almost the same as the upload time of the fragment with a larger capacity. In fact, this is a long-tail effect of file fragment upload. In view of the above situation, the present invention adopts a new method. After the large file is fragmented, when it is transmitted to the last two fragments, the file size of the last fragment is queried and verified. When its size meets the conditions for merging, the last two fragments are directly merged and processed, and only one fragment needs to be sent. When the conditions for merging are not met, the fragments are still uploaded according to the old logic. In this way, the long-tail effect can be effectively reduced in the fragment upload scenario, and the efficiency of the overall large file fragment upload can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flow chart of the method of the present invention.

[0020] Figure 2 It is a system principle block diagram of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings.

[0022] See also Figure 1 As shown, a method for optimizing file transmission delay of the present invention comprises the following steps:

[0023] Step S1: When uploading a file, the file is segmented in real time and the segments are transmitted and uploaded according to the network conditions;

[0024] Step S2: When the last two fragments are transmitted, the file sizes of the last two fragments are checked and the current network conditions are determined;

[0025] Step S3: when the size of the last fragment exceeds 50% of the size of the second last fragment or the current network condition is not smooth, the last two fragments are uploaded separately;

[0026] Step S4: When the size of the last fragment does not exceed 50% of the size of the second last fragment, and the current network condition is stable or unblocked, the last fragment and the second last fragment are merged into one fragment and uploaded.

[0027] The present invention will be further described below in conjunction with a specific embodiment:

[0028] A method for optimizing file transfer delay,

[0029] 1. When uploading files, the files are segmented in real time and the segments are uploaded and transmitted according to the network conditions. The specific network conditions can be judged based on specific business tests. Usually 10M / s is a good network transmission point.

[0030] 1.1 The real-time sharding process of the file is further as follows: the file is sharded multiple times, and the data point to which each sharding process is processed is recorded, and each time the sharding is performed from this point to the subsequent data points.

[0031] Suppose a 1000M file is uploaded, in which the size of each shard is not fixed. When the network is stable in the middle, the size of the transmitted shard may be larger. During the upload process, the transmission of the fixed shard size is not performed, but a certain shard is transmitted first. According to the size of the shard and the transmission time, the network transmission situation of the time period can be evaluated. When the network transmission reaches a certain condition, the size of the next shard is gradually increased, so that the more stable the network is, the larger the size of the shard that continues to be transmitted is, and the effect of reducing the transmission time is achieved. Similarly, when the network is unstable, the size of the shard is reduced, but the number of shards will be increased, and the impact of the network will be offset by increasing the number of shards. This is also quite special. And the sharding of the present application is real-time sharding division, which records which data point each shard is processed to, and then shards from this point to the subsequent data point each time, so that when it is known that there are only 2 shards left later, if the conditions in the second description above are met, the data of the last 2 shard ranges are directly read out and transmitted as a complete shard.

[0032] 2. When transferring to the last two fragments, check and verify the file sizes of the last two fragments and determine the current network conditions;

[0033] 2.1 The real-time fragmentation processing of files in the fragmentation processing module is further as follows: the current network condition is further judged as follows: when the network speed is less than 1m / s, it is not smooth; when the network speed is greater than 1m / s and less than 5m / s, it is smooth; when the network speed is greater than 5m / s and less than 10m / s, it is stable.

[0034] 3. When the size of the last fragment exceeds 50% of the size of the second last fragment or the current network condition is not smooth, the last two fragments will be uploaded separately;

[0035] 4. When the size of the last fragment does not exceed 50% of the size of the second last fragment, and the current network condition is stable or unobstructed, the last fragment and the second last fragment are merged into one fragment and uploaded. First, the last 50% of the conditions must be met first, and at the same time, the network condition must be stable or unobstructed at this time. When these two conditions are met at the same time, the last two fragments are merged to reduce the long tail effect.

[0036] See also Figure 2 As shown, the present invention also provides a system for optimizing file transmission delay, the system comprising a fragmentation processing module, a judgment module, a first uploading module, and a second uploading module;

[0037] The fragment processing module: when uploading a file, the file is fragmented in real time and the fragments are transmitted and uploaded according to the network conditions;

[0038] The real-time sharding process of the file is further as follows: the file is sharded multiple times, the data point to which each sharding process is performed is recorded, and sharding is performed from this point to subsequent data points each time.

[0039] Suppose a 1000M file is uploaded, in which the size of each shard is not fixed. When the network is stable in the middle, the size of the transmitted shard may be larger. During the upload process, the transmission of the fixed shard size is not performed, but a certain shard is transmitted first. According to the size of the shard and the transmission time, the network transmission situation of the time period can be evaluated. When the network transmission reaches a certain condition, the size of the next shard is gradually increased, so that the more stable the network is, the larger the size of the shard that continues to be transmitted is, and the effect of reducing the transmission time is achieved. Similarly, when the network is unstable, the size of the shard is reduced, but the number of shards will be increased, and the impact of the network will be offset by increasing the number of shards. This is also quite special. And the sharding of the present application is real-time sharding division, which records which data point each shard is processed to, and then shards from this point to the subsequent data point each time, so that when it is known that there are only 2 shards left later, if the conditions in the second description above are met, the data of the last 2 shard ranges are directly read out and transmitted as a complete shard.

[0040] The judgment module: when transmitting to the last two fragments, query and verify the file sizes of the last two fragments, and judge the current network conditions;

[0041] The real-time fragmentation processing of the file in the fragmentation processing module is further as follows: the current network condition is further judged as follows: when the network speed is less than 1m / s, it is not smooth; when the network speed is greater than 1m / s and less than 5m / s, it is smooth; when the network speed is greater than 5m / s and less than 10m / s, it is stable.

[0042] The first uploading module: when the size of the last fragment exceeds 50% of the size of the second last fragment or the current network condition is not smooth, the last two fragments are uploaded separately;

[0043] The second uploading module: when the size of the last fragment does not exceed 50% of the size of the second last fragment, and the current network condition is stable or unobstructed, the last fragment and the second last fragment are merged into one fragment and uploaded. First, the last 50% of the conditions need to be met first, and at the same time, the network condition at this time needs to be relatively stable or unobstructed. When these two conditions are met at the same time, the last two fragments are merged to reduce the long tail effect.

[0044] In summary, the present invention adopts a new method. After a large file is fragmented, when it is transmitted to the last two fragments, the file size of the last fragment is queried and verified. When its size meets the conditions for merging, the last two fragments are directly merged and processed, and only one fragment needs to be sent. When the conditions for merging are not met, the fragments are still uploaded according to the old logic. In this way, the long tail effect can be effectively reduced in the fragment upload scenario, and the efficiency of the overall large file fragment upload can be improved.

[0045] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A method for optimizing file transmission delay, characterized in that: The method comprises the following steps: Step S1: When uploading a file, the file is segmented in real time and the segments are transmitted and uploaded according to the network conditions; Step S2: When the last two fragments are transmitted, the file sizes of the last two fragments are checked and the current network conditions are determined; Step S3: when the size of the last fragment exceeds 50% of the size of the second last fragment or the current network condition is not smooth, the last two fragments are uploaded separately; Step S4: When the size of the last fragment does not exceed 50% of the size of the second last fragment, and the current network condition is stable or unblocked, the last fragment and the second last fragment are merged into one fragment and uploaded.

2. The method for optimizing file transmission delay according to claim 1, characterized in that: The real-time slicing of the file in step S1 is further as follows: slicing the file multiple times, recording which data point each slicing is processed to, and slicing from this point to subsequent data points each time.

3. The method for optimizing file transmission delay according to claim 1, characterized in that: The real-time segmentation of the file in step S1 is further as follows: the current network condition is further determined as follows: when the network speed is less than 1m / s, it is not smooth; when the network speed is greater than 1m / s and less than 5m / s, it is smooth; when the network speed is greater than 5m / s and less than 10m / s, it is stable.

4. A file transfer delay optimization system, characterized in that: The system includes a sharding processing module and a judgment module. A first upload module, a second upload module; The fragment processing module: when uploading a file, the file is fragmented in real time and the fragments are transmitted and uploaded according to the network conditions; The judgment module: when transmitting to the last two fragments, query and verify the file sizes of the last two fragments, and judge the current network conditions; The first uploading module: when the size of the last fragment exceeds 50% of the size of the second last fragment or the current network condition is not smooth, the last two fragments are uploaded separately; The second uploading module: when the size of the last fragment does not exceed 50% of the size of the second last fragment and the current network condition is stable or unblocked, the last fragment and the second last fragment are merged into one fragment and uploaded.

5. The file transmission delay optimization system according to claim 4, characterized in that: The real-time slicing processing of the file in the slicing processing module is further as follows: slicing the file multiple times, recording which data point each slicing process is to, and slicing from this point to subsequent data points each time.

6. The file transmission delay optimization system according to claim 4, characterized in that: The real-time fragmentation processing of the file in the fragmentation processing module is further as follows: the current network condition is further judged as follows: when the network speed is less than 1m / s, it is not smooth; when the network speed is greater than 1m / s and less than 5m / s, it is smooth; when the network speed is greater than 5m / s and less than 10m / s, it is stable.

Images