Fragmented downloading test method and device based on multi-thread downloading
By customizing the number of threads and shard size in the shard download test and verifying file integrity using hash algorithm, the problem of lack of transparency and coverage in traditional testing methods is solved, and more efficient and accurate test results are achieved .
Patent Information
- Application Number
- CN202411126883.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-03
AI Technical Summary
The traditional shard download testing method lacks transparency and detailed monitoring of the download process, and cannot customize the number of threads and shard size, resulting in weak coverage of the test scenarios and the testing efficiency and accuracy need to be improved.
Using the shard download test method based on multi-thread download, customize the shard size and number of threads, verify the integrity and consistency of shard download and merged files through hashing algorithm, and capture and record error information during the download process. .
It improves the accuracy and reliability of the test, can cover more test scenarios, ensures file integrity and consistency, and improves the data support for testing efficiency and quality evaluation.
Smart Images

Figure CN119071046A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of file downloading, and in particular relates to a fragment downloading test method and device based on multi-threaded downloading. Background Art
[0002] In the field of computer network technology, file downloading is one of the most common operations for users. However, the traditional single-thread download method has obvious disadvantages when it comes to downloading large files. First, single-thread download can only process the reception and writing of data packets in sequence, resulting in the download speed being limited by the bandwidth of a single network connection and the server response speed. Second, when a single-thread download encounters network fluctuations or a session connection timeout, it is often necessary to restart the entire download process, which not only wastes the user's time and bandwidth resources, but also increases the processing burden on the server.
[0003] In order to overcome the drawbacks of single-threaded downloading, multi-threaded downloading technology came into being. Multi-threaded downloading allows multiple threads to be opened at the same time for file downloading, and each thread is responsible for processing the reception and writing of a part of the data packets. This method can make full use of bandwidth resources and realize the simultaneous transmission of multiple data packets, thereby significantly improving the download speed. In addition, multi-threaded downloading also has better fault tolerance and stability. Even if a thread is interrupted due to network fluctuations or server failures, other threads can still continue to download, ensuring the continuity and integrity of the overall download process.
[0004] Existing segment download tests usually have the following problems:
[0005] The testing process is not transparent:
[0006] Traditional segmented download tests are often implemented through tools, and only verify the files that are finally downloaded, lacking detailed monitoring and recording of each stage of the download process.
[0007] This black box testing method makes it impossible for testers to observe the specific process of segment downloading in real time, such as the segment download speed, the download status of each segment, etc.
[0008] Insufficient thread and shard size control:
[0009] In traditional segmented download testing, testers usually cannot directly specify the number of threads used in the download process, nor can they accurately control the size of each segment.
[0010] This may result in incomplete coverage of test scenarios, as different numbers of threads and shard sizes may affect download speed, stability, and resource usage.
[0011] The coverage of test scenarios is weak:
[0012] Since it is impossible to directly control the number of threads and the size of the segments, traditional segment download tests are unable to cover all possible download scenarios.
[0013] For example, the performance of segmented downloads may vary in different environments such as high concurrency, low bandwidth, and network fluctuations, but traditional testing methods may not be able to fully simulate and test these scenarios.
[0014] Testing efficiency and accuracy need to be improved:
[0015] Traditional multi-segment download tests usually only focus on the integrity verification of the final file, while ignoring the monitoring and analysis of errors, exceptions and bottlenecks that may occur during the download process.
[0016] This can lead to inefficient testing because even if the final file verifies successfully, there may still be potential performance or stability issues. Summary of the invention
[0017] The purpose of the present invention is to provide a multi-threaded download test method, which can customize the fragment size and the number of threads, verify the integrity and consistency of the fragments after downloading and merging files through a hash algorithm, and improve the accuracy and reliability of the test.
[0018] To solve the above problems, the technical solution of the present invention is:
[0019] A multi-threaded download test method includes:
[0020] Customize the fragment size, divide the file to be downloaded into multiple fragments according to the set fragment size;
[0021] Customize the number of threads, create download threads according to the number of shards, and each thread is responsible for downloading a specific shard;
[0022] Perform integrity testing on each downloaded segment to ensure that the segment has not been tampered with or damaged;
[0023] Merge all downloaded segments in the order of the original files to generate a complete download file;
[0024] Perform a hash comparison on the downloaded file to ensure it is identical to the original file.
[0025] According to an embodiment of the present invention, during the process of segment downloading, errors that occur are captured and processed, and logs are recorded for subsequent analysis.
[0026] According to one embodiment of the present invention, when customizing the segment size, the segment size segmentSize is set according to system resources, network bandwidth and the size of the file to be downloaded, and the total number of segments segmentsNum = int(fileSize / segmentSize)+1 is calculated; wherein fileSize is the size of the file to be downloaded.
[0027] According to an embodiment of the present invention, when executing a multi-threaded segment download task, a thread pool is created, tasks are assigned to each thread, and the starting position i*segmentSize and the ending position (i+1)*segmentSize-1 of the bytes to be downloaded by each thread are calculated according to the thread number i, i=0,1,2,3,...,n;
[0028] In each thread, build an HTTP request, set the Rang header information, and specify the byte position range to be downloaded.
[0029] According to an embodiment of the present invention, a hash value is calculated for each downloaded segment, and the obtained hash value is compared with a preset value. If they are not equal, the segment is downloaded again;
[0030] The number of times to re-download the segments is limited. If the preset number is exceeded, an error message is recorded and an abnormal prompt is given.
[0031] A multi-threaded download testing device comprising:
[0032] The fragment customization module is used to customize the fragment size and divide the file to be downloaded according to the set fragment size to obtain multiple fragments;
[0033] The thread customization module is used to customize the number of threads and create download threads according to the number of shards. Each thread is responsible for downloading a specific shard.
[0034] The integrity test module is used to perform integrity tests on each downloaded segment to ensure that the segment has not been tampered with or damaged;
[0035] The merging module is used to merge all downloaded segments in the order of the original files to generate a complete download file;
[0036] The consistency test module is used to compare the hash consistency of the downloaded file to ensure that it is exactly the same as the original file.
[0037] According to an embodiment of the present invention, the log recording module is used to capture and process errors that occur during the segment download process and record logs for subsequent analysis.
[0038] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art:
[0039] The multi-threaded download test method in one embodiment of the present invention customizes multiple fragments according to the size of the file to be downloaded, specifies the size of each fragment, and confirms the total number of fragments; creates a corresponding number of download threads according to the total number of fragments, and each thread is responsible for downloading a specific fragment; after downloading each fragment, performs an integrity test on it to ensure that the fragment is not damaged or tampered with; merges all downloaded fragments in the order of the original files to generate a complete download file; and performs a hash consistency comparison on the merged file to ensure that it is exactly the same as the original file. By flexibly controlling the number of threads and the size of the fragments, more test scenarios can be covered, providing more comprehensive data support for the quality evaluation of the fragment download technology, and improving the test efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a flow chart of a multi-threaded download test method in one embodiment of the present invention;
[0041] Figure 2 A data flow diagram of a multi-threaded segmented downloading test method according to an embodiment of the present invention;
[0042] Figure 3 A step diagram of a multi-threaded shard downloading test method according to an embodiment of the present invention;
[0043] Figure 4 The figure is a block diagram of a fragment downloading test device based on multi-threaded downloading in one embodiment of the present invention. DETAILED DESCRIPTION
[0044] The following is a further detailed description of a multi-threaded download test method and device proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims.
[0045] The advantages of multi-threaded downloading are:
[0046] Concurrency: Multiple threads can work simultaneously, increasing download speed.
[0047] Fault tolerance: If a thread fails to download, other threads can continue downloading without causing the entire download task to fail.
[0048] Resource Utilization: Make full use of network bandwidth and server resources, especially when the network conditions are good.
[0049] As a way to implement multi-threaded downloading, segmented downloading divides the file to be downloaded into multiple segments, and each segment is downloaded independently. This method further improves download efficiency and stability. Segmented downloading, also known as breakpoint resume or block downloading, has the following advantages:
[0050] Resume downloading from breakpoints: If the download process is interrupted, you can continue downloading from the breakpoint without having to start over.
[0051] Flexibility: The downloaded chunk size can be dynamically adjusted based on network conditions to optimize download speed.
[0052] Load balancing: Distribute download tasks among multiple servers or mirror sites to reduce the pressure on a single server.
[0053] It can be seen that there are differences between multi-threaded downloading and segmented downloading in the following aspects:
[0054] Concurrency: Multi-threaded downloading emphasizes using multiple threads to download different parts of the same file at the same time, while segmented downloading focuses on splitting the file into multiple parts, each of which can be downloaded independently.
[0055] Breakpoint resume: Segmented downloading naturally supports breakpoint resume, and although multi-threaded downloading can also achieve breakpoint resume, it requires additional logic to manage the download status of each thread.
[0056] Resource allocation: Multi-threaded downloading usually allocates resources on a single server, while segmented downloading can allocate resources among multiple servers or mirror sites.
[0057] Implementation complexity: Multi-threaded downloading may require more complex thread management and synchronization mechanisms, while segmented downloading requires processing file splitting and merging logic.
[0058] In actual applications, these two technologies are often used in combination to achieve the best download efficiency. For example, a download tool may use multi-threading technology to download different fragments of the same file from multiple servers or mirror sites, while supporting breakpoint resume function. This not only takes advantage of the concurrency of multi-threading, but also ensures the flexibility and reliability of downloading.
[0059] In the process of testing multi-segment downloading, a prominent issue is how to confirm the integrity and consistency of multi-segment downloading. Since multi-segment downloading divides a file into multiple segments for independent downloading, it is necessary to ensure that each segment is not damaged or tampered with during the download process and can be restored to the complete original file after merging. This requires integrity testing of each segment and consistency confirmation after merging.
[0060] This embodiment provides a relatively complete test process method to implement the entire process of segmented downloading and confirm the consistency of files. Specifically, it is a segmented download test method based on multi-threaded downloading. Please refer to Figure 1 , the method comprises the following steps:
[0061] Customize the fragment size, divide the file to be downloaded into multiple fragments according to the set fragment size;
[0062] Customize the number of threads, create download threads according to the number of shards, and each thread is responsible for downloading a specific shard;
[0063] Perform integrity testing on each downloaded segment to ensure that the segment has not been tampered with or damaged;
[0064] Merge all downloaded segments in the order of the original files to generate a complete download file;
[0065] Perform a hash comparison on the downloaded file to ensure it is identical to the original file.
[0066] The above method provides flexible control over the fragment size, allowing the test to cover different fragment strategies. The choice of fragment size has an impact on both download speed and resource utilization. By customizing the fragment size, testers can evaluate the download performance under different fragment sizes and provide the best fragment strategy recommendations for actual applications.
[0067] In addition, testers are allowed to customize the number of threads according to test requirements or target network environment. In multi-threaded downloading, the number of threads directly affects the speed and stability of the download. By controlling the number of threads, testers can simulate different download scenarios, such as testing the performance of segmented downloading in high concurrency, low bandwidth and other environments.
[0068] The integrity test of each downloaded segment is performed through the hash algorithm, and the consistency test of the merged file is performed to ensure that the downloaded file is completely consistent with the original file. This can effectively prevent the segments from being damaged or tampered with during the download or merge process, and improve the accuracy and reliability of the test.
[0069] Furthermore, in order to accurately locate errors that occur during the test, the method also captures and processes errors that occur during the segment download process, and records logs for subsequent analysis. If download failure or file inconsistency is found during the test, the problem can be quickly located by analyzing the download logs and monitoring data, providing a basis for subsequent repairs and optimizations.
[0070] For details, please see Figure 2 and Figure 3When testing multi-threaded segment downloads, first, you need to determine the URL of the file to be downloaded and the target location to save it after downloading. By first downloading a small part of the file data, such as the first 10 bytes of the file, you can obtain basic information about the file, such as the file size, file name, and file unique encoding. You can also obtain the file size through the Content-Length field in the HTTP request header. Then you can calculate the MD5 value of the file for subsequent file consistency testing.
[0071] Second, perform file division. This embodiment provides flexible control of custom segment size. Users can reasonably customize the segment size segmentSize based on factors such as system resources, network bandwidth, and file size, and then calculate the total number of segments based on the segment size: segmentsNum = int(fileSize / segmentSize)+1, where fileSize is the file size.
[0072] Third, perform multi-threaded segment download tasks. Use the thread pool technology in the Python programming language to create n threads for downloading. Assign tasks to each thread, and each thread is responsible for downloading a specific segment. According to the thread number i, i = 0, 1, 2, ..., n, calculate the starting position (start = i * segmentSize) and the end position (end = (i + 1) * segmentSize-1) that the thread needs to download, but the last thread may need to download to the end of the file. Set the HTTP request header. In the thread, build an HTTP request and set the Range header information to specify the byte range to be downloaded (such as Range: bytes = start-end). Get the data in the specified range through the HTTP request and save it to the corresponding location of the local file.
[0073] Fourth, shard test. For each downloaded shard, calculate its hash value (such as MD5, SHA-1, etc.) to verify the integrity and accuracy of the data. Compare the calculated hash value with the expected value (which can be provided by the server or obtained from other reliable sources). If the hash value comparison fails, it means that there is a problem with the shard data and the shard needs to be downloaded again. You can set a retry limit. After exceeding the limit, stop retrying and record error information.
[0074] Fifth, merge verification. After all threads have completed the download task (including re-downloading), merge the segmented files downloaded by each thread into a complete file. You can use file operation functions (such as the open() function in Python to implement file read and write operations) to ensure the correctness and efficiency of the merge process. Perform a hash consistency comparison on the merged file to ensure that it is exactly the same as the original file. For example, calculate the MD5 value of the merged file and compare it with the MD5 value obtained before multi-threaded segment downloading.
[0075] In addition to the above steps, you can also record the exceptions encountered during the download process. For example, during the entire download process, record key information and error logs for subsequent analysis and troubleshooting. Capture and process possible network exceptions, file read and write exceptions, etc. to ensure the robustness of the program.
[0076] In summary, this method can cover more test scenarios by flexibly controlling the number of threads and the size of the fragments, providing more comprehensive data support for the quality assessment of the fragment download technology. The download process can also be monitored in real time and data can be collected through automated test scripts and monitoring tools to improve the efficiency of the test. If download failures or file inconsistencies are found during the test, the problem can be quickly located by analyzing the download logs and monitoring data, providing a basis for subsequent repairs and optimizations.
[0077] Based on the same concept, this embodiment also provides a multi-threaded download test device, see Figure 4 , the device comprises:
[0078] The fragment customization module is used to customize the fragment size and divide the file to be downloaded according to the set fragment size to obtain multiple fragments;
[0079] The thread customization module is used to customize the number of threads and create download threads according to the number of shards. Each thread is responsible for downloading a specific shard.
[0080] The integrity test module is used to perform integrity tests on each downloaded segment to ensure that the segment has not been tampered with or damaged;
[0081] The merging module is used to merge all downloaded segments in the order of the original files to generate a complete download file;
[0082] The consistency test module is used to compare the hash consistency of the downloaded file to ensure that it is exactly the same as the original file.
[0083] Furthermore, the device also includes a log recording module, which is used to capture and process errors that occur during the segment download process and record logs for subsequent analysis.
[0084] The device is used to implement the steps in the above-mentioned multi-threaded downloading-based fragment downloading test method and achieve the same functions, and the specific implementation method will not be repeated.
[0085] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Even if various changes are made to the present invention, if these changes fall within the scope of the claims of the present invention and their equivalents, they still fall within the protection scope of the present invention.
Claims
1. A multi-threaded download test method, characterized in that: include: Customize the fragment size, divide the file to be downloaded into multiple fragments according to the set fragment size; Customize the number of threads, create download threads according to the number of shards, and each thread is responsible for downloading a specific shard; Perform integrity testing on each downloaded segment to ensure that the segment has not been tampered with or damaged; Merge all downloaded segments in the order of the original files to generate a complete download file; Perform a hash comparison on the downloaded file to ensure it is identical to the original file.
2. The multi-threaded download test method according to claim 1, characterized in that: During the segment download process, any errors that occur are captured and processed, and logs are recorded for subsequent analysis.
3. The multi-threaded download test method according to claim 1, characterized in that: When customizing the segment size, set the segment size segmentSize according to system resources, network bandwidth and the size of the file to be downloaded, and calculate the total number of segments segmentsNum=int(fileSize / segmentSize)+1; where fileSize is the size of the file to be downloaded.
4. The multi-threaded download test method according to claim 3, characterized in that: When executing multi-threaded segment download tasks, create a thread pool, assign tasks to each thread, and calculate the starting position i*segmentSize and the ending position (i+1)*segmentSize-1 of the bytes that each thread needs to download according to the thread number i, i=0,1,2,3,...,n; In each thread, build an HTTP request, set the Rang header information, and specify the byte position range to be downloaded.
5. The multi-threaded download test method according to claim 1, characterized in that: Calculate the hash value for each downloaded segment and compare the obtained hash value with the preset value. If they are not equal, re-download the segment. The number of times to re-download the segments is limited. If the preset number is exceeded, an error message is recorded and an abnormal prompt is given.
6. A fragment download test device based on multi-thread download, characterized in that: include: The fragment customization module is used to customize the fragment size and divide the file to be downloaded according to the set fragment size to obtain multiple fragments; The thread customization module is used to customize the number of threads and create download threads according to the number of shards. Each thread is responsible for downloading a specific shard. The integrity test module is used to perform integrity tests on each downloaded segment to ensure that the segment has not been tampered with or damaged; The merging module is used to merge all downloaded segments in the order of the original files to generate a complete download file; The consistency test module is used to compare the hash consistency of the downloaded file to ensure that it is exactly the same as the original file.
7. The multi-threaded download testing device according to claim 6, characterized in that: It also includes a logging module, which is used to capture and process errors that occur during the segment download process and record logs for subsequent analysis.
Citation Information
Cited By
Data downloading method and device, electronic equipment and medium
CN119484516A
