A method for processing large response files in bidding process

Abstract

The invention discloses a large-volume response file processing method applied to a bidding process, S1. dividing a response file to be uploaded into a plurality of data blocks; S2. allocating each data slice to different threads for parallel uploading; S3. using the spark-md5 algorithm to respectively calculate the overall md5 value of the response file and the md5 value of each data slice; S4. in a breakpoint resume transmission scenario, judging the unfinished data slice based on the stored slice md5 value and the upload progress information; S5. in a second transmission scenario, directly feeding back to the user that the upload is successful without repeated uploading; S6. storing the preview slice file on the server side; S7. caching the loaded preview slice file data locally; S8. the client dynamically releases the preview slice file data in the cache to control the occupation of the local storage space. The invention significantly improves the upload and preview efficiency of large-volume response files in the bidding process.

Description

Technical Field

[0001] The present invention relates to the technical field of file processing, and in particular to a method for processing large-volume response files applied in a bidding process. Background Art

[0002] With the development of digital technology, the demand for file processing in the bidding process is increasing, especially in the uploading and bid evaluation stages of response files. Large-volume file processing has become a common technical problem. In traditional bidding and procurement systems, the response file processing process usually includes file uploading, storage, and bid evaluation preview. For the upload stage, existing systems mostly use fixed segment upload or single-thread upload methods, and rely on basic breakpoint resume functions. However, this technical solution has obvious shortcomings when processing large-volume files, which are specifically manifested in slow upload speeds, low breakpoint resume efficiency, and insufficient multi-thread support.

[0003] The existing technology usually processes large files by fragmenting them into fixed-size segments, splitting the files into several equal-sized data blocks, uploading them one by one, and attempting to resume the upload when the network is interrupted. Although this fixed-segment and single-threaded upload method can achieve basic file upload, its efficiency and stability are still relatively limited, especially when the network fluctuates greatly or the file size is large. The fixed-size segments will cause the upload speed to be unstable, and the size of the segments is difficult to match the current network conditions, resulting in a waste of resources during the transmission process. In addition, the resume function at the interruption point of the existing technology is usually unable to accurately record and quickly restore the upload progress. After the network is interrupted, most of the content needs to be re-uploaded from the beginning, which reduces the overall efficiency of file upload.

[0004] In terms of bid evaluation preview, the existing technology usually decrypts the entire file and displays it directly to the client without reasonably slicing and paging it according to the size of the file content. The direct display method is likely to cause system freezes when processing large-volume response files. Frequent previews during the bid evaluation process occupy a high amount of system resources, resulting in insufficient preview smoothness. In addition, many systems lack an efficient caching mechanism in file preview, resulting in the need to reload the file content for each preview request, which increases the system burden.

[0005] To sum up, the existing technology has significant deficiencies in uploading and previewing large files: the upload link has problems of slow upload speed and unstable breakpoint resumption, and the preview link has problems of system freezes and poor preview fluency. These defects directly affect the work efficiency and user experience of the bidding system. Summary of the invention

[0006] An object of the present invention is to propose a method for processing large-volume response files applied to a bidding process. The present invention significantly improves the efficiency of uploading and previewing large-volume response files during the bidding process.

[0007] A method for processing a large-volume response file applied to a bidding process according to an embodiment of the present invention comprises the following steps:

[0008] S1. Divide the response file to be uploaded into multiple data blocks, and perform data sharding on each data block, monitor the current network status and server processing capacity, and dynamically adjust the data size of each data shard;

[0009] S2. Create and manage a thread pool on the client based on the Web Worker API. By configuring an appropriate number of concurrent threads, each data shard is assigned to different threads for parallel upload.

[0010] S3. Use the spark-md5 algorithm to calculate the overall MD5 value of the response file and the MD5 value of each data shard, and send the MD5 value information together with the data shard data to the server to support breakpoint resume and instant upload functions;

[0011] S4. In the breakpoint resume scenario, based on the stored data shard md5 value and upload progress information, the unfinished data shard is determined according to the upload progress information and MD5 value comparison results, and the unfinished data shard is continued to be uploaded from the breakpoint;

[0012] S5. In the instant upload scenario, when the md5 value of the response file uploaded by the client is consistent with the md5 value of the file stored on the server, the upload success is directly fed back to the user without the need for repeated upload;

[0013] S6. For the response file uploaded by the user, after the response file is decrypted, the decrypted file is divided into several preview slice files, the number of pages of each preview slice file is configured according to the content of a single page of the file, and the preview slice files are stored on the server;

[0014] S7. During the bid evaluation process, the client requests the server to preview the slice file data as needed for the step-by-step preview display of the response file, and caches the loaded preview slice file data locally;

[0015] S8. According to the preview situation, the client dynamically releases the preview slice file data in the cache to control the occupancy of the local storage space and ensure smooth preview.

[0016] Optionally, the S1 includes:

[0017] S11. Divide the response file to be uploaded into multiple data blocks, and the size of each data block is defined as :

[0018] ;

[0019] Where i represents the serial number of the data block, and the size of the data block is defined based on the total size F of the response file and the expected number of uploaded shards N;

[0020] S12. For each data block Perform data sharding processing, divide the data block into multiple data shards, and set the size of each data shard to , where j is the data block The shard sequence number in the initial data shard size parameter , and dynamically adjust according to the real-time network status;

[0021] S13. Monitor the current network status, including network bandwidth W, network delay L, and server processing capacity S, and obtain the maximum transmission rate R of the network in real time by evaluating the fluctuation range of network bandwidth W, network delay L, and server processing capacity S:

[0022] ;

[0023] Dynamically adjust the size of data shards based on network conditions and server load , adjust the adjustment factor of the data shard size , adjustment coefficient It is derived from the ratio of the maximum transmission rate R and the server processing capacity S:

[0024] ;

[0025] Adjusted data shard size It is expressed as:

[0026] ;

[0027] in, The updated data shard size is adjusted to match the current network conditions and server processing capabilities.

[0028] Optionally, S2 includes:

[0029] S21. Create a thread pool on the client based on the Web Worker API and set the maximum number of concurrent threads T in the thread pool;

[0030] S22: According to the number N of data fragments to be uploaded, each data fragment Distribute to different threads, and evenly distribute the number of data shards and the number of threads, so that the number of data shards allocated to each thread is ;

[0031] S23. Setting an upload task for each thread, the upload task includes the allocation of data shards, status tracking, and recording of upload progress, and independently managing the upload status of the task in each thread;

[0032] S24, during the upload process, the status of each thread is monitored in real time, and if a thread task is completed, the remaining unuploaded data slices are dynamically allocated to the thread;

[0033] S25: Based on the order in which the data segments are uploaded, the uploaded data segments are merged. Reassemble into complete adjusted data blocks :

[0034] ;

[0035] in, Represents the reorganized data block i, which consists of multiple data fragments composition, Indicates the total number of data fragments contained in data block i, is the transmission rate deviation during the upload process, which indicates the difference between the actual transmission rate and the ideal transmission rate. It is used to adjust the upload order of data shards. S is the server processing capacity, which is used as a reference value for adjusting the shard merging order. The weight coefficient is the order weight coefficient of the data shards. The weight coefficient ensures the order of the data shards when the data shards are merged to avoid fluctuations in the transmission rate. Caused by disorder of sequence.

[0036] Optionally, the S3 includes:

[0037] S31. Use the spark-md5 algorithm to shard each adjusted data Perform MD5 checksum calculation to generate the md5 value of each data shard :

[0038] ;

[0039] in, Sharding data The MD5 checksum value, Function to calculate MD5 checksum for spark-md5 algorithm;

[0040] S32, md5 value of all data fragments Perform weighted hash aggregation to generate the overall md5 value H of the response file:

[0041] ;

[0042] Where K is the total number of data blocks, is the weight factor of data sharding, defined as:

[0043] ;

[0044] The total size of the response file after adjustment:

[0045] ;

[0046] S33, compare the generated overall md5 value H with the md5 value of each data fragment Attached to the corresponding data shard In the metadata, a data structure containing verification information is formed :

[0047] ;

[0048] The data structure containing the verification information is sent to the server. When the server receives it, it uses the md5 value of each data shard and the overall md5 value to verify the integrity of the data shard and the overall file.

[0049] Optionally, the S31 includes:

[0050] S311. For each adjusted data shard Perform block processing to divide the data into multiple byte blocks , each byte block size is B, where k represents the data fragment The k-th byte block in satisfies the following conditions:

[0051] ;

[0052] Among them, M represents data sharding The total number of byte blocks within;

[0053] S312: For each data shard , read and process bytes block by block using spark-md5 algorithm Generate intermediate hash values :

[0054] ;

[0055] in, is the MD5 result of the k-th byte block, Sharding data The hash gain factor represents the amplification effect of the spark-md5 algorithm on the hash value of the previous block. is the byte sequence offset value in the kth byte block, which is used to adjust the position influence weight of the current byte block in the MD5 calculation process. Adjust the length of the overall byte block sequence by a factor so that the hash result reflects the order and content changes of the byte blocks;

[0056] S313: After all byte blocks have been iteratively calculated, data fragments are generated. The complete MD5 checksum of :

[0057] ;

[0058] Among them, M is the data shard The total number of byte blocks within It is the final MD5 checksum of the shard, which is used to uniquely identify the data shard. Indicates the check weight coefficient of the data shard.

[0059] Optionally, the S4 includes:

[0060] S41. During the upload process, the client records each data segment in real time Upload status, generate upload progress information table , the upload progress information table indicates data sharding Upload status, a value of 1 indicates successful upload, and a value of 0 indicates that the upload has not been completed or has not been completed;

[0061] S42. When the upload is interrupted due to network interruption or other abnormal conditions, the client will And the md5 value of the uploaded data fragment Stored in local storage for use after reconnection;

[0062] S43. After reconnecting to the network, the client reads the upload progress information table stored locally. And the uploaded md5 value , and the local md5 value The md5 value set of the received data fragments on the server Compare and generate a set of data shards W that have not been uploaded:

[0063] ;

[0064] Where W represents the set of data shards that need to be retransmitted;

[0065] S44: For the data segment set W that has not been uploaded, the client continues to upload the corresponding data segment from the breakpoint After the multi-part upload is successful, update the upload progress information table The status of the corresponding shard in is 1;

[0066] S45. When all data are fragmented All uploaded successfully, meeting the conditions:

[0067] ;

[0068] The client sends a completion notification to the server, confirming that all data segments have been transferred and the breakpoint-resume transfer process is complete.

[0069] Optionally, the S6 includes:

[0070] S61. Decrypt the response file uploaded by the user to generate a complete decrypted file. ;

[0071] S62. The decrypted complete file Divided into several preview slice files Each preview slice file contains a number of file pages. The number of pages of each slice file is determined based on the single page content p of the file. :

[0072] ;

[0073] in, Indicates the content of the i-th slice file in the decrypted file, Indicates the average content of a single page of a document;

[0074] S63. Generate metadata for each preview slice file , metadata contains the number of pages in the slice file , page order information and the starting and ending page numbers of the slice file, so that each preview slice file can be loaded on demand;

[0075] S64. Each generated preview slice file and its corresponding metadata Stored on the server side, when the client requests a preview, the server dynamically provides the corresponding preview slice file based on the requested page range .

[0076] The beneficial effects of the present invention are:

[0077] (1) The present invention adopts an intelligent sharding algorithm to dynamically adjust the size of data shards by real-time monitoring of network conditions and server processing capabilities, so that each shard can adapt to the current network transmission rate and reduce resource waste. Compared with the fixed sharding method in the prior art, the intelligent sharding algorithm performs particularly well in an environment with large fluctuations in network speed, ensuring the stability and speed of file upload, avoiding waiting time during the upload process, improving the concurrent processing efficiency of shard transmission, and optimizing the transmission speed.

[0078] (2) The present invention uses the spark-md5 algorithm to calculate the md5 value of the response file and its fragments, thereby realizing efficient breakpoint resumption and instant transmission functions. By calculating a unique md5 value for each data fragment, the system can quickly locate the unfinished fragments and continue uploading after a network interruption, without having to re-upload all the data. At the same time, the instant transmission mechanism uses md5 value matching to directly skip existing files to avoid repeated uploading. Compared with traditional single-thread breakpoint resumption, the solution of the present invention not only significantly improves the speed of resuming uploads, but also greatly reduces the storage and transmission pressure on the server side.

[0079] (3) The present invention divides the file content into several preview slice files, and configures the number of pages of each slice based on the single-page content of the file so that the slice content is reasonable and suitable for preview. The server side only dynamically provides the corresponding slice when the client requests a preview and performs cache management on the loaded slice, avoiding loading the full file for each preview request, thereby significantly reducing system memory usage and server load. The slice preview mechanism effectively solves the jamming problem during the preview of large files and ensures the smoothness of the preview. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0081] Figure 1 The present invention provides a flowchart of a method for processing large-volume response files applied in a bidding process. DETAILED DESCRIPTION

[0082] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0083] refer to Figure 1 A method for processing a large volume response file applied in a bidding process comprises the following steps:

[0084] S1. Divide the response file to be uploaded into multiple data blocks, and perform data sharding on each data block, monitor the current network status and server processing capacity, and dynamically adjust the data size of each data shard;

[0085] S2. Create and manage a thread pool on the client based on the Web Worker API. By configuring an appropriate number of concurrent threads, each data shard is assigned to different threads for parallel upload.

[0086] S3. Use the spark-md5 algorithm to calculate the overall MD5 value of the response file and the MD5 value of each data shard, and send the MD5 value information together with the data shard data to the server to support breakpoint resume and instant upload functions;

[0087] S4. In the breakpoint resume scenario, based on the stored data shard md5 value and upload progress information, the unfinished data shard is determined according to the upload progress information and MD5 value comparison results, and the unfinished data shard is continued to be uploaded from the breakpoint;

[0088] S5. In the instant upload scenario, when the md5 value of the response file uploaded by the client is consistent with the md5 value of the file stored on the server, the upload success is directly fed back to the user without the need for repeated upload;

[0089] S6. For the response file uploaded by the user, after the response file is decrypted, the decrypted file is divided into several preview slice files, the number of pages of each preview slice file is configured according to the content of a single page of the file, and the preview slice files are stored on the server;

[0090] S7. During the bid evaluation process, the client requests the server to preview the slice file data as needed for the step-by-step preview display of the response file, and caches the loaded preview slice file data locally;

[0091] S8. According to the preview situation, the client dynamically releases the preview slice file data in the cache to control the occupancy of the local storage space and ensure smooth preview.

[0092] In this implementation, S1 includes:

[0093] S11, divide the response file to be uploaded into multiple data blocks, the size of each data block is defined as :

[0094] ;

[0095] Where i represents the serial number of the data block, and the size of the data block is defined based on the total size F of the response file and the expected number of uploaded shards N;

[0096] S12. For each data block Perform data sharding processing, divide the data block into multiple data shards, and set the size of each data shard to , where j is the data block The shard sequence number in the initial data shard size parameter , and dynamically adjust according to the real-time network status;

[0097] S13. Monitor the current network status, including network bandwidth W, network delay L, and server processing capacity S, and obtain the maximum transmission rate R of the network in real time by evaluating the fluctuation range of network bandwidth W, network delay L, and server processing capacity S:

[0098] ;

[0099] Dynamically adjust the size of data shards based on network conditions and server load , adjust the adjustment factor of the data shard size , adjustment coefficient It is derived from the ratio of the maximum transmission rate R and the server processing capacity S:

[0100] ;

[0101] Adjusted data shard size It is expressed as:

[0102] ;

[0103] in, The updated data shard size is adjusted to match the current network conditions and server processing capabilities.

[0104] In this implementation, S2 includes:

[0105] S21. Create a thread pool on the client based on the Web Worker API and set the maximum number of concurrent threads T in the thread pool;

[0106] S22: According to the number N of data fragments to be uploaded, each data fragment Distribute to different threads, and evenly distribute the number of data shards and the number of threads, so that the number of data shards allocated to each thread is ;

[0107] S23. Setting an upload task for each thread, the upload task includes the allocation of data shards, status tracking, and recording of upload progress, and independently managing the upload status of the task in each thread;

[0108] S24, during the upload process, the status of each thread is monitored in real time, and if a thread task is completed, the remaining unuploaded data slices are dynamically allocated to the thread;

[0109] S25: Based on the order in which the data segments are uploaded, the uploaded data segments are merged. Reassemble into complete adjusted data blocks :

[0110] ;

[0111] in, Represents the reorganized data block i, which consists of multiple data fragments composition, Indicates the total number of data fragments contained in data block i, is the transmission rate deviation during the upload process, which indicates the difference between the actual transmission rate and the ideal transmission rate. It is used to adjust the upload order of data shards. S is the server processing capacity, which is used as a reference value for adjusting the shard merging order. The weight coefficient is the order weight coefficient of the data shards. The weight coefficient ensures the order of the data shards when the data shards are merged to avoid fluctuations in the transmission rate. Caused by disorder of sequence.

[0112] In this implementation, S3 includes:

[0113] S31. Use the spark-md5 algorithm to shard each adjusted data Perform MD5 checksum calculation to generate the md5 value of each data shard :

[0114] ;

[0115] in, Sharding data The MD5 checksum value, Function to calculate MD5 checksum for spark-md5 algorithm;

[0116] S32, md5 value of all data fragments Perform weighted hash aggregation to generate the overall md5 value H of the response file:

[0117] ;

[0118] Where K is the total number of data blocks, is the weight factor of data sharding, defined as:

[0119] ;

[0120] The total size of the response file after adjustment:

[0121] ;

[0122] S33, compare the generated overall md5 value H with the md5 value of each data fragment Attached to the corresponding data shard In the metadata, a data structure containing verification information is formed :

[0123] ;

[0124] The data structure containing the verification information is sent to the server. When the server receives it, it uses the md5 value of each data shard and the overall md5 value to verify the integrity of the data shard and the overall file.

[0125] In this implementation, S31 includes:

[0126] S311. For each adjusted data shard Perform block processing to divide the data into multiple byte blocks , each byte block size is B, where k represents the data fragment The k-th byte block in satisfies the following conditions:

[0127] ;

[0128] Among them, M represents data sharding The total number of byte blocks within;

[0129] S312: For each data shard , read and process bytes block by block using spark-md5 algorithm Generate intermediate hash values :

[0130] ;

[0131] in, is the MD5 result of the k-th byte block, Sharding data The hash gain factor represents the amplification effect of the spark-md5 algorithm on the hash value of the previous block. is the byte sequence offset value in the kth byte block, which is used to adjust the position influence weight of the current byte block in the MD5 calculation process. Adjust the length of the overall byte block sequence by a factor so that the hash result reflects the order and content changes of the byte blocks;

[0132] S313: After all byte blocks have been iteratively calculated, data fragments are generated. The complete MD5 checksum of :

[0133] ;

[0134] Among them, M is the data shard The total number of byte blocks within It is the final MD5 checksum of the shard, which is used to uniquely identify the data shard. Indicates the check weight coefficient of the data shard.

[0135] In this implementation, S4 includes:

[0136] S41. During the upload process, the client records each data segment in real time Upload status, generate upload progress information table , the upload progress information table indicates data sharding Upload status, a value of 1 indicates successful upload, and a value of 0 indicates that the upload has not been completed or has not been completed;

[0137] S42. When the upload is interrupted due to network interruption or other abnormal conditions, the client will And the md5 value of the uploaded data fragment Stored in local storage for use after reconnection;

[0138] S43. After reconnecting to the network, the client reads the upload progress information table stored locally. And the uploaded md5 value , and the local md5 value The md5 value set of the received data fragments on the server Compare and generate a set of data shards W that have not been uploaded:

[0139] ;

[0140] Where W represents the set of data shards that need to be retransmitted;

[0141] S44: For the data segment set W that has not been uploaded, the client continues to upload the corresponding data segment from the breakpoint After the multi-part upload is successful, update the upload progress information table The status of the corresponding shard in is 1;

[0142] S45. When all data are fragmented All uploaded successfully, meeting the conditions:

[0143] ;

[0144] The client sends a completion notification to the server, confirming that all data segments have been transferred and the breakpoint-resume transfer process is complete.

[0145] In this implementation, S6 includes:

[0146] S61. Decrypt the response file uploaded by the user to generate a complete decrypted file. ;

[0147] S62. The decrypted complete file Divided into several preview slice files Each preview slice file contains a number of file pages. The number of pages of each slice file is determined based on the single page content p of the file. :

[0148] ;

[0149] in, Indicates the content of the i-th slice file in the decrypted file, Indicates the average content of a single page of a document;

[0150] S63. Generate metadata for each preview slice file , metadata contains the number of pages in the slice file , page order information and the starting and ending page numbers of the slice file, so that each preview slice file can be loaded on demand;

[0151] S64. Each generated preview slice file and its corresponding metadata Stored on the server side, when the client requests a preview, the server dynamically provides the corresponding preview slice file based on the requested page range .

[0152] Embodiment 1:

[0153] In modern bidding activities, participating companies need to upload large response files, which include project plans, technical parameters, and qualification certificates. Traditional bidding and procurement systems often encounter problems such as slow upload speed, unstable breakpoint resume, and preview freeze when processing these large response files. In this embodiment 1, we designed a simulation of the entire bidding document processing process based on the method of the present invention to demonstrate the feasibility and advantages of the present invention in improving upload speed, ensuring breakpoint resume effect, and optimizing preview performance.

[0154] The simulation scenario took place in a public resource trading center in a certain province. The bidding units included Company A, Company B, and Company C. The response files of each unit were large, about 2GB on average. The response files of each company included PDF files, drawings, video presentations, and data tables in multiple formats. The target server for file upload was configured at a medium level and experienced more network fluctuations during rush hour.

[0155] During the bidding process, Company A first logged into the bidding system and started uploading a total of 2GB of response files. In traditional systems, files are fixedly segmented and uploaded in a single-threaded manner, with 1MB of files uploaded each time. When network fluctuations occur, they may need to be re-uploaded. This method causes the upload process to take a long time, especially when the network fluctuates frequently, and more content often needs to be re-uploaded due to failure of breakpoint resumption.

[0156] Based on the method of the present invention, the system first intelligently slices the response file of Company A, and dynamically adjusts the slice size by real-time monitoring the network bandwidth and server load during the file upload process. When the network bandwidth is stable, the system automatically sets the slice size to 5MB, and when the network fluctuates greatly, the slice size is reduced to 1MB to adapt to the current network conditions. In this way, the slice size of Company A's response file always matches the transmission rate, thereby reducing the upload delay and resource waste caused by unstable network speed.

[0157] During the upload process, the system also creates a thread pool containing 8 threads through the Web Worker API to achieve multi-threaded concurrent upload. Each thread is responsible for the upload task of a shard, and the data integrity of each shard is ensured by MD5 verification. In the case of network interruption, the system uses the spark-md5 algorithm to calculate a unique md5 value for each shard, and stores the upload status and md5 value of the shard in the local device. After the network is restored, the system automatically compares the server-side and local md5 values ​​to find the unfinished uploaded shards and continue uploading from the breakpoint. In this way, the present invention effectively solves the re-upload problem caused by network fluctuations in the traditional method.

[0158] In Example 1, Company A was interrupted five times due to network fluctuations when uploading files. In the traditional method, the breakpoint resumption of fixed-size fragments often cannot accurately record the upload progress, resulting in Company A having to re-upload about 600MB of data after the network is restored. The method of the present invention only needs to upload the remaining 50MB of data through md5 comparison, which greatly shortens the upload time. In this simulation, Company A took a total of about 12 minutes to upload the file, while the traditional method took 32 minutes.

[0159] After the file is uploaded, the system previews and slices Company A's response file, dividing the 2GB file into approximately 400 slice files. Each slice contains 3-6 pages of content and generates metadata for each slice file, including the number of slice pages and page number information. When the bid evaluation begins, the system gradually loads the corresponding slice files according to the evaluators' preview requests. When the evaluators choose to view the project plan section, the system first loads the slice file of that part of the content for initial rendering. At the same time, the system temporarily stores the slice file in the cache area, and can directly read the cached data for subsequent requests to avoid repeated loading. In traditional methods, the system often needs to load the entire file at one time. This overall loading method often causes jamming and delays when processing large files, affecting the bid evaluation experience.

[0160] During the bid evaluation process, Company A and Company B respectively adopted the present invention and the traditional method in their bid evaluation stages. The average file preview time for Company A was 1.8 seconds per page, while the file preview time for Company B using the traditional method was about 4.3 seconds, and there would be short freezes during multiple loadings. During the entire bid evaluation process, the file preview operation of Company A using the present invention was smooth, and the judges could quickly switch pages and content. However, the file preview of Company B using the traditional method had the problems of slow loading and low cache utilization, and the preview efficiency and fluency were significantly inferior to those of the present invention.

[0161] In this embodiment, we recorded various data indicators during the file processing of Company A and Company B respectively to further verify the effectiveness of the present invention:

[0162] Table 1 Comparison of large response file processing methods

[0163]

[0164] The present invention realizes intelligent slicing adjustment, multi-threaded concurrent uploading and breakpoint resume during uploading, which significantly shortens the time spent on file uploading. When the network fluctuates five times, the amount of data that Company A needs to retransmit is only 50MB, while Company B needs to retransmit 600MB of data, which further demonstrates the efficiency improvement of the present invention in breakpoint resume. In the file preview stage, through the preview slicing mechanism and cache optimization, Company A only needs 1.8 seconds to preview and load each page, so the judges can smoothly view the file contents. When Company B adopts the traditional method, due to the long loading time and jamming, the overall evaluation time is about 35 minutes longer than that of Company A.

[0165] To sum up, this embodiment 1 demonstrates in detail the significant advantages of the present invention in the upload and preview links through the entire process of response file processing of Company A in the bidding process. It not only greatly improves the speed of file upload and the stability of breakpoint resumption, but also effectively optimizes the smoothness of the preview, which is particularly suitable for bidding scenarios with large-volume response files.

[0166] The present invention adopts an intelligent sharding algorithm, and dynamically adjusts the size of data shards by real-time monitoring of network conditions and server processing capabilities, so that each shard can adapt to the current network transmission rate and reduce resource waste. Compared with the fixed sharding method in the prior art, the intelligent sharding algorithm performs particularly well in an environment with large fluctuations in network speed, ensuring the stability and speed of file uploading, avoiding waiting time during the uploading process, improving the concurrent processing efficiency of shard transmission, and realizing the optimization of transmission speed.

[0167] The present invention uses the spark-md5 algorithm to calculate the md5 value of the response file and its fragments, thereby realizing efficient breakpoint resume and second transmission functions. By calculating a unique md5 value for each data fragment, the system can quickly locate the unfinished fragment after a network interruption and continue uploading without re-uploading all the data. At the same time, the second transmission mechanism directly skips existing files by using md5 value matching to avoid repeated uploading. Compared with traditional single-thread breakpoint resume, the solution of the present invention not only significantly improves the speed of resuming upload, but also greatly reduces the storage and transmission pressure on the server side.

[0168] The present invention divides the file content into several preview slice files, and configures the number of pages of each slice based on the single-page content of the file to make the slice content reasonable and suitable for preview. The server side dynamically provides the corresponding slice only when the client requests a preview and performs cache management on the loaded slices, avoiding loading the full file for each preview request, thereby significantly reducing system memory usage and server load. The slice preview mechanism effectively solves the jamming problem in the preview process of large files and ensures the smoothness of the preview.

[0169] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for processing large-volume response files used in a bidding process, characterized in that: The steps include: S1. Divide the response file to be uploaded into multiple data blocks, and perform data sharding processing on each data block; S2. Create and manage a thread pool on the client based on the Web Worker API, and assign each data shard to different threads for parallel upload; S3. Use the spark-md5 algorithm to calculate the overall md5 value of the response file and the md5 value of each data shard; S4. In the breakpoint resume scenario, based on the stored data fragment md5 value and upload progress information, the unfinished data fragment is determined according to the upload progress information and MD5 value comparison results, and the unfinished data fragment is continued to be uploaded from the breakpoint; S5. In the instant upload scenario, when the md5 value of the response file uploaded by the client is consistent with the md5 value of the file stored on the server, the upload success is directly fed back to the user; S6. For the response file uploaded by the user, after the response file is decrypted, the decrypted file is divided into several preview slice files; S7. During the bid evaluation process, the client requests the server to preview the slice file data as needed for the step-by-step preview display of the response file, and caches the loaded preview slice file data locally; The S1 includes: S11, divide the response file to be uploaded into multiple data blocks, the size of each data block is defined as B i : Where i represents the serial number of the data block, and the size of the data block is defined based on the total size F of the response file and the expected number of uploaded shards N; S12, for each data block B i Perform data sharding processing, divide the data block into multiple data shards, and set the size of each data shard to P i,j , where j is the data block B i The shard sequence number in the data, the initial data shard size parameter P0, and dynamically adjust according to the real-time network status; S13. Monitor the current network status, including network bandwidth W, network delay L, and server processing capacity S, and obtain the maximum transmission rate R of the network in real time by evaluating the fluctuation range of network bandwidth W, network delay L, and server processing capacity S: Dynamically adjust the size of data shards P based on network conditions and server load i,j , adjust the adjustment coefficient α of the data shard size. The adjustment coefficient α is derived from the ratio of the maximum transmission rate R and the server processing capacity S: Adjusted data shard size P′ i,j It is expressed as: P′ i,j =P i,j ×α; Among them, P′ i,j The updated data shard size is adjusted to match the current network status and server processing capacity; The S2 includes: S21. Create a thread pool on the client based on the Web Worker API and set the maximum number of concurrent threads T in the thread pool; S22, according to the number of data fragments to be uploaded N, each data fragment P' i,j Distribute to different threads, and evenly distribute the number of data shards and the number of threads, so that the number of data shards allocated to each thread is S23. Setting an upload task for each thread, the upload task includes the allocation of data shards, status tracking, and recording of upload progress, and independently managing the upload status of the task in each thread; S24, during the upload process, the status of each thread is monitored in real time, and if a thread task is completed, the remaining unuploaded data slices are dynamically allocated to the thread; S25, based on the order in which the data segments are uploaded, merge the uploaded data segments data P′ i,j Reorganize into a complete adjusted data block B′ i : Among them, B′ i Represents the reorganized data block i, which consists of multiple data fragments P′ i,j Composition, M i Indicates the total number of data fragments contained in data block i, ΔR j is the transmission rate deviation during the upload process, which indicates the difference between the actual transmission rate and the ideal transmission rate. It is used to adjust the upload order of data fragments. S is the server processing capacity, which is used as a reference value for adjusting the fragment merging order. i,j is the weight coefficient of the data shard order. The weight coefficient ensures the order of data shards when merging data shards to avoid transmission rate fluctuations ΔR j Caused by disorder of sequence.

2. According to claim 1, a method for processing large-volume response files applied to a bidding process is characterized in that: The S3 includes: S31. Use the spark-md5 algorithm to perform the calculation on each adjusted data shard P′ i,j Perform MD5 checksum calculation to generate the md5 value h for each data fragment i,j : h i,j =MD5 spark (P′ i,j ); Among them, h i,j P′ is the data segment i,j MD5 checksum value, MD5 spark Function to calculate MD5 checksum for spark-md5 algorithm; S32, md5 value h for all data fragments i,j Perform weighted hash aggregation to generate the overall md5 value H of the response file: Where K is the total number of data blocks, φ i,j is the weight factor of data sharding, defined as: F' is the total size of the response file after adjustment: S33, compare the generated overall md5 value H with the md5 value h of each data fragment i,j Attached to the corresponding data fragment P′ i,j In the metadata, a data structure D containing verification information is formed i,j : D i,j ={P′ i,j ,h i,j ,H}; The data structure containing the verification information is sent to the server. When the server receives it, it uses the md5 value of each data shard and the overall md5 value to verify the integrity of the data shard and the overall file.

3. The method for processing large-volume response files in a bidding process according to claim 2 is characterized in that: The S31 includes: S311, for each adjusted data slice P′ i,j Perform block processing and divide the data into multiple byte blocks b i,j,k , each byte block size is B, where k represents the data fragment P′ i,j The k-th byte block in satisfies the following conditions: Where M represents the data fragment P′ i,j The total number of byte blocks within; S312: For each data fragment P′ i,j , read and process byte block b block by block using spark-md5 algorithm i,j,k Generate an intermediate hash value H i,j,k : Among them, H i,j,k is the MD5 result of the k-th byte block, λ i,j P′ is the data segment i,j The hash gain factor represents the amplification effect of the spark-md5 algorithm on the hash value of the previous block, δ k is the byte sequence offset value in the kth byte block, which is used to adjust the position influence weight of the current byte block in the MD5 calculation process. ΔB is the overall byte block sequence length adjustment factor, so that the hash result reflects the order and content changes of the byte blocks; S313: After all byte blocks have been iteratively calculated, a data fragment P′ is generated. i,j The complete MD5 checksum h i,j : Where M is the data fragment P′ i,j The total number of byte blocks in h i,j is the final MD5 checksum of the shard, which is used to uniquely identify the data shard. i,j Indicates the check weight coefficient of the data shard.

4. The method for processing large-volume response files in a bidding process according to claim 1 is characterized in that: The S4 includes: S41. During the upload process, the client records each data segment P′ in real time i,j Upload status, generate upload progress information table U i,j , the upload progress information table represents the data segment P′ i,j Upload status, a value of 1 indicates successful upload, and a value of 0 indicates that the upload has not been completed or has not been completed; S42: When the upload is interrupted due to network interruption or other abnormal conditions, the client sends the current upload progress information to the table U. i,j And the uploaded data fragment md5 value h i,j Stored in local storage for use after reconnection; S43. After reconnecting to the network, the client reads the upload progress information table U stored locally. i,j And the uploaded md5 value h i,j , and the local md5 value h i,j The md5 value set h′ of the received data fragments on the server i,j Compare and generate a set of data shards W that have not been uploaded: Where W represents the set of data shards that need to be retransmitted; S44: For the data segment set W that has not been uploaded, the client continues to upload the corresponding data segment P′ from the breakpoint i,j After the segment upload is successful, update the upload progress information table U i,j The status of the corresponding shard in is 1; S45. When all data fragments P′ i,j All uploaded successfully, meeting the conditions: The client sends a completion notification to the server, confirming that all data segments have been transferred and the breakpoint-resume transfer process is complete.

5. The method for processing large-volume response files in a bidding process according to claim 1 is characterized in that: The S6 includes: S61, decrypt the response file uploaded by the user to generate a decrypted complete file F dec ; S62, the decrypted complete file F dec Divided into several preview slice files C i Each preview slice file contains a number of file pages. The number of pages N of each slice file is determined based on the single page content p of the file. i : Among them, L i Indicates the content of the i-th slice file in the decrypted file, p avg Indicates the average content of a single page of a document; S63. Generate metadata for each preview slice file The metadata contains the number of pages N of the slice file i , page order information S i,j and the starting and ending page numbers of the slice file, so that each preview slice file can be loaded on demand; S64, each generated preview slice file C i and its corresponding metadata Stored on the server side, when the client requests a preview, the server dynamically provides the corresponding preview slice file C according to the requested page range i .

Images