Asynchronous file generation and downloading implementation method and system

Abstract

The invention provides an asynchronous file generation and downloading realization method and system, and relates to the technical field of computer file processing, and the method comprises the steps: receiving a user file downloading request, creating a task, and generating a unique identifier; dividing the task into a plurality of sub-task segments based on a file feature recognition rule and allocating resources; distributing the subtasks to a plurality of processing nodes to generate subfiles in parallel; merging the sub-files into a target downloading file and uploading the target downloading file to a server cluster; and returning the file in response to user query. The large file downloading efficiency is improved, the system stability is enhanced, and the user experience is optimized.

Description

Technical Field

[0001] This invention relates to the field of computer file processing technology, and in particular to a method and system for asynchronous file generation and download. Background Technology

[0002] In enterprise information systems, file download functionality is a common and fundamental requirement, especially in applications such as data analysis and report export, where users frequently need to export large amounts of data as files for download. Traditional file download implementations typically employ a synchronous processing approach, meaning that when a user initiates a download request, the system immediately performs data querying, file generation, and transfer operations, and the user must wait for the entire process to complete before obtaining the file.

[0003] Existing technologies lack effective mechanisms for tracking file generation status and resuming tasks. If an error or failure occurs during file generation, the entire task needs to be re-executed, wasting system resources and prolonging user waiting time.

[0004] In recent years, although some systems have begun to adopt asynchronous processing to improve file download functionality, problems such as inflexible task scheduling, unreasonable resource allocation, and imperfect subtask failure handling mechanisms still exist. In complex business scenarios, especially when dealing with file generation tasks involving extremely large data volumes or multiple data source integration, existing technologies struggle to meet the requirements of high efficiency and high reliability. Summary of the Invention

[0005] The embodiments of the present invention provide an asynchronous file generation and download implementation method and system, which can solve the problems in the prior art.

[0006] A first aspect of the present invention provides an asynchronous file generation and download implementation method, comprising:

[0007] Receive a user's file download request, extract the data source code and create a file download task, write the file download task into the database and generate a globally unique task identifier;

[0008] The file download task is analyzed based on preset file feature recognition rules. According to the analysis results, the file download task is dynamically divided into multiple sub-task segments, and each sub-task segment is allocated an independent resource quota and processing timeout threshold.

[0009] The multiple subtask fragments are assigned to multiple target processing nodes, and a parallel file generation method is used to generate sub-files. When the generation of a sub-file fails, the corresponding sub-file generation task is redistributed to other file processing nodes. All sub-files are merged into a target download file according to a preset merging strategy, and the file integrity is checked. The globally unique task identifier is marked as completed.

[0010] The target download file is uploaded to the file server cluster, and a file status monitoring table is established. In response to the user's file list query request, the target download file is retrieved from the file server cluster and returned.

[0011] The system receives a user's file download request, extracts the data source code, creates a file download task, writes the file download task to the database, and generates a globally unique task identifier, including:

[0012] Receive the user's file download request, extract the data source type identifier, data source number, timestamp information and digital signature from it, obtain the data source code and create a file download task;

[0013] Based on the data source encoding, combined with the user level and task queue length, the priority score of the file download task is calculated. The file download task is written into the two-layer task processing structure according to the priority score. When the number of tasks in the two-layer task processing structure reaches the upper limit, an asynchronous thread is started to write the file download tasks with priority scores greater than the priority threshold into the database in batches.

[0014] Based on the database write timestamp, the priority score, and the data source code, a globally unique task identifier is generated for the file download task, and a cache expiration time is set.

[0015] The file download task is analyzed based on preset file feature recognition rules. According to the analysis results, the file download task is dynamically divided into multiple sub-task segments, and each sub-task segment is allocated an independent resource quota and processing timeout threshold, including:

[0016] Extract file data from the file download task, calculate the entropy value, data distribution density, and time series correlation of the file data, and construct a file feature vector;

[0017] The correlation degree of data blocks is calculated based on the file feature vector, a data block dependency graph is constructed, and the ratio of the in-degree value to the out-degree value of each data block node is calculated. Data block nodes with a ratio lower than the dependency threshold are used as initial splitting nodes.

[0018] Calculate the data transmission cost between the initial segmentation nodes, select the initial segmentation node whose data transmission cost is less than the cost threshold as the optimal segmentation node, and dynamically segment the file download task based on the optimal segmentation node to generate multiple subtask segments.

[0019] The file feature vector is matched with the features of historical execution tasks. Based on the similarity matching results, an independent processing timeout threshold is set for each subtask segment, and an independent resource quota is set in combination with its data complexity.

[0020] The multiple subtask fragments are assigned to multiple target processing nodes, and a parallel file generation method is used to generate subfiles. When a subfile generation failure is detected, the corresponding subfile generation task is redistributed to other file processing nodes, including:

[0021] Obtain the total resources, network bandwidth capacity and current load status of each processing node, filter out candidate processing nodes that meet the resource quota requirements of subtask segments, calculate the task affinity score of each candidate processing node, construct a priority sequence of candidate processing nodes in descending order of task affinity score, select multiple candidate processing nodes as target processing nodes, and allocate the multiple subtask segments to the target processing nodes.

[0022] Start the file generation process on the target processing node, initialize the file write handle and file checksum, write the data of the subtask fragments into the file according to the data block size, calculate the checksum of each data block and compare it with the file checksum, and rewrite the data block when a data block checksum mismatch is detected. After the sub-file generation is completed, update the file metadata information.

[0023] Monitor the processing progress of subtask segments on the target processing node. When it is detected that the file metadata information of the subtask segment has not changed and the processing time has exceeded the processing timeout threshold of each subtask segment, reselect the next processing node from the priority sequence of candidate processing nodes for task allocation.

[0024] All sub-files are merged into the target download file according to a preset merging strategy, and the file integrity is checked. The globally unique task identifier is marked as completed, including:

[0025] Obtain the file metadata information of all sub-files, and establish a sub-file merging sequence according to the numbering order of the file metadata information;

[0026] The data blocks of the sub-files are read according to the sub-file merging sequence and written into the file merging buffer for data merging. The data merging rate is controlled by the token bucket algorithm. The capacity of the token bucket is dynamically adjusted according to the system disk write bandwidth. When the number of tokens in the token bucket is insufficient, the data merging operation is paused and the read rate of the upstream sub-files is reduced to balance the data processing pipeline.

[0027] The merged data is verified in real time using a sliding window mechanism. The cyclic redundancy check code of the data block is calculated within the sliding window. The calculated cyclic redundancy check code is compared with the original check code of the sub-file. When a check code mismatch is detected, the sliding window is rolled back and the data is merged again.

[0028] The data blocks that have passed verification are read from the merge buffer and written into the target download file. The integrity of the target download file is checked to obtain the verified target download file, and the globally unique task identifier is marked as completed.

[0029] Upload the target download file to the file server cluster and establish a file status monitoring table. In response to a user's file list query request, retrieve and return the target download file from the file server cluster, including:

[0030] The target download file is uploaded to the file server cluster, the storage location information of the target download file in the file server cluster is obtained, and the corresponding file address is generated.

[0031] Establish a file status monitoring table to record the file status, file address, and number of file accesses of the target download file. When the number of file accesses exceeds the access threshold, mark the target download file as a hot file and trigger the file preloading mechanism.

[0032] The system receives a file list query request from a user, obtains a file status list based on the file status monitoring table, prioritizes displaying popular files, retrieves the target download file from the server cluster, and returns it to the user.

[0033] A second aspect of the present invention provides an asynchronous file generation and download implementation system, comprising:

[0034] The first unit is used to receive the user's file download request, extract the data source code and create a file download task, write the file download task into the database and generate a globally unique task identifier;

[0035] The second unit is used to analyze the file download task based on preset file feature recognition rules, dynamically divide the file download task into multiple sub-task segments according to the analysis results, and allocate independent resource quotas and processing timeout thresholds to each sub-task segment.

[0036] The third unit is used to distribute the multiple sub-task fragments to multiple target processing nodes, generate sub-files using a parallel file generation method, and redistribute the corresponding sub-file generation task to other file processing nodes when the sub-file generation fails. All sub-files are merged into a target download file according to a preset merging strategy, and the file integrity is checked. The globally unique task identifier is marked as completed.

[0037] The fourth unit is used to upload the target download file to the file server cluster, establish a file status monitoring table, and in response to the user's file list query request, retrieve and return the target download file from the file server cluster.

[0038] A third aspect of the present invention,

[0039] An electronic device is provided, comprising:

[0040] processor;

[0041] Memory used to store processor-executable instructions;

[0042] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0043] Fourth aspect of the embodiments of the present invention,

[0044] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0045] The beneficial effects of this application are as follows:

[0046] By implementing the asynchronous file generation and download method of the present invention, the file download task can be dynamically divided into multiple sub-task segments for parallel processing, which effectively improves the processing efficiency of large file generation and download, shortens user waiting time, and improves system response speed.

[0047] This invention employs an independent resource quota and timeout threshold control mechanism, combined with an automatic redistribution strategy for failed subtasks, which significantly enhances the system's stability and fault tolerance in high-concurrency scenarios, effectively avoids overall download failures caused by single points of failure, and ensures the reliability of the file generation process.

[0048] Furthermore, this invention enables users to query file processing progress and obtain results at any time through file server cluster storage and file status monitoring table management, significantly improving the user experience. It also facilitates system administrators in monitoring and managing the file processing process, thereby enhancing system maintainability. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the asynchronous file generation and download implementation method according to an embodiment of the present invention;

[0050] Figure 2 This is a flowchart illustrating the file download request processing method. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0053] Figure 1 This is a flowchart illustrating the asynchronous file generation and download implementation method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0054] Receive a user's file download request, extract the data source code and create a file download task, write the file download task into the database and generate a globally unique task identifier;

[0055] The file download task is analyzed based on preset file feature recognition rules. According to the analysis results, the file download task is dynamically divided into multiple sub-task segments, and each sub-task segment is allocated an independent resource quota and processing timeout threshold.

[0056] The multiple subtask fragments are assigned to multiple target processing nodes, and a parallel file generation method is used to generate sub-files. When the generation of a sub-file fails, the corresponding sub-file generation task is redistributed to other file processing nodes. All sub-files are merged into a target download file according to a preset merging strategy, and the file integrity is checked. The globally unique task identifier is marked as completed.

[0057] The target download file is uploaded to the file server cluster, and a file status monitoring table is established. In response to the user's file list query request, the target download file is retrieved from the file server cluster and returned.

[0058] Figure 2 This is a flowchart illustrating a file download request processing method. In one optional implementation, the method includes receiving a user's file download request, extracting the data source code and creating a file download task, writing the file download task into a database, and generating a globally unique task identifier, including:

[0059] Receive the user's file download request, extract the data source type identifier, data source number, timestamp information and digital signature from it, obtain the data source code and create a file download task;

[0060] Based on the data source encoding, combined with the user level and task queue length, the priority score of the file download task is calculated. The file download task is written into the two-layer task processing structure according to the priority score. When the number of tasks in the two-layer task processing structure reaches the upper limit, an asynchronous thread is started to write the file download tasks with priority scores greater than the priority threshold into the database in batches.

[0061] Based on the database write timestamp, the priority score, and the data source code, a globally unique task identifier is generated for the file download task, and a cache expiration time is set.

[0062] This embodiment describes in detail a method for receiving user file download requests, extracting data source encoding, and creating file download tasks. This method ensures the rational allocation of system resources and the priority of task processing by creating an efficient task processing mechanism.

[0063] In this specific embodiment, a file download request sent by a user needs to be received. This request is typically transmitted over the network in the form of HTTP or HTTPS protocol. The request contains several key pieces of information. The data source type identifier is extracted from the request body. This identifier is usually in string form, such as "DB" for database type source, "API" for interface type source, and "FILE" for file system type source. At the same time, the data source number is extracted. This number is usually a unique string, in the format "SRC20240501001", used to accurately locate the specific data source. The timestamp information in the request is recorded in UTC format, such as "2024-05-01T12:30:45Z", used to prevent replay attacks. The request also contains a digital signature, an encrypted string generated using the SHA256+RSA algorithm, such as "a1b2c3d4e5f6...", used to verify the legality and integrity of the request. By combining the data source type identifier and the data source number, a complete data source code is generated, such as "DB_SRC20240501001", and a new file download task object is created based on this.

[0064] After a file download task is created, its priority score is calculated to allocate system resources appropriately. Priority calculation considers three main factors: data source encoding, user level, and current task queue length. For data source encoding, a base score is assigned based on the type: 10 points for database type, 8 points for API type, and 6 points for file system type. User levels are typically divided into VIP users, regular users, and trial users, with weights of 1.5, 1.0, and 0.8 respectively. When the task queue length is less than 100, the score is not adjusted; when the length is between 100 and 500, the priority is reduced by 10%; when the length exceeds 500, the priority is reduced by 20%. These factors are considered together to calculate the final priority score. For example, if a VIP user requests to download a database source and the current queue length is 80, the priority score would be 10 × 1.5 × 1.0 = 15 points.

[0065] After calculating the priority score, the file download task is written into a two-layer task processing structure, which includes a high-priority task queue and a low-priority task queue in memory. When the priority score exceeds 12 points, the task enters the high-priority queue; otherwise, it enters the low-priority queue. Each queue is organized using a priority heap structure to ensure that high-scoring tasks are processed first. The number of tasks in the two-layer task processing structure is continuously monitored. When the total number of tasks reaches a preset limit (usually 1000), an asynchronous thread is started to process them. This thread selects tasks with priority scores greater than the priority threshold (usually set to 10 points), groups them into batches (up to 100 tasks per batch), and calls the database batch write interface to persist these tasks. The database write uses a transaction mechanism to ensure the atomicity of the batch write, i.e., all succeed or all fail.

[0066] After a successful database write, a globally unique task ID needs to be generated for each task. The generation process considers three factors: the database write timestamp, the task priority score, and the data source encoding. In the specific implementation, a timestamp accurate to milliseconds needs to be obtained, such as 1588763142358; the priority score is converted into a two-digit fixed-length string, such as priority 15.5 being converted to "16"; the last 6 characters are extracted from the data source encoding (e.g., "01001" is extracted from "DB_SRC20240501001"); these three parts are concatenated using a specific delimiter, and a 4-digit random alphanumeric string is added as a suffix to form the final task ID, for example: "1588763142358_16_01001_A7B9". This combination method ensures the global uniqueness of the ID while containing the key information of the task.

[0067] After generating a task identifier, the task information is written to the caching system to improve subsequent query efficiency. The cache record contains key information such as task identifier, data source code, priority score, and creation time. Different cache expiration times are set according to task priority: 30 minutes for high-priority tasks (score > 15); 20 minutes for medium-priority tasks (10-15 points); and 10 minutes for low-priority tasks (< 10 points). After the cache expires, the task information will be reloaded from the database to ensure data consistency.

[0068] The system returns a response to the user containing a globally unique task identifier. The user can use this identifier to query the task status. The response body is in JSON format and contains information such as the task identifier, estimated processing time, and task status. At the same time, it records a detailed operation log, including information such as user ID, operation time, request IP, and task identifier, for subsequent auditing and troubleshooting.

[0069] In one optional implementation, the file download task is analyzed based on preset file feature recognition rules. According to the analysis results, the file download task is dynamically divided into multiple sub-task segments, and each sub-task segment is allocated an independent resource quota and processing timeout threshold, including:

[0070] Extract file data from the file download task, calculate the entropy value, data distribution density, and time series correlation of the file data, and construct a file feature vector;

[0071] The correlation degree of data blocks is calculated based on the file feature vector, a data block dependency graph is constructed, and the ratio of the in-degree value to the out-degree value of each data block node is calculated. Data block nodes with a ratio lower than the dependency threshold are used as initial splitting nodes.

[0072] Calculate the data transmission cost between the initial segmentation nodes, select the initial segmentation node whose data transmission cost is less than the cost threshold as the optimal segmentation node, and dynamically segment the file download task based on the optimal segmentation node to generate multiple subtask segments.

[0073] The file feature vector is matched with the features of historical execution tasks. Based on the similarity matching results, an independent processing timeout threshold is set for each subtask segment, and an independent resource quota is set in combination with its data complexity.

[0074] In practice, the process involves receiving file download tasks, which include basic file information such as URL and file size. File data samples are extracted from these tasks, typically the first 1MB of data or uniformly sampled data segments. For the extracted file data, entropy is calculated to reflect the randomness and information density of the data. Entropy is calculated by counting the frequency of each byte in the data; high entropy values ​​(e.g., >7.5) usually indicate that the data is compressed or encrypted. Simultaneously, data distribution density is calculated, i.e., the distribution of data values ​​within a range. This can be achieved by constructing a histogram, dividing the byte value range of 0-255 into 16 intervals and counting the number of data points in each interval. Temporal correlation is also analyzed, i.e., the degree of association between adjacent data blocks. This can be quantified by calculating the correlation coefficient between adjacent data blocks; a correlation coefficient close to 1 indicates high correlation, while a coefficient close to 0 indicates almost no correlation. Through these calculations, a file feature vector is constructed, containing entropy, data distribution density feature vector, and temporal correlation index.

[0075] Based on the constructed file feature vector, the correlation degree between each data block within the file is calculated. The file is divided into basic data blocks of equal size, such as 64KB each. Based on the temporal correlation index in the feature vector, the correlation degree between blocks is calculated. For text files, the correlation degree between adjacent blocks is high (e.g., 0.8), while for different content areas in multimedia files, the correlation degree is low (e.g., 0.3). Using these correlation values, a data block dependency graph is constructed. In the graph, nodes represent data blocks, edges represent the correlation between data blocks, and the weight of the edge represents the correlation strength.

[0076] After the dependency graph is constructed, the ratio of the in-degree to the out-degree of each data block node is calculated. The in-degree value represents the number of edges pointing to that node, and the out-degree value represents the number of edges originating from that node. Nodes with a ratio lower than a preset dependency threshold (e.g., 0.5) are marked as initial splitting nodes. These nodes are usually located where the data content changes significantly and are suitable as task splitting nodes. For example, a 100MB video file has two nodes with low ratios at 33MB and 67MB, with ratios of 0.3 and 0.4 respectively. These locations correspond to scene transitions or changes in content type.

[0077] Further calculations are made regarding the data transmission cost between the initial partitioning nodes, which is the resource consumption required to transmit data from one node to another. This cost can be calculated based on the amount of data between nodes, network conditions, and data characteristics. For example, for two initial partitioning nodes 20MB apart, if the entropy value of the intermediate data is high, it indicates high data complexity, and the transmission cost reaches a high value, such as 85. Conversely, for data regions with lower entropy values, even if they are 30MB apart, the transmission cost is only 50. The initial partitioning node with a data transmission cost less than a preset cost threshold is selected as the optimal partitioning node.

[0078] Based on the determined optimal segmentation node, the file download task is dynamically divided into multiple sub-task segments. Each sub-task segment corresponds to a continuous data region of the original file. For example, a 100MB file is divided into three sub-tasks: 0-33MB, 33-67MB, and 67-100MB. This segmentation method takes into account the internal structure of the data, making the data correlation within each sub-task strong and the dependency between sub-tasks weak.

[0079] The constructed file feature vector is matched with the feature vectors of historical tasks based on similarity. Similarity calculation can use methods such as cosine similarity or Euclidean distance; a higher value indicates greater similarity in task characteristics. For example, if the current file feature vector has a similarity of 0.92 with a ZIP file in the historical record, it indicates that the two files have similar processing characteristics. Based on the similarity matching results, an independent processing timeout threshold is set for each subtask segment. For subtasks highly similar to historical tasks (e.g., similarity > 0.9), the average processing time of the historical task plus a 20% tolerance is directly used as the timeout threshold. For subtasks with moderate similarity (0.6-0.9), 150% of the historical time can be used as the threshold. For subtasks with low similarity (< 0.6), a more lenient default threshold is set, such as the subtask size divided by the minimum expected download rate.

[0080] Meanwhile, independent resource quotas are set based on the data complexity of subtask segments. Data complexity can be measured by the entropy value and distribution density in the feature vector. Subtasks with high complexity (such as entropy value > 7.0) require more processing resources, and are allocated 50% more memory and CPU time; while subtasks with low complexity (such as entropy value < 5.0) can use the standard quota. For example, for a download task containing three subtasks, the first subtask is allocated 2 processing threads and 100MB of memory, with a timeout threshold of 30 seconds; the second subtask is allocated 4 threads and 200MB of memory, with a timeout threshold of 60 seconds; and the third subtask is allocated 6 threads and 400MB of memory, with a timeout threshold of 120 seconds.

[0081] This dynamic fragmentation and resource allocation mechanism can optimize the download process based on file characteristics, improve download success rate and efficiency, and make reasonable use of system resources.

[0082] In one optional implementation, the plurality of subtask fragments are distributed to multiple target processing nodes, and a parallel file generation method is used to generate subfiles. When a subfile generation failure is detected, the corresponding subfile generation task is redistributed to other file processing nodes, including:

[0083] Obtain the total resources, network bandwidth capacity and current load status of each processing node, filter out candidate processing nodes that meet the resource quota requirements of subtask segments, calculate the task affinity score of each candidate processing node, construct a priority sequence of candidate processing nodes in descending order of task affinity score, select multiple candidate processing nodes as target processing nodes, and allocate the multiple subtask segments to the target processing nodes.

[0084] Start the file generation process on the target processing node, initialize the file write handle and file checksum, write the data of the subtask fragments into the file according to the data block size, calculate the checksum of each data block and compare it with the file checksum, and rewrite the data block when a data block checksum mismatch is detected. After the sub-file generation is completed, update the file metadata information.

[0085] Monitor the processing progress of subtask segments on the target processing node. When it is detected that the file metadata information of the subtask segment has not changed and the processing time has exceeded the processing timeout threshold of each subtask segment, reselect the next processing node from the priority sequence of candidate processing nodes for task allocation.

[0086] In this specific embodiment, resource status information of each processing node is obtained, including the number of CPU cores, memory capacity, storage space, network bandwidth capacity, and load indicators such as current CPU utilization, memory usage, and network traffic for each processing node. Taking a server cluster in a data center as an example, processing node A has a 16-core CPU, 64GB of memory, and 10Gbps network bandwidth. The current CPU utilization is 30%, memory usage is 40%, and network bandwidth utilization is 25%. Processing node B has a 32-core CPU, 128GB of memory, and 40Gbps network bandwidth. The current CPU utilization is 60%, memory usage is 70%, and network bandwidth utilization is 45%.

[0087] Based on the resource requirements of each subtask segment—for example, subtask segment 1 requires a 4-core CPU, 8GB of memory, and 1Gbps of network bandwidth—it is matched with the available resources of each processing node to select candidate processing nodes that meet the resource requirements. For subtask segment 1, processing nodes A and B both meet its resource requirements and become candidate processing nodes.

[0088] The task affinity score for each candidate processing node is calculated. This score is based on multiple factors, including the node's resource matching degree, historical task completion rate, and average processing speed. Resource matching degree considers the degree of match between the task's resource requirements and the node's available resources; historical task completion rate reflects the node's stability; and average processing speed reflects the node's efficiency. For processing node A, the resource matching degree is 85% (sufficient but not excessive), the historical task completion rate is 98%, and the average processing speed is 120MB / s, resulting in a calculated affinity score of 0.89. For processing node B, the resource matching degree is 65% (excessive resources), the historical task completion rate is 95%, and the average processing speed is 180MB / s, resulting in a calculated affinity score of 0.81.

[0089] A priority sequence of candidate processing nodes is constructed based on the task affinity score. In this example, the priority sequence is: processing node A, processing node B. Processing node A is selected as the target processing node from the priority sequence, and subtask fragment 1 is assigned to processing node A for execution. Other subtask fragments are assigned in a similar way.

[0090] When the target processing node receives the subtask fragment, it starts the file generation process, creates a file write handle, allocates file storage space, and initializes the file checksum, such as using MD5 or SHA256 algorithms. Taking subtask fragment 1 as an example, its data size is 2GB, which is divided into 4MB data blocks, for a total of 500 data blocks.

[0091] For each data block, a write operation is performed. After the content of data block 1 is written to the file, the checksum of the data block is calculated. If the CRC32 algorithm is used, the checksum value is 0x1A2B3C4D. This checksum value is compared with the expected checksum value. If the checksum values ​​match, the next data block is processed. If they do not match (e.g., 0x1A2B3C5E is obtained), the data block is rewritten until the checksum matches or the maximum number of retries is reached.

[0092] During the writing process, file metadata information, including file size, number of processed data blocks, and last modification time, is updated periodically (e.g., every 10 data blocks written). After all data blocks have been written, the checksum of the entire file is calculated, the final file metadata is updated, and the subfile generation status is marked as "complete".

[0093] Continuously monitor the processing progress of all subtask segments. For each executing subtask segment, check its file metadata information at regular intervals (e.g., 5 seconds). If the metadata information of subtask segment 1 (e.g., last modified time, number of processed data blocks) does not change in 3 consecutive checks, and the processing time has exceeded the preset processing timeout threshold (e.g., 1.5 times the expected processing time calculated based on the data volume, 150 seconds in this example), then the processing of the subtask segment is determined to have failed.

[0094] When a failure is detected in the processing of subtask fragment 1, the next processing node B is selected from the priority sequence of candidate processing nodes, and subtask fragment 1 is reassigned to processing node B for execution. After receiving the task, processing node B restarts the file generation process, initializes the file write handle and checksum, writes data blocks to the file and performs verification, until the sub-file generation is successfully completed.

[0095] Through the above-mentioned technical means, the present invention realizes intelligent allocation of subtask segments, reliable writing of file data, and automatic redistribution of failed tasks, thereby improving the parallel efficiency and reliability of file generation.

[0096] In one optional implementation, all sub-files are merged into a target download file according to a preset merging strategy, and the file integrity is checked. The globally unique task identifier is then marked as completed, including:

[0097] Obtain the file metadata information of all sub-files, and establish a sub-file merging sequence according to the numbering order of the file metadata information;

[0098] The data blocks of the sub-files are read according to the sub-file merging sequence and written into the file merging buffer for data merging. The data merging rate is controlled by the token bucket algorithm. The capacity of the token bucket is dynamically adjusted according to the system disk write bandwidth. When the number of tokens in the token bucket is insufficient, the data merging operation is paused and the read rate of the upstream sub-files is reduced to balance the data processing pipeline.

[0099] The merged data is verified in real time using a sliding window mechanism. The cyclic redundancy check code of the data block is calculated within the sliding window. The calculated cyclic redundancy check code is compared with the original check code of the sub-file. When a check code mismatch is detected, the sliding window is rolled back and the data is merged again.

[0100] The data blocks that have passed verification are read from the merge buffer and written into the target download file. The integrity of the target download file is checked to obtain the verified target download file, and the globally unique task identifier is marked as completed.

[0101] In a method for achieving efficient file merging and integrity verification, the orderly merging and verification of sub-files is achieved through multiple steps. This method retrieves file metadata information of all sub-files from the storage system, including key attributes such as filename, creation time, file size, and file block sequence number. Based on the numbering order in the file metadata information, which is usually the sequence number contained in the sub-filename, such as "file_part_001", "file_part_002", etc., a sub-file merging sequence is constructed. For example, for a download task containing 5 sub-files, a merging queue is established in the order of "file_part_001" to "file_part_005" to ensure that the merging operation is performed in the correct file block order.

[0102] The merging process uses a token bucket algorithm to control the data processing rate. A token bucket is initialized with an initial capacity of 50MB, dynamically adjusted based on the current system disk write capacity. A certain number of tokens are added to the token bucket every 100 milliseconds, the amount depending on the real-time monitored disk write bandwidth. For example, when the system disk write bandwidth is detected to be 120MB / s, approximately 12MB of tokens are added every 100 milliseconds. During the data merging process, sub-file data blocks are read sequentially according to the sub-file merging sequence. The default block size is 4MB. Each data block read consumes a corresponding number of tokens. When the number of available tokens in the token bucket falls below 4MB, the data merging operation is paused. Simultaneously, a backpressure mechanism reduces the read rate of upstream sub-files, thereby balancing the entire data processing pipeline. In the specific implementation, when tokens are insufficient, the read thread enters a brief sleep state, typically 50 milliseconds, to prevent excessive memory consumption due to rapid read operations.

[0103] Data merging quality control employs a sliding window mechanism for real-time verification. A sliding window with a default size of 16MB is created. After the merged data is written to the merge buffer, the sliding window moves within the buffer and calculates the CRC32 cyclic redundancy check code of the data within the current window. For example, when 32MB of data has been written into the buffer, the sliding window will sequentially calculate the check codes for intervals such as 0-16MB, 4MB-20MB, and 8MB-24MB, and compare them with the original check codes of the sub-files. If a check code mismatch is detected in the interval [12MB-28MB], such as the calculated check code being 0xA1B2C3D4 while the original check code is 0xA1B2C3D5, the sliding window will back to the last successfully verified position, such as 8MB, and the corresponding sub-file data block will be reread and merged. To improve verification efficiency, a block-based calculation method is adopted, dividing the 16MB window into four 4MB sub-blocks to calculate the check codes in parallel and then merging the results.

[0104] Once the data passes verification, the verified data blocks are read from the merge buffer and written to the target download file. The writing process uses asynchronous I / O, maintaining an I / O request queue with a default length of 8. Each I / O request is 8MB in size. These I / O requests are processed in a background thread, writing the data to disk. After all sub-files are merged, the generated target download file undergoes integrity verification. Integrity verification combines block verification and full file verification. The target file is divided into multiple 128MB blocks, with the last block being less than 128MB. The SHA-256 hash value of each block is calculated and compared with the original hash value obtained during download task initialization. If all blocks pass verification, the MD5 value of the entire file is calculated for secondary confirmation. After successful verification, the system marks the task as completed in the task management system with a globally unique task identifier, such as "download_task_12345678," and updates the task completion time, file size, and other relevant information.

[0105] If a mismatch is found during the file integrity verification process, the information of the failed block is recorded, and an attempt is made to re-merge the corresponding sub-files. If the verification still fails after 3 retries, a detailed error report is generated, including information such as the range of failed blocks and the difference between the expected and actual verification values, to facilitate subsequent analysis and processing.

[0106] In one optional implementation, the target download file is uploaded to a file server cluster, and a file status monitoring table is established. In response to a user's file list query request, the target download file is retrieved from the file server cluster and returned, including:

[0107] The target download file is uploaded to the file server cluster, the storage location information of the target download file in the file server cluster is obtained, and the corresponding file address is generated.

[0108] Establish a file status monitoring table to record the file status, file address, and number of file accesses of the target download file. When the number of file accesses exceeds the access threshold, mark the target download file as a hot file and trigger the file preloading mechanism.

[0109] The system receives a file list query request from a user, obtains a file status list based on the file status monitoring table, prioritizes displaying popular files, retrieves the target download file from the server cluster, and returns it to the user.

[0110] In this specific embodiment, the system receives the target download file uploaded by the user and transmits it to a preset file server cluster via a secure transmission protocol (such as HTTPS, SFTP, etc.). The file server cluster includes multiple distributed file storage servers, and distributed storage of files is achieved through a load balancing strategy. The uploaded file is processed by fragmentation, for example, dividing a 2GB video file into 20 100MB data blocks, and calculating the hash value of each data block for verification to ensure file integrity. After the upload is completed, the system obtains the storage location information of the target download file from the file server cluster, including the server IP address, storage path, file fragment location, etc. Based on the above information, a unique file access address is generated. This address adopts the URI format, such as " / cluster / file / repository / 2023 / 07 / 15 / file_id_12345.mp4", for subsequent fast file location and access.

[0111] After the upload is complete, a file status monitoring table is created to record and manage the status information of the target downloaded file. This monitoring table uses a key-value pair data structure and includes the following fields: file ID, file name, file type, file size, upload time, file status (e.g., "available", "deleted", "corrupted"), file address, file access count, last access time, and file tags. The file status is checked periodically (e.g., every 5 minutes) to ensure file accessibility. When a file's access count exceeds a preset access threshold, the target downloaded file is marked as a hot file. For example, if a video file is accessed more than 1000 times within one hour, the system updates its status field to "hot file" and adds the corresponding tag to the database.

[0112] For target download files marked as "hot files," a file preloading mechanism is triggered. This is implemented by copying the hot file from its original storage location to a high-speed cache server, such as an edge node server distributed across different geographical locations. These edge node servers are configured with high-speed SSD storage and sufficient bandwidth resources, enabling faster response to user requests. A multi-copy strategy is implemented for hot files, storing identical copies of the file on multiple service nodes to improve concurrent file access performance. For extremely large files, versions of different qualities and formats are pre-generated to accommodate user needs under varying network conditions. For example, for a 4K resolution video file, 1080p, 720p, and 480p versions are generated, allowing users to automatically select the appropriate version based on network bandwidth.

[0113] When a file list query request is received from a user, the system retrieves relevant file information from the file status monitoring table based on request parameters such as file type and upload time range. A file status list is generated based on the records in the monitoring table. This list includes basic information such as file ID, file name, file type, file size, upload time, and file status. When generating the list, files marked as "hot" are prioritized and placed at the top of the list. The list is displayed in a paginated manner, with 20 records per page by default. Users can adjust the number of records displayed per page through request parameters.

[0114] After a user selects a target file to download from the file list, the system locates and retrieves the target file from the file server cluster based on the file address recorded in the file status monitoring table. For ordinary files, the file is retrieved directly from the original storage location; for hot files, the file content is retrieved from the nearest edge node server first. The system supports the function of resuming interrupted downloads. By recording the user's download progress, the system allows the user to continue downloading from the last interrupted position after the download is interrupted, avoiding the duplication of the downloaded file portion.

[0115] During file transfer, download operations are recorded, updating the file access count field in the file status monitoring table, along with the last access time. This allows for real-time monitoring of file access frequency and dynamic adjustment of the hot file detection threshold. For example, during peak network periods (e.g., 18:00-22:00 daily), the hot file detection threshold is increased to 2000 accesses per hour, while during off-peak periods (e.g., 2:00-6:00 AM), it is reduced to 500 accesses per hour to optimize system resource allocation.

[0116] Through the above methods, this embodiment achieves efficient uploading, status monitoring, and intelligent distribution of the target download file. In particular, the preloading processing of hot files significantly improves file access performance and user experience, while effectively balancing the load pressure of the server cluster and enhancing the stability and reliability of the entire system.

[0117] An asynchronous file generation and download implementation system according to an embodiment of the present invention includes:

[0118] The first unit is used to receive the user's file download request, extract the data source code and create a file download task, write the file download task into the database and generate a globally unique task identifier;

[0119] The second unit is used to analyze the file download task based on preset file feature recognition rules, dynamically divide the file download task into multiple sub-task segments according to the analysis results, and allocate independent resource quotas and processing timeout thresholds to each sub-task segment.

[0120] The third unit is used to distribute the multiple sub-task fragments to multiple target processing nodes, generate sub-files using a parallel file generation method, and redistribute the corresponding sub-file generation task to other file processing nodes when the sub-file generation fails. All sub-files are merged into a target download file according to a preset merging strategy, and the file integrity is checked. The globally unique task identifier is marked as completed.

[0121] The fourth unit is used to upload the target download file to the file server cluster, establish a file status monitoring table, and in response to the user's file list query request, retrieve and return the target download file from the file server cluster.

[0122] A third aspect of the present invention provides an electronic device, comprising:

[0123] processor;

[0124] Memory used to store processor-executable instructions;

[0125] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0126] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0127] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An asynchronous file generation and download implementation method, characterized in that, include: Receive a user's file download request, extract the data source identifier and create a file download task, write the file download task into the database and generate a globally unique task identifier; The file download task is analyzed based on preset file feature recognition rules. According to the analysis results, the file download task is dynamically divided into multiple sub-task segments, and each sub-task segment is allocated an independent resource quota and processing timeout threshold. The multiple subtask fragments are distributed to multiple distributed processing nodes, and the file generation method is called in parallel to generate sub-files. When the generation of a sub-file fails, the corresponding sub-file generation task is redistributed to other distributed processing nodes. All sub-files are merged into the target download file according to the preset merging strategy, and the file integrity is checked. The globally unique task identifier is marked as completed. The target download file is uploaded to the distributed file server cluster. In response to the user's file list query request, the target download file is retrieved from the distributed file server cluster and returned.

2. The method according to claim 1, characterized in that, The system receives a user's file download request, extracts the data source code, creates a file download task, writes the file download task to the database, and generates a globally unique task identifier, including: Receive the user's file download request, extract the data source type identifier, data source number, timestamp information and digital signature from it, obtain the data source code and create a file download task; Based on the data source encoding, combined with the user level and task queue length, the priority score of the file download task is calculated. The file download task is written into the two-layer task processing structure according to the priority score. When the number of tasks in the two-layer task processing structure reaches the upper limit, an asynchronous thread is started to write the file download tasks with priority scores greater than the priority threshold into the database in batches. Based on the database write timestamp, the priority score, and the data source code, a globally unique task identifier is generated for the file download task, and a cache expiration time is set.

3. The method according to claim 1, characterized in that, The file download task is analyzed based on preset file feature recognition rules. According to the analysis results, the file download task is dynamically divided into multiple sub-task segments, and each sub-task segment is allocated an independent resource quota and processing timeout threshold, including: Extract file data from the file download task, calculate the entropy value, data distribution density, and time series correlation of the file data, and construct a file feature vector; The correlation degree of data blocks is calculated based on the file feature vector, a data block dependency graph is constructed, and the ratio of the in-degree value to the out-degree value of each data block node is calculated. Data block nodes with a ratio lower than the dependency threshold are used as initial splitting nodes. Calculate the data transmission cost between the initial segmentation nodes, select the initial segmentation node whose data transmission cost is less than the cost threshold as the optimal segmentation node, and dynamically segment the file download task based on the optimal segmentation node to generate multiple subtask segments. The file feature vector is matched with the features of historical execution tasks. Based on the similarity matching results, an independent processing timeout threshold is set for each subtask segment, and an independent resource quota is set in combination with its data complexity.

4. The method according to claim 1, characterized in that, The multiple subtask fragments are assigned to multiple target processing nodes, and a parallel file generation method is used to generate subfiles. When a subfile generation failure is detected, the corresponding subfile generation task is redistributed to other file processing nodes, including: Obtain the total resources, network bandwidth capacity and current load status of each processing node, filter out candidate processing nodes that meet the resource quota requirements of subtask segments, calculate the task affinity score of each candidate processing node, construct a priority sequence of candidate processing nodes in descending order of task affinity score, select multiple candidate processing nodes as target processing nodes, and allocate the multiple subtask segments to the target processing nodes. Start the file generation process on the target processing node, initialize the file write handle and file checksum, write the data of the subtask fragments into the file according to the data block size, calculate the checksum of each data block and compare it with the file checksum, and rewrite the data block when a data block checksum mismatch is detected. After the sub-file generation is completed, update the file metadata information. Monitor the processing progress of subtask segments on the target processing node. When it is detected that the file metadata information of the subtask segment has not changed and the processing time has exceeded the processing timeout threshold of each subtask segment, reselect the next processing node from the priority sequence of candidate processing nodes for task allocation.

5. The method according to claim 1, characterized in that, All sub-files are merged into the target download file according to a preset merging strategy, and the file integrity is checked. The globally unique task identifier is marked as completed, including: Obtain the file metadata information of all sub-files, and establish a sub-file merging sequence according to the numbering order of the file metadata information; The data blocks of the sub-files are read according to the sub-file merging sequence and written into the file merging buffer for data merging. The data merging rate is controlled by the token bucket algorithm. The capacity of the token bucket is dynamically adjusted according to the system disk write bandwidth. When the number of tokens in the token bucket is insufficient, the data merging operation is paused and the read rate of the upstream sub-files is reduced to balance the data processing pipeline. The merged data is verified in real time using a sliding window mechanism. The cyclic redundancy check code of the data block is calculated within the sliding window. The calculated cyclic redundancy check code is compared with the original check code of the sub-file. When a check code mismatch is detected, the sliding window is rolled back and the data is merged again. The data blocks that have passed verification are read from the merge buffer and written into the target download file. The integrity of the target download file is checked to obtain the verified target download file, and the globally unique task identifier is marked as completed.

6. The method according to claim 1, characterized in that, Upload the target download file to the file server cluster and establish a file status monitoring table. In response to a user's file list query request, retrieve and return the target download file from the file server cluster, including: The target download file is uploaded to the file server cluster, the storage location information of the target download file in the file server cluster is obtained, and the corresponding file address is generated. Establish a file status monitoring table to record the file status, file address, and number of file accesses of the target download file. When the number of file accesses exceeds the access threshold, mark the target download file as a hot file and trigger the file preloading mechanism. The system receives a file list query request from a user, obtains a file status list based on the file status monitoring table, prioritizes displaying popular files, retrieves the target download file from the server cluster, and returns it to the user.

7. An asynchronous file generation and download implementation system, used to implement the method as described in any one of claims 1-6, characterized in that, include: The first unit is used to receive the user's file download request, extract the data source code and create a file download task, write the file download task into the database and generate a globally unique task identifier; The second unit is used to analyze the file download task based on preset file feature recognition rules, dynamically divide the file download task into multiple sub-task segments according to the analysis results, and allocate independent resource quotas and processing timeout thresholds to each sub-task segment. The third unit is used to allocate the multiple subtask fragments to multiple target processing nodes, generate sub-files by using a parallel file generation method, and when the generation of the sub-file is detected to fail, redistribute the corresponding sub-file generation task to other file processing nodes. All sub-files are merged into the target download file according to the preset merging strategy, and the file integrity is checked. The globally unique task identifier is marked as completed. The fourth unit is used to upload the target download file to the file server cluster, establish a file status monitoring table, and in response to the user's file list query request, retrieve and return the target download file from the file server cluster.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images