Asynchronous file generation download implementation method and system

Abstract

The application provides an asynchronous file generation download implementation method and system, relates to the technical field of computer file processing, and comprises the following steps: after receiving a user file download request, a task is created and a unique identifier is generated; based on file feature identification rules, the task is divided into multiple subtask segments and resources are allocated; the subtasks are allocated to multiple processing nodes to generate subfiles in parallel; the subfiles are merged into a target download file and uploaded to a server cluster; and the file is returned in response to a user query. The application improves the large file download efficiency, enhances the system stability, and optimizes the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer file processing, and particularly relates to an asynchronous file generation and download implementation method and system. BACKGROUND

[0002] In an enterprise information system, file download function is a common basic requirement, especially in data analysis, report export and other application scenarios, users often need to export a large amount of data as a file for download. The traditional file download implementation usually adopts a synchronous processing method, that is, when the user initiates a download request, the system immediately performs data query, file generation and transmission operations, and the user needs to wait for the entire process to complete to obtain the file.

[0003] The prior art lacks effective file generation state tracking and task recovery mechanism, and once an exception or failure occurs in the file generation process, the entire task needs to be re-executed, wasting system resources and prolonging user waiting time.

[0004] In recent years, although some systems have begun to use asynchronous processing to improve file download function, there are still problems such as inflexible task scheduling, unreasonable resource allocation, and imperfect subtask failure processing mechanism. In complex business scenarios, especially when dealing with large data volumes or involving multi-data source integration file generation tasks, the prior art cannot meet the requirements of high efficiency and high reliability. SUMMARY

[0005] The embodiments of the present application provide an asynchronous file generation and download implementation method and system, which can solve the problems in the prior art.

[0006] In a first aspect of the embodiments of the present application, an asynchronous file generation and download implementation method is provided, comprising:

[0007] Receiving a file download request of a user, extracting a data source code and creating a file download task, writing the file download task into a database and generating a globally unique task identifier;

[0008] Analyzing the file download task based on a preset file feature recognition rule, dynamically dividing the file download task into multiple subtask segments according to the analysis result, and allocating an independent resource quota and a processing timeout threshold to each subtask segment;

[0009] Allocating the multiple subtask segments to multiple target processing nodes, generating subfiles by calling a file generation method in parallel, when detecting that the subfile generation fails, redistributing the corresponding subfile generation task to other file processing nodes; merging all subfiles into a target download file according to a preset merging strategy, detecting the file integrity, and marking the globally unique task identifier as completed;

[0010] uploading the target download file to a file server cluster and establishing a file state monitoring table, in response to a file list query request of a user, obtaining and returning the target download file from the file server cluster.

[0011] receiving a file download request of a user, extracting a data source code and creating a file download task, writing the file download task into a database and generating a globally unique task identification, comprising:

[0012] receiving a file download request of a user, and extracting a data source type identification, a data source number, timestamp information and a digital signature therefrom, obtaining a data source code and creating a file download task;

[0013] based on the data source code, combining a user level and a task queue length, calculating a priority score of the file download task, writing the file download task into a double-layer task processing structure according to the priority score, when the number of tasks of the double-layer task processing structure reaches an upper limit, starting an asynchronous thread to write a plurality of file download tasks with priority scores greater than a priority threshold into the database in batches;

[0014] generating a globally unique task identification of the file download task according to a database write timestamp, the priority score and the data source code, and setting a cache invalidation time.

[0015] analyzing the file download task based on a preset file feature recognition rule, dynamically dividing the file download task into a plurality of subtask segments according to an analysis result, and allocating an independent resource quota and a processing timeout threshold to each subtask segment, comprising:

[0016] extracting file data in the file download task, calculating an entropy value, a data distribution density and a time sequence correlation of the file data, and constructing a file feature vector;

[0017] calculating a data block correlation degree according to the file feature vector, constructing a data block dependency graph, and calculating a ratio of an in-degree value to an out-degree value of each data block node, taking a data block node with a ratio lower than a dependency threshold as an initial segmentation node;

[0018] calculating a data transmission cost between the initial segmentation nodes, selecting an initial segmentation node with a data transmission cost less than a cost threshold as an optimal segmentation node, and dynamically segmenting the file download task based on the optimal segmentation node to generate a plurality of subtask segments;

[0019] performing similarity matching of the file feature vector and historical execution task features, setting an independent processing timeout threshold for each subtask segment based on the similarity matching result, and allocating an independent resource quota in combination with the data complexity thereof.

[0020] allocating the plurality of sub-task segments to a plurality of target processing nodes, generating sub-files by using a parallel invocation file generation method, when detecting that the sub-file generation fails, re-distributing the corresponding sub-file generation task to other file processing nodes, comprising:

[0021] obtaining the total amount of resources, network bandwidth capacity and current load state of each processing node, screening out candidate processing nodes that meet the resource quota requirements of the sub-task segments, and calculating the task affinity scores of each candidate processing node, constructing a candidate processing node priority sequence in the order of task affinity scores from high to low, and selecting a plurality of candidate processing nodes as target processing nodes, and allocating the plurality of sub-task segments to the target processing nodes;

[0022] starting a file generation process on the target processing node, initializing a file write handle and a file checksum, writing the data of the sub-task segment into the file according to the data block size, calculating the checksum of each data block and comparing it with the file checksum, when detecting that the data block checksum does not match, re-writing the data block, updating the file metadata information after completing the sub-file generation;

[0023] monitoring the processing progress of the sub-task segment on the target processing node, when detecting that the file metadata information of the sub-task segment has not changed and the processing time has exceeded the processing timeout threshold of each sub-task segment, re-selecting the next processing node from the candidate processing node priority sequence for task allocation.

[0024] merging all sub-files into a target download file according to a preset merging strategy, and detecting the file integrity, marking the global unique task identifier as completed, comprising:

[0025] obtaining the file metadata information of all sub-files, and establishing a sub-file merging sequence according to the numbering order of the file metadata information;

[0026] reading the data block of the sub-file according to the sub-file merging sequence, and writing it into the file merging buffer for data merging, controlling the data merging rate by the token bucket algorithm, the capacity of the token bucket is dynamically adjusted according to the system disk write bandwidth, when the number of tokens in the token bucket is insufficient, the data merging operation is suspended, and the reading rate of the upstream sub-file is reduced to balance the data processing pipeline;

[0027] based on the sliding window mechanism, the merged data is verified in real time, the cyclic redundancy check code of the data block is calculated within the sliding window, the calculated cyclic redundancy check code is compared with the original check code of the sub-file, when detecting that the check code does not match, the sliding window is rolled back and the data merging is re-performed;

[0028] Read the data block passed the check from the merging buffer to a target download file, perform integrity check on the target download file to obtain a verified target download file, and mark the globally unique task identifier as a completed state.

[0029] Upload the target download file to a file server cluster, and establish a file state monitoring table, in response to a file list query request of a user, obtain and return the target download file from the file server cluster, including:

[0030] Upload the target download file to a file server cluster, obtain storage location information of the target download file in the file server cluster, and generate a corresponding file address;

[0031] Establish a file state monitoring table, record the file state, the file address and the file access times of the target download file, when the file access times exceed an access threshold, mark the target download file as a hot file, and trigger a file preloading mechanism;

[0032] Receive a file list query request sent by a user, obtain a file state list based on the file state monitoring table, and preferentially display a hot file, obtain a target download file from the server cluster and return to the user.

[0033] A second aspect of the embodiment of the application provides an asynchronous file generation and download implementation system, including:

[0034] A first unit is configured to receive a file download request of a user, extract a data source code and create a file download task, write the file download task into a database and generate a globally unique task identifier;

[0035] A second unit is configured to analyze the file download task based on a preset file feature recognition rule, dynamically divide the file download task into a plurality of subtask segments according to an analysis result, and allocate an independent resource quota and a processing timeout threshold to each subtask segment;

[0036] A third unit is configured to allocate the plurality of subtask segments to a plurality of target processing nodes, generate subfiles by using a parallel calling file generation method, when it is detected that the subfile generation fails, re-distribute a corresponding subfile generation task to other file processing nodes, merge all the subfiles into a target download file according to a preset merging strategy, detect the file integrity, and mark the globally unique task identifier as completed;

[0037] A fourth unit is configured to upload the target download file to a file server cluster, establish a file state monitoring table, and in response to a file list query request of a user, obtain and return the target download file from the file server cluster.

[0038] A third aspect of the embodiments of the present application,

[0039] An electronic device is provided, comprising:

[0040] a processor;

[0041] a memory for storing processor-executable instructions;

[0042] wherein the processor is configured to invoke the instructions stored by the memory to perform the method described above.

[0043] A fourth aspect of the embodiments of the present application,

[0044] A computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0045] The beneficial effects of the present application are as follows:

[0046] By implementing the asynchronous file generation and download implementation method of the present application, the file download task can be dynamically divided into multiple sub-task segments for parallel processing, effectively improving the processing efficiency of large file generation and download, shortening the user waiting time, and improving the system response speed.

[0047] The present application adopts an independent resource quota and timeout threshold control mechanism, combined with a sub-task failure automatic redistribution strategy, significantly enhances the stability and fault tolerance capability of the system in a high concurrency scenario, effectively avoids the overall download failure problem caused by single point failure, and ensures the reliability of the file generation process.

[0048] In addition, the present application realizes the function of allowing users to query the file processing progress and obtain the results at any time through file server cluster storage and file state monitoring table management, the user experience is obviously improved, and at the same time, it is convenient for the system administrator to monitor and manage the file processing process, and the maintainability of the system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The flowchart of the asynchronous file generation and download implementation method of the embodiments of the present application is shown;

[0050] Figure 2 The flowchart of the file download request processing method is shown. DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0052] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and some embodiments may not be described again for the same or similar concepts or processes.

[0053] Figure 1 The flowchart of the method for generating a download of an asynchronous file according to an embodiment of the present application is shown in FIG. 1, which comprises the following steps. Figure 1

[0054] Receiving a file download request of a user, extracting a data source code and creating a file download task, writing the file download task into a database and generating a globally unique task identification;

[0055] Analyzing the file download task based on a preset file feature recognition rule, dynamically dividing the file download task into multiple subtask segments according to the analysis result, and assigning an independent resource quota and a processing timeout threshold to each subtask segment;

[0056] Assigning the multiple subtask segments to multiple target processing nodes, generating subfiles by using a parallel call file generation method, re-distributing the corresponding subfile generation task to other file processing nodes when detecting that the subfile generation fails, merging all the subfiles into a target download file according to a preset merging strategy, detecting the file integrity, and marking the globally unique task identification as completed;

[0057] Uploading the target download file to a file server cluster, establishing a file state monitoring table, and in response to a file list query request of a user, obtaining and returning the target download file from the file server cluster.

[0058] Figure 2 The flowchart of the file download request processing method is shown in FIG. 1. In an optional embodiment, receiving a file download request of a user, extracting a data source code and creating a file download task, writing the file download task into a database and generating a globally unique task identification, comprises:

[0059] Receiving a file download request of a user, extracting a data source code and creating a file download task, writing the file download task into a database and generating a globally unique task identification, comprises:​

[0060] Based on the data source encoding, combined with user level and task queue length, the priority score of the file download task is calculated, and the file download task is written into the double-layer task processing structure according to the priority score, when the number of tasks in the double-layer task processing structure reaches the upper limit, the asynchronous thread is started to write the file download tasks with priority score greater than the priority threshold into the database in batches;

[0061] According to the database write timestamp, the priority score and the data source encoding, the globally unique task identifier of the file download task is generated, and the cache invalidation time is set.

[0062] In this embodiment, a method of receiving user file download request, extracting data source encoding and creating file download task is described in detail. This method creates an efficient task processing mechanism to ensure the reasonable allocation of system resources and the priority order of task processing.

[0063] In this embodiment, the user's file download request needs to be received, which is usually transmitted through the network in the form of HTTP or HTTPS protocol, and contains multiple key information in the request. The data source type identifier is extracted from the request body, which is usually in the form of a string, such as "DB" representing a database type source, "API" representing an interface type source, and "FILE" representing a file system type source. At the same time, the data source number is extracted, which is usually a unique string, such as "SRC20240501001", used to accurately locate the specific data source. The timestamp information in the request is recorded in UTC format, such as "2024-05-01T12:30:45Z", used to prevent replay attacks. The request also contains a digital signature, which is an encrypted string generated by the SHA256+RSA algorithm, such as "a1b2c3d4e5f6...", used to verify the legality and integrity of the request. By combining the data source type identifier and the data source number, a complete data source encoding is generated, such as "DB_SRC20240501001", and a new file download task object is created based on it.

[0064] After the file download task is created, the priority score of the task is calculated to allocate system resources reasonably. The priority calculation considers three main factors: data source encoding, user level, and current task queue length. For data source encoding, different types are assigned a base score, such as 10 points for database type, 8 points for API type, and 6 points for file system type. User levels are usually divided into VIP users, ordinary users, and trial users, with weights of 1.5, 1.0, and 0.8, respectively. When the task queue length is less than 100, the score is not adjusted; when the length is between 100 and 500, the priority is reduced by 10%; and when the length exceeds 500, the priority is reduced by 20%. The final priority score is calculated by combining these factors. For example, if a VIP user requests to download a database type source and the current queue length is 80, the priority score is 10 x 1.5 x 1.0 = 15 points.

[0065] After calculating the priority score, the file download task is written to a double-layer task processing structure, which includes a high-priority task queue and a low-priority task queue in memory. When the priority score exceeds 12 points, the task enters the high-priority queue; otherwise, it enters the low-priority queue. Each queue is organized using a priority heap structure to ensure that high-score tasks are processed first. The number of tasks in the double-layer task processing structure is continuously monitored. When the total number of tasks reaches a preset upper limit (usually 1000), an asynchronous thread is started to process. This thread filters tasks with a priority score greater than a priority threshold (usually set to 10 points) and groups them into batches (up to 100 tasks per batch). The database bulk write interface is called to persistently store these tasks. The database write uses a transaction mechanism to ensure atomicity of bulk writing, i.e., all succeed or all fail.

[0066] After successful database writing, a globally unique task identifier (Task ID) needs to be generated for each task. The generation process considers three elements: database write timestamp, task priority score, and data source encoding. In the specific implementation, a timestamp accurate to milliseconds is obtained, such as 1588763142358. The priority score is converted to a two-digit fixed-length string, such as priority 15.5 converted to "16". The last 6 characters are extracted from the data source encoding (such as "01001" from "DB_SRC20240501001"). These three parts are connected using a specific separator and a 4-digit random alphanumeric string is added as a suffix to form the final task identifier, for example: "1588763142358_16_01001_A7B9". This combination ensures the global uniqueness of the identifier while containing key information about the task.

[0067] After generating the task identifier, the task information is written into the cache system to improve subsequent query efficiency. The cache record contains task identifier, data source code, priority score, creation time, and other key information. Different cache expiration times are set according to task priority: high priority tasks (score > 15) are set to 30 minutes of cache period; medium priority tasks (10-15 minutes) are set to 20 minutes of cache period; low priority tasks (<10 minutes) are set to 10 minutes of cache period. After cache expiration, task information will be reloaded from the database to ensure data consistency.

[0068] The response containing the globally unique task identifier is returned to the user, who can query the task status through this identifier. The response body is in JSON format and contains task identifier, estimated processing time, task status, and other information. At the same time, detailed operation logs are recorded, including user ID, operation time, request IP, task identifier, and other information, to facilitate subsequent auditing and problem troubleshooting.

[0069] In an optional implementation, the file download task is analyzed based on a preset file feature recognition rule, and the file download task is dynamically divided into multiple sub-task segments according to the analysis result, and an independent resource quota and a processing timeout threshold are allocated to each sub-task segment, including:

[0070] The file data in the file download task is extracted, the entropy value, data distribution density, and time sequence correlation of the file data are calculated, and a file feature vector is constructed;

[0071] The data block correlation degree is calculated according to the file feature vector, a data block dependency graph is constructed, and the ratio of the in-degree value and the out-degree value of each data block node is calculated. The data block node with a ratio lower than the dependency threshold is selected as the initial segmentation node;

[0072] The data transmission cost between the initial segmentation nodes is calculated, and the initial segmentation node with a data transmission cost less than a cost threshold is selected as the optimal segmentation node. Based on the optimal segmentation node, the file download task is dynamically fragmented to generate multiple sub-task segments;

[0073] The file feature vector is matched with historical execution task features in terms of similarity, and an independent processing timeout threshold is set for each sub-task segment based on the similarity matching result. An independent resource quota is set in combination with the data complexity.

[0074] In a specific implementation, a file download task is received, which contains basic information of the file such as URL, file size, etc. A file data sample is extracted from the download task, which is usually the first 1 MB of data or a uniformly sampled data segment. The entropy value of the extracted file data is calculated, which reflects the randomness and information density of the data. The entropy value is calculated by counting the frequency of each byte in the data. A high entropy value (e.g., > 7.5) usually indicates that the data is compressed or encrypted. At the same time, the data distribution density is calculated, which is the distribution of data values within a range. This can be achieved by constructing a histogram, dividing the byte value range of 0-255 into 16 intervals, and counting the number of data points in each interval. The time series correlation of the data is also analyzed, which is the degree of correlation between adjacent data blocks. This can be quantified by calculating the correlation coefficient of adjacent data blocks. A correlation coefficient close to 1 indicates a high correlation, and a correlation coefficient close to 0 indicates almost no correlation. Through these calculations, a file feature vector is constructed, which includes the entropy value, data distribution density feature vector, and time series correlation index.

[0075] Based on the constructed file feature vector, the correlation between each data block within the file is calculated, and the file is divided into equal-sized basic data blocks, such as 64 KB per block. Based on the time series correlation index in the feature vector, the correlation between blocks is calculated. For text files, the correlation between adjacent blocks is high (e.g., 0.8), while for different content regions in multimedia files, the correlation is low (e.g., 0.3). These correlation values are used to construct a data block dependency graph. The nodes in the graph represent data blocks, and the edges represent the correlation between data blocks. The weight of the edge represents the correlation strength.

[0076] After the dependency graph is constructed, the ratio of the in-degree value and the out-degree value of each data block node is calculated. The in-degree value represents the number of edges pointing to the node, and the out-degree value represents the number of edges from the node. Nodes with a ratio below a preset dependency threshold (e.g., 0.5) are marked as initial segmentation nodes. These nodes are usually locations where the data content changes significantly and are suitable for task segmentation nodes. For example, a 100 MB video file has two low-ratio nodes at 33 MB and 67 MB, with a ratio of 0.3 and 0.4, respectively. These locations correspond to scene transitions or content type changes.

[0077] The data transmission cost between the initial segmentation nodes is further calculated, which is the resource consumption required to transfer data from one node to another. The transmission cost can be calculated based on the data volume between nodes, network conditions, and data characteristics. For example, for two initial segmentation nodes 20 MB apart, if the entropy value of the intermediate data is high, indicating high data complexity, the transmission cost reaches a high value such as 85. For data regions with low entropy values, even if they are 30 MB apart, the transmission cost is only 50. The initial segmentation nodes with a data transmission cost less than a preset cost threshold are selected as the optimal segmentation nodes.

[0078] Based on the determined optimal segmentation node, the file download task is dynamically fragmented into multiple sub-task fragments, each corresponding to a continuous data region of the original file, such as a 100MB file being divided into three sub-tasks: 0-33MB, 33-67MB and 67-100MB. This fragmentation method takes into account the internal structure of the data, making the data within each sub-task more strongly related and the dependency between sub-tasks weaker.

[0079] The constructed file feature vector is matched with the feature vectors of historical execution tasks in terms of similarity. The similarity calculation can use methods such as cosine similarity or Euclidean distance. The higher the value, the more similar the task characteristics. For example, the similarity between the current file feature vector and a certain ZIP file in the historical record is 0.92, indicating that the two files have similar processing characteristics. Based on the similarity matching result, an independent processing timeout threshold is set for each sub-task fragment. For sub-tasks with high similarity to historical tasks (e.g., similarity > 0.9), the average processing time of the historical task plus a 20% tolerance is directly used as the timeout threshold; for cases with medium similarity (0.6-0.9), the historical time multiplied by 150% can be used as the threshold; for cases with low similarity (<0.6), a relatively loose default threshold is set, such as the size of the sub-task divided by the lowest expected download rate.

[0080] At the same time, independent resource quotas are set according to the data complexity of the sub-task fragments. Data complexity can be measured by entropy and distribution density in the feature vector. Sub-tasks with high complexity (e.g., entropy > 7.0) require more processing resources, allocating 50% more memory and CPU time; while sub-tasks with low complexity (e.g., entropy < 5.0) can use standard quotas. For example, for a download task containing three sub-tasks, the first sub-task is allocated 2 processing threads and 100MB of memory, with a timeout threshold of 30 seconds; the second sub-task is allocated 4 threads and 200MB of memory, with a timeout threshold of 60 seconds; the third sub-task is allocated 6 threads and 400MB of memory, with a timeout threshold of 120 seconds.

[0081] Through this dynamic fragmentation and resource allocation mechanism, the download process can be optimized according to the file characteristics, improving the success rate and efficiency of the download, while reasonably utilizing system resources.

[0082] In an optional implementation, the multiple sub-task fragments are allocated to multiple target processing nodes, and a parallel file generation method is used to generate sub-files. When it is detected that the sub-file generation fails, the corresponding sub-file generation task is redistributed to other file processing nodes, including:

[0083] obtaining total resource amount, network bandwidth capacity and current load state of each processing node, screening out candidate processing nodes meeting resource quota requirement of sub-task fragments, and calculating task affinity score of each candidate processing node, constructing candidate processing node priority sequence in order of task affinity score from high to low, and selecting multiple candidate processing nodes as target processing nodes, and distributing the multiple sub-task fragments to the target processing nodes;

[0084] starting a file generation process on the target processing node, initializing a file write handle and a file check code, writing data of the sub-task fragments into a file according to data block size, calculating a check code of each data block and comparing it with the file check code, re-writing the data block when a mismatch is detected, updating file metadata information after completion of the sub-file generation;

[0085] monitoring processing progress of the sub-task fragments on the target processing node, and re-selecting the next processing node from the candidate processing node priority sequence for task distribution when detecting that file metadata information of the sub-task fragments has not changed and processing time has exceeded a processing timeout threshold of each sub-task fragment.

[0086] In the specific embodiment, resource state information of each processing node is obtained, including CPU core number, memory capacity, storage space, network bandwidth capacity, and current CPU usage, memory occupancy, network traffic and other load indicators of each processing node. Taking a server cluster of a certain data center as an example, processing node A has a 16-core CPU, 64GB memory, 10Gbps network bandwidth, current CPU usage of 30%, memory occupancy of 40%, and network bandwidth usage of 25%; processing node B has a 32-core CPU, 128GB memory, 40Gbps network bandwidth, current CPU usage of 60%, memory occupancy of 70%, and network bandwidth usage of 45%.

[0087] According to resource requirements of each sub-task fragment, for example, sub-task fragment 1 requires 4-core CPU, 8GB memory, and 1Gbps network bandwidth, the available resources of each processing node are matched, and candidate processing nodes meeting the resource requirements are screened out. For sub-task fragment 1, processing nodes A and B both meet its resource requirements and become candidate processing nodes.

[0088] The task affinity score of each candidate processing node is calculated, and the task affinity score is based on multiple factors, including resource matching degree, historical task completion rate, average processing speed, etc. of the processing node. The resource matching degree considers the degree of matching between the resource demand of the subtask and the resource that the processing node can provide; the historical task completion rate reflects the stability of the processing node; and the average processing speed embodies the efficiency of the processing node. For processing node A, the resource matching degree is 85% (resource is sufficient but not excessive), the historical task completion rate is 98%, and the average processing speed is 120 MB / s. The affinity score is 0.89 after comprehensive calculation; for processing node B, the resource matching degree is 65% (resource is excessive), the historical task completion rate is 95%, and the average processing speed is 180 MB / s. The affinity score is 0.81 after comprehensive calculation.

[0089] The priority sequence of the candidate processing nodes is constructed according to the task affinity score. In this example, the priority sequence is: processing node A, processing node B. Processing node A is selected as the target processing node from the priority sequence, and subtask fragment 1 is assigned to processing node A for execution. Other subtask fragments are also assigned in a similar manner.

[0090] When the target processing node receives the subtask fragment, it starts the file generation process, creates a file write handle, allocates file storage space, and initializes the file checksum, such as using MD5 or SHA256 algorithm. Taking subtask fragment 1 as an example, its data size is 2GB, which is divided into 500 data blocks of 4MB each.

[0091] For each data block, write operation is performed. After the content of data block 1 is written to the file, the checksum of the data block is calculated, such as using CRC32 algorithm, to obtain the checksum value 0x1A2B3C4D. The checksum value is compared with the expected checksum value. If the checksum values match, the next data block is processed; if they do not match (such as 0x1A2B3C5E), the data block is re-written until the checksum matches or the maximum number of retries is reached.

[0092] During the writing process, file metadata information is updated regularly (such as every 10 data blocks written). The file metadata information includes file size, number of processed data blocks, last modification time, etc. After all data blocks are written, the checksum of the entire file is calculated, the final file metadata is updated, and the subfile generation state is marked as "complete".

[0093] The processing progress of all sub-task segments is continuously monitored, and for each sub-task segment being executed, the file metadata information thereof is checked every certain time (e.g., 5 seconds). If the metadata information (e.g., last modification time, number of processed data blocks) of the sub-task segment 1 is found to have no change in three consecutive checks, and the processing time has exceeded a preset processing timeout threshold (e.g., 1.5 times of the expected processing time calculated according to the data amount, which is 150 seconds in this example), it is determined that the processing of the sub-task segment fails.

[0094] When the processing of the sub-task segment 1 is detected to fail, the next processing node B is selected from the candidate processing node priority sequence, and the sub-task segment 1 is reassigned to the processing node B for execution. After receiving the task, the processing node B restarts the file generation process, initializes the file write handle and the check code, writes the data blocks into the file and performs the check, until the sub-file generation is successfully completed.

[0095] Through the above technical means, the intelligent distribution of sub-task segments, the reliable writing of file data, and the automatic redistribution of failed tasks are realized, and the parallel efficiency and reliability of file generation are improved.

[0096] In an optional embodiment, all sub-files are merged into a target download file according to a preset merging strategy, and the file integrity is detected, and the global unique task identifier is marked as completed, including:

[0097] File metadata information of all sub-files is acquired, and a sub-file merging sequence is established according to the numbering order of the file metadata information;

[0098] Data blocks of the sub-files are read according to the sub-file merging sequence, and are written into a file merging buffer for data merging, a token bucket algorithm is used to control the data merging rate, the capacity of the token bucket is dynamically adjusted according to the system disk write bandwidth, and when the number of tokens in the token bucket is insufficient, the data merging operation is suspended, and the reading rate of the upstream sub-file is reduced to balance the data processing pipeline;

[0099] The merged data is checked in real time based on a sliding window mechanism, the cyclic redundancy check code of the data blocks is calculated within the sliding window, the calculated cyclic redundancy check code is compared with the original check code of the sub-file, and when the check codes are detected to be mismatched, the sliding window is rolled back and the data merging is performed again;

[0100] The data blocks that pass the check are read from the merging buffer and written into the target download file, the integrity of the target download file is verified, the target download file that passes the verification is obtained, and the global unique task identifier is marked as completed.

[0101] In a method for efficient file merging and integrity verification, the ordered merging and verification of subfiles is achieved through multiple steps. The method retrieves file metadata information of all subfiles from the storage system, including file name, creation time, file size, file block sequence number, and other key attributes. Based on the sequence number in the file metadata information, usually the sequence number contained in the subfile name, such as "file_part_001", "file_part_002", etc., a subfile merging sequence is constructed, for example, for a download task containing 5 subfiles, a merging queue is established in the order of "file_part_001" to "file_part_005", ensuring that the merging operation is performed in the correct file block sequence.

[0102] The merging process uses a token bucket algorithm to control the data processing rate. A token bucket is initialized with an initial capacity of 50MB, which is dynamically adjusted based on the current system disk write capability. Every 100 milliseconds, a certain number of tokens are added to the token bucket, and the amount added depends on the real-time monitored disk write bandwidth. For example, when the system disk write bandwidth is detected to be 120MB / s, about 12MB of tokens will be added every 100 milliseconds. During the data merging process, subfile data blocks are read in sequence according to the subfile merging sequence, with a default block size of 4MB. Each read data block consumes tokens of the corresponding size. When the available token quantity in the token bucket is less than 4MB, the data merging operation is paused, and the read rate of the upstream subfile is reduced through a backpressure mechanism, thereby balancing the entire data processing pipeline. In specific implementation, the read thread will enter a short sleep state, usually 50 milliseconds, when tokens are insufficient, to avoid excessive memory usage caused by too fast read operations.

[0103] The data merging quality control uses a sliding window mechanism for real-time verification. A sliding window with a default size of 16MB is created. When merging data is written to the merging buffer, the sliding window moves within the buffer and calculates the CRC32 cyclic redundancy check code of the data in the current window. For example, when 32MB of data has been written to the buffer, the sliding window will calculate the check code for the intervals 0-16MB, 4MB-20MB, 8MB-24MB, etc., and compare it with the original check code of the subfile. If a mismatch is detected in the interval [12MB-28MB], such as the calculated check code being 0xA1B2C3D4 and the original check code being 0xA1B2C3D5, the sliding window is rolled back to the last successful check position, such as 8MB, the corresponding subfile data block is re-read and merged. To improve verification efficiency, the 16MB window is divided into 4 4MB sub-blocks for parallel calculation of the check code and merging of the results.

[0104] When the data passes the verification, the data block that passes the verification is read from the merging buffer and written into the target download file. The writing adopts an asynchronous IO mode. An IO request queue is maintained. The default queue length is 8. The data size of each IO request is 8 MB. The IO requests are processed in a background thread, and the data is written into the disk. After the merging of all sub files is completed, the integrity of the generated target download file is verified. The integrity verification adopts a combination of block verification and full file verification. The target file is divided into multiple blocks with a size of 128 MB. The last block is smaller than 128 MB. The SHA-256 hash value of each block is calculated and compared with the original hash value obtained during the initialization of the download task. If all the blocks pass the verification, the MD5 value of the entire file is calculated for secondary confirmation. After the verification passes, the system marks the global unique task identifier, such as "download_task_12345678", as completed in the task management system, and updates the task completion time, file size and other related information.

[0105] If a mismatch is found during the file integrity verification process, the information of the failed block is recorded, and the corresponding sub file is tried to be merged again. If the verification still fails after 3 retries, a detailed error report is generated, including the range of the failed block, the difference between the expected verification value and the actual verification value and other information, to facilitate subsequent analysis and processing.

[0106] In an optional implementation, the target download file is uploaded to a file server cluster, and a file state monitoring table is established. In response to a file list query request of a user, the target download file is obtained from the file server cluster and returned, including:

[0107] The target download file is uploaded to the file server cluster, the storage location information of the target download file in the file server cluster is obtained, and a corresponding file address is generated;

[0108] A file state monitoring table is established, and the file state, the file address and the file access times of the target download file are recorded. When the file access times exceed an access threshold, the target download file is marked as a hot file, and a file preloading mechanism is triggered;

[0109] A file list query request sent by a user is received. A file state list is obtained based on the file state monitoring table, and hot files are preferentially displayed. The target download file is obtained from the server cluster and returned to the user.

[0110] In this specific embodiment, the target download file uploaded by the user is received, and the file is transmitted to a preset file server cluster through a secure transmission protocol (such as HTTPS, SFTP, etc.). The file server cluster includes multiple distributed file storage servers, and the distributed storage of the file is realized through a load balancing strategy. The uploaded file is subjected to fragmentation processing, for example, a 2GB video file is divided into 20 100MB data blocks, and the hash value of each data block is calculated for verification to ensure file integrity. After the upload is completed, the storage location information of the target download file is obtained from the file server cluster, including server IP address, storage path, file fragment location, etc. Based on the above information, a unique file access address is generated, which uses the URI format, such as " / cluster / file / repository / 2023 / 07 / 15 / file_id_12345.mp4", which is used for subsequent fast positioning and access of the file.

[0111] After the upload is completed, a file state monitoring table is established to record and manage the state information of the target download file. The monitoring table uses a key-value pair data structure to store, including the following fields: file ID, file name, file type, file size, upload time, file status (such as "available", "deleted", "damaged", etc.), file address, file access times, last access time, file tags, etc. The file state is checked regularly (for example, every 5 minutes) to ensure the accessibility of the file. When the file access times exceed the preset access threshold, the target download file is marked as a hot file, for example, when a video file is accessed more than 1000 times within 1 hour, the system updates its state field to "hot file" and adds the corresponding label in the database.

[0112] The file preloading mechanism is triggered for the target download file marked as a hot file, and the specific implementation is as follows: the hot file is copied from the original storage location to the cache server, such as the edge node server distributed in different geographical locations. These edge node servers are configured with high-speed SSD storage and sufficient bandwidth resources, which can respond to user requests more quickly. The hot file is subjected to a multi-copy strategy, and the same file copy is saved on multiple service nodes to improve the concurrent performance of file access. For large files, different quality and format versions are pre-generated to meet the needs of users under different network conditions, for example, for a 4K resolution video file, 1080p, 720p, 480p, etc. versions are additionally generated, and users can automatically select the appropriate version according to the network bandwidth.

[0113] When receiving a file list query request sent by a user, relevant file information is retrieved from a file state monitoring table according to request parameters such as a file type and an upload time range, a file state list is generated based on records in the monitoring table, the list contains basic information such as a file ID, a file name, a file type, a file size, an upload time and a file state, when the list is generated, content marked as a hot file is preferentially displayed, the hot file is arranged in a front position of the list, the list is displayed through a paging mode, and by default, 20 records are displayed on each page, and the user can adjust the number of displayed records on each page through a request parameter.

[0114] After the user selects a target download file from the file list, the target file is located and acquired from a file server cluster according to a file address recorded in the file state monitoring table, for a normal file, the file is directly acquired from an original storage position, for a hot file, file content is preferentially acquired from a nearest edge node server, a breakpoint resume function is supported, the user is allowed to continue downloading from a last interruption position after interruption, and repeated transmission of a downloaded file part is avoided.

[0115] During file transmission, a download operation is recorded, a file access frequency is monitored in real time, a file access frequency field in the file state monitoring table is updated, and a hot file determination threshold is dynamically adjusted. For example, during a network peak period (for example, 18:00-22:00 every day), the hot file determination threshold is increased to 2000 accesses per hour, and during a low peak period (for example, 2:00-6:00 every morning), the hot file determination threshold is reduced to 500 accesses per hour, so that system resource allocation is optimized.

[0116] Through the above method, the embodiment realizes efficient upload, state monitoring and intelligent distribution of a target download file, especially preloading processing of a hot file, file access performance and user experience are significantly improved, load pressure of a server cluster is effectively balanced, and stability and reliability of the entire system are improved.

[0117] The asynchronous file generation and download implementation system provided in the embodiment comprises:

[0118] A first unit is configured to receive a file download request of a user, extract a data source code, create a file download task, write the file download task into a database, and generate a globally unique task identifier.

[0119] A second unit is configured to analyze the file download task based on a preset file feature recognition rule, dynamically divide the file download task into a plurality of subtask segments according to an analysis result, and allocate an independent resource quota and a processing timeout threshold to each subtask segment.

[0120] The third unit is configured to distribute the plurality of sub-task segments to a plurality of target processing nodes, generate sub-files by using a parallel call file generation method, re-distribute a corresponding sub-file generation task to other file processing nodes when detecting that the sub-file generation fails, merge all the sub-files into a target download file according to a preset merging strategy, detect the file integrity, and mark the global unique task identifier as completed.

[0121] The fourth unit is configured to upload the target download file to a file server cluster, establish a file state monitoring table, and in response to a file list query request of a user, obtain and return the target download file from the file server cluster.

[0122] In a third aspect, an electronic device is provided, including:

[0123] a processor;

[0124] a memory for storing processor-executable instructions;

[0125] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0126] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions. When the computer program instructions are executed by a processor, the method described above is implemented.

[0127] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0128] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An implementation method of asynchronous file generation download, characterized in that, The application comprises: receiving a user's file download request, extracting a data source identifier and creating a file download task, writing the file download task into a database and generating a globally unique task identifier; analyzing the file download task based on a preset file feature recognition rule, dynamically dividing the file download task into multiple sub-task segments according to the analysis result, and assigning an independent resource quota and a processing timeout threshold to each sub-task segment, including: extracting file data in the file download task, calculating the entropy value, data distribution density and time sequence correlation of the file data, and constructing a file feature vector; calculating the data block correlation degree according to the file feature vector, constructing a data block dependency graph, and calculating the ratio of the in-degree value to the out-degree value of each data block node, taking the data block node with a ratio lower than a dependency threshold as an initial segmentation node; calculating the data transmission cost between the initial segmentation nodes, selecting the initial segmentation node with a data transmission cost less than a cost threshold as an optimal segmentation node, and dynamically segmenting the file download task based on the optimal segmentation node to generate multiple sub-task segments; matching the file feature vector with historical execution task features for similarity, setting an independent processing timeout threshold for each sub-task segment based on the similarity matching result, and setting an independent resource quota in combination with its data complexity; allocating the multiple sub-task segments to multiple distributed processing nodes, sampling and calling a file generation method in parallel to generate sub-files, and when detecting that the sub-file generation fails, redistributing the corresponding sub-file generation task to other distributed processing nodes; merging all sub-files into a target download file according to a preset merging strategy, detecting the file integrity, and marking the globally unique task identifier as completed; uploading the target download file to a distributed file server cluster, responding to a user's file list query request, obtaining and returning the target download file from the distributed file server cluster.

2. The method of claim 1, wherein, receiving a user's file download request, extracting a data source encoding and creating a file download task, writing the file download task into a database and generating a globally unique task identifier, including: receiving a user's file download request and extracting a data source type identifier, a data source number, timestamp information and a digital signature therefrom to obtain a data source encoding and create a file download task; based on the data source encoding, combining user level and task queue length to calculate the priority score of the file download task, writing the file download task into a double-layer task processing structure according to the priority score, and when the number of tasks in the double-layer task processing structure reaches the upper limit, starting an asynchronous thread to write a batch of file download tasks with a priority score greater than a priority threshold into the database; generating a globally unique task identifier for the file download task according to the database write timestamp, the priority score and the data source encoding, and setting a cache expiration time.

3. The method of claim 1, wherein, allocating the plurality of sub-task segments to a plurality of target processing nodes, generating sub-files by using a parallel call file generation method, when detecting that the sub-file generation fails, re-distributing the corresponding sub-file generation task to other file processing nodes, comprising: obtaining the total amount of resources, network bandwidth capacity and current load state of each processing node, screening out candidate processing nodes that meet the resource quota requirements of the sub-task segments, and calculating the task affinity scores of each candidate processing node, constructing a candidate processing node priority sequence in descending order of the task affinity scores, and selecting a plurality of candidate processing nodes as target processing nodes, and allocating the plurality of sub-task segments to the target processing nodes; starting a file generation process on the target processing node, initializing a file write handle and a file checksum, segmenting the data of the sub-task segments according to the data block size, writing the data into the file, calculating the checksum of each data block and comparing it with the file checksum, when detecting that the data block checksum does not match, re-writing the data block, updating the file metadata information after completing the sub-file generation; monitoring the processing progress of the sub-task segments on the target processing node, when detecting that the file metadata information of the sub-task segments has not changed and the processing time has exceeded the processing timeout threshold of each sub-task segment, selecting the next processing node from the candidate processing node priority sequence for task allocation.

4. The method of claim 1, wherein, merge all sub-files into a target download file according to a preset merging strategy, and detect the file integrity, and mark the global unique task identifier as completed, comprising: obtaining the file metadata information of all sub-files, and establishing a sub-file merging sequence according to the numbering order of the file metadata information; read the data block of the sub-file according to the sub-file merging sequence, and write it into the file merging buffer for data merging, control the data merging rate by the token bucket algorithm, the capacity of the token bucket is dynamically adjusted according to the system disk write bandwidth, when the number of tokens in the token bucket is insufficient, pause the data merging operation, and reduce the reading rate of the upstream sub-file to balance the data processing pipeline; based on the sliding window mechanism, the merged data is verified in real time, the cyclic redundancy check code of the data block is calculated in the sliding window, the calculated cyclic redundancy check code is compared with the original check code of the sub-file, when detecting that the check code does not match, the sliding window is rolled back and the data merging is re-performed; read the data block that passes the verification from the merging buffer and write it into the target download file, perform integrity test on the target download file, get the verified target download file, and mark the global unique task identifier as completed.

5. The method of claim 1, wherein, upload the target download file to the file server cluster, and establish a file state monitoring table, in response to a file list query request of a user, obtain and return the target download file from the file server cluster, comprising: upload the target download file to the file server cluster, obtain the storage location information of the target download file in the file server cluster, and generate the corresponding file address; A file state monitoring table is established to record file states of the target download file, file addresses, and file access times. When the file access times exceed an access threshold, the target download file is marked as a hotspot file, triggering a file preloading mechanism. A file list query request sent by a user is received, a file state list is obtained based on the file state monitoring table, and hotspot files are preferentially displayed. Target download files are obtained from the server cluster and returned to the user.

6. A system for implementing the method according to any one of claims 1-5, characterized in that, Comprise: A first unit configured to receive a file download request from a user, extract a data source code, and create a file download task. The file download task is written into a database and a globally unique task identifier is generated. A second unit configured to analyze the file download task based on a preset file feature recognition rule, dynamically divide the file download task into multiple subtask segments according to an analysis result, and assign an independent resource quota and a processing timeout threshold to each subtask segment. A third unit configured to assign the multiple subtask segments to multiple target processing nodes, generate subfiles using a parallel file generation method, and reassign a corresponding subfile generation task to another file processing node when the subfile generation fails. All subfiles are merged into a target download file according to a preset merging strategy, and the file integrity is detected. The globally unique task identifier is marked as completed. A fourth unit configured to upload the target download file to a file server cluster, establish a file state monitoring table, and obtain and return the target download file from the file server cluster in response to a file list query request from a user.

7. An electronic device, comprising: Comprise: A processor; A memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1 to 5.

Images