An asynchronous derivation method and system

Abstract

The present disclosure provides an asynchronous export method and system, the method comprising: in response to a user submitting an export request, creating a corresponding export task; preprocessing the export task based on a preset strategy set, and storing the preprocessed export task in a task queue, the preset strategy set including a task deduplication strategy and a task decomposition strategy; based on the dynamic priority of each export task in the task queue, executing each export task in a multi-thread concurrent manner to obtain an execution result; calculating the execution progress and the remaining time of each export task and feeding back to the user end for display, and when any export task is abnormal, combining a differentiated fault tolerance strategy to retry the export task; generating an export file for the execution result of each export task and uploading the export file to a file storage system. The present disclosure realizes high-reliability, high-efficiency and perceptible asynchronous export through task decomposition, dynamic scheduling and differentiated fault tolerance.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically to an asynchronous export method and system. Background Technology

[0002] As enterprises continue to deepen their informatization process, the need to export large-scale data from business systems to support analysis, report generation, or data archiving is increasing. Faced with massive data processing, synchronous export methods are no longer sufficient to meet actual needs due to issues such as network timeouts and server resource exhaustion. Although asynchronous export methods alleviate the request blocking and immediate response problems caused by synchronous export to some extent, existing asynchronous export methods still have significant shortcomings in their implementation.

[0003] First, processing large amounts of data takes a very long time, and users cannot obtain key status information such as the processing progress and estimated completion time of the task in real time. The lack of effective feedback for a long time can easily cause user anxiety, and even lead to misjudgment and duplicate submissions, which seriously affects the user experience.

[0004] Second, the simple first-in-first-out strategy for managing export tasks lacks priority scheduling and resource isolation mechanisms. This causes time-consuming tasks to block fast tasks, and may even lead to the paralysis of the export service under high concurrency, seriously affecting system stability.

[0005] Third, there is a lack of effective fault tolerance and recovery mechanisms during task execution. When temporary failures such as database connection timeouts, network fluctuations, or brief unavailability of middleware services occur, existing solutions typically only mark the task as failed, requiring users to resubmit and execute it in full, making it impossible to recover from the breakpoint.

[0006] Fourth, there is a lack of granular control over user export behavior, making it impossible to effectively limit the number of concurrent tasks, request frequency, or the amount of data exported in a single instance. Furthermore, there is a lack of necessary defense and circuit breaker mechanisms to address user errors or malicious requests.

[0007] Therefore, there is an urgent need in this field for an asynchronous export method that can achieve stable and efficient operation in high-concurrency, high-data-volume scenarios, in order to overcome the aforementioned deficiencies in existing technologies. Summary of the Invention

[0008] This application provides an asynchronous export method and system to solve problems such as unknown progress, coarse scheduling, and poor reliability in the prior art.

[0009] In a first aspect, this disclosure provides an asynchronous export method, the method comprising: In response to a user's export request, a corresponding export task is created; The exported tasks are preprocessed based on a preset strategy set, and the preprocessed exported tasks are stored in a task queue. The preset strategy set includes a task deduplication strategy and a task decomposition strategy. Based on the dynamic priority of each exported task in the task queue, each exported task is executed sequentially in a multi-threaded concurrent manner to obtain the execution result; The execution progress and remaining time of each export task are calculated and displayed to the user. If any export task encounters an error, the export task is retried using a differentiated fault tolerance strategy. The execution results of each export task are used to generate an export file, which is then uploaded to the file storage system.

[0010] According to an asynchronous export method provided in this disclosure, the export task includes at least a route identifier, business parameters, execution information, delivery information, and management information; the execution information includes task status, execution progress, remaining time, progress context, and delivery information; the task status includes: submitted, executing, completed, failed, and canceled.

[0011] According to an asynchronous export method provided in this disclosure, the task deduplication strategy is used to remove duplicate export requests. The preprocessing of the export tasks based on a preset strategy set includes: based on the task identifier of a first export request, checking whether there is an export request with the same task identifier as the first export request; if there is a second export request with the same task identifier as the first export request, and the task status of the second export request is completed, then the export file corresponding to the second export request is associated with the first export request, and the task status of the first export request is updated to completed; if there is a second export request with the same task identifier as the first export request, and the task status of the second export request is in execution, then the execution thread corresponding to the second export request is associated with the first export request, and the task status of the first export request is updated to in execution; if there is a second export request with the same task identifier as the first export request, and the task status of the second export request is submitted, then the second export request is associated with the first export request, and the task status of the first export request is updated to submitted.

[0012] According to the asynchronous export method provided in this disclosure, the task decomposition strategy includes at least a time-based decomposition strategy, a business-based decomposition strategy, and a data source complexity-based decomposition strategy. The preprocessing of the export task based on a preset strategy set includes: if the business parameters of the export task meet the time continuity condition, then the time-based decomposition strategy is used to process the export task; if the business parameters of the export task meet the complex business condition, then the business-based decomposition strategy is used to process the export task; if the business parameters of the export task meet the multi-source heterogeneity condition, then the data source complexity-based decomposition strategy is used to process the export task.

[0013] According to an asynchronous export method provided in this disclosure, the dynamic priority is determined based on the task type, user identifier, estimated total data volume, and waiting time of the export task.

[0014] According to the asynchronous export method provided in this disclosure, the step of calculating the execution progress and remaining time of each export task and feeding it back to the user terminal for display includes: for any export task in any thread, taking the start time of the processing thread as the start time, continuously monitoring the execution time of the export task from the start time to the current time; obtaining data using a paginated query method, calculating the average processing speed based on the number of pages of data obtained and the execution time, and calculating the execution progress based on the number of pages of data obtained and the total amount of data; predicting the remaining time based on the average processing speed and the execution progress, and feeding the execution progress and the remaining time back to the user terminal for display.

[0015] According to the asynchronous export method provided in this disclosure, any exception in the export task includes execution timeout and execution failure. The execution timeout refers to the execution time of any export task being greater than its corresponding timeout threshold.

[0016] According to the asynchronous export method provided in this disclosure, the retrying of the export task in conjunction with a differentiated fault tolerance strategy includes: analyzing the root cause of any export task anomaly; if the root cause is database lock contention, then using an exponential backoff retry strategy for retrying; if the root cause is transient service unavailability, then using an immediate retry strategy for retrying; if the root cause is insufficient system resources, then using a routing retry strategy for retrying; if the root cause is a program logic error, then using a fast failure strategy to directly update the task status of the export task to failure.

[0017] According to an asynchronous export method provided in this disclosure, before creating the corresponding export task, the method further includes: validating the export request, wherein the validity verification includes at least one of data range verification, user permission verification, and resource quota verification.

[0018] Secondly, this disclosure also provides an asynchronous export system, the system comprising: Create a unit to respond to a user's submitted export request and create the corresponding export task; The management unit is used to preprocess the exported tasks based on a preset strategy set and store the preprocessed exported tasks in a task queue. The preset strategy set includes a task deduplication strategy and a task decomposition strategy. The execution unit is used to execute each exported task sequentially in a multi-threaded concurrent manner to obtain the execution result based on the dynamic priority of each exported task in the task queue. The monitoring unit is used to calculate the execution progress and remaining time of each exported task and feed it back to the user terminal for display. When any exported task encounters an abnormality, the exported task is retried in combination with a differentiated fault tolerance strategy. The storage unit is used to generate export files from the execution results of each export task and upload the export files to the file storage system.

[0019] In summary, the asynchronous export method and system disclosed herein intercept business errors and security risks at the source by performing multi-level pre-verification of export requests regarding data, permissions, and quotas. It also creates export tasks with complete context information for user requests, achieving complete decoupling between requests and processing. Through task deduplication strategies, requests with identical parameters are transformed into reuses or subscriptions to existing results, completely eliminating invalid redundant calculations and resource consumption. Task decomposition strategies transform large-scale complex tasks into multiple parallel subtasks, significantly improving the efficiency of asynchronous export. Providing users with feedback on execution progress and remaining time makes the task execution process completely transparent and predictable, eliminating user anxiety and blind waiting, greatly improving the user experience. By combining breakpoint resumption and root cause analysis-based differentiated fault tolerance strategies, meaningless retries are avoided, significantly improving the success rate and resource utilization of large-scale tasks. In conclusion, the asynchronous export method disclosed herein systematically solves the reliability, efficiency, manageability, and user experience problems faced by asynchronous data export in high-concurrency, large-data-volume scenarios, achieving stable, efficient, and intelligent asynchronous export. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a flowchart illustrating an asynchronous export method provided in this disclosure; Figure 2 This is a flowchart illustrating a method for retrying the exported task provided in this disclosure; Figure 3 This is a schematic diagram of the structure of an asynchronous export system provided in this disclosure. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0023] Figure 1 This is a flowchart illustrating an asynchronous export method provided in this disclosure. See also... Figure 1 The method includes: Step S11: In response to the user's submitted export request, create the corresponding export task.

[0024] Specifically, it is understood that existing technologies generally only perform simple validation on the export requests from the front end, allowing invalid parameters to initiate requests without request restrictions. This can lead to problems such as resource waste and data leakage. In order to avoid spam requests or unrestricted export requests from individual users, it is necessary to validate the export requests initiated by users. That is, before creating the corresponding export task, the method further includes: validating the export request. The validity validation includes at least one of data range validation, user permission validation, and resource quota validation.

[0025] The data range validation is used to determine whether the time range is reasonable. For example, it checks whether the end time selected by the user in the time field is not earlier than the start time, or whether the department selected in the department field exists. This avoids meaningless requests occupying the task queue and wasting thread resources.

[0026] The permission verification is used to determine whether the user initiating the export request has the authority to export the requested data range. Performing permission verification when a user initiates an export request is more effective than simply verifying permissions during user login, preventing unauthorized exports, avoiding data leaks, and improving data security.

[0027] The resource quota verification is used to check whether the current export has reached the export limit, such as the limit for concurrent export tasks or the daily export limit. When a user unintentionally or even maliciously submits multiple large-scale export requests in a short period of time, consuming all processing threads, it may prevent other users from using and exporting data normally. Therefore, to ensure that a few users do not monopolize the system, it is advisable to limit the number of concurrent tasks and daily requests for a single user. In other words, to prevent resource abuse, the system sets an export limit for users or their departments / organizations.

[0028] Specifically, it can also be understood that, in order to ensure the immediate response of user interactions and the overall stability of system services, after a user submits an export request, the backend will create an export task and execute it, so that the backend can perform asynchronous queries based on the export task, reducing the user's waiting time and decoupling the request response from the task execution.

[0029] The export request is submitted by the user through the browser interface or application programming interface (API). The export request is structured data, including parameters such as data range, filtering conditions, export file format, and the identity of the submitting user.

[0030] The export task is a structured data entity within the system used to carry, manage, and execute user export requests. It decouples request response from task execution by connecting the front-end user export intent with the back-end multi-threaded business execution. The export task is typically created after verifying the export request. If verification passes, a task identifier is immediately generated, and an export task is created. Each export request and export task has a unique task identifier. To ensure that the back-end thread can accurately identify the user's export request based on the export task, the export task must not only contain complete parameter information of the export request but also all necessary information for task execution and management. That is, the export task must at least include a route identifier, business parameters, execution information, delivery information, and management information.

[0031] The routing identifier represents the unique identifier and basic attributes of the export task, including: task identifier, task description, user identifier, and creation time. The task identifier is a unique string generated when the export task is created, used to track the export task throughout its lifecycle. The task identifier directly originates from the user's export request and can generally be generated using a UUID (Universally Unique Identifier) ​​or the Snowflake algorithm. The task description is used to describe the name or summary of the export task, facilitating quick identification and understanding of the current export task, such as "2024 Sales Report Export". The user identifier refers to the identity of the user who submitted the export request. The creation time refers to the time when the export task was successfully created and the persisted system time.

[0032] The business parameters represent the content exported by the export task, that is, who executes the export task and what it specifically executes, including: business identifier, service identifier, export parameter set, output configuration, etc. The business identifier is a data-level location identifier, indicating the data range during the execution of the export task. It can be a single dataset or multiple datasets, such as an order list (order_id) or a user list (user_id). The service identifier is a system-level routing identifier, indicating the backend business service or processor responsible for handling the export task. The export parameter set is a structured container that stores all query and filtering conditions parsed from the user's export request, such as start date, end date, or filtering conditions {field, operator, value}. The output configuration defines the specifications of the exported file, including but not limited to: file format, encoding, list of included fields, file naming rules, etc. The file format includes plain text CSV, report Excel, or document PDF formats. The encoding generally chooses a character encoding scheme. The list of included fields allows users to choose to export only some fields. The file naming rules can be a fixed static prefix plus a changing dynamic suffix, such as adding the export date to a sales report, or adding the user identifier to a sales report.

[0033] The execution information indicates the execution status of the export task, primarily providing users with progress awareness information, including: task status, execution progress, remaining time, progress context, and delivery information. The task status refers to the stage of the export task in its lifecycle, including: submitted, executing, completed, failed, and canceled. When the task status is submitted, it indicates that the export task has been successfully created and is waiting to enter the task queue; when the task status is executing, it indicates that the export task has been scheduled and executed by the backend thread; when the task status is completed, it indicates that the export task has successfully completed the asynchronous export of data; when the task status is failed, it indicates that the export task encountered a problem and was interrupted during execution; when the task status is canceled, it indicates that the export task actively stopped exporting during execution. The execution progress indicates the current progress of the export task, expressed as a progress bar or numerically, with the numerical form representing the execution progress as a percentage. The remaining time indicates the remaining execution time of the current export task. The progress context represents the context information needed to calculate progress or predict the remaining time, such as the estimated total amount of data corresponding to the exported task and the amount of data already processed. The amount of data already processed can be actively obtained and updated after pagination or after a phase is completed, such as determining the amount of data already processed based on the number of pages of data obtained and the amount of data per page.

[0034] The delivery information represents delivery results and recorded issues, including file access information, result information, and exception information. The file access information indicates the file's storage path, download address, or temporary download token. The exported file refers to the result file where the exported data is stored after the export task is completed. The file access information includes access authorization and timeliness control. Access authorization means that only the user who initiated the export request can access the exported file. Timeliness control refers to the cache validity period of the exported file. Generally, the cache validity period for sensitive data is shorter, while the cache validity period for regular data is longer. For example, exported files containing user personal information are retained for 3 days, while sales reports are retained for 30 days. In other embodiments, timeliness control can also be a fixed duration, or it can be set according to the business type and user role corresponding to the export task; this disclosure does not limit this. The result information refers to metadata about the export results, including breakpoint information, the size of the exported file, and the generation time of the exported file. The exception information refers to the exception information recorded when an exception occurs in the export task. For example, when the task status of any export task is failed, the error code, error description, stack trace, or root cause analysis summary that caused the failure needs to be recorded in detail in the exception information section.

[0035] The management information is used to record the dynamic priority and timeout threshold of each exported task. The dynamic priority of each exported task is assigned based on the task type, user level, or urgency level, while the timeout threshold is the maximum allowed execution time of the exported task, which can be a system default value or a value dynamically calculated based on the task complexity.

[0036] In the above method, by performing pre-emptive, multi-layered validity verification on export requests, potential business errors, security vulnerabilities, and resource abuse risks are intercepted at the task submission stage, preventing invalid or malicious tasks from entering the execution queue and consuming system resources. By creating self-contained, self-descriptive, and manageable export task entities from simple user-initiated export requests, complete decoupling of the system architecture is achieved. Each export task carries its complete execution context, which allows task queuing, scheduling, execution, and monitoring to operate independently and scale horizontally.

[0037] Step S12: Preprocess the exported task based on a preset strategy set, and store the preprocessed exported task in a task queue. The preset strategy set includes a task deduplication strategy and a task decomposition strategy.

[0038] Specifically, to prevent duplicate export requests due to user error, repeated clicks, or script errors, or in collaborative scenarios where different users request the same data, this disclosure proposes a task deduplication strategy. A unique "task identifier" is determined based on all parameters of the user's export request. When a new request arrives, its task identifier is calculated first. If the same task identifier exists in the cache and the corresponding export file has already been completed, the generated file link is directly returned to the new user and associated with the same task record. If the export task is still in progress, the new request waits for and subscribes to the result of the same export task.

[0039] In some embodiments, the preset strategy set includes a task deduplication strategy, which is used to remove duplicate export requests. The strategy can determine whether two export requests are completely identical by comparing the task identifiers of the export tasks. The preprocessing of the export tasks based on the preset strategy set includes: Step a1: Based on the task identifier of the first export request, check if there is an export request with the same task identifier as the first export request; Specifically, the first export request is the export request corresponding to the current export task; the task queue stores all export tasks waiting to be processed, the multi-threaded system stores all export tasks currently being executed, and the file storage system caches the export files of each export task. Each export file corresponds to a unique task identifier. Therefore, the search for whether there is an export request with the same task identifier as the first export request can be performed by searching the file storage system for a cached export file with the same task identifier as the first export request, searching the multi-threaded system for a currently executing export task with the same task identifier as the first export request, or searching the task queue for a waiting export task with the same task identifier as the first export request.

[0040] Step a2: If there is a second export request with the same task identifier as the first export request, and the task status of the second export request is completed, then associate the exported file corresponding to the second export request with the first export request, and update the task status of the first export request to completed. Specifically, when the task identifier of the first export request is the same as that of the second export request, it indicates that the two export requests are export requests with completely identical parameters. Furthermore, if the task status of the second export request is "completed", then it is determined that the first export request is a duplicate request and the previous identical export request "second export request" has been completed. Instead of storing the export task corresponding to the first export request in the task queue, the export file corresponding to the second export request is directly associated with the first export request, and the task status of the first export request is updated to "completed", allowing users to directly and quickly obtain the export results and improving the user experience.

[0041] Step a3: If there is a second export request with the same task identifier as the first export request, and the task status of the second export request is in execution, then associate the execution thread corresponding to the second export request with the first export request, and update the task status of the first export request to in execution. Specifically, when the task identifier of the first export request is the same as that of the second export request, it indicates that the two export requests have completely identical parameters. Furthermore, if the task status of the second export request is "in execution", then it is determined that the first export request is a duplicate request and the previous identical export request "second export request" is currently being executed. Instead of storing the export task corresponding to the first export request in the task queue, the execution thread corresponding to the second export request is directly associated with the first export request, and the task status of the first export request is updated to "in execution". This allows users to quickly obtain the export results based on the currently executing thread, improving the user experience.

[0042] Step a4: If there is a second export request with the same task identifier as the first export request, and the task status of the second export request is submitted, then associate the second export request with the first export request, and update the task status of the first export request to submitted. Specifically, when the task identifier of the first export request is the same as that of the second export request, it indicates that the two export requests have completely identical parameters. Furthermore, if the task status of the second export request is "submitted", then it is determined that the first export request is a duplicate request and the previous identical export request "second export request" has just been submitted. Instead of storing the export task corresponding to the first export request in the task queue, the second export request is directly associated with the first export request, and the task status of the first export request is updated to "submitted". This allows the user to simultaneously execute the export task and generate the export file based on the subsequent task execution and export file generation of the second export request, thus improving the user experience.

[0043] Specifically, it can also be understood that, in order to improve the processing efficiency of complex or large-scale export tasks, this disclosure proposes a task decomposition strategy. This strategy intelligently splits a potentially large or complex export task into smaller, more easily scheduled subtasks that can be executed independently or in parallel, thereby improving the processing efficiency and reliability of the export task. The task decomposition strategy can perform horizontal decomposition based on overlapping time windows, vertical decomposition based on overlapping business dimensions, or vertical decomposition based on the complexity of the data source; this disclosure does not limit the specific decomposition in these respects.

[0044] In some embodiments, the preset strategy set includes task decomposition strategies, which at least include time-based decomposition strategies, business-based decomposition strategies, and data source complexity-based decomposition strategies. The preprocessing of the exported task based on the preset strategy set includes: Step b1: If the business parameters of the exported task meet the time continuity condition, then the exported task is processed using a time-dimension-based decomposition strategy. Specifically, the business parameters include: business identifier, service identifier, export parameter set, output configuration, etc. The export parameter set is a structured container that stores all query and filtering conditions parsed from the user's export request. The time continuity condition refers to the data query range covering a continuous time period, such as one year. The time-based decomposition strategy for processing the export task can be used to divide the continuous time period into multiple continuous sub-time periods and create a parallel sub-task for each sub-time period. Alternatively, a monthly sharding strategy can be used for horizontal decomposition. For example, for the export task of "getting orders for the whole year of 2024", monthly sharding will generate 12 sub-tasks to process data for January, February, ..., December 2024, respectively. A weekly sharding strategy can also be used for horizontal decomposition, or the time interval can be determined based on the estimated total amount of data. This disclosure does not limit this.

[0045] Step b2: If the business parameters of the exported task meet complex business conditions, then the exported task is processed using a business dimension-based decomposition strategy. Specifically, the complex business conditions refer to the fact that the exported task involves multiple independent business categories. The decomposition strategy based on business dimensions for processing the exported task can be to split the exported task into multiple sub-tasks that process data from different categories according to the business category dimension. For example, if an exported task involves multiple dimensions such as region (e.g., North China, East China), product line (e.g., mobile phones, computers), and customer level, then the exported task can be split into multiple sub-tasks based on a list of all enumerated values ​​or specified values ​​under a certain dimension. For example, it can be split into multiple sub-tasks based on region, such as North China, East China, etc., or based on product line, such as mobile phones, computers, etc.

[0046] Step b3: If the business parameters of the exported task meet the multi-source heterogeneity condition, then the exported task is processed using a decomposition strategy based on the complexity of the data source. Specifically, the multi-source heterogeneous condition refers to the situation where the export task needs to obtain data from multiple different channels. The decomposition strategy based on data source complexity can be used to process the export task, which may involve splitting the export task into multiple sub-tasks that each obtain data from a single data source. For example, the export task for a "customer order details report" needs to link the order table in the order database, the user information table in the user database, and the product information table provided by the product service. In this case, it can be split into three sub-tasks: querying order data from the order database, querying user information from the user database, and retrieving product information from the product service.

[0047] In the above embodiments, after the exported task is processed by the decomposition strategy based on the time dimension, the decomposition strategy based on the business dimension, or the decomposition strategy based on the complexity of the data source, it is also necessary to assign a globally unique task identifier to the sub-tasks obtained by splitting them, and the task identifier of the sub-task needs to be associated with the task identifier of its parent task, so as to facilitate the subsequent merging of the execution results of each sub-task into the exported file of the parent task.

[0048] In other embodiments, after preprocessing the exported task based on a preset strategy set and before storing the preprocessed exported task in a task queue, the method further includes: Step c1: In the case of adding new export tasks, determine whether there are any overlapping tasks among the newly added export tasks; Specifically, the newly added export task refers to multiple sub-tasks obtained after splitting an export task according to a time-based decomposition strategy, a business-based decomposition strategy, or a data source complexity decomposition strategy. Cross-tasks refer to whether there are any identical sub-tasks in different export tasks. For example, the export task "Sales data from January 1 to January 31, 2024" and the export task "Sales data from January 15 to February 15, 2024" both have the same "data for January 2024" in their sub-tasks that are split by month. Similarly, the export task "Sales report of all products in East China" split by product line and the export task "Sales report of smartphone category in China" split by region both have the same "Sales report of smartphone category in East China" in their sub-tasks that are split by region.

[0049] Step c2: If there are cross-tasks, each cross-task group will retain only one newly added export task in the task queue, and the rest will be associated and bound to that newly added export task. Specifically, when there are overlapping tasks, that is, when there are identical subtasks, only one subtask can be kept, and the other identical subtasks can be associated with that subtask. The result data can be read directly based on the execution result of that subtask, thus reducing resource consumption.

[0050] In other embodiments, a task decomposition strategy may be used only for exporting extremely large single tables, such as an export request for "exporting all orders from the past 5 years". The preset strategy set includes task decomposition strategies. Before preprocessing the export task based on the preset strategy set, the method further includes: Step d1: Estimate the total number of data in the exported files of the exported task to obtain the estimated total data volume. Specifically, the total data volume is estimated by forecasting the total number of orders over 5 years. This estimation typically employs a "fast pre-query" method, such as a rapid estimation based on database statistics, which can quickly obtain an approximate value of the data volume at extremely low cost.

[0051] Step d2: If the total amount of data exceeds a preset threshold, the exported task is preprocessed based on the task decomposition strategy. The preset threshold can be 1 million records, or it can be set according to actual needs; this disclosure does not limit this. If the total data volume of all orders over 5 years exceeds the preset threshold, it can be broken down into multiple sub-tasks based on the data volume. These sub-tasks are completely independent and can be executed in parallel, greatly shortening the total execution time. Alternatively, it can be segmented by time range, dividing the time-sorted data into years, months, and weeks. For example, exporting 5 years of data can be directly split into 5 sub-tasks by year.

[0052] In the above method, identical duplicate requests are intercepted before task execution and converted into subscriptions to or direct reuse of existing task results, eliminating invalid duplicate calculations, database queries, and file generation. By intelligently splitting a single large-scale or complex task into multiple independently executable subtasks, long-running tasks can be executed in parallel on multiple threads, reducing end-to-end processing time from linear accumulation to almost the time required to process the slowest subtask, achieving an order-of-magnitude improvement in processing speed. After task decomposition, overlapping tasks between different derived tasks are identified, reducing the number of tasks enqueued, avoiding duplicate calculations, and greatly improving resource reuse and overall system efficiency.

[0053] Step S13: Based on the dynamic priority of each exported task in the task queue, execute each exported task sequentially in a multi-threaded concurrent manner to obtain the execution result.

[0054] Specifically, it can be understood that the dynamic priority is determined based on the task type, user identifier, estimated total data volume, and waiting time of the export task. The task type indicates the classification of the task, such as report task export and ordinary task export, simple task export and complex task export, real-time task export and scheduled task export, etc.; the user identifier is the identifier of the person who submitted the export request; the estimated total data volume is the data volume in the exported file estimated based on the export task; and the waiting time refers to the waiting time of the current export in the task queue.

[0055] In some embodiments, the dynamic priority determined based on the task type, user identifier, estimated total data volume, and waiting time of the exported task can be achieved using a multi-level queue method. Multiple sub-queues with different priorities are pre-set. First, tasks are routed to different sub-queues based on task type and user identity. Then, within each sub-queue, tasks are sorted based on the estimated total data volume and waiting time. In this case, the dynamic priority determined based on the task type, user identifier, estimated total data volume, and waiting time of the exported task specifically includes: Step S131: Create a high-priority queue, a normal-priority queue, and a low-priority queue; Specifically, the task queue can be split into a high-priority queue, a normal-priority queue, and a low-priority queue, or the high-priority queue, normal-priority queue, and low-priority queue can be reconstructed, and then exported tasks can be retrieved from the task queue and sent to the high-priority queue, normal-priority queue, or low-priority queue respectively.

[0056] Step S132: Assign each exported task to one of the following priority queues: high priority queue, normal priority queue, or low priority queue, based on the task type and user identifier. Specifically, export tasks that serve high-level decision-making can be assigned higher priority, as can export tasks submitted by high-value users. That is, after determining the user's level based on their identifier, the dynamic priority of export tasks from member users is higher than that from ordinary users; the priority of report export tasks is higher than that of ordinary export tasks; the priority of real-time export tasks is higher than that of scheduled export tasks; and simple export tasks also have a higher priority. Taking the task types including report export tasks and ordinary export tasks as an example, the process of assigning each export task to one of the following priority queues—high-priority queue, ordinary-priority queue, and low-priority queue—based on the task type and user identifier can include: storing member users' report export tasks in the high-priority queue, member users' ordinary export tasks in the ordinary-priority queue, ordinary users' report export tasks in the ordinary-priority queue, and ordinary users' ordinary export tasks in the low-priority queue.

[0057] Step S133: In the same priority queue, sort the export tasks in the current priority queue according to the estimated total data volume and waiting time; Specifically, export tasks with smaller estimated total data volumes have higher priority than those with larger estimated total data volumes. Prioritizing export tasks with smaller data volumes can reduce the average waiting time. For a regular export task that has been waiting in the queue for a long time, its priority can be gradually increased over time to prevent starvation. For example, within the high-priority queue, tasks can be sorted by estimated total data volume from smallest to largest. If the waiting time is too long, the task can be promoted to a higher-level queue, such as directly from the regular priority queue to the high-priority queue.

[0058] In other embodiments, the dynamic priority determined based on the task type, user identifier, estimated total data volume, and waiting time of the exported task can be achieved by using a weighted scoring method. This method assigns weights and scores to the task type, user identifier, estimated total data volume, and waiting time, calculates a comprehensive priority score using a weighted sum formula, and sorts the queue based on this score.

[0059] Specifically, it can also be understood that the export task stored in the task queue after step S12 may be the original export task or multiple split export tasks. If an export task has a parent task, after determining the dynamic priority based on the export task's task type, user identifier, estimated total data volume, and waiting time, in order to ensure that multiple export tasks belonging to the same parent task can be completed in the shortest possible time, the dynamic priority can also be fine-tuned based on the real-time status of a group of export tasks corresponding to a parent task. That is, for export tasks that are executed in parallel in a group of export tasks, when most of the export tasks have been completed and only a few export tasks are slow to execute due to some reason (such as data skew), the system can temporarily increase the priority of these "delayed" export tasks to accelerate the completion of the group of export tasks.

[0060] In some embodiments, for a group of exported tasks belonging to the same parent task, the dynamic priority of each exported task is fine-tuned according to the real-time status of each exported task. Specifically, this includes: monitoring the execution status of any exported task in the current group, and when the difference in execution progress of all exported tasks in the group exceeds a preset difference threshold, automatically increasing the dynamic priority of the exported task with lagging progress.

[0061] In other embodiments, for export tasks with parent tasks, a method of "determining a dynamic priority based on the export task's task type, user identifier, estimated total data volume, and waiting time" can be used to determine a priority for the parent task. Then, each export task automatically inherits the priority of its parent task. That is, when a "high-priority" export task is split into multiple export tasks, the split export tasks are considered high-priority. In addition, if the export tasks in the task queue are interleaved tasks, their dynamic priority is the higher of the priorities of their respective parent tasks. That is, when a "low-priority" first subtask and a "high-priority" second subtask are interleaved tasks, the export task in the task queue has a high priority.

[0062] Specifically, it can also be understood that the execution of at least two export tasks in a multi-threaded concurrent manner can be achieved by retrieving high-priority export tasks from the task queue based on the dynamic priority of each export task, and then using multi-threading to schedule and execute these export tasks. Specifically, the execution can be achieved by waking up the corresponding thread to perform business processing based on the service identifier and business identifier of each export task, transmitting the query parameters of the user request, and marking the task status as executing.

[0063] In some embodiments, to avoid multi-threaded load, it is also necessary to limit the number of concurrent threads. That is, the execution of at least two export tasks in a multi-threaded concurrent manner also includes: counting the number of export tasks being executed concurrently. If the number of concurrently executed export tasks is less than a preset threshold, the multi-threaded business execution program is invoked in real time. Otherwise, the export task needs to enter the task queue to continue waiting until there are available resources before the multi-threaded business execution program is invoked to execute the export task.

[0064] It should be noted that the execution result refers to the data obtained by each thread when processing the export task; during the execution of each export task, it is necessary to periodically mark the breakpoint information of the successfully processed data boundary and persist it to the delivery information associated with the export task, the delivery information including the breakpoint information.

[0065] The above method, by comprehensively considering multi-dimensional information such as task type, user identity, estimated data volume, and waiting time, can dynamically and fairly assign priorities to each exported task; for exported tasks obtained after splitting, the priorities can be fine-tuned according to their respective execution status to ensure the coordination between a group of exported tasks; by actively limiting the number of concurrent threads, overload is prevented; and by recording breakpoint information, redundant calculations are reduced for subsequent retries, greatly improving the success rate and execution efficiency of large-scale tasks.

[0066] Step S14: Calculate the execution progress and remaining time of each exported task and display them to the user terminal. If any exported task encounters an error, the exported task will be retried using a differentiated fault tolerance strategy.

[0067] Specifically, it is understood that in the prior art, when exporting data, users cannot obtain key status information such as the processing progress of the task in real time. In order to make it easier for users to perceive the real-time export status of the export task, this disclosure proposes a method to monitor the execution thread in real time during the export task process and obtain the execution progress and remaining time of each export task.

[0068] In some embodiments, calculating the execution progress and remaining time of each exported task and displaying it to the user includes: Step S141: For any export task in any thread, take the start time of the processing thread as the start time, and continuously monitor the execution time of the export task from the start time to the current time. Specifically, each export task is executed concurrently in a multi-threaded manner, with each thread monitoring the export tasks it is processing.

[0069] Step S142: Data is obtained using a paginated query method. The average processing speed is calculated based on the number of pages of data obtained and the execution time. The execution progress is calculated based on the number of pages of data obtained and the total amount of data. Specifically, the total data volume is the total data volume estimated for each export task based on a fast estimation method using database statistics. During execution, data is obtained using a paginated query method, that is, the obtained data is written to the target export file, and a record is made once each page is completed. In this way, the current amount of data processed can be known by counting the number of pages of data obtained. Furthermore, the current execution progress can be determined by the ratio of the amount of data processed to the total amount of data.

[0070] Step S143: Based on the average processing speed and the execution progress, predict the remaining time, and feed back the execution progress and the remaining time to the user terminal for display.

[0071] Specifically, the average processing speed is calculated based on the current amount of data processed and the execution time. In other embodiments, it can also be calculated by combining historical processing data. The remaining time refers to how much time is expected to take for the export task to complete. The remaining amount can be calculated first based on the estimated total amount of data and the number of pages of data to be acquired, and then the remaining time can be calculated based on the remaining amount and the average processing speed. Alternatively, the remaining time can be predicted by combining historical data to achieve more accurate progress prediction. This disclosure does not limit this.

[0072] More specifically, the execution progress and remaining time are fed back to the user for display. This can be achieved by the user client receiving the feedback on the execution progress and remaining time, and then dynamically updating the interface components, such as displaying the execution progress as a progress bar or as a percentage value, while simultaneously displaying the remaining time near the progress bar or in other areas of the interface. This allows the user to perceive the specific progress of the export task being processed in the background in near real-time. For export tasks with a parent task, that is, a group of export tasks belonging to the same parent task, the execution progress and remaining time of each export task in the group can be fed back to the user for display. Alternatively, the total execution progress and total remaining time can be calculated based on the execution progress and remaining time of each export task in the group, and then the total execution progress and total remaining time can be fed back to the user for display. Then, a detailed progress button can be set up to trigger the display of the execution progress and remaining time of each export task. This disclosure does not limit this approach.

[0073] Specifically, it is also understandable that thread errors may occur due to imprecise business execution program code or environmental fluctuations, but these errors are not explicitly reported. This can cause the export task's status to remain in execution, even though the actual business execution program has already exited with an error. To avoid this problem, this disclosure also sets a timeout threshold for each export task to monitor its execution status. By monitoring the execution time of the business execution program on each thread, if the execution time exceeds the preset timeout threshold and the task has not been completed or has not actively reported an error, the task is considered to be abnormal, i.e., it has failed or timed out without response.

[0074] Any anomalies encountered in the export task include execution timeout and execution failure. Execution timeout refers to the execution time of any export task exceeding its corresponding timeout threshold. The initial default value of the timeout threshold can be set to 30 minutes, and then adjusted according to the average processing time during actual execution. For example, the timeout threshold = (number of data records to be downloaded in the current export task / 1000) * average execution time per thousand data records of historical tasks * (1 + 30%).

[0075] Specifically, it can also be understood that the retrying of the exported task using a differentiated fault-tolerance strategy includes: analyzing the root cause of any exported task's anomaly; if the root cause is database lock contention, then using an exponential backoff retry strategy; if the root cause is transient service unavailability, then using an immediate retry strategy; if the root cause is insufficient system resources, then using a routing retry strategy; and if the root cause is a program logic error, then using a fast failure strategy to directly update the exported task's status to failure.

[0076] The differentiated fault tolerance strategies include exponential backoff retry strategy, immediate retry strategy, route retry strategy, and route retry strategy, all of which work in conjunction with the breakpoint resumption mechanism. These strategies periodically persist breakpoint states during task execution. When a retry is triggered, the new execution instance first loads the breakpoint state and reconstructs the query or computation context based on this state, ensuring that each retry starts from the previous successful completion point, not the task's starting point. In other words, the retry supports resuming from breakpoints. The retry includes: reading the breakpoint information recorded by the exported task before retrying the exported task; and continuing execution of the exported task from the data position identified by the breakpoint information.

[0077] In the above method, by calculating and displaying the execution progress and remaining time of the exported task in real time, users can clearly perceive the actual processing status of the exported task, thus improving the user experience. By adopting a differentiated fault tolerance strategy for retries, different retry methods are selected for different types of execution exceptions, rather than blindly repeating, ensuring that the system significantly improves the task success rate and system resource utilization.

[0078] Step S15: Generate export files from the execution results of each export task and upload the export files to the file storage system.

[0079] Specifically, it can be understood that in a file storage system, a file is created for each export task to store the exported data, such as creating a new Excel file. Since some export tasks are actually subtasks split into smaller ones, when storing export files, each export file can be stored separately first and then merged into a single file. That is, each export task has a corresponding export file to store the execution results. For a group of export tasks belonging to the same parent task, the export files of each export task within the group need to be merged into the same file as the final export result fed back to the user. Alternatively, when storing export files, the execution results of a group of export tasks belonging to the same parent task can be directly stored in the same file. This requires creating a file for export tasks without a parent task and a file for a group of export tasks belonging to the same parent task, and then writing the execution results of each export task within the group into this export file.

[0080] In some embodiments, generating export files from the execution results of each export task typically employs paginated querying and streaming write techniques; that is, the exported data is streamed to an Excel file in a local temporary directory. This process is synchronized with progress updates to avoid loading all data into memory at once, thus preventing memory overflow. Once all data for the entire task has been successfully written to the Excel file, the task status of the export task is updated to "Completed," and the temporary or final storage path of the file is recorded. A time-sensitive or access-restricted download link is displayed on the user's client interface, allowing the user to download the export file at any time within the validity period.

[0081] The file storage system provides persistent file storage services, such as cloud storage S3, OSS, or a distributed file system. The timeliness of the file storage system refers to the caching time of the exported files corresponding to each export task. Generally, sensitive data is stored for a shorter time, while regular data is stored for a longer time. For example, exported files containing user personal information are retained for 3 days, while sales reports are retained for 30 days. The duration can also be fixed or determined based on the business type and user role corresponding to the export task; this disclosure does not impose any limitations.

[0082] The aforementioned asynchronous export method intercepts business errors and security risks at the source by performing multi-level pre-verification of export requests, including data, permissions, and quotas. It creates export tasks with complete context information for user requests, achieving complete decoupling between requests and processing. A task deduplication strategy transforms requests with identical parameters into reuses or subscriptions to existing results, completely eliminating unnecessary redundant calculations and resource consumption. A task decomposition strategy transforms large-scale complex tasks into multiple parallel subtasks, significantly improving the efficiency of asynchronous export. Providing users with feedback on execution progress and remaining time makes the task execution process completely transparent and predictable, eliminating user anxiety and blind waiting, greatly improving the user experience. Combining breakpoint resumption and root cause analysis-based differentiated fault tolerance strategies avoids meaningless retries, significantly improving the success rate and resource utilization of large-scale tasks. In summary, the asynchronous export method disclosed herein systematically solves the reliability, efficiency, manageability, and user experience problems faced by asynchronous data export in high-concurrency, large-data-volume scenarios, achieving stable, efficient, and intelligent asynchronous export.

[0083] Figure 2 This is a flowchart illustrating a method for retrying the exported task provided in this disclosure. The retry supports resuming from a breakpoint. The retry includes: reading the breakpoint information recorded during the execution of the exported task, and continuing execution from the point of interruption based on the breakpoint information, so as to realize resuming from a breakpoint.

[0084] See Figure 2 The method includes: Step S21: Analyze the root cause of any abnormality in the exported task.

[0085] Step S22: If the root cause is database lock contention, then an exponential backoff retry strategy is adopted for retrying.

[0086] Specifically, database lock contention refers to the phenomenon where multiple transactions or threads simultaneously request locks on the same resource (such as data rows, tables, etc.), causing some transactions to wait for the lock to be released before they can continue execution. This can be determined by analyzing database monitoring logs and wait events. If the execution thread of a certain exported task experiences an anomaly, and a query reveals that there are indeed "waiting for locks" and blockers, then it can be determined that the root cause of the anomaly is database lock contention. The exponential backoff retry strategy refers to retrying after a certain period of time to avoid escalating contention. In particular, there is a first delay before the first retry, and if it fails again, there is a second, even longer delay, such as 2 seconds, 4 seconds, and 8 seconds before retrying.

[0087] Step S23: If the root cause is transient service unavailability, then an immediate retry strategy is adopted to retry.

[0088] Specifically, the transient service unavailability refers to a transient timeout of an external interface. This can generally be determined by checking the health status and network connectivity of the external interface. For example, through error snapshot analysis, if the error message is found to be a clear network anomaly, web client anomaly, or distributed communication framework anomaly, then the root cause can be preliminarily determined to be transient service unavailability. The immediate retry strategy refers to immediately retrying 1-2 times. This is because transient service unavailability may only be caused by network jitter. If it can be executed normally after 1-2 retries, it can be confirmed that it is a transient failure. If it still fails and an error is not responded after more than 2 retries, then the table name failure may persist. It can be determined that the root cause is the continuous unavailability of the external service, and then a fast failure strategy can be used to directly update the task status of the exported task to failure.

[0089] Step S24: If the root cause is insufficient system resources, then a route retry strategy is adopted for retrying.

[0090] Specifically, insufficient system resources refer to excessively high memory usage. This can be determined by checking the actual memory usage of each thread to see if the execution anomaly is caused by insufficient system resources. For example, if during the period when the export task anomaly occurs, the utilization rate of one or more key resources on the node where the export task resides consistently reaches or approaches a saturation threshold, then the root cause can be identified as insufficient system resources. The execution routing retry strategy refers to reassigning the task to a thread with lower load or more abundant resources for execution. That is, the export task is republished to the task queue and reassigned to another thread with lower load in the multi-threaded process.

[0091] Step S25: If the root cause is a program logic error, then a fast failure strategy is adopted to directly update the task status of the exported task to failure.

[0092] Specifically, the program logic errors include typical characteristics such as logical infinite loops, business rule defects, missing data exception handling, etc. Generally, the existence of program logic errors can be determined by analyzing the task execution log. If an abnormal export task has no progress updates for a long time during execution and the memory usage continues to increase linearly or exponentially, the task status of the export task is directly marked as failed, the error log is recorded, and the administrator is notified to check the code logic.

[0093] In some embodiments, each export task has a corresponding timeout threshold, and the execution timeout refers to the execution time of any export task exceeding its corresponding timeout threshold. However, if an export task times out but its execution progress is constantly being updated and the remaining time is very short, it is advisable to appropriately relax the timeout threshold for the export task. That is, the estimated total data volume and the actual processed data volume of the export task can be analyzed first. If the current average processing speed is fast or the remaining time is short, the timeout threshold of the export task can be extended to allow it to continue to complete the execution.

[0094] The aforementioned method for retrying the exported task utilizes a differentiated fault-tolerance strategy based on root cause analysis. This approach selects the optimal recovery path for different types of execution anomalies, avoiding blind repetition and preventing meaningless retries for logical errors or immediate retrying for resource contention. This significantly improves task success rate and system resource utilization. Furthermore, by retrying at breakpoints, it ensures that even in severe situations such as process interruption, the existing results of large-scale tasks are not lost; tasks can simply resume from the breakpoint, greatly enhancing the reliability and execution efficiency of large-scale data export tasks.

[0095] Figure 3 This is a schematic diagram of the structure of an asynchronous export system provided in this disclosure. See also... Figure 3 The system 300 includes: Create unit 310 to create a corresponding export task in response to a user's submitted export request; Management unit 320 is used to preprocess the exported tasks based on a preset strategy set and store the preprocessed exported tasks in a task queue. The preset strategy set includes a task deduplication strategy and a task decomposition strategy. The execution unit 330 is used to execute each exported task sequentially in a multi-threaded concurrent manner to obtain the execution result based on the dynamic priority of each exported task in the task queue. The monitoring unit 340 is used to calculate the execution progress and remaining time of each exported task and feed it back to the user terminal for display. When any exported task encounters an abnormality, the exported task is retried in combination with a differentiated fault tolerance strategy. Storage unit 350 is used to generate export files from the execution results of each export task and upload the export files to the file storage system.

[0096] For a detailed description of the asynchronous export system described above, please refer to the description of the relevant method steps in the above embodiments; repeated details will not be repeated. The embodiments of the asynchronous export method and system described above are merely illustrative. The "units" and "modules" used as separate components can be combinations of software and / or hardware that implement a predetermined function, and may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and not to limit them; although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this invention.

Claims

1. An asynchronous export method, characterized in that, The method includes: In response to a user's export request, a corresponding export task is created; The exported tasks are preprocessed based on a preset strategy set, and the preprocessed exported tasks are stored in a task queue. The preset strategy set includes a task deduplication strategy and a task decomposition strategy. Based on the dynamic priority of each exported task in the task queue, each exported task is executed sequentially in a multi-threaded concurrent manner to obtain the execution result. The dynamic priority is determined according to the task type, user identifier, estimated total data volume and waiting time of the exported task. After determining the dynamic priority, for a group of exported tasks belonging to the same parent task, when the difference in execution progress of all exported tasks in the group exceeds a preset difference threshold, the dynamic priority of the exported task with the lagging progress is increased. The execution progress and remaining time of each export task are calculated and displayed to the user. If any export task encounters an anomaly, a differentiated fault tolerance strategy is used to retry the task. This retrying strategy includes: analyzing the root cause of any export task's anomaly; if the root cause is database lock contention, an exponential backoff retry strategy is used; if the root cause is transient service unavailability, an immediate retry strategy is used; if the root cause is insufficient system resources, a routing retry strategy is used; and if the root cause is a program logic error, a fast failure strategy is used to directly update the task status of the export task to failure. The execution results of each export task are used to generate an export file, which is then uploaded to the file storage system.

2. The asynchronous export method according to claim 1, characterized in that, The exported task includes at least a route identifier, business parameters, execution information, delivery information, and management information; the execution information includes task status, execution progress, remaining time, progress context, and delivery information; the task status includes: submitted, executing, completed, failed, and canceled.

3. The asynchronous export method as described in claim 2, characterized in that, The task deduplication strategy is used to remove duplicate export requests. The preprocessing of the export tasks based on a preset strategy set includes: Based on the task identifier of the first export request, check if there is an export request with the same task identifier as the first export request; If a second export request has the same task identifier as the first export request, and the task status of the second export request is completed, then the exported file corresponding to the second export request is associated with the first export request, and the task status of the first export request is updated to completed. If a second export request has the same task identifier as the first export request, and the task status of the second export request is in execution, then the execution thread corresponding to the second export request is associated with the first export request, and the task status of the first export request is updated to in execution. If a second export request has the same task identifier as the first export request, and the task status of the second export request is submitted, then the second export request is associated with the first export request, and the task status of the first export request is updated to submitted.

4. The asynchronous export method as described in claim 1, characterized in that, The task decomposition strategy includes at least a time-based decomposition strategy, a business-based decomposition strategy, and a data source complexity-based decomposition strategy. The preprocessing of the exported task based on a preset strategy set includes: If the business parameters of the exported task meet the time continuity condition, then the exported task is processed using a time-dimension-based decomposition strategy. If the business parameters of the exported task meet complex business conditions, then a business-dimensional decomposition strategy is adopted to process the exported task. If the business parameters of the exported task meet the multi-source heterogeneity condition, then the exported task is processed using a decomposition strategy based on the complexity of the data source.

5. The asynchronous export method as described in claim 1, characterized in that, The calculation of the execution progress and remaining time of each exported task and its feedback to the user terminal for display includes: For any exported task in any thread, the execution time of the exported task from the start time of the processing thread to the current time is continuously monitored, with the thread's startup time as the start time. Data is retrieved using a paginated query method. The average processing speed is calculated based on the number of pages retrieved and the execution time. The execution progress is calculated based on the number of pages retrieved and the total amount of data. The remaining time is predicted based on the average processing speed and the execution progress, and the execution progress and the remaining time are fed back to the user for display.

6. The asynchronous export method according to claim 1, characterized in that, Any abnormality in any exported task includes execution timeout and execution failure. Execution timeout means that the execution time of any exported task exceeds its corresponding timeout threshold.

7. The asynchronous export method according to claim 1, characterized in that, Before creating the corresponding export task, the method further includes: The export request is validated, and the validation includes at least one of data range validation, user permission validation, and resource quota validation.

8. An asynchronous export system, characterized in that, The system includes: Create a unit to respond to a user's submitted export request and create the corresponding export task; The management unit is used to preprocess the exported tasks based on a preset strategy set and store the preprocessed exported tasks in a task queue. The preset strategy set includes a task deduplication strategy and a task decomposition strategy. The execution unit is used to execute each exported task sequentially in a multi-threaded concurrent manner to obtain the execution result based on the dynamic priority of each exported task in the task queue. The dynamic priority is determined according to the task type, user identifier, estimated total data volume and waiting time of the exported task. After determining the dynamic priority, for a group of exported tasks belonging to the same parent task, when the difference in execution progress of all exported tasks in the group exceeds a preset difference threshold, the dynamic priority of the exported task with the lagging progress is increased. The monitoring unit calculates the execution progress and remaining time of each exported task and displays it to the user. When any exported task encounters an anomaly, it retryes the task using a differentiated fault-tolerance strategy. This retrying includes: analyzing the root cause of any exported task's anomaly; if the root cause is database lock contention, an exponential backoff retry strategy is used; if the root cause is transient service unavailability, an immediate retry strategy is used; if the root cause is insufficient system resources, a routing retry strategy is used; and if the root cause is a program logic error, a fast failure strategy is used to directly update the exported task's status to failure. The storage unit is used to generate export files from the execution results of each export task and upload the export files to the file storage system.

Images