Micro-service asynchronous exporting method

Abstract

The invention discloses a micro-service asynchronous exporting method. The method comprises the following steps that a user initiates an exporting request through a front end; the business service receives an export request initiated by the front end and assembles the query statement and the export template parameters to the export request according to the business; the business service forwards the processed export request to an asynchronous export service; after receiving an export request of the business service, the asynchronous export service creates an export task and returns a success state to the business service; the asynchronous export service evaluates the complexity, the dependency relationship and the execution time of the task according to the data volume and the system performance of the exported task, selects a task splitting strategy, splits the task into a plurality of sub-tasks and puts the sub-tasks into a task pool; the task pool triggers scheduling, parallel execution of subtasks is achieved through multiple threads or multiple processes, data are read in batches, and the data are written into temporary files. Through asynchronous processing, the user does not need to wait for export completion, and the user experience is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of software development, and in particular relates to a microservice asynchronous export method. Background Art

[0002] Among the existing file export technologies, common solutions include HTTP requests and responses, the use of message queues, and sharded exports. These solutions have their own advantages and disadvantages, but there are generally some technical problems when processing large files or high-concurrency requests.

[0003] HTTP request and response: The user initiates an export request, and the server returns a task ID. The user can query the task status through polling or long polling until the export is completed. Frequent polling requests will increase the server load, especially when a large number of users are exporting at the same time. In addition, the user needs to wait for the export to complete, which is a poor user experience.

[0004] Use of message queues: To achieve asynchronous processing, the system sends export request information to a message queue (such as RabbitMQ, Apache Kafka, etc.). This decouples the processing of requests from the user's request, and the user does not have to wait for the file generation to be completed. The use of message queues may lead to additional resource consumption, especially in high concurrency situations, where more computing and storage resources may be required. In addition, message order issues may lead to data inconsistency.

[0005] Sliced ​​export: For large files, the export task can be divided into multiple small tasks and processed asynchronously. Finally, multiple small files are merged into a complete file and provided to users for download. Sliced ​​export requires the management of multiple small tasks, which increases the complexity of the system. The merge operation may become a performance bottleneck, especially when processing a large number of small files.

[0006] The existing technical solutions generally have the following technical problems when processing large files or high-concurrency requests: High resource consumption: Frequent polling requests and the use of message queues will increase server resource consumption and affect system performance. Poor user experience: Users need to wait for the export to complete, especially when the export time is long, the user experience is poor. Increased complexity: Sharded export requires the management of multiple small tasks, which increases the complexity and maintenance difficulty of the system. Summary of the invention

[0007] The purpose of the present invention is to provide a microservice asynchronous export method, which solves the problems raised in the above background technology by asynchronousizing the export tasks and splitting the tasks.

[0008] To achieve the above purpose, the present invention adopts the following technical solution: a microservice asynchronous export method, comprising the following steps:

[0009] S1: The user initiates an export request through the front end; S2: The business service receives the export request initiated by the front end, and assembles the query statement and export template parameters into the export request according to the business; S3: The business service forwards the processed export request to the asynchronous export service; S4: After the asynchronous export service receives the export request from the business service, it creates an export task and returns a success status to the business service; S5: The asynchronous export service first evaluates the complexity, dependencies and execution time of the export task based on the data volume and system performance, selects the task splitting strategy according to the characteristics of the task, and splits a task into multiple subtasks and puts them into the task pool; S6: The task pool triggers scheduling, uses multi-threading or multi-process to realize parallel execution of subtasks, executes multiple export subtasks concurrently, and reads data in batches and writes temporary files in each subtask.

[0010] Preferably, the method further includes S7: after the task is executed, merging the temporary files into a summary file, and uploading the file to the file service.

[0011] Preferably, the task splitting strategy includes: data-based splitting: dividing the task into multiple subtasks, each subtask processes a different part of the data set; function-based splitting: splitting the task according to functional modules, and different modules can be executed independently; time-based splitting: splitting the task according to time periods, which is suitable for tasks with time series properties.

[0012] Preferably, the task pool maintains a globally unique task pool and task queue.

[0013] Preferably, it also includes S8: after receiving the file, the file service saves the file to the storage medium and returns the file download address to the asynchronous export service. S9: the asynchronous export service receives the file download address and updates the export result. S10: the user front end initiates a request to the asynchronous export service to query the export result; S11: the asynchronous export service returns the file download address to the user front end; S12: the front end user clicks on the returned file download address and initiates a request to the file service to download the file; S13: after receiving the request, the file service reads the file from the storage medium and returns it to the user terminal.

[0014] Preferably, task scheduling is implemented in the asynchronous export service for concurrently processing multiple export tasks.

[0015] Preferably, batch processing is implemented in the asynchronous export service, and data is read in batches and written into temporary files.

[0016] Preferably, streaming processing is implemented in the asynchronous export service, and a streaming processing framework is used to process data.

[0017] Technical effects and advantages of the present invention: Compared with the prior art, the microservice asynchronous export method proposed in the present invention has the following advantages:

[0018] Through asynchronous processing, the present invention allows users to continue with other operations without waiting for the export to be completed, thereby improving the user experience. Task splitting and parallel processing reduce the resource consumption of a single task and improve the overall performance of the system. Task pool management and task scheduling algorithms simplify task management, ensure the order and reliability of tasks, and reduce the complexity of the system. The asynchronous export method under the microservice architecture effectively solves the problems of high resource consumption, poor user experience, and increased system complexity in the prior art when processing large files or high-concurrency requests, thereby improving the response speed and overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention is a flowchart of a microservice asynchronous export method. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] The present invention provides a microservice asynchronous export method, such as Figure 1 As shown, the following steps are included:

[0022] S1: The user initiates an export request through the front end;

[0023] S2: The business service receives the export request initiated by the front end, and assembles the query statement and export template parameters into the export request according to the business.

[0024] S3: The business service forwards the processed export request to the asynchronous export service.

[0025] S4: After receiving the export request from the business service, the asynchronous export service creates an export task and returns a success status to the business service.

[0026] S5: The asynchronous export service first evaluates the complexity, dependencies, and execution time of the task based on the data volume and system performance of the export task, selects a task splitting strategy based on the characteristics of the task, and splits a task into multiple subtasks and puts them into the task pool;

[0027] Specifically, the task splitting strategies include: data-based splitting: dividing the task into multiple subtasks, each subtask processes a different part of the data set; function-based splitting: splitting the task according to functional modules, and different modules can be executed independently; time-based splitting: splitting the task by time period, which is suitable for tasks with time series characteristics.

[0028] S6: Task pool triggers scheduling, uses multi-threading or multi-processing to implement parallel execution of subtasks, executes multiple export subtasks concurrently, and reads data and writes temporary files in batches in each subtask.

[0029] S7: After the task is executed, the temporary files are merged into a summary file and the file is uploaded to the file service.

[0030] S8: After receiving the file, the file service saves the file to the storage medium and returns the file download address to the asynchronous export service.

[0031] S9: The asynchronous export service receives the file download address and updates the export result.

[0032] S10: The user front end initiates a request to the asynchronous export service to query the export results.

[0033] S11: The asynchronous export service returns the file download address to the user front end.

[0034] S12: The front-end user clicks the returned file download address and initiates a request to the file service to download the file.

[0035] S13: After receiving the request, the file service reads the file from the storage medium and returns it to the user terminal.

[0036] Furthermore, task scheduling is implemented in the asynchronous export service to process multiple export tasks concurrently. Batch processing is implemented in the asynchronous export service to read data in batches and write them to temporary files. Streaming processing is implemented in the asynchronous export service to process data using a streaming processing framework.

[0037] Business services, asynchronous export services, and file services are encapsulated as microservices, and servers can be freely selected for placement. They can be expanded independently, especially during peak traffic periods, when only the most commonly used microservices need to be expanded, and the others are not affected. The technology is also relatively flexible, and microservices can be built in multiple languages ​​or platforms because they are connected by communicating with each other, and the actual language used is not important. This flexibility ensures that microservices can adapt to changing needs and technical environments.

[0038] The task pool involves the management of the entire process of task creation, scheduling, execution, and completion, including task queuing, execution order control, and task status tracking, and encapsulates the task pool into an asynchronous export microservice.

[0039] The asynchronous file export solution based on microservices effectively solves technical problems such as task scheduling, resource optimization, error handling, data consistency and user experience, and provides a guarantee for the efficient operation of the system. The asynchronous file export solution based on microservices effectively solves technical problems such as task scheduling, resource optimization, error handling, data consistency and user experience, and provides a guarantee for the efficient operation of the system.

[0040] Asynchronous export file technology is suitable for many scenarios, especially when you need to process large amounts of data or high concurrent requests. The following are some of the main application scenarios.

[0041] (1) Big Data Processing

[0042] Data migration: When performing large-scale data migration, asynchronous export can process data in shards to reduce the impact on system performance.

[0043] Data backup: When backing up large amounts of data regularly, asynchronous export can ensure that the backup process does not affect normal business operations.

[0044] (2) File export service

[0045] User self-service export: Users choose to export data (such as CSV, Excel, etc.) on the platform. The system processes the export request asynchronously, and users can download the file after the export is completed.

[0046] Batch export: In e-commerce, CRM and other systems, users may need to export customer information, order records, etc. in batches. Asynchronous export can effectively handle these requests.

[0047] (3) Content Management System

[0048] Content export: In a content management system, users may need to export articles, media files, etc. Asynchronous export can be processed in the background, and users can obtain files when needed.

[0049] Data migration: Asynchronous export ensures data integrity and consistency during content migration or backup

[0050] The following is an example based on specific application scenarios:

[0051] Application scenario 1

[0052] In e-commerce, CRM and other systems, users may need to export customer information, order records, etc. in batches. Asynchronous export can effectively handle these requests and implement the batch export process in the file export service.

[0053] Step 1: Initiate an export request:

[0054] Execution entity: user front end.

[0055] Trigger conditions: The user selects the data to be exported in the system (such as customer information, order records, etc.) and clicks the "Export" button.

[0056] Processing action: The front end encapsulates the user-selected data type, export format and other information into a request and sends it to the business service.

[0057] Result: The business service successfully receives the export request.

[0058] Function of this step: Start the entire export process to ensure that the user's data export requirements are processed.

[0059] Step 2: Build the export request:

[0060] Executing entity: business services.

[0061] Trigger condition: receiving an export request from the front end.

[0062] Processing action: Assemble query statements, export template and other parameters according to the request content, and prepare the export request.

[0063] Result: The generated export request encapsulates the query details and the export configuration.

[0064] Function of this step: Prepare the information required for export and ensure the correctness and completeness of the data.

[0065] Step 3: Forward the export request:

[0066] Executing entity: business services.

[0067] Triggered by: After successfully building an export request.

[0068] Processing action: forward the generated export request to the asynchronous export service for processing.

[0069] Result: The asynchronous export service receives a pending export request.

[0070] Function of this step: Separate business logic from export tasks, optimize system architecture, and improve maintainability.

[0071] Step 4: Create an export task:

[0072] Execution entity: asynchronous export service.

[0073] Trigger condition: Receiving an export request for a business service.

[0074] Processing action: Create an export task and generate a unique identifier, and return a success status to the business service.

[0075] Result: The export task is successfully created and feedback is given to the business service.

[0076] Function of the step: Asynchronous processing avoids blocking the user front end and improves the system's responsiveness and concurrent processing capabilities.

[0077] Step 5: Feedback the export task creation status:

[0078] Executing entity: business services.

[0079] Trigger condition: receiving a successful response from the asynchronous export service.

[0080] Processing action: Return the success status to the user front end.

[0081] Result: The user frontend receives a prompt indicating that the task was created successfully.

[0082] Function of this step: Enhance the user experience and give users visual feedback on the export progress.

[0083] Step 6: Task storage and scheduling:

[0084] Execution entity: asynchronous export service.

[0085] Trigger condition: The asynchronous export service successfully creates multiple export tasks.

[0086] Processing actions: Evaluate the complexity, dependencies, and execution time of the task, select a function-based splitting strategy based on the characteristics of the task, split a task into multiple subtasks and put them into the task pool. Put the export task into the task pool, and add the tasks that need to be executed asynchronously to the task queue.

[0087] Result: Tasks are efficiently stored, waiting for background threads to execute them in order.

[0088] Function of the steps: realize concurrent processing, optimize resource utilization, and reduce memory burden.

[0089] Step 7: Reading data and writing files:

[0090] Execution body: background thread (task scheduler).

[0091] Trigger condition: There are export tasks to be executed in the task queue.

[0092] Processing action: Read data in batches according to task requirements and write them to temporary files.

[0093] Result: One or more temporary files are generated.

[0094] What this step does: Process large amounts of data and ensure that the export operation does not affect system performance.

[0095] Step 8: Merge and upload files:

[0096] Execution entity: asynchronous export service.

[0097] Trigger condition: All temporary files are generated.

[0098] Processing action: Merge the temporary files into a summary file and upload it to the file service.

[0099] Result: The summary file is successfully saved in the file service.

[0100] Function of this step: Ensure that the exported data is complete and stable to facilitate subsequent downloads by users.

[0101] Step 9: Return to the file download address:

[0102] Execution entity: file service.

[0103] Trigger condition: receiving an upload request from the asynchronous export service.

[0104] Processing action: Return the file download address to the asynchronous export service.

[0105] Result: The asynchronous export service obtains the actual file download address.

[0106] Function of this step: The file service implements the data storage and access interface, making it easy for users to retrieve files.

[0107] Step 10: Update the export results:

[0108] Execution entity: asynchronous export service.

[0109] Trigger condition: receiving the download address returned by the file service.

[0110] Processing action: Update the export task status in the database record or memory and store the download address. Result: The export task status is updated, including the address of the downloaded file.

[0111] Function of this step: Ensure that users can obtain the required file download information in a timely manner.

[0112] Step 11: Query the export results:

[0113] Execution entity: user front end.

[0114] Trigger: The user wants to see the export results, such as through a status page or an alert.

[0115] Processing action: Send a query request to the asynchronous export service.

[0116] Result: The asynchronous export service receives the query request.

[0117] Function of this step: Enhance the user interaction experience and make it easier for users to understand the status of the export task.

[0118] Step 12: Return to the file download address:

[0119] Execution entity: asynchronous export service.

[0120] Trigger condition: receiving a user query request.

[0121] Processing action: Return the file download address of the export result to the user front end.

[0122] Result: The user obtains the address to download the file.

[0123] Function of this step: Maintain the transmission of feedback information between users and the system, and ensure that users can access information conveniently.

[0124] Step 13: Initiate a file download request:

[0125] Execution entity: user front end.

[0126] Trigger condition: The user clicks on the returned download address.

[0127] Processing action: Send a file download request to the file service.

[0128] Result: The user's download request is sent to the file service.

[0129] Function of this step: Enable users to actually access and download the exported files.

[0130] Step 14: Return to download file:

[0131] Execution entity: file service.

[0132] Trigger condition: receiving a file download request from the user.

[0133] Processing action: Read the file and return it to the user's terminal.

[0134] Result: The user successfully downloads the requested file.

[0135] Function of this step: Complete the goal of the entire export function and meet the user's needs for exporting data.

[0136] The effect is as follows:

[0137] Asynchronous export allows multiple export requests to be processed simultaneously, making full use of system resources and reducing user waiting time. By assigning export tasks to different service instances, the load can be effectively dispersed to avoid single-point bottlenecks. After initiating an export request, users can continue with other operations without having to wait for the export to complete, which improves the responsiveness of the system. Users can view the export progress and status in real time, enhancing the interactive experience.

[0138] Computing resources can be dynamically allocated according to the complexity of the export task to avoid resource waste. Export requests are managed through the task queue to ensure that the system can still run stably under high load.

[0139] The export service can be independently expanded according to demand, supporting more export formats and data sources to adapt to business changes. It is easy to integrate with other microservices (such as data storage, notification services, etc.) to form a complete export solution.

[0140] Asynchronous processing can achieve better error capture and handling mechanisms, ensuring that export tasks can be automatically retried after failure. The status and error information of each export task can be recorded to facilitate subsequent auditing and troubleshooting.

[0141] Application scenario 2

[0142] When performing large-scale data migration or data backup in big data processing, asynchronous export can process data in shards to reduce the impact on system performance.

[0143] Step 1. Task initialization:

[0144] Execution subject: user request.

[0145] Trigger condition: The user initiates a data migration or backup request.

[0146] Processing action: The system receives the request and initializes the export task, including setting the task ID, data source, target location, etc.

[0147] Result: A unique task identifier is generated, ready for subsequent processing.

[0148] Function of this step: Lay the foundation for the entire export process.

[0149] Step 2. Data sharding configuration:

[0150] Executing entity: system.

[0151] Trigger condition: Task initialization is completed.

[0152] Processing action: Determine the size and number of data shards based on the data volume and system performance.

[0153] Result: A shard configuration is generated, recording the start and end positions of each shard.

[0154] Function of this step: Optimize data processing efficiency and ensure that the size of each shard is suitable for processing.

[0155] Step 3. Asynchronous task enqueue:

[0156] Execution entity: task management system.

[0157] Trigger condition: Data sharding configuration is completed.

[0158] Processing action: Evaluate the complexity, dependencies, and execution time of the task, select a data-based splitting strategy based on the characteristics of the task, queue the processing tasks of each data shard, and wait for the background thread to execute them in sequence.

[0159] Result: The task queue contains multiple shard tasks to be processed.

[0160] Function of the step: Realize concurrent processing of tasks and improve overall efficiency.

[0161] Step 4. Data reading and processing:

[0162] Execution body: asynchronous processing thread.

[0163] Trigger condition: The task is scheduled for execution.

[0164] Processing action: Read the data of the specified shard from the data source and perform necessary conversion or cleaning.

[0165] Result: The data of each shard is processed and stored as a temporary file.

[0166] Step function: Ensure data integrity and consistency.

[0167] Step 5. Temporary file storage:

[0168] Execution body: asynchronous processing thread.

[0169] Trigger condition: data reading and processing completed.

[0170] Processing action: Write the processed data shards to a temporary storage location.

[0171] Result: Multiple temporary files are generated, each file corresponds to a data shard.

[0172] Function of this step: Prepare for subsequent file merging and uploading.

[0173] Step 6. File merging:

[0174] Execution body: asynchronous processing thread.

[0175] Trigger condition: All data shards are processed.

[0176] Processing action: Merge all temporary files into a complete export file.

[0177] Result: The final export file is generated.

[0178] Function of this step: Integrate data to form the complete data set required by the user.

[0179] Step 7. File upload:

[0180] Execution body: asynchronous processing thread.

[0181] Trigger condition: The summary file is generated.

[0182] Action: Upload the final export file to the target storage location (such as cloud storage). Result: The file is successfully stored and ready for the user to download.

[0183] Step function: Ensure data is stored securely for easy subsequent access.

[0184] Step 8. Update the task status:

[0185] Execution body: asynchronous processing thread.

[0186] Trigger condition: The file is uploaded successfully.

[0187] Processing action: Update the status of the export task, record the file storage location and download link. Result: The task status is marked as completed, and the user can obtain the download information.

[0188] What this step does: Provide user feedback and ensure that users are informed.

[0189] Step 9. User downloads the file:

[0190] Execution entity: user interface.

[0191] Trigger condition: User requests to download a file.

[0192] Processing action: The user obtains the file through the download link.

[0193] Result: The user successfully downloads the required export file.

[0194] Function of this step: It is the final step to complete data migration or backup.

[0195] The effect is as follows:

[0196] The asynchronous export service under the microservice architecture can easily integrate different data sources and storage systems, support multiple data sources, improve the flexibility of data migration, and support multiple data formats and storage solutions. At the same time, it improves the reliability and consistency of data and reduces the risk of data loss. Asynchronous processing improves the user experience, and users can continue to perform other operations without being hindered by the long export process. The microservice architecture improves the reliability of the system and ensures stability during data migration and backup. Increase scalability, and during peak periods, it can quickly respond to and process more export requests to ensure high availability of the system.

[0197] On the other hand, this embodiment also provides a microservice asynchronous export system, including:

[0198] The front-end module is used to receive the user's export request;

[0199] The business service module is used to receive the export request from the front-end module, assemble the query statement, export template and other parameters into the export request according to the business, and forward the processed export request to the asynchronous export service module;

[0200] The asynchronous export service module is used to receive export requests from the business service module, create export tasks, select appropriate splitting strategies based on the data volume of the task and system performance, split the task into multiple subtasks and put them into the task pool, trigger the task pool scheduling, and use multi-threading or multi-processing to realize the parallel execution of subtasks;

[0201] The file service module is used to receive the summary file uploaded by the asynchronous export service module, save the file to the storage medium, and return the file download address to the asynchronous export service module;

[0202] The download management module is used to receive download requests from the user front end, read files from the file service module and return them to the user terminal.

[0203] The asynchronous export service module includes a task splitting unit and a task pool unit. The task splitting unit is used to evaluate the complexity, dependency and execution time of the task and select an appropriate splitting strategy; the task pool unit is used to maintain a globally unique task queue to ensure the order and reliability of the task.

[0204] The file service module includes a file merging unit and a file uploading unit. The file merging unit is used to merge temporary files into a summary file; the file uploading unit is used to upload the summary file to the file service through the REST API to ensure the integrity and security of the file.

[0205] In this embodiment, the system also includes a dynamic expansion module for dynamically expanding or reducing service instances according to load conditions to improve the elasticity of the system.

[0206] In this embodiment, the system also includes a load balancing module for distributing export tasks to multiple microservice instances, optimizing resource usage, and avoiding single point bottlenecks.

[0207] In this embodiment, the system also includes an error handling module for implementing fault-tolerant processing and task retry mechanisms to improve the reliability of the system.

[0208] In this embodiment, the system also includes a data consistency module, which is used to ensure the consistency and integrity of data during the export process and avoid data errors caused by concurrent operations.

[0209] In this embodiment, the system also includes a user experience module for providing a user-friendly feedback mechanism and convenient download management to enhance the user experience.

[0210] In addition, the present invention also provides a terminal device. The microservice asynchronous export method involved in this embodiment is mainly applied to the terminal device, which can be a PC, a portable computer, a mobile terminal, or other device with display and processing functions.

[0211] Specifically, the terminal device may include a processor (such as a CPU), a communication bus, a user interface, a network interface, and a memory. The communication bus is used to realize the connection and communication between these components; the user interface may include a display screen (Displ ay), an input unit such as a keyboard (Keyboard); the network interface may optionally include a standard wired interface, a wireless interface (such as a WIFI interface); the memory may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory, and the memory may optionally be a storage device independent of the aforementioned processor.

[0212] Among them, the memory stores a readable storage medium, the readable storage medium stores an asynchronous export program, and the processor can call the asynchronous export program stored in the memory and execute the microservice asynchronous export method provided by an embodiment of the present invention.

[0213] It is understood that a readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or raised structures in grooves on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used here is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0214] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0215] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, by utilizing state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0216] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A microservice asynchronous export method, characterized in that: The following steps are involved: S1: The user initiates an export request through the front end; S2: The business service receives the export request initiated by the front end, and assembles the query statement and export template parameters into the export request according to the business; S3: The business service forwards the processed export request to the asynchronous export service; S4: After receiving the export request from the business service, the asynchronous export service creates an export task and returns a success status to the business service; S5: The asynchronous export service first evaluates the complexity, dependencies, and execution time of the task based on the data volume and system performance of the export task, selects a task splitting strategy based on the characteristics of the task, and splits a task into multiple subtasks and puts them into the task pool; S6: Task pool triggers scheduling, uses multi-threading or multi-processing to implement parallel execution of subtasks, executes multiple export subtasks concurrently, and reads data and writes temporary files in batches in each subtask.

2. A microservice asynchronous export method according to claim 1, characterized in that: The method also includes S7: merging temporary files into a summary file after executing the task, and uploading the file to the file service.

3. A microservice asynchronous export method according to claim 2, characterized in that: The task splitting strategy includes: Data-based splitting: split the task into multiple subtasks, each of which processes a different part of the dataset; Function-based splitting: tasks are split into functional modules, and different modules can be executed independently; Time-based splitting: split the task into time periods, which is suitable for tasks with time series characteristics.

4. A microservice asynchronous export method according to claim 1, characterized in that: The task pool maintains a globally unique task pool and task queue.

5. A microservice asynchronous export method according to claim 2, characterized in that: The process also includes S8: after receiving the file, the file service saves the file to a storage medium and returns the file download address to the asynchronous export service.

6. A microservice asynchronous export method according to claim 5, characterized in that: The method also includes S9: the asynchronous export service receives the file download address and updates the export result.

7. A microservice asynchronous export method according to claim 6, characterized in that: Also includes the steps: S10: The user front end initiates a request to the asynchronous export service to query the export result; S11: The asynchronous export service returns the file download address to the user front end; S12: The front-end user clicks the returned file download address and initiates a request to the file service to download the file; S13: After receiving the request, the file service reads the file from the storage medium and returns it to the user terminal.

8. A microservice asynchronous export method according to claim 1, characterized in that: Task scheduling is implemented in the asynchronous export service to process multiple export tasks concurrently.

9. A microservice asynchronous export method according to claim 1, characterized in that: The asynchronous export service implements batch processing, reading data in batches and writing them into temporary files.

10. A microservice asynchronous export method according to claim 1, characterized in that: Stream processing is implemented in the asynchronous export service, and a stream processing framework is used to process data.

Images