Pluggable data exporting method and device suitable for multi-micro-service architecture and electronic equipment
By configuring mapping and reflection invocation mechanisms, combined with dynamic configuration and distributed task scheduling, pluggable data export under a multi-microservice architecture is realized, which solves the problems of inefficient cross-service collaboration and imbalance between resource utilization and export efficiency in existing technologies, and improves system stability and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- RENLIWO NETWORK TECHNOLOGY (SHANGHAI) CO LTD
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-15
Smart Images
Figure CN122045291A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of microservice data processing technology, and in particular to a pluggable data export method, apparatus, and electronic device suitable for multi-microservice architectures. Background Technology
[0002] With the development of distributed technology, multi-microservice architecture has become the mainstream architectural pattern for enterprise application systems. Data export, as a core function that meets users' needs for data statistics, backup, and analysis, is widely used in multi-microservice architecture.
[0003] Currently, in the industry, data export functionality in multi-microservice architectures generally adopts a "distributed implementation" approach. This means that each microservice node independently develops and deploys its own data export module, defining cross-service data call rules through hard coding and designing separate processing logic for different export scenarios. Existing technologies categorize data export processing modes into synchronous and asynchronous export: synchronous export is suitable for small data volumes, directly returning a file stream to the user; asynchronous export is suitable for large data volumes, requiring serial execution via a task queue and notification to the user of the download results.
[0004] However, existing distributed data export solutions have many technical shortcomings: (1) Cross-service export collaboration is inefficient and development and maintenance costs are high: each microservice repeatedly develops export functions, resulting in functional redundancy; cross-service data calls rely on hard coding, and interface changes have a wide impact; new export scenarios need to be redeveloped, which is time-consuming and costly.
[0005] (2) Rigid synchronous and asynchronous modes lead to an imbalance between resource utilization and export efficiency: A single processing mode is used in a fixed manner or manual configuration is required, which cannot be dynamically adapted to the amount of data. Using asynchronous mode for small amounts of data leads to resource waste and delays; using synchronous mode for large amounts of data is prone to timeouts (for example, the timeout rate for exporting more than 100,000 data items exceeds 60%), which seriously affects the user experience.
[0006] (3) Multitasking congestion and poor response during peak hours: Asynchronous export tasks are generally executed serially, resulting in a large number of tasks queuing up during peak business hours, and users have to wait too long.
[0007] (4) Service restart interruption and high operation and maintenance costs: When the service restarts or the node fails, the task being executed may become a "zombie task", resulting in data loss or abnormal status. Manual intervention is required to investigate and restore, which is a lot of operation and maintenance work. (5) Untimely anomaly alarms lead to an expansion of the scope of the fault: The lack of a unified anomaly handling and alarm mechanism means that maintenance personnel need to regularly check the logs scattered across various nodes, resulting in delayed fault detection and easy expansion of the problem.
[0008] Therefore, a pluggable data export method, apparatus, and electronic device suitable for multi-microservice architecture are proposed. Summary of the Invention
[0009] This specification provides a pluggable data export method, device, and electronic device suitable for multi-microservice architectures. By configuring mapping and reflection invocation mechanisms, when adding new export scenarios, only the configuration and data model need to be extended, without modifying the core code of each service or performing hard-coded adaptation, which improves development efficiency and truly achieves "pluggability".
[0010] This specification provides a pluggable data export method suitable for multi-microservice architectures, including: Obtain the client's export request; The target service information corresponding to the export request is parsed using a preset configuration mapping relationship; Based on the export request, the data volume threshold corresponding to the current export scenario is determined from the dynamic configuration center, and the estimated export data volume of the export request is obtained through SQL query or interface. The estimated amount of exported data is compared with the data amount threshold to obtain the comparison result; Based on the comparison results, a synchronous export mode or an asynchronous export mode is selected, and an export request is executed based on the target service information to complete the data export.
[0011] Optionally, the step of executing an export request based on the target service information to complete the data export includes: The target service information includes the target service identifier and the target data acquisition method name; In synchronous export mode, the data interface of the corresponding microservice is dynamically called by using reflection mechanism based on the target service identifier and the target data to obtain the method name, generate a file stream in real time and return it. In asynchronous export mode, the target service identifier and the target data acquisition method name are encapsulated into an asynchronous task and submitted to the task scheduling center.
[0012] Optionally, after encapsulating the target service identifier and the target data acquisition method name into an asynchronous task and submitting it to the task scheduling center, the method further includes: The task scheduling center schedules executor nodes, which locate and call the data interface of the corresponding microservice based on the target service identifier, obtain data in a paginated manner, generate an export file, and update the task status.
[0013] Optionally, selecting the synchronous export mode or the asynchronous export mode based on the comparison result includes: When the estimated amount of data to be exported is less than or equal to the data volume threshold, select the synchronous export mode; When the estimated amount of data to be exported is greater than the data volume threshold, the asynchronous export mode is selected.
[0014] Optionally, after selecting the asynchronous export mode but before executing the asynchronous export request, the following steps are also included: The estimated amount of exported data is returned to the client for secondary confirmation, and execution continues after receiving the confirmation instruction.
[0015] Optional, also includes: When the project goes live, the microservice nodes restart and automatically query the task scheduling center to see if there are any abnormal tasks among all tasks in the running state. If it exists, interrupt the thread of the abnormal task and output an alarm message.
[0016] Optional, also includes: Collect exception logs generated by each microservice node during the data export process; The exception logs are filtered and formatted. The processed alarm information is pushed to the operation and maintenance terminal through a preset Webhook interface.
[0017] This specification provides a pluggable data export device suitable for multi-microservice architectures, comprising: The acquisition module is used to acquire the client's export request; The parsing module is used to parse the target service information corresponding to the export request through a preset configuration mapping relationship; The determination module is used to determine the data volume threshold corresponding to the current export scenario from the dynamic configuration center based on the export request, and obtain the estimated export data volume of the export request through SQL query or interface. The comparison module is used to compare the estimated amount of exported data with the data amount threshold to obtain a comparison result; The export module is used to select a synchronous export mode or an asynchronous export mode based on the comparison results, and to execute an export request based on the target service information to complete the data export.
[0018] Optionally, the export module includes: The target service information includes the target service identifier and the target data acquisition method name; In synchronous export mode, the data interface of the corresponding microservice is dynamically called by using reflection mechanism based on the target service identifier and the target data to obtain the method name, generate a file stream in real time and return it. In asynchronous export mode, the target service identifier and the target data acquisition method name are encapsulated into an asynchronous task and submitted to the task scheduling center.
[0019] Optionally, after encapsulating the target service identifier and the target data acquisition method name into an asynchronous task and submitting it to the task scheduling center, the method further includes: The task scheduling center schedules executor nodes, which locate and call the data interface of the corresponding microservice based on the target service identifier, obtain data in a paginated manner, generate an export file, and update the task status.
[0020] Optionally, the export module includes: When the estimated amount of data to be exported is less than or equal to the data volume threshold, select the synchronous export mode; When the estimated amount of data to be exported is greater than the data volume threshold, the asynchronous export mode is selected.
[0021] Optionally, after selecting the asynchronous export mode but before executing the asynchronous export request, the following steps are also included: The estimated amount of exported data is returned to the client for secondary confirmation, and execution continues after receiving the confirmation instruction.
[0022] Optional, also includes: When the project goes live, the microservice nodes restart and automatically query the task scheduling center to see if there are any abnormal tasks among all tasks in the running state. If it exists, interrupt the thread of the abnormal task and output an alarm message.
[0023] Optional, also includes: Collect exception logs generated by each microservice node during the data export process; The exception logs are filtered and formatted. The processed alarm information is pushed to the operation and maintenance terminal through a preset Webhook interface.
[0024] This specification also provides an electronic device, wherein the electronic device includes: A processor; and a memory storing computer-executable instructions, which, when executed, cause the processor to perform any of the methods described above.
[0025] This specification also provides a computer-readable storage medium that stores one or more programs that, when executed by a processor, implement any of the methods described above.
[0026] In this invention, by configuring mapping and reflection invocation mechanisms, adding new export scenarios only requires extending the configuration and data model, without modifying the core code of each service or performing hard-coded adaptations, thus improving development efficiency and truly achieving "pluggable" functionality. Relying on dynamically configured intelligent thresholds, the system can automatically and precisely switch between synchronous (instant response) and asynchronous (reliable execution) modes based on the data volume, solving the problem of mode rigidity, improving the response speed of small data volume exports, ensuring the success rate of large data volume exports, and eliminating timeout bottlenecks. Through a distributed task scheduling framework, parallel execution and fault transfer of asynchronous tasks are achieved, reducing the total processing time of multiple tasks during peak business periods and effectively avoiding task congestion. Simultaneously, the service startup self-healing mechanism can automatically identify and handle "zombie tasks" interrupted by restarts, improving overall task stability and greatly reducing manual maintenance intervention. A unified log collection, processing, and alarm chain ensures that any anomalies can be detected and pushed in real time, fundamentally improving the efficiency of fault detection and location, and effectively controlling the scope of fault impact. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0028] Figure 1 A schematic diagram illustrating the principle of a pluggable data export method suitable for a multi-microservice architecture, provided in the embodiments of this specification. Figure 2 A schematic diagram of a pluggable data export device suitable for a multi-microservice architecture, provided as an embodiment of this specification; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification; Figure 4 This is a schematic diagram of a computer-readable medium provided for embodiments of this specification. Detailed Implementation
[0029] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.
[0030] The following is in conjunction with the appendix Figure 1-4Exemplary embodiments of the invention will be described more fully here. However, exemplary embodiments can be implemented in many forms and should not be construed as limiting the invention to the embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention more comprehensive and complete, and to facilitate a full communication of the inventive concept to those skilled in the art. The same reference numerals in the figures denote the same or similar elements, components, or parts, and therefore repeated descriptions of them are omitted.
[0031] Subject to the technical concept of this invention, the features, structures, characteristics or other details described in a particular embodiment may be combined in one or more other embodiments in a suitable manner.
[0032] In the description of specific embodiments, the features, structures, characteristics, or other details described in this invention are intended to enable those skilled in the art to fully understand the embodiments. However, it is not excluded that those skilled in the art can practice the technical solutions of this invention without one or more of the specific features, structures, characteristics, or other details.
[0033] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0035] The terms “and / or” or “and / or” include all combinations of any one or more of the listed items.
[0036] Figure 1 This is a schematic diagram illustrating the principle of a pluggable data export method suitable for a multi-microservice architecture, provided in an embodiment of this specification. The method may include: S110: Obtain the client's export request; In a specific implementation of this specification, the request processing module deployed on each microservice node receives export requests from the web client (e.g., when a user clicks the "Export Orders" button).
[0037] S120: Parse the target service information corresponding to the export request through the preset configuration mapping relationship; In the specific implementation of this specification, the local preset enumeration mapping relationship (such as an enumeration class that defines "scenario -> service") is queried, and the corresponding target service information is parsed out according to the scenario identifier (such as EXPORT_ORDER) in the request.
[0038] Pre-defined configuration mappings are typically implemented at the code level using enumeration classes. For example, an `ExportSceneEnum` enumeration can be defined, where each enumeration value (e.g., `ORDER_EXPORT`) is associated with three key attributes: `serviceId` (e.g., "order-service"), `methodName` (e.g., "getOrderListForExport"), and `voClassName` (e.g., "com.example.OrderExportVO"). When a new export scenario is added, developers only need to add an entry to this enumeration and complete the writing of the relevant `VO` class to achieve "hot-swappable" functionality, without modifying any core scheduling and processing code. Dynamic information such as threshold configurations and service instance addresses for all scenarios is uniformly maintained in the Nacos configuration center. The request processing modules of each microservice node listen for changes in relevant configurations in Nacos, ensuring real-time updates and effectiveness of the configurations, completely avoiding service restarts due to threshold adjustments or service address changes.
[0039] S130: Based on the export request, determine the data volume threshold corresponding to the current export scenario from the dynamic configuration center, and obtain the estimated export data volume of the export request through SQL query or interface; In the specific implementation of this specification, a specific data volume threshold configured for this scenario (e.g., 20,000 records for the "order export" scenario) is obtained through a remote Nacos dynamic configuration center. The total amount of data involved in this export is estimated by executing a COUNT(*) type SQL query or calling the fast counting interface provided by the target microservice.
[0040] S140: Compare the estimated amount of exported data with the data amount threshold to obtain a comparison result; S150: Select synchronous export mode or asynchronous export mode based on the comparison result, and execute the export request based on the target service information to complete the data export.
[0041] In the specific implementation of this specification, the estimated data volume is compared with the threshold obtained from Nacos, and a decision is made on whether to adopt a synchronous or asynchronous export mode based on the comparison result. Then, the subsequent export execution logic is triggered based on the parsed target service information (such as the service name order-service).
[0042] Optionally, S150 includes: The target service information includes the target service identifier and the target data acquisition method name; In synchronous export mode, the data interface of the corresponding microservice is dynamically called by using reflection mechanism based on the target service identifier and the target data to obtain the method name, generate a file stream in real time and return it. In asynchronous export mode, the target service identifier and the target data acquisition method name are encapsulated into an asynchronous task and submitted to the task scheduling center.
[0043] In the specific implementation of this specification, the target service information specifically includes the target service identifier (e.g., order-service) and the target data retrieval method name (e.g., getOrderListForExport). In synchronous export mode, the system utilizes Java reflection to locate the instance of order-service through the service discovery component based on the service identifier, and dynamically calls its getOrderListForExport method. After this method returns a data list, the system uses the POI library to instantly generate an Excel file stream in memory and directly returns it to the front-end user for download via an HTTP response. In asynchronous export mode, the system encapsulates the aforementioned service identifier, method name, and client request parameters (e.g., time range, filter conditions) into an asynchronous task object, and then submits the task to an independently deployed "export center" (i.e., task scheduling center) via an HTTP call.
[0044] Optionally, after encapsulating the target service identifier and the target data acquisition method name into an asynchronous task and submitting it to the task scheduling center, the method further includes: The task scheduling center schedules executor nodes, which locate and call the data interface of the corresponding microservice based on the target service identifier, obtain data in a paginated manner, generate an export file, and update the task status.
[0045] In the specific implementation of this specification, the subsequent processing flow for asynchronous tasks submitted to the task scheduling center is as follows: After receiving the task, the task scheduling center (implemented using the XXL-JOB distributed task scheduling framework) first persists the task information (including task ID, parameters, pending execution status, creation time, etc.) to a MySQL database. Subsequently, the XXL-JOB scheduler selects one of multiple executor nodes to execute the task based on a load balancing strategy. The selected executor node obtains the specific network address (IP and port) of the target service identifier (order-service) in the task information by querying the service address mapping in Nacos. The executor calls the data retrieval method corresponding to the address via RPC or HTTP protocol, using a paginated query method (e.g., fetching 5000 records at a time) to cyclically fetch all data to avoid memory overflow due to excessive data loading at one time. After the data is fetched, the executor assembles and formats it into an Excel file, uploads it to an object storage service such as Alibaba Cloud OSS, and obtains a permanent file download URL. Finally, the executor calls back the interface of the task scheduling center to update the status of the task in the database to complete and fills in the file download URL.
[0046] The task scheduling center acts as the scheduler (Admin) for XXL-JOB, while multiple executor nodes can be horizontally scaled. When a large number of export tasks arrive simultaneously, the export center will distribute the tasks in parallel to different executor nodes based on the load of each executor. Each executor can be configured with an independent thread pool, achieving resource isolation between tasks and preventing a single time-consuming task from blocking other tasks.
[0047] Optionally, S150 includes: When the estimated amount of data to be exported is less than or equal to the data volume threshold, select the synchronous export mode; When the estimated amount of data to be exported is greater than the data volume threshold, the asynchronous export mode is selected.
[0048] In the specific implementation of this specification, the rule for selecting the export mode based on the comparison result is as follows: The system compares the estimated amount of data to be exported with a dynamic threshold. If the estimated amount of data is less than or equal to the threshold (e.g., estimated 5000 records ≤ threshold 20000 records), it is determined to be a small data volume scenario, and the synchronous export mode is selected to pursue immediate response. If the estimated amount of data is greater than the threshold (e.g., estimated 100000 records > threshold 20000 records), it is determined to be a large data volume scenario, and the asynchronous export mode is selected to ensure reliable task execution and non-blocking of requests. This rule is embedded in the request processing module through code logic, realizing intelligent and automatic adaptation of the export mode.
[0049] Optionally, after selecting the asynchronous export mode but before executing the asynchronous export request, the following steps are also included: The estimated amount of exported data is returned to the client for secondary confirmation, and execution continues after receiving the confirmation instruction.
[0050] In the specific implementation of this specification, to optimize user experience and prevent accidental operations, a user confirmation step is included after the system decides to adopt the asynchronous export mode and before formally submitting the task to the task scheduling center. Specifically, the request processing module returns the calculated estimated amount of data to be exported (e.g., "100,000 records") to the web client. The front-end page then pops up a confirmation dialog box, displaying "The current estimated number of records to be exported is 100,000. Confirming execution will generate an asynchronous task. Continue?". Only after the user clicks "Confirm" does the front-end send the final execution instruction, and the request processing module then continues the task submission process. Clicking "Cancel" will abort the current export operation.
[0051] Optional, also includes: When the project goes live, the microservice nodes restart and automatically query the task scheduling center to see if there are any abnormal tasks among all tasks in the running state. If it exists, interrupt the thread of the abnormal task and output an alarm message.
[0052] In the specific implementation described in this specification, to achieve task self-healing during service restart, each microservice node integrates an initialization processor based on the Spring Boot ApplicationRunner interface during the startup phase. This processor runs automatically when the service (e.g., order-service) starts. It first calls the RESTful API provided by the task scheduling center to query a list of all tasks whose executor is the current node and whose task status is "in execution". Then, it iterates through the list, checking the "last update time" field of each task. If the difference between the current time and this time exceeds a preset failure time threshold (e.g., 30 minutes), the task is determined to be a "zombie task" left over from the previous abnormal service termination. The system records the ID, type, and creation time of these abnormal tasks as ERROR-level logs, providing clear evidence for subsequent manual intervention or automatic retries, thereby significantly reducing the troubleshooting costs for operations personnel. The relevant personnel are notified via DingTalk to log in to the platform to check the task list and manually set the "in execution" status to "not executed". After correction, the next timer will automatically run again without code modification.
[0053] Optional, also includes: Collect exception logs generated by each microservice node during the data export process; The exception logs are filtered and formatted. The processed alarm information is pushed to the operation and maintenance terminal through a preset Webhook interface.
[0054] In the specific implementation described in this specification, each microservice node and application component uses the Logback logging framework to locally collect all ERROR-level logs generated during runtime (including API call timeouts, reflection call failures, file generation anomalies, zombie tasks discovered through task self-healing, etc.). These logs are then automatically sent to Alibaba Cloud Log Service (SLS). In SLS, logs are filtered, aggregated, and standardized in real time using pre-configured query and analysis dashboards. The processed alarm information (including key fields such as anomaly type, occurrence time, microservice node IP, and task ID) is encapsulated into a specific format (such as Markdown or JSON). Finally, by calling the Webhook interface provided by DingTalk Robot, the formatted alarm message is automatically pushed to the designated operations and maintenance DingTalk group, thereby achieving real-time anomaly detection and notification, significantly shortening fault discovery and response time.
[0055] In this invention, by configuring mapping and reflection invocation mechanisms, adding new export scenarios only requires extending the configuration and data model, without modifying the core code of each service or performing hard-coded adaptations, thus improving development efficiency and truly achieving "pluggable" functionality. Relying on dynamically configured intelligent thresholds, the system can automatically and precisely switch between synchronous (instant response) and asynchronous (reliable execution) modes based on the data volume, solving the problem of mode rigidity, improving the response speed of small data volume exports, ensuring the success rate of large data volume exports, and eliminating timeout bottlenecks. Through a distributed task scheduling framework, parallel execution and fault transfer of asynchronous tasks are achieved, reducing the total processing time of multiple tasks during peak business periods and effectively avoiding task congestion. Simultaneously, the service startup self-healing mechanism can automatically identify and handle "zombie tasks" interrupted by restarts, improving overall task stability and greatly reducing manual maintenance intervention. A unified log collection, processing, and alarm chain ensures that any anomalies can be detected and pushed in real time, fundamentally improving the efficiency of fault detection and location, and effectively controlling the scope of fault impact.
[0056] Figure 2 This is a schematic diagram of a pluggable data export device suitable for a multi-microservice architecture, provided in an embodiment of this specification. The device may include: Module 10 is used to obtain the client's export request; The parsing module 20 is used to parse the target service information corresponding to the export request through a preset configuration mapping relationship; The determination module 30 is used to determine the data volume threshold corresponding to the current export scenario from the dynamic configuration center based on the export request, and obtain the estimated export data volume of the export request through SQL query or interface. Comparison module 40 is used to compare the estimated amount of exported data with the data amount threshold to obtain a comparison result; The export module 50 is used to select a synchronous export mode or an asynchronous export mode based on the comparison result, and execute an export request based on the target service information to complete the data export.
[0057] Optionally, the export module 50 includes: The target service information includes the target service identifier and the target data acquisition method name; In synchronous export mode, the data interface of the corresponding microservice is dynamically called by using reflection mechanism based on the target service identifier and the target data to obtain the method name, generate a file stream in real time and return it. In asynchronous export mode, the target service identifier and the target data acquisition method name are encapsulated into an asynchronous task and submitted to the task scheduling center.
[0058] Optionally, after encapsulating the target service identifier and the target data acquisition method name into an asynchronous task and submitting it to the task scheduling center, the method further includes: The task scheduling center schedules executor nodes, which locate and call the data interface of the corresponding microservice based on the target service identifier, obtain data in a paginated manner, generate an export file, and update the task status.
[0059] Optionally, the export module 50 includes: When the estimated amount of data to be exported is less than or equal to the data volume threshold, select the synchronous export mode; When the estimated amount of data to be exported is greater than the data volume threshold, the asynchronous export mode is selected.
[0060] Optionally, after selecting the asynchronous export mode but before executing the asynchronous export request, the following steps are also included: The estimated amount of exported data is returned to the client for secondary confirmation, and execution continues after receiving the confirmation instruction.
[0061] Optional, also includes: When the project goes live, the microservice nodes restart and automatically query the task scheduling center to see if there are any abnormal tasks among all tasks in the running state. If present, the thread of the abnormal task is interrupted, and an alarm message is output. Optionally, it may also include: Collect exception logs generated by each microservice node during the data export process; The exception logs are filtered and formatted. The processed alarm information is pushed to the operation and maintenance terminal through a preset Webhook interface.
[0062] The functions of the apparatus in this embodiment have been described in the above method embodiments. Therefore, for any parts not detailed in this embodiment, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here.
[0063] Based on the same inventive concept, embodiments of this specification also provide an electronic device.
[0064] The following describes embodiments of the electronic device of the present invention, which can be considered as specific implementations of the methods and apparatus embodiments of the present invention described above. Details described in the embodiments of the electronic device of the present invention should be considered as supplements to the methods or apparatus embodiments described above; details not disclosed in the embodiments of the electronic device of the present invention can be implemented with reference to the methods or apparatus embodiments described above.
[0065] Figure 3 This is a schematic diagram of an electronic device provided as an embodiment of this specification. Refer to the following... Figure 3 The electronic device 300 according to this embodiment of the present invention will be described. Figure 3 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0066] like Figure 3 As shown, the electronic device 300 is presented in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including storage unit 320 and processing unit 310), a display unit 340, etc.
[0067] The storage unit stores program code that can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the processing method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 310 can perform, for example... Figure 1 The steps are shown.
[0068] The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache storage unit 3202, and may further include a read-only memory unit (ROM) 3203.
[0069] The storage unit 320 may also include a program / utility 3204 having a set (at least one) program module 3205, such program module 3205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0070] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0071] Electronic device 300 can also communicate with one or more external devices 400 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable viewers to interact with electronic device 300, and / or with any device that enables electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. Network adapter 360 can communicate with other modules of electronic device 300 via bus 330. It should be understood that, although... Figure 3 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0072] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described in this invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method described above according to this invention. When the computer program is executed by a data processing device, it enables the computer-readable medium to implement the method described above, i.e.: as... Figure 1 The method shown.
[0073] Figure 4 This is a schematic diagram of a computer-readable medium provided for embodiments of this specification.
[0074] accomplish Figure 1 The computer program of the method shown can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0075] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0076] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the audience's computing device, partially on the audience's device, as a standalone software package, partially on the audience's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the audience's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0077] In summary, the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that in practice, general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.
[0078] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
[0079] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0080] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A pluggable data export method suitable for multi-microservice architectures, characterized in that, include: Obtain the client's export request; The target service information corresponding to the export request is parsed using a preset configuration mapping relationship; Based on the export request, the data volume threshold corresponding to the current export scenario is determined from the dynamic configuration center, and the estimated export data volume of the export request is obtained through SQL query or interface. The estimated amount of exported data is compared with the data amount threshold to obtain the comparison result; Based on the comparison results, a synchronous export mode or an asynchronous export mode is selected, and an export request is executed based on the target service information to complete the data export.
2. The pluggable data export method for multi-microservice architecture as described in claim 1, characterized in that, The step of executing an export request based on the target service information to complete the data export includes: The target service information includes the target service identifier and the target data acquisition method name; In synchronous export mode, the data interface of the corresponding microservice is dynamically called by using reflection mechanism based on the target service identifier and the target data to obtain the method name, generate a file stream in real time and return it. In asynchronous export mode, the target service identifier and the target data acquisition method name are encapsulated into an asynchronous task and submitted to the task scheduling center.
3. The pluggable data export method applicable to multi-microservice architecture as described in claim 2, characterized in that, After encapsulating the target service identifier and the target data acquisition method name into an asynchronous task and submitting it to the task scheduling center, the process further includes: The task scheduling center schedules executor nodes, which locate and call the data interface of the corresponding microservice based on the target service identifier, obtain data in a paginated manner, generate an export file, and update the task status.
4. The pluggable data export method for multi-microservice architecture as described in claim 1, characterized in that, The step of selecting synchronous export mode or asynchronous export mode based on the comparison result includes: When the estimated amount of data to be exported is less than or equal to the data volume threshold, select the synchronous export mode; When the estimated amount of data to be exported is greater than the data volume threshold, the asynchronous export mode is selected.
5. The pluggable data export method suitable for multi-microservice architecture as described in claim 1 or 4, characterized in that, After selecting the asynchronous export mode but before executing the asynchronous export request, the following is also included: The estimated amount of exported data is returned to the client for secondary confirmation, and execution continues after receiving the confirmation instruction.
6. The pluggable data export method for multi-microservice architecture as described in claim 2 or 3, characterized in that, Also includes: When the project goes live, the microservice nodes restart and automatically query the task scheduling center to see if there are any abnormal tasks among all tasks in the running state. If it exists, interrupt the thread of the abnormal task and output an alarm message.
7. The pluggable data export method for multi-microservice architecture as described in claim 1, characterized in that, Also includes: Collect exception logs generated by each microservice node during the data export process; The exception logs are filtered and formatted. The processed alarm information is pushed to the operation and maintenance terminal through a preset Webhook interface.
8. A pluggable data export device suitable for multi-microservice architecture, characterized in that, include: The acquisition module is used to acquire the client's export request; The parsing module is used to parse the target service information corresponding to the export request through a preset configuration mapping relationship; The determination module is used to determine the data volume threshold corresponding to the current export scenario from the dynamic configuration center based on the export request, and obtain the estimated export data volume of the export request through SQL query or interface. The comparison module is used to compare the estimated amount of exported data with the data amount threshold to obtain a comparison result; The export module is used to select a synchronous export mode or an asynchronous export mode based on the comparison results, and to execute an export request based on the target service information to complete the data export.
9. An electronic device, wherein, The electronic device includes: A processor; and a memory storing computer-executable instructions, which, when executed, cause the processor to perform the method according to any one of claims 1-7.
10. A computer-readable storage medium, wherein, The computer-readable storage medium stores one or more programs that, when executed by a processor, implement the method of any one of claims 1-7.