Business data export method and system

Abstract

This application relates to the field of data export technology, and provides a method and system for exporting business data. This application, through a business server, converts data export requests sent from user terminals to business client nodes into data export tasks and corresponding task identifiers. This decomposes the data export process between the user terminal and the business system server into a data feedback flow between the business server and the user terminal, and a data acquisition flow between the business server and the business client nodes. This allows the global data export operation, originally handled by the business client nodes, to be transferred to the business server for unified management and data acquisition in stages. This avoids memory overflow or single point-of-failure issues on the business client nodes, resulting in higher stability and reliability for the business system server with business client nodes.

Description

Technical Field

[0001] This application relates to the field of data export technology, specifically to a method and system for exporting business data. Background Technology

[0002] With the deepening of digital transformation across industries, especially in fields such as intelligent connected vehicles, e-commerce, and big data analytics, the amount of data carried and processed by systems is growing exponentially. In daily operations and data analysis, system operators frequently need to export large-scale datasets from business system servers for offline analysis, such as full user profile analysis, massive order record backtracking, or evaluation of vehicle operation logs over long periods. To meet export requirements, after receiving the user's export instruction, the business system server will directly perform a series of operations such as data querying, format conversion, and file generation within the business server, and provide the generated Excel file for the user to download.

[0003] However, since the export operation is executed directly inside the business server, in order to generate a complete data file, the business system server needs to first query the database to retrieve all the data that meets the conditions and load it into the memory of the business server all at once. This consumes the memory resources of the business server rapidly, posing a risk of memory overflow (OOM, Out Of Memory), which may lead to the business server crashing. This not only causes the export task to fail, but also seriously affects the normal operation of core business and the overall stability of the system, resulting in low reliability of the data exported by the business server. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0005] In view of the shortcomings of the prior art described above, this application provides a business data export method and system to improve the reliability of data exported from the business server.

[0006] This application provides a business data export method applied to a business server. The method includes: if a user terminal sends a data export request to a business client node, obtaining a data export task corresponding to the data export request and a task identifier corresponding to the data export task; sending the task identifier to the user terminal and executing the data export task corresponding to the task identifier, wherein the data export task includes performing one or more data collections using the business client node to obtain a paginated table file; if the task identifier sent by the user terminal is received, exporting the paginated table file corresponding to the task identifier to the user terminal.

[0007] In one embodiment of this application, the business client node is a business system server with a client toolkit deployed. The method further includes: if a registration request is received from the business system server, registering the business system server as a business client node and setting the registration status of the business client node to online; performing heartbeat monitoring on the business client node, obtaining heartbeat monitoring results, and counting the number of abnormal samplings where the heartbeat monitoring results are abnormal; if the number of abnormal samplings is greater than or equal to a first preset threshold and less than a second preset threshold, setting the registration status of the business client node to observation; if the number of abnormal samplings is greater than or equal to the second preset threshold, setting the registration status of the business client node to offline; if the registration status of the business client node is observation and the heartbeat monitoring result of the business client node is normal, setting the registration status of the business client node to online.

[0008] In one embodiment of this application, executing the data export task corresponding to the task identifier includes: obtaining the task identifier according to a preset message queue storage; sequentially determining the target identifier from the message queue; responding to the target identifier, requesting the target thread corresponding to the target identifier from a preset asynchronous thread pool; and using the target thread to execute the data export task corresponding to the target identifier.

[0009] In one embodiment of this application, the process of collecting data once or multiple times using the business client node to obtain a paginated table file includes: determining a target node from the business client nodes that are online; if the data export mode of the data export task is the default mode, then calling the data retrieval interface of the target node according to different data page numbers to retrieve the business data corresponding to each data page number; and generating a paginated table file based on the business data corresponding to each data page number.

[0010] In one embodiment of this application, the client toolkit deployed by the business client node includes a default export abstract class. The default export abstract class carries: default export attributes, including at least one of data type, service identifier, template path, and table header mapping relationship; and default export interfaces, including at least one of attribute constructor, mapping relationship interface, task creation interface, and data retrieval interface. The attribute constructor is used to set the data type, service identifier, and template path according to preset initialization parameters. The mapping relationship interface is used to set the table header mapping relationship according to the field annotation corresponding to the data type. The task creation interface is used to request the business server to generate a data export task corresponding to the data export request.

[0011] In one embodiment of this application, data collection is performed once or multiple times using the business client node to obtain a paginated table file, including: determining a target node from the business client nodes that are online; if the data export mode of the data export task is a custom mode, then calling the custom export interface of the target node, the custom export interface being used to generate intermediate files according to a preset file generation logic based on the collected business data, and pushing the intermediate files to the business server in the form of file fragments; generating a paginated table file based on the received file fragments.

[0012] In one embodiment of this application, the client toolkit deployed by the business client node includes a custom export abstract class. The custom export abstract class carries: custom export attributes, including a service identifier; and a custom export interface, including at least one of a quantity statistics interface, a custom generation interface, and a data push method. The quantity statistics interface is used to count the number of exported business data. The custom generation interface is used to generate intermediate files based on the collected business data according to a preset file generation logic. The data push method is used to push the intermediate files to the business server in the form of file fragments.

[0013] In one embodiment of this application, after executing the data export task corresponding to the task identifier, the method further includes: querying unfinished tasks from a preset task pool, the task pool being used to store the acquired data export tasks; querying the task identifier corresponding to the unfinished task from a preset identifier database, the identifier database being used to store the task identifier of the data export task if the data export task is in an execution state; if the identifier database does not store the task identifier corresponding to the unfinished task, and the number of retry attempts for the task corresponding to the unfinished task is less than a preset retry count threshold, then re-executing the unfinished task using an asynchronous thread pool, and updating the number of retry attempts for the task corresponding to the unfinished task.

[0014] In one embodiment of this application, after executing the data export task corresponding to the task identifier, the method further includes: if the user terminal initiates a retry request corresponding to the task identifier, then querying the task identifier from the identifier database and querying the data export task corresponding to the task identifier from the task pool; if the identifier database stores the task identifier, or the task pool does not store the data export task corresponding to the task identifier, then sending a retry error notification to the user terminal; if the identifier database does not store the task identifier, and the task pool stores the data export task corresponding to the task identifier, then re-executing the data export task using an asynchronous thread pool and sending a retry success notification to the user terminal.

[0015] This application also provides a business data export system, including: a business client node; a user terminal, used to send a data export request to the business client node; and a business server, used to obtain a data export task corresponding to the data export request and a task identifier corresponding to the data export task, send the task identifier to the user terminal, and execute the data export task corresponding to the task identifier. The data export task includes using the business client node to perform one or more data collections to obtain a paginated table file, and, if the task identifier sent by the user terminal is received, exporting the paginated table file corresponding to the task identifier to the user terminal.

[0016] The beneficial effects of this application are:

[0017] The business server obtains the data export task and its corresponding task identifier for the data export request, sends the task identifier to the user terminal, and executes the data export task corresponding to the task identifier. If a task identifier is received from the user terminal, the server exports the paginated table file corresponding to that task identifier to the user terminal. In this way, the business server converts the data export request sent from the user terminal to the business client node into a data export task and its corresponding task identifier. This breaks down the data export process between the user terminal and the business system server into a data feedback flow between the business server and the user terminal, and a data acquisition flow between the business server and the business client node. This allows the global data export operation, originally handled by the business client node, to be managed centrally by the business server, and data is collected in stages. This avoids memory overflow or single point-of-failure issues on the business client node, resulting in higher stability and reliability for the business system server with the business client node. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] In the attached diagram: Figure 1 This is a schematic diagram of the system architecture for implementing a business data export method in an embodiment of this application; Figure 2 This is a flowchart illustrating a business data export method in an embodiment of this application; Figure 3 This is a flowchart illustrating another business data export method in an embodiment of this application; Figure 4 This is a flowchart illustrating a method for setting the registration status of a business client node in an embodiment of this application. Figure 5 This is a UML (Unified Modeling Language) class diagram of a default exported abstract class in an embodiment of this application; Figure 6 This is a UML class diagram of a custom exported abstract class in an embodiment of this application; Figure 7 This is a flowchart illustrating another business data export method in an embodiment of this application; Figure 8 This is a flowchart illustrating a data export method corresponding to a default mode in an embodiment of this application. Figure 9 This is a flowchart illustrating a data export method corresponding to a custom mode in an embodiment of this application. Figure 10 This is a schematic diagram of the structure of a business data export system in an embodiment of this application. Detailed Implementation

[0020] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0023] The terms "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0024] Unless otherwise stated, the term "multiple" means two or more.

[0025] In this application, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0026] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0027] Combination Figure 1 As shown, this application provides a system architecture for implementing a business data export method, including a user terminal 101, a business client node 102, and a business service center 103. The business service center 103 includes a business server 10301 and middleware.

[0028] User terminal 101 is used to send user data export requests to business client node 102, receive task identifiers sent by business server 10301, and obtain the pagination table file stored by business server 10301 according to the task identifiers.

[0029] The business client node 102 is set on the business system server. The business client node 102 is used to apply to the business server 10301 for the data export task corresponding to the data export request sent by the user terminal 101.

[0030] In some embodiments, applied to vehicle manufacturing scenarios, the business client node 102 is embedded in the business system servers of different business systems. These business systems include Manufacturing Execution System (MES), Quality Management System (QMS), Enterprise Resource Planning (ERP), Office Automation (OA), Supply Chain Management (SCM), Warehouse Management System (WMS), Product Lifecycle Management (PLM), etc. At least some of these business systems include multiple business system servers. For example, the MES system maintains multiple MES system servers. When a user requests the MES system to export an Excel file containing a large amount of production data, it will instantly occupy the CPU and memory resources of a single MES system server, which can easily cause the business system server to respond slowly or even crash due to memory overflow. Any lag or downtime of the MES system server can lead to production line stoppage and cause huge economic losses.

[0031] Based on the above vehicle manufacturing scenario, this application provides a system architecture for implementing a business data export method, which decouples the business server and the data export task. The business client node 102 no longer bears the heavy responsibility of data aggregation and file generation. Its responsibility is only to respond to the lightweight data pull requests of the business server 10301. Since each request in the batch requests only involves a small part of the data, the memory and CPU consumption of the business client node 102 is minimal and controllable, thereby fundamentally eliminating the risk of performance degradation or crash of the core business system server due to data export, and ensuring the continuity and stability of production and operation.

[0032] The business server 10301 is used to execute the following instructions: obtain the data export task corresponding to the data export request and the task identifier corresponding to the data export task; send the task identifier to the user terminal and execute the data export task corresponding to the task identifier. The data export task includes using the business client node to perform one or more data collections to obtain a paginated table file; if the task identifier sent by the user terminal 101 is received, the paginated table file corresponding to the task identifier is exported to the user terminal.

[0033] In some embodiments, the number of business server 10301 is one or more.

[0034] In some embodiments, the business server 10301 is built on Java 8 and Spring Boot 2.7.12, and the business server 10301 builds a persistence layer, message queue, and buffer middleware through middleware. The persistence layer used includes JPA (Java Persistence API) and MySQL (My Structured Query Language). The message queue used includes Redis (Remote Dictionary Server), RocketMQ (Rocket Message Queue), and Kafka (Distributed Stream Processing Platform). The caching middleware used includes Redis and OSS (Object Storage Service).

[0035] Combination Figure 2 As shown, this application provides a method for exporting business data, applied to a business server. The method includes: Step S201: If the user terminal sends a data export request to the business client node, then obtain the data export task corresponding to the data export request and the task identifier corresponding to the data export task. The task identifier is generated by the business client node and sent from the business client node to the business server through the export task application; Step S202: Send a task identifier to the user terminal and execute the data export task corresponding to the task identifier; The data export task includes collecting data once or multiple times using business client nodes to obtain paginated table files. The pagination table file is an Excel file; Step S203: If a task identifier is received from the user terminal, then export the pagination table file corresponding to the task identifier to the user terminal.

[0036] The business data export method provided in this application obtains the data export task and its corresponding task identifier from the data export request through the business server, sends the task identifier to the user terminal, and executes the data export task corresponding to the task identifier. If the task identifier sent by the user terminal is received, the paginated table file corresponding to the task identifier is exported to the user terminal. In this way, the business server converts the data export request sent from the user terminal to the business client node into a data export task and its corresponding task identifier. This decomposes the data export process between the user terminal and the business system server into a data feedback flow between the business server and the user terminal, and a data acquisition flow between the business server and the business client node. This allows the global data export operation, originally handled by the business client node, to be transferred to the business server for unified management and data acquisition in stages. This avoids memory overflow or single point-of-failure issues on the business client node, resulting in higher stability and reliability for the business system server with the business client node.

[0037] Combination Figure 3 As shown, this application provides a method for exporting business data, including: Step S301: The user terminal sends a data export request to the service client node; Step S302: The business client node generates an export task application based on the export request parameters of the data export request. Export request parameters include filename, service identifier (Provider), query parameters (queryParam), user identifier (userId), tenant information (tenantId), server route identifier (serviceName), header information (headers), data export mode (excelExportMode), and task identifier (taskId); Step S303: The business client node sends an export task request to the business server. Step S304: The business server generates a data export task; Step S305: The service server sends a task identifier to the user terminal; Step S306: The business server requests a target thread from the preset asynchronous thread pool and executes the data export task through the target thread; Step S307: The business server calls the data export interface of the business client node according to the data export task; The data export interface includes a data retrieval interface or a custom export interface; The business server estimates the export time based on the total amount of data returned by the data export interface. Step S308: The business client node sends business data back to the business server. Step S309: The business server generates a paginated table file based on the business data fed back by the business client node; Step S310: The user terminal sends a task identifier to the service server. Step S311: The service server sends the pagination table file corresponding to the task identifier to the user terminal; The business server stores the pagination table file through OSS and sends the file storage address corresponding to the pagination table file to the user terminal according to the task label, triggering the user terminal to download the complete pagination table file itself according to the file storage address.

[0038] Optionally, the business client node is a business system server with a client toolkit deployed.

[0039] Optionally, if a registration request is received from the business system server, the business system server is registered as a business client node, and the registration status of the business client node is set to online. Heartbeat monitoring is performed on the business client node to obtain the heartbeat monitoring results, and the number of abnormal samplings with abnormal heartbeat monitoring results is counted. If the number of abnormal samplings is greater than or equal to a first preset threshold and less than a second preset threshold, the registration status of the business client node is set to observation. If the number of abnormal samplings is greater than or equal to the second preset threshold, the registration status of the business client node is set to offline. If the registration status of the business client node is observation and the heartbeat monitoring result of the business client node is normal, the registration status of the business client node is set to online.

[0040] In some embodiments, by introducing a client toolkit (e.g., the easy-download-support.jar package) into the business system server, and adding the configuration of the business client node in the business client node, the business system server automatically sends a registration request to the business server via HTTP (HyperText Transfer Protocol), k8s (Kubernetes, an open-source system for automatically deploying, scaling and managing containerized applications), etc.

[0041] In some embodiments, the client toolkit includes a Controller layer interface and a task request interface. Users send data export requests to business client nodes through user terminals. The web container of the business client node distributes the received data export requests to the Controller layer interface, so that the Controller layer interface sends an export task application to the business server by calling the task request interface.

[0042] In some embodiments, the business system server with the client toolkit deployed automatically registers with the business server when the business client node starts. The business server needs to record the client's registration information, including service address, business service port, service context, service identifier, etc. The business server periodically checks the health status of the business client node (e.g., heartbeat monitoring results). If there are multiple business servers, each business server locks the health status of the business client node through a distributed lock when checking the health status of the business client node, so as to achieve coordination among multiple business servers. The business server polls the health status of each business client node by calling the / health interface in the client toolkit.

[0043] Combination Figure 4 As shown, this application provides a method for setting the registration status of a business client node, including: Step S401: Obtain the heartbeat monitoring results of the business client node; Step S402: Determine whether the heartbeat monitoring result is abnormal. If yes, proceed to step S403; otherwise, proceed to step S406. Step S403: Determine whether the number of abnormal samplings in the heartbeat monitoring results is greater than or equal to the second preset threshold. If yes, proceed to step S405; otherwise, proceed to step S404. Step S404: Set the registration status of the business client node to the observation status, and proceed to step S401.

[0044] Step S405: Set the registration status of the business client node to offline.

[0045] Step S406: Set the registration status of the business client node to online status, then proceed to step S401.

[0046] Optionally, the client toolkit deployed on the business client node includes a default export abstract class, which carries: default export attributes, including at least one of data type, service identifier, template path, and table header mapping relationship; and default export interfaces, including at least one of attribute constructor, mapping relationship interface, task creation interface, and data retrieval interface. The attribute constructor is used to set the data type, service identifier, and template path according to preset initialization parameters. The mapping relationship interface is used to set the table header mapping relationship according to the field annotations corresponding to the data type. The task creation interface is used to request the business server to generate the data export task corresponding to the data export request.

[0047] Combination Figure 5As shown, this application provides a UML class diagram for a default exported abstract class (AbstractSimpleDownloadClient.class).

[0048] In some embodiments, the default exported properties of the default exported abstract class include data type (dataType), service identifier (Provider), template path (templatePath), and fieldHeaders. The data type (dataType) is used to extract the table headers via reflection and to register the format converter. The service identifier (Provider) is a unique identifier for the business client node, used to provide the server with a factory to find and obtain the corresponding bean instance. The template path (templatePath) is used to customize the cover and template of the paginated table file and can be null. The fieldHeaders are used to carry the mapping relationship between fields and table headers in the paginated table file.

[0049] In some embodiments, the default export interface of the default export abstract class includes a property constructor, a mapping relationship interface, a task creation interface, and a data retrieval interface. The property constructor is used to initialize the data type (dataType), service identifier (Provider), and template path (templatePath) of the parent class by calling the super() method. The mapping relationship interface is called during initialization and is used to automatically generate table header mapping relationships (fieldHeaders) by reading the field annotations on the data type (dataType). The task creation interface is used to request a data export task from the business server. The data retrieval interface is used to return the business data corresponding to the pagination request based on the pagination request sent by the business server.

[0050] In some embodiments, business client nodes need to inherit the default exported abstract class and initialize the default exported properties by declaring a parameterless constructor. The service identifier in the default exported properties needs to be globally unique so that specific business logic can be executed by querying the SpringBean and calling its interface.

[0051] Optionally, the client toolkit deployed on the business client node includes a custom export abstract class, which carries: custom export attributes, including service identifiers; and custom export interfaces, including at least one of a quantity statistics interface, a custom generation interface, and a data push method. The quantity statistics interface is used to count the number of exported business data, the custom generation interface is used to generate intermediate files based on the collected business data according to a preset file generation logic, and the data push method is used to push the intermediate files to the business server in the form of file fragments.

[0052] Combination Figure 6 As shown, this application provides a UML class diagram for a custom exported abstract class (AbstractUserDefinedDownloadClient class).

[0053] In some embodiments, the custom export attribute of the custom export abstract class includes a service identifier, which is a unique identifier for the business client node and is used to provide the server with a factory to find and obtain the corresponding bean instance.

[0054] In some embodiments, the custom export interface of the custom export abstract class includes a quantity statistics interface, a custom generation interface, and a data push method. The quantity statistics interface is used to provide the business server with the total amount of data to be exported. The custom generation interface is used to receive the export request parameters of the data export request, obtain all the necessary data from the database or other data sources according to the export request parameters, and create a custom Excel temporary file locally on the business client node. The custom content includes creating multiple sheets, merging cells, setting font colors, inserting charts, using complex formulas, etc., and finally returning the file path corresponding to the Excel temporary file. The data push method is used to push the local Excel temporary file to the business server in the form of file fragments.

[0055] In some embodiments, the business client node supports user-defined pagination table files through a custom export abstract class.

[0056] In some embodiments, the custom export abstract class deletes the temporary Excel file after implementing the data push method.

[0057] In some embodiments, the custom exported abstract class implements an exception handling mechanism based on AOP (Aspect-Oriented Programming). That is, if an exception occurs during the implementation of the custom generation interface or data push method, the error reason is automatically submitted to the business server, triggering the business server to set the task status to task failure and generate an error log.

[0058] Combination Figure 7 As shown, this application provides a method for exporting business data, applied to a business server. The method includes: Step S701: Obtain the data export task corresponding to the data export request and the task identifier corresponding to the data export task; Step S702: Store the task identifier in the message queue; Step S703: Listen to the message queue using a message listener; Step S704: If a task identifier is detected in the message queue, request the target thread corresponding to the task identifier from the preset asynchronous thread pool. The target thread is used to execute the data export task; Step S705: Using the target thread, determine the target node from the business client nodes that are online; Step S706: Determine whether the data export mode is the default mode. If yes, proceed to step S707; otherwise, proceed to step S708. Step S707: Call the data retrieval interface of the target node according to different data page numbers to retrieve the business data corresponding to each data page number, and then jump to step S709. Step S708: Call the custom export interface of the target node and receive the file fragment returned by the custom export interface, then proceed to step S709. Among them, the custom export interface is used to generate intermediate files based on the collected business data according to the preset file generation logic, and push the intermediate files to the business server in the form of file fragments; Step S709: Generate a paginated table file based on the business data or file fragments fed back by the target node.

[0059] In some embodiments, the target node is not necessarily the business client node originally corresponding to the data export request, but can also be other business client nodes belonging to the same business system. For example, the business system server includes human resource management system A and human resource management system B belonging to the same business system. If a user sends a data export request to human resource management system A, the business client node set in human resource management system B can be used as the target node when executing the data export task corresponding to the data export request.

[0060] Optionally, executing the data export task corresponding to the task identifier includes: obtaining the task identifier according to the preset message queue storage; determining the target identifier from the message queue in sequence; requesting the target thread corresponding to the target identifier from the preset asynchronous thread pool in response to the target identifier; and using the target thread to execute the data export task corresponding to the target identifier.

[0061] In some embodiments, through a unified business server design, the business server communicates with the business client nodes via the HTTP protocol, enabling functions such as data caching, file generation, file storage, and message intermediary layer to directly interface with the service layer, completely isolated from the business client nodes, and without interfering with the existing business of the business system server.

[0062] In this way, compared to directly allowing business client nodes to export business data to user terminals in batches by importing SDKs, this application, through the design of business server and business client nodes, eliminates the need for business client nodes to access persistence layers, caching layers, asynchronous components, etc., thereby reducing the business intrusion on the business system server. At the same time, unifying file generation, task scheduling, and other functions to be handled by the business server can also reduce the performance impact of data export on the business system server.

[0063] In some embodiments, if there are multiple business servers, a distributed lock is used to consume the task identifier stored in the message queue to ensure that the task is not executed repeatedly by multiple business servers.

[0064] In some embodiments, users can customize the format of paginated table files using different data export modes, providing a high degree of flexibility in data export.

[0065] Optionally, data can be collected once or multiple times using business client nodes to obtain a paginated table file, including: determining the target node from the business client nodes that are online; if the data export mode of the data export task is the default mode, calling the data retrieval interface of the target node according to different data page numbers to retrieve the business data corresponding to each data page number; and generating a paginated table file based on the business data corresponding to each data page number.

[0066] Combination Figure 8 As shown, this application provides a data export method corresponding to the default mode, applied to the business server. The method includes: Step S801: Obtain data export task; Step S802: Determine the target node corresponding to the data export task from the business client nodes that are online; Step S803: Generate data retrieval requests corresponding to different data page numbers based on the export request parameters in the data export task. Step S804: Determine the target page number from the data page numbers; Step S805: Call the data retrieval interface of the target node according to the data retrieval request corresponding to the target page number, and receive the business data fed back by the data retrieval interface; Step S806: Determine whether the service data has been fully received. If yes, proceed to step S807; otherwise, proceed to step S804. Step S807: Generate a pagination table file based on the received business data; Among them, the easy-excel open-source component is used to generate paginated table files in Excel format; Among them, pagination table files are stored through OSS service.

[0067] Optionally, data is collected once or multiple times using business client nodes to obtain a paginated table file, including: determining the target node from the business client nodes that are online; if the data export mode of the data export task is a custom mode, then calling the custom export interface of the target node, which is used to generate intermediate files based on the collected business data according to a preset file generation logic, and pushing the intermediate files to the business server in the form of file fragments; and generating a paginated table file based on the received file fragments.

[0068] Combination Figure 9 As shown, this application provides a data export method corresponding to a custom mode, applied to a business server. The method includes: Step S901: Obtain data export task; Step S902: Determine the target node corresponding to the data export task from the business client nodes that are online; Step S903: Based on the export request parameters in the data export task, call the custom export interface of the target node; Among them, the custom export interface is used to generate intermediate files based on the collected business data according to the preset file generation logic, and push the intermediate files to the business server in the form of file fragments; The intermediate files are in Excel format. Step S904: Receive the file fragment returned by the custom export interface; Step S905: Determine whether the file segment has been completely received. If yes, proceed to step S906; otherwise, proceed to step S904. Step S906: Generate a pagination table file based on the received file fragments; The paginated table file is in Excel format. Store paginated table files using OSS service.

[0069] Optionally, after executing the data export task corresponding to the task identifier, the method further includes: querying unfinished tasks from a preset task pool, wherein the task pool is used to store the acquired data export tasks; querying the task identifier corresponding to the unfinished task from a preset identifier database, wherein the identifier database is used to store the task identifier of the data export task if the data export task is in the execution state; if the identifier database does not store the task identifier corresponding to the unfinished task, and the number of retries for the task corresponding to the unfinished task is less than a preset retry threshold, then the unfinished task is re-executed using an asynchronous thread pool, and the number of retries for the task corresponding to the unfinished task is updated.

[0070] In some embodiments, the business server is equipped with an automatic retry mechanism, which includes: the business server queries the unfinished tasks every 5 minutes using a timer, and simultaneously queries the identifier database to see if a corresponding task identifier exists; if the task identifier of the unfinished task exists in the identifier database, it means that the data export task is being executed and no retry is performed; if the task identifier of the unfinished task does not exist in the identifier database, it is re-executed according to the export request parameters of the unfinished task, and the task identifier of the re-executed unfinished task is stored in the identifier database. Each data export task can only be retried 3 times. If the data export task still fails to complete after three executions, the task is abandoned and an error log is generated.

[0071] Optionally, after executing the data export task corresponding to the task identifier, the method further includes: if the user terminal initiates a retry request corresponding to the task identifier, querying the task identifier from the identifier database and querying the data export task corresponding to the task identifier from the task pool; if the identifier database stores the task identifier, or the task pool does not store the data export task corresponding to the task identifier, sending a retry error notification to the user terminal; if the identifier database does not store the task identifier, and the task pool stores the data export task corresponding to the task identifier, re-executing the data export task using an asynchronous thread pool and sending a retry success notification to the user terminal.

[0072] In some embodiments, the business server is equipped with a manual retry mechanism, which includes: the user sending a retry request to the business server through the user terminal; the business server querying the task identifier from the identifier database; if the task identifier does not exist in the identifier database, querying the data export task corresponding to the task identifier from the task pool, and re-executing the data export task according to the thread in the asynchronous thread pool.

[0073] Combination Figure 10 As shown, this application provides a business data export system, including a user terminal 101, a business client node 102, and a business server 10301.

[0074] User terminal 101 is used to send data export requests to business client nodes.

[0075] The business server 10301 is used to obtain the data export task corresponding to the data export request and the task identifier corresponding to the data export task, send the task identifier to the user terminal, and execute the data export task corresponding to the task identifier. The data export task includes using the business client node to perform one or more data collections to obtain a paginated table file, and if the task identifier sent by the user terminal is received, exporting the paginated table file corresponding to the task identifier to the user terminal.

[0076] The business data export system provided in this application obtains the data export task and its corresponding task identifier from the data export request through the business server, sends the task identifier to the user terminal, and executes the data export task corresponding to the task identifier. If a task identifier is received from the user terminal, the system exports the paginated table file corresponding to the task identifier to the user terminal. In this way, the business server converts the data export request sent from the user terminal to the business client node into a data export task and its corresponding task identifier. This decomposes the data export process between the user terminal and the business system server into a data feedback flow between the business server and the user terminal, and a data acquisition flow between the business server and the business client node. This allows the global data export operation, originally handled by the business client node, to be transferred to the business server for unified management and data acquisition in stages. This avoids memory overflow or single point-of-failure issues on the business client node, resulting in higher stability and reliability for the business system server with the business client node.

[0077] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0078] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some sub-samples may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. Furthermore, the functional units in this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than those disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for exporting business data, characterized in that, Applied to the business server, the method includes: If a user terminal sends a data export request to a business client node, then the data export task corresponding to the data export request and the task identifier corresponding to the data export task are obtained. Send the task identifier to the user terminal and execute the data export task corresponding to the task identifier. The data export task includes using the business client node to collect data once or multiple times to obtain a paginated table file. If a task identifier is received from the user terminal, then the pagination table file corresponding to the task identifier is exported to the user terminal.

2. The method according to claim 1, characterized in that, The business client node is a business system server with a client toolkit deployed. The method further includes: If a registration request is received from the business system server, the business system server is registered as a business client node, and the registration status of the business client node is set to online. Heartbeat monitoring is performed on the business client nodes to obtain heartbeat monitoring results, and the number of abnormal samplings where the heartbeat monitoring results are abnormal is counted. If the number of abnormal samplings is greater than or equal to a first preset threshold and the number of abnormal samplings is less than a second preset threshold, then the registration status of the business client node is set to observation status. If the number of abnormal samplings is greater than or equal to the second preset threshold, then the registration status of the business client node is set to offline. If the registration status of the business client node is in the observation state and the heartbeat monitoring result of the business client node is normal, then the registration status of the business client node is set to the online state.

3. The method according to claim 1, characterized in that, Executing the data export task corresponding to the task identifier includes: The task identifier is obtained based on the preset message queue storage; The target identifier is determined sequentially from the message queue; In response to the target identifier, a target thread corresponding to the target identifier is requested from a preset asynchronous thread pool; The target thread is used to execute the data export task corresponding to the target identifier.

4. The method according to claim 1, characterized in that, By using the aforementioned business client node to perform one or more data collections, a paginated table file is obtained, including: Identify the target node from the business client nodes that are currently online; If the data export mode of the data export task is the default mode, then the data retrieval interface of the target node is called according to different data page numbers to retrieve the business data corresponding to each data page number respectively; Generate a pagination table file based on the business data corresponding to each of the data page numbers.

5. The method according to claim 4, characterized in that, The client toolkit deployed on the business client node includes a default exported abstract class, which carries: The default exported attributes include at least one of the following: data type, service identifier, template path, and table header mapping relationship; The default export interface includes at least one of the following: attribute constructor, mapping relationship interface, task creation interface, and data retrieval interface. The attribute constructor is used to set the data type, the service identifier, and the template path according to preset initialization parameters. The mapping relationship interface is used to set the table header mapping relationship according to the field annotations corresponding to the data type. The task creation interface is used to request the business server to generate the data export task corresponding to the data export request.

6. The method according to claim 1, characterized in that, By using the aforementioned business client node to perform one or more data collections, a paginated table file is obtained, including: Identify the target node from the business client nodes that are currently online; If the data export mode of the data export task is a custom mode, then the custom export interface of the target node is called. The custom export interface is used to generate intermediate files according to the collected business data according to the preset file generation logic, and push the intermediate files to the business server in the form of file fragments respectively. Generate a paginated table file based on the received file fragments.

7. The method according to claim 6, characterized in that, The client toolkit deployed on the business client node includes a custom exported abstract class, which carries: Custom export attributes, including service identifier; A custom export interface includes at least one of a quantity statistics interface, a custom generation interface, and a data push method. The quantity statistics interface is used to count the number of exported business data. The custom generation interface is used to generate intermediate files based on the collected business data according to a preset file generation logic. The data push method is used to push the intermediate files to the business server in the form of file fragments.

8. The method according to any one of claims 1 to 7, characterized in that, After executing the data export task corresponding to the task identifier, the method further includes: Unfinished tasks are retrieved from a preset task pool, which is used to store the acquired data export tasks; The system queries the preset identifier database to find the task identifier corresponding to the incomplete task. The identifier database is used to store the task identifier of the data export task if the data export task is in the execution state. If the identifier database does not store the task identifier corresponding to the incomplete task, and the number of retries for the incomplete task is less than a preset retry threshold, then the incomplete task is re-executed using an asynchronous thread pool, and the number of retries for the incomplete task is updated.

9. The method according to claim 8, characterized in that, After executing the data export task corresponding to the task identifier, the method further includes: If the user terminal initiates a retry request corresponding to the task identifier, then the task identifier is queried from the identifier database, and the task is exported from the task pool by querying the data corresponding to the task identifier. If the identifier database stores the task identifier, or if the task pool does not store the data export task corresponding to the task identifier, a retry error notification is sent to the user terminal. If the identifier database does not store the task identifier, but the task pool stores the data export task corresponding to the task identifier, then the data export task is re-executed using an asynchronous thread pool, and a retry success notification is sent to the user terminal.

10. A business data export system, characterized in that, include: Business client node; The user terminal is used to send a data export request to the business client node; The business server is used to obtain the data export task corresponding to the data export request and the task identifier corresponding to the data export task, send the task identifier to the user terminal, and execute the data export task corresponding to the task identifier. The data export task includes using the business client node to perform one or more data collections to obtain a paginated table file, and if the task identifier sent by the user terminal is received, exporting the paginated table file corresponding to the task identifier to the user terminal.

Images