Data export method, application service system, computer device and storage medium

Abstract

Embodiments of the present application disclose a data export method, an application service system, a computer device and a storage medium. The method is applied to the application service system, and the method comprises the following steps: a service node receives a data export request sent by a user terminal, the data export request comprising a data filtering condition; a data export task is created according to the data filtering condition and a preset paging logic, a corresponding task record is created in a database, and the data export task is sent to a thread pool; a plurality of paging data are sequentially exported from the database by the thread pool; a CPU load of the application service system is monitored in real time by an export component, a task pause instruction of the data export task is sent to the thread pool when the CPU load is greater than a preset load threshold; and a task continue execution instruction of the data export task is sent to the thread pool by the export component according to the task record when the CPU load is less than or equal to the load threshold. According to the present application, the stability of a core business in the application service system can be avoided from being affected by the export task.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to data export methods, application service systems, computer equipment, and storage media. Background Technology

[0002] In enterprise-level application service systems (such as enterprise data systems, big data platforms, cloud service platforms, or government systems), users often need to export large amounts of data from the database of the application service system.

[0003] However, when the amount of data to be exported is extremely large, the export task will occupy system resources for a long time when exporting data through traditional data export methods. Especially during peak business hours, the export task will affect the stability of the core business of the application service system. Summary of the Invention

[0004] This application provides a data export method, an application service system, a computer device, and a storage medium, which can prevent the export task from affecting the stability of the core business in the application service system.

[0005] In a first aspect, embodiments of this application provide a data export method, which is applied to an application service system. The application service system includes service nodes, an export component, a thread pool, and a database. The method includes:

[0006] The service node receives data export requests sent by user terminals, and the data export requests include data filtering conditions.

[0007] The service node creates a data export task based on the data filtering conditions and preset pagination logic, creates a task record corresponding to the data export task in the database, and sends the data export task to the thread pool. The task record is used to record the execution status, data filtering conditions, and real-time updated data export position of the data export task.

[0008] The thread pool responds to the data export task, sequentially exporting multiple sets of paginated data from the database, and storing the exported paginated data in a local temporary file in real time.

[0009] During the execution of the data export task by the thread pool, the CPU load of the application service system is monitored in real time by the export component. When the CPU load exceeds a preset load threshold, a task pause instruction for the data export task is sent to the thread pool, and the execution status is updated to paused.

[0010] When the CPU load is less than or equal to the load threshold, the export component sends a task continuation instruction for the data export task to the thread pool according to the task record, and updates the execution status from the paused state to the export state.

[0011] In some embodiments, the step of responding to the data export task through the thread pool, sequentially exporting multiple sets of paginated data from the database, and storing the exported paginated data to a local temporary file in real time includes:

[0012] The target worker thread is allocated to the data export task through the thread pool;

[0013] The target worker thread receives the data export task and determines the pagination start position of the current export page based on the data export position in the task record.

[0014] The target worker thread exports the pagination data corresponding to the current exported page that meets the data filtering conditions from the database according to the pagination start position and the pagination data volume corresponding to the pagination logic, and stores the exported pagination data of the current exported page to the local temporary file in real time.

[0015] Once the current export page has been exported, update the current export page and return to the step of determining the pagination start position of the current export page based on the data export position in the task record, until all data in the database that meets the data filtering conditions has been exported.

[0016] In some embodiments,

[0017] The step of allocating target worker threads for the data export task through the thread pool includes:

[0018] If there are idle core threads in the thread pool, then the target worker thread is determined from the idle core threads;

[0019] If there are no idle core threads in the thread pool and the task queue in the thread pool is not full, then the data export task is added to the task queue, and the target worker thread is determined through the task queue.

[0020] If there are no idle core threads in the thread pool and the task queue in the thread pool is full, then a temporary thread is created as the target worker thread.

[0021] The data export task is assigned to the target worker thread.

[0022] In some embodiments, after storing the pagination data of the currently exported page to the local temporary file in real time, the method further includes:

[0023] The total amount of data in the database that meets the data filtering criteria is determined by the target worker thread.

[0024] Based on the target worker thread, the current export progress is determined according to the total data volume and the total exported data volume corresponding to all paginated data in the local temporary file.

[0025] When a progress query request is received from the user terminal, the current export progress is returned to the user terminal.

[0026] In some embodiments, after sending the data export task to the thread pool, the method further includes:

[0027] If the application service system is in a restart state, start the service node;

[0028] Once the service node is started, the task record is queried through the service node to determine whether the execution status of the data export task is in the export status.

[0029] If the execution status of the data export task is in the export state, then return to the execution step of sending the data export task to the thread pool.

[0030] In some embodiments, after sending the data export task to the thread pool, the method further includes:

[0031] The system returns information indicating that the export task has started execution to the user terminal.

[0032] In some embodiments, after responding to the data export task through the thread pool, sequentially exporting multiple sets of paginated data from the database, and storing the exported paginated data in a local temporary file in real time, the method further includes:

[0033] If the data export task is completed, the multiple pages of data in the local temporary file are identified as the target export data. The target export data is sent to the cloud server through the thread pool, and the target export data in the local temporary file is deleted.

[0034] The target download link is returned to the user terminal through the thread pool. The target download link is the download link of the target exported data in the cloud server.

[0035] Secondly, embodiments of this application also provide an application service system, which includes:

[0036] The application service system includes service nodes, export components, thread pools, and a database, wherein:

[0037] The service node is used to receive data export requests sent by user terminals, the data export requests including data filtering conditions; create data export tasks according to the data filtering conditions and preset pagination logic; create task records corresponding to the data export tasks in the database; and send the data export tasks to the thread pool. The task records are used to record the execution status, data filtering conditions, and real-time updated data export positions corresponding to the data export tasks.

[0038] The thread pool is used to execute the data export task, sequentially exporting multiple sets of paginated data from the database, and storing the exported paginated data to a local temporary file in real time.

[0039] The export component is used to monitor the CPU load of the application service system in real time during the execution of the data export task in the thread pool. When the CPU load is greater than a preset load threshold, it sends a task pause instruction for the data export task to the thread pool and updates the execution status to paused. When the CPU load is less than or equal to the load threshold, it sends a task continue execution instruction for the data export task to the thread pool according to the task record and updates the execution status from paused to export.

[0040] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor. The memory stores a computer program, and the computer device deploys an application service system. The application service system includes service nodes, export components, thread pools, and a database. When the processor executes the computer program, it implements the above-described method.

[0041] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the above-described method.

[0042] This application provides a data export method, an application service system, a computer device, and a storage medium. The method is applied to an application service system, which includes service nodes, an export component, a thread pool, and a database. On one hand, this application exportes corresponding data from the database in pages and stores the exported data in local temporary files (such as a disk), thereby reducing the memory footprint of the export task on the application service system. On the other hand, during the execution of the data export task in the thread pool, the export component monitors the CPU load of the application service system in real time. When the CPU load exceeds a preset load threshold, a task pause instruction for the data export task is sent to the thread pool, achieving automatic protection against overload and preventing the data export task from affecting the stability of core business operations within the application service system. Furthermore, when the detected CPU load is less than or equal to the load threshold, since task records for the data export task are persisted in the database beforehand, the data export task can continue to be executed based on these records, thus achieving automatic resumption of the data export task. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram illustrating an application scenario of the data export method provided in the embodiments of this application;

[0045] Figure 2 A flowchart illustrating the data export method provided in this application embodiment;

[0046] Figure 3 A schematic diagram of a sub-process of the data export method provided in the embodiments of this application;

[0047] Figure 4 A schematic block diagram of an application service system provided in the embodiments of this application;

[0048] Figure 5 A schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0051] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0052] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0053] This application provides a data export method, an application service system, a computer device, and a storage medium.

[0054] The execution entity of this data export method can be the application service system provided in the embodiments of this application, or a computer device that integrates the application service system. The application service system can be implemented in hardware or software. Specifically, the application service system can be a system corresponding to an enterprise data system, a big data platform, a cloud service platform, or a government system.

[0055] Please see Figure 1 , Figure 1 This diagram illustrates an application scenario of the data export method provided in this embodiment. The data export method is applied to... Figure 1 In the application service system 10, the application service system 10 includes service nodes, export components, thread pools and databases. Users can export the data they need from the application service system 10 through user terminal 20.

[0056] Figure 2 This is a flowchart illustrating the data export method provided in an embodiment of this application. For example... Figure 2 As shown, the method includes the following steps S110-S150.

[0057] S110. Receive a data export request sent by a user terminal through the service node. The data export request includes data filtering conditions.

[0058] In this embodiment, the user can open the data export interface of the application service system through a webpage or APP client on the user terminal, then select the data to be exported (including data type and data range) in the data export interface to determine the data filtering conditions, and then click the "Export" control to send a data export request to the application service system. The data export request includes data filtering conditions, and may also include the user identifier that sent the data export request, so as to generate download records for different users in the application service system.

[0059] S120. The service node creates a data export task based on the data filtering conditions and preset pagination logic, creates a task record corresponding to the data export task in the database, and sends the data export task to the thread pool. The task record is used to record the execution status, data filtering conditions, and real-time updated data export position corresponding to the data export task.

[0060] In this embodiment, when the service node receives a data export request, it does not immediately begin processing the data. Instead, it first initializes the task, performing corresponding data export task initialization processing based on the data export request. A data export task is created according to data filtering conditions (e.g., data of type C within the time period from time A to time B) and preset pagination logic (e.g., 1000 data entries per page). A local temporary file corresponding to the data export task is created, and a task record for the data export task is persisted in the database. The local temporary file stores the data exported from the database. After storing one page of exported data, the corresponding intermediate data in memory is deleted to reduce memory usage. This task record is used for resuming interrupted data transfers when the task is paused (active pause due to excessive CPU load) or interrupted abnormally (passive interruption during server restart).

[0061] S130. The thread pool responds to the data export task, sequentially exporting multiple sets of paginated data from the database, and storing the exported paginated data in a local temporary file in real time.

[0062] In this embodiment, the data export task is executed through a thread pool, and then the data is exported from the database page by page, without exporting all the data at once. After the data is exported, it is directly stored in a local temporary file on the disk. After exporting a page of data, the corresponding intermediate data in memory will also be deleted. It can be seen that this embodiment reduces the memory usage of the export task by pagination and local temporary files. The local temporary file can be a temporary Excel file.

[0063] Specifically, in some embodiments, please refer to Figure 3 Step S130 includes:

[0064] S1301. Assign a target worker thread to the data export task through the thread pool;

[0065] S1302. Receive the data export task through the target worker thread, and determine the pagination start position of the current export page according to the data export position in the task record;

[0066] S1303. The target working thread exports the pagination data corresponding to the current exported page that meets the data filtering conditions from the database according to the pagination start position and the pagination data volume corresponding to the pagination logic, and stores the exported pagination data of the current exported page to the local temporary file in real time.

[0067] S1304. After the current export page is exported, update the current export page and return to the step of determining the pagination start position of the current export page based on the data export position in the task record, until all data in the database that meets the data filtering conditions has been exported.

[0068] In this embodiment, the data export task is executed through a thread pool. After receiving the data export task, the thread pool first allocates a target worker thread specifically for it. This target worker thread is dedicated to executing the data export task. During the export process, the target worker thread updates the data export position in the task record in real time. After the current page is exported, the end position of the current page is updated in the task record to indicate the data export position. Then, the target worker thread proceeds to export the next page. At this point, the target thread determines the pagination start position of the current page based on the latest data export position in the task record (the end position of the previous page), until all data in the database that meets the data filtering criteria has been exported.

[0069] For example, if the pagination logic corresponds to a pagination data volume of 10,000 records, meaning 10,000 records are exported per page, and 10,000 records are exported at a time, the target worker thread searches the database for data that meets the data filtering conditions. If 1 million records meet the conditions, the first 10,000 records are exported, and the current export position is recorded, for example, the 10,000th row is exported, completing the export of the current page's data. Then, the target worker thread executes the export of the next page's data. Querying the task record, it finds that the 10,000th row has been exported (the data export position). At this point, the 10,001st row is used as the starting position for the next page's data export task, until all 1 million records are exported to a local temporary file.

[0070] In the step of storing the paginated data of the currently exported page to the local temporary file in real time, it can be done by exporting each page of data to memory first, and then exporting the currently exported page of data to the local temporary file. In this case, it is only necessary to update the end position of the current page in the task record. Alternatively, each line of data can be exported directly from memory to the local temporary file. In this case, the current exported line needs to be updated in the task record. It can be seen that the real-time data export position in the task record can be the end position of each page or the current exported line of each page.

[0071] In some embodiments, allocating a target worker thread for the data export task through the thread pool includes: if there are idle core threads in the thread pool, determining the target worker thread from the idle core threads; if there are no idle core threads in the thread pool and the task queue in the thread pool is not full, adding the data export task to the task queue and determining the target worker thread through the task queue; if there are no idle core threads in the thread pool and the task queue in the thread pool is full, creating a temporary thread as the target worker thread; and allocating the data export task to the target worker thread.

[0072] When there are no idle core threads in the thread pool and the task queue in the thread pool is full, it is necessary to further determine whether the total number of threads in the current thread pool is greater than the preset total number of threads. If it is not greater, a temporary thread is created as the target worker thread. If it is greater, the thread pool resolves the task and reports the thread pool full information to the service node. At this time, the service node records the execution status of the task to be started in the task record corresponding to the task. The service node polls the database periodically (e.g., every 3 minutes) to see if there is a task record with the execution status of task to be started. If it exists, the corresponding data export task is sent to the thread pool. At this time, the user terminal is unaware of the data export request and waits for the data export to be completed.

[0073] In this embodiment, the service node submits the task to the thread pool for asynchronous processing. After the service node sends the data export task to the thread pool, it can return information about the start of the export task to the user terminal. For example, a prompt "Export task has been created and is being processed in the background" will pop up on the data export interface of the user terminal, and a task ID and an empty progress bar will be displayed. The user can minimize this window and continue to do other work.

[0074] Furthermore, after storing the pagination data of the currently exported page to the local temporary file in real time, the method further includes:

[0075] The target worker thread determines the total amount of data in the database that meets the data filtering conditions; based on the target worker thread, the current export progress is determined according to the total amount of data and the total amount of exported data corresponding to all paginated data in the local temporary file; when a progress query request is received from the user terminal, the current export progress is returned to the user terminal.

[0076] In this embodiment, the user terminal will periodically (e.g., once every 30 seconds) send a progress query request (task processing percentage) to the application service system, so that the front-end user can transparently perceive the execution status of the background task.

[0077] In other embodiments, when the current export progress is updated, the application service system can also proactively send the current export progress to the corresponding user terminal.

[0078] Furthermore, users can track the execution status of export tasks (export status (running) / paused / completed) and estimated remaining time in real time on the user terminal. In addition, abnormal states can be visualized on the user terminal, such as displaying warning prompts when paused due to high load, and displaying recovery progress when a task is interrupted, making the task status transparent and traceable.

[0079] S140. During the execution of the data export task in the thread pool, the CPU load of the application service system is monitored in real time through the export component. When the CPU load is greater than a preset load threshold, a task pause instruction for the data export task is sent to the thread pool, and the execution status is updated to paused.

[0080] In this embodiment, during the execution of the data export task, the thread pool monitors the CPU load of the application service system in real time through the export component. If the current CPU load is greater than a preset load threshold (such as 80%), a task pause instruction for the data export task is sent to the thread pool. At this time, the thread pool pauses the execution of the data export task to reduce the CPU load occupied by the export task, and at the same time, the execution status of the task in the task record is changed to the paused state.

[0081] Specifically, the export component can periodically detect the CPU load of the application service system based on a preset time interval (e.g., 1 minute).

[0082] For example, when the evening rush hour arrives and online order traffic surges, the CPU utilization rate reaches 85%. The export component detects in real time that the current CPU utilization rate is greater than 80% and sends a task pause command to the thread pool to avoid the export of large amounts of data affecting the main service response.

[0083] S150. When the CPU load is less than or equal to the load threshold, the export component sends a task continuation instruction for the data export task to the thread pool according to the task record, and updates the execution status from the paused state to the export state.

[0084] In this embodiment, when the export component detects that the current CPU load is less than or equal to the load threshold, the export component will also check whether there is a task in the database whose execution status is paused. If there is, the component will send a task continuation execution instruction to the thread pool so that the corresponding thread in the thread pool can continue to execute the export task. At the same time, the execution status in the task record of the export task will be updated from paused to export, thereby realizing automatic protection when the server system load exceeds the limit and automatic task recovery.

[0085] Furthermore, in some embodiments, after sending the data export task to the thread pool, the method further includes:

[0086] If the application service system is in a restart state, start the service node;

[0087] Once the service node is started, the task record is queried through the service node to determine whether the execution status of the data export task is in the export status.

[0088] If the execution status of the data export task is in the export state, then return to the execution step of sending the data export task to the thread pool.

[0089] In this embodiment, if the application service system restarts (such as a scheduled server restart or an abnormal power outage restart), the traditional export method will lose the export task, resulting in export failure. However, in this embodiment, the application service system will automatically resume the export task after restarting. Specifically, the service node will be automatically started, and after the service node starts, it will query the task records of all tasks in the database to check if there are any data export tasks with an execution status of export. If so, it means that these data export tasks were not completed before the restart. At this time, the export task needs to be continued. Since the data exported before the restart has been stored in a local temporary file, there is no need to start the export from the beginning. At this time, the data export task is regenerated and resent to the thread pool. The thread pool will find the breakpoint position (data export position) in the task record and continue to execute the export operation from the power outage position, realizing the breakpoint resumption of the export task and improving network bandwidth utilization.

[0090] Furthermore, in some embodiments, after responding to the data export task through the thread pool, sequentially exporting multiple sets of paginated data from the database, and storing the exported paginated data to a local temporary file in real time, the method further includes:

[0091] If the data export task is completed, the multiple pages of data in the local temporary file are identified as the target export data. The target export data is sent to the cloud server through the thread pool, and the target export data in the local temporary file is deleted.

[0092] The target download link is returned to the user terminal through the thread pool. The target download link is the download link of the target exported data in the cloud server.

[0093] In this embodiment, after the data export task is completed, the execution status in the task record is updated to "completed." Multiple pages of data in the local temporary file are identified as the target export data, generating the final data file. The target export data is then uploaded to a cloud server, such as Object Storage Service (OSS). At this point, a corresponding cloud download link can be returned to the user's terminal, allowing the user to repeatedly download the exported data using this link.

[0094] In other embodiments, it may not be necessary to upload the exported data to the cloud server. In this case, after the application service system finishes exporting the data, it returns the exported data in the local temporary file to the user terminal.

[0095] In this embodiment, the application service system can specifically be a server corresponding to an enterprise data system, a big data platform, a cloud service platform, or a government system. Specifically, in an enterprise data system, the exported data can be Enterprise Resource Planning (ERP) report export, Customer Relationship Management (CRM) customer data export, and financial system statement generation (export); in a big data platform, the exported data can be Hive / Spark query results export, data warehouse batch export, and real-time computing engine result output; in a cloud service platform, the exported data can be cloud database query results export, log analysis system output, and monitoring data report generation (export); in the Internet of Things (IoT) field, the exported data can be device historical data export, sensor data analysis report export, and maintenance record batch export; in a government system, the exported data can be population statistics export, administrative approval record archiving, and public service data export.

[0096] In summary, the method provided in this application embodiment is applied to an application service system, which includes service nodes, an export component, a thread pool, and a database. On one hand, this application embodiment exports the corresponding data from the database in pages and stores the exported data in a local temporary file (such as a disk), thereby reducing the memory occupation of the export task on the application service system. On the other hand, during the execution of the data export task by the thread pool, this application embodiment monitors the CPU load of the application service system in real time through the export component. When the CPU load exceeds a preset load threshold, a task pause instruction for the data export task is sent to the thread pool to achieve automatic protection when the load exceeds the threshold, preventing the data export task from affecting the stability of the core business in the application service system. Moreover, when the CPU load is detected to be less than or equal to the load threshold, since the task record of the data export task is persisted in the database in advance, the data export task can continue to be executed according to the task record, thereby achieving automatic resumption of the data export task.

[0097] Figure 4 This is a schematic block diagram of an application service system provided in an embodiment of this application. For example... Figure 4 As shown, corresponding to the above data export method, this application also provides an application service system 10. The application service system 10 includes a service node 401, an export component 402, a thread pool 403, and a database 404, wherein:

[0098] The service node 401 is used to receive a data export request sent by a user terminal, the data export request including data filtering conditions; create a data export task according to the data filtering conditions and preset pagination logic; create a task record corresponding to the data export task in the database 404; and send the data export task to the thread pool 403. The task record is used to record the execution status, data filtering conditions and real-time updated data export position corresponding to the data export task.

[0099] The thread pool 403 is used to execute the data export task, sequentially exporting multiple sets of paginated data from the database 404, and storing the exported paginated data in a local temporary file in real time.

[0100] The export component 402 is used to monitor the CPU load of the application service system 10 in real time during the execution of the data export task by the thread pool 403. When the CPU load is greater than a preset load threshold, it sends a task pause instruction for the data export task to the thread pool 403 and updates the execution status to a paused state. When the CPU load is less than or equal to the load threshold, it sends a task continue execution instruction for the data export task to the thread pool 403 according to the task record and updates the execution status from a paused state to an export state.

[0101] In some embodiments, the thread pool 403 performs the data export task through the following steps:

[0102] A target worker thread is assigned to the data export task; the target worker thread receives the data export task, determines the pagination start position of the current export page based on the data export position in the task record, and exports the pagination data corresponding to the current export page that meets the data filtering conditions from the database 404 according to the pagination start position and the pagination data volume corresponding to the pagination logic, and stores the exported pagination data of the current export page in the local temporary file in real time; when the current export page is exported, the current export page is updated, and the step of determining the pagination start position of the current export page based on the data export position in the task record is returned to be executed until all data that meets the data filtering conditions in the database 404 has been exported.

[0103] In some embodiments, the thread pool 403 allocates target worker threads to the data export task through the following steps:

[0104] If there are idle core threads in the thread pool 403, the target worker thread is determined from the idle core threads; if there are no idle core threads in the thread pool 403 and the task queue in the thread pool 403 is not full, the data export task is added to the task queue, and the target worker thread is determined through the task queue; if there are no idle core threads in the thread pool 403 and the task queue in the thread pool 403 is full, a temporary thread is created as the target worker thread; and the data export task is assigned to the target worker thread.

[0105] In some embodiments, after the pagination data of the currently exported page is stored in the local temporary file in real time:

[0106] The target worker thread is also used to determine the total amount of data in the database 404 that meets the data filtering conditions; determine the current export progress based on the total amount of data and the total amount of exported data corresponding to all paginated data in the local temporary file; and return the current export progress to the user terminal when a progress query request is received from the user terminal.

[0107] In some embodiments, after sending the data export task to the thread pool 403, the application service system 10 further includes:

[0108] A startup node is used to start the service node 401 when the application service system 10 is in a restart state;

[0109] The service node 401 is also configured to query the task record after startup to determine whether the execution status of the data export task is in the export state; if the execution status of the data export task is in the export state, then return to the step of sending the data export task to the thread pool 403.

[0110] In some embodiments, after sending the data export task to the thread pool 403, the method further includes:

[0111] The system returns information indicating that the export task has started execution to the user terminal.

[0112] In some embodiments, after the thread pool 403 responds to the data export task, sequentially exports multiple sets of paginated data from the database 404, and stores the exported paginated data in a local temporary file in real time, the thread pool 403 is further used for:

[0113] If the data export task is completed, the multiple pages of data in the local temporary file are identified as the target export data, the target export data is sent to the cloud server, and the target export data in the local temporary file is deleted; a target download link is returned to the user terminal, the target download link being the download link for the target export data in the cloud server.

[0114] In summary, the method provided in this application embodiment is applied to an application service system 10, which includes a service node 401, an export component 402, a thread pool 403, and a database 404. On one hand, this application embodiment exports the corresponding data in the database 404 through pagination and stores the exported data in a local temporary file (such as a disk), thereby reducing the memory occupation of the export task on the application service system 10. On the other hand, during the execution of the data export task by the thread pool 403, this application embodiment monitors the CPU load of the application service system 10 in real time through the export component 402. When the CPU load exceeds a preset load threshold, a task pause instruction for the data export task is sent to the thread pool 403 to achieve automatic protection when the load exceeds the threshold, preventing the data export task from affecting the stability of the core business in the application service system 10. Moreover, when the CPU load is detected to be less than or equal to the load threshold, since the task record of the data export task is persisted in the database 404 in advance, the data export task can continue to be executed according to the task record, thereby achieving automatic resumption of the data export task.

[0115] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above application service system and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0116] The aforementioned application service system can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0117] Please see Figure 5 , Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 deploys an application service system provided in this application, which includes service nodes, export components, thread pools, and a database.

[0118] See Figure 5 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0119] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a data export method.

[0120] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0121] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a data export method.

[0122] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps:

[0124] The service node receives data export requests sent by user terminals, and the data export requests include data filtering conditions.

[0125] The service node creates a data export task based on the data filtering conditions and preset pagination logic, creates a task record corresponding to the data export task in the database, and sends the data export task to the thread pool. The task record is used to record the execution status, data filtering conditions, and real-time updated data export position of the data export task.

[0126] The thread pool responds to the data export task, sequentially exporting multiple sets of paginated data from the database, and storing the exported paginated data in a local temporary file in real time.

[0127] During the execution of the data export task by the thread pool, the CPU load of the application service system is monitored in real time by the export component. When the CPU load exceeds a preset load threshold, a task pause instruction for the data export task is sent to the thread pool, and the execution status is updated to paused.

[0128] When the CPU load is less than or equal to the load threshold, the export component sends a task continuation instruction for the data export task to the thread pool according to the task record, and updates the execution status from the paused state to the export state.

[0129] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0130] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0131] Therefore, this application also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the following steps:

[0132] The service node receives data export requests sent by user terminals, and the data export requests include data filtering conditions.

[0133] The service node creates a data export task based on the data filtering conditions and preset pagination logic, creates a task record corresponding to the data export task in the database, and sends the data export task to the thread pool. The task record is used to record the execution status, data filtering conditions, and real-time updated data export position of the data export task.

[0134] The thread pool responds to the data export task, sequentially exporting multiple sets of paginated data from the database, and storing the exported paginated data in a local temporary file in real time.

[0135] During the execution of the data export task by the thread pool, the CPU load of the application service system is monitored in real time by the export component. When the CPU load exceeds a preset load threshold, a task pause instruction for the data export task is sent to the thread pool, and the execution status is updated to paused.

[0136] When the CPU load is less than or equal to the load threshold, the export component sends a task continuation instruction for the data export task to the thread pool according to the task record, and updates the execution status from the paused state to the export state.

[0137] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0140] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0142] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data export method, characterized by, The method is applied to an application service system, and the application service system comprises a service node, an export component, a thread pool and a database. A data export request sent by a user terminal is received by the service node, and the data export request comprises a data filtering condition; A data export task is created by the service node according to the data filtering condition and a preset paging logic, a task record corresponding to the data export task is created in the database, and the data export task is sent to the thread pool, wherein the task record is used to record an execution state, a data filtering condition and a real-time updated data export position corresponding to the data export task; A plurality of pages of data are exported from the database by the thread pool in response to the data export task, and the exported pages of data are stored in a local temporary file in real time; During execution of the data export task by the thread pool, the CPU load of the application service system is monitored by the export component in real time, and when the CPU load is greater than a preset load threshold, a task pause instruction of the data export task is sent to the thread pool, and the execution state is updated to a pause state; When the CPU load is less than or equal to the load threshold, a task continue execution instruction of the data export task is sent to the thread pool by the export component according to the task record, and the execution state is updated from the pause state to an export state; The data export task is allocated a target worker thread by the thread pool; The target worker thread receives the data export task, and determines a page start position of a current export page according to the data export position in the task record; The target worker thread exports, from the database, page data corresponding to the current export page that meets the data filtering condition according to the page start position and a page data amount corresponding to the paging logic, and stores the exported page data of the current export page in the local temporary file in real time; When the current export page is exported, the current export page is updated, and the step of determining the page start position of the current export page according to the data export position in the task record is returned, until all data meeting the data filtering condition in the database are exported. The data export task is allocated a target worker thread by the thread pool, comprising:

2. The method of claim 1, wherein, If there is an idle core thread in the thread pool, the target worker thread is determined from the idle core thread; If there is no idle core thread in the thread pool, and a task queue in the thread pool is not full, the data export task is added to the task queue, and the target worker thread is determined through the task queue; If there is no idle core thread in the thread pool, and the task queue in the thread pool is full, a temporary thread is created as the target worker thread. ​ The data export task is assigned to the target worker thread.

3. The method of claim 1, wherein, After the exported paging data of the current export page is stored into the local temporary file in real time, the method further comprises: determining, by the target worker thread, a total data amount in the database that meets the data filtering condition; determining, based on the target worker thread, a current export progress according to the total data amount and a total export data amount corresponding to all the paging data in the local temporary file; when receiving the progress query request sent by the user terminal, returning the current export progress to the user terminal.

4. The method of claim 1, wherein, After the data export task is sent to the thread pool, the method further comprises: if the application service system is in a restarting state, starting the service node; after the service node is started, querying the task record by the service node to determine whether the execution state of the data export task is an export state; if the execution state of the data export task is the export state, returning to the execution step of sending the data export task to the thread pool.

5. The method of claim 1, wherein, After the data export task is sent to the thread pool, the method further comprises: returning export task start execution information to the user terminal.

6. The method according to any one of claims 1 to 5, characterized in that, After the thread pool responds to the data export task to sequentially export multiple paging data from the database and stores the exported paging data into a local temporary file in real time, the method further comprises: if the data export task is executed completely, determining multiple paging data in the local temporary file as target export data, sending the target export data to a cloud server by the thread pool, and deleting the target export data in the local temporary file; returning a target download link to the user terminal by the thread pool, the target download link being a download link of the target export data in the cloud server.

7. An application service system characterized by comprising: The application service system comprises a service node, an export component, a thread pool, and a database, wherein: the service node is configured to receive a data export request sent by a user terminal, the data export request comprising a data filtering condition; create a data export task according to the data filtering condition and a preset paging logic, create a task record corresponding to the data export task in the database, and send the data export task to the thread pool, the task record being used to record an execution state corresponding to the data export task, a data filtering condition, and a data export position updated in real time; the thread pool is configured to execute the data export task, sequentially export multiple paging data from the database, and store the exported paging data into a local temporary file in real time; and the database is configured to store the multiple paging data exported by the thread pool. The export component is configured to monitor a CPU load of the application service system in real time during execution of the data export task by the thread pool, and send a task suspension instruction of the data export task to the thread pool when the CPU load is greater than a preset load threshold, and update the execution state to a suspension state; when the CPU load is less than or equal to the load threshold, send a task execution instruction of the data export task to the thread pool according to the task record, and update the execution state from the suspension state to an export state. The thread pool specifically executes the data export task by the following steps: allocating a target work thread for the data export task; receiving the data export task through the target work thread, determining a paging start position of a current export page according to the data export position in the task record, exporting paging data corresponding to the current export page that meets the data filtering condition from the database according to the paging start position and a paging data amount corresponding to the paging logic of the target work thread, and storing the exported paging data of the current export page in the local temporary file in real time; when the current export page is exported, updating the current export page, and returning to the step of determining the paging start position of the current export page according to the data export position in the task record until all data meeting the data filtering condition in the database is exported.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer device is deployed with an application service system, and the application service system includes a service node, an export component, a thread pool and a database.

9. A storage medium, characterized by The storage medium stores a computer program, and the computer program includes program instructions, which, when executed by a processor, cause the processor to execute the data export method of any one of claims 1-6.

Images