Data downloading method and system

Abstract

The invention provides a data downloading method and system, and the method comprises the steps: writing a downloading task request initiated by a user into a virtual task pool corresponding to the user, mapping the downloading task requests in the virtual task pools of different users to a public task pool, and adding the public task pool to a message queue, and identifying the downloading task requests with the same downloading condition parameters in the public task pool as the same downloading task request, and adopting a user no-reference automatic skipping strategy until the current downloading task request is acquired from the message queue, according to the method, the data needing to be downloaded in the current downloading task request is stored in the temporary table in a streaming query mode, and the data in the temporary table is converted to obtain the downloading result of the current downloading task request, so that the consumption of database computing resources and the load of an application server can be greatly reduced, and the controllability of the resources is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a data downloading method and system. Background Art

[0002] In big data scenarios, when faced with the export or download of massive amounts of data (millions to tens of millions), it often causes huge consumption of computing resources, uncontrollable resource services, and even resource crashes and data leaks.

[0003] How to provide a low-load and highly resource-controllable method for downloading massive data will play a huge role in reducing the load on application servers and reducing database resource consumption. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data download method and system, which can significantly reduce the application server load and database computing resource consumption, and support full-link support and resource controllability.

[0005] In order to solve the above problem, the present invention provides a data downloading method, comprising:

[0006] A virtual task pool is created for a user; a download task request initiated by a user is written into the virtual task pool corresponding to the user, each download task request includes a download condition parameter and a user reference parameter, the download condition parameter is used to limit the data range downloaded by the download task request, and the user reference parameter is used to indicate the association between the download task request and the user; download task requests in virtual task pools of different users are mapped to a public task pool, and download task requests with the same download condition parameter in the public task pool are identified as the same download task request; the download task request in the public task pool is added to a message queue and the download task request is persisted; a user-free reference automatic skip strategy is adopted until the current download task request is obtained from the message queue; a temporary table is created, and the temporary table is used to store the data that needs to be downloaded by the current download task request; a streaming query method is used to perform a database query operation, and the data that needs to be downloaded by the current download task request is stored in the temporary table through a single database access and batch reading of the query results; the data in the temporary table is converted to obtain the download result of the current download task request.

[0007] In some embodiments, the step of adopting an automatic skipping strategy when there is no user reference until the current download task request is obtained from the message queue further includes: taking out a download task request from the message queue as a pre-download task request; judging whether the pre-download task request has a user association through a user reference parameter of the pre-download task request, and if the pre-download task request has a user association, taking the pre-download task request as the current download task request; if the pre-download task request has no user association, skipping the pre-download task request, taking out the next download task request from the message queue as a new pre-download task request, and judging whether the new pre-download task request has a user association.

[0008] In some embodiments, in the step of creating a temporary table, the data is stored in MaxCompute, and the temporary table is formed using the Tunnel function of MaxCompute.

[0009] In some embodiments, the step of adopting a user-free automatic skipping strategy until obtaining the current download task request from the message queue further includes: regulating the consumption speed of the message queue.

[0010] In some embodiments, in the step of performing database query operations in a streaming query manner, the batch size of each batch is adjustable, and the resource consumption of the database is controlled by adjusting the batch size.

[0011] In some embodiments, the method further includes: monitoring each download task request in the message queue.

[0012] In some embodiments, the method further includes: after obtaining the download result, deleting the temporary table.

[0013] In some embodiments, the method further includes: setting an upper limit on the amount of downloaded data for the application.

[0014] In order to solve the above problem, the present invention also provides a data download system, comprising:

[0015] A user request processing module, the user request processing module is used to process download requests issued by different users. When a user initiates a download task request, the user request processing module creates a virtual task pool for the user, and writes the download task request initiated by the user into the virtual task pool corresponding to the user. Each download task request includes a download condition parameter and a user reference parameter. The download condition parameter is used to limit the data range downloaded by the download task request, and the user reference parameter is used to indicate the association between the download task request and the user;

[0016] A public task processing module, the public task processing module includes a public task pool and a message queue, the public task processing module maps download task requests in virtual task pools of different users to the public task pool, and the public task processing module identifies download task requests with the same download condition parameters in the public task pool as the same download task request, and the public task processing module also adds the download task requests in the public task pool to the message queue and persists the download task requests;

[0017] A task execution module, wherein the task execution module adopts an automatic skipping strategy without user reference until the current download task request is obtained from the message queue, the task execution module creates a temporary table for storing the data that needs to be downloaded by the current download task request, the task execution module adopts a streaming query method to perform database query operations, accesses the database once and reads the query results in batches, the task execution module stores the data that needs to be downloaded in the temporary table, and the task execution module transforms the data in the temporary table to obtain the download result of the current download task request.

[0018] In some embodiments, the data download system further includes a task monitoring module, and the task monitoring module is used to monitor each download task request in the message queue.

[0019] The above technical scheme, by writing the download task request initiated by the user into the virtual task pool corresponding to the user, maps the download task requests in the virtual task pools of different users to the public task pool and adds them to the message queue, identifies the download task requests with the same download condition parameters in the public task pool as the same download task request, adopts the user's non-reference automatic skip strategy until the current download task request is obtained from the message queue, adopts the streaming query method to store the data that needs to be downloaded by the current download task request into a temporary table, transforms the data in the temporary table to obtain the download result of the current download task request, which can greatly reduce the consumption of database computing resources and the load of the application server and improve the controllability of resources.

[0020] It should be understood that the above general description and the detailed description below are exemplary and explanatory only and cannot limit the present invention. The techniques, methods and devices known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered as part of the authorization specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a data downloading method provided by an embodiment of the present invention;

[0023] Figure 2 It is a schematic diagram of the composition of a data download system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Figure 1 FIG. 1 is a flow chart of a data downloading method provided by an embodiment of the present invention. Figure 1 As shown, the data download method includes: step S11, creating a virtual task pool for a user; step S12, writing a download task request initiated by a user into the virtual task pool corresponding to the user, each of the download task requests including a download condition parameter and a user reference parameter, the download condition parameter being used to limit the data range downloaded by the download task request, and the user reference parameter being used to indicate the association between the download task request and the user; step S13, mapping the download task requests in the virtual task pools of different users to a public task pool, and identifying the download task requests in the public task pool with the same download condition parameter as the same download task request; step S1 4. Add the download task request in the public task pool to a message queue and make the download task request persistent; step S15, adopt the user-free automatic skip strategy until the current download task request is obtained from the message queue; step S16, create a temporary table, the temporary table is used to store the data that needs to be downloaded by the current download task request; step S17, use streaming query method to perform database query operation, through one database access and batch reading of query results, the data that needs to be downloaded by the current download task request is stored in the temporary table; step S18, transform the data in the temporary table to obtain the download result of the current download task request.

[0026] Referring to step S11, a virtual task pool is created for the user. The virtual task pool is used to store download task requests initiated by the user. Specifically, a virtual task pool is created for each user who logs into the application.

[0027] Referring to step S12, the download task request initiated by the user is written into the virtual task pool corresponding to the user, each of the download task requests includes a download condition parameter and a user reference parameter, the download condition parameter is used to limit the data range downloaded by the download task request, and the user reference parameter is used to indicate the association between the download task request and the user.

[0028] In this embodiment, the virtual task pool is a virtual task queue. When a user initiates a download task request, after authority verification, the download task request is written into the virtual task queue corresponding to the user.

[0029] The download condition parameter is used to limit the data range requested to be downloaded by the download task. For example, the download condition parameter limits the generation time of the data to be downloaded, and limits that the data to be downloaded must contain specific keywords.

[0030] The user reference parameter is used to indicate the association between the download task request and the user. When a user creates the download task request, the user reference parameter of the download task request contains the user's information; when the user cancels the download task request, the user reference parameter of the download task request deletes the user's information.

[0031] Referring to step S13, the download task requests in the virtual task pools of different users are mapped to the public task pool, and the download task requests with the same download condition parameters in the public task pool are identified as the same download task request. In a multi-user environment, the data ranges requested to be downloaded by the download task requests initiated by different users may be the same. This step maps all the download task requests initiated by different users to the public task pool by setting the public task pool, and identifies the download task requests with the same download data range as the same download task request according to the download condition parameters of each download task request, thereby avoiding the download task requests with substantially the same query range being executed multiple times in the database, thereby reducing the consumption of database computing resources.

[0032] In this embodiment, an execution flag is set for the download task request to distinguish whether there is an identical task request. When the execution flag is True, it indicates that the download task request is a new download task request and a database query operation needs to be performed. When the execution flag is False, the download task request is the same as the previous download task request and no database query operation needs to be performed. When the download condition parameter of a download task request in the public task pool is the same as the download condition parameter of another download task request in the public task pool, and the execution flag of the other download task request is True, the execution flag of the download task request is False.

[0033] In this embodiment, the download task requests in the virtual task pools of different users are periodically mapped to the public task pool, ensuring that the download task requests with the same download condition parameters in the same period will not be executed repeatedly.

[0034] Each time the download task requests in the virtual task pools of different users are mapped to the public task pool, the public task pool is cleared first, and the time interval of the periodic operation is set according to the number of actual download task requests initiated by the users.

[0035] Referring to step S14, the download task request in the public task pool is added to a message queue and the download task request is persisted. This step executes the download task request asynchronously through a message queue. This embodiment uses a Kafka message queue.

[0036] In this embodiment, the download task requests in the public task pool are stored in a database to achieve persistence of the download task requests, thereby avoiding loss of the download task requests caused by unexpected downtime.

[0037] In this embodiment, each download task request in the message queue is monitored, so as to track the status of each download task request.

[0038] Referring to step S15, an automatic skipping strategy without user reference is adopted until the current download task request is obtained from the message queue.

[0039] The current download task request is a download task request associated with a user. Specifically, this step further includes: (1) taking out a download task request from the message queue as a pre-download task request; (2) judging whether the pre-download task request is associated with a user through a user reference parameter of the pre-download task request; if the pre-download task request is associated with a user, taking the pre-download task request as the current download task request; (3) if the pre-download task request is not associated with a user, skipping the pre-download task request and continuing to execute (1) to (3) until the current download task request is obtained from the message queue.

[0040] In this embodiment, when a user creates a download task request, the user reference parameter of the download task request contains relevant information of the user; when the user cancels the download task request, the user reference parameter of the download task request deletes the relevant information of the user. If the download task request is no longer associated with a user, continuing to execute the download task request will result in a waste of resources. The user reference parameter of the current download task request obtained in this step contains user-related information, and this step is used to avoid the waste of resources caused by the user canceling the download task request.

[0041] In this step, the consumption speed of the message queue is regulated to avoid the crash of program and database resources caused by too many download task requests executed simultaneously.

[0042] Referring to step S16, a temporary table is created, and the temporary table is used to store the data required to be downloaded by the current download task request. When the amount of data is large and more data calculation or filtering is required, setting the temporary table can speed up the download speed.

[0043] In this embodiment, the data is stored in MaxCompute, and the temporary table is formed by using the Tunnel function of MaxCompute.

[0044] Referring to step S17, a streaming query method is used to perform a database query operation, and the query results are read in batches through a single database access, and the data required to be downloaded by the current download task request is stored in the temporary table.

[0045] In conventional large data download solutions, batch (page) download is generally used, and the number of requests is too high. For example, for two million data, each request downloads 200 data items, and a total of 10,000 download requests are required, which puts too much pressure on the application and database due to the high access frequency. In addition, under paging query, as the page number increases, the execution efficiency of the database query SQL gradually decreases.

[0046] The streaming query method is used to perform database query operations. Only one request, one database access, and batch reading of query results are required. This can greatly reduce the frequency of access to the server and database. Streaming query is a method of reading data line by line in batches. It is an iterator that can retrieve data result sets by traversing the iterator instead of loading all data into memory at once.

[0047] When the batch size is set to be relatively large, the memory utilization and compression rate can be improved, but it will consume too many server resources when caching data, and even increase the risk of memory overflow; when the batch size is set to be relatively small, it will lead to an increase in the number of database connections. The batch size of each batch in this step is adjustable, and the resource consumption of the database is controlled by adjusting the batch size, and the batch size is adjusted according to the actual business data volume.

[0048] During the execution of this step, each download task request in the message queue is monitored. In this embodiment, when the current download task request fails to execute this step, a retry is performed to improve the fault tolerance of the application. In this embodiment, the number of retries is 2 times.

[0049] In this embodiment, the data download system also sets an upper limit on the amount of data to be downloaded for the application. When the amount of data required to be downloaded by the current download task request exceeds the maximum amount of data, only the data of the maximum amount of data is downloaded to avoid resource waste, resource crash, and data leakage caused by exporting too much data. In addition, in this embodiment, the maximum amount of data is configurable.

[0050] Referring to step S18, the data in the temporary table is converted to obtain the download result of the current download task request. The data in the temporary table is the data required to be downloaded obtained from the database query. In this step, the data required to be downloaded is converted accordingly to generate the download result of the current download task request. In this embodiment, the data in the temporary table is filtered and packaged in a format to form the download result that meets the requirements of the current download task request.

[0051] In this embodiment, when executing step S13, an execution flag is set for the download task request. When the execution flag is True, it indicates that the download task request is a new download task request and a database query operation needs to be performed. When the execution flag is False, the download task request is the same as the previous download task request and no database query operation needs to be performed. For the download task request whose execution flag value is False contained in the message queue, it is not necessary to perform a data query operation from the database. Data can be directly read from the temporary table and converted into a download result that meets the requirements of the corresponding download task request.

[0052] After obtaining the download result, the temporary table is deleted to release the server resources. In this embodiment, the download task requests in the virtual task pools of different users are periodically mapped to the public task pool, and the public task pool is emptied when each cycle starts to execute, and the temporary table is cleared at the same time.

[0053] During the execution of this step, each download task request in the message queue is monitored. In this embodiment, the download task request that failed to execute is retried, and the number of retries is 2. When the current download task request fails to execute this step, it is also retried twice, further improving the fault tolerance of the application.

[0054] Based on the same inventive concept, an embodiment of the present invention further provides a data downloading system.

[0055] Figure 2 FIG. 1 is a schematic diagram of the composition of a data download system provided by an embodiment of the present invention. Figure 2 As shown, the data download system includes: a user request processing module 11, a public task processing module, and a task execution module 13.

[0056] The user request processing module 11 is used to process download requests issued by different users. When a user initiates a download task request, the user request processing module 11 creates a virtual task pool for the user and writes the download task request initiated by the user into the virtual task pool corresponding to the user. Each of the download task requests includes a download condition parameter and a user reference parameter. The download condition parameter is used to limit the data range downloaded by the download task request, and the user reference parameter is used to indicate the association between the download task request and the user.

[0057] like Figure 2 As shown, user U1, user U2, ... user Un respectively initiate download task requests, and the download task requests of different users all enter the user request processing module 11 for processing.

[0058] The public task processing module 12 includes a public task pool and a message queue. The public task processing module 12 maps the download task requests in the virtual task pools of different users to the public task pool, and the public task processing module 12 identifies the download task requests in the public task pool with the same download condition parameters as the same download task request. The public task processing module 12 also adds the download task requests in the public task pool to the message queue and persists the download task requests.

[0059] The task execution module 13 adopts an automatic skip strategy without user reference until the current download task request is obtained from the message queue. The task execution module 13 creates a temporary table for storing the data that needs to be downloaded by the current download task request. The task execution module 13 uses a streaming query method to perform database query operations, accesses the database once, reads the query results in batches, and stores the data that needs to be downloaded in the temporary table. The task execution module 13 transforms the data in the temporary table to obtain the download result of the current download task request.

[0060] The current download task request is a download task request associated with a user. The step of adopting the user-free reference automatic skipping strategy until the current download task request is obtained from the message queue further includes: (1) taking a download task request from the message queue as a pre-download task request; (2) judging whether the pre-download task request is associated with a user through the user reference parameter of the pre-download task request, and if the pre-download task request is associated with a user, taking the pre-download task request as the current download task request; (3) if the pre-download task request is not associated with a user, skipping the pre-download task request and continuing to execute (1) to (3) until the current download task request is obtained from the message queue.

[0061] like Figure 2 As shown, the common task processing module 12 generates download results corresponding to the download task requests initiated by the user U1, the user U2, ... the user Un.

[0062] In this embodiment, the data download system further includes a task monitoring module 14, which is used to monitor each download task request in the message queue, thereby tracking the status of each download task request, avoiding task loss caused by unexpected downtime, and further improving the controllability of resources. When the task monitoring module 14 monitors that the download task request in the message queue fails to download, a retry operation is performed, and the number of retries is 2.

[0063] The above technical scheme, by writing the download task request initiated by the user into the virtual task pool corresponding to the user, maps the download task requests in the virtual task pools of different users to the public task pool and adds them to the message queue, identifies the download task requests with the same download condition parameters in the public task pool as the same download task request, adopts the user's non-reference automatic skip strategy until the current download task request is obtained from the message queue, adopts the streaming query method to store the data that needs to be downloaded by the current download task request into a temporary table, transforms the data in the temporary table to obtain the download result of the current download task request, which can greatly reduce the consumption of database computing resources and the load of the application server and improve the controllability of resources.

[0064] It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion. The various embodiments in this specification are described in a related manner, and the same and similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0065] The above is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. It should be noted that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A data downloading method, characterized in that: include: Create a virtual task pool for users; Writing the download task request initiated by the user into the virtual task pool corresponding to the user, each download task request includes a download condition parameter and a user reference parameter, the download condition parameter is used to limit the data range downloaded by the download task request, and the user reference parameter is used to indicate the association between the download task request and the user; Mapping download task requests in virtual task pools of different users to a public task pool, and identifying download task requests with the same download condition parameters in the public task pool as the same download task request; Adding the download task request in the public task pool to a message queue and persisting the download task request; Adopting an automatic skipping strategy without user reference until obtaining the current download task request from the message queue; Create a temporary table, the temporary table is used to store the data that needs to be downloaded by the current download task request; use a streaming query method to perform a database query operation, access the database once, read the query results in batches, and store the data that needs to be downloaded by the current download task request in the temporary table; The data in the temporary table is converted to obtain the download result of the current download task request.

2. The method according to claim 1, characterized in that The step of adopting the user-free reference automatic skipping strategy until obtaining the current download task request from the message queue further includes: taking out a download task request from the message queue as a pre-download task request; Determine whether the pre-download task request is associated with a user by using a user reference parameter of the pre-download task request, and if the pre-download task request is associated with a user, use the pre-download task request as the current download task request; If the pre-download task request has no user association, the pre-download task request is skipped, the next download task request is taken out from the message queue as a new pre-download task request, and it is determined whether the new pre-download task request has a user association.

3. The method according to claim 1, characterized in that The step of adopting the automatic skipping strategy without user reference until obtaining the current download task request from the message queue further includes: regulating the consumption speed of the message queue.

4. The method according to claim 1, characterized in that In the step of creating a temporary table, the data is stored in MaxCompute, and the temporary table is formed using the Tunnel function of MaxCompute.

5. The method according to claim 1, characterized in that In the step of performing database query operations in a streaming query manner, the batch size of each batch is adjustable, and the resource consumption of the database is controlled by adjusting the batch size.

6. The method according to claim 1, characterized in that The method further comprises: monitoring each download task request in the message queue.

7. The method according to claim 1, characterized in that The method further comprises: after obtaining the download result, deleting the temporary table.

8. The method according to claim 1, characterized in that The method further includes: setting an upper limit on the amount of downloaded data for the application.

9. A data download system, characterized in that: include: A user request processing module, the user request processing module is used to process download requests issued by different users. When a user initiates a download task request, the user request processing module creates a virtual task pool for the user, and writes the download task request initiated by the user into the virtual task pool corresponding to the user. Each download task request includes a download condition parameter and a user reference parameter. The download condition parameter is used to limit the data range downloaded by the download task request, and the user reference parameter is used to indicate the association between the download task request and the user; A public task processing module, the public task processing module includes a public task pool and a message queue, the public task processing module maps download task requests in virtual task pools of different users to the public task pool, and the public task processing module identifies download task requests with the same download condition parameters in the public task pool as the same download task request, and the public task processing module also adds the download task requests in the public task pool to the message queue and persists the download task requests; A task execution module, wherein the task execution module adopts an automatic skipping strategy without user reference until the current download task request is obtained from the message queue, the task execution module creates a temporary table for storing the data that needs to be downloaded by the current download task request, the task execution module adopts a streaming query method to perform database query operations, accesses the database once and reads the query results in batches, the task execution module stores the data that needs to be downloaded in the temporary table, and the task execution module transforms the data in the temporary table to obtain the download result of the current download task request.

10. The data download system according to claim 9, characterized in that: The data download system further comprises a task monitoring module, and the task monitoring module is used to monitor each download task request in the message queue.

Images