Data downloading method and system

Abstract

The application provides a data downloading method and system, wherein a user-initiated downloading task request is written into a virtual task pool corresponding to the user, downloading task requests in virtual task pools of different users are mapped to a common task pool and added to a message queue, downloading task requests with the same downloading condition parameters in the common task pool are identified as the same downloading task request, a user-unreferenced automatic skipping strategy is adopted until a current downloading task request is obtained from the message queue, data required to be downloaded by the current downloading task request is stored in a temporary table in a streaming query mode, and the data in the temporary table is converted to obtain a downloading result of the current downloading task request, so that the consumption of database computing resources and the load of an application server can be greatly reduced, and the controllability of resources is improved.

Description

Technical Field

[0001] This invention relates to the field of big data, and in particular to a data download method and system. Background Technology

[0002] In big data scenarios, exporting or downloading massive amounts of data (millions to tens of millions) often leads to huge consumption of computing resources, resulting in uncontrollable resource services, and even resource crashes and data leaks.

[0003] Providing a low-load and highly resource-controllable method for downloading massive amounts of data can greatly reduce application server load and database resource consumption. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a data download method and system that can significantly reduce the load on application servers and the consumption of database computing resources, and support resource controllability throughout the entire process.

[0005] To address the above problems, the present invention provides a data download method, comprising:

[0006] A virtual task pool is created for each user; download task requests initiated by the user are written into the virtual task pool corresponding to that user. Each download task request includes download condition parameters and user reference parameters. The download condition parameters are used to limit the range of data to be downloaded by the download task request, and the user reference parameters are used to identify the association between the download task request and the user; download task requests from different users' virtual task pools are mapped to a common task pool, and download task requests with the same download condition parameters in the common task pool are identified as the same download task request; the download task requests in the common task pool are added to a message queue and persisted; an automatic skip strategy for users without references is adopted until the current download task request is retrieved from the message queue; a temporary table is created to store the data to be downloaded by the current download task request; a streaming query method is used to perform database query operations, and the query results are read in batches through a single database access, and the data to be downloaded by the current download task request is stored in the temporary table; the data in the temporary table is transformed to obtain the download result of the current download task request.

[0007] In some embodiments, the step of adopting an automatic skip strategy with no user reference until the current download task request is obtained from the message queue further includes: retrieving a download task request from the message queue as a pre-download task request; determining whether the pre-download task request has a user association based on the user reference parameter of the pre-download task request; if the pre-download task request has a user association, then using the pre-download task request as the current download task request; if the pre-download task request has no user association, then skipping the pre-download task request, retrieving the next download task request from the message queue as a new pre-download task request, and determining whether the new pre-download task request has a user association.

[0008] In some embodiments, during the step of creating a temporary table, the data is stored in MaxCompute, and the temporary table is formed using MaxCompute's Tunnel feature.

[0009] In some embodiments, the step of employing a user-free automatic skip strategy until the current download task request is obtained from the message queue further includes: regulating the consumption speed of the message queue.

[0010] In some embodiments, in the step of performing database query operations using a streaming query method, the batch size of each batch is adjustable, and the resource consumption of the database is controlled by adjusting the batch size.

[0011] In some embodiments, the method further includes: listening to each download task request in the message queue.

[0012] In some embodiments, the method further includes: deleting the temporary table after obtaining the download result.

[0013] In some embodiments, the method further includes setting a maximum download data limit for the application.

[0014] To address the above problems, the present invention also provides a data download system, comprising:

[0015] The user request processing module is used to process download requests issued by different users. When a user initiates a download task request, the user request processing module creates a virtual task pool for the user and writes the download task request initiated by the user into the virtual task pool corresponding to the user. Each download task request includes download condition parameters and user reference parameters. The download condition parameters are used to limit the data range to be downloaded by the download task request, and the user reference parameters are used to indicate the association between the download task request and the user.

[0016] A public task processing module includes a public task pool and a message queue. The public task processing module maps download task requests from virtual task pools of different users to the public task pool. The public task processing module identifies download task requests with the same download condition parameters in the public task pool as the same download task request. The public task processing module also adds the download task request in the public task pool to the message queue and persists the download task request.

[0017] The task execution module employs a user-free, no-reference automatic skip strategy until it retrieves the current download task request from the message queue. The task execution module creates a temporary table to store the data to be downloaded for the current download task request. It performs database queries using a streaming query method, reading query results in batches through a single database access. The task execution module then stores the data to be downloaded into the temporary table and transforms the data in the temporary table to obtain the download result of the current download task request.

[0018] In some embodiments, the data download system further includes a task monitoring module, which is used to monitor each download task request in the message queue.

[0019] The above technical solution, by writing user-initiated download task requests into the user's corresponding virtual task pool, mapping download task requests from different users' virtual task pools to a common task pool and adding them to a message queue, identifying download task requests with the same download condition parameters in the common task pool as the same download task request, adopting a user-free automatic skip strategy until the current download task request is obtained from the message queue, using a streaming query method to store the data to be downloaded by the current download task request into a temporary table, and transforming the data in the temporary table to obtain the download result of the current download task request, can significantly reduce the consumption of database computing resources and the load on the application server, and improve resource controllability.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a data download method provided in an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the composition of a data download system provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Figure 1 This is a flowchart of a data download method provided in an embodiment of the present invention. Figure 1 As shown, the data download method includes: Step S11, creating a virtual task pool for a user; Step S12, writing the download task request initiated by the user into the virtual task pool corresponding to that user, each download task request including download condition parameters and user reference parameters, the download condition parameters being used to limit the data range to be downloaded by the download task request, and the user reference parameters being used to identify the association between the download task request and the user; Step S13, mapping download task requests in the virtual task pools of different users to a common task pool, and identifying download task requests with the same download condition parameters in the common task pool as the same download task request; Step S1 4. Add the download task request from the public task pool to a message queue and persist the download task request; Step S15. Use a user-free reference automatic skip strategy until the current download task request is obtained from the message queue; Step S16. Create a temporary table to store the data that the current download task request needs to download; Step S17. Use a streaming query method to perform a database query operation, and through a single database access and batch reading of query results, store the data that the current download task request needs to download into the temporary table; Step S18. Transform the data in the temporary table to obtain the download result of the current download task request.

[0026] Referring to step S11, a virtual task pool is created for each user. This virtual task pool is used to store download task requests initiated by the user. Specifically, one virtual task pool is created for each user logged into the application.

[0027] Referring to step S12, the download task request initiated by the user is written into the virtual task pool corresponding to the user. Each download task request includes download condition parameters and user reference parameters. The download condition parameters are used to limit the data range to be downloaded by the download task request, and the user reference parameters are used to indicate the association between the download task request and the user.

[0028] In this embodiment, the virtual task pool is a virtual task queue. When a user initiates a download task request, after permission verification, the download task request is written into the virtual task queue corresponding to that user.

[0029] The download condition parameters are used to limit the range of data requested by the download task. For example, the download condition parameters may limit the generation time of the data to be downloaded, and may also limit the data to be downloaded to contain specific keywords.

[0030] The user reference parameter is used to identify the association between the download task request and the user. When a user creates the download task request, the user reference parameter of the download task request contains the user's information; when a user cancels the download task request, the user reference parameter of the download task request deletes the user's information.

[0031] Referring to step S13, download task requests from different users' virtual task pools are mapped to a common task pool, and download task requests with the same download condition parameters in the common task pool are identified as the same download task request. In a multi-user environment, download task requests initiated by different users may request the same range of data. This step, by setting up the common task pool, maps download task requests initiated by different users to the common task pool, and identifies download task requests with the same data range as the same download task request based on the download condition parameters of each download task request. This avoids the multiple executions of download task requests with substantially the same range in the database, thereby reducing the consumption of database computing resources.

[0032] In this embodiment, an execution flag is set for the download task request to distinguish whether there are identical task requests. When the execution flag is True, it indicates that the download task request is a new download task request and a database query operation is required. When the execution flag is False, the download task request is the same as a previous download task request and no database query operation is required. When the download condition parameters of one download task request in the public task pool are the same as the download condition parameters of another download task request in the public task pool, and the execution flag of the other download task request is True, then the execution flag of the first download task request is set to False.

[0033] In this embodiment, download task requests from different users' virtual task pools are periodically mapped to the public task pool to ensure that download task requests with the same download condition parameters are not executed repeatedly within the same period.

[0034] Each time a download task request from a different user's virtual task pool is mapped to the public task pool, the public task pool is first cleared. The time interval for this periodic operation is set according to the actual number of download task requests initiated by the user.

[0035] Referring to step S14, the download task request in the public task pool is added to a message queue and persisted. This step executes the download task request asynchronously through the message queue. This embodiment uses a Kafka message queue.

[0036] In this embodiment, the download task requests in the public task pool are persisted by storing them in the database, thus avoiding the loss of download task requests due to unexpected downtime.

[0037] In this embodiment, each download task request in the message queue is monitored to track the status of each download task request.

[0038] Referring to step S15, a user-free automatic skip strategy is adopted until the current download task request is obtained from the message queue.

[0039] The current download task request is a download task request associated with a user. Specifically, this step further includes: (1) retrieving a download task request from the message queue as a pre-download task request; (2) determining whether the pre-download task request has a user association through the user reference parameter of the pre-download task request. If the pre-download task request has a user association, then the pre-download task request is used as the current download task request; (3) if the pre-download task request has no user association, then the pre-download task request is skipped, and (1) to (3) are continued until the current download task request is obtained from the message queue.

[0040] In this embodiment, when a user creates a download task request, the user reference parameter of the download task request contains the user's relevant information; when the user cancels the download task request, the user reference parameter of the download task request removes the user's relevant information. If the download task request is no longer associated with any user, continuing to execute the download task request would lead to resource waste. The user reference parameter of the current download task request obtained in this step contains user-related information, thus avoiding resource waste caused by the user canceling the download task request.

[0041] In this step, the consumption speed of the message queue is controlled to prevent the program and database resources from crashing due to too many simultaneous download task requests.

[0042] Referring to step S16, a temporary table is created to store the data that the current download task requests to download. When the data volume is large and requires extensive data calculation or filtering, setting up the temporary table can speed up the download process.

[0043] In this embodiment, the data is stored in MaxCompute, and the temporary table is formed using MaxCompute's Tunnel function.

[0044] Referring to step S17, a streaming query method is used to perform database query operations. By accessing the database once and reading the query results in batches, the data that the current download task requests to download is stored in the temporary table.

[0045] In conventional large-scale data download solutions, batch (pagination) downloads are typically used. This results in an excessive number of requests. For example, with two million records, downloading 200 records at a time would require 10,000 requests, placing excessive pressure on the application and database. Furthermore, with paginated queries, the execution efficiency of database SQL queries gradually decreases as the page number increases.

[0046] Streaming queries for database operations require only a single request and database access, retrieving query results in batches. This significantly reduces the frequency of access to the server and database. Streaming queries read data row by row in batches; it uses an iterator to retrieve the result set, rather than loading all data into memory at once.

[0047] Setting a large batch size can improve memory utilization and compression rate, but it consumes too many server resources when caching data and may even increase the risk of memory overflow. Setting a small batch size will lead to an increase in the number of database connections. The batch size of each batch in this step is adjustable. By adjusting the batch size, database resource consumption can be controlled. The batch size is adjusted according to the actual amount of data in the business.

[0048] During this step, each download task request in the message queue is monitored. In this embodiment, if the current download task request fails to execute this step, a retry is performed to improve the application's fault tolerance. In this embodiment, the number of retries is 2.

[0049] In this embodiment, the data download system also sets a maximum download data volume for the application. When the current download task requests more data than the maximum data volume, only the maximum data volume is downloaded, avoiding resource waste, resource crashes, and data leaks caused by exporting too much data. Furthermore, in this embodiment, the maximum data volume is configurable.

[0050] Referring to step S18, the data in the temporary table is transformed to obtain the download result requested by the current download task. The data in the temporary table is the data to be downloaded obtained from the database. This step transforms the data to be downloaded accordingly to generate the download result requested by the current download task. In this embodiment, the data in the temporary table is filtered and packaged into a download result that meets the requirements of the current download task request.

[0051] In this embodiment, during step S13, an execution flag is set for the download task request. When the execution flag is True, it indicates that the download task request is a new download task request and requires a database query operation. When the execution flag is False, the download task request is the same as a previous download task request and does not require a database query operation. For download task requests in the message queue with an execution flag value of False, there is no need to perform a data query operation from the database; data can be directly read from the temporary table and converted into download results that meet the requirements of the corresponding download task request.

[0052] After obtaining the download result, the temporary table is deleted to release server resources. In this embodiment, download task requests from different users' virtual task pools are periodically mapped to the common task pool. At the start of each period, the common task pool is emptied, and the temporary table is also cleared.

[0053] During this step, each download task request in the message queue is monitored. This embodiment retryes failed download task requests up to two times. If the current download task request fails to execute this step, it also retries twice, further improving the application's fault tolerance.

[0054] Based on the same inventive concept, an embodiment of the present invention also provides a data download system.

[0055] Figure 2 This is a schematic diagram of the composition of a data download system provided in an embodiment of the present invention. Figure 2 As shown, the data download system includes: a user request processing module 11, a public task processing module, and a task execution module 13.

[0056] The user request processing module 11 is used to process download requests issued by different users. When a user initiates a download task request, the user request processing module 11 creates a virtual task pool for the user and writes the download task request initiated by the user into the virtual task pool corresponding to that user. Each download task request includes download condition parameters and user reference parameters. The download condition parameters are used to limit the data range to be downloaded by the download task request, and the user reference parameters are used to identify the association between the download task request and the user.

[0057] like Figure 2 As shown, users U1, U2, ..., Un each initiated a download task request, and the download task requests from different users were all processed by the user request processing module 11.

[0058] The public task processing module 12 includes a public task pool and a message queue. The public task processing module 12 maps download task requests from virtual task pools of different users to the public task pool. The public task processing module 12 identifies download task requests with the same download condition parameters in the public task pool as the same download task request. The public task processing module 12 also adds the download task request in the public task pool to the message queue and persists the download task request.

[0059] The task execution module 13 adopts a user-free automatic skip strategy until it obtains the current download task request from the message queue. The task execution module 13 creates a temporary table to store the data to be downloaded by the current download task request. The task execution module 13 performs database query operations using a streaming query method. By accessing the database once and reading the query results in batches, the data to be downloaded is stored in the temporary table. The task execution module 13 transforms the data in the temporary table to obtain the download result of the current download task request.

[0060] The current download task request is a download task request associated with a user. The step of adopting the user-unreferenced automatic skip strategy until the current download task request is obtained from the message queue further includes: (1) taking a download task request from the message queue as a pre-download task request; (2) determining whether the pre-download task request has a user association through the user reference parameter of the pre-download task request. If the pre-download task request has a user association, then the pre-download task request is taken as the current download task request; (3) if the pre-download task request has no user association, then the pre-download task request is skipped, and (1) to (3) are continued until the current download task request is obtained from the message queue.

[0061] like Figure 2 As shown, the public task processing module 12 generates download results corresponding to the download task requests initiated by users U1, U2, ..., Un.

[0062] In this embodiment, the data download system further includes a task monitoring module 14. The task monitoring module 14 monitors each download task request in the message queue to track the status of each download task request, preventing task loss due to unexpected system crashes and further improving resource controllability. When the task monitoring module 14 detects a download task request in the message queue that has failed to download, it performs a retry operation, with a maximum of two retries.

[0063] The above technical solution, by writing user-initiated download task requests into the user's corresponding virtual task pool, mapping download task requests from different users' virtual task pools to a common task pool and adding them to a message queue, identifying download task requests with the same download condition parameters in the common task pool as the same download task request, adopting a user-free automatic skip strategy until the current download task request is obtained from the message queue, using a streaming query method to store the data to be downloaded by the current download task request into a temporary table, and transforming the data in the temporary table to obtain the download result of the current download task request, can significantly reduce the consumption of database computing resources and the load on the application server, and improve resource controllability.

[0064] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion. The various embodiments in this specification are described in a related manner, and similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data download method, characterized in that, include: Create a virtual task pool for users; The download task request initiated by the user is written into the virtual task pool corresponding to the user. Each download task request includes download condition parameters and user reference parameters. The download condition parameters are used to limit the data range to be downloaded by the download task request, and the user reference parameters are used to indicate the association between the download task request and the user. Download task requests from different users' virtual task pools are mapped to a common task pool, and download task requests with the same download condition parameters in the common task pool are identified as the same download task request. Add the download task request from the public task pool to a message queue and persist the download task request; A user-free automatic skip strategy is adopted until the current download task request is obtained from the message queue; A temporary table is created to store the data that the current download task needs to download. A streaming query is used to perform database query operations, and the query results are read in batches through a single database access, and the data that the current download task needs to download is stored in the temporary table. The data in the temporary table is transformed to obtain the download result requested by the current download task.

2. The method according to claim 1, characterized in that, The step of adopting the user-free automatic skip strategy until the current download task request is obtained from the message queue further includes: retrieving a download task request from the message queue as a pre-download task request; The user reference parameter of the pre-download task request is used to determine whether the pre-download task request has a user association. If the pre-download task request has a user association, then the pre-download task request is used as the current download task request. If the pre-download task request has no user association, then the pre-download task request is skipped, and the next download task request is taken from the message queue as a new pre-download task request. It is then determined whether the new pre-download task request has a user association.

3. The method according to claim 1, characterized in that, The step of adopting the user-free automatic skip strategy until the current download task request is obtained from the message queue further includes: regulating the consumption speed of the message queue.

4. The method according to claim 1, characterized in that, In the step of creating a temporary table, the data is stored in MaxCompute, and the temporary table is formed using MaxCompute's Tunnel function.

5. The method according to claim 1, characterized in that, In the steps of performing database query operations using the streaming query method, the batch size of each batch is adjustable, and the resource consumption of the database is controlled by adjusting the batch size.

6. The method according to claim 1, characterized in that, The method further includes: listening to each download task request in the message queue.

7. The method according to claim 1, characterized in that, The method further includes: after obtaining the download result, deleting the temporary table.

8. The method according to claim 1, characterized in that, The method further includes setting a maximum download data limit for the application.

9. A data download system, characterized in that, include: The user request processing module is used to process download requests issued by different users. When a user initiates a download task request, the user request processing module creates a virtual task pool for the user and writes the download task request initiated by the user into the virtual task pool corresponding to the user. Each download task request includes download condition parameters and user reference parameters. The download condition parameters are used to limit the data range to be downloaded by the download task request, and the user reference parameters are used to indicate the association between the download task request and the user. A public task processing module includes a public task pool and a message queue. The public task processing module maps download task requests from virtual task pools of different users to the public task pool. The public task processing module identifies download task requests with the same download condition parameters in the public task pool as the same download task request. The public task processing module also adds the download task request in the public task pool to the message queue and persists the download task request. The task execution module employs a user-free, no-reference automatic skip strategy until it retrieves the current download task request from the message queue. The task execution module creates a temporary table to store the data to be downloaded for the current download task request. It performs database queries using a streaming query method, reading query results in batches through a single database access. The task execution module then stores the data to be downloaded into the temporary table and transforms the data in the temporary table to obtain the download result of the current download task request.

10. The data download system according to claim 9, characterized in that, The data download system also includes a task monitoring module, which is used to monitor each download task request in the message queue.

Images