Data export method, device, electronic device and storage medium

Abstract

The present invention discloses a data export method, device, electronic device and storage medium, which relate to the field of data transmission. The method includes: storing pre-data obtained from a database into a memory; setting the number of pages according to the total number of data to be exported, and sequentially obtaining the paged data from the database through a main thread according to the retrieval of associated data, so as to store the paged data into the memory; obtaining the paged data in the memory and performing data processing through at least one slave thread to obtain a paged data file after the data processing is completed. The technical solution of the embodiment of the present invention makes it possible to completely export the data to be exported by occupying only the memory resources required for a small number of paged data, thereby reducing the occupation of memory resources and avoiding the occurrence of memory overflow. At the same time, the main thread can directly obtain the paged data each time it accesses the database, thereby reducing the number of read and write operations on the database disk interface and improving the efficiency of data export.

Description

Technical Field

[0001] The present invention relates to the field of data transmission, and in particular to a data export method, device, electronic device and storage medium. Background Art

[0002] As business needs continue to expand, a large amount of business data is often stored in the database of the business platform. How to quickly export business data from the business platform has also become an important condition for the development of the business platform.

[0003] Existing online real-time data export technology mainly exports data through concurrent threads to store the exported data completely in memory. However, this data export method occupies a large amount of memory resources and has a serious risk of memory overflow, making the business processing process more cumbersome.

[0004] At the same time, when exporting data from the business platform according to the search conditions, each access to the database requires multiple read and write operations on the database's disk I / O (input / output) interface, which greatly reduces the efficiency of data export. Summary of the Invention

[0005] The present invention provides a data export method, device, electronic device and storage medium to solve the problem of memory overflow risk when exporting business data from a database.

[0006] According to one aspect of the present invention, a data export method is provided, comprising:

[0007] According to the acquired data export task, matching pre-data is obtained from the database, and the pre-data is stored in the memory; wherein the pre-data includes the total number of data to be exported and the search-related data;

[0008] Setting the number of pages according to the total number of data to be exported, and sequentially acquiring page data from the database through the main thread according to the search-related data, so as to store the page data into the memory;

[0009] The paging data in the memory is acquired and processed by at least one slave thread to acquire a paging data file after the data processing is completed, and the paging data file is stored in a shared disk.

[0010] The method of setting the number of pages according to the total number of data to be exported specifically includes: determining a standard single-page threshold according to the business type and data type of the data export task, and setting the number of pages according to the total number of data to be exported and the standard single-page threshold. The standard single-page threshold not only ensures that each page of data occupies the same memory resources and implements a fixed call to memory resources, but also avoids excessive data loading that causes the user to wait too long when generating the corresponding data file for each page of data and presenting it to the user, and avoids too little data loading that causes an excessive number of data files, thereby improving the convenience of data file storage and user browsing.

[0011] Determining the standard single-page threshold based on the business type and data type of the data export task specifically includes: determining the standard single-page threshold and the maximum single-page threshold based on the business type and data type of the data export task; wherein the maximum single-page threshold is greater than the standard single-page threshold; if the total amount of data to be exported is less than or equal to the maximum single-page threshold, setting the number of pages to one; and setting the number of pages based on the total amount of data to be exported and the standard single-page threshold specifically includes: if the total amount of data to be exported is greater than the maximum single-page threshold, setting the number of pages based on the total amount of data to be exported and the standard single-page threshold. Compared to the standard single-page threshold, the maximum single-page threshold only increases the memory resource usage of the paging data and the data file loading wait time to a limited extent, but can avoid data paging. In particular, if the first page of data is generated based on the standard single-page threshold, the second page of data composed of the remaining exported data may be mistaken for an empty file or an invalid file due to its small amount of data. Therefore, if the total amount of data to be exported is less than or equal to the maximum single-page threshold, the exported data can be aggregated into a complete data file to avoid the above phenomenon.

[0012] After setting the number of pages based on the total number of data to be exported and the standard single-page threshold, the method further includes: obtaining an average single-page threshold based on the total number of data to be exported and the number of pages, and using the average single-page threshold as the threshold for the number of data items on each page. Using the average single-page threshold as the threshold for the number of data items on each page improves the processing efficiency of the data export task and ensures that the number of data items in each data file is the same, so that when displaying the paginated data files, each page of data can provide the user with a sufficient amount of related data.

[0013] After storing the pre-data in memory, the method further includes: obtaining a thread response coefficient based on the total amount of data to be exported and the business type of the data export task; wherein the thread response coefficient is the ratio of thread waiting time to thread calculation time; and setting the number of threads in the thread pool based on the number of cores in the central processing unit, the utilization rate of the central processing unit, and the thread response coefficient. This allows for dynamic adjustment of the number of threads without adding new functional components to monitor thread running status or memory occupancy, ensuring that the number of threads in the thread pool meets the data processing requirements of the data export task while avoiding the occurrence of redundant threads.

[0014] The method of sequentially obtaining paging data from the database through the main thread according to the retrieval-related data to store the paging data in the memory specifically includes: obtaining a first quantity threshold according to the number of idle slave threads and / or the time consumed by slave thread data processing; obtaining paging data of the first quantity threshold from the database through the main thread according to the retrieval-related data to store the paging data of the first quantity threshold in the memory. The first quantity threshold is obtained by the number of idle slave threads and / or the time consumed by slave thread data processing, so that the quantity of paging data obtained each time by the main thread can meet the data processing requirements of each slave thread, reduce the existence of idle threads, and avoid excessive occupation of memory resources.

[0015] After the paging data is sequentially obtained from the database through the main thread according to the retrieval associated data to store the paging data into the memory, it also includes: setting a first expiration time for the pre-set data and starting the timing; if the first expiration time has expired, judging whether the main thread has been used to store all the paging data of the data export task into the memory; if the main thread has not been used to store all the paging data of the data export task into the memory, resetting the first expiration time to restart the timing operation of the first expiration time; if the main thread has been used to store all the paging data of the data export task into the memory, deleting the pre-set data. The setting and resetting of the first expiration time ensures that the cached data in the memory is cleared in time to avoid memory overflow caused by long-term memory occupation, and also ensures that the export of the data to be exported is complete to avoid the occurrence of data missing.

[0016] According to another aspect of the present invention, there is provided a data exporting device, comprising:

[0017] A pre-data acquisition module is used to acquire matching pre-data from the database according to the acquired data export task, and store the pre-data into the memory; wherein the pre-data includes the total number of data to be exported and the search-related data;

[0018] A paging data storage module is used to set the number of pages according to the total number of data to be exported, and to sequentially obtain paging data from the database through a main thread according to the retrieval associated data, so as to store the paging data in a memory;

[0019] The paging data processing module is used to obtain the paging data in the memory and perform data processing through at least one slave thread to obtain a paging data file after data processing, and store the paging data file to a shared disk.

[0020] According to another aspect of the present invention, an electronic device is provided, comprising:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data export method described in any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data export method described in any embodiment of the present invention when executed.

[0025] According to the technical solution of the embodiment of the present invention, the main thread obtains matching pre-data from the database, sets the number of pages, and then obtains the paged data from the database in sequence according to the retrieval associated data and stores it in the memory. The slave thread extracts the paged data in the memory and processes the data, and then stores the paged data file in the shared disk, so that only the memory resources required for a small number of paged data are occupied, and the data to be exported can be completely exported, which reduces the occupation of memory resources and avoids the occurrence of memory overflow. At the same time, based on the retrieval associated data stored in the memory, the main thread can directly obtain the current paged data each time it accesses the database, which reduces the number of read and write operations on the database disk interface and improves data export efficiency.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 This is a flow chart of a data export method provided according to the first embodiment of the present invention;

[0029] Figure 2 This is a flow chart of a data export method provided according to the second embodiment of the present invention;

[0030] Figure 3A This is a flow chart of a data export method provided in accordance with Embodiment 3 of the present invention;

[0031] Figure 3B This is a flow chart of a data export method provided according to a specific application scenario 1 of the present invention;

[0032] Figure 4 This is a schematic structural diagram of a data export device provided according to a fourth embodiment of the present invention;

[0033] Figure 5 3 is a schematic diagram of the structure of an electronic device for implementing the data export method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] Example 1

[0037] Figure 1 This is a flow chart of a data export method provided in the first embodiment of the present invention. This embodiment is applicable to the case of exporting business data from a business database of a business platform. The method can be executed by a data export device. The data export device can be implemented in the form of hardware and / or software and configured in an electronic device such as a terminal device. Typically, the electronic device can be used to export business data from a business database configured in a server cluster. Figure 1 As shown, the method includes:

[0038] S101. According to the acquired data export task, matching pre-data is obtained from the database, and the pre-data is stored in the memory; wherein the pre-data includes the total number of data to be exported and retrieval-related data.

[0039] A data export task is a process of obtaining required data from a database based on user-configured search conditions. For example, a data export task includes obtaining the transaction flow of a business platform in January, i.e., exporting the transaction flow information of the business platform in January from the database of a business server. The total number of data to be exported is the number of data items to be exported, which also reflects the number of data rows to be exported in a data table. For example, the total number of data items to be exported is 2 million. Retrieving associated data is the intermediate data obtained based on the database's search logic when searching according to the search conditions in a data export task. The database obtains the final search results based on the intermediate data. Obviously, different data export tasks correspond to different retrieved associated data.

[0040] Taking the aforementioned data export task as an example, when searching the database based on the search criteria, the search logic might be to first retrieve account information with transaction flows in January, for example, obtaining 500 account identifiers; then, based on these 500 accounts, retrieve 2 million specific transaction flow information; the retrieval-related data is the aforementioned 500 account identifiers, which are also intermediate data in the process of obtaining the aforementioned 2 million transaction data based on the search criteria. After the data export task retrieves the matching pre-query data from the database, the total number of data to be exported and the retrieval-related data are both stored in memory; the Redis database can be used as a storage device for the pre-query data and paging data in memory.

[0041] In particular, if the search conditions are more complex, multiple intermediate data (i.e., search-related data) may be obtained based on the search logic in the database, and therefore the memory also needs to record the above-mentioned multiple search-related data at the same time; in addition, in the embodiment of the present invention, the statistics and acquisition of user transaction flow information have been pre-authorized by each user; in addition, in the embodiment of the present invention, there is no specific limitation on the database type in the business platform.

[0042] S102: setting the number of pages according to the total number of data to be exported, and sequentially acquiring page data from the database through a main thread according to the search associated data, so as to store the page data into a memory.

[0043] Due to the large amount of data managed in the business platform, each data export task may involve exporting a large amount of data. If all data is exported to the memory at the same time, it will occupy too many memory resources. Therefore, it is often necessary to page the data to be exported, that is, to achieve complete acquisition of the data to be exported through multiple export operations; among them, a fixed data number threshold can be set for each page, that is, the maximum number of data items that can be included in each page; and then the number of pages is obtained based on the ratio of the total number of data to be exported to the single-page number threshold.

[0044] At the same time, after the main thread in the thread pool stores the pre-data obtained from the database into the memory, each time the paging data is obtained, the retrieval-related data in the memory is synchronously sent to the business database as the known information in the paging data acquisition request; in the traditional technical solution, each time the business database is accessed to obtain paging data, it is actually necessary to repeatedly obtain the retrieval-related data through the disk I / O interface of the business database, and then complete the paging data acquisition based on the retrieval-related data.

[0045] Taking the above technical solution as an example, each time the disk I / O interface of the business database is accessed, it is necessary to first obtain the above 500 account identifiers, and then obtain the paging data to be obtained this time based on the 500 account identifiers. One access is actually equivalent to performing two read operations on the disk I / O interface of the database; and for a data export task with more complex retrieval conditions, if N (N is greater than 2) retrieval-related data are obtained, one access is equivalent to performing N+1 read operations on the disk I / O interface of the database; and in the embodiment of the present invention, it is obvious that each access only requires one interface read operation to directly obtain the current paging data.

[0046] Optionally, in an embodiment of the present invention, the paging data is sequentially obtained from the database through the main thread according to the retrieval associated data to store the paging data in the memory, specifically including: obtaining a first quantity threshold according to the number of idle slave threads and / or the time consumed in slave thread data processing; obtaining paging data of the first quantity threshold from the database through the main thread according to the retrieval associated data to store the paging data of the first quantity threshold in the memory.

[0047] Specifically, each time the main thread obtains paging data, it can obtain only one paging data at a time, or it can obtain multiple paging data, as long as the memory resources can meet the storage requirements of the above number of paging data; however, if the main thread puts too much paging data at a time, there are no idle slave threads to retrieve the paging data from the memory in time, which will cause a waste of memory resources; therefore, the number of paging data obtained this time (i.e., the first number threshold) can be estimated based on the number of idle slave threads; obviously, the more idle slave threads there are, the larger the value of the first number threshold; the fewer idle slave threads there are, the smaller the value of the first number threshold.

[0048] At the same time, the slave thread data processing time reflects the time taken by each slave thread to process a page of data for the current data processing task. The average time taken by the slave threads that have completed the page data processing can be used as the slave thread data processing time. Therefore, the shorter the slave thread data processing time, the faster the slave thread processes the page data, and the larger the first quantity threshold. The longer the slave thread data processing time, the slower the slave thread processes the page data, and the smaller the first quantity threshold. The first quantity threshold is obtained by the number of idle slave threads and / or the slave thread data processing time, so that the number of page data obtained by the main thread each time can meet the data processing needs of each slave thread, reduce the existence of idle threads, and avoid excessive occupation of memory resources.

[0049] S103 . Obtain the paging data in the memory and perform data processing through at least one slave thread to obtain a paging data file after data processing, and store the paging data file in a shared disk.

[0050] The master thread stores the pre-placed data and paging data from the business database into memory. The slave threads extract the paging data from memory and perform data processing on it, including data filtering, data format conversion, and data calculations. This generates data files in a specified format, such as "xlsx" and "text," and finally saves the generated data files to a shared disk. Obviously, the slave threads typically spend more time processing paging data than the master thread. Therefore, a single master thread writing paging data to memory can satisfy the data processing needs of multiple slave threads. Furthermore, multiple master threads can be configured to further improve the efficiency of exporting paging data from the database.

[0051] Optionally, in an embodiment of the present invention, after obtaining the paging data in sequence from the database through the main thread according to the retrieval associated data to store the paging data in the memory, it also includes: setting a first expiration time for the preceding data and starting the timing; if the first expiration time has expired, determining whether the main thread has been used to store all the paging data of the data export task in the memory; if the main thread has not been used to store all the paging data of the data export task in the memory, resetting the first expiration time to restart the timing operation of the first expiration time; if the main thread has been used to store all the paging data of the data export task in the memory, deleting the preceding data.

[0052] Specifically, after the main thread writes the last page data of the data export task into the memory, the export of all data of the data export task has been completed. At this time, after the first expiration time ends, the previous data can be directly deleted without waiting for the data processing of all slave threads to be completed; if the first expiration time has ended, but the main thread has not written all the page data into the memory, the first expiration time will be reset to restart the timing of the first expiration time to ensure that the complete data of the data export task is written into the memory; the setting and resetting of the first expiration time ensures that the cached data in the memory is cleared in time to avoid memory overflow caused by long-term memory occupation, and ensures the complete export of the data to be exported to avoid the occurrence of data missing.

[0053] According to the technical solution of the embodiment of the present invention, the main thread obtains matching pre-data from the database, sets the number of pages, and then obtains the paged data from the database in sequence according to the retrieval associated data and stores it in the memory. The slave thread extracts the paged data in the memory and processes the data, and then stores the paged data file in the shared disk, so that only the memory resources required for a small number of paged data are occupied, and the data to be exported can be completely exported, which reduces the occupation of memory resources and avoids the occurrence of memory overflow. At the same time, based on the retrieval associated data stored in the memory, the main thread can directly obtain the current paged data each time it accesses the database, which reduces the number of read and write operations on the database disk interface and improves data export efficiency.

[0054] Example 2

[0055] Figure 2 This is a flowchart of a data export method provided by the second embodiment of the present invention. The relationship between this embodiment and the above embodiment is that different standard single-page thresholds are set for different data export tasks according to different business types and data types. Figure 2 As shown, the method includes:

[0056] S201. According to the acquired data export task, matching pre-data is obtained from the database, and the pre-data is stored in the memory; wherein the pre-data includes the total number of data to be exported and the search-related data.

[0057] S202: Determine a standard single-page threshold according to the business type and data type of the data export task, and set the number of pages according to the total amount of data to be exported and the standard single-page threshold.

[0058] Different business types require different exported parameter items, and the number of exported parameter items may also vary. Therefore, the business type determines the parameter item composition of the exported data; the data type determines the number of bytes occupied by the parameter information in each parameter item; among them, the data type can include image data, text data, list data, and hyperlink data, etc., and can also include multiple data types such as integer data and floating-point data; based on the business type and data type of the current data export task, the standard single-page threshold can be determined, that is, the number of data items that can be carried in each page.

[0059] Therefore, after setting the number of pages, the standard single-page threshold can not only ensure that each page data occupies the same memory resources and realize fixed call to memory resources, but also when each page data generates the corresponding data file and displays it to the user, it can avoid excessive data loading causing long user waiting time, and avoid insufficient data loading resulting in an excessive number of data files, thereby improving the convenience of data file storage and user browsing.

[0060] Optionally, in an embodiment of the present invention, determining the standard single-page threshold based on the business type and data type of the data export task specifically includes: determining the standard single-page threshold and the maximum single-page threshold based on the business type and data type of the data export task; wherein the maximum single-page threshold is greater than the standard single-page threshold; if the total amount of data to be exported is less than or equal to the maximum single-page threshold, setting the number of pages to one; setting the number of pages based on the total amount of data to be exported and the standard single-page threshold specifically includes: if the total amount of data to be exported is greater than the maximum single-page threshold, setting the number of pages based on the total amount of data to be exported and the standard single-page threshold.

[0061] Specifically, the maximum single-page threshold is numerically slightly larger than the standard single-page threshold, but much smaller than twice the standard single-page threshold; for example, the maximum single-page threshold can be 1.1 times the standard single-page threshold; taking the above technical solution as an example, the standard single-page threshold is set to 200,000 data items, and the maximum single-page threshold is set to 220,000 data items; compared with the standard single-page threshold, the maximum single-page threshold only increases the memory resource usage of the paging data and the loading waiting time of the data file to a limited extent, but can avoid data paging, especially if the first paging data is generated according to the standard single-page threshold, then the second paging data composed of the remaining exported data may be mistaken for an empty file or an invalid file due to its small amount of data. Therefore, if the total amount of data to be exported is less than or equal to the maximum single-page threshold, the exported data can be summarized into a complete data file to avoid the above phenomenon.

[0062] S203: Obtain an average single-page threshold according to the total number of data to be exported and the number of pages, and use the average single-page threshold as a data item number threshold for each page.

[0063] The number of data items in each page of data is related to the data processing time of the slave thread. Taking the above technical solution as an example, in the traditional technical solution, if the total number of data to be exported is 450,000 data items, and the standard single-page threshold obtained is 200,000 data items, 200,000 data items need to be processed by slave thread A, 200,000 data items need to be processed by slave thread B, and 30,000 data items need to be processed by thread C. Since the main thread does not need to perform any data processing, it imports the paged data into the memory at a faster speed. Therefore, the time when slave threads A, B and C obtain the paged data is relatively close. At this time, the completion time of the data export task is mainly determined by the processing completion time of the slave thread with a larger number of data items. That is, after slave thread A or slave thread B completes processing 200,000 data items, the data export task is considered to be completed.

[0064] In the embodiment of the present invention, since the average single-page threshold is obtained, that is, the paging data corresponding to slave thread A, slave thread B and slave thread C are all 150,000 data items, as mentioned above, since the time when the paging data are obtained from slave thread A, slave thread B and slave thread C is relatively close, at this time, the completion time of the data export task is mainly determined by the processing completion time of the slave thread with a larger number of data items; that is, after slave thread A, slave thread B or slave thread C completes processing of 150,000 data items, the data export task is deemed to be completed; obviously, using the average single-page threshold as the data item number threshold for each paging improves the processing efficiency of the data export task, and at the same time, ensures that the number of data items in each data file is the same, so that when the paging data files are displayed, each paging data can provide the user with a sufficient amount of previous and subsequent related data.

[0065] S204 , sequentially acquiring paging data from the database through the main thread according to the retrieved associated data, and storing the paging data in a memory.

[0066] S205 . Obtain the paging data in the memory and perform data processing through at least one slave thread to obtain a paging data file after data processing, and store the paging data file in a shared disk.

[0067] The technical solution of the embodiment of the present invention determines the standard single-page threshold according to the business type and data type of the data export task, and then sets the number of pages according to the total number of data to be exported and the standard single-page threshold. This not only realizes the fixed call to memory resources, but also avoids excessive data loading resulting in a long waiting time for users and insufficient data loading resulting in an excessive number of data files when each page data generates a corresponding data file and displays it to the user. In addition, based on the total number of data to be exported and the number of pages, the obtained average single-page threshold is used as the data item number threshold for each page, which improves the processing efficiency of the data export task and, at the same time, ensures that the number of data items in each data file is the same, so that when the paged data files are displayed, each page data can provide the user with a sufficient amount of previous and subsequent related data.

[0068] Example 3

[0069] Figure 3A This is a flow chart of a data export method provided by embodiment 3 of the present invention. In this embodiment of the present invention, the number of threads in the thread pool is related to the total amount of data to be exported and the business type of the data export task. Figure 3A As shown, the method includes:

[0070] S301. According to the acquired data export task, matching pre-data is obtained from the database, and the pre-data is stored in the memory; wherein the pre-data includes the total number of data to be exported and the search-related data.

[0071] S302. Obtain a thread response coefficient according to the total amount of data to be exported and the business type of the data export task; wherein the thread response coefficient is a ratio of thread waiting time to thread calculation time.

[0072] If the task operation executed by the current thread needs to wait for the task processing results in other threads before continuing to execute its own task, then the time the current thread waits for other threads to finish processing tasks is the thread waiting time; and the time required for the current thread to read the data to be processed, perform data processing operations, and perform other processing tasks is the thread calculation time.

[0073] Different business types require different data processing operations. For example, taking the transaction flow of the business platform in January as an example, after obtaining the paging data, the slave thread needs to mark the amount range to which the transaction amount of each transaction belongs, the time range to which the transaction time of each transaction belongs, and other operations for the current paging data to provide preprocessing operations for subsequent data statistics. For other types of business data, the slave thread will also perform other processing operations or no operations on the current paging data. Therefore, different business types require the slave threads to perform different data processing operations on the paging data, and thus the data processing time is also different. Therefore, if the data processing operations that each slave thread needs to perform under this business type are more complicated, then the calculation time of each thread is longer, that is, a thread response coefficient with a smaller value is obtained. If the data processing operations that each slave thread needs to perform under this business type are relatively simple, then the calculation time of each thread is shorter, that is, a thread response coefficient with a larger value is obtained.

[0074] At the same time, if the total amount of data to be exported is larger, the number of pages will be larger, and the slave thread will execute data processing, which needs to rely on the main thread to complete the memory writing operation of the paged data, that is, the slave thread will have a longer waiting time, waiting to obtain the required paged data through the memory; therefore, if the total amount of data to be exported is larger, the thread waiting time of each thread will be longer, that is, a larger thread response coefficient will be obtained; if the total amount of data to be exported is smaller, the thread waiting time of each thread will be shorter, that is, a smaller thread response coefficient will be obtained.

[0075] S303: Setting the number of threads in the thread pool according to the number of cores of the central processing unit, the utilization rate of the central processing unit, and the thread response coefficient.

[0076] The number of threads in the thread pool can be obtained by the following formula:

[0077] N=N 内核数 +N 利用率 +(1+W / C)

[0078] Where N is the number of threads in the thread pool; N 内核数 N is the number of cores of the central processing unit of the current electronic device; 利用率 is the expected utilization of the CPU, and its value is in the interval (0, 1); W / C is the thread response coefficient; W is the thread waiting time; C is the thread calculation time.

[0079] In traditional technical solutions, the thread response coefficient is usually estimated by installing a plug-in to monitor the running status of existing threads; or a fixed W / C value is set based on empirical values; in the embodiment of the present invention, the matching thread waiting time is determined by the total number of data to be exported, and the matching thread calculation time is determined by the business type of the data export task. Therefore, without adding new functional components to monitor the thread running status or memory occupancy, the number of threads can be dynamically adjusted, which ensures that the number of threads in the thread pool meets the data processing requirements of the data export task and avoids the emergence of redundant threads.

[0080] S304: setting the number of pages according to the total number of data to be exported, and sequentially acquiring page data from the database through the main thread according to the search-related data, so as to store the page data into the memory.

[0081] S305 . Obtain the paging data in the memory and perform data processing through at least one slave thread to obtain a paging data file after data processing, and store the paging data file in a shared disk.

[0082] The technical solution of the embodiment of the present invention, after obtaining the thread response coefficient based on the total number of data to be exported and the business type of the data export task, sets the number of threads in the thread pool according to the number of cores of the central processing unit, the utilization rate of the central processing unit and the thread response coefficient. Without adding new functional components to monitor the thread running status or memory occupancy rate, dynamic adjustment of the number of threads can be achieved, which ensures that the number of threads in the thread pool meets the data processing requirements of the data export task and avoids the emergence of redundant threads.

[0083] Specific application scenario 1

[0084] Figure 3B This is a flowchart of a data export method provided for a specific application scenario of the present invention. In this embodiment of the present invention, the user exports the required data from the business database through a client that matches the business platform. Figure 3B As shown, the method includes:

[0085] S401 . In response to obtaining a data export task, sending a search condition to a database, so that the database performs a search operation based on the search condition.

[0086] S402: Acquire matching pre-data from a database and store the pre-data in a memory; wherein the pre-data includes the total number of data to be exported and search-related data.

[0087] S403: Set the number of pages according to the total amount of data to be exported.

[0088] S404: Initialize the door bolt.

[0089] The bolt is the completion indicator of the data export task.

[0090] S405 , sequentially acquiring paging data from the database through the main thread according to the retrieved associated data, storing the paging data in a memory, and setting a second expiration time for the paging data.

[0091] S406 , obtaining the paging data in the memory and performing data processing through at least one slave thread to obtain a paging data file after data processing, and storing the paging data file in a shared disk.

[0092] S407: Release the door bolt.

[0093] S408: compress the multiple paging files and output them in binary stream format.

[0094] S409: Delete temporary files.

[0095] The technical solution of the embodiment of the present invention determines the standard single-page threshold according to the business type and data type of the data export task, and then sets the number of pages according to the total number of data to be exported and the standard single-page threshold. This not only realizes the fixed call to memory resources, but also avoids excessive data loading resulting in a long waiting time for users and insufficient data loading resulting in an excessive number of data files when each page data generates a corresponding data file and displays it to the user. In addition, based on the total number of data to be exported and the number of pages, the obtained average single-page threshold is used as the data item number threshold for each page, which improves the processing efficiency of the data export task and, at the same time, ensures that the number of data items in each data file is the same, so that when the paged data files are displayed, each page data can provide the user with a sufficient amount of previous and subsequent related data.

[0096] Example 4

[0097] Figure 4 : is a structural block diagram of a data export device provided by a fourth embodiment of the present invention, the device specifically includes:

[0098] The pre-data acquisition module 401 is used to acquire matching pre-data from the database according to the acquired data export task, and store the pre-data into the memory; wherein the pre-data includes the total number of data to be exported and the search-related data;

[0099] The paging data storage module 402 is configured to set the number of pages according to the total number of data to be exported, and sequentially obtain the paging data from the database through the main thread according to the retrieval associated data, so as to store the paging data in the memory;

[0100] The paging data processing module 403 is configured to obtain the paging data in the memory and perform data processing through at least one slave thread to obtain a paging data file after data processing, and store the paging data file in a shared disk.

[0101] According to the technical solution of the embodiment of the present invention, the main thread obtains matching pre-data from the database, sets the number of pages, and then obtains the paged data from the database in sequence according to the retrieval associated data and stores it in the memory. The slave thread extracts the paged data in the memory and processes the data, and then stores the paged data file in the shared disk, so that only the memory resources required for a small number of paged data are occupied, and the data to be exported can be completely exported, which reduces the occupation of memory resources and avoids the occurrence of memory overflow. At the same time, based on the retrieval associated data stored in the memory, the main thread can directly obtain the current paged data each time it accesses the database, which reduces the number of read and write operations on the database disk interface and improves data export efficiency.

[0102] Optionally, the paging data storage module 402 is specifically configured to determine a standard single-page threshold according to the business type and data type of the data export task, and set the number of pages according to the total amount of data to be exported and the standard single-page threshold.

[0103] Optionally, the paging data storage module 402 is further used to determine the standard single-page threshold and the maximum single-page threshold based on the business type and data type of the data export task; wherein the maximum single-page threshold is greater than the standard single-page threshold; if the total amount of data to be exported is less than or equal to the maximum single-page threshold, the number of pages is set to one; if the total amount of data to be exported is greater than the maximum single-page threshold, the number of pages is set according to the total amount of data to be exported and the standard single-page threshold.

[0104] Optionally, the paging data storage module 402 is further configured to obtain an average single-page threshold according to the total amount of data to be exported and the number of pages, and use the average single-page threshold as a data item number threshold for each page.

[0105] Optionally, the data export device further includes:

[0106] A thread configuration module is used to obtain a thread response coefficient based on the total amount of data to be exported and the business type of the data export task; wherein the thread response coefficient is the ratio of thread waiting time to thread calculation time; and set the number of threads in the thread pool based on the number of cores of the central processing unit, the utilization rate of the central processing unit and the thread response coefficient.

[0107] Optionally, the paging data storage module 402 is further specifically used to obtain a first quantity threshold based on the number of idle slave threads and / or the time consumed in slave thread data processing; obtain paging data of the first quantity threshold from the database through the main thread according to the retrieval associated data, so as to store the paging data of the first quantity threshold to the memory.

[0108] Optionally, the paging data storage module 402 is further used to set a first expiration time for the preceding data and start timing; if the first expiration time has expired, determine whether all the paging data of the data export task has been stored in the memory through the main thread; if all the paging data of the data export task has not been stored in the memory through the main thread, reset the first expiration time to restart the timing operation of the first expiration time; if all the paging data of the data export task has been stored in the memory through the main thread, delete the preceding data.

[0109] The above device can execute the data export method provided by any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the data export method provided by any embodiment of the present invention.

[0110] Example 5

[0111] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0112] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0113] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0114] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data derivation method.

[0115] In some embodiments, the data export method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on a heterogeneous hardware accelerator via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the data export method described above can be performed. Alternatively, in other embodiments, the processor can be configured to execute the data export method by any other appropriate means (e.g., by means of firmware).

[0116] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0121] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0122] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0123] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data export method, characterized in that: include: According to the acquired data export task, matching pre-data is obtained from the database, and the pre-data is stored in the memory; wherein the pre-data includes the total number of data to be exported and search-related data; the search-related data is the intermediate data obtained according to the search logic of the database when searching according to the search conditions in the data export task, and the database obtains the search results based on the intermediate data; Setting the number of pages according to the total number of data to be exported, and sequentially acquiring page data from the database through the main thread according to the search-related data, so as to store the page data into the memory; The step of sequentially acquiring paging data from the database through the main thread according to the retrieval associated data, and storing the paging data in the memory, specifically includes: acquiring a first quantity threshold according to the number of idle slave threads and / or the slave thread data processing time; acquiring paging data of the first quantity threshold from the database through the main thread according to the retrieval associated data, and storing the paging data of the first quantity threshold in the memory; The paging data in the memory is acquired and processed by at least one slave thread to acquire a paging data file after the data processing is completed, and the paging data file is stored in a shared disk.

2. The method according to claim 1, characterized in that The step of setting the number of pages according to the total number of data to be exported specifically includes: A standard single-page threshold is determined according to the business type and data type of the data export task, and the number of pages is set according to the total amount of data to be exported and the standard single-page threshold.

3. The method according to claim 2, characterized in that Determining the standard single-page threshold according to the business type and data type of the data export task specifically includes: Determining a standard single-page threshold and a maximum single-page threshold according to the business type and data type of the data export task; wherein the maximum single-page threshold is greater than the standard single-page threshold; If the total amount of data to be exported is less than or equal to the maximum single-page threshold, the number of pages is set to one; The step of setting the number of pages according to the total amount of data to be exported and the standard single-page threshold specifically includes: If the total amount of data to be exported is greater than the maximum single-page threshold, the number of pages is set according to the total amount of data to be exported and the standard single-page threshold.

4. The method according to claim 2 or 3, characterized in that After setting the number of pages according to the total amount of data to be exported and the standard single-page threshold, the method further includes: According to the total number of data to be exported and the number of pages, an average single-page threshold is obtained, and the average single-page threshold is used as the data item number threshold for each page.

5. The method according to claim 1, wherein After storing the pre-data into the memory, the method further includes: Obtaining a thread response coefficient according to the total amount of data to be exported and the business type of the data export task; wherein the thread response coefficient is a ratio of thread waiting time to thread calculation time; The number of threads in the thread pool is set according to the number of cores of the central processing unit, the utilization rate of the central processing unit and the thread response coefficient.

6. The method according to claim 1, wherein After sequentially acquiring paging data from the database through the main thread according to the retrieved associated data and storing the paging data in the memory, the method further includes: Setting a first expiration time for the pre-data and starting a timer; If the first expiration time has expired, determining whether the main thread has passed, and storing all the paging data of the data export task into the memory; If all the paged data of the data export task are not stored in the memory through the main thread, resetting the first expiration time to restart the timing operation of the first expiration time; If all the paging data of the data export task have been stored in the memory through the main thread, the preceding data is deleted.

7. A data export device, characterized in that: include: A pre-data acquisition module is used to acquire matching pre-data from the database according to the acquired data export task, and store the pre-data in the memory; wherein the pre-data includes the total number of data to be exported and search-related data; the search-related data is the intermediate data acquired according to the database's search logic when searching according to the search conditions in the data export task, and the database acquires the search results based on the intermediate data; A paging data storage module is used to set the number of pages according to the total number of data to be exported, and to sequentially obtain paging data from the database through a main thread according to the retrieval associated data, so as to store the paging data in a memory; The paging data storage module is specifically configured to obtain a first quantity threshold according to the number of idle slave threads and / or the slave thread data processing time; obtain paging data of the first quantity threshold from the database through the main thread according to the retrieval associated data, and store the paging data of the first quantity threshold in the memory; The paging data processing module is used to obtain the paging data in the memory and perform data processing through at least one slave thread to obtain a paging data file after data processing, and store the paging data file in a shared disk.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data export method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data export method according to any one of claims 1 to 6 when executed.

Images