A data batch export method and system based on data packet sorting

Abstract

The application discloses a data batch export method and system based on data packet sorting, and belongs to the technical field of data export. The method comprises the following steps: traversing a table file to be exported, and constructing a corresponding key-value pair by acquiring attribute information of each data table; wherein the key-value pair comprises a table ID and a table occupied space; based on the table occupied space, all data tables are sorted in a preset order, the sorted data tables are grouped according to a current available thread number, and a plurality of table groups are obtained, each table group corresponding to an available thread; and the data in the table groups is simultaneously exported by each available thread. Therefore, by implementing the application, the problem of unbalanced thread load caused by unreasonable data allocation and low data export efficiency in the prior art can be solved.

Description

Technical Field

[0001] This application belongs to the field of data export technology, specifically relating to a method and system for batch data export based on data grouping and sorting. Background Technology

[0002] In our work, we often need to process and export large amounts of data tables. Existing multi-threaded export technology is widely used because it can effectively utilize system resources and shorten the overall export time. However, most existing multi-threaded export methods adopt a random task allocation approach, that is, they do not preprocess the data to be exported, but directly allocate the data randomly to each thread.

[0003] Because different data tables vary significantly in size, random allocation may result in one thread being assigned multiple large data tables, while other threads primarily process small data tables. This leads to a severe imbalance in thread load, with the thread processing large data tables consuming a significant amount of time, thus prolonging the overall export time and failing to fully leverage the advantages of multithreading. Consequently, during multithreaded export processes, situations may arise where some threads work for extended periods while others complete their export tasks early and remain idle, resulting in a severe waste of system resources and low export efficiency. Summary of the Invention

[0004] This application proposes a method and system for batch data export based on data grouping and sorting, which can solve the problems of uneven thread load and low data export efficiency caused by unreasonable data allocation in the prior art.

[0005] The first aspect of this application provides a method for batch exporting data based on data grouping and sorting, the method comprising: Traverse the table files to be exported and construct corresponding key-value pairs by obtaining the attribute information of each data table; wherein, the key-value pairs include table ID and table space occupied; Based on the space occupied by the table, all data tables are sorted in a preset order. According to the current number of available threads, the sorted data tables are grouped to obtain several table groups, and each table group corresponds to one available thread. The data within the table group is exported simultaneously using each available thread.

[0006] The above scheme first iterates through the table files to be exported to determine the amount of data to be exported and the total number of data tables, and constructs a temporary database structure with key-value pairs to store information for each data table. The data tables are then sorted according to their size. First, each available thread is assigned a data table of approximately the same size, and then smaller data tables are assigned sequentially until all data tables are assigned to their corresponding table groups, achieving uniform grouping of the data tables. This grouping ensures a uniform distribution of the total size of the data tables within each group, providing a basis for load balancing when threads process the export task. Because the amount of data to be processed by each available thread is roughly the same, and the processing time of each thread is not significantly different, exporting data simultaneously can improve data export efficiency and fully utilize the system's multi-threaded resources.

[0007] In one possible implementation of the first aspect, the table file to be exported is traversed, and corresponding key-value pairs are constructed by obtaining the attribute information of each data table, specifically as follows: During the traversal, the attribute information of each data table is read, including file attributes and storage address; Based on the field attributes corresponding to the file attributes and the storage address, the table space occupied by each data table is obtained; Using the table ID of the data table as the key and the space occupied by the table as the value, construct a corresponding key-value pair for each data table.

[0008] The above scheme obtains the storage space occupied by each data table by reading its storage address; this space size reflects the amount of data in each table. Key-value pairs serve as attribute labels for the data tables, providing a basis for subsequent data sorting.

[0009] In one possible implementation of the first aspect, the table space occupied by each data table is obtained based on the field attributes corresponding to the file attributes and the storage address, specifically as follows: If the storage address points to a database, then the table space occupied by the data table is obtained from the database based on the storage address; If the storage address points to the local file system, then the interface of the local file system is called according to the storage address to obtain the number of bytes of the data table, and thus the space occupied by the table.

[0010] In one possible implementation of the first aspect, based on the space occupied by the tables, all data tables are sorted in a preset order, and the sorted data tables are grouped according to the current number of available threads to obtain several table groups, each table group corresponding to one available thread, specifically: The data tables are sorted according to their size in descending order of space usage to obtain a table sequence. Based on the current number of available threads, each data table within the table sequence is sequentially assigned to its corresponding table group, and the number of table groups is consistent with the number of available threads. Allocate one of the available threads to each table group.

[0011] The above scheme determines the number of table groups based on the number of available threads, ensuring that each table group corresponds to one available thread. To ensure a balanced amount of data processed by each available thread, each data table is sorted by size, resulting in a table sequence. Data tables are then extracted sequentially from this sequence and assigned to each table group, guaranteeing even data distribution and preventing large data tables from being concentrated in one place, which would increase the burden on available threads.

[0012] In one possible implementation of the first aspect, each data table within the table sequence is sequentially assigned to its corresponding table group, specifically as follows: The number of groups in the table is used as the first threshold. Extract a first threshold number of data tables sequentially from the table sequence, and assign the first threshold number of data tables to all table groups according to the table ID, as one element of each table group; After all data tables have been assigned, output all table groups.

[0013] In one possible implementation of the first aspect, the data within the table group is exported simultaneously by each available thread, specifically as follows: Using the available thread, the corresponding data is read from the storage address of the data table according to the table ID provided by the table group, and the data is exported to a preset file template; All available threads execute the export task simultaneously, and export the data in the order of the data tables within the table groups.

[0014] The above solution ensures that the amount of data exported by each available thread is balanced, preventing any single thread from processing a large number of large-sized data tables. Therefore, simultaneous export can save data processing time and improve data export efficiency.

[0015] One possible implementation of the first aspect also includes: sorting all data tables using a quicksort algorithm.

[0016] A second aspect of this application provides a data batch export system based on data grouping and sorting, the system comprising: a key-value pair construction module, a thread allocation module, and a data export module; The key-value pair construction module is used to traverse the table files to be exported and construct corresponding key-value pairs by obtaining the attribute information of each data table; wherein, the key-value pairs include table ID and table space occupied; The thread allocation module is used to sort all data tables in a preset order based on the space occupied by the table, and group the sorted data tables according to the current number of available threads to obtain several table groups, each table group corresponding to one available thread; The data export module is used to export data within the table groups simultaneously using each available thread.

[0017] A third aspect of this application provides a terminal device, the device comprising: a terminal device including a processor and a memory, the memory storing a computer program, wherein the processor executes the computer program to implement the steps of a data batch export method based on data grouping and sorting as described in any one embodiment of this application.

[0018] A fourth aspect of this application provides a storage medium that stores computer-readable program code, which, when executed, implements the steps of a data batch export method based on data grouping and sorting as described in any one embodiment of this application. Attached Figure Description

[0019] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating the specific process of a batch data export method based on data grouping and sorting, provided in an embodiment of this application. Figure 2 This is a detailed structural diagram of a data batch export system based on data grouping and sorting, provided in an embodiment of this application; Figure 3 This is a structural diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0023] First Embodiment like Figure 1 As shown, to address the problem of uneven thread load and low data export efficiency caused by unreasonable data allocation in the prior art, the first embodiment of this application provides a detailed flowchart of a data batch export method based on data grouping and sorting. This embodiment's data batch export method based on data grouping and sorting includes steps S1 to S3, detailed below: Step S1: Traverse the table files to be exported and construct corresponding key-value pairs by obtaining the attribute information of each data table.

[0024] In existing technologies, due to the significant differences in the size of different data tables, random thread allocation may result in a single thread having to export an excessive amount of data, causing a severe imbalance in the load between threads. Furthermore, the thread processing large data tables consumes a significant amount of time, prolonging the overall export time and failing to fully leverage the advantages of multithreading technology.

[0025] For example, when 10 threads are processing 100 data tables simultaneously, if the distribution is random, one thread might continuously process multiple large tables of tens or even hundreds of megabytes, while other threads process small tables of only a few kilobytes. In this situation, the completion time of the entire export task depends on the time taken by the thread processing the large tables. Other threads finish their export tasks early and remain idle, resulting in a serious waste of system resources and low export efficiency.

[0026] To address the aforementioned challenges, this application's embodiments sort and group data by size to ensure that the total amount of data processed by each thread is as balanced as possible, thus avoiding excessive differences in load between threads.

[0027] First, the exported table file is traversed, and the attribute information of each data table in the file is read. The attribute information includes file attributes and storage address.

[0028] Then, based on the field attributes corresponding to the file attributes and the storage address, the table space occupied by each data table is obtained from the location where the data tables are stored. For example, if the storage address points to a database, the table space occupied by the data table is obtained from the database based on the storage address; if the storage address points to a local file system, the local file system's interface is called based on the storage address to obtain the number of bytes of the data table, thus obtaining the table space occupied.

[0029] Therefore, the embodiments of this application do not depend on a specific data storage format or export tool, and are applicable to various scenarios that require multi-threaded batch export of data tables, such as database report export and big data analysis result export.

[0030] Finally, using the table ID as the key and the space occupied by the table as the value, a corresponding key-value pair is constructed for each data table. The table ID is the ID of the data table in the table file, and the key-value pair is actually a temporary data structure, which can be considered as a dictionary, used to store the key attribute information of the data table.

[0031] Based on the table ID, each data table is bound to its corresponding key-value pair, providing a basis for subsequent data table sorting and grouping.

[0032] Step S2: Based on the space occupied by the table, sort all the data tables in a preset order, and group the sorted data tables according to the current number of available threads to obtain several table groups, each table group corresponding to one available thread.

[0033] Based on the key-value pairs, the size of the table's storage space is compared, and all data tables are sorted in descending order of storage space to obtain a table sequence. For example, if table A occupies 50MB of storage space and table B occupies 2MB of storage space, then table A will appear before table B in the table sequence. The table sequence is actually a sorting result of table IDs, arranged in descending order of storage space.

[0034] In other embodiments, the tables can also be sorted in ascending order of the space they occupy.

[0035] Optionally, this application embodiment uses the quicksort algorithm to sort all data tables. In other embodiments, other efficient sorting algorithms such as merge sort can also be used to ensure good performance when processing large amounts of data.

[0036] The system retrieves the number of currently idle and available threads, and sequentially assigns each data table to its corresponding table group within the table sequence. The number of table groups is determined based on the number of available threads, ensuring that each table group is allocated the required number of threads.

[0037] The data table allocation process is as follows: the number of table groups is used as a first threshold. The first threshold number of data tables are extracted sequentially from the table sequence. The first threshold number of data tables are allocated to all table groups according to the table ID, as an element of each table group. Then, the remaining first threshold number of data tables are extracted from the table sequence again for allocation. After all data tables are allocated, all table groups are output.

[0038] For example, suppose there are 100 data tables to be exported and 10 table groups. Extract the top ten data tables in the table sequence and assign them to each table group as the first element of the 10 table groups. Then, assign the data tables ranked 11th to 20th to the 10 table groups as the second element. And so on, until all data tables have been assigned to the corresponding table groups.

[0039] Finally, the data tables in the resulting table groups are arranged in descending order of the table space occupied, and the total size of the data tables in each table group is as balanced as possible to avoid a situation where the amount of data to be processed by one available thread far exceeds that of other threads.

[0040] Since the data tables in each table group are arranged from largest to smallest and the grouping is balanced, the processing time of each thread is not much different. The maximum export time of the overall export task is effectively controlled within the time for a single available thread to process the largest data table plus the small time for processing other smaller data tables, which is much less than the maximum export time under the existing random allocation method, thus effectively improving the efficiency of data export.

[0041] Step S3: Export the data within the table group simultaneously using each available thread.

[0042] After the data table grouping is completed, each available thread reads the corresponding data table from the database or other data storage address according to the table ID, and exports the data table to a preset file template (if the template is an Excel file, then it is exported to that file).

[0043] All available threads execute the export task simultaneously, exporting data tables sequentially according to the order of the data tables within each table group. Because the data size in each table group is relatively uniform, the processing time of each available thread is not significantly different. For example, an available thread processing a 50MB data table will process smaller data tables after finishing that large data table, while the total size of the data tables processed by other available threads is also similar to that of the first available thread. Actual testing shows that using the method of this embodiment reduces the overall export time to approximately 40% of the original random allocation method, greatly improving data export efficiency.

[0044] Implementing the embodiments of this application has the following beneficial effects: This embodiment first iterates through the table file to be exported to determine the amount of data to be exported and the total number of data tables, and constructs a temporary database structure with key-value pairs to store information for each data table. The data tables are then sorted according to their size. First, data tables of similar size are allocated to each available thread, and then smaller data tables are allocated sequentially until all data tables are assigned to their corresponding table groups, achieving uniform grouping of the data tables. This grouping ensures a uniform distribution of the total size of the data tables within each group, providing a basis for load balancing when threads process the export task. Because the amount of data to be processed by each available thread is roughly the same, and the processing time of each thread is not significantly different, exporting data simultaneously can improve data export efficiency and fully utilize the system's multi-threaded resources.

[0045] Second Embodiment Furthermore, in order to implement the data batch export system based on data grouping and sorting corresponding to the above method embodiments, and to achieve the corresponding functions and technical effects, Figure 2 A structural diagram of a data batch export system based on data grouping and sorting is provided. For ease of explanation, only the parts relevant to this embodiment are shown. The data batch export system based on data grouping and sorting provided in this embodiment includes: The key-value pair construction module 201 is used to traverse the table file to be exported and construct corresponding key-value pairs by obtaining the attribute information of each data table; wherein, the key-value pair includes table ID and table space occupied.

[0046] In existing technologies, due to the significant differences in the size of different data tables, random thread allocation may result in a single thread having to export an excessive amount of data, causing a severe imbalance in the load between threads. Furthermore, the thread processing large data tables consumes a significant amount of time, prolonging the overall export time and failing to fully leverage the advantages of multithreading technology.

[0047] For example, when 10 threads are processing 100 data tables simultaneously, if the distribution is random, one thread might continuously process multiple large tables of tens or even hundreds of megabytes, while other threads process small tables of only a few kilobytes. In this situation, the completion time of the entire export task depends on the time taken by the thread processing the large tables. Other threads finish their export tasks early and remain idle, resulting in a serious waste of system resources and low export efficiency.

[0048] To address the aforementioned challenges, this application's embodiments sort and group data by size to ensure that the total amount of data processed by each thread is as balanced as possible, thus avoiding excessive differences in load between threads.

[0049] First, the exported table file is traversed, and the attribute information of each data table in the file is read. The attribute information includes file attributes and storage address.

[0050] Then, based on the field attributes corresponding to the file attributes and the storage address, the table space occupied by each data table is obtained from the location where the data tables are stored. For example, if the storage address points to a database, the table space occupied by the data table is obtained from the database based on the storage address; if the storage address points to a local file system, the local file system's interface is called based on the storage address to obtain the number of bytes of the data table, thus obtaining the table space occupied.

[0051] Finally, using the table ID as the key and the space occupied by the table as the value, a corresponding key-value pair is constructed for each data table. The table ID is the ID of the data table in the table file, and the key-value pair is actually a temporary data structure, which can be considered as a dictionary, used to store the key attribute information of the data table.

[0052] Based on the table ID, each data table is bound to its corresponding key-value pair, providing a basis for subsequent data table sorting and grouping.

[0053] The thread allocation module 202 is used to sort all data tables in a preset order based on the space occupied by the table, and group the sorted data tables according to the current number of available threads to obtain several table groups, each table group corresponding to one available thread.

[0054] Based on the key-value pairs, the size of the table's storage space is compared, and all data tables are sorted in descending order of storage space to obtain a table sequence. For example, if table A occupies 50MB of storage space and table B occupies 2MB of storage space, then table A will appear before table B in the table sequence. The table sequence is actually a sorting result of table IDs, arranged in descending order of storage space.

[0055] In other embodiments, the tables can also be sorted in ascending order of the space they occupy.

[0056] Optionally, this application embodiment uses the quicksort algorithm to sort all data tables. In other embodiments, other efficient sorting algorithms such as merge sort can also be used to ensure good performance when processing large amounts of data.

[0057] The system retrieves the number of currently idle and available threads, and sequentially assigns each data table to its corresponding table group within the table sequence. The number of table groups is determined based on the number of available threads, ensuring that each table group is allocated the required number of threads.

[0058] The data table allocation process is as follows: the number of table groups is used as a first threshold. The first threshold number of data tables are extracted sequentially from the table sequence. The first threshold number of data tables are allocated to all table groups according to the table ID, as an element of each table group. Then, the remaining first threshold number of data tables are extracted from the table sequence again for allocation. After all data tables are allocated, all table groups are output.

[0059] For example, suppose there are 100 data tables to be exported and 10 table groups. Extract the top ten data tables in the table sequence and assign them to each table group as the first element of the 10 table groups. Then, assign the data tables ranked 11th to 20th to the 10 table groups as the second element. And so on, until all data tables have been assigned to the corresponding table groups.

[0060] Finally, the data tables in the resulting table groups are arranged in descending order of the table space occupied, and the total size of the data tables in each table group is as balanced as possible to avoid a situation where the amount of data to be processed by one available thread far exceeds that of other threads.

[0061] Since the data tables in each table group are arranged from largest to smallest and the grouping is balanced, the processing time of each thread is not much different. The maximum export time of the overall export task is effectively controlled within the time for a single available thread to process the largest data table plus the small time for processing other smaller data tables, which is much less than the maximum export time under the existing random allocation method, thus effectively improving the efficiency of data export.

[0062] The data export module 203 is used to export data within the table group simultaneously using each available thread.

[0063] After the data table grouping is completed, each available thread reads the corresponding data table from the database or other data storage address according to the table ID, and exports the data table to a preset file template (if the template is an Excel file, then it is exported to that file).

[0064] All available threads execute the export task simultaneously, exporting data tables sequentially according to the order of the data tables within each table group. Because the data size in each table group is relatively uniform, the processing time of each available thread is not significantly different. For example, an available thread processing a 50MB data table will process smaller data tables after finishing that large data table, while the total size of the data tables processed by other available threads is also similar to that of the first available thread. Actual testing shows that using the method of this embodiment reduces the overall export time to approximately 40% of the original random allocation method, greatly improving data export efficiency.

[0065] Implementing the embodiments of this application has the following beneficial effects: This embodiment first iterates through the table file to be exported to determine the amount of data to be exported and the total number of data tables, and constructs a temporary database structure with key-value pairs to store information for each data table. The data tables are then sorted according to their size. First, data tables of similar size are allocated to each available thread, and then smaller data tables are allocated sequentially until all data tables are assigned to their corresponding table groups, achieving uniform grouping of the data tables. This grouping ensures a uniform distribution of the total size of the data tables within each group, providing a basis for load balancing when threads process the export task. Because the amount of data to be processed by each available thread is roughly the same, and the processing time of each thread is not significantly different, exporting data simultaneously can improve data export efficiency and fully utilize the system's multi-threaded resources.

[0066] Third Embodiment Furthermore, Figure 3 This is a structural diagram of a terminal device provided in one embodiment of this application. Figure 3As shown, the terminal device 3 of this embodiment includes: at least one processor 30 (in... Figure 3 (Only one is shown in the image) and a memory 31 and a computer program 32 stored in the memory 31 and executable on the at least one processor, wherein when the processor 30 executes the computer program 32, it can implement the steps of a data batch export method based on data grouping and sorting as described in any one embodiment of the present application.

[0067] The terminal device 3 may be a computing device such as a desktop computer, a cloud server, or a laptop computer, and the computing device may include, but is not limited to, a processor 30 and a memory 31. Figure 3 This is merely an example of terminal device 3 and does not constitute a limitation on terminal device 3. It may include more or fewer components than those shown in the figure.

[0068] This application provides a storage medium that stores computer-readable program code. When the computer-readable program code is executed, it implements the steps of the above-described method for batch exporting data based on data grouping and sorting.

[0069] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, or improvements made by those skilled in the art within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for batch exporting data based on data grouping and sorting, characterized in that, include: The table files to be exported are traversed, and corresponding key-value pairs are constructed by obtaining the attribute information of each data table; wherein, the key-value pairs include the table ID and the table space occupied. Based on the space occupied by the table, all data tables are sorted in a preset order. According to the current number of available threads, the sorted data tables are grouped to obtain several table groups, and each table group corresponds to one available thread. The data within the table group is exported simultaneously using each available thread.

2. The method for batch exporting data based on data grouping and sorting according to claim 1, characterized in that, Iterate through the table files to be exported, and construct corresponding key-value pairs by obtaining the attribute information of each data table, specifically as follows: During the traversal, the attribute information of each data table is read, including file attributes and storage address; Based on the field attributes corresponding to the file attributes and the storage address, the table space occupied by each data table is obtained; Using the table ID of the data table as the key and the space occupied by the table as the value, construct a corresponding key-value pair for each data table.

3. The method for batch exporting data based on data grouping and sorting according to claim 2, characterized in that, Based on the field attributes corresponding to the file attributes and the storage address, the table space occupied by each data table is obtained, specifically: If the storage address points to a database, then the table space occupied by the data table is obtained from the database based on the storage address; If the storage address points to the local file system, then the interface of the local file system is called according to the storage address to obtain the number of bytes of the data table, and the space occupied by the table is obtained.

4. The method for batch exporting data based on data grouping and sorting according to claim 1, characterized in that, Based on the space occupied by the tables, all data tables are sorted in a preset order. Then, according to the current number of available threads, the sorted data tables are grouped into several table groups, each corresponding to one available thread. Specifically: The data tables are sorted according to their size in descending order of space usage to obtain a table sequence. Based on the current number of available threads, each data table in the table sequence is sequentially assigned to its corresponding table group, and the number of table groups is consistent with the number of available threads. Allocate one of the available threads to each table group.

5. The method for batch exporting data based on data grouping and sorting according to claim 4, characterized in that, Within the table sequence, each data table is sequentially assigned to its corresponding table group, specifically as follows: The number of groups in the table is used as the first threshold. Extract a first threshold number of data tables sequentially from the table sequence, and assign the first threshold number of data tables to all table groups according to the table ID, as one element of each table group; After all data tables have been assigned, output all table groups.

6. The method for batch exporting data based on data grouping and sorting according to claim 1, characterized in that, The data within the table groups is exported simultaneously using each available thread, specifically as follows: Using the available thread, the corresponding data is read from the storage address of the data table according to the table ID provided by the table group, and the data is exported to a preset file template; All available threads execute the export task simultaneously, and export the data in the order of the data tables within the table groups.

7. The method for batch exporting data based on data grouping and sorting according to claim 4, characterized in that, Also includes: The quicksort algorithm is used to sort all the data tables.

8. A data batch export system based on data grouping and sorting, characterized in that, include: Key-value pair building module, thread allocation module, and data export module; The key-value pair construction module is used to traverse the table files to be exported and construct corresponding key-value pairs by obtaining the attribute information of each data table; wherein, the key-value pairs include table ID and table space occupied; The thread allocation module is used to sort all data tables in a preset order based on the space occupied by the table, and group the sorted data tables according to the current number of available threads to obtain several table groups, each table group corresponding to one available thread; The data export module is used to export data within the table groups simultaneously using each available thread.

9. A terminal device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the steps of the data batch export method based on data grouping and sorting as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores computer-readable program code, which, when executed, implements the steps of a data batch export method based on data grouping and sorting as described in any one of claims 1 to 7.

Images