Processing method and system for quickly processing table data import through asynchronous multithreading

Abstract

The invention provides a processing method and system for rapidly processing table data import through asynchronous multithreading, and relates to the technical field of data processing.The method comprises the steps that information analysis is conducted on table data needing to be imported, set data is generated, and one piece of data in a table corresponds to one piece of data in a set; judging the size of the set data, and if the number of pieces of the set data is greater than a preset threshold value, segmenting the set data to form a plurality of subsets; according to the number of the divided subsets, starting a corresponding number of asynchronous threads, and respectively processing data of one subset by each asynchronous thread; each asynchronous thread carries out table storage persistence processing on the sub-set data processed by the asynchronous thread; and performing asynchronous counting by using CountDownLatch, and when all asynchronous threads are successfully processed, returning a flag bit, which indicates that the data is successfully imported. The method has the advantages of high efficiency, low resource occupation and easiness in implementation, and is suitable for scenes needing to process large-data-volume table import tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for asynchronous multi-threaded rapid processing of table data import. Background Art

[0002] In the current field of data processing technology, the processing and importing of tabular data is an important link. However, existing table processing tools often take a long time when faced with large data database writing tasks, which limits the data processing efficiency to a certain extent. In order to solve this problem, some other solutions have emerged on the Internet, such as decoupling using message queues and using asynchronous thread processing. However, these solutions often involve more resource usage in practical applications, which not only increases the system burden but also increases the operating cost. In view of this situation, it is necessary to develop a more efficient and less resource-intensive tabular data processing method. Summary of the invention

[0003] To this end, an embodiment of the present invention provides a method and system for asynchronous multi-threaded rapid processing of table data import, which is used to solve the problem that table data processing tools in the prior art take too long to write to the database after parsing the data, and other asynchronous processing solutions occupy more resources.

[0004] In order to solve the above problems, an embodiment of the present invention provides a method for asynchronous multi-threaded rapid processing of table data import, the method comprising:

[0005] Step S1: parsing the table data to be imported to generate set data, where one piece of data in the table corresponds to one piece of data in the set;

[0006] Step S2: Determine the size of the set data. If the number of the set data is greater than a preset threshold, divide the set data into multiple subsets.

[0007] Step S3: according to the number of the divided subsets, a corresponding number of asynchronous threads are started, and each asynchronous thread processes the data of one subset respectively;

[0008] Step S4: Each asynchronous thread stores and persists the processed subset data in a table;

[0009] Step S5: Use CountDownLatch to perform asynchronous counting. When all asynchronous threads are processed successfully, a flag is returned to indicate that the data import is successful.

[0010] Preferably, in step S2, the preset threshold is dynamically adjusted according to system resource conditions and data processing requirements.

[0011] Preferably, in step S2, the method for segmenting the set data is:

[0012] The collection data is packaged according to a preset threshold, and each packaged collection contains a number of data items that does not exceed the preset threshold.

[0013] Preferably, before step S3, the method includes: configuring a thread pool for managing asynchronous threads, wherein the thread pool attributes include the number of core threads, the maximum number of threads and the queue capacity.

[0014] Preferably, the thread pool is implemented using ThreadPoolTaskExecutor.

[0015] Preferably, in step S3, each asynchronous thread uses the @Async annotation to perform asynchronous processing.

[0016] Preferably, in step S4, the table storage persistence processing includes performing a database write operation on the data in the subset.

[0017] The embodiment of the present invention further provides a processing system for asynchronous multi-threaded rapid processing of table data import, which is used to implement the above-mentioned processing method for asynchronous multi-threaded rapid processing of table data import, specifically comprising:

[0018] The data analysis module is used to analyze the table data to be imported and generate set data, where one piece of data in the table corresponds to one piece of data in the set;

[0019] The data segmentation module is used to determine the size of the set data. If the number of the set data is greater than a preset threshold, the set data is segmented to form multiple subsets.

[0020] The thread management module is used to start a corresponding number of asynchronous threads according to the number of the divided subsets, and each asynchronous thread processes the data of one subset respectively;

[0021] The data processing module is used by each asynchronous thread to store and persist the subset data processed by it;

[0022] The counting module is used to use CountDownLatch for asynchronous counting. When all asynchronous threads are processed successfully, a flag is returned to indicate that the data import is successful.

[0023] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned asynchronous multi-threaded fast processing method for importing table data.

[0024] An embodiment of the present invention further provides a computer storage medium storing a computer software product, wherein the computer software product includes a number of instructions for enabling a computer device to execute the above-mentioned asynchronous multi-threaded fast processing method for importing table data.

[0025] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0026] (1) The present invention uses asynchronous multi-threading to process table data import, can divide and group tables according to their size, and use a thread pool to manage asynchronous threads. This method significantly improves the parallelism of data processing, thereby greatly shortening the time for data import into the database and improving overall work efficiency.

[0027] (2) Compared with the scheme using message queues in the prior art, the present invention realizes asynchronous multi-threaded processing by integrating the existing framework technology without introducing external resources. This reduces the additional resource occupation, reduces the system burden, and makes the data import process more efficient and economical.

[0028] (3) The asynchronous multi-threaded table data importing method provided by the present invention is not only flexible in technical implementation, but also highly reusable. Through simple configuration and adjustment, it can adapt to data import requirements of different scales and types, and provide a convenient data processing solution for various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0030] Figure 1 A flowchart of a method for asynchronous multi-threaded rapid processing of table data import provided in an embodiment;

[0031] Figure 2 A block diagram of a processing system for asynchronous multi-threaded rapid processing of table data import provided in an embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Embodiment 1

[0034] In order to solve the problem that the table data processing tools in the prior art take too long to write to the database after parsing the data, and other asynchronous processing solutions occupy a lot of resources. Figure 1 As shown, an embodiment of the present invention proposes a method for asynchronous multi-threaded rapid processing of table data import, the method comprising:

[0035] Step S1: parsing the table data to be imported to generate set data, where one piece of data in the table corresponds to one piece of data in the set;

[0036] Step S2: Determine the size of the set data. If the number of the set data is greater than a preset threshold, divide the set data into multiple subsets.

[0037] Step S3: according to the number of the divided subsets, a corresponding number of asynchronous threads are started, and each asynchronous thread processes the data of one subset respectively;

[0038] Step S4: Each asynchronous thread stores and persists the processed subset data in a table;

[0039] Step S5: Use CountDownLatch to perform asynchronous counting. When all asynchronous threads are processed successfully, a flag is returned to indicate that the data import is successful.

[0040] It can be seen from the above technical solution that the present invention provides a method for processing table data import by asynchronous multi-threading fast processing, through the steps of information analysis, data segmentation, asynchronous thread processing, table persistence and asynchronous counting. The method realizes data parallel processing by judging the size of the collection data and segmenting the processing, and managing the asynchronous threads by the thread pool. Its advantage is that it can flexibly adjust the processing strategy according to the size of the table, effectively reduce the time consumption of database writing, and improve data processing efficiency. Asynchronous counting ensures that all data processing is successful before returning the flag bit, which ensures the integrity and accuracy of the data import, and provides an efficient and reliable solution for big data processing.

[0041] In step S1, information of the table data to be imported is parsed to generate set data, wherein one piece of data in the table corresponds to one piece of data in the set.

[0042] Specifically, first, through professional parsing tools or custom parsing algorithms, read and parse each item in the table data to ensure that each data is extracted accurately.

[0043] Next, the parsed data is organized according to a specific data structure (such as List, Map and other collection types in Java) to generate collection data. In this process, it is necessary to ensure that a piece of data in the table can correspond to a piece of data in the collection to maintain data integrity and consistency.

[0044] Finally, the generated collection data is preliminarily verified to ensure the accuracy and completeness of the data and prepare for subsequent data processing.

[0045] In step S2, the size of the set data is determined. If the number of the set data is greater than a preset threshold, the set data is segmented to form a plurality of subsets.

[0046] Specifically, first, a reasonable preset threshold (such as 500 data) is set, and the threshold can be dynamically adjusted according to system resource conditions and data processing requirements.

[0047] Next, it is determined whether the size of the collection data exceeds a preset threshold. If the number of collection data is greater than the preset threshold, the collection data needs to be segmented. That is, the collection data is packaged according to the preset threshold, and each packaged collection contains a number of data that does not exceed the preset threshold.

[0048] When splitting, you can use various strategies such as uniform splitting and on-demand splitting to split the collection data into multiple subsets of similar size or on-demand allocation. At the same time, it is necessary to ensure that the data in each subset is complete and independent for subsequent asynchronous processing.

[0049] In step S3, a corresponding number of asynchronous threads are started according to the number of the divided subsets, and each asynchronous thread processes the data of one subset respectively.

[0050] Specifically, first, the number of asynchronous threads to be started is determined according to the number of subsets after segmentation, and this number can correspond to the number of subsets one by one, or can be dynamically adjusted according to factors such as system resources and processing speed.

[0051] Next, use the thread pool (such as ThreadPoolTaskExecutor) to manage these asynchronous threads. The thread pool includes properties such as the number of core threads, the maximum number of threads, and the queue capacity. The thread pool can provide convenient functions such as thread creation, destruction, and reuse, effectively reducing thread management overhead.

[0052] When an asynchronous thread is started, each subset needs to be assigned to a corresponding thread for processing. At the same time, it is necessary to ensure the independence between threads to avoid data interference and competition conditions. In the present invention, each asynchronous thread uses the @Async annotation for asynchronous processing.

[0053] In step S4, each asynchronous thread performs table storage and persistence processing on the subset data processed by itself.

[0054] Specifically, in an asynchronous thread, it is first necessary to preprocess the subset data assigned to itself, such as data format conversion, data verification, etc.

[0055] Then, using efficient tools such as database connection pool, the pre-processed data is written into the database in batches. In this process, it is necessary to ensure the accuracy and completeness of the data while minimizing the overhead of database operations.

[0056] If an exception or error occurs during data writing, it is necessary to handle the exception in a timely manner and record the error information for subsequent troubleshooting and processing.

[0057] In step S5, CountDownLatch is used for asynchronous counting. When all asynchronous threads are processed successfully, a flag is returned to indicate that the data import is successful.

[0058] Specifically, before all asynchronous threads start processing data, a CountDownLatch object is initialized and its count value is set to the number of subsets (ie, the number of asynchronous threads).

[0059] After each asynchronous thread has processed the sub-collection data it is responsible for, it will call the countDown() method of CountDownLatch to decrement the counter value.

[0060] When all asynchronous threads have been processed (that is, the counter value is decremented to 0), CountDownLatch will trigger an event (such as waking up the waiting thread, executing the callback method, etc.). In this event, you can check the processing results of all asynchronous threads and return the corresponding flag to indicate whether the data import is successful.

[0061] If the data import is successful, subsequent operations can be performed (such as notifying users, updating status, etc.); if the data import fails, corresponding processing is required according to the cause of the failure (such as retrying, recording logs, etc.).

[0062] Embodiment 2

[0063] like Figure 2As shown, the present invention provides a processing system for asynchronous multi-threaded rapid processing of table data import, which is used to implement the processing method for asynchronous multi-threaded rapid processing of table data import of the above embodiment 1, specifically comprising:

[0064] The data analysis module 100 is used to analyze the table data to be imported and generate set data, wherein one piece of data in the table corresponds to one piece of data in the set;

[0065] The data segmentation module 200 is used to determine the size of the set data, and if the number of the set data is greater than a preset threshold, the set data is segmented to form multiple subsets;

[0066] The thread management module 300 is used to start a corresponding number of asynchronous threads according to the number of the divided subsets, and each asynchronous thread processes the data of one subset respectively;

[0067] The data processing module 400 is used for each asynchronous thread to store and persist the subset data processed by the thread;

[0068] The counting module 500 is used to perform asynchronous counting using CountDownLatch. When all asynchronous threads are processed successfully, a flag is returned to indicate that the data is imported successfully.

[0069] The present embodiment provides a processing system for asynchronous multi-threaded rapid processing of table data import, which is used to implement the aforementioned processing method for asynchronous multi-threaded rapid processing of table data import. Therefore, the specific implementation method of the processing system for asynchronous multi-threaded rapid processing of table data import can be seen in the embodiment part of the processing method for asynchronous multi-threaded rapid processing of table data import mentioned above. For example, the data parsing module 100, the data segmentation module 200, the thread management module 300, the data processing module 400, and the counting module 500 are respectively used to implement steps S1, S2, S3, S4, and S5 in the aforementioned processing method for asynchronous multi-threaded rapid processing of table data import. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here.

[0070] Embodiment 3

[0071] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned asynchronous multi-threaded fast processing method for importing table data.

[0072] Embodiment 4

[0073] An embodiment of the present invention further provides a computer storage medium storing a computer software product. The computer software product includes a number of instructions for enabling a computer device to execute the above-mentioned asynchronous multi-threaded fast processing method for importing table data.

[0074] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0075] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0077] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A method for asynchronous multi-threaded rapid processing of table data import, characterized in that: include: Step S1: parsing the table data to be imported to generate set data, where one piece of data in the table corresponds to one piece of data in the set; Step S2: Determine the size of the set data. If the number of the set data is greater than a preset threshold, divide the set data into multiple subsets. Step S3: according to the number of the divided subsets, a corresponding number of asynchronous threads are started, and each asynchronous thread processes the data of one subset respectively; Step S4: Each asynchronous thread stores and persists the processed subset data in a table; Step S5: Use CountDownLatch to perform asynchronous counting. When all asynchronous threads are processed successfully, a flag is returned to indicate that the data import is successful.

2. The method for asynchronous multi-threaded rapid processing of table data import according to claim 1, characterized in that: In step S2, the preset threshold is dynamically adjusted according to system resource conditions and data processing requirements.

3. The method for asynchronous multi-threaded rapid processing of table data import according to claim 1, characterized in that: In step S2, the method for segmenting the set data is: The collection data is packaged according to a preset threshold, and each packaged collection contains a number of data items that does not exceed the preset threshold.

4. The method for asynchronous multi-threaded rapid processing of table data import according to claim 1, characterized in that: Before step S3, the process includes: configuring a thread pool for managing asynchronous threads, wherein the thread pool attributes include the number of core threads, the maximum number of threads, and the queue capacity.

5. The method for asynchronous multi-threaded rapid processing of table data import according to claim 4, characterized in that: The thread pool is implemented using ThreadPoolTaskExecutor.

6. The method for asynchronous multi-threaded rapid processing of table data import according to claim 1, characterized in that: In step S3, each asynchronous thread uses the @Async annotation to perform asynchronous processing.

7. The method for asynchronous multi-threaded rapid processing of table data import according to claim 1, characterized in that: In step S4, the table storage persistence processing includes performing a database write operation on the data in the subset.

8. A processing system for asynchronous multi-threaded rapid processing of table data import, characterized in that: The system is used to implement the asynchronous multi-threaded fast processing table data import processing method according to any one of claims 1 to 7, specifically comprising: The data analysis module is used to analyze the table data to be imported and generate set data, where one piece of data in the table corresponds to one piece of data in the set; The data segmentation module is used to determine the size of the set data. If the number of the set data is greater than a preset threshold, the set data is segmented to form multiple subsets. The thread management module is used to start a corresponding number of asynchronous threads according to the number of the divided subsets, and each asynchronous thread processes the data of one subset respectively; The data processing module is used by each asynchronous thread to store and persist the subset data processed by it; The counting module is used to use CountDownLatch for asynchronous counting. When all asynchronous threads are processed successfully, a flag is returned to indicate that the data import is successful.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the asynchronous multi-threaded fast processing method for table data import as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, which includes a number of instructions for enabling a computer device to execute the asynchronous multi-threaded fast processing method for importing table data as described in any one of claims 1 to 7.

Images