Multi-thread single-task asynchronous import method for mass data

Through the multi-threaded single-task asynchronous import method, the performance bottlenecks and stability problems of traditional synchronous import methods are solved, the system performance and user experience are improved, and data integrity and consistency are ensured.

CN119065808AInactive Publication Date: 2024-12-03HEBEI TONGFU SHARING TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411071839.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional synchronous data import methods lead to performance bottlenecks, response delays and system stability problems when processing large batches of data, resulting in slow system response, poor user experience, and may cause memory overflow and database lock contention.

Method used

Using multi-threaded single-task asynchronous import method, the data import process is broken down into multiple stages, and the background task or message queue is used to execute asynchronously. Through the shared queue and snapshot storage mechanism, task parallel processing and resource optimization are decomposed to ensure system stability and User interface response.

Benefits of technology

It significantly improves data import performance and system stability, avoids unresponsive state, enhances user experience and system reliability, and ensures data integrity and consistency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065808A_ABST
    Figure CN119065808A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-thread single-task asynchronous import method for mass data. The method comprises the following steps: setting a rule and generating an import configuration template; when an import task is started in a multi-node distributed system, the task is allocated through a shared queue, and nodes process data according to a configured template rule and store snapshots. According to the method, a multi-thread single-task asynchronous import scheme is adopted, the data import process is decomposed into a plurality of stages, and asynchronous execution is carried out by utilizing background tasks or message queues, so that the performance can be effectively improved, the response delay is reduced, and the stability of the system is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of data import, in particular to a multi-threaded single-task asynchronous import method for large batches of data. Background Art

[0002] As the business scale grows, especially when large-volume data import operations involving Excel, CSV and other files are involved, the following problems may occur when executing traditional synchronization methods:

[0003] 1. Performance bottleneck: When the amount of imported data is huge, synchronous processing will cause computing resources to be occupied for a long time and may even trigger a timeout error.

[0004] When the amount of data that the system needs to process is very large, if the synchronous processing method is used, that is, processing one task at a time and not processing the next one until it is completed, this will cause the system to be unable to process other tasks while processing the current task, thereby reducing efficiency.

[0005] Performance bottlenecks are usually manifested as slower system responses and reduced processing speed, especially when resources are limited, such as CPU, memory, or disk I / O.

[0006] 2. Response delay: The user interface will become unresponsive while waiting for data import to complete, seriously affecting the user experience.

[0007] When synchronously processing data import, if a task takes a long time to complete, the user interface may not respond to other user operations before the task is completed, causing the user to wait. This state is called "unresponsive state".

[0008] Response delay directly affects the user experience, making users feel that the system is slow or unstable.

[0009] 3. System stability: Large-scale synchronous import may cause problems such as memory overflow and database lock contention, posing a threat to the robustness of the system.

[0010] Large-scale synchronous data import may consume a lot of memory. If it exceeds the system's carrying capacity, it may cause memory overflow, that is, the system does not have enough memory to process the current task.

[0011] Database lock contention occurs when multiple processes or threads try to access or modify the same data resource at the same time. Without proper synchronization mechanisms, it may cause data inconsistency or system crash.

[0012] The above problems are all related to resource management and task scheduling during the data import process. When faced with large amounts of data, traditional synchronous processing methods cannot efficiently process tasks due to the limitations of resource allocation and task scheduling, resulting in performance degradation and stability issues.

[0013] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0014] In view of the defects existing in the prior art, the purpose of the present invention is to provide a multi-threaded single-task asynchronous import method for large amounts of data. By adopting a multi-threaded single-task asynchronous import scheme, the data import process is decomposed into multiple stages and asynchronously executed using background tasks or message queues, which can effectively improve performance, reduce response delays, and enhance system stability.

[0015] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0016] A multi-threaded single-task asynchronous import method for large batches of data, characterized by comprising the following steps:

[0017] Set rules and generate import configuration templates;

[0018] When starting an import task in a multi-node distributed system, the task is assigned through a shared queue, and the node processes the data according to the configured template rules and saves the snapshot.

[0019] Based on the above technical solution, the setting rules include the following steps:

[0020] Set the template field name and its corresponding json data structure;

[0021] Set the field type and the corresponding header comment one by one according to the field name;

[0022] Configure the callback addresses of each business system, the number of callbacks each time, and the data verification rules.

[0023] Based on the above technical solution, after starting the import task, first start a public queue, each node shares the public queue, and configure the number of threads of each node according to the preset usage rules;

[0024] After the import task is created, it first enters the public queue. The free nodes will go to the public nodes to get the task. After the import task comes in, the corresponding thread is assigned to perform corresponding processing according to the configured template rules, and a snapshot storage of the configuration of the template rules is saved.

[0025] Based on the above technical solution, when executing each import task, the data import process is decomposed into multiple stages, and these stages are executed asynchronously using background tasks or message queues;

[0026] The multiple stages include: rule configuration stage, scheduling allocation stage, subsystem data processing stage, result query display stage, and snapshot storage stage.

[0027] Based on the above technical solution, all source data corresponding to the import task are archived and stored in the asynchronous import management data center.

[0028] Based on the above technical solutions, each subsystem of the multi-node distributed system processes data according to unified specifications and records the success and failure of core business data;

[0029] The record results are forwarded to the asynchronously imported management data center for backup.

[0030] Based on the above technical solution, the asynchronously imported management data center sets a unified access specification. Each subsystem of the multi-node distributed system processes the corresponding data according to the access specification. After each subsystem completes the data processing, it returns the corresponding standard data format. The core business data is recorded according to the success or failure of the returned data.

[0031] Based on the above technical solution, the asynchronous import management data center provides an import result viewing portal, showing the number and reasons of successful and failed data, and allows downloading failed results for modification and re-upload.

[0032] The multi-threaded single-task asynchronous import method for large batches of data described in the present invention has the following beneficial effects:

[0033] 1. Can significantly improve the performance of data import.

[0034] Through the asynchronous processing mechanism, the import task is decomposed into multiple independent subtasks, which can be executed in parallel in different threads. Compared with the traditional synchronous processing method, this method can make better use of the computing power of multi-core processors and reduce the processing time of a single task. At the same time, through the introduction of background tasks or message queues, the decoupling of tasks and the optimal allocation of resources are further realized, so that the system can process more import tasks at the same time, thereby greatly improving the overall import efficiency.

[0035] 2. The user interface is no longer unresponsive while waiting for data to be imported.

[0036] The import task is moved to the background for execution, and the front-end interface can continue to respond to other user operations. At the same time, by providing a unified import result viewing portal and detailed import result display (including the number of successful items, the number of failed items, and the reasons for failure, etc.), users can understand the import progress and results anytime and anywhere, which enhances the transparency and convenience of user operations.

[0037] 3. The stability of the system has been significantly enhanced by dispersing processing pressure, reducing resource contention and introducing a snapshot storage mechanism.

[0038] The asynchronous processing mechanism enables the system to handle multiple import tasks at the same time, avoiding problems such as system crashes or slow responses caused by a single task occupying a large amount of resources for a long time. At the same time, the snapshot storage mechanism ensures the integrity and consistency of historical data, avoiding data display problems caused by template changes.

[0039] In addition, through reasonable resource allocation and error handling mechanisms, the system can recover in time and continue to perform tasks when abnormal situations occur.

[0040] In summary, the multi-threaded single-task asynchronous import method for large batches of data described in the present invention has shown significant beneficial effects in improving performance, optimizing user experience, and enhancing system stability. These effects not only improve the efficiency and accuracy of data processing, but also enhance the reliability of the system and the satisfaction of user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention has the following accompanying drawings:

[0042] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0043] Figure 1 A flowchart of Embodiment 1 of a multi-threaded single-task asynchronous import method for large batch data described in the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings. The detailed description is a description made in conjunction with exemplary embodiments of the present invention, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0045] Conventional asynchronous multi-threaded import does not store snapshots of the configuration of the imported data, and the presentation of historical data will be incomplete, for example:

[0046] When template001 is first generated, it includes columns A, B, and C. During use, template001 saves 1 million records. When template001 is opened, the data in columns A, B, and C can be displayed normally.

[0047] The modified template template001 includes columns B, C, and D. When the modified template template001 is opened, column A of the 1 million data saved before the modification will be missing, and column D will be empty.

[0048] like Figure 1 As shown, the present invention provides a multi-threaded single-task asynchronous import method for large batches of data, comprising the following steps:

[0049] Set rules and generate import configuration templates;

[0050] Exemplarily, the setting rule includes the following steps:

[0051] Set the template field name and its corresponding json data structure;

[0052] Set the field type and the corresponding header comment one by one according to the field name;

[0053] Configure the callback addresses of each business system, the number of callbacks each time, and the data verification rules;

[0054] When starting an import task in a multi-node distributed system, the task is assigned through a shared queue, and the node processes the data according to the configured template rules and saves the snapshot.

[0055] The asynchronous import scheme allows the system to adapt to different workloads by adjusting the number of threads and resource allocation when facing different data volumes and complexities, thereby improving the flexibility of the system.

[0056] Based on the above technical solution, after starting the import task, first start a public queue, each node shares the public queue, and configure the number of threads of each node according to the preset usage rules;

[0057] After the import task is created, it first enters the public queue. The free nodes will go to the public nodes to get the task. After the import task comes in, the corresponding thread is assigned to perform corresponding processing according to the configured template rules, and a snapshot storage of the configuration of the template rules is saved.

[0058] When executing each import task, a snapshot storage is made for the configuration of the template rules, and each import task is bound to a snapshot storage. Although the field mapping of the template will change due to the modification, the display of historical data does not change. The data is still obtained from the bound snapshot storage, avoiding the problem of incorrect data display after the template is modified.

[0059] By storing template rule configurations in snapshots, the display rules for historical data can remain unchanged even after changes to the template field mapping. This means that users can view and analyze historical data without having to worry about changes in data structure affecting data integrity and consistency.

[0060] The snapshot mechanism provides a reliable backup for historical data, making it possible to trace and access the original data format even after the template is updated or changed. When the template or data processing logic needs to be updated, the system can be maintained and upgraded more easily due to the existence of snapshots without affecting the display and analysis of existing data.

[0061] Based on the above technical solution, when executing each import task, the data import process is decomposed into multiple stages, and these stages are executed asynchronously using background tasks or message queues;

[0062] The multiple stages include: rule configuration stage, scheduling allocation stage, subsystem data processing stage, result query display stage, and snapshot storage stage.

[0063] Based on the above technical solution, all source data corresponding to the import task are archived and stored in the asynchronous import management data center.

[0064] Based on the above technical solutions, each subsystem of the multi-node distributed system processes data according to unified specifications and records the success and failure of core business data;

[0065] The record results are forwarded to the asynchronously imported management data center for backup.

[0066] Based on the above technical solution, the asynchronously imported management data center sets a unified access specification. Each subsystem of the multi-node distributed system processes the corresponding data according to the access specification. After each subsystem completes the data processing, it returns the corresponding standard data format. The core business data is recorded according to the success or failure of the returned data.

[0067] To ensure the consistency of data processing, a unified interface parameter specification is defined to ensure the consistency of data processing.

[0068] To ensure the traceability of data processing results, record the data processing results, distinguish between success and failure, and record the reasons for failure.

[0069] Based on the above technical solution, the asynchronous import management data center provides an import result viewing portal, showing the number and reasons of successful and failed data, and allows downloading failed results for modification and re-upload.

[0070] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

[0071] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by those skilled in the art based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A multi-threaded single-task asynchronous import method for large batches of data, characterized in that: The steps include: Set rules and generate import configuration templates; When starting an import task in a multi-node distributed system, the task is assigned through a shared queue, and the node processes the data according to the configured template rules and saves the snapshot.

2. A multi-threaded single-task asynchronous import method for large batches of data as claimed in claim 1, characterized in that: The setting rules include the following steps: Set the template field name and its corresponding json data structure; Set the field type and the corresponding header comment one by one according to the field name; Configure the callback addresses of each business system, the number of callbacks each time, and the data verification rules.

3. A multi-threaded single-task asynchronous import method for large batches of data as claimed in claim 1, characterized in that: After starting the import task, a public queue is started first. Each node shares the public queue, and the number of threads of each node is configured according to the preset usage rules. After the import task is created, it first enters the public queue. The free nodes will go to the public nodes to get the task. After the import task comes in, the corresponding thread is assigned to perform corresponding processing according to the configured template rules, and a snapshot storage of the configuration of the template rules is saved.

4. A multi-threaded single-task asynchronous import method for large batches of data as claimed in claim 3, characterized in that: When executing each import task, break down the data import process into multiple stages and use background tasks or message queues to execute these stages asynchronously; The multiple stages include: rule configuration stage, scheduling allocation stage, subsystem data processing stage, result query display stage, and snapshot storage stage.

5. A multi-threaded single-task asynchronous import method for large batches of data as claimed in claim 3, characterized in that: All source data corresponding to the import task are archived and stored in the asynchronous import management data center.

6. A multi-threaded single-task asynchronous import method for large batches of data as claimed in claim 3, characterized in that: Each subsystem of the multi-node distributed system processes data according to unified specifications and records the success and failure of core business data; The record results are forwarded to the asynchronously imported management data center for backup.

7. A multi-threaded single-task asynchronous import method for large batches of data as claimed in claim 3, characterized in that: The asynchronously imported management data center sets a unified access specification. Each subsystem of the multi-node distributed system processes the corresponding data according to the access specification. After each subsystem completes the data processing, it returns the corresponding standard data format. The core business data is recorded according to the success or failure of the returned data.

8. A multi-threaded single-task asynchronous import method for large batches of data as claimed in claim 3, characterized in that: The management data center for asynchronous import provides an import result viewing portal, which displays the number and reasons of successful and failed data, and allows downloading failed results, modifying them, and uploading them again.