A method and system for multi-threaded data processing based on a custom condition queue

Abstract

This invention provides a method and system for multi-threaded data processing based on a custom condition queue. The method includes creating a data processing class object, initializing the maximum number of child threads that can be created, creating a custom double-ended condition queue based on the maximum number of child threads, having the main thread call the data processing class object to process data, decomposing each item into a processable segment, encapsulating the processable item and adding it to the queue, and assigning a child thread to handle it. Each child thread retrieves an element from the double-ended condition queue and processes the data within the element according to its task type. This invention improves system resource utilization, enhances the system's scalability for processing various types of data, avoids the possibility of multiple threads competing for shared data, eliminates the time overhead of introducing lock mechanisms to synchronize data, and improves the utilization of child threads and program processing performance.

Description

Technical Field

[0001] This invention belongs to the technical field, specifically relating to a method and system for multi-threaded data processing based on a custom condition queue. Background Technology

[0002] Application software is required to have higher processing performance. To improve the processing performance of application software, parallel technology can be used in program design, or more CPUs and more memory can be configured at the hardware level. However, hardware upgrades often require a larger investment in procurement costs, while software-level improvements using parallel processing technology are relatively cheaper and are often the preferred solution.

[0003] Parallelism in a broad sense refers to a program processing multiple tasks simultaneously. To improve program performance, two common parallel processing techniques are multi-processing and multi-threading. Compared to multi-threading, multi-processing offers advantages such as resource isolation between processes, simpler programming, and higher reliability, but it also has disadvantages such as higher memory consumption and lower CPU utilization.

[0004] Therefore, multi-process technology is suitable for weakly correlated parallel processing of multiple tasks, but not for applications requiring significant CPU usage for computation. Multi-threading technology, on the other hand, is widely used in data processing due to its advantages such as simple data sharing, low memory consumption, and high CPU utilization.

[0005] However, for data computing and data analysis tasks, such as the calculation of modulation characteristics or the synthesis of signaling data in the chip testing industry, if traditional multi-threaded processing technology is used, it is often necessary to introduce mutexes, read-write locks, spinlocks or semaphores to solve the problem of resource contention or data synchronization between multiple threads. If not handled carefully, deadlock bugs may be introduced.

[0006] Furthermore, most current multithreaded data processing methods in software development are limited to decomposing a large task into several homogeneous subtasks for parallel processing, but they are helpless when it comes to parallel processing of heterogeneous subtasks, meaning they can only improve the processing performance of some data. Once the number of different types of subtasks is unbalanced, scenarios will arise where some queued tasks are completed quickly, leaving threads running idly and consuming system resources, while other queued tasks accumulate, causing threads to be unable to keep up, ultimately leading to unstable program processing performance.

[0007] While remedial measures can be taken by combining business data analysis when the above situations occur, it requires creating more processing queues to balance the load for those subtask types that are numerous or time-consuming to process, which also requires creating more processing threads. This reactive approach has low CPU utilization, is complex and not universally applicable; and becomes inadequate once business needs change and the number of various subtasks changes as well. Summary of the Invention

[0008] To overcome the shortcomings of the prior art, this invention proposes a multi-threaded data processing method based on a custom condition queue, the method comprising:

[0009] Create a data processing class object and initialize the maximum number of child threads that can be created;

[0010] Create a custom double-ended conditional queue based on the maximum number of child threads that can be created;

[0011] The main thread calls the data processing class object to process the data, breaking it down into items to be processed one by one;

[0012] The main thread encapsulates the items to be processed, submits them to the queue, and assigns a child thread to process them.

[0013] The sub-thread retrieves an element from the double-ended condition queue;

[0014] The sub-thread processes the data within the element according to the element's task type.

[0015] Specifically, the phrase "creating a data processing class object and initializing the maximum number of child threads that can be created" includes:

[0016] Create the data processing class object;

[0017] Obtain the number of CPU cores in the system, and determine the maximum number of child threads that can be created based on the number of CPU cores;

[0018] Create a custom double-ended condition queue array based on the maximum number of child threads that can be created.

[0019] Specifically, the phrase "the main thread calls the data processing class object to process the data, and decomposes the items to be processed one by one" includes:

[0020] The main thread calls the data processing class object to process the data;

[0021] Determine whether a single item to be processed can be extracted from the data;

[0022] If a processing item can be extracted from the data, the processing item is encapsulated into a processing sub-unit, the processing sub-unit is delivered, and a processing sub-thread is allocated.

[0023] If a pending item cannot be extracted from the data, determine whether it is necessary to wait for all queue elements to be processed.

[0024] If it is necessary to wait for all queue elements to be processed, determine whether each queue is empty;

[0025] If none of the queues are empty, the queue load is balanced by round-robin.

[0026] If each queue is empty, determine whether the thread in each queue is idle;

[0027] If the threads in each of the queues are not idle, repeat the check to determine whether the threads in each of the queues are idle.

[0028] Specifically, the phrase "the main thread encapsulates the items to be processed, submits them to the queue, and allocates child threads for processing" includes:

[0029] Set the queue number used for traversing the queue to 0;

[0030] The number of queue elements is obtained based on the queue number;

[0031] Determine whether the number of elements in the queue is greater than 0;

[0032] If the number of queue elements is greater than 0, save the queue number with the fewest queue elements;

[0033] The queue number value is incremented by 1;

[0034] Determine whether the queue number is equal to the total number of queues;

[0035] If the queue number is equal to the total number of queues, push the sub-unit to be processed into the queue with the fewest elements. If the result is negative, repeat the step of "obtaining the number of queue elements according to the queue number".

[0036] Furthermore, it also includes:

[0037] If the number of elements in the queue is not greater than 0, determine whether there is a processing thread for that queue number in the thread pool;

[0038] If there is no processing thread for the queue number in the thread pool, a processing sub-thread is created for the queue, and the queue number is specified for the thread processing function. The processing sub-thread of the queue is included in the thread pool. According to the queue number, the sub-unit to be processed is pushed into the corresponding queue and the processing thread is woken up.

[0039] If the thread pool has a processing thread for the queue number, determine whether the queue processing thread is idle;

[0040] If the queue processing thread is idle, the sub-unit to be processed is pushed into the corresponding queue according to the queue number and the processing thread is woken up.

[0041] If the queue processing thread is not idle, execute the step of "saving the queue number with the fewest queue elements".

[0042] Furthermore, the "round-robin load balancing of the queue" includes:

[0043] Get the number of child threads in the thread pool;

[0044] Set the queue number used for traversing the queue to 0, and set the loop counter value to 1;

[0045] Based on the queue number, check whether the number of elements in the queue is greater than 1;

[0046] If the number of elements in the queue is not greater than 1, record the empty queue number and increment the queue number by 1;

[0047] If the number of queue elements is greater than 1, determine if there is an empty queue number;

[0048] If there is an empty queue number, remove an element to be adjusted from the queue, remove an empty queue number from the empty queue number record, push the element to be adjusted into the empty queue according to the empty queue number and wake up the processing thread, and increment the queue number by 1;

[0049] If there is no empty queue number, the queue number is incremented by 1;

[0050] After incrementing the queue number by 1, determine whether the queue number is equal to the number of child threads in the thread pool;

[0051] If the queue number is not equal to the number of child threads in the thread pool, execute the step of "checking whether the number of queue elements is greater than 1 based on the queue number";

[0052] If the queue number is equal to the number of child threads in the thread pool, determine whether the loop counter value is greater than 1;

[0053] If the loop count value is not greater than 1, increment the loop count value by 1, set the queue number to 0, and execute the step of "checking whether the number of queue elements is greater than 1 based on the queue number".

[0054] Furthermore, the phrase "the main thread calls the data processing class object to process the data, decomposing the items to be processed one by one" also includes:

[0055] The child thread accesses the queue based on the queue number;

[0056] The step of "delivering the sub-unit to be processed and assigning a sub-thread for processing" is achieved by waking up the processing thread and executing the step of "the sub-thread obtaining an element from the double-ended condition queue", thus obtaining the queue element.

[0057] Determine if the data is valid;

[0058] If the data is valid, the corresponding processing function is called according to the task type of the sub-unit to be processed obtained from the queue, the business-related data is calculated and stored, and the step of "the sub-thread accesses the queue according to the queue number" is repeated.

[0059] Specifically, the phrase "the sub-thread retrieves an element from the double-ended condition queue" includes:

[0060] Try to retrieve an element from the queue;

[0061] Determine if the queue is empty;

[0062] If the queue is not empty, remove one element from the queue;

[0063] If the queue is empty, determine whether the program requires a stop.

[0064] If the program does not require stopping, the queue enters a waiting state until the main thread pushes a new sub-unit to be processed into the queue and wakes up the processing thread.

[0065] Furthermore, the phrase "the sub-thread processes the data within the element according to the task type of the element" includes:

[0066] Determine the task type of the subunit to be processed;

[0067] Select the appropriate processing function based on the task type of the sub-unit to be processed.

[0068] The present invention also proposes a system for multi-threaded data processing based on a custom condition queue, the system being used to implement the method for multi-threaded data processing based on a custom condition queue as described above.

[0069] The present invention has at least the following beneficial effects:

[0070] Releasing CPU and lock resources when threads are idle improves system resource utilization, avoids the possibility of multiple threads competing for shared data, eliminates the time consumption of introducing lock mechanisms to synchronize data, and improves the utilization of child threads and program processing performance.

[0071] Furthermore, this invention supports free delivery queues, which improves the stability, versatility, and flexibility of the software architecture and achieves dynamic load balancing during operation.

[0072] Therefore, this invention provides a method and system for multi-threaded data processing based on a custom condition queue, which improves the performance of multi-threaded data processing. The method provided by this invention improves the utilization of system resources, enhances the scalability of the system in processing various types of data, avoids the possibility of multi-threaded contention for shared data, eliminates the time consumption caused by introducing a locking mechanism to synchronize data, realizes dynamic load balancing in the running state, and improves the stability, versatility and flexibility of the software architecture. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 A flowchart illustrating a multi-threaded data processing method based on a custom condition queue provided by the present invention;

[0075] Figure 2 A schematic diagram illustrating the creation and initialization process of data processing class objects;

[0076] Figure 3 This is a schematic diagram illustrating the structural relationship between queues and threads;

[0077] Figure 4 A flowchart illustrating the process of delivering sub-units to be processed;

[0078] Figure 5 A flowchart illustrating the process of handling the sub-units to be processed;

[0079] Figure 6 A flowchart illustrating the process of intelligent delivery and allocation of sub-threads;

[0080] Figure 7 A flowchart illustrating the process of polling and balancing the queue load;

[0081] Figure 8 A flowchart illustrating the process of a child thread retrieving an element from a double-ended condition queue;

[0082] Figure 9 This is a flowchart illustrating the process by which a child thread processes the data within an element based on its task type. Detailed Implementation

[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0084] This invention provides a method for multi-threaded data processing based on a custom condition queue. The method provided by this invention creates threads reasonably, has high resource utilization efficiency, uses few or no locks, and is relatively flexible and versatile.

[0085] Please refer to Figure 1 The method includes:

[0086] S100: Create a data processing class object, initialize the maximum number of child threads that can be created, and proceed to step S200.

[0087] The method proposed in this invention creates and initializes data processing class objects based on the number of CPU cores, which determines the upper limit of the number of threads created, thus avoiding wasting memory due to creating too many threads and wasting time due to frequent CPU switching.

[0088] By allocating a dedicated queue to each child thread, the possibility of multiple threads competing for shared data is avoided, and the time consumption caused by introducing a locking mechanism to synchronize data is eliminated.

[0089] S200: Create a custom double-ended condition queue based on the maximum number of child threads that can be created, and proceed to step S300.

[0090] This embodiment uses a custom double-ended condition queue. In other embodiments, a double-ended condition queue can be used, and when the queue is empty, the CPU usage is released through the yield function.

[0091] Assigning a dedicated queue to each child thread ensures that the data processing of each child thread is independent, avoiding contention for shared data between child threads. This eliminates the time overhead of introducing mutex locks to resolve shared data access conflicts. For the access relationships between the main thread, child threads, and queues, please refer to [reference needed]. Figure 2 .

[0092] S300: The main thread calls the data processing class object to process the data, breaking it down into items to be processed one by one.

[0093] When processing data, this invention supports decomposing items into processing items one by one according to business needs. These items can be various types of sub-data blocks or sub-tasks. Each item is de-differentiated and encapsulated into a processing sub-unit, which contains a source data pointer, processing type, and other fields.

[0094] The uniformly formatted sub-units to be processed can be delivered to the dedicated queues of each thread without being restricted by business differences. That is, the overall architecture of the program does not need to be adjusted due to changes in the amount of various business data. It is only necessary to distinguish and process them according to the queue element type in the processing function of the sub-thread. Even if it is necessary to add processing of new business data types, it is basically only necessary to add the relevant logic for the new processing type in the processing function of the sub-thread.

[0095] For processing sub-units, a real-time decomposition and intelligent delivery queue approach is adopted to trigger sub-threads to process queue elements in real time, allowing the sub-threads to start working as quickly as possible.

[0096] After submitting all pending sub-units, the main thread can either return directly or wait until all queue elements have been processed before returning. For the case where it needs to wait for all queue elements to be processed, the main thread will poll the dedicated queues of each sub-thread. When it finds a scenario with both free and backlogged queues, it will utilize the characteristic of its dedicated queues being double-ended conditional queues to perform load balancing.

[0097] S400: The main thread encapsulates the items to be processed, submits them to the queue, and assigns a child thread to process them.

[0098] Specifically, when delivering the sub-units to be processed, the main thread will intelligently deliver the sub-units based on the queue backlog and create sub-threads as needed to ensure load balancing across the queues as much as possible.

[0099] If the main thread returns immediately after submitting the pending sub-unit, the load balancing of the queue is achieved by the intelligent submission of the pending sub-units of subsequent requests; if the main thread waits for all queue elements to be processed before returning after submitting the pending sub-unit, the main thread will periodically poll the load of the dedicated queues of each sub-thread.

[0100] The main thread iterates through the dedicated queues of two child threads in a single poll, moving elements from the backlog queue to the empty queue for processing. This dynamically balances the load, enabling the reuse of idle child threads and improving the overall performance of the system.

[0101] S500: The child thread retrieves an element from the double-ended condition queue.

[0102] The child thread's dedicated queue is a custom double-ended condition queue constructed based on condition variables. The custom double-ended condition queue exists in the form of a class template and supports instantiation of the queue through custom types. Through the main thread's queue polling mechanism and the double-ended queue feature, dynamic load balancing is achieved in the running state, improving program processing performance.

[0103] After a child thread finishes processing the elements in its dedicated queue, the queue enters a waiting state, relinquishing CPU and lock resources until the main thread pushes a new child unit into the queue and wakes up the processing thread. Only then does the child thread regain CPU resources and retrieve an element from the queue.

[0104] S600: The child thread processes the data within the element according to the task type of the element.

[0105] When processing data, the main thread encapsulates multiple items into a uniformly formatted sub-unit. Sub-threads, when processing queue elements, perform targeted processing based on different processing types, which is the foundation of queue load balancing.

[0106] For details, please refer to Figure 3 Step S100 includes:

[0107] S110: Create a data processing class object, proceed to step S120.

[0108] S120: Obtain the number of CPU cores in the system, determine the maximum number of child threads that can be created, and proceed to step S130.

[0109] Specifically, the number of CPU cores is mCoreNum, and the maximum number of child threads that can be created is mThNum = mCoreNum - 1; if it is a hyper-threaded case, then the maximum number of child threads that can be created is mThNum = 2 * mCoreNum - 1.

[0110] S130: Create a custom double-ended condition queue array based on the maximum number of child threads that can be created.

[0111] Specifically, the number of elements in the custom double-ended conditional queue array is mQueNum = mThNum.

[0112] For details, please refer to Figure 4 Step S300 includes:

[0113] S310: Begin, proceed to step S311.

[0114] S311: The main thread calls the data processing class object to process the data, and proceeds to step S312.

[0115] S312: Determine whether a single item to be processed can be extracted from the data.

[0116] If a processable item can be extracted from the data, proceed to step S320; if a processable item cannot be extracted from the data, proceed to step S330.

[0117] S320: Encapsulate the item to be processed into a sub-unit to be processed, and proceed to step S321.

[0118] The sub-unit to be processed contains task type, sequence number, etc.

[0119] S321: Deliver the sub-unit to be processed and allocate the sub-thread for processing, then proceed to step S312.

[0120] It should be noted that step S321 is equivalent to step 400.

[0121] S330: Determine whether it is necessary to wait for all queue elements to be processed.

[0122] If the judgment result is yes, proceed to step S340; if the judgment result is no, proceed to step S360.

[0123] S340: Determine if each queue is empty.

[0124] If the judgment result is yes, proceed to step S350; if the judgment result is no, proceed to step S341.

[0125] S341: Poll the load balancing queue and repeat step S340.

[0126] S350: Determine whether the threads in each queue are idle.

[0127] If the judgment result is yes, proceed to step S360; if the judgment result is no, repeat step S350.

[0128] S360: End.

[0129] Further, please refer to Figure 5 Step S300 also includes:

[0130] S370: Begin, proceed to step S371.

[0131] S371: The child thread accesses the queue according to the queue number and proceeds to step S372.

[0132] S372: Obtain a queue element and proceed to step S373.

[0133] It should be noted that obtaining a queue element in step S372 is achieved by waking up the processing thread in step 321 and executing step 500.

[0134] S373: Determine if the data is valid.

[0135] If the data is valid, proceed to step S380; if the data is invalid, proceed to step S390.

[0136] S380: Process the data, proceed to step S371.

[0137] It should be noted that the data processing described in this embodiment includes: calling the corresponding processing function according to the task type of the sub-unit to be processed obtained from the queue, and performing business-related data calculation and storage.

[0138] S390: The thread exits and proceeds to step S391.

[0139] S391: End.

[0140] For details, please refer to Figure 6 Step S400 includes:

[0141] S410: Begin, proceed to step S411.

[0142] S411: Queue number, proceed to step S412.

[0143] In this embodiment, the queue number used for traversing the queue is set to 0, and the queues are numbered starting from 0.

[0144] S412: Obtain the number of queue elements based on the queue number, and proceed to step S413.

[0145] S413: Determine if the number of elements in the queue is greater than 0.

[0146] If the judgment result is yes, proceed to step S431; if the judgment result is no, proceed to step S420.

[0147] S420: Determine if there is a processing thread for the given queue number in the thread pool.

[0148] If the judgment result is yes, proceed to step S430; if the judgment result is no, proceed to step S440.

[0149] S430: Determine if the queue processing thread is idle.

[0150] If the judgment result is yes, proceed to step S442; if the judgment result is no, proceed to step S440.

[0151] S431: Save the queue number with the fewest queue elements, and proceed to step S432.

[0152] S432: Increment the queue number value by 1, proceed to step S433.

[0153] S433: Determine if the queue number is equal to the total number of queues.

[0154] It should be noted that the total number of queues is mQueMum.

[0155] If the judgment result is yes, proceed to step S434; if the judgment result is no, proceed to step S412.

[0156] S434: Push the sub-unit to be processed into the queue with the fewest elements.

[0157] S440: Create a child thread for processing the queue and specify the queue number for the thread processing function, then proceed to step S441.

[0158] S441: Add the queue processing sub-thread to the thread pool and proceed to step S442.

[0159] S442: Based on the queue number, push the sub-unit to be processed into the corresponding queue and wake up the processing thread, then proceed to step S443.

[0160] S443: End.

[0161] This invention, by encapsulating the differences in data items to be processed, supports a flexible delivery queue. Differential processing based on the processing type is only required within the sub-thread processing function, improving the stability, versatility, and flexibility of the software architecture. Furthermore, the intelligent delivery queue element enhances the utilization rate of sub-threads, thereby improving program processing performance.

[0162] Furthermore, step S400 also includes polling and balancing the queue load; please refer to [reference needed]. Figure 7 The step of polling and balancing the queue load includes:

[0163] S450: Begin, proceed to step S451.

[0164] S451: Get the number of child threads in the thread pool, then proceed to step S452.

[0165] S452: Set the queue number used for traversing the queue to 0, set the loop counter value to 1, and proceed to step S453.

[0166] S453: Check if the number of queue elements is greater than 1 based on the queue number.

[0167] If the number of queue elements is greater than 1, proceed to step S460; if the number of queue elements is not greater than 1, proceed to step S470.

[0168] S460: Determine if there is an empty queue number.

[0169] If the judgment result is yes, proceed to step S461; if the judgment result is no, proceed to step S471.

[0170] S461: Remove an element to be adjusted from the queue and proceed to step S462.

[0171] S462: Remove an empty queue number from the empty queue number record and proceed to step S463.

[0172] S463: Push the element to be adjusted into the empty queue according to the empty queue number and wake up the processing thread, then proceed to step S471.

[0173] S470: Record the empty queue number and proceed to step S471.

[0174] S471: Increment the queue number by 1, proceed to step S472.

[0175] S472: Determine if the queue number is equal to the number of child threads in the thread pool.

[0176] If the judgment result is yes, proceed to step S473; if the judgment result is no, proceed to step S453.

[0177] S473: Determine if the loop counter value is greater than 1.

[0178] If the loop count is greater than 1, proceed to step S490; if the loop count is not greater than 1, proceed to step S480.

[0179] S480: Increment the loop counter value by 1, set the queue number to 0, and proceed to step S453.

[0180] S490: End.

[0181] For details, please refer to Figure 8 Step S500 includes:

[0182] S510: Begin, proceed to step S520.

[0183] S520: Attempt to retrieve an element from the queue, proceed to step S530.

[0184] S530: Determine if the queue is empty.

[0185] If the queue is empty, proceed to step S540; if the queue is not empty, proceed to step S560.

[0186] S540: Determine if the program requires a stop.

[0187] If the judgment result is yes, proceed to step S570; if the judgment result is no, proceed to step S550.

[0188] S550: The queue enters a waiting state until the main thread pushes a new sub-unit to be processed into the queue and wakes up the processing thread, then proceeds to step S520.

[0189] It should be noted that after the queue enters the waiting state, it will relinquish central processing unit resources and lock resources accordingly, thereby improving the utilization rate of system resources.

[0190] S560: Remove one queue element and proceed to step S570.

[0191] S570: End.

[0192] For details, please refer to Figure 9 Step S600 includes:

[0193] S610: Begin, proceed to step S620.

[0194] S620: Determine the task type of the sub-unit to be processed, and proceed to step S630.

[0195] In this embodiment, it is assumed that the task types of the sub-unit to be processed include processing type 1, processing type 2 and processing type 3.

[0196] S630: Select the appropriate processing function according to the task type of the sub-unit to be processed, and proceed to step S640.

[0197] In this embodiment, when the type of the sub-unit to be processed is processing type 1, the process is transferred to processing function 1; when the type of the sub-unit to be processed is processing type 2, the process is transferred to processing function 2; and when the type of the sub-unit to be processed is processing type 3, the process is transferred to processing function 3.

[0198] S640: End.

[0199] The present invention also proposes a system for multi-threaded data processing based on a custom condition queue, which is used to implement the method for multi-threaded data processing based on a custom condition queue as described above.

[0200] In summary, this invention provides a method and system for multi-threaded data processing based on a custom condition queue, which improves the performance of multi-threaded data processing. The method provided by this invention improves the utilization rate of system resources, enhances the scalability of the system in processing various types of data, avoids the possibility of multi-threaded contention for shared data, eliminates the time consumption caused by introducing a locking mechanism to synchronize data, realizes dynamic load balancing in the running state, and improves the stability, versatility and flexibility of the software architecture.

[0201] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for multi-threaded data processing based on a custom condition queue, characterized in that, The method includes: Create a data processing class object and initialize the maximum number of child threads that can be created; Create a custom double-ended conditional queue based on the maximum number of child threads that can be created; The main thread calls the data processing class object to process the data, breaking it down into items to be processed one by one; The main thread encapsulates the items to be processed, submits them to the queue, and assigns a child thread to process them. The sub-thread retrieves an element from the double-ended condition queue; The sub-thread processes the data within the element according to the task type of the element; The phrase "creating a data processing class object and initializing the maximum number of child threads that can be created" includes: Create the data processing class object; Obtain the number of CPU cores in the system, and determine the maximum number of child threads that can be created based on the number of CPU cores; Based on the maximum number of child threads that can be created, create a custom double-ended conditional queue array; The phrase "the main thread calls the data processing class object to process the data, and decomposes the items to be processed one by one" includes: The main thread calls the data processing class object to process the data; Determine whether a single item to be processed can be extracted from the data; If a processing item can be extracted from the data, the processing item is encapsulated into a processing sub-unit, the processing sub-unit is delivered, and a processing sub-thread is allocated. If a pending item cannot be extracted from the data, determine whether it is necessary to wait for all queue elements to be processed. If it is necessary to wait for all queue elements to be processed, determine whether each queue is empty; If none of the queues are empty, the queue load is balanced by round-robin. If each queue is empty, determine whether the thread in each queue is idle; If the threads in each of the queues are not idle, repeat the check to see if the threads in each of the queues are idle. The phrase "the main thread encapsulates the items to be processed, submits them to the queue, and assigns a child thread to process them" includes: Set the queue number used for traversing the queue to 0; The number of queue elements is obtained based on the queue number; Determine whether the number of elements in the queue is greater than 0; If the number of queue elements is greater than 0, save the queue number with the fewest queue elements; The queue number value is incremented by 1; Determine whether the queue number is equal to the total number of queues; If the queue number is equal to the total number of queues, push the sub-unit to be processed into the queue with the fewest elements. If the result is negative, repeat the step of "obtaining the number of queue elements according to the queue number".

2. The method for multi-threaded data processing based on a custom condition queue according to claim 1, characterized in that, Also includes: If the number of elements in the queue is not greater than 0, determine whether there is a processing thread for that queue number in the thread pool; If there is no processing thread for the queue number in the thread pool, a processing sub-thread is created for the queue, and the queue number is specified for the thread processing function. The processing sub-thread of the queue is included in the thread pool. According to the queue number, the sub-unit to be processed is pushed into the corresponding queue and the processing thread is woken up. If the thread pool has a processing thread for the queue number, determine whether the queue processing thread is idle; If the queue processing thread is idle, the sub-unit to be processed is pushed into the corresponding queue according to the queue number and the processing thread is woken up. If the queue processing thread is not idle, execute the step of "saving the queue number with the fewest queue elements".

3. The method for multi-threaded data processing based on a custom condition queue according to claim 2, characterized in that, The "round-robin load balancing of the queue" includes: Get the number of child threads in the thread pool; Set the queue number used for traversing the queue to 0, and set the loop counter value to 1; Based on the queue number, check whether the number of elements in the queue is greater than 1; If the number of elements in the queue is not greater than 1, record the empty queue number and increment the queue number by 1; If the number of queue elements is greater than 1, determine if there is an empty queue number; If there is an empty queue number, remove an element to be adjusted from the queue, remove an empty queue number from the empty queue number record, push the element to be adjusted into the empty queue according to the empty queue number and wake up the processing thread, and increment the queue number by 1; If there is no empty queue number, the queue number is incremented by 1; After incrementing the queue number by 1, determine whether the queue number is equal to the number of child threads in the thread pool; If the queue number is not equal to the number of child threads in the thread pool, execute the step of "checking whether the number of queue elements is greater than 1 based on the queue number"; If the queue number is equal to the number of child threads in the thread pool, determine whether the loop counter value is greater than 1; If the loop count value is not greater than 1, increment the loop count value by 1, set the queue number to 0, and execute the step of "checking whether the number of queue elements is greater than 1 based on the queue number".

4. The method for multi-threaded data processing based on a custom condition queue according to any one of claims 1-3, characterized in that, The phrase "the main thread calls the data processing class object to process the data, and decomposes the items to be processed one by one" also includes: The child thread accesses the queue based on the queue number; The step of "delivering the sub-unit to be processed and assigning a sub-thread for processing" is achieved by waking up the processing thread and executing the step of "the sub-thread obtaining an element from the double-ended condition queue" to obtain the queue element. Determine if the data is valid; If the data is valid, the corresponding processing function is called according to the task type of the sub-unit to be processed obtained from the queue, the business-related data is calculated and stored, and the step of "the sub-thread accesses the queue according to the queue number" is repeated.

5. The method for multi-threaded data processing based on a custom condition queue according to claim 1, characterized in that, The phrase "the sub-thread retrieves an element from the double-ended condition queue" includes: Try to retrieve an element from the queue; Determine if the queue is empty; If the queue is not empty, remove one element from the queue; If the queue is empty, determine whether the program requires a stop. If the program does not require stopping, the queue enters a waiting state until the main thread pushes a new sub-unit to be processed into the queue and wakes up the processing thread.

6. The method for multi-threaded data processing based on a custom condition queue according to claim 5, characterized in that, The phrase "the sub-thread processes the data within the element according to the task type of the element" includes: Determine the task type of the subunit to be processed; Select the appropriate processing function based on the task type of the sub-unit to be processed.

7. A multi-threaded data processing system based on a custom condition queue, characterized in that, The system is used to implement the multi-threaded data processing method based on a custom condition queue as described in any one of claims 1-6.

Images