Data processing method and device based on different-speed thread pool and electronic equipment
By employing a multi-threaded, rate-divided thread pool method using a balanced binary tree priority blocking queue in the batch thread pool, the problem of task delay in multi-batch data processing is solved, and task processing efficiency is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PEOPLE'S INSURANCE COMPANY OF CHINA
- Filing Date
- 2025-11-27
- Publication Date
- 2026-04-10
AI Technical Summary
In existing technologies, when processing multiple batches of data, task batches with fewer records need to wait for batches with more records to complete, resulting in delays for high-priority tasks and low processing efficiency.
The batch thread pool is converted into a multi-threaded one, the number of record thread pools is increased, and a balanced binary tree priority blocking queue based on array is used to allocate tasks to record thread pools with different processing speeds according to the number of records and their priorities.
This allows for the priority processing of task batches with fewer records, avoiding delays for high-priority tasks and improving data processing efficiency.
Smart Images
Figure CN121833155A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical fields of computer software, programming languages, etc., and in particular to a data processing method, apparatus and electronic device based on a rate-division thread pool. Background Technology
[0002] Thread pools play a role in task scheduling in data processing. By properly managing thread resources, performance optimization, load control, and reliability improvement can be achieved. They are an important means of handling large-scale concurrent tasks (such as batch processing, real-time computing, and network request processing). Summary of the Invention
[0003] This application aims to at least partially address one of the technical problems in the related art.
[0004] Therefore, the first objective of this application is to propose a data processing method based on a speed-based thread pool, which allocates each batch of tasks to a thread pool with different processing speeds according to the number of records in each batch. This allows batches of tasks with fewer records to be processed first when multiple batches of data are processed in a concentrated manner, without having to wait for the earlier, longer-running batches to complete, thus avoiding delays in high-priority tasks and improving task processing efficiency.
[0005] The second objective of this application is to propose a data processing device based on a rate-division thread pool.
[0006] The third objective of this application is to propose an electronic device.
[0007] The fourth objective of this application is to provide a computer-readable storage medium.
[0008] The fifth objective of this application is to provide a computer program product.
[0009] To achieve the above objectives, the first aspect of this application proposes a data processing method based on a rate-division thread pool, comprising: Based on the number of available threads in the batch thread pool and the first priority corresponding to each candidate batch in the batch thread pool, one or more target batches are obtained in descending order of the first priority, wherein each target batch contains a first number of records; Determine a second number of record thread pools in the list, and a sorting result of the second number of record thread pools, wherein the sorting result is determined by a third number of records contained in the record thread pool; For each target batch, a target record thread pool for processing the target batch is determined based on at least one of the first quantity, the first quantity threshold, the second quantity, the second quantity threshold, and the sorting result. Determine the second priority of each record in the target batch in the target record thread pool, and process the target batch in descending order of the second priority.
[0010] To achieve the above objectives, a second aspect of this application provides a data processing apparatus based on a rate-division thread pool, comprising: The acquisition module is used to acquire one or more target batches according to the number of available threads in the batch thread pool and the first priority corresponding to each candidate batch in the batch thread pool, in descending order of the first priority, wherein each target batch contains a first number of records; The first determining module is used to determine a second number of thread pools recorded in the list, and a sorting result of the second number of thread pools recorded, wherein the sorting result is determined by a third number of records contained in the thread pools recorded. The second determining module is used to determine, for each target batch, a target record thread pool for processing the target batch based on at least one of the first quantity, the first quantity threshold, the second quantity, the second quantity threshold, and the sorting result. The processing module is used to determine the second priority of each record in the target batch in the target record thread pool, and process the target batch in descending order of the second priority.
[0011] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the data processing method based on a rate-division thread pool as described in the first aspect embodiment.
[0012] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data processing method based on a rate-division thread pool as described in the first aspect embodiment.
[0013] To achieve the above objectives, a fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method based on a rate-division thread pool as described in the first aspect embodiment.
[0014] The data processing method, apparatus, and electronic device based on a rate-based thread pool provided in this application distribute multiple batch tasks to record thread pools with different processing speeds according to the number of records they contain, using a multi-threaded batch thread pool. Then, the processing order within each record in the record thread pool is determined based on the priority of each record in each batch, thus completing the data processing for that batch. Therefore, when processing multiple batches of data centrally, different record thread pools are used to process batch tasks containing different numbers of records. This allows batches with fewer records to be processed first and quickly, without waiting for earlier, longer-running batches to complete, avoiding delays for high-priority tasks and improving task processing efficiency.
[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0016] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a data processing method based on a rate-division thread pool provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a data processing device based on a rate-division thread pool, provided in an embodiment of this application. Detailed Implementation
[0017] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0018] In this application, a thread pool is a thread usage pattern. A thread pool can maintain multiple threads, waiting for a supervisor to assign concurrently executable tasks. Batch thread pools and record thread pools are two thread pool models designed based on task granularity in batch data processing scenarios.
[0019] A batch thread pool uses batches as the basic unit of processing. A batch may contain tens of thousands of records. The batch thread pool determines which batch is processed first and distributes the records in a batch to the record thread pool.
[0020] A Record Thread Pool, which uses a single record as the basic processing unit. The Record Thread Pool is used to receive specific record processing tasks from the Batch Thread Pool and perform business logic processing on the data.
[0021] In the prior art, both the Batch Thread Pool and the Record Thread Pool use a blocking queue implemented based on a linked list data structure, that is, the order of enqueueing and dequeueing in the queue is first in, first out. In addition, the Batch Thread Pool is a single-threaded thread pool, that is, at the same time, at most one task in the Batch Thread Pool can be processed. Due to these two reasons, the tasks in the Record Thread Pool will be processed in a first-in, first-out manner strictly according to the order (chronological order) of the tasks pushed by the Batch Thread Pool and the order of the threads allocated by the Record Thread Pool. For example, at time t1, operator A releases Batch 1, and the number of records in this batch is x; at time t2, operator B releases Batch 2, and the number of records in this batch is y. t1 < t2, that is, Batch 1 enters the Batch Thread Pool earlier than Batch 2. x >> y, that is, the number of records in Batch 1 is much larger than the number of records in Batch 2. When Batch 1 has not been processed yet, Batch 2 has also entered the Batch Thread Pool. In the prior art method, the y records in Batch 2 will wait until all the records in Batch 1 are processed before being processed, resulting in the waiting time of operator B far exceeding the expected processing time, and the data processing efficiency is extremely low.
[0022] To address this problem, the embodiments of this application provide a data processing method based on a speed-splitting thread pool, which changes the Batch Thread Pool from single-threaded to multi-threaded, increases the number of Record Thread Pools, and changes the blocking queues of the Batch Thread Pool and the Record Thread Pool from sequential blocking queues based on a linked list data structure to priority blocking queues based on an array implementing a balanced binary tree, so as to enable batch tasks with fewer records to obtain a higher priority, obtain CPU shard time first and be processed. Even if they enter the system after batches with more records, they can be processed before batches with more records, greatly reducing the waiting time of small batch tasks and improving data processing efficiency.
[0023] It should be noted that in this application, a speed-splitting thread pool refers to determining the speed of data processing corresponding to a Record Thread Pool according to the amount of pending data contained in different Record Thread Pools at a certain moment, so that batches can be assigned to Record Thread Pools with different speeds for data processing according to the speed sorting among multiple Record Thread Pools.
[0024] Next, the data processing method and device based on a speed-splitting thread pool according to the embodiments of this application will be described with reference to the accompanying drawings.
[0025] Figure 1 It is a schematic flowchart of a data processing method based on a speed-splitting thread pool provided by the embodiments of this application.
[0026] like Figure 1 As shown, the data processing method based on the rate-division thread pool includes the following steps: Step 101: Based on the number of available threads in the batch thread pool and the first priority corresponding to each candidate batch in the batch thread pool, obtain one or more target batches in descending order of the first priority.
[0027] Each target batch contains a first number of records.
[0028] In this embodiment, a batch thread pool can be defined, with multiple core threads configured, such as four, and a priority blocking queue based on an array-based balanced binary tree. Because some core threads in the batch thread pool may be executing task distribution at any given time, the number of available threads in the batch thread pool may be one or more, not exceeding the configured core thread count. Each available thread can then select a candidate batch as the target batch from the blocking queue in descending order of priority.
[0029] In this embodiment, the first priority of candidate batches in the batch thread pool can be determined based on the time they are pushed into the batch thread pool; the earlier the push, the higher the first priority. Alternatively, other methods can be used to determine the first priority, and this application does not limit this method.
[0030] Step 102: Determine the second number of thread pools recorded in the list, and the sorting result of the second number of thread pools recorded.
[0031] The sorting result is determined by the third number of records contained in the record thread pool. For example, if the sorting result is from smallest to largest, the fewer records to be processed in the record thread pool, the higher the record thread pool will rank in the sorting result.
[0032] In this embodiment, the list is a list of recorded thread pools, and the length of the list indicates the maximum number of recorded thread pools that can process data simultaneously. The second number of recorded thread pools in the list is the current number of recorded thread pools used for data processing, and this second number should not exceed the length of the list.
[0033] In this embodiment, the third quantity is the number of records to be processed in each record thread pool. The sorting result can be obtained by sorting the record thread pools from smallest to largest according to the third quantity, or it can be obtained by sorting the record thread pools from largest to smallest according to the third quantity. This application does not limit the comparison.
[0034] It should be noted that since the record thread pool is constantly processing the record data it contains, and the record thread pool may also receive new records distributed by the batch thread pool at any time, the third quantity is changing in real time. Therefore, the sorting result of all record thread pools in the list is also changing at all times, and the record thread pool can be added or removed from the sorting result.
[0035] Step 103: For each target batch, determine the target record thread pool for processing the target batch based on at least one of the first quantity, the first quantity threshold, the second quantity, the second quantity threshold, and the sorting result.
[0036] The first quantity threshold, which can be called the high-speed / low-speed dividing value, can be set empirically or dynamically using an algorithm to enable it to adapt to different task environments. This application does not limit this.
[0037] The second quantity threshold is the length of the list of recorded thread pools.
[0038] In this embodiment, when the first number of records in the target batch is less than or equal to a first quantity threshold, a record thread pool with fewer pending tasks can be selected as the target thread pool; alternatively, when the first number of records in the target batch is greater than the first quantity threshold, a record thread pool with more pending tasks can be selected as the target thread pool. This allows for batch processing to be distributed to different record thread pools according to the batch processing speed, achieving speed-based processing and effectively avoiding the problems of excessively long waiting times and low processing efficiency for small batches of tasks after large batches of tasks.
[0039] In this embodiment, it can be further determined whether a record thread pool already exists in the record thread pool list. If not, or if the number is less than the length of the record thread pool list, a new record thread pool can be generated as the target record thread pool. If the number of record thread pools has reached the length of the record thread pool list, the relationship between the first number and the first number threshold can be determined, and the record thread pool with the fewest pending tasks or the record thread pool with the most pending tasks in the current record thread pool list can be selected as the target record thread pool.
[0040] Optionally, if there are multiple target batches, the first target batch with the highest priority can be determined first.
[0041] Then, if the second quantity is less than the second quantity threshold, a new record thread pool is generated as the target record thread pool for the first target batch. The second quantity is incremented by 1, and the sorting result is updated.
[0042] In this embodiment, after generating a new record thread pool, the new record thread pool can be added to the record thread pool list, and the second quantity is incremented by 1. This prevents errors in judgment caused by the record thread pool quantity not being updated in time when selecting the target record thread pool in the next target batch. Because this new record thread pool can also serve as a candidate for selecting the target record thread pool in the next target batch, the sorting result of all record thread pools can be updated after the first quantity of records in the target batch are allocated to the record thread pool.
[0043] Next, the second target batch with the highest priority (excluding the first target batch) among multiple target batches can be identified. Based on the updated second batch quantity and sorting result, the target record thread pool for determining the second target batch is returned, and so on, until the target record thread pool for all target tasks is determined.
[0044] It should be noted that when determining the target record thread pool for the second target batch, the second quantity after increasing by 1 may still be less than the second quantity threshold. In this case, a new record thread pool can be generated as the target record thread pool. Alternatively, the second quantity may reach the second quantity threshold after increasing by 1. In this case, one of the record thread pools can be determined as the target record thread pool based on the updated sorting result.
[0045] Optionally, if the second quantity is greater than or equal to the second quantity threshold and the first quantity is less than or equal to the first quantity threshold, the thread pool with the smallest corresponding third quantity in the sorting results can be determined as the target thread pool.
[0046] Alternatively, if the second quantity is greater than or equal to the second quantity threshold, and the first quantity is greater than the first quantity threshold, the thread pool with the largest third quantity in the sorting results can be determined as the target record thread pool.
[0047] It should be noted that in some embodiments, the blocking queue in the recording thread pool has a capacity limit. If the capacity limit is exceeded, records cannot be distributed to the recording thread pool, which may cause data processing failures and affect processing efficiency. Therefore, in this application, the sum of the first quantity in the target batch and the third quantity of records to be processed in the recording thread pool can also be used to determine whether the capacity limit is exceeded, and to select the target recording thread pool.
[0048] Optionally, if the second quantity is greater than or equal to the second quantity threshold, and the first quantity is greater than the first quantity threshold, first determine the value obtained by adding each third quantity to the first quantity in the sorting result. Then, determine the record thread pool corresponding to the maximum value less than the third quantity threshold as the target record thread pool.
[0049] The third quantity threshold is the upper limit of the capacity of the blocking queue in the thread pool.
[0050] In this embodiment of the application, when the number of recorded thread pools has reached the length of the recorded thread pool list and the first number is greater than the first number threshold, i.e. the target batch is a large batch of tasks with a long processing time, the recorded thread pools that will not exceed the capacity limit when adding the first number of records to all recorded thread pools and have a large number of tasks to be processed are selected. This allows large batches of tasks to be concentrated in thread pools with low processing speed, without affecting the processing speed of small batches of tasks. It also avoids the situation where there are too many tasks to be processed in a single recorded thread pool, which would lead to a significant increase in the average latency and tail latency of task processing.
[0051] Step 104: Determine the second priority of each record in the target batch in the target record thread pool, and process the target batch in descending order of the second priority.
[0052] The second priority is the processing order of each record in a batch, which can be determined based on the importance of each record in the business logic, or it can be determined based on other rules. This application does not limit this.
[0053] It should be noted that the second priority of records within a batch reflects the order of all records within the batch and does not affect the first priority of the batch itself. When any batch is dispatched to the record thread pool for processing, the integrity of the batch should not be disrupted.
[0054] In this embodiment, CPU time can be automatically allocated in the thread pool according to the second priority of each record in the blocking queue. Records with higher priority will be processed first, while those with lower priority will be processed later.
[0055] In this embodiment, multiple batch tasks are distributed to record thread pools with different processing speeds based on the number of records they contain, using a multi-threaded batch thread pool. The processing order within each record thread pool is then determined according to the priority of each record in each batch, thus completing the data processing for that batch. Therefore, when processing multiple batches of data centrally, different record thread pools are used to process batch tasks with varying numbers of records. This allows batches with fewer records to be processed quickly and efficiently, without waiting for earlier, longer-running batches to complete, avoiding delays for high-priority tasks and improving overall task processing efficiency.
[0056] To implement the above embodiments, this application also proposes a data processing device based on a rate-division thread pool.
[0057] Figure 2 This is a schematic diagram of a data processing device based on a rate-division thread pool, provided in an embodiment of this application.
[0058] like Figure 2 As shown, the data processing device 20 based on a rate-division thread pool includes: The acquisition module 201 is used to acquire one or more target batches according to the number of available threads in the batch thread pool and the first priority corresponding to each candidate batch in the batch thread pool, in descending order of the first priority, wherein each target batch contains a first number of records; The first determining module 202 is used to determine the second number of thread pools recorded in the list, and the sorting result of the second number of thread pools recorded, wherein the sorting result is determined by the third number of records contained in the thread pools recorded. The second determining module 203 is used to determine, for each target batch, the target record thread pool for processing the target batch based on at least one of the first quantity, the first quantity threshold, the second quantity, the second quantity threshold, and the sorting result. Processing module 204 is used to determine the second priority of each record in the target batch in the target record thread pool, and process the target batch in descending order of the second priority.
[0059] Furthermore, in one possible implementation of this application embodiment, the second determining module 203 may specifically be used for: When there are multiple target batches, the first target batch with the highest priority is determined. If the second quantity is less than the second quantity threshold, a new record thread pool is generated as the target record thread pool for the first target batch. Increment the second quantity by 1 and update the sorting result; Among multiple target batches, identify the second target batch with the highest priority, excluding the first target batch; Based on the updated second quantity and sorting results, return the target record thread pool for determining the second target batch, and so on, until the target record thread pool for all target tasks is determined.
[0060] Furthermore, in one possible implementation of this application embodiment, the second determining module 203 can also be used for: If the second quantity is greater than or equal to the second quantity threshold, and the first quantity is less than or equal to the first quantity threshold, then the thread pool with the smallest corresponding third quantity in the sorting results is determined as the target record thread pool; or, If the second quantity is greater than or equal to the second quantity threshold, and the first quantity is greater than the first quantity threshold, then the thread pool with the largest third quantity in the sorting results is determined as the target record thread pool. Furthermore, in one possible implementation of this application embodiment, the second determining module 203 can also be used for: If the second quantity is greater than or equal to the second quantity threshold, and the first quantity is less than or equal to the first quantity threshold, then the thread pool with the smallest corresponding third quantity in the sorting results is determined as the target record thread pool; or, If the second quantity is greater than or equal to the second quantity threshold, and the first quantity is greater than the first quantity threshold, then the thread pool with the largest third quantity in the sorting results is determined as the target record thread pool.
[0061] Furthermore, in one possible implementation of this application embodiment, the second determining module 203 can also be used for: Determine the value obtained by adding the third quantity to the first quantity in the sorting result; The record thread pool corresponding to the maximum value that is less than the third threshold is determined as the target record thread pool.
[0062] It should be noted that the foregoing explanation of the data processing method embodiment based on the rate-division thread pool also applies to the data processing device based on the rate-division thread pool in this embodiment, and will not be repeated here.
[0063] In this embodiment, multiple batch tasks are distributed to record thread pools with different processing speeds based on the number of records they contain, using a multi-threaded batch thread pool. The processing order within each record thread pool is then determined according to its priority, thus completing the data processing for that batch. Therefore, when processing multiple batches of data centrally, different record thread pools are used to process batch tasks with varying numbers of records. This allows batches with fewer records to be processed quickly and efficiently, without waiting for earlier, longer-running batches to complete, avoiding delays for high-priority tasks and improving overall task processing efficiency.
[0064] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided in the foregoing embodiments. To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.
[0065] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.
[0066] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0067] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.
[0068] This application is intended to provide an implementation scheme for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.
[0069] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0070] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0071] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0072] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.
[0073] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0074] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0075] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0076] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. A data processing method based on a rate-division thread pool, characterized in that, Includes the following steps: Based on the number of available threads in the batch thread pool and the first priority corresponding to each candidate batch in the batch thread pool, one or more target batches are obtained in descending order of the first priority, wherein each target batch contains a first number of records; Determine a second number of record thread pools in the list, and a sorting result of the second number of record thread pools, wherein the sorting result is determined by a third number of records contained in the record thread pool; For each target batch, a target record thread pool for processing the target batch is determined based on at least one of the first quantity, the first quantity threshold, the second quantity, the second quantity threshold, and the sorting result. Determine the second priority of each record in the target batch in the target record thread pool, and process the target batch in descending order of the second priority.
2. The method according to claim 1, characterized in that, For each target batch, determining the target record thread pool for processing the target batch based on at least one of the first quantity, the first quantity threshold, the second quantity, the second quantity threshold, and the sorting result includes: If there are multiple target batches, the first target batch with the highest priority is determined; If the second quantity is less than the second quantity threshold, a new record thread pool is generated as the target record thread pool for the first target batch. Increment the second quantity by 1 and update the sorting result; Among the plurality of target batches, identify the second target batch with the highest first priority, excluding the first target batch; Based on the updated second quantity and sorting results, return the target record thread pool for determining the second target batch, and so on, until the target record thread pool for all target tasks is determined.
3. The method according to claim 2, characterized in that, Also includes: If the second quantity is greater than or equal to the second quantity threshold and the first quantity is less than or equal to the first quantity threshold, the record thread pool with the smallest corresponding third quantity in the sorting result is determined as the target record thread pool. or, If the second quantity is greater than or equal to the second quantity threshold, and the first quantity is greater than the first quantity threshold, then the thread pool with the largest corresponding third quantity in the sorting results is determined as the target thread pool.
4. The method according to claim 3, characterized in that, The step of determining the target record thread pool as the record thread pool with the largest corresponding third quantity in the sorting results when the second quantity is greater than or equal to the second quantity threshold and the first quantity is greater than the first quantity threshold includes: Determine the value obtained by adding each third quantity to the first quantity in the sorting result; The record thread pool corresponding to the maximum value that is less than the third threshold is determined as the target record thread pool.
5. A data processing device based on a rate-division thread pool, characterized in that, include: The acquisition module is used to acquire one or more target batches according to the number of available threads in the batch thread pool and the first priority corresponding to each candidate batch in the batch thread pool, in descending order of the first priority, wherein each target batch contains a first number of records; The first determining module is used to determine a second number of thread pools recorded in the list, and a sorting result of the second number of thread pools recorded, wherein the sorting result is determined by a third number of records contained in the thread pools recorded. The second determining module is used to determine, for each target batch, a target record thread pool for processing the target batch based on at least one of the first quantity, the first quantity threshold, the second quantity, the second quantity threshold, and the sorting result. The processing module is used to determine the second priority of each record in the target batch in the target record thread pool, and process the target batch in descending order of the second priority.
6. The apparatus according to claim 5, characterized in that, The second determining module is specifically used for: If there are multiple target batches, the first target batch with the highest priority is determined; If the second quantity is less than the second quantity threshold, a new record thread pool is generated as the target record thread pool for the first target batch. Increment the second quantity by 1 and update the sorting result; Among the plurality of target batches, identify the second target batch with the highest first priority, excluding the first target batch; Based on the updated second quantity and sorting results, return the target record thread pool for determining the second target batch, and so on, until the target record thread pool for all target tasks is determined.
7. The apparatus according to claim 6, characterized in that, The second determining module is further configured to: If the second quantity is greater than or equal to the second quantity threshold and the first quantity is less than or equal to the first quantity threshold, the record thread pool with the smallest corresponding third quantity in the sorting result is determined as the target record thread pool. or, If the second quantity is greater than or equal to the second quantity threshold, and the first quantity is greater than the first quantity threshold, then the thread pool with the largest corresponding third quantity in the sorting results is determined as the target thread pool.
8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes the computer execution instructions stored in the memory to implement the data processing method based on the rate-division thread pool as described in any one of claims 1-4.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method based on a rate-division thread pool as described in any one of claims 1-4.
10. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the data processing method based on a rate-division thread pool as described in any one of claims 1-4.