Task concurrent processing method and terminal

Abstract

The invention discloses a task concurrent processing method and a terminal, and the method comprises the steps: carrying out the task type discrimination of all to-be-processed tasks, and carrying out the concurrent processing of different types of tasks; and for tasks of the same type and when the unique identifiers of the tasks are consistent, a serial processing mode in the message queue is adopted to carry out serial processing on the tasks of the situation. Through the mode, the integrity of the data can be ensured and the processing capability of the data can be further improved through a mode of not adopting a distributed lock in a high-concurrency scene.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and terminal for concurrent task processing. Background Art

[0002] To support massive amounts of user operations and data activity, most server-side terminals currently require high concurrency and high throughput performance. Furthermore, these requirements must be met while ensuring data integrity. To rapidly support data integrity in high-concurrency scenarios, distributed locking is essential. Prior to performing data operations, relevant resources are locked and unlocked only after the task has completed processing the data, allowing subsequent task requests to perform other data operations. However, the use of distributed locking inherently limits data processing capabilities in high-concurrency scenarios. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and terminal for concurrent task processing, which can ensure data integrity and further improve data processing capabilities in high-concurrency scenarios without adopting distributed locks.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for concurrent task processing, comprising the steps of: receiving a plurality of pending tasks within a preset time range, dividing the pending tasks according to task types to obtain task sets of different task types, and concurrently processing each task set; If there are first tasks with the same unique identifier in the task set, the first tasks are processed serially; otherwise, the pending tasks in the task set are processed concurrently.

[0005] In order to solve the above technical problems, another technical solution adopted by the present invention is: A terminal for concurrent task processing includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the above-mentioned method for concurrent task processing is implemented.

[0006] The beneficial effects of the present invention are as follows: first, all pending tasks are distinguished by task type, and different types of tasks are processed concurrently; and then, for tasks of the same type with identical unique identifiers, tasks in this case are processed serially using the serial processing method in the message queue. This approach ensures data integrity and further improves data processing capabilities in high-concurrency scenarios without using distributed locks. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A flowchart of a method for concurrent task processing according to an embodiment of the present invention; Figure 2 A schematic diagram of a terminal for concurrent task processing according to an embodiment of the present invention; Description of labels: 1. A terminal that processes tasks concurrently; 2. Memory; 3. Processor. DETAILED DESCRIPTION

[0008] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0009] Please refer to Figure 1 , an embodiment of the present invention provides a method for concurrent task processing, comprising the steps of: receiving a plurality of pending tasks within a preset time range, dividing the pending tasks according to task types to obtain task sets of different task types, and concurrently processing each task set; If there are first tasks with the same unique identifier in the task set, the first tasks are processed serially; otherwise, the pending tasks in the task set are processed concurrently.

[0010] As can be seen from the above description, the beneficial effects of the present invention are as follows: first, all pending tasks are distinguished by task type, and different types of tasks are processed concurrently; and when tasks of the same type and with the same unique task identifier are processed serially using the serial processing method in the message queue. This approach can ensure data integrity and further improve data processing capabilities in high-concurrency scenarios without using distributed locks.

[0011] Furthermore, the tasks to be processed are divided according to task types to obtain task sets of different task types, including: Acquire a processing data type of the task to be processed, and set a task type of the task to be processed according to the processing data type; Multiple pending tasks are classified according to task types, and pending tasks of the same task type are stored in a task set to obtain task sets of different task types.

[0012] As can be seen from the above description, the task type is determined by the processing data type of the to-be-processed task, so that the to-be-processed tasks of the same task type are stored in the task set, so that different types of tasks process different types of data.

[0013] Furthermore, we obtain a set of tasks of different task types, which include: Sending the task set to a corresponding task type area of a message queue according to the task type; Concurrently process pending tasks of different task types, including: The task sets of each task type area in the message queue are consumed by multithreading.

[0014] From the above description, it can be seen that for task sets of different task types, they are assigned to different task type areas of the message queue, and concurrent processing can be quickly achieved through multi-threaded concurrent consumption of the message queue.

[0015] Furthermore, if there is a first task with the same unique identifier in the task set, the above includes: Hash the unique ID of each pending task in the task collection.

[0016] As can be seen from the above description, by performing hash processing on the unique identifier of the task to be processed, the security of the task processing process can be improved.

[0017] Furthermore, if there are first tasks with the same unique identifier in the task set, processing the first tasks serially includes: If there is a first task with the same unique identification hash value in the task set, the first task is stored in a preset data storage and distribution unit of the message queue, and the preset data storage and distribution unit corresponds to one thread processing.

[0018] From the above description, it can be seen that if there are pending tasks of the same task type and the same unique identification hash value within the preset time range, these tasks with the same hash value will be placed in the preset data storage and distribution unit of the same message queue to ensure that these tasks with the same hash value will only be consumed by one thread in the preset data storage and distribution unit. Compared with the locking processing method of the existing technology, it is possible to avoid locking the tasks and at the same time ensure a certain amount of concurrent task processing.

[0019] Please refer to Figure 2 Another embodiment of the present invention provides a terminal for concurrent task processing, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the various steps of the above-mentioned method for concurrent task processing.

[0020] The method and terminal for concurrent task processing described above in the present invention are suitable for performing task processing in high-concurrency scenarios without using distributed locks. The following is an explanation of the specific implementation methods: Please refer to Figure 1 , embodiment 1 of the present invention is: A method for concurrent task processing, comprising the steps of: S1. Receive multiple tasks to be processed within a preset time range, divide the tasks to be processed according to task types to obtain task sets of different task types, and process each task set concurrently.

[0021] In this embodiment, the preset time range can be set as a unit time.

[0022] Among them, the processing data type of the task to be processed is obtained, the task type of the task to be processed is set according to the processing data type, multiple tasks to be processed are classified according to the task type, and the tasks to be processed of the same task type are stored in a task set to obtain a task set of different task types.

[0023] The task set is sent to a corresponding task type area of a message queue according to the task type, and the task set of each task type area in the message queue is consumed by multiple threads.

[0024] Specifically, all tasks need to be differentiated by type, with different tasks identified by task type. Assuming that, under the existing approach, 100 tasks arrive per unit time, only one task will be processed at a time using a distributed lock. However, in this embodiment, distributed locks are no longer used for locking; only tasks processed per unit time are processed for subsequent processing.

[0025] Therefore, in this embodiment, if 100 tasks arrive per unit time, the task types are first distinguished. Different tasks process different data types. All different types of tasks can be sent to the task type area of the message queue. The task type area can be a preset partition or group. Subsequently, multiple threads can concurrently consume messages from this partition or group. Partitions or groups that receive tasks of different types can also process tasks in parallel.

[0026] S2. If there is a first task with the same unique identifier in the task set, the first task is processed serially; otherwise, the pending tasks in the task set are processed concurrently.

[0027] Among them, the unique identifier of each task to be processed in the task set is hashed. If there is a first task with the same unique identifier hash value in the task set, the first task is stored in the preset data storage and distribution unit of the message queue, and the preset data storage and distribution unit corresponds to a thread processing.

[0028] Specifically, when a partition or group receives tasks of the same type, each task will carry a unique primary key for its own data, such as the task ID, data ID, account ID, organization ID, and so on, creating a unique data identifier. Within each partition or group, there is a minimum data storage and distribution unit. For example, RabbitMQ's minimum data storage and distribution unit is the topic, and Kafka's minimum data storage and distribution unit is the partition. If tasks of the same type occur within a unit of time, hashing based on the unique data identifier will result in the same unique data task being processed only within the minimum data storage and distribution unit in the same message queue. Because the minimum data storage and distribution unit in a message queue can only be consumed by one thread, data with the same unique data identifier at the same time is guaranteed to be consumed and processed by only one thread.

[0029] Therefore, in this embodiment, the distinction is made based on the unique identifier in each task. If it is a task to process the same data, it will be assigned to the same data storage and distribution unit through the message queue mechanism. This unit can only execute tasks serially, while multiple units can execute tasks concurrently, and messages are sent to different units, which distinguishes the tasks that need to be locked. The underlying mechanism of MQ is used to avoid locking in tasks. Therefore, the underlying principle of the message queue is adopted, and the task can be processed without locking. At the same time, a certain amount of concurrent task processing can be guaranteed. In this way, the distributed lock method can be avoided. In high-concurrency scenarios, the integrity of the data can be guaranteed and the data processing capability can be further improved.

[0030] Please refer to Figure 2 , the second embodiment of the present invention is: A terminal 1 for concurrent task processing includes a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, each step of a method for concurrent task processing in embodiment 1 is implemented.

[0031] In summary, the method and terminal for concurrent task processing provided by the present invention adopt a non-distributed lock architecture to solve the problem in the prior art of processing tasks by using distributed locks, which limits the data processing capability in high-concurrency scenarios. First, the task types are distinguished for all tasks. When a task arrives, the locking operation of the distributed lock is no longer performed. Concurrent processing is performed according to the task type. When there are tasks of the same type within a unit cycle and the primary key fields in the tasks are also consistent, the serial processing method in the message queue is used to serially process tasks of this type. The remaining tasks can quickly realize the concurrent processing of concurrent tasks through multi-threaded concurrent consumption of the message queue. In this way, it is possible to ensure the integrity of the data in high-concurrency scenarios without using distributed locks, and further improve the data processing capability.

[0032] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for concurrent task processing, characterized in that: Including steps: receiving a plurality of pending tasks within a preset time range, dividing the pending tasks according to task types to obtain task sets of different task types, and concurrently processing each task set; If there are first tasks with the same unique identifier in the task set, the first tasks are processed serially; otherwise, the pending tasks in the task set are processed concurrently.

2. The method for concurrent task processing according to claim 1, wherein: The tasks to be processed are divided according to task types to obtain task sets of different task types, including: Acquire a processing data type of the task to be processed, and set a task type of the task to be processed according to the processing data type; Multiple pending tasks are classified according to task types, and pending tasks of the same task type are stored in a task set to obtain task sets of different task types.

3. The method for concurrent task processing according to claim 1, wherein: Get a collection of tasks of different task types, including: Sending the task set to a corresponding task type area of a message queue according to the task type; Concurrently process pending tasks of different task types, including: The task sets of each task type area in the message queue are consumed by multithreading.

4. The method for concurrent task processing according to claim 1, wherein: If there is a first task with the same unique identifier in the task set, the preceding steps include: Hash the unique ID of each pending task in the task collection.

5. The method for concurrent task processing according to claim 4, characterized in that: If there are first tasks with the same unique identifier in the task set, processing the first tasks serially includes: If there is a first task with the same unique identification hash value in the task set, the first task is stored in a preset data storage and distribution unit of the message queue, and the preset data storage and distribution unit corresponds to one thread processing.

6. A terminal for concurrent task processing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: receiving a plurality of pending tasks within a preset time range, dividing the pending tasks according to task types to obtain task sets of different task types, and concurrently processing each task set; If there are first tasks with the same unique identifier in the task set, the first tasks are processed serially; otherwise, the pending tasks in the task set are processed concurrently.

7. The terminal for concurrent task processing according to claim 6, characterized in that: The tasks to be processed are divided according to task types to obtain task sets of different task types, including: Acquire a processing data type of the task to be processed, and set a task type of the task to be processed according to the processing data type; Multiple pending tasks are classified according to task types, and pending tasks of the same task type are stored in a task set to obtain task sets of different task types.

8. The terminal for concurrent task processing according to claim 6, characterized in that: Get a collection of tasks of different task types, including: Sending the task set to a corresponding task type area of a message queue according to the task type; Concurrently process pending tasks of different task types, including: The task sets of each task type area in the message queue are consumed by multithreading.

9. The terminal for concurrent task processing according to claim 6, characterized in that: If there is a first task with the same unique identifier in the task set, the preceding steps include: Hash the unique ID of each pending task in the task collection.

10. The terminal for concurrent task processing according to claim 9, characterized in that: If there are first tasks with the same unique identifier in the task set, processing the first tasks serially includes: If there is a first task with the same unique identification hash value in the task set, the first task is stored in a preset data storage and distribution unit of the message queue, and the preset data storage and distribution unit corresponds to one thread processing.

Images