A method for cluster parallel processing of multiple tasks

Abstract

The present invention discloses a method for cluster parallel processing of multiple tasks. Application programs are deployed in a cluster. Multiple applications simultaneously receive a main task submitted by an external request and split the main task into N subtasks for parallel execution. Finally, the subtasks themselves submit the final processed result data of the subtasks to a unified external storage. The submitted subtask status data is modulo a fixed value K according to the main task ID to obtain a partition number, and the result data is accompanied by the partition number. The status data of the N subtasks of the same main task correspond to the same partition, and the total number of partitions is K. Finally, the cluster master node creates K threads to concurrently process the data in the K partitions, judges whether all the subtasks of each main task have submitted their own processing results, and calls back the main task whose subtasks are all completed, that is, notifies the task submitter that the task has been completed. The present invention effectively improves the concurrent processing and throughput efficiency of tasks in the application of the task collection and distribution type working principle.

Description

Technical Field

[0001] The present invention is applicable to the field of software development, and in particular relates to a method for cluster parallel processing of multiple tasks. Background Art

[0002] In a large system, there is usually a "task center" for executing various tasks. A task may be divided into multiple subtasks, each of which completes part of the work of the main task, and finally the results of the subtasks are processed together to produce the processing results of the main task. For example, in the field of financial credit, when a credit request is received, the request only contains the characteristic data of some users, such as the user's mobile phone number and ID card information. When the data is not sufficient to support the credit decision, the system needs to associate other data based on the existing characteristic values, which is a "data expansion" task. When the task is submitted, it may be necessary to associate three data sources A, B, and C at one time. The traditional approach is to start three threads to execute the subtask association data in parallel, and these three tasks will insert the status into the external storage after the task is completed, and use a scheduled task to scan the external storage to determine whether all subtasks have been completed, and call back the completed main task.

[0003] However, when deployed in a cluster, the above-mentioned scheduled task approach will have design flaws, because each node scans the external storage. If the same batch of status data is scanned by multiple application nodes, the same batch of data will be processed multiple times, resulting in repeated callbacks and eventually causing system abnormalities. Therefore, how to determine whether all subtasks have been completed needs to be redesigned. Summary of the invention

[0004] In view of the above problems, the present invention proposes a method for cluster parallel processing of multiple tasks to improve the efficiency of split sub-task processing and the efficiency of main task overall progress perception, thereby effectively improving the throughput of the system.

[0005] The objective of the present invention is achieved through the following technical solution: a method for cluster parallel processing of multiple tasks, specifically comprising the following steps:

[0006] (1) Receiving the main task: The application is deployed in the cluster. As a node in the cluster, every node in the cluster can be called by external requests and submit the main task. The main task contains a unique primary key: the main task ID.

[0007] (2) Calculate the partition number and execute subtasks in parallel: After receiving the main task request, any node in the cluster first calculates the partition to which the main task processing status data is finally stored based on the main task ID in step (1) and the number of data partitions K configured when the application is started. The specific calculation formula is:

[0008] Partition number = |hash(main task ID)%K|

[0009] In the formula, % represents the modulo operation. Hash generally refers to a hash algorithm that can calculate the main task ID as a numerical value.

[0010] The node will then split the main task into N subtasks and start N threads to execute these N subtasks in parallel.

[0011] (3) After the subtask is completed, the status is reported to the specified partition: According to the partition number calculated in step (2), each subtask thread will store the final result of the subtask to the external storage middleware at the end of the subtask, regardless of whether the task is ultimately successful or failed, and the data will be marked with the partition number.

[0012] (4) The master node scans the total progress of the main task: Based on the data with partition numbers in step (3), the master node in the cluster will start K threads to scan the task status data of these K data partitions in parallel. The scanning work will call back the total results of the main task whose subtasks have been completed to the main task submitter in step (1), that is, the sender of the external request.

[0013] Furthermore, in step (1), the submitting of the main task is an asynchronous request.

[0014] Furthermore, in step (1), the main task ID can be a string type or a numeric type.

[0015] Furthermore, in step (2), the contents of the N subtasks split from the main task may be logic codes that can be executed in any order.

[0016] Furthermore, in step (3), data partitioning can be implemented specifically according to the type of external storage middleware of the application. In cache middleware, it can be implemented by concatenating the partition number with a unified key prefix. In relational databases, it can be implemented through a column "partition number" in the data table.

[0017] Furthermore, in step (4), when the master node is scanning the task, if the master node crashes for some reason, the cluster will elect a new master node, and the new master node will take over the work of the original master node and continue to perform the scanning work of step (4).

[0018] Beneficial effects of the present invention: The present invention provides an efficient solution for cluster parallel processing of multiple tasks, which effectively improves the concurrent processing and throughput efficiency of tasks in the application of task distribution working principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram of the application cluster (Master-Slave) deployment;

[0020] Figure 2 A schematic diagram of a cluster parallel processing multi-task implementation method of the present invention;

[0021] Figure 3 Implement logical partitioning diagrams for Aerospike or Redis;

[0022] Figure 4 Schematic diagram of implementing logical partitioning for traditional relational databases. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] The present invention provides a method for cluster parallel processing of multiple tasks, which specifically includes the following steps:

[0025] (1) Receiving the main task: The application is deployed in a cluster. As a node in the cluster, each node in the cluster can be called by an external request and submit the main task. Submitting the main task is an asynchronous request. The main task contains a unique primary key: the main task ID. The main task ID can be a string type or a numeric type.

[0026] (2) Calculate the partition number and execute subtasks in parallel: After receiving the main task request, any node in the cluster first calculates the partition to which the main task processing status data is finally stored based on the main task ID in step (1) and the number of data partitions K configured when the application is started. The specific calculation formula is:

[0027] Partition number = |hash(main task ID)%K|

[0028] In the formula, % represents modulo operation, hash refers to a hash algorithm that can calculate the main task ID as a value; the main task ID is a String object in the system, that is, string type data, and HashCode is a method of the String class, which can get the hash value of the string, that is, an integer value. "HashCode of the main task ID" in the formula can be understood as a way to convert a String string, that is, a main task ID, into an integer while maintaining its uniqueness.

[0029] After calculating the partition number, the node will split the main task into N subtasks and start N threads to execute the N subtasks in parallel. The content of the N subtasks split from the main task can be logic code executed in any order. The status data of the N subtasks of the same main task corresponds to the same partition, and the total number of partitions is K.

[0030] (3) After the subtask is completed, the status is reported to the specified partition: Data partitioning can be implemented according to the type of external storage middleware of the application. According to the partition number calculated in step (2), each subtask thread will store the final result of the subtask to the external storage middleware at the end of the subtask, regardless of whether the task is ultimately successful or failed, and the data will be marked with the partition number.

[0031] (4) The master node scans the total progress of the main task: Based on the data with partition numbers in step (3), the master node in the cluster will start K threads to scan the task status data of these K data partitions in parallel. The scanning work will call back the total results of the main task whose subtasks have been completed to the main task submitter in step (1), that is, the sender of the external request.

[0032] When the master node is scanning, if it crashes for some reason, the cluster will elect a new master node, which will take over the work of the original master node and continue the scanning work in step (4).

[0033] A specific embodiment of the present invention is as follows:

[0034] like Figure 1 and 2 As shown, the present invention provides an efficient method for cluster parallel processing of multiple tasks, comprising the following steps:

[0035] 1. Use Spring Boot application framework to build web applications, and integrate ZooKeeper middleware to implement the cluster function of the application, so that multiple applications in the cluster can realize the election of master and slave nodes, and reverse proxy can be realized through Nginx middleware.

[0036] 2. After deploying the application according to step 1, use postman or jmeter tool to send a request to the Nginx proxy address to submit the main task.

[0037] 3. According to the main task submitted in step 2, the cluster will process the submitted main task in steps:

[0038] a. Submit and receive the main task: A node can be called by an external request and submit the main task. The main task submission here is an asynchronous request. The main task result is actively called back to the submitter by the main task processor after judging that the main task is completed.

[0039] b. Calculate the partition number and execute subtasks in parallel: After receiving the main task request, any node of the application first calculates the partition to which the task processing status data is finally stored based on the main task ID in step a and the number of data partitions K configured when the application is started. The number of data partitions K can be selected as 8. The specific calculation formula is:

[0040] Partition number = |hash(main task ID)%K|

[0041] In the formula, % represents the modulo operation. Hash generally refers to a hash algorithm that can calculate the main task ID as a numerical value.

[0042] After calculating the partition number, the node will split the main task into N subtasks and start N threads to execute the N subtasks in parallel. The contents of the N subtasks split from the main task can be logic codes that can be executed in any order.

[0043] c. After the subtask is completed, report the status to the specified partition: According to the partition number calculated in step b, each subtask thread will store the final result of the subtask to the external storage middleware at the end of the subtask, regardless of whether the task is ultimately successful or failed, and the data itself will be accompanied by the partition number. The data partitioning here can be implemented according to the type of external storage middleware actually used. For example, cache middleware such as Redis and Aerospike can be implemented by concatenating the partition number with a unified key prefix, such as Figure 3 As shown. It can also be implemented in relational databases such as Oracle and MySQL through a column "partition number" in the data table, such as Figure 4 shown.

[0044] d. The master node scans the total progress of the main task: Based on the data with partition numbers in step c, the master node in the cluster will start K threads to scan the task status data of these K data partitions in parallel. The scanning work will call back the total results of the main task whose subtasks have all been completed to the main task submitter in step a, that is, the external request sender.

[0045] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for cluster parallel processing of multiple tasks, characterized in that: The specific steps include: (1) Receiving the main task: The application is deployed in a cluster. As a node in the cluster, each node in the cluster can be called by an external request and submit the main task. The submission of the main task is an asynchronous request. The main task contains a unique primary key: main task ID; (2) Calculate the partition number and execute subtasks in parallel: After receiving the main task request, any node in the cluster first calculates the partition to which the main task processing status data is finally stored based on the main task ID in step (1) and the number of data partitions K configured when the application is started. The specific calculation formula is: Partition number = |hash(main task ID)%K| In the formula, % represents the modulo operation, and hash refers to the hash algorithm that can calculate the main task ID as a numerical value; The node will then split the main task into N subtasks and start N threads to execute the N subtasks in parallel; the contents of the N subtasks split from the main task can be logic codes executed in any order; the status data of the N subtasks of the same main task correspond to the same partition; (3) After the subtask is completed, the status is reported to the specified partition: According to the partition number calculated in step (2), each subtask thread will store the final result of the subtask to the external storage middleware at the end of the subtask, regardless of whether the task is ultimately successful or failed, and the data will be marked with the partition number; Data partitioning can be implemented based on the type of external storage middleware of the application. For cache middleware, it can be implemented by concatenating the partition number with a unified key prefix. For relational databases, it can be implemented by a column "partition number" in the data table. (4) The master node scans the total progress of the main task: Based on the data with partition numbers in step (3), the master node in the cluster will start K threads to scan the task status data of these K data partitions in parallel; the scanning work will call back the total result of the main task whose subtasks have been completed to the main task submitter in step (1), that is, the sender of the external request.

2. The method for cluster parallel processing of multiple tasks according to claim 1, characterized in that: In step (1), the main task ID can be a string type or a numeric type.

3. The method for cluster parallel processing of multiple tasks according to claim 1, characterized in that: In step (4), when the master node is scanning the task, if the master node crashes for some reason, the cluster will elect a new master node, and the new master node will take over the work of the original master node and continue to perform the scanning work of step (4).

Images