Main task processing method, device, server and storage medium

Abstract

The embodiments of the present application provide a main task processing method, device, server and storage medium, by constructing a first main task message, the content of the first main task message is the currently executable first main task, the subject of the first main task message is the state of the first main task, the first main task message is sent to the message middleware, and a second main task message is obtained from the message middleware, the content of the second main task message is the second main task, and the subject of the second main task message is the state of the second main task, and the second main task message is processed through a thread pool. The method is suitable for scheduling and processing main tasks under a cluster architecture, and by introducing the message middleware, the distribution, claiming and execution processes of the main tasks are decoupled, thereby improving the parallel processing capability and processing efficiency of the main tasks.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a main task processing method, device, server and storage medium. Background Art

[0002] Batch processing tasks, also called main tasks, refer to a group of tasks suitable for batch processing that are obtained by combining a series of subtasks according to certain rules. Since batch processing does not require manual intervention, it has the advantages of high resource utilization and high throughput, and is highly favored in business scenarios involving the processing of massive data, such as large banks. With the advent of the big data era, the original single-machine deployment system has gradually evolved into a distributed system, and the main task processing mechanism of the existing technology can no longer meet the processing needs of a large number of high-frequency main tasks.

[0003] Therefore, how to improve the parallel processing capability of the main task has become a technical problem that needs to be solved urgently in the prior art. Summary of the invention

[0004] The embodiments of the present application provide a main task processing method, device, server and storage medium to solve the problems of low verification efficiency and accuracy in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a main task processing method, including:

[0006] Constructing a first main task message, wherein the content of the first main task message is the currently executable first main task, and the subject of the first main task message is the state of the first main task;

[0007] Sending the first main task message to the message middleware;

[0008] Acquire a second main task message from the message middleware, wherein the content of the second main task message is the second main task, and the subject of the second main task message is the state of the second main task;

[0009] The second main task message is processed through the thread pool.

[0010] Optionally, constructing the first main task message includes:

[0011] According to the status of the main task, the main task data table is screened to obtain a target main task data table, in which the status of the main task in the target main task data table is an unstarted state or a subtask execution completed state;

[0012] According to the target main task data table, extracting the first main task from the shared database;

[0013] The first main task message is constructed according to the extracted first main task and its status in the main task data table.

[0014] Optionally, the processing of the second main task message by using a thread pool includes:

[0015] Allocate threads through the thread pool;

[0016] The second main task message is processed according to the thread.

[0017] Optionally, the processing the second main task message includes:

[0018] Extracting the second main task from the second main task message;

[0019] Determining a state of the second main task according to a subject of the second main task message;

[0020] The second main task is processed according to the state of the second main task.

[0021] Optionally, the state of the second main task includes an unstarted state or a subtask execution completed state, and the processing of the second main task according to the state of the second main task includes:

[0022] If the state of the second main task is not started, preprocessing the second main task;

[0023] If the state of the second main task is the subtask execution completion state, the second main task is post-processed.

[0024] Optionally, the preprocessing the second main task includes:

[0025] Splitting the second main task into at least two subtasks;

[0026] The state of the second main task in the main task data table is modified to a preprocessing completion state.

[0027] Optionally, the post-processing the second main task includes:

[0028] Summarizing the execution status of at least two subtasks obtained by splitting the second main task;

[0029] The status of the second main task in the main task data table is modified to a post-processing completion status.

[0030] Optionally, before processing the second main task message according to the thread, the method further includes:

[0031] Determine whether the threads that can be allocated by the thread pool meet the processing requirements of the second main task message;

[0032] If not, put the redundant second main task message back into the message middleware.

[0033] Optionally, before constructing the first main task message, the method further includes:

[0034] Determine a state of the distributed lock, the state comprising an occupied state and an unoccupied state;

[0035] If the state of the distributed lock is unoccupied, the distributed lock is acquired.

[0036] Optionally, after sending the first main task message to the message middleware, the method further includes:

[0037] Release the distributed lock.

[0038] In a second aspect, an embodiment of the present application provides a main task processing device, including:

[0039] A main task distribution module is used to construct a first main task message, wherein the content of the first main task message is the currently executable first main task, and the subject of the first main task message is the state of the first main task; and send the first main task message to the message middleware;

[0040] The main task claiming module is used to obtain the second main task message from the message middleware, the content of the second main task message is the second main task, and the subject of the second main task message is the status of the second main task; the second main task message is processed through the thread pool.

[0041] In a third aspect, an embodiment of the present application provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the main task processing method as described in the first aspect above is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the main task processing method as described in the first aspect above.

[0043] The main task processing method, device, server and storage medium provided in the embodiments of the present application construct a first main task message, the content of the first main task message is the currently executable first main task, the subject of the first main task message is the status of the first main task, the first main task message is sent to the message middleware, and a second main task message is obtained from the message middleware, the content of the second main task message is the second main task, and the subject of the second main task message is the status of the second main task, and the second main task message is processed through a thread pool. The method is suitable for scheduling and processing main tasks under a cluster architecture, and by introducing the message middleware, the distribution, claiming and execution processes of the main tasks are decoupled, thereby improving the parallel processing capability and processing efficiency of the main tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0045] Figure 2 A flowchart of the main task processing method provided in Example 1 of the present application;

[0046] Figure 3 A schematic diagram of the execution logic of the main task processing method provided in the second embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of the main task processing device provided in Embodiment 3 of the present application;

[0048] Figure 5 A schematic diagram of the structure of a server provided in Example 4 of the present application. DETAILED DESCRIPTION

[0049] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0050] First, the nouns involved in this application are explained as follows:

[0051] Main task: For ease of description, the batch task is called the main task, and each main task includes at least two subtasks.

[0052] Subtask: A single task obtained by splitting the main task is called a subtask.

[0053] The main idea of ​​the technical solution of this application: The embodiment of this application provides a technical solution for processing the main task, by introducing the message middleware, and the main task publisher sends the main task to the message middleware in the form of a message, and the main task claimant obtains the message from the message middleware and processes the main task, thereby realizing the decoupling of the distribution, claiming, and execution of the main task, which is conducive to better utilizing the characteristics of the cluster system, realizing highly parallel processing of the main task, and improving the processing efficiency of the main task. In addition, the technical solution of the embodiment of this application can also parameterize the number of claimed messages, so that the concurrency capacity of each instance (node) can be parameterized and dynamically adjusted, and has a strong scene adaptability.

[0054] The technical solution for main task processing provided in this embodiment is mainly applied to a cluster system. The cluster system is characterized by having multiple nodes (or multiple instances), each of which is an independent server running its own process. For example, Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application, such as Figure 1 As shown, it is assumed that the cluster system includes three nodes, namely node A, node B and node C. The message middleware is a software or service program independent of the three nodes. The three nodes can interact with the message middleware to realize message distribution (i.e. sending messages to the message middleware) or message claiming (i.e. retrieving messages from the message middleware), that is, each node can independently execute the main task processing method provided in this embodiment.

[0055] The introduction of message middleware decouples message distribution and message claiming in time and space, that is, messages distributed to the message middleware may not be claimed immediately, but may be claimed after a period of time. At the same time, messages distributed to the message middleware may be claimed by other nodes or by the current node, that is, the nodes that distribute and claim the same message may be the same or different. It can be understood that in a cluster system, only one node is responsible for message distribution at a time, and each node in the cluster system can claim messages. For example, in a task scheduling process, node A is responsible for message distribution, and nodes A, B, and C can all claim the message.

[0056] In order to ensure that only one node in the cluster system distributes messages at a time, in the embodiment of the present application, a timed patroller is set for each node, and a cache server D is introduced at the same time. The cache server D is used to store the status of the distributed lock. Each node can read the status of the distributed lock through its timed patroller and preempt the distributed lock. The node that preempts the distributed lock can be used as a message distribution node to distribute messages. Exemplarily, when the timed patroller A1 of node A reads that the status of the distributed lock is unoccupied, the timed patroller A1 modifies the status of the distributed lock to occupied, and node A can perform the execution process of message distribution. At this time, the status of the distributed lock read by the timed patroller B1 of node B and the timed patroller C1 of node C is occupied, and nodes B and C cannot distribute messages.

[0057] In addition, since the ultimate goal of the technical solution of the present application is to realize the scheduling and processing of the main task, the content of the message is the main task that needs to be processed. Therefore, the cluster system provided by the embodiment of the present application should also include a shared database E, which is used to store the main tasks. All nodes in the system have the authority to use the shared database E or process the main tasks in the shared database E.

[0058] In addition, the embodiment of the present application may also set an executor for each node in the system, and the executor may be divided into a pre-processing executor and a post-processing executor. After each node completes the claim of the message, the corresponding executor processes the main task in the message.

[0059] It should be noted that in the following embodiments, for the sake of distinction, the main task message sent when the main task is distributed is called the first main task message, and the main task message obtained when the main task is claimed is called the second main task message. Accordingly, the main task in the first main task message is called the first main task, and the main task in the second main task message is called the second main task.

[0060] Embodiment 1

[0061] Figure 2 This is a flow chart of the main task processing method provided in the first embodiment of the present application. The execution subject of this embodiment is any node in the system, such as Figure 1 Node A in Figure 2 As shown, the main task processing method of this embodiment includes:

[0062] S101, constructing a first main task message.

[0063] Message construction is the first step to complete message distribution. In this step, the first main task message is constructed based on the currently executable first main task as the content and the subject of the state of the first main task according to the message generation rules.

[0064] Among them, the message generation rules are determined by the protocol used by the message middleware. For example, if the message middleware is Kafka, the message should be constructed according to the message generation rules defined in the Kafka protocol to ensure the availability of the constructed information.

[0065] Exemplarily, in this step, the main task data table can be screened according to the status of the main task to obtain a target main task data table, in which the status of the main task in the target main task data table is either an unstarted state or a subtask execution completed state. According to the target main task data table, the first main task is extracted from the shared database, and the first main task message is constructed based on the extracted first main task and its status in the main task data table.

[0066] Among them, the status of the main task is determined according to the stage of the main task in the processing process. In this embodiment, the processing of the main task is divided into three stages, namely the main task preprocessing stage, the subtask execution processing stage and the main task postprocessing stage. The main task preprocessing stage is mainly responsible for splitting the main task into individual subtasks, the subtask execution processing stage is mainly responsible for executing each subtask, and the main task postprocessing stage is mainly responsible for summarizing the execution status of each subtask and forming a report. Accordingly, the status of the main task can be not started, preprocessing, preprocessing completed, subtask executing, subtask execution completed, postprocessing, postprocessing completed and completed. Among them, "not started" refers to the status of the main task that has not yet started processing, and "completed" refers to the status of the main task that has completed all three stages.

[0067] The main task data table, that is, the data list of the main task, is mainly used to record the main task's mark (such as number, name, etc.), status and other information.

[0068] The target main task data table is obtained by filtering and only includes information of main tasks that are in a not started state or a subtask execution completed state.

[0069] A shared database refers to a database used to store main tasks. The database is independent of each node, and all nodes in the system can access or operate the shared database.

[0070] It should be noted that the main task method of this embodiment is mainly applicable to the main task preprocessing stage and the main task postprocessing stage. Therefore, when screening the main task data table, in this step, the main focus is on the main tasks that are in the unstarted state or the subtask execution completed state, so as to select the main tasks that need to be preprocessed or post-processed.

[0071] After obtaining the target main task data table through screening, further, in this step, according to the records of the target main task data table, the corresponding first main task is extracted from the shared database, and according to the message generation rules, the first main task is used as the content of the message, and the status of the first main task is used as the subject of the message to realize the construction of the first main task message.

[0072] From the above analysis, it can be seen that only the node that seizes the distributed lock has the authority to distribute messages. Therefore, before S101, the method of this embodiment should also include: determining the state of the distributed lock, and if the state of the distributed lock is unoccupied, acquiring the distributed lock. That is, determining that the node that executes message distribution seizes the distributed lock.

[0073] Among them, the state of the distributed lock includes an occupied state and an unoccupied state. Assuming that the state of the distributed lock is digitized with a binary number, for example, 1 represents the occupied state and 0 represents the unoccupied state, and the digitized rules are pre-stored on each node, then in this step, the state value of the distributed lock can be stored through a cache server (such as Redis), and the state of the distributed lock can be determined by reading the state value of the distributed lock on the cache server, and then the read state value is compared with the digitized rules to determine the state of the distributed lock. If the state of the determined distributed lock is an unoccupied state, the distributed lock is acquired, that is, the state of the distributed lock is changed to an occupied state, and the message distribution process is started. In this step, the state value of the distributed lock can be periodically read on the cache server through a timed patroller and the distributed lock preemption work can be performed.

[0074] It should be noted that, when the screened target main task data table includes multiple main tasks, there are multiple first main task messages constructed in this step, and each first main task message corresponds to one first main task.

[0075] S102: Send the first main task message to the message middleware.

[0076] In this step, after S101, the generated first main task message is sent to the message middleware for consumption (message processing) by consumers (nodes in the system), completing the distribution of the first main task message. Since there may be multiple first main task messages generated at one time in S101, there may also be multiple first main task messages sent to the message middleware in this step.

[0077] Among them, the message middleware is a network application with functions such as receiving requests, saving data and sending data. The message middleware used in the embodiment of the present application can be any one of ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, and RocketMQ, and there is no limitation here.

[0078] Optionally, after S102, the method of this embodiment may further include: releasing the distributed lock.

[0079] S103: Obtain a second main task message from the message middleware.

[0080] In this step, the node in the system, as the message claimant, obtains the second main task message from the message middleware according to the pre-agreed logic. The content of the second main task message is the second main task, and the subject of the second main task message is the status of the second main task.

[0081] Among them, the pre-agreed logic is the rule for obtaining the second main task message. In one possible implementation, the message sender (node ​​in the system) can subscribe to the message in advance from the message middleware in accordance with the "publish-subscribe" mode. When a message is published to the message middleware, the message middleware sends a notification to the node that has subscribed to the message. Correspondingly, the node that receives the notification comes to obtain the second main task message from the message middleware. In another possible implementation, each node can also periodically access the message middleware through a timed patroller to determine whether the message middleware has unclaimed messages. When it is determined that there are unclaimed messages, the corresponding second main task message is obtained from the message middleware.

[0082] It should be noted that, since the claimer and publisher of the same message may be the same or different, the second main task message may be the same as or different from the first main task message, and there is no strict restriction here.

[0083] In addition, in order to fully utilize the data processing capability of each node, in this step, it is supported to obtain multiple second main task messages from the message middleware at one time to realize parallel processing of the second main task messages. The number of second main task messages obtained each time can be configured in advance or determined according to the current processing capability (such as the number of available threads).

[0084] S104: Process the second main task message through the thread pool.

[0085] In this step, since there may be multiple second main task messages, in order to avoid memory consumption caused by repeated creation and destruction of threads, a thread pool is pre-established for each node in this step. A certain number of threads are stored in the thread pool. During use, only a certain number of idle threads need to be obtained from the thread pool. Specifically, the required number of threads is determined according to the number of second main task messages, threads are allocated according to the determined number of threads, and tasks are established for each thread, so that each thread can realize parallel processing of the second main task device.

[0086] Among them, the essence of processing the second main task message is to process the second main task therein. To this end, it is necessary to first extract the second main task from the second main task message, and then determine the state of the second main task according to the subject of the second main task message. Finally, according to the state of the second main task, take corresponding measures to process the second main task accordingly.

[0087] Since the status of the second main task includes the not started status or the subtask execution completed status, accordingly:

[0088] (1) If the status of the second main task is not started, the second main task is preprocessed. Specifically, according to the actual situation of the second main task, the second main task is split into at least two subtasks, and the status of the second main task in the main task data table is modified to the preprocessing completed state. Of course, other operations can also be performed according to actual needs during the preprocessing process, which is not limited here.

[0089] (2) If the status of the second main task is the subtask execution completion status, the second main task is post-processed. Specifically, the execution status of at least two subtasks obtained by splitting the second main task is summarized, and the status of the second main task in the main task data table is modified to the post-processing completion status. Of course, other operations can also be performed according to actual needs during the post-processing process, which is not limited here.

[0090] In actual applications, different executors may be called to perform corresponding processing according to different topics of the second main task message.

[0091] Since the thread pool needs to allocate threads according to actual conditions, for example, the total number of threads in a thread pool is 20, if 12 of the threads are already occupied, the maximum number of threads that can be allocated is 8. Therefore, optionally, the method of this embodiment also includes: determining whether the number of threads that can be allocated by the thread pool meets the processing requirements of the second main task message, and if not, putting the excess second main task messages back into the message middleware.

[0092] For example, when the number of threads that can be allocated in a thread pool is 8, and there are currently 10 second main task messages that need to be processed, that is, the threads that can be allocated by the thread pool cannot meet the processing requirements of the second main task messages, the excess 2 main task messages will be put back into the message middleware to allow other nodes to claim and process the messages, thereby realizing dynamic control of parallel processing capabilities and ensuring maximum processing efficiency of the cluster system.

[0093] Optionally, the process of claiming a message may also include a process of acquiring and releasing a distributed lock, and its specific implementation method is similar to that of the message publishing process, which will not be described in detail here.

[0094] It is understandable that the above S101-S104 is only used as a possible implementation logic. In actual application, the above steps are likely to be performed alternately without a strict sequence. Therefore, the above numbers cannot be used as a strict execution sequence to limit the technical solution of the present invention.

[0095] In this embodiment, by constructing a first main task message, the content of the first main task message is the currently executable first main task, and the subject of the first main task message is the status of the first main task, the first main task message is sent to the message middleware, and the second main task message is obtained from the message middleware, the content of the second main task message is the second main task, and the subject of the second main task message is the status of the second main task, and the second main task message is processed through the thread pool, which is suitable for scheduling and processing main tasks under a cluster architecture, and by introducing the message middleware, the distribution, claiming and execution processes of the main tasks are decoupled, thereby improving the parallel processing capability and processing efficiency of the main tasks.

[0096] Embodiment 2

[0097] Figure 3 The execution logic diagram of the main task processing method provided in the second embodiment of the present application is based on the first embodiment. Figure 3 As shown, in this embodiment, according to the method in Embodiment 1, an abstract class of the main task timing patroller is designed, and the main task timing patroller abstract class mainly has the following functions:

[0098] (1) Realize the acquisition and allocation of distributed locks among multiple instances (nodes) of a cluster system;

[0099] (2) Realize the distribution of main task messages;

[0100] (3) Abstract the processing flow of the main task timing patroller;

[0101] (4) Abstract the interface for interacting with the shared database;

[0102] (5) Abstract the timer patroller task processing logic interface.

[0103] In this embodiment, according to the method in the first embodiment, an abstract class of the main task executor is designed. The main task executor abstract class mainly has the following functions:

[0104] (1) Realize the acquisition and allocation of distributed locks among multiple instances (nodes) of a cluster system.

[0105] (2) Realize the pulling of main task messages.

[0106] (3) Implement control of the main task concurrent thread pool.

[0107] (4) Abstract the executor task processing logic interface.

[0108] Embodiment 3

[0109] Figure 4 This is a schematic diagram of the structure of the main task processing device provided in Example 3 of the present application, such as Figure 4 As shown, the main task processing device 10 in this embodiment includes:

[0110] A main task distribution module 11 and a main task claiming module 12.

[0111] The main task distribution module 11 is used to construct a first main task message, the content of the first main task message is the currently executable first main task, and the subject of the first main task message is the state of the first main task; and send the first main task message to the message middleware;

[0112] The main task claiming module 12 is used to obtain the second main task message from the message middleware, the content of the second main task message is the second main task, and the subject of the second main task message is the status of the second main task; the second main task message is processed through the thread pool.

[0113] Optionally, the main task distribution module 11 is specifically used for:

[0114] According to the status of the main task, the main task data table is screened to obtain a target main task data table, in which the status of the main task in the target main task data table is an unstarted state or a subtask execution completed state;

[0115] According to the target main task data table, extracting the first main task from the shared database;

[0116] The first main task message is constructed according to the extracted first main task and its status in the main task data table.

[0117] Optionally, the main task claiming module 12 is specifically used for:

[0118] Allocate threads through the thread pool;

[0119] The second main task message is processed according to the thread.

[0120] Optionally, the main task claiming module 12 is specifically used for:

[0121] Extracting the second main task from the second main task message;

[0122] Determining a state of the second main task according to a subject of the second main task message;

[0123] The second main task is processed according to the state of the second main task.

[0124] Optionally, the state of the second main task includes an unstarted state or a subtask execution completed state, and the main task claiming module 12 is specifically used to:

[0125] If the state of the second main task is not started, preprocessing the second main task;

[0126] If the state of the second main task is the subtask execution completion state, the second main task is post-processed.

[0127] Optionally, the main task claiming module 12 is specifically used for:

[0128] Splitting the second main task into at least two subtasks;

[0129] The state of the second main task in the main task data table is modified to a preprocessing completion state.

[0130] Optionally, the main task claiming module 12 is specifically used for:

[0131] Summarizing the execution status of at least two subtasks obtained by splitting the second main task;

[0132] The status of the second main task in the main task data table is modified to a post-processing completion status.

[0133] Optionally, the main task claiming module 12 is further used for:

[0134] Determine whether the threads that can be allocated by the thread pool meet the processing requirements of the second main task message;

[0135] If not, put the redundant second main task message back into the message middleware.

[0136] Optionally, the main task distribution module 11 is further used for:

[0137] Determine a state of the distributed lock, the state comprising an occupied state and an unoccupied state;

[0138] If the state of the distributed lock is unoccupied, the distributed lock is acquired.

[0139] Optionally, the main task distribution module 11 is further used for:

[0140] Release the distributed lock.

[0141] The main task processing device provided in this embodiment can execute the main task processing method provided in the above method embodiment, and has the corresponding functional modules and beneficial effects of the execution method. The implementation principle and technical effects of this embodiment are similar to those of the above method embodiment, and will not be repeated here.

[0142] Embodiment 4

[0143] Figure 5 A schematic diagram of the structure of a server provided in Embodiment 4 of the present application, such as Figure 5 As shown, the server 20 includes a memory 21, a processor 22, and a computer program stored in the memory and executable on the processor; the number of the processor 22 in the server 20 may be one or more. Figure 5 A processor 22 is taken as an example; the processor 22 and the memory 21 in the server 20 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0144] The memory 21 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions corresponding to the main task distribution module 11 and the main task claiming module 12 in the embodiment of the present application. The processor 22 runs the software programs, instructions and modules stored in the memory 21, thereby implementing various functional applications and data processing of the server, that is, realizing the above-mentioned main task processing method.

[0145] The memory 21 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 21 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 21 may further include a memory remotely arranged relative to the processor 22, and these remote memories may be connected to the server via a grid. Examples of the above-mentioned grid include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] Embodiment 5

[0147] Embodiment 5 of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer processor, the computer program is used to execute a main task processing method, the method comprising:

[0148] Constructing a first main task message, wherein the content of the first main task message is the currently executable first main task, and the subject of the first main task message is the state of the first main task;

[0149] Sending the first main task message to the message middleware;

[0150] Acquire a second main task message from the message middleware, wherein the content of the second main task message is the second main task, and the subject of the second main task message is the state of the second main task;

[0151] The second main task message is processed through the thread pool.

[0152] Of course, the computer-readable storage medium provided in the embodiment of the present application, whose computer program is not limited to the method operations described above, can also execute the relevant operations in the main task processing method provided in any embodiment of the present application.

[0153] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present application can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or grid device, etc.) to execute the methods described in each embodiment of the present application.

[0154] It is worth noting that in the above-mentioned embodiment of the main task processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0155] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A main task processing method, characterized in that: include: Constructing a first main task message, wherein the content of the first main task message is the currently executable first main task, and the subject of the first main task message is the state of the first main task; Sending the first main task message to the message middleware; Acquire a second main task message from the message middleware, wherein the content of the second main task message is the second main task, and the subject of the second main task message is the state of the second main task; Processing the second main task message through the thread pool; The step of constructing the first main task message includes: According to the status of the main task, the main task data table is screened to obtain a target main task data table, in which the status of the main task in the target main task data table is an unstarted state or a subtask execution completed state; According to the target main task data table, extracting the first main task from the shared database; Constructing the first main task message according to the extracted first main task and its status in the main task data table; wherein the status of the main task includes: not started, preprocessing, preprocessing completed, subtask executing, subtask execution completed, postprocessing, postprocessing completed and completed; The processing of the second main task message by the thread pool includes: Allocate threads through the thread pool; extracting the second main task from the second main task message according to the thread; Determining a state of the second main task according to a subject of the second main task message; processing the second main task according to the state of the second main task; The state of the second main task includes an unstarted state or a subtask execution completed state, and the processing of the second main task according to the state of the second main task includes: If the state of the second main task is not started, split the second main task into at least two subtasks, and modify the state of the second main task in the main task data table to be a preprocessing completed state; If the status of the second main task is the subtask execution completion status, post-processing the second main task; Wherein, the constructing the first main task message further includes: The status value of the distributed lock is periodically read and the distributed lock is preempted through the timing patroller.

2. The method according to claim 1, characterized in that The post-processing of the second main task includes: Summarizing the execution status of at least two subtasks obtained by splitting the second main task; The status of the second main task in the main task data table is modified to a post-processing completion status.

3. The method according to claim 1, characterized in that Before processing the second main task message according to the thread, the method further includes: Determine whether the threads that can be allocated by the thread pool meet the processing requirements of the second main task message; If not, put the redundant second main task message back into the message middleware.

4. The method according to any one of claims 1 to 3, characterized in that: Before constructing the first main task message, the method further includes: Determine a state of the distributed lock, the state comprising an occupied state and an unoccupied state; If the state of the distributed lock is unoccupied, the distributed lock is acquired.

5. The method according to claim 3, characterized in that: After sending the first main task message to the message middleware, the method further includes: Release the distributed lock.

6. A main task processing device, characterized in that: include: A main task distribution module is used to construct a first main task message, wherein the content of the first main task message is the currently executable first main task, and the subject of the first main task message is the state of the first main task; and send the first main task message to the message middleware; A main task claiming module is used to obtain a second main task message from the message middleware, the content of the second main task message is the second main task, and the subject of the second main task message is the state of the second main task; and process the second main task message through a thread pool; The main task distribution module is specifically used to: screen the main task data table according to the status of the main task to obtain a target main task data table, in which the status of the main task in the target main task data table is an unstarted state or a subtask execution completed state; extract the first main task from the shared database according to the target main task data table; construct the first main task message according to the extracted first main task and its status in the main task data table; wherein the status of the main task includes: unstarted, preprocessing, preprocessing completed, subtask executing, subtask execution completed, postprocessing, postprocessing completed and completed; The main task claiming module is specifically used to: allocate threads through a thread pool; extract the second main task from the second main task message according to the threads; determine the state of the second main task according to the subject of the second main task message; and process the second main task according to the state of the second main task; The state of the second main task includes an unstarted state or a subtask execution completed state, and the main task claiming module is specifically used to: if the state of the second main task is an unstarted state, split the second main task into at least two subtasks; modify the state of the second main task in the main task data table to a pre-processing completed state; if the state of the second main task is a subtask execution completed state, post-process the second main task; Wherein, the constructing the first main task message further includes: The status value of the distributed lock is periodically read and the distributed lock is preempted through the timing patroller.

7. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the main task processing method as described in any one of claims 1-5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the main task processing method as described in any one of claims 1 to 5 is implemented.

Images