Task processing method and device, computer device, storage medium and program product

Abstract

The application relates to a task processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a first batch of tasks in a current polling process; performing initialization processing on the first batch of tasks in a task pool according to the first batch of tasks, and obtaining a plurality of first task groups; determining whether there is a to-be-processed task group according to a maximum thread number and polling identifiers of a plurality of second task groups of a second batch of tasks in a previous polling process; when it is determined that there is a to-be-processed task group, selecting the to-be-processed task group and at least one first task group from the task pool according to the maximum thread number; and calling a worker thread to process the to-be-processed task group and the at least one first task group. The maximum thread number is the number of the worker threads. The method can reduce a task processing failure rate.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a task processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the continuous development of computer technology, financial institutions have to handle a large number of tasks every day. For example, financial institutions need to schedule and process a series of transaction tasks, such as interest settlement and balance buffer accounting in the financial industry, in order to centrally manage transaction-related tasks.

[0003] However, if a large number of batch tasks are received in a short period of time, the server may experience interface blockage or even server crash when processing these tasks, resulting in task processing failure. Summary of the Invention

[0004] Therefore, it is necessary to provide a task processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can reduce the task processing failure rate in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a task processing method, the method comprising:

[0006] Retrieve the first batch of tasks in the current polling process;

[0007] Based on the first batch of tasks, the first batch of tasks are initialized in the task pool to obtain multiple first task groups;

[0008] Based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process, determine whether there are any pending task groups;

[0009] When it is determined that there is a group of tasks to be processed, the group of tasks to be processed and at least one first task group are selected from the task pool according to the maximum number of threads.

[0010] Worker threads are invoked to process the pending task group and the at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0011] In one embodiment, determining whether there is a pending task group based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process includes:

[0012] Based on the maximum number of threads, configure a polling flag for at least one second task group that was not polled in the previous polling process; the polling flag indicates that it was not polled.

[0013] Based on the polling identifier being "not polled", it is determined that there is a task group to be processed in the task pool; the task group to be processed is the at least one second task group.

[0014] In one embodiment, determining whether there is a pending task group based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process includes:

[0015] Based on the maximum number of threads, determine the multiple second task groups that were polled in the previous polling process;

[0016] If the execution result of at least one of the multiple second subtasks fails, the polling identifier of each corresponding second task group is determined to be polled but the execution result failed;

[0017] Based on the polling identifier of each of the corresponding second task groups, which indicates that the polling has failed, it is determined that there are task groups to be processed in the task pool; the task groups to be processed are the second task groups corresponding to the polled but failed task groups.

[0018] In one embodiment, determining the multiple second task groups polled in the previous polling process based on the maximum number of threads includes:

[0019] Obtain the priority of the plurality of second task groups;

[0020] Based on the priority of the plurality of second task groups, select the plurality of second task groups that have the same number of threads as the maximum number of threads.

[0021] In one embodiment, the initialization process of the first batch of tasks in the task pool to obtain multiple first task groups includes:

[0022] Get the task group configuration table;

[0023] Based on the task group configuration table and the first batch of tasks, the first batch of tasks are initialized in the task pool to obtain the plurality of first task groups.

[0024] In one embodiment, the worker thread includes a first worker thread and at least one second worker thread; the invocation of the worker thread to process the pending task group and the at least one first task group respectively includes:

[0025] The at least one second worker thread is invoked to process the at least one first task group respectively;

[0026] When at least one subtask in the pending task group fails to execute, the first worker thread is invoked to process the subtask that failed to execute, according to the execution order of the subtasks in the pending task group, until the pending task group is completed.

[0027] Secondly, this application provides a task processing apparatus, the apparatus comprising:

[0028] The acquisition module is used to acquire the first batch of tasks in the current polling process;

[0029] The grouping module is used to initialize the first batch of tasks in the task pool according to the first batch of tasks, and obtain multiple first task groups;

[0030] The determination module is used to determine whether there is a pending task group based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process.

[0031] The selection module is used to select, when it is determined that there is a task group to be processed, the task group to be processed and at least one first task group from the task pool according to the maximum number of threads.

[0032] The processing module is used to call worker threads to process the task group to be processed and the at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0033] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0034] Retrieve the first batch of tasks in the current polling process;

[0035] Based on the first batch of tasks, the first batch of tasks are initialized in the task pool to obtain multiple first task groups;

[0036] Based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process, determine whether there are any pending task groups;

[0037] When it is determined that there is a group of tasks to be processed, the group of tasks to be processed and at least one first task group are selected from the task pool according to the maximum number of threads.

[0038] Worker threads are invoked to process the pending task group and the at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0040] Retrieve the first batch of tasks in the current polling process;

[0041] Based on the first batch of tasks, the first batch of tasks are initialized in the task pool to obtain multiple first task groups;

[0042] Based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process, determine whether there are any pending task groups;

[0043] When it is determined that there is a group of tasks to be processed, the group of tasks to be processed and at least one first task group are selected from the task pool according to the maximum number of threads.

[0044] Worker threads are invoked to process the pending task group and the at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0045] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0046] Retrieve the first batch of tasks in the current polling process;

[0047] Based on the first batch of tasks, the first batch of tasks are initialized in the task pool to obtain multiple first task groups;

[0048] Based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process, determine whether there are any pending task groups;

[0049] When it is determined that there is a group of tasks to be processed, the group of tasks to be processed and at least one first task group are selected from the task pool according to the maximum number of threads.

[0050] Worker threads are invoked to process the pending task group and the at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0051] The aforementioned task processing method, apparatus, computer equipment, storage medium, and computer program product obtain the first batch of tasks in the current polling process. Based on the first batch of tasks, they initialize the first batch of tasks in a task pool to obtain multiple first task groups. Based on the maximum number of threads and the polling identifiers of multiple second task groups from the second batch of tasks in the previous polling process, they determine whether there are any pending task groups. By combining the polling identifiers of task groups from the previous polling process, they ensure that no task groups needing processing are missed in the current polling process, guaranteeing the integrity of the task processing results. When it is determined that there are pending task groups, based on the maximum number of threads, they select the pending task group and at least one first task group from the task pool, and call worker threads to process the pending task group and at least one first task group respectively. By setting the maximum number of threads, even if a large number of batch tasks are received in a short period, the number of tasks processed by worker threads in the current polling process can be limited, thereby avoiding interface blocking and server downtime issues and reducing the task processing failure rate. Attached Figure Description

[0052] Figure 1 This is an application environment diagram of a task processing method in one embodiment;

[0053] Figure 2 This is a flowchart illustrating a task processing method in one embodiment;

[0054] Figure 3 This is a flowchart illustrating the process of determining whether there is a pending task group based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process.

[0055] Figure 4 This is a flowchart illustrating the process of determining whether there is a task group to be processed in the task pool based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process.

[0056] Figure 5 This is a flowchart illustrating the process of determining multiple second task groups polled in the previous polling process based on the maximum number of threads in one embodiment.

[0057] Figure 6 This is a flowchart illustrating the process of initializing the first batch of tasks in a task pool to obtain multiple first task groups, based on a first batch of tasks in one embodiment.

[0058] Figure 7 This is a schematic diagram of a process in one embodiment where worker threads are invoked to process a group of tasks to be processed and at least one first task group.

[0059] Figure 8 This is a flowchart illustrating the task processing method in another embodiment;

[0060] Figure 9 This is a structural block diagram of a task processing device in one embodiment;

[0061] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The task processing method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, the task processing system 102 includes a server 104. Specifically, based on the first batch of tasks obtained by the task processing system 102 in the current polling process, the server 104 can initialize the first batch of tasks in the task pool to obtain multiple first task groups. It then determines whether there are any pending task groups based on the maximum number of threads and the polling identifiers of multiple second task groups in the previous polling process. When it is determined that there are pending task groups, it selects the pending task group and at least one first task group from the task pool based on the maximum number of threads, and calls worker threads to process the pending task group and at least one first task group respectively. The maximum number of threads is the number of worker threads.

[0064] In one embodiment, such as Figure 2 As shown, a task processing method is provided, which is applied to Figure 1 Taking server 104 as an example, the following steps are included:

[0065] S202, retrieve the first batch of tasks in the current polling process.

[0066] The first batch of tasks consists of tasks to be processed. For example, the first batch of tasks can be transaction confirmation files received by the task processing system from a third party, or transaction confirmation files manually imported into the task processing system.

[0067] In some embodiments, the task processing system may include a task scheduler (UTS) that can acquire a first batch of tasks at a preset time and perform polling processing on the first batch of tasks based on the server.

[0068] In some embodiments, the task processing system has a management console, through which a person can import a first batch of tasks. The task processing system can then execute the corresponding process based on the first batch of tasks. The first batch of tasks may be tasks that failed to execute due to customer account attribute issues during the previous polling process, and which have been correctly processed by the person after the customer account attributes have been corrected.

[0069] S204: Based on the first batch of tasks, initialize the first batch of tasks in the task pool to obtain multiple first task groups.

[0070] The task pool is used to store data during the polling process. When the polling process is performed for the first time, there are no task groups in the task pool. The first batch of tasks includes multiple first target tasks, which may include tasks for generating fund market data files and tasks for pushing fund market data files. Each first target task includes corresponding target product parameters. Specifically, the target product parameters include target business type, target customer identifier, target product code, and target transfer agent (TA) code.

[0071] In some embodiments, the first batch of tasks can be grouped in the task pool according to the target product code and target TA code of the first target task in the first batch of tasks to obtain multiple first task groups. For example, the first task group can be a fund market information push task group, which consists of the task of generating fund market information files and the task of generating fund market information files.

[0072] S206, based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process, determine whether there is a task group to be processed.

[0073] The maximum number of threads is a pre-set number of worker threads, which are used to process task groups selected from the task pool. In some embodiments, the task processing system includes a poller for selecting task groups from the task pool, and worker threads for processing the task groups selected by the poller.

[0074] The polling flag is used to indicate the polling result of the task group during the polling process. The polling flag includes not polled, polled but the execution result failed, and polled. Polled indicates that the task group has been polled and the execution result of the task group is successful.

[0075] The first batch of tasks can be different from the second batch of tasks, or it can be a task updated by the second batch of tasks during the current polling process; this is not limited here. If the previous polling process was the first polling process, then the current polling process is the second polling process. The method for obtaining multiple second task groups of the second batch of tasks can refer to the method for obtaining multiple first task groups of the first batch of tasks, and will not be repeated here.

[0076] In some embodiments, if the polling identifier of at least one second task group in the second batch of tasks is not polled during the previous polling process, it is determined that there is a task group to be processed; the task group to be processed is at least one second task group in the second batch of tasks that is not polled.

[0077] In some embodiments, if the polling identifier of at least one second task group in the second batch of tasks in the previous polling process is polled but the execution result fails, it is determined that there is a task group to be processed; the task group to be processed is at least one second task group in the second batch of tasks that has been polled but the execution result fails.

[0078] S208, when it is determined that there is a task group to be processed, select the task group to be processed and at least one first task group from the task pool according to the maximum number of threads.

[0079] In some embodiments, when it is determined that there are task groups to be processed in the task pool, the tasks to be processed can be stored in the task pool; and according to the maximum number of threads, task groups to be processed and at least one first task group are selected from the task pool; wherein, the total number of task groups to be processed and at least one first task group is the same as the maximum number of threads.

[0080] The task processing system includes a poller. In some embodiments, when the poller selects a task group from the task pool based on the maximum number of threads, if the time taken for the poller to select a task group from the task pool (i.e., the actual polling time) exceeds a preset polling timeout, the polling process ends. This is to avoid the situation where the system crashes due to the inability to release resources and the resulting high CPU usage, and also to avoid the situation where subsequent batch tasks cannot run due to the inability to release resources.

[0081] S210, call worker threads to process the task group to be processed and at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0082] The system consists of multiple worker threads, each handling a task group, ensuring that the processing of different task groups is not interfered with and reducing the failure rate of task processing.

[0083] In summary, through Figure 2The method shown has two advantages. First, by combining the polling identifiers of the task groups in the previous polling process, it ensures that no task groups that need to be processed are missed in the current polling process, thus guaranteeing the integrity of the task processing results. Second, by setting a maximum number of threads, even if a large number of batch tasks are received in a short period of time, the number of tasks processed by the worker threads in the current polling process can be limited, thereby avoiding interface blocking and server downtime issues and reducing the failure rate of task processing.

[0084] In one embodiment, such as Figure 3 The diagram illustrates a process for determining the existence of a pending task group based on the maximum number of threads and the polling identifiers of multiple second task groups in the previous polling process. This method is then applied to… Figure 1 Taking server 104 as an example, the following steps are included:

[0085] S302, based on the maximum number of threads, configure a polling flag for at least one second task group that was not polled in the previous polling process; the polling flag is "not polled".

[0086] The maximum number of threads can limit the number of task groups processed by the worker threads in the previous polling process. Each second task group in the second batch of tasks includes corresponding product parameters. Specifically, the product parameters include business type, customer identifier, product code, and TA code.

[0087] In some embodiments, a second task group with the same maximum number of threads can be selected from multiple second task groups based on the priority of product parameters, and an unpolled flag can be configured for at least one second task group that is not selected.

[0088] S304, based on the polling identifier being not polled, it is determined that there is a pending task group in the task pool; the pending task group is at least one second task group.

[0089] As can be seen from the description in S302, at least one second task group is a task group that has not been polled. Specifically, at least one second task group is a task group that was not selected from the task pool in the previous polling process, and the task group still exists in the task pool. Therefore, based on the polling identifier being not polled, it can be determined that there is a task group to be processed in the task pool.

[0090] In summary, through Figure 3 The method shown identifies task groups that exceed the maximum number of threads as task groups that were not polled in the previous polling process. This allows these unpolled task groups to wait for the next polling process, avoiding server crashes during the previous polling process and reducing the failure rate of task processing.

[0091] In one embodiment, such as Figure 4 The diagram illustrates a process for determining whether a task pool contains a group of tasks to be processed, based on the maximum number of threads and the polling identifiers of multiple second task groups in the previous polling process. This method is then applied to… Figure 1 Taking server 104 as an example, the following steps are included:

[0092] S402, based on the maximum number of threads, determine the multiple second task groups that were polled in the previous polling process.

[0093] The maximum number of threads limits the number of task groups processed by worker threads in the previous polling process. Therefore, the number of multiple second task groups polled in the previous polling process is the same as the maximum number of threads.

[0094] S404, if the execution result of at least one of the multiple second subtasks is a failure, determine the polling flag of each corresponding second task group as polled but the execution result failed.

[0095] The execution results of multiple second subtasks can include success or failure. If the execution result of at least one of the multiple second subtasks is failure, then the execution result of the task group corresponding to the multiple second subtasks is also failure. Therefore, a polling flag is configured for the task group corresponding to the multiple second subtasks that has been polled but has failed.

[0096] In some embodiments, a batch task configuration table can be obtained, which includes the execution order of multiple subtasks in a task group. According to the batch task configuration table, when each worker thread executes multiple second subtasks of its corresponding second task group, the execution result of multiple second subtasks of each second task group can be obtained according to the execution order of the multiple second subtasks of each second task group. And when the execution result of at least one subtask of multiple second subtasks in each second task group fails, a polling flag is configured for each second task group that has been polled but whose execution result has failed.

[0097] S406, based on the polling identifier of each corresponding second task group as polled but with execution failure, it is determined that there are pending task groups in the task pool; the pending task groups are the second task groups corresponding to those polled but with execution failure.

[0098] When the polling flag of each corresponding second task group is marked as polled but the execution result fails, since at least one subtask in each corresponding second task group has failed to execute, in order to ensure the integrity of the result, each corresponding second task group still needs to be processed in the next polling process.

[0099] In summary, through Figure 4The method shown, by determining whether there is a group of tasks to be processed, ensures the integrity of the task processing results by preventing situations where a task group is polled but its subtasks fail to process, even if a task group is selected from the task groups.

[0100] In one embodiment, such as Figure 5 As shown, a flowchart illustrates a method for determining multiple second task groups polled in the previous polling process based on the maximum number of threads. This method is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0101] S502, obtain the priority of multiple second task groups.

[0102] The multiple second task groups include corresponding product parameters, which include business type, customer identifier, product code, and TA code. The business type has a higher priority than the product code, which in turn has a higher priority than the TA code, and the TA code has a higher priority than the customer identifier. It is understood that the priority of the product parameters can also be set according to the actual application scenario; this embodiment does not impose specific limitations.

[0103] S504 selects multiple second task groups with the same maximum number of threads based on their priorities.

[0104] In some embodiments, the corresponding second task groups can be sorted in descending order according to the priority order among business type, product code, and TA code, and the first N task groups with the same maximum number of threads can be selected from the multiple second task groups after descending sorting, and the first N task groups can be determined as multiple second task groups.

[0105] In summary, through Figure 5 The method shown can prioritize task groups based on their priority, and can also limit the number of task groups that worker threads can process by setting a maximum number of threads, thus avoiding server crashes during task processing and reducing the failure rate of task processing.

[0106] In one embodiment, such as Figure 6 As shown, a flowchart illustrates a method for initializing a first batch of tasks in a task pool to obtain multiple first task groups, and how this method can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0107] S602, Get the task group configuration table.

[0108] In some embodiments, the task group configuration table includes grouping rules corresponding to the business type; for example, if the business type is 03, the grouping rule corresponding to the business type is a grouping rule consisting of the business type and the product code. When the business type is 05, the grouping rule corresponding to the business type is a grouping rule consisting of the business type and the TA code.

[0109] In some embodiments, the task group configuration table includes rules composed of business type, product code, and TA code. For example, the task group configuration table includes: a rule composed of business type 03, product code P1, and TA code TA3, and a rule composed of business type 05, product code P4, and TA code TA4.

[0110] S604: Based on the task group configuration table and the first batch of tasks, initialize the first batch of tasks in the task pool to obtain multiple first task groups.

[0111] In some embodiments, the first batch of tasks can be grouped according to the grouping rules corresponding to the business type and the business type of each first target task in the first batch of tasks to obtain multiple first task groups of the first batch of tasks.

[0112] As shown in Table 1, a first batch task is provided, wherein the first batch task includes 8 first target tasks, each of which includes target business type, target customer identifier, target product code, target TA code and other data.

[0113] Table 1

[0114]

[0115]

[0116] Table 2 provides a task group configuration table, as follows:

[0117] Table 2

[0118] Business type Grouping Dimensions Other data 03 Product Code ...... 05 TA code ...... ...... ...... ......

[0119] Furthermore, based on the first batch of tasks shown in Table 1 and the task group configuration table shown in Table 2, the grouping results shown in Table 3 can be obtained:

[0120] Table 3

[0121] Business type Product Code Other data 03 P1 Customer 1, Customer 3, TA1, ... 03 P2 Customer 2, Customer 4, TA1, ... 05 TA3 Customer 5, Customer 6, P3, P6, ... 05 TA4 Customer 7, Customer 8, P4, P5, ...

[0122] In summary, through Figure 6The method shown can group batch tasks according to a pre-set task group configuration table to obtain multiple corresponding task groups. Different task group configuration tables can be set according to different business types and corresponding grouping dimensions to meet different data analysis needs.

[0123] In one embodiment, the worker threads include a first worker thread and at least one second worker thread; specifically, such as... Figure 7 The diagram illustrates a process for calling worker threads to process a group of tasks to be processed and at least one first task group, respectively. This method is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0124] S702, call at least one second worker thread to process at least one first task group respectively.

[0125] Each second worker thread processes one first task group, ensuring that the processing of different task groups is not interfered with, thus reducing the failure rate of task processing.

[0126] S704: When the execution result of at least one subtask in the task group to be processed is a failure, the first worker thread is called to start processing from the subtask that failed, according to the execution order of the subtasks in the task group to be processed, until the task group to be processed is completed.

[0127] The execution order of multiple subtasks in a task group to be processed can be obtained through a batch task configuration table. For example, if a task group to be processed includes four subtasks, and the execution order of each subtask is A1->A2->A3->A4, when the first worker thread fails to process subtask A3 in the previous polling process, it will start processing directly from subtask A3 in the current polling process. That is, subtasks A3 and A4 will be processed in the current polling process, while subtasks A1 and A2 will not be processed. The processing results of subtasks A1 and A2 in the previous polling process will be used.

[0128] In summary, through Figure 7 The method shown reprocesses the task groups that were polled but had not yet been executed in the previous polling process, which can ensure the integrity of the task results; at the same time, it can save task processing time and improve the processing efficiency of task groups by starting processing directly from the failed subtasks.

[0129] In one embodiment, taking the current polling process as the next polling process, the previous polling process as the first polling process, and the first batch of tasks as the tasks updated by the second batch of tasks during the current polling process as an example, such as... Figure 8 The diagram illustrates a task processing method, the details of which are described below:

[0130] During the initial polling process, the second batch of tasks includes a first task, a second task, and a third task, etc. These tasks can be tasks for generating fund market data files and tasks for pushing fund market data files. Based on the task group configuration table, the second batch of tasks is initialized in the task pool to obtain multiple second task groups. These multiple second task groups include the first task group, the second task group, and the third task group. For example, the second task group is a task group created for tasks that generate and push fund market data files.

[0131] During the initial polling process, the poller selects the first and third task groups from the task pool based on the number of worker threads (i.e., the maximum number of threads). Worker thread 1 and worker thread 2 then process the first and third task groups respectively. Since worker thread 1 executes multiple subtasks in the first task group successfully, the polling of the first task group is marked as polled, and therefore, it will not be processed again in the next polling. Worker thread 2 executes subtask 3 in the third task group unsuccessfully, causing subtask 4, which follows subtask 3, to also fail. Therefore, the polling of the third task group is marked as polled but the execution result is unsuccessful, and it will continue to be processed in the next polling. The execution results of the subtasks of the batch tasks can be recorded in the batch task execution record table, which can be stored in the batch task execution pool for easy retrieval of the execution results of each subtask to determine whether there are any pending task groups.

[0132] During the re-polling process, the first batch of tasks is obtained. The first batch of tasks includes a first task, a second task, a third task, and a fourth task. The first task in the first batch of tasks is the updated data of the first task in the second batch of tasks. The second task in the first batch of tasks is the updated task of the second task in the second batch of tasks. The third task in the first batch of tasks is the updated data of the third task in the second batch of tasks. According to the task group configuration table, the first batch of tasks is initialized in the task pool to obtain multiple first task groups. The multiple first task groups include a fourth task group and a fifth task group.

[0133] Furthermore, if it is determined that there are pending task groups, including the third task group (i.e., the task group that was polled in the last polling but failed to execute) and the second task group (i.e., the task group that was not polled in the last polling process), then the poller selects the second and fourth task groups from the task pool based on the number of worker threads (i.e., the maximum number of threads). Worker thread 1 and worker thread 2 respectively process the subtasks in the second and fourth task groups. If the execution results of all subtasks in the second and fourth task groups are successful, then the second and fourth task groups will not be processed again in the next polling process. Subsequent polling processes follow the same logic, and will not be elaborated further here.

[0134] Understandably, the poller can also select the second and third task groups from the task pool based on the number of worker threads (i.e., the maximum number of threads). When a worker thread processes the third task group, it starts processing from the failed subtasks within that group; that is, the worker thread starts processing subtask 3 within the third task group and continues processing subtask 4. The fourth and fifth task groups, which are not selected, can be processed in the next polling cycle. The specific processing method can be determined based on the actual application scenario; this embodiment does not impose specific limitations.

[0135] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0136] Based on the same inventive concept, this application also provides a task processing apparatus for implementing the task processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more task processing apparatus embodiments provided below can be found in the limitations of the task processing method described above, and will not be repeated here.

[0137] In one embodiment, such as Figure 9 As shown, a task processing device is provided, including: an acquisition module 902, a grouping module 904, a determination module 906, a selection module 908, and a processing module 910, wherein:

[0138] Module 902 is used to obtain the first batch of tasks in the current polling process;

[0139] Grouping module 904 is used to initialize the first batch of tasks in the task pool based on the first batch of tasks, and obtain multiple first task groups;

[0140] The determination module 906 is used to determine whether there is a pending task group based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process.

[0141] Selection module 908 is used to select a task group to be processed and at least one first task group from the task pool based on the maximum number of threads when it is determined that there is a task group to be processed.

[0142] The processing module 910 is used to call worker threads to process the task group to be processed and at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0143] In one embodiment, the determining module 906 is further configured to: configure a polling identifier for at least one second task group that was not polled in the previous polling process based on the maximum number of threads; the polling identifier is not polled; and determine that there is a task group to be processed in the task pool based on the polling identifier being not polled; the task group to be processed is at least one second task group.

[0144] In one embodiment, the determining module 906 is further configured to: determine, based on the maximum number of threads, multiple second task groups that were polled in the previous polling process; if the execution result of at least one of the multiple second subtasks is a failure, determine that the polling identifier of each corresponding second task group is polled but the execution result is failed; determine that there are pending task groups in the task pool based on the polling identifier of each corresponding second task group being polled but the execution result is failed; the pending task groups are the second task groups that were polled but the execution result is failed.

[0145] In one embodiment, the determining module 906 is further configured to: obtain the priorities of multiple second task groups; and select multiple second task groups with the same maximum number of threads from the multiple second task groups according to the priorities of the multiple second task groups.

[0146] In one embodiment, the grouping module 904 is further configured to: obtain a task group configuration table; and initialize the first batch of tasks in the task pool according to the task group configuration table and the first batch of tasks to obtain multiple first task groups.

[0147] In one embodiment, the worker thread includes a first worker thread and at least one second worker thread; the processing module 910 is further configured to: call at least one second worker thread to process at least one first task group respectively; when the execution result of at least one subtask in the task group to be processed is failure, according to the execution order of the subtasks in the task group to be processed, call the first worker thread to start processing from the subtask with the failure result until the task group to be processed is completed.

[0148] Each module in the aforementioned task processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0149] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores task group configuration tables and batch task configuration tables. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a task processing method.

[0150] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0151] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0152] Retrieve the first batch of tasks in the current polling process;

[0153] Based on the first batch of tasks, initialize the first batch of tasks in the task pool to obtain multiple first task groups;

[0154] Based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process, determine whether there are any pending task groups;

[0155] When it is determined that there is a group of tasks to be processed, select the group of tasks to be processed and at least one first task group from the task pool based on the maximum number of threads.

[0156] The worker threads are invoked to process the task group to be processed and at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0157] In one embodiment, when the processor executes the computer program, it further performs the following steps: configuring a polling identifier for at least one second task group that was not polled in the previous polling process based on the maximum number of threads; the polling identifier is not polled; determining that there is a task group to be processed in the task pool based on the polling identifier being not polled; the task group to be processed is at least one second task group.

[0158] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining multiple second task groups that were polled in the previous polling process based on the maximum number of threads; if the execution result of at least one of the multiple second subtasks is a failure, determining that the polling identifier of each corresponding second task group is polled but the execution result is a failure; determining that there are pending task groups in the task pool based on the polling identifier of each corresponding second task group being polled but the execution result is a failure; the pending task groups are the second task groups corresponding to the polled but failed execution results.

[0159] In one embodiment, when the processor executes a computer program, it further performs the following steps: obtaining the priorities of a plurality of second task groups; and selecting a plurality of second task groups that are the same as the maximum number of threads from the plurality of second task groups according to the priorities of the plurality of second task groups.

[0160] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a task group configuration table; and initializing the first batch of tasks in the task pool according to the task group configuration table and the first batch of tasks to obtain multiple first task groups.

[0161] In one embodiment, the worker thread includes a first worker thread and at least one second worker thread; when the processor executes the computer program, it further implements the following steps: calling at least one second worker thread to process at least one first task group respectively; when the execution result of at least one subtask in the task group to be processed is a failure, according to the execution order of the subtasks in the task group to be processed, calling the first worker thread to start processing from the subtask with the failure result until the task group to be processed is completed.

[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0163] Retrieve the first batch of tasks in the current polling process;

[0164] Based on the first batch of tasks, initialize the first batch of tasks in the task pool to obtain multiple first task groups;

[0165] Based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process, determine whether there are any pending task groups;

[0166] When it is determined that there is a group of tasks to be processed, select the group of tasks to be processed and at least one first task group from the task pool based on the maximum number of threads.

[0167] The worker threads are invoked to process the task group to be processed and at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0168] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: configuring a polling flag for at least one second task group that was not polled in the previous polling process based on the maximum number of threads; the polling flag is not polled; determining that there is a task group to be processed in the task pool based on the polling flag being not polled; the task group to be processed is at least one second task group.

[0169] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining multiple second task groups that were polled in the previous polling process based on the maximum number of threads; if the execution result of at least one of the multiple second subtasks is a failure, determining that the polling identifier of each corresponding second task group is polled but the execution result is a failure; determining that there are pending task groups in the task pool based on the polling identifier of each corresponding second task group being polled but the execution result is a failure; the pending task groups are the second task groups corresponding to the polled but failed execution results.

[0170] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the priorities of a plurality of second task groups; and selecting a plurality of second task groups that are the same as the maximum number of threads from the plurality of second task groups according to the priorities of the plurality of second task groups.

[0171] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a task group configuration table; and initializing the first batch of tasks in a task pool according to the task group configuration table and the first batch of tasks to obtain multiple first task groups.

[0172] In one embodiment, the worker thread includes a first worker thread and at least one second worker thread; when the computer program is executed by the processor, it further implements the following steps: calling at least one second worker thread to process at least one first task group respectively; when the execution result of at least one subtask in the task group to be processed is a failure, according to the execution order of the subtasks in the task group to be processed, calling the first worker thread to start processing from the subtask with the failure result until the task group to be processed is completed.

[0173] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0174] Retrieve the first batch of tasks in the current polling process;

[0175] Based on the first batch of tasks, initialize the first batch of tasks in the task pool to obtain multiple first task groups;

[0176] Based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process, determine whether there are any pending task groups;

[0177] When it is determined that there is a group of tasks to be processed, select the group of tasks to be processed and at least one first task group from the task pool based on the maximum number of threads.

[0178] The worker threads are invoked to process the task group to be processed and at least one first task group respectively; the maximum number of threads is the number of worker threads.

[0179] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: configuring a polling flag for at least one second task group that was not polled in the previous polling process based on the maximum number of threads; the polling flag is not polled; determining that there is a task group to be processed in the task pool based on the polling flag being not polled; the task group to be processed is at least one second task group.

[0180] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining multiple second task groups that were polled in the previous polling process based on the maximum number of threads; if the execution result of at least one of the multiple second subtasks is a failure, determining that the polling identifier of each corresponding second task group is polled but the execution result is a failure; determining that there are pending task groups in the task pool based on the polling identifier of each corresponding second task group being polled but the execution result is a failure; the pending task groups are the second task groups corresponding to the polled but failed execution results.

[0181] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the priorities of a plurality of second task groups; and selecting a plurality of second task groups that are the same as the maximum number of threads from the plurality of second task groups according to the priorities of the plurality of second task groups.

[0182] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a task group configuration table; and initializing the first batch of tasks in a task pool according to the task group configuration table and the first batch of tasks to obtain multiple first task groups.

[0183] In one embodiment, the worker thread includes a first worker thread and at least one second worker thread; when the computer program is executed by the processor, it further implements the following steps: calling at least one second worker thread to process at least one first task group respectively; when the execution result of at least one subtask in the task group to be processed is a failure, according to the execution order of the subtasks in the task group to be processed, calling the first worker thread to start processing from the subtask with the failure result until the task group to be processed is completed.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0185] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0186] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0187] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A task processing method, characterized in that, The method includes: Retrieve the first batch of tasks in the current polling process; Based on the first batch of tasks, the first batch of tasks are initialized in the task pool to obtain multiple first task groups; Based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process, determine whether there are any pending task groups; When it is determined that there is a group of tasks to be processed, the group of tasks to be processed and at least one first task group are selected from the task pool according to the maximum number of threads. Worker threads are invoked to process the pending task group and the at least one first task group respectively; the maximum number of threads is the number of worker threads; The step of determining whether there is a pending task group based on the maximum number of threads and the polling identifiers of multiple second task groups in the previous polling process includes: Based on the maximum number of threads, configure a polling identifier for at least one second task group that was not polled in the previous polling process; the polling identifier indicates that it was not polled; based on the polling identifier indicating that it was not polled, determine that there is a task group to be processed in the task pool; the task group to be processed is the at least one second task group. Alternatively, the priorities of the plurality of second task groups are obtained; based on the priorities of the plurality of second task groups, the plurality of second task groups with the same number of threads are selected from the plurality of second task groups; if the execution result of at least one of the plurality of second subtasks fails, the polling identifier of the corresponding second task group is determined to be polled but the execution result has failed; based on the polling identifier of the corresponding second task group being polled but the execution result has failed, it is determined that there is a task group to be processed in the task pool; the task group to be processed is the second task group corresponding to the polled but the execution result has failed.

2. The method according to claim 1, characterized in that, The step involves initializing the first batch of tasks in a task pool to obtain multiple first task groups, including: Get the task group configuration table; Based on the task group configuration table and the first batch of tasks, the first batch of tasks are initialized in the task pool to obtain the plurality of first task groups.

3. The method according to claim 1, characterized in that, The worker threads include a first worker thread and at least one second worker thread; the invocation of the worker threads processes the pending task group and the at least one first task group respectively, including: The at least one second worker thread is invoked to process the at least one first task group respectively; When at least one subtask in the pending task group fails to execute, the first worker thread is invoked to process the subtask that failed to execute, according to the execution order of the subtasks in the pending task group, until the pending task group is completed.

4. The method according to claim 1, characterized in that, The step involves initializing the first batch of tasks in a task pool to obtain multiple first task groups, including: Based on the target product code and target transfer registration code of the first target task in the first batch of tasks, the first batch of tasks are grouped in the task pool to obtain multiple first task groups.

5. A task processing device, characterized in that, The device includes: The acquisition module is used to acquire the first batch of tasks in the current polling process; The grouping module is used to initialize the first batch of tasks in the task pool according to the first batch of tasks, and obtain multiple first task groups; The determination module is used to determine whether there is a pending task group based on the maximum number of threads and the polling identifiers of multiple second task groups of the second batch of tasks in the previous polling process. The selection module is used to select, when it is determined that there is a task group to be processed, the task group to be processed and at least one first task group from the task pool according to the maximum number of threads. The processing module is used to call worker threads to process the task group to be processed and the at least one first task group respectively; the maximum number of threads is the number of worker threads; The determining module is further configured to: configure a polling identifier for at least one second task group that was not polled in the previous polling process according to the maximum number of threads; the polling identifier is not polled; determine that there is a pending task group in the task pool based on the polling identifier being not polled; the pending task group is the at least one second task group; or, obtain the priority of the plurality of second task groups; select the plurality of second task groups with the same number of threads as the maximum number of threads from the plurality of second task groups according to the priority of the plurality of second task groups; if the execution result of at least one subtask among the plurality of second subtasks is a failure, determine that the polling identifier of each corresponding second task group is polled but the execution result is failed; determine that there is a pending task group in the task pool based on the polling identifier of each corresponding second task group being polled but the execution result is failed; the pending task group is each second task group corresponding to the polled but the execution result is failed.

6. The apparatus according to claim 5, characterized in that, The grouping module is further configured to: obtain a task group configuration table; and initialize the first batch of tasks in the task pool according to the task group configuration table and the first batch of tasks to obtain the plurality of first task groups.

7. The apparatus according to claim 5, characterized in that, The working thread includes a first working thread and at least one second working thread; the processing module is further configured to: call the at least one second working thread to process the at least one first task group respectively; when the execution result of at least one subtask in the task group to be processed is a failure, according to the execution order of the subtasks in the task group to be processed, call the first working thread to start processing from the subtask with the failure result until the task group to be processed is completed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images