Task processing method, electronic equipment and program product

Abstract

The invention provides a task processing method, electronic equipment and a program product. The task processing method is applied to the task processing system, the task processing system comprises a plurality of processing nodes, and the task processing method comprises the steps of creating a to-be-processed task; determining the data volume of the to-be-processed task; splitting the to-be-processed task into a plurality of sub-tasks according to the data volume of the to-be-processed task and the load condition of the task processing system; and distributing the subtasks to the processing nodes to process the subtasks.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a task processing method, electronic device, storage medium, and program product. Background Technology

[0002] In the field of large-scale, massive data processing, such as tax filing or financial calculations, distributed computing has become a core technology for dealing with high concurrency and complex logic.

[0003] However, in existing distributed systems, task splitting typically employs fixed batch sizes or static patterns, failing to dynamically adapt to changes in data size and system load. When the data volume is small, this fixed splitting leads to excessive scheduling overhead, wasting resources and reducing efficiency; when the data volume is large, the overall progress is slowed down due to the excessively long execution time of individual tasks, and may even cause system bottlenecks.

[0004] Furthermore, task allocation often ignores the real-time status and historical performance of nodes, leading to uneven load, idle or overloaded resources, which affects system stability and scalability.

[0005] Therefore, an adaptive task processing mechanism is needed to optimize task splitting and allocation, thereby achieving efficient resource utilization and processing performance. Summary of the Invention

[0006] This disclosure provides a task processing method, an electronic device, a storage medium, and a program product.

[0007] According to one aspect of this disclosure, a task processing method is provided, applied to a task processing system, the task processing system including multiple processing nodes, the task processing method comprising: Create a task to be processed; Determine the amount of data for the task to be processed; Based on the amount of data in the task to be processed and the load of the task processing system, the task to be processed is divided into multiple sub-tasks. The subtask is assigned to the processing node for processing.

[0008] The above technical solution dynamically considers data volume and system load when splitting tasks, avoiding the problems of excessive scheduling overhead for small data volumes or excessively long execution time for large data volumes caused by fixed patterns. This achieves adaptive splitting, improving overall processing efficiency and resource utilization. Simultaneously, combined with load balancing, it ensures stable system operation in high-concurrency scenarios, significantly shortening task completion time and reducing system bottleneck risks.

[0009] According to at least one embodiment of the task processing method of this disclosure, the step of dividing the task to be processed into multiple sub-tasks based on the data volume of the task to be processed and the load of the task processing system includes: When the amount of data in the task to be processed is in the first data volume range, the task to be processed is divided into multiple sub-tasks according to the load rate of the task processing system.

[0010] The above technical solution, for medium-sized data volumes (the first range, such as 10,000-100,000 records), dynamically adjusts the splitting granularity based on the load rate. This avoids excessive splitting under low load, which increases overhead, or coarse-grained splitting causing bottlenecks under high load, thereby improving system responsiveness and balanced resource utilization. Simultaneously, this strategy adapts to real-time load changes, ensuring optimized processing efficiency in different scenarios and reducing overall latency.

[0011] According to at least one embodiment of the task processing method of this disclosure, the step of splitting the task to be processed into multiple sub-tasks according to the load rate of the task processing system includes: When the load rate is greater than the first load rate threshold, the task to be processed is split into a first number of subtasks; When the load rate is less than or equal to the first load rate threshold and greater than the second load rate threshold, the task to be processed is split into a second number of sub-tasks; When the load rate is less than the second load rate threshold, the task to be processed is split into a third number of subtasks; The first load rate threshold is greater than the second load rate threshold, the first quantity is greater than the second quantity, and the second quantity is greater than the third quantity.

[0012] The above technical solution uses multi-level load thresholds (e.g., 80%, 50%) to divide the number of subtasks into layers (e.g., 8-16, 4-7, 2-3), enabling fine-grained parallelism to accelerate processing under high load and coarse-grained parallelism to reduce scheduling overhead under low load, thereby improving system adaptability and efficiency. Simultaneously, it avoids resource waste, ensures stable performance under load fluctuations, and optimizes overall task throughput.

[0013] According to at least one embodiment of the task processing method of this disclosure, the step of dividing the task to be processed into multiple sub-tasks based on the data volume of the task to be processed and the load of the task processing system includes: When the amount of data in the task to be processed is in the second data range, the task to be processed is divided into multiple subtasks according to the number of processing nodes, the memory capacity of the task processing system, and the processing time required for a single subtask.

[0014] The above technical solution, targeting large-scale data volumes (second range, e.g., >100,000 records), comprehensively addresses constraints across multiple dimensions, including node count, memory, and processing time, avoiding imbalances caused by a single factor and improving parallel processing capabilities and resource utilization. Simultaneously, this method prevents memory overflow or timeout issues, ensuring reliable system operation in massive data scenarios and significantly improving processing speed.

[0015] According to at least one embodiment of the task processing method of this disclosure, the step of splitting the task to be processed into multiple subtasks based on the number of processing nodes, the memory capacity of the task processing system, and the processing time required for a single subtask includes: The first number of subtasks is determined based on the number of processing nodes; The data volume of a single subtask is determined based on the memory capacity, and a second number of subtasks is determined based on the data volume of the single subtask. The third number of subtasks is determined based on the processing time required for each individual subtask; The final number of the sub-tasks is determined based on the first number, the second number, and the third number.

[0016] The above technical solution achieves optimized partitioning under multiple constraints by calculating three types of quantities (based on nodes, memory, and time), avoiding resource waste or bottlenecks caused by excessive or insufficient partitioning, and improving system flexibility and efficiency. At the same time, it ensures that each subtask is executable within memory and time limits, enhancing the stability and scalability of distributed processing.

[0017] A task processing method according to at least one embodiment of the present disclosure, which determines the final number of sub-tasks based on a first quantity, a second quantity, and a third quantity, includes: The minimum of the first quantity, the second quantity, and the third quantity is determined as the final quantity.

[0018] The above technical solution uses the minimum of the three quantities as the final split to ensure that the strictest constraints are met, avoiding the risk of memory overruns or timeouts, and improving system security and resource efficiency. At the same time, this conservative strategy optimizes parallelism through multi-dimensional balancing, reducing the occurrence of failures and improving the overall task completion rate.

[0019] According to at least one embodiment of the task processing method of this disclosure, the step of assigning the subtask to the processing node for processing the subtask includes: The matching score of a processing node for different subtasks is determined based on the matching degree between the subtask and the processing node, the priority of the subtask, the load rate of the processing node, and the historical execution score of the processing node; and Subtasks are assigned to the processing nodes based on their matching scores for different subtasks.

[0020] The above technical solution achieves load balancing and efficient resource utilization through intelligent allocation based on multi-factor matching scores (matching degree, priority, load, and historical scores), avoiding the overload or idle node problems of traditional random allocation. Simultaneously, it improves task execution reliability and speed, optimizing system performance under high concurrency.

[0021] According to at least one embodiment of the task processing method of this disclosure, the step of assigning subtasks to the processing node based on the matching scores of the processing node corresponding to different subtasks includes: The corresponding subtask is selected based on the matching score of the processing node for different subtasks; The corresponding subtask is assigned according to the task lock, which is used to assign the corresponding subtask to the processing node.

[0022] The above technical solution allocates the corresponding subtask according to the task lock, and the task lock is used to allocate the corresponding subtask to the processing node.

[0023] By combining matching score selection and task lock (distributed lock) mechanisms, atomicity and intelligent allocation are ensured, conflicts are avoided, and decentralized scheduling is achieved, improving system reliability and efficiency. Simultaneously, resource contention is prevented, enhancing parallel processing capabilities and fault tolerance.

[0024] According to at least one embodiment of the task processing method of this disclosure, the subtask is executed according to business processing logic, and the business processing logic is set according to business requirements; The business process corresponding to the business requirement has multiple processing nodes, and there is an execution order relationship among the multiple processing nodes; The task processing method includes: Trigger the first processing node and execute the business processing logic corresponding to the first processing node to process the subtasks assigned to the first processing node; In response to the completion of the subtask assigned to the first processing node, the subtask status of the subtask is updated and a subtask completion event is published; Based on the subtask completion event, determine the second processing node that should be executed after the first processing node, and trigger the second processing node to execute the business processing logic corresponding to the second processing node to process the subtask assigned to the second processing node.

[0025] The above technical solution decouples business processes from logic through event-driven node triggering and state updates, improving system scalability and maintenance efficiency, and avoiding the complexity of code modifications caused by traditional tight coupling. Simultaneously, it supports automatic retries and exception handling, ensuring process reliability and rapid deployment of new services.

[0026] According to at least one embodiment of the task processing method of this disclosure, the task processing system further includes middleware that interacts with the processing node; The task processing method further includes: The running status of the task processing system is determined based on at least one of the resource usage information of the task processing system, the status information of the middleware, and the processing status of the processing nodes of the task processing system. When the operating status is detected to meet the first condition, the operation of the processing node is paused; If the operating status is detected to meet the second condition, the operation of the processing node is not suspended or the operation of the suspended processing node is resumed.

[0027] The above technical solution dynamically regulates system operation through multi-indicator monitoring (resources, middleware, processing status), achieving adaptive pause / resumption, avoiding cascading failures caused by overload, and improving system stability and availability. Simultaneously, integration with task scheduling ensures efficient resource utilization and rapid recovery to normal processing.

[0028] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a task processing method according to any embodiment of this disclosure.

[0029] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement a task processing method according to any embodiment of this disclosure.

[0030] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a task processing method according to any embodiment of this disclosure. Attached Figure Description

[0031] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0032] Figure 1 This is a schematic diagram illustrating an application scenario of a task processing method according to one embodiment of the present disclosure.

[0033] Figure 2 This is a flowchart illustrating a task processing method according to one embodiment of the present disclosure.

[0034] Figure 3 This is a schematic diagram of a task processing architecture according to one embodiment of the present disclosure.

[0035] Figure 4 A flowchart illustrating task splitting according to one embodiment of this disclosure is shown.

[0036] Figure 5 A flowchart illustrating a preferred embodiment of the task splitting of this disclosure is shown.

[0037] Figure 6 A flowchart illustrating task splitting according to one embodiment of this disclosure is shown.

[0038] Figure 7 A flowchart illustrating task allocation according to a preferred embodiment of this disclosure is shown.

[0039] Figure 8 A schematic diagram illustrating the subtask preemption interaction of one embodiment of this disclosure is shown.

[0040] Figure 9 A schematic diagram of the process node triggering in one embodiment of this disclosure is shown.

[0041] Figure 10 A schematic diagram of a processing node transition according to one embodiment of this disclosure is shown.

[0042] Figure 11 A schematic diagram illustrating the transition of task execution states according to one embodiment of this disclosure is shown.

[0043] Figure 12 A schematic diagram of the operation status monitoring process of a task processing system according to one embodiment of the present disclosure is shown.

[0044] Figure 13 An exemplary embodiment of the structured health check system of this disclosure is shown.

[0045] Figure 14 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0046] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0047] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] In large-scale data processing, such as tax data processing, massive amounts of taxpayer data need to be handled. Taking corporate tax data processing as an example, in the scenario of calculating individual income tax, it is necessary to calculate individual income tax for a large number of employees every month. The calculation of individual income tax involves complex calculation logic such as employee wages and salaries, special additional deductions, and cumulative withholding. In the existing data processing process, the breakdown of tasks to be processed (i.e., data processing tasks) usually adopts a fixed batch size, i.e., a fixed subtask size, which cannot be dynamically adjusted. When the data volume is small, the granularity of the breakdown is too fine, resulting in large scheduling overhead. When the data volume is large, the granularity of the breakdown is too coarse, resulting in excessively long execution time for individual subtasks. It is unable to adapt to changes in system load over different time periods.

[0049] Therefore, this disclosure proposes a task processing method.

[0050] Figure 1 A schematic diagram illustrating an application scenario of a task processing method according to an embodiment of this disclosure is shown.

[0051] Scenario 100 includes a terminal device 110 (e.g., a computer) and a server 120. The terminal device 110 communicates with the server 120 via a network. A distributed task processing system can be configured using one or more servers 120.

[0052] The task processing system consists of multiple processing nodes and middleware that interacts with these nodes. The processing nodes are used to process tasks.

[0053] Terminal device 110 can send a data processing request to server 120. After receiving the data processing request, server 120 creates a task to be processed according to the data processing request, determines the amount of data in the task to be processed, and, based on the amount of data in the task to be processed and the load of the task processing system, splits the task to be processed into multiple sub-tasks, and assigns the split sub-tasks to the corresponding processing nodes for processing, until all sub-tasks are processed, and returns the processing result to terminal device 110.

[0054] Figure 2 A flowchart illustrating a task processing method according to one embodiment of this disclosure is shown.

[0055] refer to Figure 2 The task processing method 200 of this embodiment includes steps S210 to S230.

[0056] Task processing method 200 can be executed by the aforementioned server or other electronic devices.

[0057] A task processing system consists of multiple processing nodes and middleware that interacts with these nodes. Processing nodes are used to process tasks and can be physical machines, virtual machines running on those machines, etc.

[0058] First, execute step S210 to create a task to be processed.

[0059] In this context, "tasks to be processed" refers to data processing tasks that need to be processed within the task processing system. These tasks may include task content and related parameters. For example, task content might involve calculating tax data. Related parameters could be data used in the task content, such as income data, for calculating tax data.

[0060] Terminal device 110 sends a data processing request to server 120, such as a data processing request for Company A's tax declaration for this month.

[0061] After receiving a data processing request, server 120 creates a corresponding task execution process, i.e., creates a task to be processed, based on the task content and data identifier in the data processing request.

[0062] The monthly tax filing process is used as an example of a data processing request. The server can create a business process instance for this data processing request through the service orchestration layer or service orchestrator, execute the business process instance, and instantiate the first processing node (such as a data synchronization node) according to the predefined process template, generating a corresponding parent task (i.e., the task to be processed).

[0063] A process template can include multiple processing nodes, the execution order or execution dependency between processing nodes, and the business processing logic corresponding to each processing node.

[0064] Configure the data processing layer 360 (or data processor) according to the business processing logic. This data processing component is associated with the process template and calls the corresponding data processing component when the corresponding processing node is executed.

[0065] Server 120 can obtain the aforementioned parent task through the task management layer or task manager. Next, when executing a processing node, the corresponding processing node, such as the tax calculation node and the declaration submission node, is instantiated through the service orchestration layer or service orchestrator.

[0066] Figure 3 A schematic diagram of a task processing architecture (server) according to one embodiment of the present disclosure is shown.

[0067] refer to Figure 3The task processing architecture 300 includes a task management layer 310, a service orchestration layer 320, and a business processing logic layer 350.

[0068] The business processing logic layer 350 calls various business processing logics or business processing logic services in the business processing logic layer to execute corresponding tasks.

[0069] Business processing logic or business processing logic services may include data synchronization services, tax calculation services, tax declaration processing services, and tax payment processing services.

[0070] The TaskManager in the Task Management Layer 310 is used for the creation, scheduling and lifecycle management of pending tasks, the registration, discovery and load balancing of executors (processing nodes), and the implementation of task preemption.

[0071] For example, the parent task of the data synchronization node needs to store the data to be processed in the data processing request into the database of the server 120. For example, by pulling the latest employee data, such as employee onboarding data, employee offboarding data, and employee basic information data, the system can obtain employee salary, bonus, and other data for the current settlement period, and collect information on special additional deductions declared by employees themselves.

[0072] The data obtained from the request processing can be converted into a unified standardized format and stored in a database, awaiting use by processing nodes such as tax calculation nodes.

[0073] In step S220, the amount of data (i.e., the total amount of data) of the task to be processed is determined.

[0074] The server 120 can determine the amount of data to be processed based on the amount of data requested in the data processing request.

[0075] Alternatively, the server (120) can query the database based on the data processing request to determine the amount of data that needs to be processed. For example, XX records of employee income data.

[0076] refer to Figure 3 The server 120 determines the total amount of data to be processed by the parent task (i.e., the task to be processed) through the task management layer 310 or the task manager. For example, the amount of data for the task to be processed can be queried from the database using a query statement.

[0077] In step S230, the task to be processed is divided into multiple subtasks based on the amount of data in the task to be processed and the load of the task processing system.

[0078] Among them, load refers to the current workload or resource utilization of the task processing system, such as CPU utilization, memory utilization, database connection status, and the number of currently executing tasks.

[0079] like Figure 3 As shown, the server 120, through the task management layer 310, splits the parent task into several sub-tasks based on the data volume of the tasks to be processed and the real-time load of the task processing system. For example, the parent task is split into 20 sub-tasks, and each sub-task processes the data of employee A.

[0080] The task processing architecture 300 also includes an executor layer 330 and a tool component layer 340. Tasks or subtasks are executed by the executors in the executor layer 330. Tasks are split using a splitting algorithm in the tool component layer 340. It should be understood that the following splitting implementation methods can be packaged into algorithmic form for task splitting.

[0081] Preferably, based on the data volume of the task to be processed and the load of the task processing system, the task to be processed is divided into multiple sub-tasks, including: When the amount of data to be processed is in the first data volume range, the task to be processed is divided into multiple subtasks according to the load rate of the task processing system.

[0082] The first data volume range is a pre-set range of data volume used to distinguish different data volumes. For example, the first data volume range is 10,000 to 100,000 records (including two endpoints).

[0083] Load rate is a quantitative indicator of load conditions. For example, load rate can be determined by using a weighted algorithm to measure the memory usage, CPU usage, database connection status, and number of currently executing tasks in a task processing system.

[0084] Preferably, weighting coefficients are set for memory utilization, CPU utilization, database connection status, and the number of currently executing tasks. The load rate is obtained by multiplying memory utilization, CPU utilization, database connection status, and the number of currently executing tasks by their respective weighting coefficients and summing them.

[0085] Continue to refer to Figure 2 In step S240, the multiple subtasks obtained by splitting the task to be processed are assigned to the processing nodes of the task processing system for processing.

[0086] The Task Manager dynamically allocates subtasks to relatively idle processing nodes based on the current load status of each processing node, ensuring that system resources are used in a balanced manner.

[0087] During the allocation process, subtasks are encapsulated into independent task units and transmitted to the target processing node, while a task status tracking mechanism is established to monitor the execution status.

[0088] This disclosure enables the task processing system to fully utilize the parallel processing capabilities of multiple processing nodes through reasonable subtask allocation, thereby significantly improving the overall task processing efficiency.

[0089] Figure 4 A flowchart illustrating task splitting according to one embodiment of this disclosure is shown.

[0090] The server 120 executes step S410 to determine the amount of data to be processed, and then executes step S420 to determine the scale of the amount of data to be processed.

[0091] It should be noted that, Figure 4 The specific values ​​and ranges mentioned are exemplary and should not be construed as limitations on the technical solutions disclosed herein.

[0092] For example, when the data volume is between 10,000 and 100,000 records, step S450 is executed to determine it as a medium-sized data volume.

[0093] Next, proceed to step S460 to determine the load rate of the task processing system.

[0094] Next, execute steps S470, S480, or S490 to split the task to be processed into multiple subtasks based on the load rate of the task processing system.

[0095] In some implementations, when the load rate is greater than a load rate threshold, the task to be processed is split into a first number of subtasks (e.g., 8-16). When the load rate is less than the load rate threshold, the task to be processed is split into a second number of subtasks (e.g., 2-7).

[0096] Figure 5 A flowchart illustrating a preferred embodiment of the task splitting of this disclosure is shown.

[0097] refer to Figure 5 In step S510, when the load rate of the task processing system is greater than the first load rate threshold, the task to be processed is split into a first number of subtasks.

[0098] In step S520, when the load rate is less than or equal to the first load rate threshold and greater than the second load rate threshold, the task to be processed is split into a second number of subtasks.

[0099] In step S530, when the load rate is less than the second load rate threshold, the task to be processed is split into a third number of subtasks.

[0100] The first load rate threshold is greater than the second load rate threshold.

[0101] For example, the first load rate threshold is 80%, and the second load rate threshold is 50%. The first number is greater than the second number, and the second number is greater than the third number. For example, the first number is 8-16, the second number is 4-7, and the third number is 2-3.

[0102] Refer again Figure 4 For example, when the load rate is less than 50%, the task is split into 2-3 subtasks. When the load rate is between 50% and 80% (including both endpoints), the task to be processed is split into 4-7 subtasks. When the load rate is greater than 80%, the task to be processed is split into 8-16 subtasks.

[0103] According to a preferred embodiment of this disclosure, the task to be processed is divided into multiple sub-tasks based on the data volume of the task and the load of the task processing system, including: When the amount of data to be processed is in the second data volume range, the task to be processed is divided into multiple subtasks based on the number of processing nodes, the memory capacity of the task processing system, and the processing time required for a single subtask.

[0104] The second data volume range has a larger data volume than the first data volume range. For example, the second data volume range has more than 100,000 data entries.

[0105] Memory capacity refers to the amount of memory in a task processing system, such as the amount of available memory. This disclosure does not specifically limit the method for determining the amount of available memory, and all such methods fall within the protection scope of this disclosure.

[0106] The processing time required for a single subtask refers to the estimated time needed to complete a single subtask.

[0107] For example, when the amount of data to be processed exceeds 100,000 records, such as Figure 4 As shown, step S491 is executed to determine the data volume of the task to be processed as a large-scale data volume.

[0108] For example, with 5 processing nodes, 512GB of available memory in the task processing system, and an estimated processing time of 30 minutes for a single subtask, the task to be processed is divided into 5 subtasks.

[0109] Figure 6 A flowchart illustrating task splitting according to one embodiment of this disclosure is shown.

[0110] In some embodiments of this disclosure, the task to be processed is divided into multiple subtasks based on the number of processing nodes, the memory capacity of the task processing system, and the processing time required for a single subtask, including steps S610 to S650.

[0111] In step S610, the first number of subtasks is determined based on the number of processing nodes.

[0112] In step S620, the data volume of a single subtask is determined based on the memory capacity of the task processing system, and a second number of subtasks is determined based on the data volume of a single subtask.

[0113] In step S630, a third number of subtasks is determined based on the processing time required for a single subtask.

[0114] In step S640, the final number of subtasks is determined based on the first quantity, the second quantity, and the third quantity.

[0115] In step S650, the tasks to be processed are split according to the final number of subtasks.

[0116] The first quantity is the number of subtasks determined based on the number of processing nodes.

[0117] The second number is the number of subtasks determined based on the memory capacity limitations of the task processing system.

[0118] The third quantity is the number of subtasks determined based on the processing time required for a single subtask.

[0119] The first quantity can be the number of subtasks determined for each processing node based on the number of processing nodes (e.g., each processing node is configured to process 3 subtasks).

[0120] The second quantity can be determined based on a preset percentage that does not exceed the available memory size of the task processing system. For example, no more than 20% of the available memory size of the task processing system could be used as the data volume for each subtask.

[0121] Refer again Figure 4Step S491: Determine the data volume of the task to be processed as a large-scale data volume. Then, execute step S492 (determine that each processing node processes 3 sub-tasks) to determine the first number described above (the number of sub-tasks determined based on the number of processing nodes); execute step S493 (determine that the data volume of each sub-task does not exceed 20% of the memory capacity (e.g., available memory capacity)) to determine the second number described above (the number of sub-tasks determined based on the memory capacity limit of the task processing system); execute step S494 (determine that the execution time required for a single sub-task does not exceed 30 minutes) to determine the third number described above (the number of sub-tasks determined based on the processing time required for a single sub-task).

[0122] Preferably, the minimum value among the first quantity, the second quantity, and the third quantity is determined as the final quantity of the subtask.

[0123] In the preferred embodiment of this disclosure, the constraints of the above three dimensions enable the entire task to be processed to be reasonably divided, avoiding the situation where the task cannot be processed or requires a long time to be processed.

[0124] It should be noted that, preferably, the data volume of each of the above sub-tasks is determined as follows: Optimal subtask data size = MIN(data size of tasks to be processed / (number of available processing nodes × 2), subtask data size with memory limit, subtask data size with time limit, maximum subtask data size).

[0125] The maximum subtask data size is the preset maximum data size for a subtask.

[0126] Preferably, the minimum value among the above values ​​is selected as the optimal subtask data volume.

[0127] In some embodiments of this disclosure, the task to be processed (i.e., the data processing request) is divided into multiple sub-tasks based on the data volume of the task to be processed and the load of the task processing system, including: When the amount of data for the task to be processed is in the second data volume range, determine the optimal subtask data volume, and split the task to be processed into multiple subtasks based on the optimal subtask data volume.

[0128] The task processing method disclosed herein also includes at least one of the following task decomposition optimization methods: The boundary values ​​for task splitting are determined based on the data range of the task to be processed (e.g., the data identifier range); the boundary values ​​for task splitting are determined based on the data order in the data list of the task to be processed; and the boundary values ​​for task splitting are determined based on the skewness of the data distribution of the task to be processed.

[0129] For example, the boundary values ​​for task splitting are determined based on the data range of the task to be processed (e.g., the data identifier range). The data is then divided into multiple consecutive ranges based on the data identifier range (ID range), with each range serving as a task shard or subtask. To ensure that the amount of data in each task shard is approximately equal, boundary values ​​need to be calculated to make the amount of data contained in each range as uniform as possible. For example, if the primary key ID of a data table ranges from 1 to 1000, and the total number of records is 1000, splitting it into 4 task shards or subtasks, such that each task shard or subtask has 250 data records, then the boundary values ​​would be 250, 500, 750, and 1000.

[0130] If the data identifiers are not continuous, determine the boundary values ​​based on the actual data identifiers to ensure that the amount of data in each task partition is uniform.

[0131] For example, when determining the boundary values ​​for task splitting based on the data order in the data list of tasks to be processed, if the query condition is a list, the task can be split according to the data order in the list.

[0132] For example, for a data list containing 1000 IDs, it is necessary to split it into 4 shards, dividing the data list into 4 segments, each containing 250 IDs. The boundary values ​​are 250, 500, 750, and 1000.

[0133] For example, when determining the boundary value for task splitting based on the skewness of the data distribution of the tasks to be processed, the skewness of the data distribution refers to the fact that the amount of data in some task fragments is greater than the amount of data in other task fragments, or that the data processing difficulty of some task fragments is greater than that of other task fragments. When splitting tasks, by controlling the skewness of the data distribution, hot data (such as frequently accessed data) is avoided from being concentrated in the same task fragment, thereby avoiding the difficulty of processing that task fragment.

[0134] Continue to refer to Figure 4 In some embodiments of this disclosure, when the amount of data for the task to be processed is within a third data volume range (e.g., less than 10,000 records), such as Figure 4 As shown, step S430 is executed, the data volume of the task to be processed is determined to be a small-scale data volume, and step S440 is executed, without splitting the task to be processed.

[0135] For tasks with small datasets (e.g., fewer than 10,000 records), task splitting would result in scheduling overhead accounting for a significant proportion of the total processing time. The process of task splitting, allocation, and coordination itself consumes system resources. Therefore, this implementation adopts a strategy of not splitting small-scale data tasks, allowing them to be processed entirely by a single processing node. This approach avoids unnecessary scheduling overhead and improves system resource utilization and task processing efficiency.

[0136] In the implementation methods described above, after splitting the task to be processed into subtasks, the task management layer 310 puts the split subtasks into a task pool (e.g., a Redis in-memory database).

[0137] Preferably, each processing node can retrieve a subtask from the task pool through a task preemption mechanism and call the corresponding data processor (business processing logic) to execute the subtask.

[0138] Continue to refer to Figure 3 The task management layer 310 stores the subtasks in the Redis (in-memory database) cluster in the infrastructure layer 380. The task management layer 310 generates an event message to notify each processing node that there are subtasks that need to be processed, and stores the event message in the message queue of the infrastructure layer 380.

[0139] The processing node acts as a consumer, pulling (reading and removing) this event message from the message queue. Based on the time message, the processing node attempts to preempt the corresponding subtask from the Redis cluster.

[0140] In existing methods, task processing systems can use polling or random allocation strategies to assign subtasks, but they cannot intelligently allocate tasks based on the actual load and processing capacity of the processing nodes. This results in some processing nodes being overloaded and processing subtasks slowly, while other processing nodes are idle, leading to significant resource waste.

[0141] This disclosure achieves dynamic allocation of subtasks through a "task release-preemption" mechanism.

[0142] After the task management layer 310 stores the split subtasks into the Redis task pool, it publishes a task availability event to all processing nodes via a message queue. Each processing node, acting as a consumer, actively retrieves the event from the message queue. Upon learning that a new task is available, it immediately initiates a task preemption request to the Redis task pool. The preemption process is implemented using a Redis distributed lock, ensuring that only one processing node acquires the same subtask. This mechanism avoids the single point of failure of centralized allocation and achieves decentralized task scheduling. Each processing node actively acquires tasks based on its idle state, naturally achieving load balancing and effectively solving the problem that traditional round-robin or random allocation strategies cannot intelligently allocate tasks based on the actual load of nodes. Simultaneously, the task preemption mechanism ensures efficient utilization of system resources, allowing idle nodes to quickly acquire tasks and preventing resource idleness.

[0143] Figure 7 A flowchart illustrating task allocation according to a preferred embodiment of this disclosure is shown.

[0144] S240. Assign the subtask to the processing node for processing, including steps S710 and S720.

[0145] In step S710, the matching score of the processing node for different subtasks is determined based on the matching degree between the subtask and the processing node, the priority of the subtask, the load rate of the processing node, and the historical execution score of the processing node.

[0146] In step S720, subtasks are assigned to the processing node according to the matching scores of the processing node corresponding to different subtasks.

[0147] In step S710, preferably, the matching score between the processing node and the subtask is determined based on at least one of the following evaluation strategies: (1) Matching degree of processing node-subtask: determined according to the association between subtask and processing node, the association is inherited from the node binding strategy when the task to be processed is created; (2) Subtask priority: dynamically graded according to the urgency of subtask, the higher the priority, the greater the weight of resource scheduling; (3) Processing node load rate: quantified based on real-time resource occupancy indicators to determine the busy status of processing node; (4) Historical execution score: reflects the reliability and efficiency of long-term task processing of processing node.

[0148] In step S720, subtasks are dynamically allocated based on the matching score: The task management layer 310 generates scores based on the weights of various dimensions and prioritizes assigning subtasks to processing nodes with high matching scores.

[0149] Alternatively, the processing node can proactively acquire tasks through a preemption mechanism, whereby the preemption is based on a distributed lock to ensure the atomicity of task allocation.

[0150] The allocation mechanism disclosed herein achieves dual optimization of load balancing and resource utilization by dynamically balancing resource load and task characteristics, thus avoiding the node overload or idleness problems caused by traditional polling strategies.

[0151] In step S720, subtasks are assigned according to the matching scores of the processing nodes corresponding to different subtasks, preferably including the following collaborative steps: (1) Task selection driven by matching score: Based on the matching score calculated in step S710, each processing node prioritizes the subtask with the highest matching score for preemption; or only preempts the subtask with a matching score exceeding the preset matching score threshold (to ensure efficient use of resources). (2) Atomic allocation guaranteed by task lock: When a processing node selects a target subtask, it locks the subtask through a distributed lock mechanism to ensure that the same subtask is only acquired by a single processing node.

[0152] Task locks can be implemented based on distributed locks such as Redis, and this disclosure does not impose any particular restrictions on them.

[0153] The above-mentioned collaborative steps disclosed herein organically integrate matching scores and task locks, thereby achieving both intelligent scheduling (dynamic allocation based on matching scores) and ensuring the reliability of task allocation (avoiding conflicts through task locks).

[0154] Figure 8 A schematic diagram illustrating the subtask preemption interaction of one embodiment of this disclosure is shown.

[0155] Specifically, processing node A executes step S810: sending a request to Redis (in-memory database) to retrieve the list of subtasks. After receiving the request, Redis executes step S820: returning the list of available subtasks in the current task pool to processing node A.

[0156] After receiving the list of subtasks, processing node A calculates the matching score between each subtask and itself, based on the multi-dimensional evaluation mechanism described above. Then, processing node A selects the subtask with the highest matching score for preemption.

[0157] Subsequently, processing node A executes step S830: sending a request to Redis to preempt a specific subtask, the request containing the unique identifier of the target subtask. When sending the preemption request, processing node A attempts to acquire exclusive access to the subtask using Redis's distributed lock mechanism to prevent other processing nodes from simultaneously preempting the same subtask.

[0158] When Redis receives a preemption request, it executes step S840: Checking whether the target subtask has been locked by another processing node. If the subtask has not been locked, Redis will successfully acquire the distributed lock and send a preemption success notification to processing node A; if the subtask has been locked by another processing node, Redis will send a preemption failure notification to processing node A.

[0159] After receiving the successful preemption notification, processing node A executes step S850: update the subtask status. Preferably, processing node A uses an optimistic locking mechanism in the database to update the subtask status from "pending execution" to "preempted and waiting to be executed" in the task status database. If the database update operation is successful, processing node A begins executing the subtask; if the database update operation fails (usually because other processing nodes have already updated the status of the same subtask), processing node A releases the Redis distributed lock, records the preemption failure log, and then attempts to preempt other subtasks.

[0160] After the database status update is successful, execute step S860: send a confirmation notification of successful status update to processing node A. At this time, the subtask status is marked as "in execution" in the database.

[0161] To illustrate the scenario of a failed preemption, assume that processing node B also attempts to preempt the same subtask.

[0162] Processing node B executes step S870: sends a request to Redis to retrieve the subtask list. Upon receiving this request, Redis executes step S880: returns the subtask list to processing node B. Processing node B executes step S890: sends a request to Redis to preempt the same subtask. Since the subtask has already been successfully locked by processing node A, Redis executes step S891: sends a notification of preemption failure to processing node B. Processing node B can then choose other available subtasks to preempt.

[0163] This implementation method achieves intelligent task allocation based on a multi-factor scoring mechanism that considers the matching degree between processing nodes and subtasks, the load status of processing nodes, and the priority of subtasks.

[0164] By employing a dual control mechanism of Redis distributed locks and database optimistic locking, the atomicity of the task preemption process and data consistency are guaranteed. Simultaneously, by dynamically considering the load status of processing nodes, a balanced distribution of system load is achieved, effectively preventing some nodes from being overloaded while others are idle. Parallel processing of subtasks by multiple processing nodes significantly improves the utilization efficiency of the system's memory and processor resources.

[0165] In practical task processing systems, traditional business processes are often tightly coupled with specific business logic, making the system difficult to expand and maintain. When new business processes need to be added, developers must modify a large amount of core code, and the state management of these processes is complex and prone to errors. Furthermore, coordination between different business processes is difficult, new business deployment cycles are long, development efficiency is low, system maintenance is complex, code vulnerability fixing is difficult, and business expansion is severely limited.

[0166] To address the aforementioned issues, this disclosure decouples the business process from the business processing logic. Specifically, subtasks are executed according to predefined business processing logic, which is then configured based on specific business requirements. The corresponding business process is determined based on these requirements, and this process consists of multiple processing nodes with a clearly defined execution order.

[0167] Figure 9 A schematic diagram of the process node triggering in one embodiment of this disclosure is shown.

[0168] refer to Figure 9Preferably, in some embodiments, the task processing method of this disclosure further includes: S910. The business processing system first triggers the first processing node and executes the business processing logic corresponding to the first processing node to process the subtasks assigned to the first processing node. S920. When the subtask assigned to the first processing node is completed, the business processing system updates the subtask status and publishes a subtask completion event. S930. The business processing system determines the second processing node that should be executed after the first processing node based on the completion event, and triggers the second processing node to execute the corresponding business processing logic.

[0169] This decoupled design gives the business processing system high flexibility and scalability. When adding or modifying business processes, only the process definition and corresponding business processing logic need to be adjusted, without modifying the core task processing framework.

[0170] Business process status management becomes simpler and more reliable, coordination between different businesses becomes smoother, the launch cycle of new businesses is significantly shortened, system maintenance becomes more convenient, and business expansion capabilities are greatly enhanced.

[0171] The business process decoupling design is achieved through the collaboration between the service orchestration layer 320 and the task management layer 310: after the task management layer 310 completes the subtask allocation, the service orchestration layer 320 automatically manages the node flow based on the event-driven mechanism.

[0172] Specifically, after the first processing node is triggered (S910) and the subtask is completed, the task management layer 310 updates the subtask status in real time and publishes a completion event through the event bus (S920). After the service orchestration layer 320 listens to the event, it immediately verifies the preconditions of the next node. When the conditions are met, the second processing node is automatically triggered to execute (S930).

[0173] This mechanism enables the data processor of the business processing logic layer 350 to be unaware of the process state and only need to implement standardized interfaces to process subtasks; at the same time, the service orchestration layer 320 manages the node state transitions in a unified manner through a centralized state machine to ensure strict adherence to the execution order.

[0174] When a subtask fails to process, the task management layer 310 automatically makes a decision based on the type of exception: if the exception is retryable, the task is rescheduled; if the exception is not retryable, the service orchestration layer 320 terminates the process and issues an alert.

[0175] This decoupled design completely solves the problem of tight coupling between business processes and processing logic in traditional systems. Adding new business processes only requires defining the node sequence and corresponding data processors, without modifying the core framework, which significantly improves system scalability and maintenance efficiency.

[0176] Figure 10 A schematic diagram of a processing node transition according to one embodiment of this disclosure is shown.

[0177] like Figure 10 As shown, business process instance 1000 executes each processing node sequentially according to the predefined execution order, forming a complete business processing chain.

[0178] In this embodiment, when the service orchestration layer 320 triggers the "data synchronization" processing node, the task management layer 310 assigns the corresponding subtasks to the processing node for execution.

[0179] refer to Figure 9 When a subtask assigned to the "data synchronization" processing node is completed, the task management layer 310 will update the subtask status and publish a subtask completion event.

[0180] After the service orchestration layer 320 detects the completion event, it verifies the preconditions of the "tax calculation" processing node (such as whether the data synchronization result is valid). If the conditions are met, the "tax calculation" processing node is automatically triggered. Similarly, after the "tax calculation" is successfully processed, "declaration submission" is triggered, and after the "declaration submission" is successfully processed, "tax payment processing" is triggered, until all processing nodes are completed and the process enters the "complete" state.

[0181] When any processing node fails to execute (e.g., subtask execution timeout, data anomaly, or system error), the task management layer 310 will capture the exception and record the exception type, and at the same time transfer the process status to the "Exception Handling" node.

[0182] During the exception handling process, the system determines the handling strategy based on the exception type: for retryable exceptions (such as temporary network failures, temporary resource shortages, etc.), the system will perform a "rescheduling" operation, and the process will return to the "data synchronization" node to start processing again; for non-retryable exceptions (such as data format errors, business rule conflicts, etc.), the system will send the exception information to the "alarm system" to request manual intervention.

[0183] It is worth noting that the "rescheduling" mechanism works closely with the task splitting and allocation strategies described above.

[0184] When rescheduling is required, the system does not simply repeat the entire process. Instead, it uses the status tracking mechanism of the task management layer 310 to identify successfully executed processing nodes (such as "data synchronization" already completed) and resumes execution directly from the node preceding the failed node, avoiding the duplication of completed steps. For example, if the "declaration submission" node fails, rescheduling will start from the "tax calculation" node instead of restarting from the "data synchronization" node, thereby improving recovery efficiency.

[0185] This implementation achieves reliable execution and automatic recovery of business processes through its intelligent node switching mechanism and anomaly handling strategy. When the system detects an anomaly, it can automatically handle recoverable anomalies and promptly notify manual intervention for unrecoverable anomalies, ensuring the integrity of business processes and data consistency.

[0186] Figure 11 A schematic diagram illustrating the transition of task execution states according to one embodiment of this disclosure is shown.

[0187] Figure 11 The state transition mechanism of the subtask execution state shown is tightly integrated with the event-driven architecture of the task management layer 310, together forming the core support for the reliable execution of business processes.

[0188] The lifecycle of a subtask begins in the "Pending Execution" state, indicating that the task has been created but not yet assigned. Once a processing node successfully acquires the subtask through the preemption mechanism, its state automatically changes to "Preempted and Awaiting Execution," indicating that the subtask has been assigned to the processing node but has not yet started execution. When a processing node begins executing the subtask, its state enters "Executing," representing that the task is being processed.

[0189] In the "Executing" state, the system branches its status based on the execution result: if the subtask completes successfully, the status changes to "Execution Successful," indicating that the task result is valid and the process can proceed; if execution fails but the retry conditions are met (such as a temporary network failure), the status changes to "Execution Failed, Waiting to Re-execute," triggering the retry mechanism. Subsequently, the subtask returns to the "Pending Execution" state through the "Release" operation, re-enters the task pool to wait for preemption, and achieves automatic retry.

[0190] The state transition process is monitored in real time by the task management layer through distributed locks and an event bus to ensure the atomicity and consistency of state changes.

[0191] This state transition mechanism is organically integrated with the business process decoupling mechanism described above: the task management layer continuously tracks the subtask status, and when a subtask enters the "execution successful" state, it publishes a completion event, driving the service orchestration layer to automatically trigger the next processing node (e.g., ...). Figure 9 , Figure 10 As shown); when entering the retry path, the task splitting strategy is combined ( Figures 4 to 6 It can intelligently adjust the granularity of subtasks to avoid repeatedly executing completed steps.

[0192] This disclosure embeds state transitions into an event-driven framework, enabling precise scheduling of tasks in a distributed environment that avoids duplication and loss. It also provides fundamental support for exception rollback and log tracing, effectively addressing the problem of insufficient fault tolerance in traditional systems.

[0193] Figure 12 A schematic diagram of the operation status monitoring process of a task processing system according to one embodiment of the present disclosure is shown.

[0194] The task processing method disclosed herein also includes: S1210. Determine the running status of the task processing system based on at least one of the resource usage information of the task processing system, the status information of the middleware, and the processing status of the processing nodes of the task processing system. S1220. When the operating status is detected to meet the first condition, the operation of the processing node is paused; S1230. When the operating status is detected to meet the second condition, the operation of the processing node is not suspended or the operation of the suspended processing node is resumed.

[0195] In this embodiment, the task processing system monitors the management layer 370 (reference). Figure 3 It collects system operation data in real time and works in collaboration with the task management layer 310 to achieve adaptive system control.

[0196] Preferably, the task management system performs the following steps: S1210: Determine the operational status of the task processing system. The monitoring and management layer 370 preferably dynamically assesses the system's health status based on three types of indicators: (1) Basic resource indicators: CPU utilization, memory utilization, disk space and network connection status; (2) Middleware status: database connection pool status, Redis connection status, and message queue backlog status; (3) Business processing status: task execution success rate, abnormal task ratio and average task execution time.

[0197] The system monitors these metrics to assess its operational status in real time. Preferably, if any metric remains abnormal (e.g., CPU usage exceeds a threshold for 5 minutes), it indicates that the system is operating abnormally.

[0198] S1220: When the operating status is detected to meet the first condition, the operation of the processing node is suspended. The first condition refers to any monitoring indicator continuously exceeding a preset threshold. The monitoring management layer 370 sends a pause command to the task management layer 310, and the task management layer 310 performs the following operations: Stop the node from accepting new subtasks; Allow the currently executing subtask to complete; Release the tasks that have been seized but not yet executed.

[0199] S1230: When the operating status is detected to meet the second condition, resume the operation of the processing node.

[0200] The second condition refers to all monitored indicators returning to normal and remaining stable. After the monitoring management layer 370 confirms that the system status is normal, it notifies the task management layer 310 to gradually restore processing capacity. Gradually increase task concurrency; The task processing channel has been reopened.

[0201] The processing mechanism in this implementation is similar to the task preemption mechanism described above (see reference). Figure 8 ) and subtask allocation mechanism (refer to Figure 7 Close coordination: When the system pauses processing nodes, subtasks in the task pool will not be preempted temporarily; when the system resumes, processing nodes will re-participate in subtask preemption to ensure the rational use of system resources.

[0202] The processing mechanism in this implementation is decoupled from the business process design described above. Figures 9 to 10 Collaborative work: When a processing node is paused, the business process automatically enters a waiting state; when the processing node resumes, the business process automatically continues to execute without manual intervention.

[0203] This implementation method can be... Figure 13 The exemplary structured health check system enables accurate determination and automated control of system operating status, effectively avoiding cascading failures caused by system overload and ensuring the stable operation of the task processing system.

[0204] Figure 14 This is a schematic block diagram of an electronic device 1400 according to one embodiment of the present disclosure.

[0205] The hardware architecture of electronic device 1400 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1410 connects various circuits, including one or more processors 1420, memory 1430, and / or hardware modules. Bus 1410 can also connect various other circuits 1440, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0206] Bus 1410 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0207] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0208] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0209] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0210] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0211] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0212] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0213] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0214] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0215] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0216] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A task processing method, characterized in that, Applied to a task processing system, the task processing system including multiple processing nodes, the task processing method includes: Create a task to be processed; Determine the amount of data for the task to be processed; Based on the data volume of the task to be processed and the load of the task processing system, the task to be processed is divided into multiple sub-tasks; and The subtask is assigned to the processing node for processing.

2. The task processing method according to claim 1, characterized in that: The step of dividing the task to be processed into multiple sub-tasks based on the data volume of the task to be processed and the load of the task processing system includes: When the amount of data in the task to be processed is in the first data volume range, the task to be processed is divided into multiple sub-tasks according to the load rate of the task processing system.

3. The task processing method according to claim 2, characterized in that, The step of dividing the task to be processed into multiple sub-tasks based on the load rate of the task processing system includes: When the load rate is greater than the first load rate threshold, the task to be processed is split into a first number of subtasks; When the load rate is less than or equal to the first load rate threshold and greater than the second load rate threshold, the task to be processed is split into a second number of sub-tasks; When the load rate is less than the second load rate threshold, the task to be processed is split into a third number of subtasks; The first load rate threshold is greater than the second load rate threshold, the first quantity is greater than the second quantity, and the second quantity is greater than the third quantity.

4. The task processing method according to claim 1 or 2, characterized in that, The step of dividing the task to be processed into multiple sub-tasks based on the data volume of the task to be processed and the load of the task processing system includes: When the amount of data in the task to be processed is in the second data range, the task to be processed is divided into multiple subtasks according to the number of processing nodes, the memory capacity of the task processing system, and the processing time required for a single subtask.

5. The task processing method according to claim 4, characterized in that, The task to be processed is divided into multiple subtasks based on the number of processing nodes, the memory capacity of the task processing system, and the processing time required for a single subtask, including: The first number of subtasks is determined based on the number of processing nodes; The data volume of a single subtask is determined based on the memory capacity, and a second number of subtasks is determined based on the data volume of the single subtask. The third number of subtasks is determined based on the processing time required for each individual subtask; and The final number of the sub-tasks is determined based on the first number, the second number, and the third number.

6. The task processing method according to claim 5, characterized in that, Determining the final number of the sub-tasks based on the first quantity, the second quantity, and the third quantity includes: The minimum of the first quantity, the second quantity, and the third quantity is determined as the final quantity.

7. The task processing method according to claim 1, characterized in that, The step of assigning the subtask to the processing node for processing includes: The matching score of a processing node for different subtasks is determined based on the matching degree between the subtask and the processing node, the priority of the subtask, the load rate of the processing node, and the historical execution score of the processing node; and Subtasks are assigned to the processing nodes based on their matching scores for different subtasks.

8. The task processing method according to claim 7, characterized in that, The step of assigning subtasks to the processing node based on the matching scores of the processing node corresponding to different subtasks includes: The corresponding subtask is selected based on the matching score of the processing node for different subtasks; The corresponding subtask is assigned according to the task lock, which is used to assign the corresponding subtask to the processing node.

9. The task processing method according to claim 1, characterized in that, The subtasks are executed according to the business processing logic, which is set according to business requirements. The business process corresponding to the business requirement has multiple processing nodes, and there is an execution order relationship among the multiple processing nodes; The task processing method includes: Trigger the first processing node and execute the business processing logic corresponding to the first processing node to process the subtasks assigned to the first processing node; In response to the completion of the subtask assigned to the first processing node, the subtask status of the subtask is updated and a subtask completion event is published; Based on the subtask completion event, determine the second processing node that should be executed after the first processing node, and trigger the second processing node to execute the business processing logic corresponding to the second processing node to process the subtask assigned to the second processing node.

10. The task processing method according to claim 1, characterized in that, The task processing system also includes middleware that interacts with the processing nodes; The task processing method further includes: The running status of the task processing system is determined based on at least one of the resource usage information of the task processing system, the status information of the middleware, and the processing status of the processing nodes of the task processing system. When the operating status is detected to meet the first condition, the operation of the processing node is paused; If the operating status is detected to meet the second condition, the operation of the processing node is not suspended or the operation of the suspended processing node is resumed.

11. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the task processing method according to any one of claims 1 to 10.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the task processing method according to any one of claims 1 to 10.

Images