Distributed task parallel control method and device, equipment and storage medium

Abstract

The invention provides a distributed task parallel control method and device, equipment and a storage medium, and the method comprises the steps: pre-defining concurrency control parameters corresponding to different types of task steps; before any task step is executed, according to the concurrency control parameters corresponding to the type of the task step, two-stage permission checking is conducted on the concurrency number, the first-stage checking is used for controlling the concurrency degree of the task step on the current service node layer, and the second-stage checking is used for controlling the concurrency degree of the task step on the whole distributed cluster layer; if the two-stage check is passed, occupying a concurrent permission and starting to execute the task step; after the task step is executed, the concurrent permission occupied by the task step is released. By means of the method, the resource utilization rate and safety of task processing can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, device, and storage medium for distributed task parallel control. Background Technology

[0002] With the development of distributed computing technology, multi-step distributed tasks are increasingly being used in scenarios such as data annotation and legal document processing within enterprises. These tasks require parallel execution across multiple nodes to improve processing efficiency. However, different task steps consume significantly different amounts of system resources such as CPU, memory, and database connections, placing higher demands on the precision of concurrency control.

[0003] In existing technologies, distributed task parallel control often adopts a task-level coarse-grained rate limiting scheme, which limits the number of concurrent executions at the node level by configuring a fixed maximum number of parallel tasks for each service node (e.g., a single node can execute a maximum of 3 tasks at the same time), thereby avoiding excessive load on a single node.

[0004] The existing technology has obvious defects: because it does not take into account the differences in resource consumption characteristics of different task steps, and uniformly limits concurrency based on the task as a whole, it is easy to cause resource utilization imbalance. Either resources are idle when low-consumption steps are executed, or nodes are overloaded or backend shared resources are overwhelmed when high-consumption steps are run in a concentrated manner, making it impossible to achieve efficient adaptation and rational utilization of system resources. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a distributed task parallel control method, apparatus, device and storage medium that can improve the resource utilization and security of task processing.

[0006] In a first aspect, embodiments of this application provide a distributed task parallel control method, the method comprising: Predefine the corresponding concurrency control parameters for different types of task steps; Before executing any task step, a two-level permission check is performed on the concurrency number according to the concurrency control parameters corresponding to its type. The first level check is used to control the concurrency of the task step at the current service node level, and the second level check is used to control the concurrency of the task step at the entire distributed cluster level. If both levels of checks pass, a concurrent permit is acquired and the task step begins execution; After the task step is completed, the concurrent license it occupied is released.

[0007] Optionally, the concurrency control parameters include the maximum number of parallel operations per node and the maximum number of parallel operations for the entire cluster; The two-level permission check for concurrency includes: Perform the first-level check to determine whether the number of concurrent executions of the same type of steps on the current service node has reached its maximum number of concurrent executions per node; If not, the second-level check is performed to determine whether the total number of concurrent executions of the same type of steps in the entire distributed cluster has reached its maximum number of parallel executions.

[0008] Optionally, the two-level permission check is implemented based on a distributed counter, including: The value of the distributed counter is queried and updated using atomic operations; The concurrent permission to occupy includes an atomically incrementing counter; The release of concurrent permissions includes atomically decrementing the corresponding counter; The distributed counters include node-level counters and cluster-level counters; The key of the node-level counter is associated with the current service node identifier and the task step type, and is used for the first-level check; The key of the cluster-level counter is associated with the task step type and is used for the second-level check.

[0009] Optionally, the two-level permission check is implemented based on the token bucket algorithm, including: Query and update the token bucket status using atomic operations; The concurrent permission to occupy includes atomically obtaining a token from the corresponding token bucket; The release of concurrent permissions includes atomically returning a token to the corresponding token bucket; or, The two-level permission check is implemented based on the leaky bucket algorithm, including: Query and update the status of the leaky bucket using atomic operations; The concurrent permission occupancy includes the request to atomically inject a request into the corresponding leaky bucket; The release of concurrent permissions includes freeing up capacity in the leaky bucket after it has completed request processing at a constant rate.

[0010] Optionally, a task scheduling step may be included before executing any task step: The task steps to be executed are distributed to different message queues based on their resource consumption characteristics or preset priorities. The task execution node listens to the corresponding message queue and prioritizes consuming messages in the high-priority queue to trigger the two-level permission check.

[0011] Optionally, after performing a two-level permission check on the number of concurrent connections based on the concurrency control parameters corresponding to their types, the method further includes: If the two-level checks fail, the execution of the task step is delayed, and the two-level permission check for the task step is re-initiated after a preset delay time. Alternatively, the task step can be marked as pending retry, and the two-level permission check can be re-initiated according to a predetermined strategy.

[0012] Optionally, during the execution of the task steps, the method further includes: Based on the total amount of data to be processed and the amount of data already processed in this task step, the processing progress and estimated remaining time are calculated in real time. The processing progress and the estimated remaining time can be provided to external users through a query interface or displayed visually.

[0013] Secondly, embodiments of this application provide a distributed task parallel control device, the device comprising: The control parameter definition module is used to predefine the corresponding concurrency control parameters for different types of task steps; The two-level permission check module is used to perform a two-level permission check on the concurrency number according to the concurrency control parameters corresponding to its type before executing any task step. The first level check is used to control the concurrency of the task step at the current service node level, and the second level check is used to control the concurrency of the task step at the entire distributed cluster level. The task step execution module is used to occupy a concurrent license and start executing the task step if both levels of checks pass. The concurrent license release module is used to release the concurrent license it occupies after the task step is completed.

[0014] Optionally, the concurrency control parameters include the maximum number of parallel operations per node and the maximum number of parallel operations for the entire cluster; The two-level permission check for concurrency includes: Perform the first-level check to determine whether the number of concurrent executions of the same type of steps on the current service node has reached its maximum number of concurrent executions per node; If not, the second-level check is performed to determine whether the total number of concurrent executions of the same type of steps in the entire distributed cluster has reached its maximum number of parallel executions.

[0015] Optionally, the two-level permission check is implemented based on a distributed counter, including: The value of the distributed counter is queried and updated using atomic operations; The concurrent permission to occupy includes an atomically incrementing counter; The release of concurrent permissions includes atomically decrementing the corresponding counter; The distributed counters include node-level counters and cluster-level counters; The key of the node-level counter is associated with the current service node identifier and the task step type, and is used for the first-level check; The key of the cluster-level counter is associated with the task step type and is used for the second-level check.

[0016] Optionally, the two-level permission check is implemented based on the token bucket algorithm, including: Query and update the token bucket status using atomic operations; The concurrent permission to occupy includes atomically obtaining a token from the corresponding token bucket; The release of concurrent permissions includes atomically returning a token to the corresponding token bucket; or, The two-level permission check is implemented based on the leaky bucket algorithm, including: Query and update the status of the leaky bucket using atomic operations; The concurrent permission occupancy includes the request to atomically inject a request into the corresponding leaky bucket; The release of concurrent permissions includes freeing up capacity in the leaky bucket after it has completed request processing at a constant rate.

[0017] Optionally, the device further includes a task scheduling module, used to perform a task scheduling step before executing any task step: The task steps to be executed are distributed to different message queues based on their resource consumption characteristics or preset priorities. The task execution node listens to the corresponding message queue and prioritizes consuming messages in the high-priority queue to trigger the two-level permission check.

[0018] Optionally, the apparatus further includes a task step delayed execution module, used for: After performing a two-level permission check on the number of concurrent requests based on the concurrency control parameters corresponding to its type, if the two-level checks fail, the execution of the task step is delayed, and the two-level permission check on the task step is re-initiated after a preset delay time. Alternatively, the task step can be marked as pending retry, and the two-level permission check can be re-initiated according to a predetermined strategy.

[0019] Optionally, the device further includes a visualization module for: During the execution of the task step, the processing progress and estimated remaining time are calculated in real time based on the total amount of data to be processed and the amount of data already processed in the task step. The processing progress and the estimated remaining time can be provided to external users through a query interface or displayed visually.

[0020] Thirdly, embodiments of this application provide a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the distributed task parallel control method described in any of the optional embodiments of the first aspect are performed.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the distributed task parallel control method described in any of the optional embodiments of the first aspect.

[0022] The technical solution provided in this application includes, but is not limited to, the following beneficial effects: By predefining corresponding concurrency control parameters for different types of task steps, differentiated adaptation of concurrency control can be achieved, breaking the limitations of coarse-grained task-level control in existing technologies. By setting specific parameters for task steps with different resource consumption characteristics, a clear basis is provided for subsequent precise concurrency control, avoiding the insufficient adaptability problem caused by a "one-size-fits-all" control method.

[0023] Performing node and cluster-level permission checks before executing any task step ensures both local load and global resource protection. The first-level node-level check effectively prevents individual service nodes from becoming overloaded due to concentrated execution of high-consumption steps, while the second-level cluster-level check prevents concurrent access from multiple nodes from impacting backend shared resources. This dual protection enhances the stability of system operation.

[0024] If both levels of checks pass, the concurrent license is used and the task steps are executed, ensuring the legality and security of concurrent execution. Execution only starts if both node and cluster concurrency limits are met, avoiding resource contention and service lag caused by exceeding limits and ensuring smooth task execution.

[0025] Releasing the concurrent licenses used by a task step after its execution enables efficient and cyclical use of resources. The released licenses can then be promptly acquired by other pending task steps, preventing idle waste caused by prolonged resource occupation and improving overall system resource utilization and task throughput.

[0026] In summary, the invention, through a complete process of "predefined parameters - two-level checks - license occupancy - license release," not only solves the problem of resource imbalance in the prior art, but also achieves dual stability assurance for both local and global aspects, while improving resource utilization and task processing security, and comprehensively optimizing the effect of distributed task parallel control.

[0027] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart of a distributed task parallel control method provided in Embodiment 1 of this application is shown; Figure 2 A flowchart of a two-level license inspection method provided in Embodiment 1 of this application is shown; Figure 3 A flowchart of the second two-level license inspection method provided in Embodiment 1 of this application is shown; Figure 4 A flowchart of the third two-level license inspection method provided in Embodiment 1 of this application is shown; Figure 5 A flowchart of a two-level license check triggering method provided in Embodiment 1 of this application is shown; Figure 6 A flowchart of a visualization method provided in Embodiment 1 of this application is shown; Figure 7 This paper shows a schematic diagram of the structure of a distributed task parallel control device provided in Embodiment 2 of this application; Figure 8 A schematic diagram of the structure of a computer device provided in Embodiment 3 of this application is shown. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0031] Example 1 To facilitate understanding of this application, the following is combined with... Figure 1 The flowchart of the distributed task parallel control method provided in Embodiment 1 of this application illustrates Embodiment 1 in detail.

[0032] See Figure 1 As shown, Figure 1 A flowchart of a distributed task parallel control method provided in Embodiment 1 of this application is shown, wherein the method includes steps S101 to S103: S101: Predefine the corresponding concurrency control parameters for different types of task steps.

[0033] Specifically, the core application scenario of this application focuses on the processing of data annotation tasks within enterprises. In this scenario, tasks are often broken down into multiple steps with significantly different resource consumption. Predefined operations need to be completed in an independent configuration center. The configuration center supports online modification, saving, and real-time synchronization of parameters, ensuring that parameter adjustments take effect without restarting the system.

[0034] The step types need to cover the key stages of the entire task process. In addition to common ones such as "document renaming" (IO-intensive, IO stands for Input / Output), "md to excel" (CPU-intensive, CPU stands for Central Processing Unit), and "synchronize es" (database write-intensive, ES stands for Elasticsearch), it also includes "data format validation" (lightweight CPU type) and "annotation result review" (IO + database type). Each type needs to be bound to two core concurrency control parameters to achieve differentiated rate limiting.

[0035] Parameter configuration needs to be precisely set according to the resource consumption characteristics of each step: for example, the "Document Renaming" step has low IO consumption and low CPU usage, so the maximum parallelism per node can be set to 8 and the maximum parallelism of the entire cluster can be set to 30; the "md to excel" step has high CPU computation, so the maximum parallelism per node can be set to 2 and the maximum parallelism of the entire cluster can be set to 10; the "Synchronize ES" step requires frequent writing to the database, so the maximum parallelism per node can be set to 3 and the maximum parallelism of the entire cluster can be set to 15 to avoid overloading the backend database.

[0036] S102: Before executing any task step, a two-level permission check is performed on the concurrency number according to the concurrency control parameters corresponding to its type. The first-level check is used to control the concurrency of the task step at the current service node level, and the second-level check is used to control the concurrency of the task step at the entire distributed cluster level.

[0037] Specifically, the two-level checks strictly follow the fixed order of "single node first, then cluster". This order design is a core improvement to address the existing technology's "lack of global view" and "easy to cause system bottlenecks". It solves the problem of excessive load on a single node and avoids the risk of multiple nodes overwhelming shared resources.

[0038] The first level of inspection focuses on the local load of a single service node to prevent CPU and memory resources from being exhausted due to a single node running multiple high-consumption steps (such as three "md to excel" steps), resulting in service lag or crashes. The second level of inspection focuses on the cluster's globally shared resources (such as a database or Elasticsearch engine deployed on a single node) to prevent multiple nodes (such as five nodes each running three "synchronize es" steps) from accessing the cluster at high concurrency, which could lead to backend service response timeouts or crashes.

[0039] The inspection process relies on Redis (Remote Dictionary Server) as a distributed storage medium. Redis needs to be deployed in a multi-node cluster mode (at least 3 masters and 3 slaves) to ensure high data availability and low query latency. Node-level and cluster-level counter data are stored in Redis in real time, and the query response time is controlled in milliseconds to avoid affecting task execution efficiency.

[0040] The server-client architecture supports the inspection process: the server is deployed on multiple physical servers or cloud servers and runs continuously as a background daemon process, with each server being an independent service node; the client provides a visual operation interface, through which users can initiate task start and stop commands, and monitor the execution progress, concurrency, and remaining time of each task step in real time. All operations of the client are converted into messages and pushed to the corresponding queue of RabbitMQ (Rabbit Message Queue) according to the task configuration. The server node consumes messages by listening to the queue and triggers subsequent inspection and execution processes.

[0041] S103: If both levels of checks pass, then a concurrent license is acquired and the task step is executed.

[0042] Specifically, the concurrent access permission is acquired through Redis's atomic increment operation (such as the INCR command). This requires simultaneously incrementing both the node-level counter and the cluster-level counter to ensure that the data of the two counters is updated synchronously, thus avoiding the inconsistency of "the node count has increased but the cluster count has not increased".

[0043] The implementation of atomic operations offers two options to adapt to different concurrency scenarios: Option 1 is a distributed lock mechanism, which acquires a distributed lock (the lock key format is "ines_lock_[step name]_[node number]") through the Redis SETNX command. After successfully acquiring the lock, it performs a check and increment operation. After the operation is completed, it releases the lock through the DEL command. If the lock acquisition fails, it waits for a preset time (such as 500 milliseconds) and then retryes. Option 2 is a Redis Lua (Lua Programming Language) script, which encapsulates the three-step operation of "reading the counter - judging the threshold - conditional increment" into a single script.

[0044] S104: After the task step is completed, release the concurrent license it occupies.

[0045] Specifically, releasing a permit is accomplished through Redis's atomic decrement operation (such as the DECR command). This requires synchronously decrementing both node-level and cluster-level counters to ensure that the counter values ​​match the actual number of concurrently executed task steps. The released permit can then be immediately acquired by subsequent task steps, improving resource utilization.

[0046] The system additionally deploys a background verification process, which executes verification logic every 30 seconds. It uses the Redis KEYS command to query all node-level counters (key prefixed with "ines_speedlimit_node_") and cluster-level counters (key prefixed with "ines_speedlimit_cluster_"), calculates the sum of all node-level counters under the same step type, and compares it with the corresponding cluster-level counter value. If the absolute value of the difference is greater than 1 (allowing for errors caused by slight network latency), an alarm message is sent to the technical contact via WeChat Work robot and email, including information such as step name, total node count, cluster count, and difference value, to facilitate manual anomaly investigation.

[0047] Redis clusters ensure high availability through master-slave replication combined with sentinel mode. If all Redis nodes fail, the system will immediately stop the execution of all task steps. Once the Redis cluster recovers, the system will automatically resume task execution.

[0048] In one optional implementation, the concurrency control parameters include the maximum number of parallel operations per node and the maximum number of parallel operations across the entire cluster.

[0049] Specifically, the maximum number of parallel operations per node is defined as the maximum number of steps of the same type of task that can be executed simultaneously on a single service node. This is used to limit the local load on a single node and prevent performance bottlenecks caused by resource exhaustion. The maximum number of parallel operations across the entire cluster is defined as the total number of steps of the same type of task that can be executed simultaneously in the entire distributed cluster. This is used to protect backend shared resources and prevent overload caused by concurrent access from multiple nodes.

[0050] The cluster threshold is set to "the sum of all node thresholds + fault tolerance redundancy". For example, if the cluster contains 5 service nodes, the single node threshold for the "synchronize ES" step is 3, and the total is 15, the cluster threshold can be set to 18, reserving 3 fault tolerance slots to avoid the normal task from failing to execute due to individual node counting errors (such as network latency causing failure to decrement in time), thus achieving traffic degradation.

[0051] See Figure 2 As shown, Figure 2 The flowchart of a two-level license check method provided in Embodiment 1 of this application is shown, wherein the two-level license check for concurrency includes steps S201 to S203: S201: Perform the first-level check to determine whether the number of concurrent executions of the same type of steps on the current service node has reached its maximum number of parallel executions per node.

[0052] Specifically, the core of the first-level check is to query the real-time concurrency of the current node. The value of the node-level counter is obtained through the Redis GET command. The key format of the counter is defined as "[fixed prefix]_[node number]:[step name]", where the fixed prefix is ​​uniformly set to "ines_speedlimit_node", the node number is the unique identifier of the service node (such as the last three digits of the server IP + port number, 12231), and the step name is a uniformly named step type (such as SyncEs, which means synchronizing Elasticsearch). A complete example is "ines_speedlimit_node_12231:SyncEs".

[0053] If the counter value is null (on the first execution of this step), the current concurrency is set to 0 by default. The real-time concurrency is compared with the predefined maximum parallelism of a single node. If it is greater than or equal to the threshold, the check fails and the task step enters the waiting queue or is marked for retry. If it is less than the threshold, the second-level check is performed. This design can limit the scope of the fault to the current node and avoid the entire cluster from being affected by the abnormal count of a single node.

[0054] S202: If not, perform the second-level check to determine whether the total number of concurrent executions of the same type of steps in the entire distributed cluster has reached its maximum number of parallel executions.

[0055] Specifically, the second level of check queries the value of the cluster-level counter using the Redis GET command. The key format is "[fixed prefix]:[step name]", and the fixed prefix is ​​uniformly set to "ines_speedlimit_cluster". A complete example is "ines_speedlimit_cluster:SyncEs". This key is unique in the entire cluster, ensuring that the total concurrency of the entire cluster is counted.

[0056] The design of performing node checks before cluster checks serves three core purposes: First, it distributes traffic pressure. The number of keys in the node-level counter is positively correlated with the number of nodes and the number of step types (e.g., 5 nodes + 10 step types = 50 node keys), which can distribute the access pressure on Redis. However, the cluster-level counter only has 10 keys. If cluster checks are performed first, a large number of requests will concentrate on accessing a small number of keys, causing hot key issues. Second, it strengthens risk control. If the node counting is wrong in extreme scenarios (e.g., exceeding the threshold without being intercepted), node checks can intercept it in advance, preventing the error from spreading to the cluster level. Third, it optimizes traffic degradation. When multiple tasks are started in batches, if cluster checks are performed first, a large number of tasks may enter the node check stage after passing the cluster check, causing a surge in node key access. Performing node checks first can filter some tasks in advance, ensuring the effectiveness of cluster checks.

[0057] If the cluster-level counter value is less than the maximum number of parallel operations in the entire cluster, both levels of checks pass; if it is greater than or equal to the threshold, the check fails, and the task step needs to wait or enter the retry queue.

[0058] In an optional implementation, the two-level permission check is implemented based on a distributed counter, including: The value of the distributed counter is queried and updated using atomic operations.

[0059] Specifically, the core of atomic operations is to ensure the uninterruptibility of the "query-judgment-update" process and avoid the problem of "two tasks simultaneously querying the counter value of 2 (threshold is 3), both performing increment operations, causing the counter value to become 4, exceeding the threshold" in high-concurrency scenarios. This problem is one of the main defects of the coarse-grained control of existing technologies. This application solves this problem perfectly through Redis Lua scripts or distributed locks.

[0060] When using Lua scripts, the scripts are executed in a single thread on the Redis server, and will not be interrupted by other client commands during execution, ensuring the atomicity of the GET, conditional, and INCR operations, thus fundamentally avoiding exceeding limits. When using distributed locks, the exclusivity of the locks ensures that only one task can perform the check and update operations at the same time, making it suitable for low-concurrency scenarios.

[0061] The concurrent license occupancy includes an atomically incrementing counter.

[0062] Specifically, the increment operation is triggered only after both levels of checks have passed. During increment, both the node-level and cluster-level counters must be operated simultaneously to ensure that "either both increment or neither increments" to avoid data inconsistency. If a Lua script is used, the increment operation is completed by the INCR command inside the script. If a distributed lock is used, the increment operation is executed separately by the Redis INCR command and must be completed during the lock holding period.

[0063] The release of concurrent permissions includes atomically decrementing the corresponding counter.

[0064] Specifically, the decrement operation is triggered after the task steps are completed. Regardless of whether the step is successful (e.g., data synchronization is complete), fails (e.g., network interruption), or encounters an exception (e.g., data format error), the decrement must be executed. This logic is guaranteed by the try{} finally{} syntax. The decrement operation is also implemented through Redis's DECR command. If the counter value is 0 after decrementing, Redis will not automatically delete the key, and it can still be queried normally (returning 0) during subsequent executions. If it is necessary to clean up idle keys, it can be achieved through Redis's expiration policy (e.g., setting a 24-hour expiration time).

[0065] The distributed counters include node-level counters and cluster-level counters.

[0066] Specifically, the node-level counter is maintained separately for each service node and step type combination, counting only the concurrency of that type of step on that node, thus achieving fine-grained rate limiting on a single node; the cluster-level counter is maintained separately for each step type, counting the total concurrency of that type of step across the entire cluster, thus achieving global resource protection. The two work together to form the core data support for two-level rate limiting.

[0067] The initialization logic of the counter is as follows: When the task step is executed for the first time, the counter key is queried by the GET command. If null is returned, it means that the counter has not been created. At this time, the key value is initialized to 0 by the SET command, and then the increment operation is performed. The initialization and increment operations can be encapsulated in Lua scripts to ensure atomicity and avoid counting errors caused by "multiple tasks detecting that the key does not exist at the same time and performing initialization".

[0068] The key of the node-level counter is associated with the current service node identifier and the task step type, and is used for the first-level check.

[0069] Specifically, the service node identifier must be unique within the cluster. It can be a server IP address, hostname, or custom node ID (such as node-001, node-002). The step type must have a uniform and standardized name (to avoid different meanings with the same name). The key combination format ensures that each step type of each node has an independent counter, so that statistical confusion will not occur.

[0070] The key of the cluster-level counter is associated with the task step type and is used for the second-level check.

[0071] Specifically, by associating counters only with step types, the total concurrency count of the same type of steps across the entire cluster is accurately counted, avoiding interference from node identifiers with the global count. This design can effectively control the total number of accesses to backend shared resources by the cluster.

[0072] In an optional implementation, see Figure 3 As shown, Figure 3 The flowchart of the second two-level permission check method provided in Embodiment 1 of this application is shown. The two-level permission check is implemented based on the token bucket algorithm and includes steps S301 to S303: S301: Query and update the token bucket status using atomic operations.

[0073] Specifically, the token bucket algorithm is an alternative suitable for scenarios requiring adaptive flow control. Its status includes the number of tokens remaining in the bucket, the token generation rate, and the maximum bucket capacity. This status data is stored in Redis. The key format for a node-level token bucket is "ines_tokenbucket_node_[node number]:[step name]", and the key format for a cluster-level token bucket is "ines_tokenbucket_cluster:[step name]".

[0074] Atomic operations are implemented using Redis Lua scripts. The scripts need to complete the entire process of "calculating the number of tokens that should be generated - updating the number of tokens in the bucket (not exceeding the maximum capacity) - checking if there are any remaining tokens" to ensure the accuracy of the state update and avoid abnormal token counts caused by concurrent modifications.

[0075] The algorithm can be flexibly configured according to the characteristics of the steps and complements the counter scheme. For example, the "document renaming" step is suitable for the token bucket algorithm, which can achieve compatibility with burst traffic and improve resource utilization.

[0076] S302: The concurrent permission to occupy includes atomically obtaining a token from the corresponding token bucket.

[0077] Specifically, each step type corresponds to independent node-level and cluster-level token buckets. When obtaining a token, it is necessary to first check whether there are any remaining tokens in the node-level token bucket. If there are, it is further checked in the cluster-level token bucket. If there are remaining tokens in both, one token is obtained from each of the two buckets simultaneously (atomic decrementing the number of tokens). If there are no remaining tokens in either bucket, the acquisition fails.

[0078] Token generation rate is dynamically adjusted: CPU and memory usage are collected in real time through server monitoring tools. When CPU usage is below 80% and memory usage is below 70%, tokens are generated at a preset rate (e.g., 2 tokens per second). If either indicator exceeds the limit, token generation is stopped to avoid server overload. The maximum capacity of the token bucket can be set to 1.5 times the maximum number of parallel nodes in a single node / cluster (e.g., if the maximum number of parallel nodes in a single node is 8, the bucket capacity is set to 12) to reserve space for sudden traffic.

[0079] S303: The release of concurrent permissions includes atomically returning a token to the corresponding token bucket.

[0080] Specifically, after the steps are completed, the tokens are atomically returned to the node-level and cluster-level token buckets. The returned tokens can be immediately acquired by subsequent tasks, realizing the recycling of tokens. This mechanism allows for burst traffic. When there are accumulated tokens in the bucket, multiple tasks can acquire tokens and execute simultaneously. It is suitable for IO-intensive steps that can tolerate short-term bursts (such as "document renaming") and can maximize resource utilization.

[0081] In an optional implementation, see Figure 4 As shown, Figure 4 The flowchart of the third two-level license check method provided in Embodiment 1 of this application is shown, wherein, alternatively, the two-level license check is implemented based on the leaky bucket algorithm, including steps S401~S403: S401: Query and update the state of the leaky bucket using atomic operations.

[0082] Specifically, the leaky bucket algorithm is another alternative solution, suitable for scenarios that require stable protection of backend resources. Its status includes the current number of requests in the bucket, the maximum capacity of the bucket, and the leakage rate. The status data is stored in Redis. The key format of the node-level leaky bucket is "ines_leakybucket_node_[node number]:[step name]", and the key format of the cluster-level leaky bucket is "ines_leakybucket_cluster:[step name]".

[0083] Atomic operations are implemented using Redis Lua scripts. The scripts need to complete the logic of "checking if the bucket has remaining capacity - injecting requests (incrementing the number of requests in the bucket) - processing requests according to the leakage rate (decrementing the number of requests in the bucket)" to ensure the stability of traffic processing and avoid sudden traffic surges impacting backend resources.

[0084] S402: The concurrent license occupancy includes requesting the atomic injection of a request into the corresponding leaky bucket.

[0085] Specifically, each step type corresponds to an independent node-level and cluster-level leaky bucket. Before injecting a request, it is necessary to check whether the node-level leaky bucket has remaining capacity (the number of requests in the bucket < the maximum capacity of the bucket). If it does, the cluster-level leaky bucket is checked. If both have capacity, the request is injected into both buckets simultaneously (atomic incrementing the number of requests in the bucket). If either bucket is full, the injection fails and the task step enters the waiting queue.

[0086] The leaky bucket parameters need to be determined through prior stress testing: Taking the "Synchronize ES" step as an example, stress test the ES database, gradually increase the number of concurrent write requests, and record its maximum stable write speed (e.g., 10 successful writes per second). Since the batch data volume of each request to the ES database is fixed (e.g., 100 records / batch), the allowable number of concurrent requests per second can be calculated to be 10. Based on this, the leakage rate of the cluster-level leaky bucket is set to 10 times / second, and the maximum bucket capacity is 5 (with a small amount of buffer reserved); the leakage rate of the node-level leaky bucket is set to 2 times / second, and the maximum bucket capacity is 3, to adapt to the processing capacity of a single node.

[0087] S403: The release of concurrent permissions includes freeing up capacity after the leaky bucket completes request processing at a constant rate.

[0088] Specifically, the leaky bucket is unaffected by the input request rate and always processes requests within the bucket at a preset leakage rate. After processing, the number of requests in the bucket decreases by 1, freeing up the corresponding capacity and achieving stable traffic output. This mechanism can effectively limit the access rate of backend resources, avoid resource overload due to sudden traffic, and is particularly suitable for database write-intensive steps (such as "synchronizing Elasticsearch"), thus improving system stability.

[0089] In an optional implementation, see Figure 5 As shown, Figure 5 The flowchart of a two-level license check triggering method provided in Embodiment 1 of this application is shown, wherein, before executing any task step, task scheduling steps S501~S502 are further included: S501: Distribute the task steps to be executed to different message queues according to the resource consumption characteristics or preset priority corresponding to their step types.

[0090] Specifically, the message queue uses RabbitMQ and supports multiple exchange types (such as DirectExchange). Different queues are divided according to the resource consumption characteristics of the task steps (CPU-intensive, IO-intensive, database-intensive) or priority (high, medium, low), such as "cpu_high_queue" (CPU-intensive high priority), "io_normal_queue" (IO-intensive medium priority), and "db_low_queue" (database-intensive low priority) to achieve traffic distribution.

[0091] Client operations on tasks (such as starting, stopping, and modifying configurations) are converted into structured messages (containing information such as task ID, step type, parameters, and priority), which are then pushed to the corresponding queues via RabbitMQ's producer API. The push process supports a message acknowledgment mechanism to ensure that messages are not lost; the queues support persistent configuration to prevent message loss due to RabbitMQ restarts.

[0092] Task scheduling also supports two queue partitioning implementation methods: Method 1 partitions queues according to server resource type, with the granularity being tasks, meaning all steps of a task are distributed to queues of the same resource type (e.g., all steps of a CPU-intensive task enter "cpu_queue"); Method 2 partitions queues according to step priority and resource requirements, and after each step is completed, a message is pushed to the queue corresponding to the next step through the RabbitMQ producer API to achieve asynchronous flow between steps. For example, after the "data format verification" step is completed, a message is pushed to the "synchronous es" queue.

[0093] S502: The task execution node listens to the corresponding message queue and prioritizes consuming messages in the high-priority queue to trigger the two-level permission check.

[0094] Specifically, each service node can be configured to listen to multiple queues. The listening logic is implemented through RabbitMQ's consumer API and supports a manual ACK (Acknowledgment) mechanism to ensure that messages are acknowledged only after task steps are completed, thus avoiding message loss. If a node failure causes a message to be unprocessed, RabbitMQ will redeliver the message to another healthy node.

[0095] The consumption strategy adopts "high-priority queue first consumption", that is, the consumer first retrieves messages from the high-priority queue (such as "cpu_high_queue"). If the high-priority queue is empty, the consumer then consumes the medium-priority queue, and finally consumes the low-priority queue, so as to ensure the execution efficiency of core business steps (such as the review of labeling results). At the same time, it supports setting the queue consumption rate to avoid excessive load caused by a single node consuming too many messages.

[0096] The client monitors the queue status in real time through a visual interface, including queue length, message consumption rate, number of unconsumed messages, etc. Users can adjust the listening configuration of service nodes according to the queue status. For example, when too many messages accumulate in "db_low_queue", the number of service nodes listening to this queue can be increased to improve consumption efficiency.

[0097] In an optional implementation, after performing a two-level permission check on the number of concurrent connections based on the concurrency control parameters corresponding to their types, the method further includes: If the two-level checks fail, the execution of the task step is delayed, and the two-level permission check for the task step is re-initiated after a preset delay time.

[0098] Specifically, the delay time can be preset in the configuration center, and it supports configuring different delay times according to the step type. For example, if the "synchronize es" step fails the check, it will be delayed by 1 second to retry, and the "md to excel" step will be delayed by 2 seconds to retry, so as to avoid the system pressure caused by frequent retries. The delay logic is implemented through thread sleep (such as Java's Thread.sleep()) or the delayed queue function of message queue, the latter being more suitable for distributed scenarios.

[0099] If a delayed queue is used, the message of the task step that fails the check will be pushed to the delayed queue of RabbitMQ. After setting the preset delay time, the message will be automatically forwarded to the original business queue, and the service node will consume it again and trigger the check. This method does not require the client or server to maintain the retry state, reducing the system complexity.

[0100] Alternatively, the task step can be marked as pending retry, and the two-level permission check can be re-initiated according to a predetermined strategy.

[0101] Specifically, pre-defined strategies include fixed-interval retries and exponential backoff retries. Fixed-interval retries retries in a loop at a preset time (e.g., 3 seconds), suitable for scenarios with stable traffic. Exponential backoff retries retries in a loop at "initial interval × 2". n The incrementing delay time (n is the number of retries) (e.g., 1 second for the first time, 2 seconds for the second, 4 seconds for the third, with a maximum of 30 seconds) is suitable for high-concurrency scenarios and avoids retry storms.

[0102] The status of tasks to be retried (task ID, step type, number of retries, next retry time) is stored in a database (such as MySQL). An independent retry scheduling process scans the database, and when the next retry time is reached, the task step message is pushed to the corresponding queue to re-trigger the check. The number of retries can be set to an upper limit (such as 5 times). If the upper limit is exceeded, the task is marked as failed and the user is notified to handle it manually.

[0103] In an optional implementation, see Figure 6 As shown, Figure 6 The flowchart of a visualization method provided in Embodiment 1 of this application is shown, wherein, during the execution of the task steps, the method further includes steps S601-S602: S601: Based on the total amount of data to be processed and the amount of data already processed in this task step, calculate the processing progress and estimated remaining time in real time.

[0104] Specifically, the total amount of data to be processed can be obtained through task configuration (e.g., the number of documents specified by the user when initiating a task is 1000) or database query (e.g., the amount of data to be synchronized in the "Synchronize es" step is stored in a MySQL table); the amount of data already processed is obtained through real-time statistics during the execution of the steps. For example, in the "Document Rename" step, the processed count is incremented for each document successfully renamed, and the count information is stored in a local cache (e.g., Redis) to ensure real-time performance.

[0105] The processing progress is calculated using the formula: "Processing progress = (Amount of data processed / Total amount of data to be processed) × 100%". The real-time processing rate is calculated using "Amount of data processed in the last minute / 60 seconds". The estimated remaining time is calculated as: "(Total amount of data to be processed - Amount of data processed) / Real-time processing rate". If the real-time processing rate is 0 (the step has just started or is paused), the estimated remaining time will be displayed as "Unknown".

[0106] S602: Provide a query interface or a visual display of the processing progress and the estimated remaining time.

[0107] Specifically, the query interface adopts a RESTful API design, and the visualization is implemented through the client interface. The interface presents the status of all task steps in the form of tables or cards, including task ID, step type, processing progress (progress bar + percentage), estimated remaining time, concurrency, execution node and other information. Users can filter and sort tasks through the interface, and it also supports exporting progress reports (such as Excel format). The client interface also provides task start and stop control buttons, allowing users to stop a task in execution at any time. After stopping, the system will release the concurrency license occupied by the task step to avoid resource waste.

[0108] To better illustrate the distributed task parallel control method provided in this application, the "synchronize Elasticsearch" step in an internal enterprise data annotation task is used as an example to fully demonstrate the execution flow of this application: 1. Parameter Configuration: Configure concurrency control parameters for the "Synchronize ES" step in the configuration center: maximum parallelism per node = 2, maximum parallelism for the entire cluster = 5; deploy a Redis (Remote Dictionary Server) cluster with 3 masters and 3 slaves, with the node-level counter key format "ines_speedlimit_node_[node number]:SyncEs" and the cluster-level counter key format "ines_speedlimit_cluster:SyncEs"; use Redis Lua (Lua Programming Language) scripts to implement atomic operations.

[0109] 2. Task Scheduling: Users initiate 10 data labeling tasks through the client. Each task includes a "synchronize Elasticsearch" step. The client pushes these step messages to the "db_normal_queue" of RabbitMQ (Rabbit Message Queue) according to priority (medium priority). The cluster contains 3 service nodes (node-001, node-002, node-003), all of which listen to this queue.

[0110] 3. Two-level inspection and permission occupancy: Consumer node-001 receives the first "Synchronize ES" message and triggers a check: it calls a Lua script to check if the node-level counter value of node-001 is 0 (below threshold 2) and the cluster-level counter value is 0 (below threshold 5), and the check passes; the script executes the INCR command, both counter values ​​become 1, occupy the concurrency license, and begin to perform the synchronization of ES data.

[0111] One second later, the node-001 consumer receives the second "Synchronize es" message, checks that the node-level counter value is 1 (not reaching 2) and the cluster-level counter value is 1 (not reaching 5), and the check passes; the counter values ​​both become 2, the license is used, and the synchronization operation is performed.

[0112] Two seconds later, the node-001 consumer retrieves the third "synchronize es" message, checks the node-level counter value to be 2 (the threshold of 2 has been reached), and fails the check; the message is pushed to the RabbitMQ delayed queue and redelivered after a 1-second delay.

[0113] Meanwhile, consumer node-002 receives two "synchronize es" messages, passes the checks, and the node-level counter value becomes 2 and the cluster-level counter value becomes 4; consumer node-003 receives one "synchronize es" message, passes the checks, and the node-level counter value becomes 1 and the cluster-level counter value becomes 5 (reaching the cluster threshold).

[0114] 4. Handling of failed checks: When the node-003 consumer obtains the next "Synchronize ES" message, it checks that the cluster-level counter value is 5 (the threshold of 5 has been reached), and the check fails. The message is marked as pending retry and retried using an exponential backoff strategy, with a delay of 1 second for the first time, 2 seconds for the second time, and 4 seconds for the third time.

[0115] 5. Execution and Progress Monitoring: The amount of data processed in real time is counted for all "Synchronize Elasticsearch" steps in execution (e.g., the count is updated every 100 data items synchronized). The processing progress of each step is displayed on the client interface (e.g., the progress of the first message of node-001 is 68.30%, with an estimated remaining time of 61 seconds). At the same time, the server CPU and memory usage are monitored through the monitoring system to ensure that the node load is normal.

[0116] 6. Permit Release: After the first "Synchronize ES" step of node-001 is completed (1000 data records are synchronized), the DECR command is executed through the finally block, the node-level counter value becomes 1, and the cluster-level counter value becomes 4; the released permit is retrieved by the message delivered by the delayed queue, the node-001 consumer re-checks and passes, the counter value is updated to 2, and the new synchronization task continues to be executed.

[0117] 7. Anomaly Detection: The background verification process performs a check every 30 seconds, calculating the sum of the node-level counters for the "Synchronize ES" of the three nodes (node-001:2, node-002:2, node-003:1, total 5). If the sum matches the cluster-level counter value of 5, there is no alarm. If a node's counter fails to decrement in time due to network latency (node-level total 6, cluster-level 5), an alarm is sent to relevant personnel via WeChat and email.

[0118] This example fully covers the core processes such as parameter configuration, task scheduling, two-level checks, license occupancy and release, progress monitoring, and exception handling. It intuitively demonstrates how this application achieves step-level and two-level rate limiting, solving the problem of unreasonable utilization of existing technical resources and easy occurrence of system bottlenecks.

[0119] Example 2 See Figure 7 As shown, Figure 7 This illustration shows a schematic diagram of a distributed task parallel control device according to Embodiment 2 of this application, wherein the device includes: The control parameter definition module 701 is used to predefine the corresponding concurrency control parameters for different types of task steps; The two-level permission check module 702 is used to perform a two-level permission check on the number of concurrent requests based on the concurrency control parameters corresponding to the type of any task step before execution. The first-level check is used to control the concurrency of the task step at the current service node level, and the second-level check is used to control the concurrency of the task step at the entire distributed cluster level. The task step execution module 703 is used to occupy a concurrent license and start executing the task step if both levels of checks pass. The concurrent license release module 704 is used to release the concurrent license it occupies after the task step is completed.

[0120] In an optional implementation, the concurrency control parameters include the maximum number of parallel operations per node and the maximum number of parallel operations across the entire cluster; The two-level permission check for concurrency includes: Perform the first-level check to determine whether the number of concurrent executions of the same type of steps on the current service node has reached its maximum number of concurrent executions per node; If not, the second-level check is performed to determine whether the total number of concurrent executions of the same type of steps in the entire distributed cluster has reached its maximum number of parallel executions.

[0121] In an optional implementation, the two-level permission check is implemented based on a distributed counter, including: The value of the distributed counter is queried and updated using atomic operations; The concurrent permission to occupy includes an atomically incrementing counter; The release of concurrent permissions includes atomically decrementing the corresponding counter; The distributed counters include node-level counters and cluster-level counters; The key of the node-level counter is associated with the current service node identifier and the task step type, and is used for the first-level check; The key of the cluster-level counter is associated with the task step type and is used for the second-level check.

[0122] In an optional implementation, the two-level permission check is implemented based on the token bucket algorithm, including: Query and update the token bucket status using atomic operations; The concurrent permission to occupy includes atomically obtaining a token from the corresponding token bucket; The release of concurrent permissions includes atomically returning a token to the corresponding token bucket; or, The two-level permission check is implemented based on the leaky bucket algorithm, including: Query and update the status of the leaky bucket using atomic operations; The concurrent permission occupancy includes the request to atomically inject a request into the corresponding leaky bucket; The release of concurrent permissions includes freeing up capacity in the leaky bucket after it has completed request processing at a constant rate.

[0123] In an optional implementation, the apparatus further includes a task scheduling module for performing a task scheduling step before executing any task step: The task steps to be executed are distributed to different message queues based on their resource consumption characteristics or preset priorities. The task execution node listens to the corresponding message queue and prioritizes consuming messages in the high-priority queue to trigger the two-level permission check.

[0124] In an optional implementation, the apparatus further includes a task step delayed execution module, for: After performing a two-level permission check on the number of concurrent requests based on the concurrency control parameters corresponding to its type, if the two-level checks fail, the execution of the task step is delayed, and the two-level permission check on the task step is re-initiated after a preset delay time. Alternatively, the task step can be marked as pending retry, and the two-level permission check can be re-initiated according to a predetermined strategy.

[0125] In an optional implementation, the device further includes a visualization module for: During the execution of the task step, the processing progress and estimated remaining time are calculated in real time based on the total amount of data to be processed and the amount of data already processed in the task step. The processing progress and the estimated remaining time can be provided to external users through a query interface or displayed visually.

[0126] Example 3 Based on the same application concept, see [link / reference] Figure 8 As shown, Figure 8 This illustration shows a structural schematic diagram of a computer device provided in Embodiment 3 of this application, wherein, as shown... Figure 8 As shown, the computer device 800 provided in Embodiment 3 of this application includes: The computer device 800 includes a processor 801, a memory 802, and a bus 803. The memory 802 stores machine-readable instructions that can be executed by the processor 801. When the computer device 800 is running, the processor 801 communicates with the memory 802 through the bus 803. The machine-readable instructions are executed by the processor 801 to perform the steps of the distributed task parallel control method shown in Embodiment 1 above.

[0127] Example 4 Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the distributed task parallel control method described in any of the above embodiments.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0129] The computer program product for distributed task parallel control provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0130] The distributed task parallel control device provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0131] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0134] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0136] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A distributed task parallel control method, characterized in that, The method includes: Predefine the corresponding concurrency control parameters for different types of task steps; Before executing any task step, a two-level permission check is performed on the concurrency number according to the concurrency control parameters corresponding to its type. The first level check is used to control the concurrency of the task step at the current service node level, and the second level check is used to control the concurrency of the task step at the entire distributed cluster level. If both levels of checks pass, a concurrent permit is acquired and the task step begins execution; After the task step is completed, the concurrent license it occupied is released.

2. The method according to claim 1, characterized in that, The concurrency control parameters include the maximum number of parallel operations per node and the maximum number of parallel operations for the entire cluster; The two-level permission check for concurrency includes: Perform the first-level check to determine whether the number of concurrent executions of the same type of steps on the current service node has reached its maximum number of concurrent executions per node; If not, the second-level check is performed to determine whether the total number of concurrent executions of the same type of steps in the entire distributed cluster has reached its maximum number of parallel executions.

3. The method according to claim 2, characterized in that, The two-level permission check is implemented based on a distributed counter, including: The value of the distributed counter is queried and updated using atomic operations; The concurrent permission to occupy includes an atomically incrementing counter; The release of concurrent permissions includes atomically decrementing the corresponding counter; The distributed counters include node-level counters and cluster-level counters; The key of the node-level counter is associated with the current service node identifier and the task step type, and is used for the first-level check; The key of the cluster-level counter is associated with the task step type and is used for the second-level check.

4. The method according to claim 2, characterized in that, The two-level permission check is implemented based on the token bucket algorithm, including: Query and update the token bucket status using atomic operations; The concurrent permission to occupy includes atomically obtaining a token from the corresponding token bucket; The release of concurrent permissions includes atomically returning a token to the corresponding token bucket; or, The two-level permission check is implemented based on the leaky bucket algorithm, including: Query and update the status of the leaky bucket using atomic operations; The concurrent permission occupancy includes the request to atomically inject a request into the corresponding leaky bucket; The release of concurrent permissions includes freeing up capacity in the leaky bucket after it has completed request processing at a constant rate.

5. The method according to claim 1, characterized in that, Before executing any task step, a task scheduling step is also included: The task steps to be executed are distributed to different message queues based on their resource consumption characteristics or preset priorities. The task execution node listens to the corresponding message queue and prioritizes consuming messages in the high-priority queue to trigger the two-level permission check.

6. The method according to claim 1, characterized in that, After performing a two-level permission check on the number of concurrent requests based on the concurrency control parameters corresponding to their types, the method further includes: If the two-level checks fail, the execution of the task step is delayed, and the two-level permission check for the task step is re-initiated after a preset delay time. Alternatively, the task step can be marked as pending retry, and the two-level permission check can be re-initiated according to a predetermined strategy.

7. The method according to claim 1, characterized in that, During the execution of the task steps, the method further includes: Based on the total amount of data to be processed and the amount of data already processed in this task step, the processing progress and estimated remaining time are calculated in real time. The processing progress and the estimated remaining time can be provided to external users through a query interface or displayed visually.

8. A distributed task parallel control device, characterized in that, The device includes: The control parameter definition module is used to predefine the corresponding concurrency control parameters for different types of task steps; The two-level permission check module is used to perform a two-level permission check on the concurrency number according to the concurrency control parameters corresponding to its type before executing any task step. The first level check is used to control the concurrency of the task step at the current service node level, and the second level check is used to control the concurrency of the task step at the entire distributed cluster level. The task step execution module is used to occupy a concurrent license and start executing the task step if both levels of checks pass. The concurrent license release module is used to release the concurrent license it occupies after the task step is completed.

9. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the distributed task parallel control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the distributed task parallel control method as described in any one of claims 1 to 7.

Images