Task allocation method and device, equipment, medium and program product

Abstract

The invention provides a task allocation method and device, equipment, a medium and a program product, and relates to the field of cloud computing, and the method is characterized in that the method comprises the steps: obtaining the matching degree of each node of a task processing system and an actuator demand; constructing a matching degree matrix of the actuators and the nodes based on the matching degree, and arranging the actuators to the corresponding nodes based on the matching degree matrix; obtaining the maximum parallel node value of the target task; splitting the target task to obtain a sub-task set, and based on the maximum parallel node value, obtaining a load-balanced task allocation plan; and distributing the subtasks in the subtask set to the executors of the corresponding nodes according to the distribution plan. The resource utilization rate and the task execution efficiency can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and in particular to a task allocation method, apparatus, device, medium, and program product. Background Technology

[0002] Telecom operators' intelligent middleware platforms have extremely high requirements for task execution efficiency and resource utilization. Related technologies often break down tasks into multiple subtasks, which are then sequentially or randomly assigned to different nodes for execution. This simple allocation strategy fails to fully utilize node resources, resulting in low resource utilization. Summary of the Invention

[0003] This application provides a task allocation method and apparatus that can solve the problem of low resource utilization due to the inability to fully utilize node resources during task allocation.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a task allocation method, the method comprising: To determine the degree of matching between the various nodes of the task processing system and the requirements of the executor; Based on the matching degree, a matching degree matrix between the executor and the node is constructed, and the executor is deployed to the corresponding node based on the matching degree matrix; Obtain the maximum number of parallel nodes for the target task; The target task is split into a set of subtasks, and a load-balanced task allocation plan is obtained based on the maximum number of parallel nodes. According to the allocation plan, the subtasks in the subtask set are allocated to the executors of the corresponding nodes.

[0005] Secondly, embodiments of this application provide a task allocation device, the device comprising: The first acquisition module is used to acquire the degree of matching between each node of the task processing system and the requirements of the executor; The placement module is used to construct a matching degree matrix between the executor and the node based on the matching degree, and to place the executor to the corresponding node based on the matching degree matrix; The second acquisition module is used to acquire the maximum number of parallel nodes for the target task. The third acquisition module is used to split the target task into a set of subtasks and obtain a load-balanced task allocation plan based on the maximum parallel node value. The allocation module is used to allocate subtasks in the subtask set to the executors of the corresponding nodes according to the allocation plan.

[0006] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the task allocation method as described in the first aspect.

[0007] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the task allocation method as described in the first aspect.

[0008] In this embodiment, the matching degree between each node of the task processing system and the executor's requirements is obtained; a matching degree matrix between the executor and the node is constructed based on the matching degree; the executor is deployed to the corresponding node based on the matching degree matrix; the maximum parallel node value of the target task is obtained; the target task is split into a set of subtasks; a load-balanced task allocation plan is obtained based on the maximum parallel node value; and the subtasks in the subtask set are allocated to the executors of the corresponding nodes according to the allocation plan. Thus, by obtaining the matching degree between each node of the task processing system and the executor's requirements, constructing a matching degree matrix between the executor and the node based on the matching degree, deploying the executor to the corresponding node, obtaining the maximum parallel node value of the target task, splitting the target task into a set of subtasks, obtaining a load-balanced task allocation plan based on the maximum parallel node value, and finally allocating the subtasks to the executors of the corresponding nodes according to the allocation plan, precise adaptation between the executor and the node, and balanced distribution of system load are achieved, effectively improving resource utilization and task execution efficiency, and better meeting the needs of efficient and intelligent task processing. Attached Figure Description

[0009] Figure 1 One of the flowcharts illustrating the task allocation method provided in the embodiments of this application; Figure 2 A second flowchart illustrating the task allocation method provided in this application embodiment; Figure 3 This is a schematic diagram of task execution for the task allocation method provided in the embodiments of this application; Figure 4 A schematic diagram of the structure of the task allocation device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] The following description, in conjunction with the accompanying drawings, further illustrates the task allocation method, apparatus, device, and storage medium proposed in the embodiments of the application.

[0012] Please see Figure 1 , Figure 1 A flowchart illustrating a task allocation method provided in this application embodiment is shown in the figure. The method includes: Step 110: Obtain the degree of matching between each node of the task processing system and the requirements of the executor; In this step, the task allocation method described above can be understood as a series of ordered operations that rationally allocate tasks to nodes and corresponding executors in the task processing system to achieve efficient task execution. This enables optimized resource utilization and rapid task response. The task processing system described above can be understood as a system for receiving and processing various tasks. It can contain multiple distributed processing units, enabling parallel task processing and collaborative resource scheduling, and is suitable for various application scenarios such as cloud computing, big data processing, and intelligent platforms. The nodes described above can be understood as hardware or software units in the task processing system that undertake task execution functions; they are the actual carriers of task execution. Each node can be configured with various types of executors, possessing certain computing, storage, or I / O processing capabilities. The executors described above can be understood as functional modules deployed on nodes for specifically executing tasks or subtasks. They can possess different resource characteristics according to task requirements, adapting to the processing requirements of various tasks.

[0013] The aforementioned executor requirements can be understood as the resource conditions an executor needs to complete a specific task, including requirements for computing power, memory capacity, I / O processing speed, etc., and are the core basis for matching nodes and executors. The aforementioned matching degree can be understood as the degree of fit between the node's resource characteristics and the executor's requirements. Its value reflects the node's adaptability to the executor; the higher the fit, the higher the matching degree.

[0014] To determine the compatibility between the requirements of each node in the task processing system and the executor, we first need to assess the degree of matching between the nodes and the executor requirements. Each node in the task processing system possesses its own resource characteristics, such as CPU processing speed, memory storage capacity, and I / O transfer rate. Different types of executors have specific requirements for completing their respective tasks. To achieve accurate resource matching, we can first determine the degree of matching between each node and the executor requirements. There are several ways to obtain this degree of matching. For example, we can calculate the degree of matching based on the compatibility between the node's CPU performance and the computing power required by the executor. Alternatively, we can combine the node's memory capacity, I / O speed, and the corresponding resource requirements of the executor for a comprehensive evaluation to determine the degree of matching. These methods can objectively reflect the node's adaptability to the executor.

[0015] For example, in the resource matching process between executors and nodes, it is assumed that executors are divided into three types: compute-intensive (CE), memory-intensive (ME), and I / O-intensive (IOE), and each type of executor has different demand weights for CPU, memory, and I / O resources. At the same time, nodes also provide these three types of resources, and the supply of each resource can be quantified.

[0016] For a computational actuator (CE), its resource requirement vector can be represented as (α_CE*CPU, β_CE*Mem, γ_CE*IO), where α_CE, β_CE, and γ_CE are the weighting coefficients of CE for CPU, memory, and I / O resources, respectively, and CPU, Mem, and IO represent the standardized values ​​of the resources (i.e., relative quantities, which are convenient for calculation).

[0017] Similarly, the resource requirement vector for a memory-type executor ME is (α_ME*CPU, β_ME*Mem, γ_ME*IO), and the resource requirement vector for an I / O-type executor IOE is (α_IOE*CPU, β_IOE*Mem, γ_IOE*IO).

[0018] The resource supply vector of a node is uniformly represented as (CPU_j, Mem_j, IO_j), where j represents a different node, and CPU_j, Mem_j, and IO_j represent the supply of CPU, memory, and I / O resources on node j, respectively (also standardized values).

[0019] The matching degree M(Ei,Nj) between actuator Ei and node Nj can be obtained by calculating the dot product of their resource demand vector and supply vector, as shown in the following formula: M(Ei,Nj)=(α_Ei*CPU,β_Ei*Mem,γ_Ei*IO)·(CPU_j,Mem_j,IO_j) =α_Ei*CPU*CPU_j+β_Ei*Mem*Mem_j+γ_Ei*IO*IO_j; Where Ei represents any type of actuator (CE, ME, or IOE). This formula reflects the degree of matching between the actuator's resource requirements and the node's resource supply. The higher the matching value, the better the resource type matching between the actuator and the node.

[0020] Step 120: Construct a matching degree matrix between the executor and the node based on the matching degree, and deploy the executor to the corresponding node based on the matching degree matrix; In this step, the matching degree matrix can be understood as structured data that organizes and presents the matching degree between each node and each type of actuator requirement in matrix form. The elements in the matrix correspond to the matching degree value between a specific node and a specific actuator requirement.

[0021] Based on the matching degree, a matching degree matrix between executors and nodes is constructed, and executors are deployed to the corresponding nodes based on this matching degree matrix. After obtaining the matching degree of each node and executor requirement, this matching degree data can be structured and organized to construct the matching degree matrix between executors and nodes. When constructing the matching degree matrix, the executor type can be used as the rows of the matrix and the nodes as the columns, with each element in the matrix representing the matching degree value between the corresponding executor type and the node; alternatively, the nodes can be used as the rows of the matrix and the executor type as the columns, with each element representing the matching degree. The matrix clearly presents the adaptation relationship between various types of executors and various nodes. When deploying executors based on this matching degree matrix, various methods can be used. For example, each type of executor can be evenly deployed to several nodes with the highest matching degree to ensure balanced resource utilization; alternatively, the nodes with the most remaining resources can be selected from the nodes whose matching degree meets preset conditions, and executors can be allocated according to resource weights so that the executors can fully utilize the idle resources of the nodes.

[0022] Step 130: Obtain the maximum number of parallel nodes for the target task; In this step, the target task mentioned above can be understood as a specific task to be assigned and executed, which may have different types and characteristics, such as computationally intensive, memory-intensive, or I / O-intensive tasks.

[0023] Obtain the maximum number of parallel nodes for the target task. The maximum number of parallel nodes for the target task is one of the key parameters determining the scale of parallel execution. Its acquisition can be based on task characteristics and system resource availability. There are several ways to obtain this value. For example, one can first determine the task's parallelism by analyzing its computational logic and data dependencies, and then calculate the maximum number of parallel nodes using a preset mapping relationship between parallelism and the number of nodes. Alternatively, one can dynamically adjust the base number of nodes corresponding to the parallelism, taking into account factors such as the current availability of system node resources and task priority, to obtain the maximum number of parallel nodes. This ensures that the maximum number of parallel nodes meets the needs of parallel task execution without exceeding the system's resource capacity.

[0024] Step 140: The target task is split into a set of subtasks, and a load-balanced task allocation plan is obtained based on the maximum parallel node value. In this step, to achieve parallel execution of tasks, the target task can be split first. The splitting method can be flexibly selected according to the task characteristics. For example, the target task can be split into multiple independent subtasks according to the functional modules of the task, with each subtask corresponding to a specific function implementation; alternatively, the target task can be evenly split into several subtasks according to the data processing volume of the task, ensuring that the processing volume of each subtask is relatively balanced. In an optional implementation, after obtaining the set of subtasks, each subtask can be further subdivided to obtain sub-subtasks. The subdivision method can be flexibly set according to the processing capacity of the executor. For example, the subtask can be split into multiple sub-subtasks according to the maximum amount of data that the executor can process at one time, or the subtask can be split into sub-subtasks adapted to its processing logic according to the type characteristics of the executor. This subdivision can make the task more closely match the actual processing capacity of the executor, further improving execution efficiency. After obtaining the set of subtasks or the set of tasks containing sub-subtasks, a load balancing allocation plan needs to be obtained based on the maximum parallel node value. When obtaining the plan, various optimization algorithms can be used. For example, a greedy algorithm can be used to simulate different allocation schemes, calculate the load difference index of each node under each scheme, and select the scheme with the smallest load difference as the allocation plan. Alternatively, a genetic algorithm can be used to obtain a load-balanced allocation plan through multiple generations of iterative optimization under the constraint of not exceeding the maximum number of parallel nodes. The plan needs to specify the number of subtasks or secondary subtasks that should be allocated to each node and the corresponding executor.

[0025] Step 150: According to the allocation plan, the subtasks in the subtask set are allocated to the executors of the nodes of the task processing system.

[0026] In this step, the maximum number of parallel nodes mentioned above can be understood as the maximum number of nodes that the target task can achieve parallel execution in the task processing system. This value needs to be determined comprehensively based on the task characteristics and system resource conditions. The subtask set mentioned above can be understood as a set of multiple independent or related task units obtained by splitting the target task according to preset rules. Each subtask can be executed independently on its corresponding node and executor. The load-balanced task allocation plan mentioned above can be understood as a task allocation scheme that can make the load distribution of each node tend to be balanced without exceeding the maximum number of parallel nodes. This plan clarifies the number of subtasks allocated to each node and the execution method.

[0027] According to the allocation plan, subtasks in the subtask set can be assigned to the executors of the nodes in the task processing system. After determining the load-balanced task allocation plan, each subtask or secondary subtask in the subtask set can be distributed to the executors on the corresponding nodes according to the allocation rules specified in the plan. If a secondary subtask is assigned, it can be directly received and executed by the executor; if a subtask is assigned, the executor can decide whether to further split it based on its own processing capacity, or it can execute it directly. In some optional implementations, during the allocation process, the amount of tasks received by each node can be matched with its own resource carrying capacity, and each task unit can be assigned to a suitable executor type. After receiving the task unit, the executor can efficiently complete the task execution based on its own resource characteristics and the processing requirements of the task unit, ultimately achieving rapid processing of the target task and optimized utilization of resources.

[0028] In the task allocation method of this application embodiment, the matching degree between each node of the task processing system and the executor requirements is obtained. Based on the matching degree, a matching degree matrix between the executor and the node is constructed and the executor is deployed to the corresponding node. At the same time, the maximum parallel node value of the target task is obtained. After the target task is split into subtask sets, a load-balanced task allocation plan is obtained based on the maximum parallel node value. Finally, the subtasks are allocated to the executors of the corresponding nodes according to the allocation plan. This achieves accurate matching between the executor and the node and balanced distribution of system load, effectively improving resource utilization and task execution efficiency, and better meeting the needs of efficient and intelligent task processing.

[0029] Optionally, obtaining the maximum number of parallel nodes for the target task includes: Obtain task attribute parameters that affect task parallelism, wherein the task attribute parameters include at least one of subtask dependency, data interaction complexity, computational logic correlation, and data processing scale; Each task attribute parameter is quantified, and a comprehensive quantized value is obtained based on the quantified task attribute parameters and their corresponding weights. The comprehensive quantized value is normalized to obtain the task parallelism. A parallelism-node number mapping function is constructed based on historical task data, and the parallelism of the task is mapped to a candidate value of the maximum number of parallel nodes through the parallelism-node number mapping function; Based on the candidate value of the maximum number of parallel nodes, a variety of task allocation schemes are simulated through a load balancing algorithm to obtain the load imbalance degree of each scheme, and the number of nodes corresponding to the scheme with the smallest load imbalance degree is determined as the maximum number of parallel nodes.

[0030] In this implementation, the aforementioned task attribute parameters can be understood as parameters related to the task's own characteristics that can affect the parallel execution effect of the task, and are a key basis for judging the task's parallel capability. The aforementioned subtask dependency relationship can be understood as the relationship of mutual association and constraint between the various subtasks after the target task is broken down, which directly affects whether the subtasks can be executed independently in parallel. The aforementioned data interaction complexity can be understood as the complexity of data transfer, sharing, or synchronization between subtasks; the more complex the interaction, the less conducive it is to the parallel execution of the task.

[0031] The aforementioned computational logic correlation can be understood as the degree of connection between subtasks in terms of computational flow and processing logic; the stronger the correlation, the greater the difficulty of parallel execution. The aforementioned data processing scale can be understood as the total amount of data that the target task needs to process; the amount of data affects the number of nodes required for parallel execution.

[0032] The parallelism of a task is not determined by a single factor, but rather by multiple attribute parameters. These parameters can include at least one of the following: subtask dependencies, data interaction complexity, computational logic correlation, and data processing scale. In practical applications, the parameters to be acquired can be flexibly selected based on the type of the target task. For example, when dealing with computationally intensive tasks, the computational logic correlation and data processing scale can be prioritized; when dealing with tasks involving frequent data transfer, data interaction complexity and subtask dependencies can be prioritized. These parameters can be acquired in various ways, either directly by parsing the task's description and execution requirements, or indirectly by analyzing the task's historical execution records.

[0033] The aforementioned comprehensive quantized value can be understood as a numerical value obtained by comprehensively calculating multiple quantized task attribute parameters and their corresponding weights, which can fully reflect the parallel potential of the task. The aforementioned normalization process can be understood as a method of mapping the comprehensive quantized value to a specific numerical range according to preset rules, with the aim of unifying the data scale and facilitating subsequent calculations and comparisons.

[0034] Each task attribute parameter is quantified, and a comprehensive quantified value is obtained based on the quantified task attribute parameters and their corresponding weights. Since different task attribute parameters have different manifestations, they cannot be directly comprehensively evaluated; therefore, each parameter needs to be quantified first. The quantification method can be flexibly set according to the parameter characteristics. For example, for subtask dependencies, quantification scores from 0 to 10 can be assigned according to the degree of dependency, with higher scores for lower dependencies. For data processing scale, different levels can be assigned based on the data volume, with higher scores for larger data volumes. After quantization, a corresponding weight can be assigned to each parameter. The weight allocation can be adjusted according to the application scenario and execution requirements of the task. For example, in scenarios emphasizing execution efficiency, the weight of computational logic correlation can be increased; in scenarios emphasizing data processing integrity, the weight of data interaction complexity can be increased. There are multiple ways to obtain the comprehensive quantified value: either multiplying the quantified value of each parameter by its corresponding weight and then summing the results, or normalizing the quantified values ​​first and then calculating a weighted average based on the weights.

[0035] The comprehensive quantized value is normalized to obtain the task parallelism. The comprehensive quantized value obtained through the preceding steps may fall within different numerical ranges. To unify the measurement standard and adapt to the subsequent mapping function, it can be normalized. Normalization maps the comprehensive quantized value to a numerical range of 0 to 1, where the closer the value is to 1, the stronger the task's parallelism capability; the closer it is to 0, the weaker the task's parallelism capability. The specific normalization method can be flexibly chosen. For example, a linear normalization method can be used to map the comprehensive quantized value proportionally to its numerical range to the target range; a nonlinear normalization method can also be used, achieving numerical scale uniformity through logarithmic or exponential transformations. Through normalization, the task parallelism, reflecting the task's parallelism capability, can be clearly and intuitively obtained.

[0036] In some alternative implementations, task parallelism can be obtained through the following steps: Feature definition and extraction: First, the internal operational features of task T can be defined and extracted, including but not limited to: Operational independence: measures whether each operation can be performed independently, without depending on the results of other operations.

[0037] Data Dependencies: Analyze the data flow between operations to determine if there are any data dependencies that restrict the execution order.

[0038] Operation execution time: Records the expected execution time of each operation to evaluate the potential for parallel execution to improve overall performance.

[0039] Operation graph construction: Based on the internal operation characteristics of task T, a task operation graph G=(V,E) is constructed, where V is the set of operation nodes and E is the set of dependency edges between operations. The operation graph intuitively displays the operations within the task and their dependencies.

[0040] Parallelism calculation formula: The parallelism P(T) of a task can be calculated using the following formula: P(T)=1 Total operation path length / serial dependency path length in: Serial dependency path length: In the operation graph, find the longest serial dependency path from the start operation to the end operation (i.e., all operations on the path must be executed in sequence), and calculate the sum of the number of operations on these paths.

[0041] Total operation path length: The sum of the number of all operations without considering dependencies, i.e., the number of nodes V in the operation graph G.

[0042] Calculation process: Construct the operation graph: Based on the internal operations of task T and their dependencies, construct the operation graph G.

[0043] Identify serial dependency paths: In the operation graph G, use depth-first search (DFS) or breadth-first search (BFS) algorithms to find all the longest serial dependency paths from the start operation to the end operation.

[0044] Calculate path length: Count the number of operations on the serial dependency path to obtain the serial dependency path length; at the same time, count the number of all operations in the operation graph G to obtain the total operation path length.

[0045] Calculation using the formula: Substitute the serial dependency path length and the total operation path length into the formula to calculate the task parallelism P(T).

[0046] Using the method described above, the parallelism P(T) of task T can be accurately calculated, providing an important basis for optimizing task allocation and execution strategies. This technical solution fully discloses the calculation process of P(T), ensuring its feasibility and verifiability.

[0047] The aforementioned historical task data can be understood as relevant data accumulated by the task processing system when it has executed various tasks in the past. It includes key information and result data during the task execution process and can provide a reference for the parameter calculation of the current task.

[0048] The parallelism-node number mapping function can be understood as a function that establishes the correspondence between the parallelism of a task and the maximum number of parallel nodes, and can output the corresponding node number reference value according to the parallelism of the task.

[0049] A parallelism-node number mapping function is constructed based on historical task data. This function maps the task parallelism to a candidate value for the maximum number of parallel nodes. Historical task data contains a wealth of matching information between different task parallelisms and their corresponding optimal number of parallel nodes, forming a crucial foundation for constructing the mapping function. During the construction of the mapping function, the historical task data can be organized and filtered to ensure its accuracy and validity. Then, the parallelism of historical tasks is used as the input feature, and the corresponding optimal number of parallel nodes is used as the output label to construct the mapping function model. Various training algorithms can be used to construct the model. For example, a linear regression algorithm can be used to establish a linear correspondence between parallelism and the number of nodes; a decision tree algorithm can be used to construct a nonlinear mapping model through the classification and summarization of historical data; or a neural network algorithm can be used to construct a high-precision mapping function using its powerful fitting ability. After the mapping function is constructed, the parallelism of the current target task is input into the function to obtain the corresponding candidate value for the maximum number of parallel nodes. This candidate value provides a basic reference for subsequent scheme optimization.

[0050] The aforementioned candidate values ​​for the maximum number of parallel nodes can be understood as preliminary reference values ​​for the maximum number of parallel nodes that meet the task's parallel requirements, calculated using a mapping function, and are not the final determined values. The aforementioned load imbalance can be understood as an indicator that measures the load difference between nodes under different task allocation schemes; the smaller the value of this indicator, the more balanced the node load distribution.

[0051] The candidate values ​​for the maximum number of parallel nodes mentioned above are only theoretical references and can be optimized and adjusted based on the actual load of the system. First, using the candidate values ​​as a basic constraint, determine the range of nodes that can participate in task allocation to ensure that subsequent simulated allocation schemes do not exceed this range. Then, use load balancing algorithms to simulate and generate various task allocation schemes. The load balancing algorithms used can be diverse, such as a greedy algorithm that generates an overall scheme by gradually selecting the optimal local allocation scheme; a genetic algorithm that generates multiple optimized schemes by simulating selection, crossover, and mutation operations in biological evolution; or a simulated annealing algorithm that finds the globally optimal allocation scheme through gradual cooling. For each simulated allocation scheme, the corresponding load imbalance degree needs to be calculated. There are several ways to calculate the load imbalance degree, such as by calculating the standard deviation of the load of each node (a smaller standard deviation indicates a more balanced load); or by calculating the difference between the maximum and minimum load values ​​of each node (a smaller difference indicates a more uniform load distribution). Finally, among all simulation schemes, the scheme with the smallest load imbalance is selected. The number of nodes corresponding to this scheme is the final determined maximum number of parallel nodes. The value determined in this way not only meets the parallel requirements of the task, but also ensures the load balance of the system nodes, thus providing a guarantee for the efficient execution of the task.

[0052] In some alternative implementations, the maximum number of parallel nodes Nmax(T) can be obtained as follows: Determine the total system resource capacity: First, it is necessary to determine the total computing resource capacity of the system, which includes: Total number of CPU cores: The sum of the number of CPU cores on all nodes in the system.

[0053] Total memory: The sum of the memory capacity of all nodes in the system.

[0054] Total I / O bandwidth: The sum of the I / O bandwidth of all nodes in the system.

[0055] The total amount of these resources forms the basis of the maximum concurrent processing capacity that the system can support.

[0056] Assess the extent to which tasks can be executed in parallel: The parallelism P(T) of the task has been calculated using the method described earlier, and it reflects the degree to which task T can be processed in parallel. The closer the value of P(T) is to 1, the more fully parallel the task can be executed; the closer the value of P(T) is to 0, the more closely the task can be executed serially.

[0057] Calculate the maximum parallel resource requirements: Next, we calculate the proportion of resources that can theoretically be used in parallel during task execution based on the task parallelism degree P(T). This proportion can be simply represented by P(T) itself, as it is already a value between 0 and 1.

[0058] Then, multiply this ratio by the total system resource capacity to obtain the theoretical maximum parallel resource requirement. Taking CPU resources as an example, the calculation formula is as follows: Maximum parallel CPU requirement = P(T) × total number of CPU cores Similarly, the maximum parallel memory requirement and the maximum parallel I / O bandwidth requirement can be calculated.

[0059] Determine the maximum number of parallel nodes Nmax(T): Finally, based on the node's resource configuration (such as the number of CPU cores and memory per node), the theoretical maximum parallel resource requirement is mapped to the specific number of nodes. This typically involves a resource allocation optimization problem, but it can be simplified to a basic proportional allocation problem.

[0060] For example, assuming each node is allocated an average number of CPU cores, the maximum number of parallel nodes Nmax(T) can be calculated using the following formula (taking CPU resources as an example): Nmax(T)CPU= Average number of CPU cores per node / Maximum parallel CPU requirement in, This indicates rounding up to ensure there are enough nodes to meet resource requirements.

[0061] Similarly, the maximum number of parallel nodes based on memory and I / O bandwidth can be calculated, and the minimum of these values ​​can be taken as the final maximum number of parallel nodes Nmax(T) to ensure balanced utilization of system resources in all aspects: Nmax(T)=min(Nmax(T)CPU,Nmax(T)Mem,Nmax(T)IO) This method allows for the precise calculation of the maximum number of nodes, Nmax(T), that can be executed in parallel for task T, providing a crucial basis for optimizing task allocation strategies. This technical solution fully discloses the calculation process of Nmax(T), ensuring its feasibility and verifiability.

[0062] Furthermore, the specific method for obtaining the load imbalance degree I can be as follows: When simulating various task allocation schemes, the load imbalance degree I needs to be calculated to evaluate the load balancing degree of each scheme. The load imbalance degree I is a key indicator that measures the load difference between nodes in task allocation; the lower the value, the more balanced the load allocation.

[0063] 1. Collect node load data First, for each allocation scheme, we need to collect the current load data for each node. This load data may include, but is not limited to: CPU utilization: The current CPU usage level on the node.

[0064] Memory utilization: The percentage of memory currently used on the node.

[0065] I / O bandwidth utilization: The current I / O bandwidth usage on the node.

[0066] To simplify calculations, one or more key metrics can be selected as the load measurement standard, or multiple metrics can be combined into a single load value through weighted summation.

[0067] 2. Calculate the average load. Next, the average load of all nodes is calculated as a reference for the ideal balanced state. Assuming there are n nodes, and the load of the i-th node is Li, the average load Lavg is calculated as follows: 3. Calculate the load imbalance degree I. Load imbalance I can be measured in several ways; the following are two commonly used methods: Method 1: Standard Deviation Method Standard deviation is a common method for measuring the dispersion of data and can be used to calculate load imbalance. The formula for calculating the standard deviation of load imbalance I is: The smaller the value of Istd, the smaller the load difference between nodes, that is, the more balanced the load.

[0068] Method 2: Difference between maximum and minimum values Another simple and intuitive method is to calculate the difference between the maximum and minimum load values. While this method is not as precise as the standard deviation method, it is simple to calculate and easy to understand. The formula for calculating the difference between the maximum and minimum load imbalance I is: Imax-min=max(Li) min(Li); The smaller the value of Imax-min, the more balanced the load.

[0069] Furthermore, the allocation scheme with the minimum load imbalance I is selected as the optimal allocation plan. This plan specifies the tasks to be allocated to S nodes, and the number of subtasks or executors to be allocated to each node.

[0070] This step, based on the previous calculations of task parallelism, maximum number of parallel nodes, simulation of various allocation schemes, and calculation of load imbalance, selects the optimal allocation scheme. By comparing the load imbalance under different allocation schemes, the scheme with the minimum load imbalance is selected as the optimal allocation plan. This plan specifies which specific nodes the tasks will be assigned to, and the number of subtasks or executors to be allocated to each node. This selection process ensures full utilization of system resources and load balancing during task execution, improving task execution efficiency and overall system performance.

[0071] In the task allocation method of this application embodiment, the task attribute parameters affecting task parallelism are obtained and quantified and weighted to obtain a comprehensive quantized value. The comprehensive quantized value is then normalized to obtain the task parallelism. A parallelism-node number mapping function is constructed based on historical task data to obtain the candidate value of the maximum number of parallel nodes. Then, a load balancing algorithm is used to simulate various task allocation schemes and select the number of nodes corresponding to the scheme with the smallest load imbalance. This achieves accurate determination of the maximum number of parallel nodes, providing reliable support for the load balancing allocation and efficient execution of subsequent tasks, and further improving the rationality of task processing and the optimization and utilization level of system resources.

[0072] Optionally, the construction of the parallelism-node number mapping function based on historical task data includes: Collect historical task datasets, which include the task parallelism of each historical task, the optimal number of parallel nodes actually running for each historical task, resource capacity parameters and resource utilization data during the execution of each historical task. The historical task dataset is standardized to obtain a standardized training dataset; Using the task parallelism of historical tasks as input features and the corresponding optimal number of parallel nodes as output labels, a mapping function model is constructed. Based on the model training algorithm and the standardized training dataset, the mapping function model is trained to determine that the trained mapping function model is a parallelism-node number mapping function. The model training algorithm includes at least one of linear regression algorithm, decision tree algorithm and neural network algorithm.

[0073] In this implementation, the aforementioned historical task dataset can be understood as a collection of relevant data accumulated by the task processing system during the execution of various tasks in the past, serving as the basic data support for constructing the mapping function. The aforementioned resource capability parameters can be understood as resource-related characteristic parameters possessed by the node during task execution, reflecting the node's resource carrying capacity and processing potential. The aforementioned resource occupancy rate data can be understood as the proportion of various node resources used during task execution, reflecting the task's consumption of node resources.

[0074] The standardization process described above can be understood as adjusting historical task datasets of different scales and ranges to a uniform data scale. The aim is to eliminate scale differences between data and improve the accuracy of model training. The standardized training dataset described above can be understood as a set of data that, after standardization, meets the requirements for model training data and can be directly used in the training process of the mapping function model.

[0075] The input features mentioned above can be understood as variables that serve as input to the model during training, providing the basis for the model's calculations and predictions. The output labels mentioned above can be understood as variables that serve as the model's output targets during training, representing the results the model needs to learn and predict. The mapping function model mentioned above can be understood as a model structure used to establish the correspondence between input features and output labels, capable of outputting corresponding prediction results based on the input feature values.

[0076] The above model training algorithm can be understood as a calculation method used to train the mapping function model so that it can accurately learn the relationship between input features and output labels.

[0077] The aforementioned historical task dataset must contain key information to support the construction of the mapping function. Specifically, this may include the task parallelism of each historical task, the optimal number of parallel nodes actually running for each historical task, resource capability parameters during the execution of each historical task, and resource utilization data. This data can be extracted from the task processing system's runtime logs or aggregated from a historical task execution database. For resource capability parameters, data such as node CPU speed, memory capacity, and I / O transfer rate can be collected. For resource utilization data, data such as CPU utilization, memory utilization, and I / O utilization during task execution can be collected. Comprehensive collection of this data provides a rich and effective data foundation for the subsequent construction of the mapping function.

[0078] Furthermore, since the collected historical task data comes from different task scenarios and execution periods, the scale and range of different data types may vary significantly, which could negatively impact training performance if used directly for model training. Therefore, the historical task dataset can be standardized. Various methods can be used, such as linear standardization to map the data to a value range of 0 to 1, standard deviation standardization to convert the data into standardized data with a mean of 0 and a standard deviation of 1, or other suitable methods like normalization based on the data's distribution characteristics. After standardization, all types of data in the historical task dataset will be on a uniform scale, eliminating interference between data points and forming a standardized training dataset that can be directly used for model training.

[0079] The aforementioned mapping function model can establish the correlation between task parallelism and the optimal number of parallel nodes. When constructing the model, the historical task parallelism in the standardized training dataset is explicitly used as the input feature, and the optimal number of parallel nodes actually used in the historical task is used as the output label. The model structure can be flexibly selected according to actual needs. For example, a simple linear model structure can be constructed, suitable for scenarios with relatively simple data relationships. A complex nonlinear model structure can also be constructed to fit complex mapping relationships between data. Furthermore, an appropriate model complexity can be selected based on the size and characteristics of the data to ensure that the model can accurately capture the correspondence between input features and output labels.

[0080] The training algorithm for the aforementioned model can be at least one of linear regression, decision tree, and neural network algorithms. For example, when there is a clear linear relationship between the data, linear regression can be used for training to obtain the mapping relationship by fitting a linear equation. When the relationship between the data is more complex and there are multiple branch decision logics, decision tree can be used to learn the classification rules and mapping relationships in the data by constructing a decision tree structure. When the amount of data is large and the nonlinear relationship between the data is significant, neural network can be used to construct a high-precision mapping function model by utilizing its powerful nonlinear fitting and feature learning capabilities. During the training process, a standardized training dataset can be continuously input, and the model parameters can be continuously adjusted until the error between the model's prediction result and the actual output label reaches the preset requirement. After training, the mapping function model can accurately output the corresponding maximum number of parallel nodes reference value according to the input task parallelism. At this point, the trained mapping function model is determined to be the required parallelism-node number mapping function.

[0081] In the task allocation method of this application embodiment, a standardized training dataset is obtained by collecting a historical task dataset containing the task parallelism of each historical task, the optimal number of parallel nodes actually running, the resource capability parameters during execution, and the resource occupancy rate data. The dataset is then standardized. A mapping function model is constructed using the task parallelism of the historical tasks as the input feature and the corresponding optimal number of parallel nodes as the output label. The model is trained using at least one of the following model training algorithms: linear regression, decision tree, and neural network. This achieves the accurate construction of the parallelism-node number mapping function, providing reliable support for the accurate derivation of the maximum number of parallel nodes for the target task, and further improving the rationality of task allocation and the efficient utilization of system resources.

[0082] Optionally, obtaining the load balancing task allocation plan based on the maximum number of parallel nodes includes: Based on the maximum parallel node value, obtain the range of nodes that can participate in task allocation and the available executor resource information of each node; Obtain the total amount of the subtasks; Based on the total number of subtasks, the range of nodes that can participate in task allocation, and executor resource information, an allocation optimization algorithm is used to simulate and generate multiple different subtask allocation schemes under the constraint of not exceeding the maximum number of parallel nodes. Each subtask allocation scheme includes the number of subtasks allocated to each node. For each subtask allocation scheme, based on the number of subtasks allocated to each node and the node's resource carrying capacity, the load imbalance degree corresponding to each subtask allocation scheme is obtained through preset load balancing evaluation indicators. Obtain the subtask allocation scheme with the minimum load imbalance.

[0083] In this implementation, the aforementioned range of nodes eligible for task allocation can be understood as the set of nodes that meet the conditions for participating in the current task allocation, under the constraint of the maximum number of parallel nodes, and represents the range of potential targets for subsequent task allocation. The aforementioned available executor resource information can be understood as information related to executors currently idle or ready to be deployed for task execution on a node, including the type, quantity, and remaining resource capacity of the executors.

[0084] The total number of subtasks mentioned above can be understood as the total number of all subtasks obtained after breaking down the target task, serving as the basic quantitative basis for task allocation. The allocation optimization algorithm mentioned above can be understood as various algorithms used to optimize the subtask allocation method to achieve the goal of load balancing, capable of generating optimal or near-optimal allocation schemes under constraints. The load balancing evaluation index mentioned above can be understood as a standard used to measure the degree of load balancing of each node in the subtask allocation scheme, serving as the core basis for calculating the degree of load imbalance.

[0085] The maximum parallel node value clearly defines the upper limit of the number of nodes that can execute a task in parallel. This value can be used to filter the range of nodes eligible for allocation. There are several filtering methods. For example, nodes with resource idle rates meeting a preset standard can be selected from all nodes in the system to form the range of eligible nodes. Alternatively, nodes that match the executor type required by the target task can be selected, and then nodes not exceeding the maximum parallel node value can be chosen to form the range. While determining the node range, the available executor resource information for each node can also be obtained. This can be done by directly extracting from the executor status records maintained by the system, or by querying the type, quantity, and remaining resources of currently available executors through real-time communication with each node, ensuring that the obtained resource information accurately reflects the real-time carrying capacity of the nodes.

[0086] The total number of subtasks is the basic data for task allocation, and the method of obtaining it can be flexibly chosen according to the actual situation of task splitting. For example, you can directly count the specific number of subtasks obtained after splitting the target task to get the accurate total number of subtasks; or you can estimate the total number of subtasks by calculating based on the total data volume of the target task and the preset data volume of each subtask, to ensure that the subsequent allocation plan can be formulated based on reasonable total data.

[0087] The choice of the aforementioned allocation optimization algorithm can be flexible and diverse. For example, a greedy algorithm can be used, allocating subtasks to each node sequentially according to the remaining resource quantity, from most to least, until all subtasks are allocated or the maximum number of parallel nodes is reached. Alternatively, a genetic algorithm can be used, treating the subtask allocation scheme as chromosomes and iteratively generating multiple optimized allocation schemes under constraints through selection, crossover, and mutation operations. Simulated annealing can also be used, starting with a randomly generated initial allocation scheme and gradually adjusting the number of subtasks to generate multiple different schemes. Each allocation scheme needs to clearly define the number of subtasks allocated to each node, and the number of nodes involved in the scheme must not exceed the maximum number of parallel nodes, ensuring that all schemes meet the basic constraints of parallel task execution.

[0088] For each subtask allocation scheme, the load imbalance degree corresponding to each scheme can be obtained based on the number of subtasks allocated to each node and the node's resource carrying capacity, using preset load balancing evaluation indicators. Load balancing evaluation indicators can be set in various ways. For example, the standard deviation of the load on each node can be used as the evaluation indicator; the smaller the standard deviation, the more balanced the load on each node. Alternatively, the difference between the maximum and minimum load values ​​on each node can be used as the evaluation indicator; the smaller the difference, the more uniform the load distribution. When calculating the load imbalance degree, it is necessary to consider the node's resource carrying capacity, such as the node's CPU and memory carrying capacity. Based on the number of allocated subtasks, the load of each node is estimated, and then quantified using preset evaluation indicators to obtain the load imbalance degree corresponding to each scheme, providing a basis for subsequent scheme selection.

[0089] The load imbalance of each of the generated subtask allocation schemes can be compared and ranked, and the scheme with the smallest load imbalance can be selected as the final load balancing task allocation plan.

[0090] The task allocation method in this application embodiment can ensure that subtasks are evenly distributed among the nodes that can participate in the allocation, avoiding the situation where some nodes are overloaded while others are idle, giving full play to the overall efficiency of system resources, and providing strong support for the efficient and stable execution of tasks.

[0091] Optionally, after allocating the subtasks in the subtask set to the executors of the corresponding nodes according to the allocation plan, the method further includes: real-time monitoring of node load and task execution status, and adjusting the task allocation scheme based on the monitoring results.

[0092] In this implementation, the aforementioned node load can be understood as the resource occupancy status of a node during task execution, reflecting the node's current workload and remaining resource capacity. The aforementioned task execution status can be understood as the running status of subtasks on the executor, including execution progress, whether it is running normally, and whether any abnormal interruptions have occurred. The aforementioned monitoring results can be understood as the relevant data and status feedback information collected and organized after real-time monitoring of node load and task execution status.

[0093] Adjusting the task allocation scheme can be understood as optimizing and adjusting the existing subtask allocation method based on monitoring results to adapt to changes in node load and task execution requirements.

[0094] To promptly grasp the system's operational status, continuous monitoring of node load and task execution status is possible. When monitoring node load, metrics such as CPU utilization, memory usage, and I / O activity can be monitored, providing a comprehensive picture of node resource consumption. When monitoring task execution status, the progress of subtasks, data processing volume, and the occurrence of errors or stutters can be tracked, ensuring timely detection of problems during task execution. Multiple monitoring methods are available. For example, periodic data collection can be used, gathering node load and task execution status data at preset intervals, which can be flexibly set according to system sensitivity requirements. Alternatively, real-time streaming can be employed to continuously monitor the relevant status of nodes and tasks, ensuring the latest operational information is obtained as soon as possible.

[0095] After obtaining monitoring results, the data can be analyzed and judged. If it is found that the current task allocation scheme can no longer meet the needs of efficient system operation or load balancing, an adjustment process can be initiated. The adjustment method can be flexibly selected according to the actual situation. For example, when the monitoring results show that the load of a certain node is too high and is close to the resource capacity limit, the unfinished subtasks on that node can be migrated to other nodes with lower load. During migration, nodes with high matching degree with the subtask requirements and idle resources can be selected first, or the nodes can be allocated sequentially according to the load balancing priority. If it is found that some nodes have low load and a lot of idle resources, while other nodes still have unfinished subtasks, some subtasks on other nodes can be scheduled to these idle nodes to make full use of system resources. In addition, if abnormal task execution status is monitored, such as a subtask execution being stuck or reporting an error, the allocation scheme can also be adjusted to reassign the subtask to other executors, or split it and distribute it to multiple executors to complete it collaboratively, ensuring that the task can proceed smoothly.

[0096] In some alternative implementations, the load on the actuator can be monitored in real time, and task allocation can be dynamically adjusted according to the actual situation to ensure that load imbalance is minimized.

[0097] Load change scenarios and targeted adjustment strategies: In scenarios with sudden load increases: Load change: Due to the sudden high computational load of a certain actuator (such as batch data analysis), the CPU utilization rate rose sharply from 30% to 90%, and the memory utilization rate was also close to full load.

[0098] Adjust strategy: The task processing system immediately detected the abnormal load on the executor and initiated the task migration process.

[0099] Migrate some non-critical or parallelizable tasks (such as data preprocessing and logging) to other low-load executors.

[0100] Meanwhile, if the task processing system resources allow, additional computing resources can be dynamically allocated (such as starting a standby executor) to share the load.

[0101] In scenarios with unbalanced load: Load variation: In a distributed task processing system, some executors may experience excessive load due to uneven task distribution, while other executors may remain idle.

[0102] Adjust strategy: The task processing system uses load monitoring tools to detect load imbalances and calculates the load differences between each actuator.

[0103] Based on load differences and task characteristics, some tasks on executors with excessive loads are migrated to executors with lower loads.

[0104] During the migration process, priority should be given to the dependencies between tasks and the compatibility of executors to ensure a smooth migration and efficient execution.

[0105] In scenarios with fluctuating loads: Load variation: The load of an actuator fluctuates within a specific time period due to handling periodic tasks (such as scheduled data backup).

[0106] Adjust strategy: The task processing system can predict the load fluctuation pattern of the actuator by analyzing historical load data.

[0107] Before peak load periods, migrate some non-urgent tasks to other actuators in advance to reserve sufficient resources to cope with load fluctuations.

[0108] At the same time, we optimized the task scheduling strategy to ensure that the task processing system can still operate stably during load fluctuations.

[0109] Through the above process, not only is it possible to intelligently allocate tasks to the most suitable executor based on task characteristics and node resource availability, but also to minimize load imbalance through dynamic adjustment and real-time monitoring mechanisms, thereby improving the overall performance and stability of the task processing system.

[0110] In the task allocation method of this application embodiment, such dynamic adjustment enables the system to always maintain a load-balanced state, thereby improving the stability and efficiency of task execution.

[0111] Optionally, the real-time monitoring of node load and task execution status, and the adjustment of the task allocation scheme based on the monitoring results, includes: Monitor node load and task execution status; When the change in node load exceeds a preset threshold or the delay in task execution exceeds a preset time, the tasks of the overloaded node are migrated to the node that meets the first condition. The first condition includes: the degree of matching between the node's resource capability parameters and the executor requirements corresponding to the migration task is within a preset matching range, and the node's load is lower than the system average load.

[0112] In this implementation, the aforementioned preset threshold can be understood as a pre-set benchmark value used to determine whether changes in node load have reached a level requiring adjustment of task allocation, and is a key criterion for triggering task migration. The aforementioned task execution delay can be understood as the time difference between the start of task execution and the expected completion time, i.e., the duration by which the actual execution progress lags behind the planned progress.

[0113] The aforementioned preset time can be understood as the maximum allowed delay for task execution. Exceeding this time is considered an excessive delay, requiring the activation of an adjustment mechanism. The aforementioned overload node can be understood as a node whose load exceeds its reasonable carrying capacity or whose load variation exceeds a preset threshold, resulting in the inability to efficiently execute the current task.

[0114] The first condition mentioned above can be understood as a set of core conditions used to screen nodes that can accept migration tasks, ensuring that the migrated tasks can be executed efficiently without disrupting the system's load balance.

[0115] The aforementioned preset matching range can be understood as a reasonable range of matching between the pre-defined node resource capability parameters and the executor requirements corresponding to the migration task. It is an important basis for determining whether a node is suitable for the migration task. The aforementioned average system load can be understood as the average load of all nodes participating in task execution in the task processing system, which can reflect the overall load level of the system.

[0116] When monitoring node load, core metrics such as CPU utilization, memory usage, and I / O activity can be collected in real time to dynamically track load changes. When monitoring task execution status, information such as task progress, data processing volume, and whether there are any stutters or anomalies can be recorded in real time, providing a comprehensive understanding of the task's operational status. The monitoring frequency can be flexibly set according to system requirements. For example, a high-frequency real-time monitoring approach can be used to ensure that status changes are captured immediately; alternatively, a dynamic adjustment approach can be adopted, reducing the monitoring frequency when the system load is stable and increasing the monitoring frequency when the load fluctuates significantly, balancing monitoring accuracy and resource consumption.

[0117] Determining whether the task migration trigger conditions are met involves real-time analysis of collected information based on continuous monitoring to determine if the conditions for task migration have been met. There are two trigger conditions; meeting either one will initiate migration. The first condition is that the change in node load exceeds a preset threshold. This preset threshold can be flexibly adjusted based on the node's resource characteristics and system operational requirements. For example, it can be set to a change in node load not exceeding 30% per unit time, or it can be set to a fixed load value change, such as a change in CPU utilization not exceeding 20 percentage points. The second condition is that the task execution delay exceeds a preset time. The preset time can be set according to the task type and priority. For example, for tasks with high real-time requirements, the preset time can be set to 5 seconds, and for non-real-time tasks, it can be set to 30 seconds. Clearly defined trigger conditions prevent unnecessary task migrations and ensure targeted adjustments.

[0118] When the triggering condition is met, nodes that meet the first condition need to be selected as target nodes for task migration. The first condition includes two core requirements, both of which must be met for a node to become a target node. The first requirement is that the node's resource capacity parameters match the executor requirements of the migration task within a preset matching range. This preset matching range can be set according to the importance of the task and execution requirements. For example, the matching degree can be set to no less than 80%, or it can be set to a matching degree between 70% and 95%, ensuring that the target node can adapt to the execution requirements of the migration task. The second requirement is that the node's load is lower than the system average load. The system average load can be calculated in various ways, such as calculating the average load of all nodes participating in task execution, or calculating the moving average of node load over a recent period, ensuring that the target node has sufficient idle resources to handle the migration task. When selecting target nodes, they can be selected in descending order of matching degree, or a comprehensive sorting based on node load redundancy can be used, prioritizing nodes with high matching degree and large load redundancy. Once the target node is identified, tasks on the overloaded node can be smoothly migrated to that node. During the migration process, breakpoint resume can be used to ensure that task data is not lost, or the scheduler can coordinate to achieve a seamless transition of task execution status, ensuring that tasks can continue to execute efficiently after migration, while restoring the system to a load-balanced state.

[0119] In the task allocation method of this application embodiment, by monitoring the node load and task execution status, when the change in node load exceeds a preset threshold or the delay in task execution exceeds a preset time, the tasks of overloaded nodes are migrated to nodes whose resource capacity parameters and the executor requirements corresponding to the migrated tasks are within a preset matching range and whose load is lower than the average load of the system. This achieves timely response to system operation anomalies and dynamic optimization of task allocation, effectively alleviates node overload problems, ensures continuous and efficient task execution, maintains overall system load balance, and further improves the stability of task processing and the rational utilization of system resources.

[0120] In some alternative implementations, reference may be made to Figure 2 , Figure 2 This is the second flowchart illustrating the task allocation method provided in this application. The task processing system can consist of three parts: a scheduler, multiple distributed nodes, and a user interface. The scheduler is the core component of the system, responsible for intelligently deploying executors on the distributed nodes and rationally allocating tasks according to task requirements. The scheduler maintains an executor table, which records in detail the status (e.g., used, available) of different types of executors on each node, ensuring that the most suitable execution resources are accurately matched during task allocation. The scheduler employs an advanced load balancing algorithm to dynamically adjust the task allocation strategy to maximize resource utilization and achieve rapid task response.

[0121] Distributed nodes are the actual carriers of task execution. Each node is equipped with different types of executors according to system requirements to support diverse task processing. Nodes maintain close communication with the scheduler, updating the executor status in real time to ensure that the scheduler can accurately grasp resource availability and allocate resources effectively. Simultaneously, nodes also support collaborative execution of complex tasks, achieving parallel processing and result aggregation through efficient network communication and data processing capabilities.

[0122] The user interface provides a convenient platform for task submission and management. Users submit task requirements through the interface, including key information such as task type, parallelism requirements, and expected execution time. Based on user needs, the system automatically matches the most suitable executor and node resources to ensure efficient and accurate task completion. The user interface also provides functions such as task status query and result feedback, allowing users to easily monitor task progress and results at any time.

[0123] In summary, this system architecture, through the close collaboration of the scheduler, distributed nodes, and user interface, achieves efficient task execution and flexible resource management, providing users with stable and reliable task processing services.

[0124] For a detailed execution diagram, please refer to... Figure 3 Node responsibilities and key relationships: User interface: Responsibilities: Provide a task submission and management platform that allows users to submit task requirements (such as task type, parallelism requirements, expected execution time, etc.), query task status, and obtain result feedback.

[0125] Relationship: Communicates closely with the scheduler, passing user-submitted task requests to the scheduler for processing.

[0126] Scheduler: Responsibilities: As a core component of the system, responsible for intelligently deploying executors on distributed nodes and rationally allocating tasks according to task requirements. Maintaining the executor table to record the status of different types of executors on each node, and dynamically adjusting the task allocation strategy using a load balancing algorithm.

[0127] Relationships: Communicates with the user interface to receive task requests, and communicates with distributed nodes to update the executor status in real time and allocate tasks.

[0128] Distributed nodes: Responsibilities: As the actual vehicle for task execution, it deploys different types of executors to support diverse task processing. It maintains close communication with the scheduler, updates the executor status in real time, and supports collaborative execution of complex tasks between nodes.

[0129] Relationships: Communicates with the scheduler to receive task assignments and update executor status, and collaborates with other distributed nodes to process tasks in parallel and aggregate results.

[0130] Actuator Table: Responsibilities: Record the status of different types of executors on each node (such as used, available, etc.) to provide the scheduler with a basis for task allocation decisions.

[0131] Association: As part of the scheduler, it is used by the scheduler to manage and query the executor status.

[0132] In some optional implementations, the task allocation method of this application embodiment can be specifically implemented through the following steps: Step 1: Based on the matching of executor type and node type, place the corresponding type of executor on the node and record it in the executor table.

[0133] The executor type can be classified according to the following dimensions, but the same executor may involve only one dimension or multiple dimensions.

[0134] Computationally intensive: If the task mainly involves a large number of mathematical operations or logical processing, an executor with strong computing power should be selected, such as an executor equipped with a high-performance CPU or GPU.

[0135] Memory-intensive: If the task needs to process a large amount of data or generate a large number of intermediate results, an executor with a large memory capacity should be selected.

[0136] I / O intensive: If the task involves frequent input / output operations, such as reading and writing disks or network transfers, an actuator with superior I / O performance should be selected.

[0137] Actuator type classification: Classified according to the following dimensions, but the same node may involve only one dimension or multiple dimensions.

[0138] Computational Executors (CE): Focused on high-performance computing, equipped with high-performance CPUs or GPUs.

[0139] Memory-based executors (ME): These have a large memory capacity and are suitable for processing large datasets.

[0140] I / O actuators (IOE): Optimized I / O performance, suitable for tasks requiring frequent read and write operations.

[0141] For example, the specific execution process can be as follows: Step 1.1 Define the set of executor types E={E1,E2,...,Em} and the set of nodes N={N1,N2,...,Nn}.

[0142] Step 1.2 For each actuator type Ei and node Nj, calculate the matching degree M(Ei,Nj), which is based on the degree of matching between the node's CPU performance, memory capacity, I / O speed and other characteristics and the actuator requirements.

[0143] Step 1.3 uses the matching degree matrix M to evenly distribute each type of executor across the nodes with the highest matching degree. There are several possible allocation methods: evenly distribute executors across all matching nodes, select the nodes with the most resources from all matching nodes and allocate them according to resource weights, or select the nodes with the highest matching degree to allocate executors.

[0144] Step 2: Determine the maximum number of parallel nodes: Task parallelism (P(T)): For task T, its parallelism P(T) is a value between 0 and 1, representing the degree to which the task can be parallelized. P(T)=1 indicates that the task is fully parallel, and P(T)=0 indicates that the task is fully serial.

[0145] Maximum number of parallel nodes (Nmax(T)): The maximum number of nodes that a task can be executed in parallel, determined based on the parallelism P(T) of task T and the system resources.

[0146] Load imbalance (I): A metric that measures the difference in load distribution among nodes, with the goal of minimizing this metric to achieve load balancing.

[0147] Step 2.1 Parallelism-Node Number Mapping Function: Define a function f(P(T)) that maps the task parallelism P(T) to the maximum number of parallel nodes Nmax(T). This function can be customized according to system resources and task characteristics; for example, it can be a linear, logarithmic, or other form of function.

[0148] Step 2.2 Calculate the maximum number of parallel nodes: For each task T, calculate the maximum number of parallel nodes Nmax(T) based on its parallelism P(T) and mapping function f(P(T)).

[0149] Step 3: Task Assignment: Step 3.1 Pre-allocation planning: Step 3.1.1 Determine the maximum number of nodes that can be allocated based on the maximum number of parallel nodes Nmax(T) of task T.

[0150] Step 3.1.2 Using variations or optimization methods of the load balancing algorithm (such as greedy algorithm, genetic algorithm, simulated annealing, etc.), simulate various allocation schemes without exceeding Nmax(T), and calculate the load imbalance degree I under each scheme. The load imbalance degree I can be measured by indicators such as standard deviation and the difference between the maximum and minimum values.

[0151] Step 3.1.3 Select the allocation scheme with the minimum load imbalance I as the optimal allocation plan. This plan specifies the tasks to be allocated to S nodes, and the number of subtasks or executors to be allocated to each node.

[0152] Step 3.2 Task Assignment and Execution: Step 3.2.1 Task division: Based on the optimal allocation plan, the split subtasks are further subdivided into smaller tasks suitable for the executor on each node.

[0153] Step 3.2.2 Task Allocation: Distribute the divided subtasks to the executors on different nodes according to the optimal allocation plan. Ensure that the number of executors or the amount of tasks on each node are as close as possible to minimize the load imbalance I.

[0154] For example, the assignment of nodes and executors can be as follows: Node 1: It is assigned two computational executors (CE1, CE2), which are focused on high-performance computing and are suitable for handling mathematical operations and logical processing tasks.

[0155] Node 2: It is allocated one memory-based executor (ME1) and two I / O-based executors (IOE1, IOE2). The memory-based executor ME1 has a large memory capacity and is suitable for processing large datasets; the I / O-based executors IOE1 and IOE2 optimize I / O performance and are suitable for tasks with frequent read and write operations, with IOE2 having higher performance.

[0156] Node 3: It is allocated one computational executor (CE3) and one memory-based executor (ME2). Although the performance of CE3 is slightly lower than that of CE1 and CE2, it still has computing power; ME2 has a medium memory capacity and can handle medium-sized datasets.

[0157] Task Description: Task 1: A computationally intensive task involving a large number of mathematical operations.

[0158] Task 2: A memory-intensive task that requires processing large amounts of data and generating intermediate results.

[0159] Task 3: An I / O intensive task involving frequent input / output operations.

[0160] Task allocation and execution process: Task 1 Assignment: Since Task 1 is a computationally intensive task, it should be prioritized to be assigned to a computationally intensive executor.

[0161] Of all the nodes, the computational executors on nodes 1 and 3 have the highest matching degree with task 1.

[0162] To minimize load imbalance, the subtasks of task 1 are equally distributed to CE1 on node 1 and CE3 on node 3, so that the load on the two nodes is balanced.

[0163] Task 2 Assignment: Task 2 is a memory-intensive task, requiring a large amount of memory to store and process data.

[0164] Among all nodes, the memory executor (ME1) on node 2 and the memory executor (ME2) on node 3 have a high degree of matching with task 2.

[0165] Considering the large amount of data processing in Task 2, the main data processing part of Task 2 is assigned to the ME1 executor on Node 2, which has a larger memory capacity, while some auxiliary subtasks with smaller data processing volume are assigned to the ME2 executor on Node 3.

[0166] To further minimize load imbalance, task allocation is dynamically adjusted based on the resource status and workload of the two nodes, ensuring that the load on each node is as close as possible.

[0167] Task 3 Assignment: Task 3 is an I / O intensive task, requiring frequent input and output operations.

[0168] Among all nodes, the two I / O actuators (IOE1, IOE2) on node 2 have the highest matching degree with task 3.

[0169] To fully utilize the high performance of IOE2, subtasks requiring high I / O performance in Task 3 are assigned to IOE2, while other subtasks are assigned to IOE1.

[0170] Meanwhile, the load status of the two actuators is monitored in real time, and the task allocation is dynamically adjusted according to the actual situation to ensure that the load imbalance is minimized.

[0171] Through the above process, not only is it possible to intelligently allocate tasks to the most suitable executor based on task characteristics and node resource availability, but also to minimize load imbalance through dynamic adjustment and real-time monitoring mechanisms, thereby improving the overall performance and stability of the system.

[0172] Please see Figure 4 , Figure 4 This is a schematic diagram of a task allocation device 200 provided in an embodiment of this application. As shown in the figure, the task allocation device 200 includes: The first acquisition module 210 is used to acquire the degree of matching between each node of the task processing system and the requirements of the executor; The placement module 220 is used to construct a matching degree matrix between the executor and the node based on the matching degree, and to place the executor to the corresponding node based on the matching degree matrix; The second acquisition module 230 is used to acquire the maximum number of parallel nodes of the target task; The third acquisition module 240 is used to split the target task into a set of subtasks and obtain a load-balanced task allocation plan based on the maximum parallel node value. The allocation module 250 is used to allocate subtasks in the subtask set to the executors of the nodes of the task processing system according to the allocation plan.

[0173] Optionally, the second acquisition module 230 can also be used for: Obtain task attribute parameters that affect task parallelism, wherein the task attribute parameters include at least one of subtask dependency, data interaction complexity, computational logic correlation, and data processing scale; Each task attribute parameter is quantified, and a comprehensive quantized value is obtained based on the quantified task attribute parameters and their corresponding weights. The comprehensive quantized value is normalized to obtain the task parallelism. A parallelism-node number mapping function is constructed based on historical task data, and the parallelism of the task is mapped to a candidate value of the maximum number of parallel nodes through the parallelism-node number mapping function; Based on the candidate value of the maximum number of parallel nodes, a variety of task allocation schemes are simulated through a load balancing algorithm to obtain the load imbalance degree of each scheme, and the number of nodes corresponding to the scheme with the smallest load imbalance degree is determined as the maximum number of parallel nodes.

[0174] Optionally, the second acquisition module 230 can also be used for: Collect historical task datasets, which include the task parallelism of each historical task, the optimal number of parallel nodes actually running for each historical task, resource capacity parameters and resource utilization data during the execution of each historical task. The historical task dataset is standardized to obtain a standardized training dataset; Using the task parallelism of historical tasks as input features and the corresponding optimal number of parallel nodes as output labels, a mapping function model is constructed. Based on the model training algorithm and the standardized training dataset, the mapping function model is trained to determine that the trained mapping function model is a parallelism-node number mapping function. The model training algorithm includes at least one of linear regression algorithm, decision tree algorithm and neural network algorithm.

[0175] Optionally, the third acquisition module 240 can also be used for: Based on the maximum parallel node value, obtain the range of nodes that can participate in task allocation and the available executor resource information of each node; Obtain the total amount of the subtasks; Based on the total number of subtasks, the range of nodes that can participate in task allocation, and executor resource information, an allocation optimization algorithm is used to simulate and generate multiple different subtask allocation schemes under the constraint of not exceeding the maximum number of parallel nodes. Each subtask allocation scheme includes the number of subtasks allocated to each node. For each subtask allocation scheme, based on the number of subtasks allocated to each node and the node's resource carrying capacity, the load imbalance degree corresponding to each subtask allocation scheme is obtained through preset load balancing evaluation indicators. Obtain the subtask allocation scheme with the minimum load imbalance.

[0176] Optionally, the task allocation device 200 can also be used for: Monitor node load and task execution status in real time, and adjust the task allocation scheme based on the monitoring results.

[0177] Optionally, the task allocation device 200 can also be used for: Monitor node load and task execution status; When the change in node load exceeds a preset threshold or the delay in task execution exceeds a preset time, the tasks of the overloaded node are migrated to the node that meets the first condition. The first condition includes: the degree of matching between the node's resource capability parameters and the executor requirements corresponding to the migration task is within a preset matching range, and the node's load is lower than the system average load.

[0178] The task allocation device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiments shown achieve the same technical effects, and will not be described again here to avoid repetition.

[0179] For details, see Figure 5 As shown in the figure, this application embodiment also provides an electronic device, including a bus 301, a transceiver 302, an antenna 303, a bus interface 304, a processor 305, and a memory 306.

[0180] Processor 305, used for: To determine the degree of matching between the various nodes of the task processing system and the requirements of the executor; Based on the matching degree, a matching degree matrix between the executor and the node is constructed, and the executor is deployed to the corresponding node based on the matching degree matrix; Obtain the maximum number of parallel nodes for the target task; The target task is split into a set of subtasks, and a load-balanced task allocation plan is obtained based on the maximum number of parallel nodes. According to the allocation plan, the subtasks in the subtask set are allocated to the executors of the corresponding nodes.

[0181] exist Figure 5 In this context, a bus architecture (represented by bus 301) is used. Bus 301 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 305 and memory represented by memory 306. Bus 301 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 304 provides an interface between bus 301 and transceiver 302. Transceiver 302 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 305 is transmitted over a wireless medium via antenna 303, which further receives data and transmits it to processor 305.

[0182] Processor 305 manages bus 301 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 306 can be used to store data used by processor 305 during operation.

[0183] Alternatively, the processor 305 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0184] Optionally, the processor 305 can also be used for: Obtain task attribute parameters that affect task parallelism, wherein the task attribute parameters include at least one of subtask dependency, data interaction complexity, computational logic correlation, and data processing scale; Each task attribute parameter is quantified, and a comprehensive quantized value is obtained based on the quantified task attribute parameters and their corresponding weights. The comprehensive quantized value is normalized to obtain the task parallelism. A parallelism-node number mapping function is constructed based on historical task data, and the parallelism of the task is mapped to a candidate value of the maximum number of parallel nodes through the parallelism-node number mapping function; Based on the candidate value of the maximum number of parallel nodes, a variety of task allocation schemes are simulated through a load balancing algorithm to obtain the load imbalance degree of each scheme, and the number of nodes corresponding to the scheme with the smallest load imbalance degree is determined as the maximum number of parallel nodes.

[0185] Optionally, the processor 305 can also be used for: Collect historical task datasets, which include the task parallelism of each historical task, the optimal number of parallel nodes actually running for each historical task, resource capacity parameters and resource utilization data during the execution of each historical task. The historical task dataset is standardized to obtain a standardized training dataset; Using the task parallelism of historical tasks as input features and the corresponding optimal number of parallel nodes as output labels, a mapping function model is constructed. Based on the model training algorithm and the standardized training dataset, the mapping function model is trained to determine that the trained mapping function model is a parallelism-node number mapping function. The model training algorithm includes at least one of linear regression algorithm, decision tree algorithm and neural network algorithm.

[0186] Optionally, the processor 305 can also be used for: Based on the maximum parallel node value, obtain the range of nodes that can participate in task allocation and the available executor resource information of each node; Obtain the total amount of the subtasks; Based on the total number of subtasks, the range of nodes that can participate in task allocation, and executor resource information, an allocation optimization algorithm is used to simulate and generate multiple different subtask allocation schemes under the constraint of not exceeding the maximum number of parallel nodes. Each subtask allocation scheme includes the number of subtasks allocated to each node. For each subtask allocation scheme, based on the number of subtasks allocated to each node and the node's resource carrying capacity, the load imbalance degree corresponding to each subtask allocation scheme is obtained through preset load balancing evaluation indicators. Obtain the subtask allocation scheme with the minimum load imbalance.

[0187] Optionally, the processor 305 can also be used for: Monitor node load and task execution status in real time, and adjust the task allocation scheme based on the monitoring results.

[0188] Optionally, the processor 305 can also be used for: Monitor node load and task execution status; When the change in node load exceeds a preset threshold or the delay in task execution exceeds a preset time, the tasks of the overloaded node are migrated to the node that meets the first condition. The first condition includes: the degree of matching between the node's resource capability parameters and the executor requirements corresponding to the migration task is within a preset matching range, and the node's load is lower than the system average load.

[0189] It should be noted that the electronic device provided in this application embodiment is a device capable of executing the above-described task allocation method. Therefore, all implementation methods in the above-described task allocation method embodiments are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0190] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described task allocation method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.

[0191] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described task allocation method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0192] This application also provides a computer program product, including computer instructions. When these computer instructions are executed by a processor, they implement the various processes of the above-described task allocation method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0195] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A task allocation method, characterized in that, The method includes: To determine the degree of matching between the various nodes of the task processing system and the requirements of the executor; Based on the matching degree, a matching degree matrix between the executor and the node is constructed, and the executor is deployed to the corresponding node based on the matching degree matrix; Obtain the maximum number of parallel nodes for the target task; The target task is split into a set of subtasks, and a load-balanced task allocation plan is obtained based on the maximum number of parallel nodes. According to the allocation plan, the subtasks in the subtask set are allocated to the executors of the corresponding nodes.

2. The method according to claim 1, characterized in that, The method of obtaining the maximum number of parallel nodes for the target task includes: Obtain task attribute parameters that affect task parallelism, wherein the task attribute parameters include at least one of subtask dependency, data interaction complexity, computational logic correlation, and data processing scale; Each task attribute parameter is quantified, and a comprehensive quantized value is obtained based on the quantified task attribute parameters and their corresponding weights. The comprehensive quantized value is normalized to obtain the task parallelism. A parallelism-node number mapping function is constructed based on historical task data, and the parallelism of the task is mapped to a candidate value of the maximum number of parallel nodes through the parallelism-node number mapping function; Based on the candidate value of the maximum number of parallel nodes, a variety of task allocation schemes are simulated through a load balancing algorithm to obtain the load imbalance degree of each scheme, and the number of nodes corresponding to the scheme with the smallest load imbalance degree is determined as the maximum number of parallel nodes.

3. The method according to claim 2, characterized in that, The parallelism-node number mapping function constructed based on historical task data includes: Collect historical task datasets, which include the task parallelism of each historical task, the optimal number of parallel nodes actually running for each historical task, resource capacity parameters and resource utilization data during the execution of each historical task. The historical task dataset is standardized to obtain a standardized training dataset; Using the task parallelism of historical tasks as input features and the corresponding optimal number of parallel nodes as output labels, a mapping function model is constructed. Based on the model training algorithm and the standardized training dataset, the mapping function model is trained to determine that the trained mapping function model is a parallelism-node number mapping function. The model training algorithm includes at least one of linear regression algorithm, decision tree algorithm and neural network algorithm.

4. The method according to any one of claims 1 to 3, characterized in that, The process of obtaining a load-balanced task allocation plan based on the maximum number of parallel nodes includes: Based on the maximum parallel node value, obtain the range of nodes that can participate in task allocation and the available executor resource information of each node; Obtain the total amount of the subtasks; Based on the total number of subtasks, the range of nodes that can participate in task allocation, and executor resource information, an allocation optimization algorithm is used to simulate and generate multiple different subtask allocation schemes under the constraint of not exceeding the maximum number of parallel nodes. Each subtask allocation scheme includes the number of subtasks allocated to each node. For each subtask allocation scheme, based on the number of subtasks allocated to each node and the node's resource carrying capacity, the load imbalance degree corresponding to each subtask allocation scheme is obtained through preset load balancing evaluation indicators. Obtain the subtask allocation scheme with the minimum load imbalance.

5. The method according to any one of claims 1 to 3, characterized in that, After allocating the subtasks in the subtask set to the corresponding node executors according to the allocation plan, the method further includes: real-time monitoring of node load and task execution status, and adjusting the task allocation scheme based on the monitoring results.

6. The method according to claim 5, characterized in that, The real-time monitoring of node load and task execution status, and the adjustment of the task allocation scheme based on the monitoring results, include: Monitor node load and task execution status; Nodes whose load changes exceed a preset threshold or whose task execution delay exceeds a preset time are identified as overloaded nodes, and their tasks are migrated to nodes that meet the first condition. The first condition includes: the degree of matching between the node's resource capability parameters and the executor requirements corresponding to the migration task is within a preset matching range, and the node's load is lower than the system average load.

7. A task allocation device, characterized in that, include: The first acquisition module is used to acquire the degree of matching between each node of the task processing system and the requirements of the executor; The placement module is used to construct a matching degree matrix between the executor and the node based on the matching degree, and to place the executor to the corresponding node based on the matching degree matrix; The second acquisition module is used to acquire the maximum number of parallel nodes for the target task. The third acquisition module is used to split the target task into a set of subtasks and obtain a load-balanced task allocation plan based on the maximum parallel node value. The allocation module is used to allocate subtasks in the subtask set to the executors of the corresponding nodes according to the allocation plan.

8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the task allocation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the task allocation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the task allocation method as described in any one of claims 1 to 6.

Images