Intelligent allocation method and system of computing resources for multi-task parallel training

Abstract

This invention provides a method and system for intelligent allocation of computing resources for multi-task parallel training, relating to the field of computing resource management technology. It includes acquiring resource request information such as the computational requirements, storage capacity requirements, and communication bandwidth requirements of multiple tasks to be trained; abstracting computing resources into a resource pool composed of divisible and composable resource atoms; and combining these atoms to form resource quota units with computational capacity and available time indicators. A matching function is established between training tasks and resource quota units to quantify the supply-demand matching degree; a system of resource competition constraint equations is constructed to solve for the optimal allocation scheme, forming initial resource binding relationships and planning execution time slices. After parallel training starts, the resource consumption rate and training progress speed are monitored in real time; when a nonlinear deviation is detected, the resource quota units are dynamically disassembled and reassembled.

Description

Technical Field

[0001] This invention relates to the field of computing resource management technology, and in particular to a method and system for intelligent allocation of computing resources for multi-task parallel training. Background Technology

[0002] With the rapid development of deep learning applications, running multiple training tasks in parallel on the same computing cluster has become a common practice to improve resource utilization. In existing technologies, computing resources are typically allocated using a static partitioning strategy, meaning that resource requirements are configured once and for all based on the submission of the task. Administrators or scheduling systems allocate hardware resources such as GPUs, CPUs, and memory to each training task according to a preset ratio, and maintain this resource configuration throughout the task's execution cycle. This approach is simple to operate, easy to manage and monitor, provides clear resource boundaries, and avoids mutual interference between tasks.

[0003] Existing static resource allocation methods have revealed significant limitations in practical applications. The resource requirements of training tasks vary across different training phases. For example, the model initialization phase has lower computational resource requirements, while the parameter update intensive phase demands a surge in computational and communication bandwidth. Static allocation cannot respond to these dynamic changes, resulting in resources being idle at some times and insufficient at others. When multiple tasks run in parallel, resource competition exists between them, and static allocation schemes struggle to achieve optimal resource supply and demand matching at the global level, often leading to an imbalance where some tasks have excess resources while others are underrepresented. Current technologies lack flexible control mechanisms for resource granularity, typically allocating GPUs and other hardware as indivisible units, failing to provide fine-grained resource combinations and adjustments based on actual task requirements, resulting in resource waste. These shortcomings hinder the overall resource utilization of computing clusters, extend task completion times, and restrict the efficiency of multi-task parallel training. Summary of the Invention

[0004] The present invention provides a method and system for intelligent allocation of computing resources for multi-task parallel training, which can solve the problems in the prior art.

[0005] A first aspect of the present invention provides a method for intelligent allocation of computing resources for multi-task parallel training, comprising:

[0006] Obtain resource request information for multiple tasks to be trained, including computational requirements, storage capacity requirements, and communication bandwidth requirements;

[0007] Computing resources are abstracted into a resource pool composed of multiple resource atoms. These resource atoms are divisible and composable. Multiple resource quota units are formed by combining these resource atoms. Each resource quota unit has a computing power index and an available duration index.

[0008] Based on the resource request information and the computing power index, a matching degree function between training tasks and resource quota units is established to quantify the matching degree between the demand of each training task and the supply of resources.

[0009] Based on the matching degree, a set of constraint equations is constructed for the resource competition relationship among multiple tasks in the resource pool. The optimal resource quota unit allocation scheme of each training task under the competition constraint is solved to form an initial resource binding relationship. Execution time slices are planned for each training task according to the available time index.

[0010] Initiate multi-task parallel training, monitor the resource consumption rate and training progress speed of each training task in real time, and when a non-linear deviation is detected in the resource consumption rate and training progress speed of each training task, disassemble the corresponding resource quota unit and reorganize the new resource quota unit and allocate it to the corresponding training task.

[0011] Computing resources are abstracted as a resource pool composed of multiple resource atoms. These resource atoms are divisible and composable. Multiple resource quota units are formed by combining these resource atoms. Each resource quota unit has a computing capacity indicator and an available duration indicator, including:

[0012] The computing resources are hierarchically decomposed, and the computing resources are divided into computing resource atoms, storage resource atoms and communication resource atoms according to their functional attributes, and the resource capacity attribute is labeled for each type of resource atom;

[0013] Establish combination constraint rules for resource atoms. The combination constraint rules define compatibility conditions and combination ratio boundaries between different types of resource atoms. The compatibility conditions determine whether resource atoms can jointly form a resource quota unit. The combination ratio boundaries limit the quantity ratio of each type of resource atoms within the same resource quota unit. Based on the combination constraint rules, select multiple resource atoms that meet the compatibility conditions and aggregate them to form a resource quota unit.

[0014] For each resource quota unit, based on the resource capacity attributes of the resource atoms participating in the combination and the combination ratio boundary, fusion weights are assigned to the resource capacity attributes of different types of resource atoms, and the computing capability index is calculated by weighted summation of the fusion weights.

[0015] Based on the expected allocation cycle and release cycle of the resource quota unit, the available duration index is determined, and the resource quota unit and the corresponding computing power index and available duration index are stored in the resource pool.

[0016] Based on the resource request information and the computing power index, a matching function is established between training tasks and resource quota units to quantify the matching degree between the demand for each training task and the supply of resources, including:

[0017] The computational requirements, storage capacity requirements, and communication bandwidth requirements in the resource request information are normalized to obtain standardized requirement values ​​for each requirement dimension.

[0018] Using the intensity of the obstruction to the progress of training tasks due to insufficient resources in each demand dimension as matrix elements, a demand sensitivity matrix is ​​constructed. Based on the demand sensitivity matrix, the standardized demand values ​​of each demand dimension are weighted and mapped to obtain the sensitivity weighted demand vector of each training task.

[0019] The computing power index is decomposed into supply components of each type according to resource type. The deviation between the sensitivity-weighted demand vector and each supply component is calculated dimension by dimension. The maximum value of the deviation of each dimension is the bottleneck constraint term, and the dispersion of the deviation of each dimension is the equilibrium constraint term. A matching degree function is constructed based on the composite relationship between the bottleneck constraint term and the equilibrium constraint term, so that the matching degree value is negatively correlated with both the bottleneck constraint term and the equilibrium constraint term.

[0020] Substitute the sensitivity-weighted demand vector of each training task and the computational capability index of each resource quota unit into the matching degree function to output the matching degree between each training task and each resource quota unit.

[0021] Based on the matching degree, a set of constraint equations is constructed to address the resource competition relationship among multiple tasks in the resource pool. The optimal resource quota unit allocation scheme for each training task under the competition constraint is solved to form an initial resource binding relationship. Furthermore, execution time slices are planned for each training task according to the available time index, including:

[0022] The matching degree between each training task and each resource quota unit is used to form a matching degree feature vector. The cosine similarity between any two matching degree feature vectors is calculated as the competition correlation coefficient, and a competition correlation matrix between all tasks is constructed.

[0023] Using the inter-task competition correlation matrix as the quadratic coefficient matrix, the total amount of each resource quota unit as the linear capacity constraint, and the minimum requirement of each training task as the lower bound constraint of the requirement, a set of constraint equations is constructed with the goal of minimizing the weighted penalty of competition correlation.

[0024] The constraint equations are solved using the interior point obstacle function method. The linear capacity constraint and the lower bound of demand constraint are internalized into the objective function using logarithmic obstacle terms. The resource quota units and dual variables of each training task are iteratively updated until the KKT optimal conditions are met, and the optimal resource quota unit allocation scheme is output.

[0025] According to the optimal resource quota unit allocation scheme, each training task is bound to the corresponding resource quota unit to form an initial resource binding relationship; based on the available duration index and the throughput capacity of the resource quota unit bound to each training task, the execution duration of each training task is divided into ordered time slices according to the load balancing principle to obtain execution time slices.

[0026] The constraint equations are solved using the interior-point barrier function method. Logarithmic barrier terms are used to internalize the linear capacity constraint and the lower bound of demand constraint into the objective function. The resource quota units and dual variables for each training task are iteratively updated until the KKT optimality condition is met. The optimal resource quota unit allocation scheme is then output, including:

[0027] Extract the linear capacity constraints and demand lower bound constraints from the constraint equation set, construct a logarithmic barrier function for each inequality constraint, initialize the barrier parameters, and add the product of the logarithmic barrier function and the barrier parameters to the competition-related weighted penalty objective function to form an augmented objective function;

[0028] Set the initial values ​​of the quota allocation for each training task and the dual variable, calculate the gradient vector and Hessian matrix of the augmented objective function, perform diagonal dominance preprocessing on the Hessian matrix, improve the condition number through diagonal scaling transformation, and construct the preconditioned Newton equation.

[0029] The preconditional Newton equation is solved using the conjugate gradient iteration method. The process terminates when the preset number of inner layer iterations is reached or the residual decreases to the inner layer convergence threshold. An approximate Newton direction is output. Based on the approximate Newton direction, a line search algorithm is used to calculate the step size parameter that satisfies the sufficient descent condition. The resource quota unit and dual variable are updated synchronously along the approximate Newton direction and the step size parameter.

[0030] The centrality deviation index is quantified by the degree of violation of the complementary relaxation conditions. The decay rate of the barrier parameters is adaptively adjusted according to the centrality deviation index. The KKT conditional residual is calculated. If it is less than the outer convergence threshold, the current resource quota unit is output as the optimal resource quota unit allocation scheme. Otherwise, the barrier parameters are updated and the augmented objective function formation step is returned.

[0031] When a non-linear deviation is detected between the resource consumption rate and the training progress speed of each training task, the corresponding resource quota unit is disassembled and a new resource quota unit is reassembled and allocated to the corresponding training task, including:

[0032] After the multi-task parallel training is started, the resource consumption rate and training progress speed of each training task are collected synchronously. Using the deviation of the resource consumption rate and the deviation of the training progress speed as the two axes, a two-dimensional deviation trajectory of each training task is constructed in each monitoring cycle. The trajectory curvature sequence is constructed using the change rate of the direction angle of adjacent segments of the two-dimensional deviation trajectory.

[0033] When the trajectory curvature sequence of each training task shows a continuously monotonically increasing inflection point, it is determined that the resource consumption rate deviates non-linearly from the training progress speed. The angle between the trajectory tangent direction at the inflection point and the dual axes is extracted, and the axis corresponding to the smallest angle is taken as the dominant deviation axis. The dismantling intensity is determined by the curvature increment at the inflection point.

[0034] The dismantling is driven by the deviation from the dominant axis and the dismantling intensity. The dismantling quota of each resource subclass is determined based on the dismantling intensity. The quota is then adjusted differentially by combining the tangent direction component of the trajectory at the inflection point, and reorganized into a new resource quota unit to be allocated to the corresponding training task.

[0035] The dismantling is driven by the deviation from the dominant axis and the dismantling intensity. The dismantling quota for each resource subclass is determined based on the dismantling intensity. Differential adjustments are made based on the trajectory tangent direction component at the inflection point, and the resulting units are reorganized into new resource quota units and allocated to the corresponding training tasks. This includes:

[0036] The axial allocation weight of each resource subclass is determined by the proportion of the absolute value of the component obtained by orthogonally projecting the deviation from the dominant axis to the reference axis of each resource subclass in the sum of the absolute values ​​of all resource subclass projections. The initial dismantling quota of each resource subclass is determined by the product of the axial allocation weight and the dismantling intensity.

[0037] The cosine of the angle between the trajectory tangent vector at the inflection point and the direction deviating from the dominant axis is taken as the tangent dominant coupling coefficient. The ratio of the absolute value of the orthogonal projection component of the tangent vector onto the reference axis of each resource subclass to the magnitude of the tangent vector is taken as the tangent component coefficient of each resource subclass. The product of the tangent dominant coupling coefficient and the tangent component coefficient is normalized to obtain the tangent adjustment weight of each resource subclass. The initial decomposition quota of each resource subclass is proportionally scaled by the tangent adjustment weight to obtain the adjusted quota of each resource subclass.

[0038] Using the dismantling intensity as the upper bound of the total, the difference between the sum of the adjusted quotas of all resource subclasses and the dismantling intensity is used to backfill the quotas according to the axial allocation weight ratio of each resource subclass. The backfilled quotas of each resource subclass are then encapsulated into new resource quota units and allocated to the corresponding training tasks.

[0039] A second aspect of the present invention provides an intelligent allocation system for computing resources in multi-task parallel training, comprising:

[0040] An information acquisition unit is used to acquire resource request information for multiple tasks to be trained, including computational requirements, storage capacity requirements, and communication bandwidth requirements.

[0041] The resource abstraction unit is used to abstract computing resources into a resource pool composed of multiple resource atoms. The resource atoms are divisible and composable. Multiple resource quota units are formed by combining the resource atoms. Each resource quota unit has a computing power index and an available time index.

[0042] The matching degree building unit is used to establish a matching degree function between training tasks and resource quota units based on the resource request information and the computing power index, and to quantify the matching degree between the needs of each training task and the resource supply.

[0043] The allocation planning unit is used to construct a set of constraint equations for the resource competition relationship among multiple tasks in the resource pool based on the matching degree, solve the optimal resource quota unit allocation scheme for each training task under the competition constraint, form an initial resource binding relationship, and plan an execution time slice for each training task according to the available time index.

[0044] The dynamic adjustment unit is used to initiate multi-task parallel training, monitor the resource consumption rate and training progress speed of each training task in real time, and when it is detected that the resource consumption rate and training progress speed of each training task deviate non-linearly, the corresponding resource quota unit is disassembled and a new resource quota unit is reorganized and allocated to the corresponding training task.

[0045] A third aspect of the present invention provides an electronic device, comprising:

[0046] processor;

[0047] Memory used to store processor-executable instructions;

[0048] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0049] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0050] The beneficial effects of this method are as follows:

[0051] Abstracting computing resources into divisible and composable resource atoms breaks through the limitations of fixed-specification allocation in traditional resource allocation methods. It allows for flexible combination into resource quota units based on the actual needs of training tasks, achieving fine-grained resource management and improving resource utilization efficiency. By establishing a matching function between training tasks and resource quota units, the matching degree between the needs of each training task and the resource supply can be quantified, transforming resource allocation from experience-driven to data-driven. This avoids resource waste or insufficiency caused by blind allocation and ensures that each training task receives a resource combination that matches its characteristics.

[0052] By constructing a system of constraint equations based on matching degree and solving for the optimal allocation scheme of resource competition relationships among multiple tasks, the resource conflict problem in multi-task parallel scenarios is effectively solved. This maximizes global resource utilization while meeting the basic needs of each task, avoiding situations where some tasks wait for extended periods due to insufficient resources. Furthermore, by combining the available time index of resource quota units for time-slice planning, reasonable resource scheduling across time dimensions is achieved. This allows multiple training tasks to utilize key resources at off-peak times, further improving resource concurrency utilization and shortening the overall training cycle.

[0053] By monitoring the nonlinear deviation between the resource consumption rate and the training progress speed in real time, changes in resource requirements during the execution of training tasks can be detected in a timely manner. Resource quota units can be dynamically decomposed and reorganized for adjustment, avoiding the problem that static allocation schemes cannot cope with the dynamic characteristics of training. This ensures that resources are always highly matched with the actual needs of the task and effectively prevents resource idleness or overload. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the intelligent allocation method for computing resources in multi-task parallel training according to an embodiment of the present invention.

[0055] Figure 2 This is a flowchart illustrating a method for organizing computing resources into multiple resource quota units according to an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0058] Figure 1 This is a flowchart illustrating the intelligent allocation method for computing resources in multi-task parallel training according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0059] Obtain resource request information for multiple tasks to be trained, including computational requirements, storage capacity requirements, and communication bandwidth requirements;

[0060] Computing resources are abstracted into a resource pool composed of multiple resource atoms. These resource atoms are divisible and composable. Multiple resource quota units are formed by combining these resource atoms. Each resource quota unit has a computing power index and an available duration index.

[0061] Based on the resource request information and the computing power index, a matching degree function between training tasks and resource quota units is established to quantify the matching degree between the demand of each training task and the supply of resources.

[0062] Based on the matching degree, a set of constraint equations is constructed for the resource competition relationship among multiple tasks in the resource pool. The optimal resource quota unit allocation scheme of each training task under the competition constraint is solved to form an initial resource binding relationship. Execution time slices are planned for each training task according to the available time index.

[0063] Initiate multi-task parallel training, monitor the resource consumption rate and training progress speed of each training task in real time, and when a non-linear deviation is detected in the resource consumption rate and training progress speed of each training task, disassemble the corresponding resource quota unit and reorganize the new resource quota unit and allocate it to the corresponding training task.

[0064] Figure 2 This is a flowchart illustrating a method for grouping computing resources into multiple resource quota units in an embodiment of the present invention. In an optional implementation, computing resources are abstracted into a resource pool composed of multiple resource atoms. These resource atoms are divisible and composable. Multiple resource quota units are formed by combining these resource atoms. Each resource quota unit has a computing capacity indicator and an availability time indicator, including:

[0065] The computing resources are hierarchically decomposed, and the computing resources are divided into computing resource atoms, storage resource atoms and communication resource atoms according to their functional attributes, and the resource capacity attribute is labeled for each type of resource atom;

[0066] Establish combination constraint rules for resource atoms. The combination constraint rules define compatibility conditions and combination ratio boundaries between different types of resource atoms. The compatibility conditions determine whether resource atoms can jointly form a resource quota unit. The combination ratio boundaries limit the quantity ratio of each type of resource atoms within the same resource quota unit. Based on the combination constraint rules, select multiple resource atoms that meet the compatibility conditions and aggregate them to form a resource quota unit.

[0067] For each resource quota unit, based on the resource capacity attributes of the resource atoms participating in the combination and the combination ratio boundary, fusion weights are assigned to the resource capacity attributes of different types of resource atoms, and the computing capability index is calculated by weighted summation of the fusion weights.

[0068] Based on the expected allocation cycle and release cycle of the resource quota unit, the available duration index is determined, and the resource quota unit and the corresponding computing power index and available duration index are stored in the resource pool.

[0069] Computational resources are hierarchically decomposed into computational resource atoms, storage resource atoms, and communication resource atoms based on their functional attributes. Computational resource atoms mainly include CPU cores, GPU computing units, and neural network processors, used to handle various computational tasks in model training. Storage resource atoms mainly include RAM, video memory, and solid-state storage, used to store training datasets and model parameters. Communication resource atoms include network bandwidth, bus channels, and RDMA connections, used to ensure data transmission and inter-node communication. Each type of resource atom is labeled with a resource capacity attribute. The capacity attribute of computational resource atoms can be represented as floating-point operations per second (Floats Per Second) or instruction throughput; the capacity attribute of storage resource atoms can be represented as storage space size or read / write speed; and the capacity attribute of communication resource atoms can be represented as bandwidth size or latency level. This labeling method provides a quantitative basis for subsequent combinations of resource atoms.

[0070] Establish combination constraint rules for resource atoms. These rules include two parts: compatibility conditions and combination ratio boundaries. Compatibility conditions determine whether different resource atoms can jointly constitute a resource quota unit. For example, a specific type of GPU computing unit can only be used with a specific type of GPU memory, while some high-performance computing cores require high-speed cache to achieve optimal performance. For instance, a GPU core must be paired with PCIe bus bandwidth, and a CPU thread must be paired with DDR memory blocks; otherwise, they are considered incompatible. Combination ratio boundaries limit the ratio of the number of different types of resource atoms within the same resource quota unit. For example, in deep learning training scenarios, each GPU computing unit typically needs to match a certain capacity of memory and corresponding communication bandwidth. This ratio can be obtained through analysis of historical training data. For deep learning training tasks, the ratio of computational to storage resource atoms is set between 1:8 and 1:16, meaning each GPU core is configured with 8 to 16 memory blocks; the ratio of computational to communication resource atoms is set between 1:2 and 1:4 to ensure that data transmission does not become a computational bottleneck.

[0071] Based on the combination constraint rules, resource atoms that meet the compatibility conditions are selected from the resource pool for aggregation. The aggregation process requires determining the core resource type of the resource quota unit. For example, for computationally intensive training tasks, computational resource atoms can be selected as the core, and then an appropriate number of storage and communication resource atoms can be matched according to the combination ratio boundary. For data-intensive training tasks, storage resource atoms can be selected as the core, and then corresponding computational and communication resources can be matched. In this way, resource quota units with specific functional tendencies are formed, providing differentiated support for different types of training tasks.

[0072] For each resource quota unit, its comprehensive computing power index needs to be calculated. The calculation method involves assigning fusion weights to different types of resource atoms based on their resource capacity attributes and the combination ratio boundaries. The allocation of fusion weights can be based on the resource's impact on the training task performance. For example, for computationally intensive training tasks, the fusion weight for computationally intensive resource atoms is set to 0.6, for storage-intensive ones to 0.25, and for communication-intensive ones to 0.15; for data-intensive tasks, the fusion weight for storage-intensive resource atoms is increased to 0.5, for computationally intensive ones to 0.3, and for communication-intensive ones to 0.2. The resource capacity attributes of each type of resource atom are normalized to the range of 0 to 1, multiplied by the corresponding fusion weight, and then summed to obtain a computing power index value between 0 and 1. A higher value indicates a stronger comprehensive processing capability for the resource quota unit.

[0073] Based on the resource allocation cycle and task scheduling strategy of the resource scheduling system, the availability time index of the resource quota unit is determined. In a cluster environment, the availability time is related to the system load, priority strategy and reservation mechanism. For example, for high-priority training tasks, a longer continuous availability time can be allocated; for training tasks that can be interrupted, multiple discontinuous time segments can be allocated. By predictively analyzing historical resource usage patterns and the current cluster status, the expected allocation cycle and release cycle of the resource quota unit can be reasonably estimated, thereby determining its availability time index.

[0074] The constructed resource quota units and their corresponding computing power and available time indicators are stored in the resource pool to form a resource list that can be scheduled. The resource list is stored using a hierarchical index structure, which supports fast retrieval by multiple dimensions such as computing power, available time, and resource composition, providing efficient query support for matching subsequent training tasks with resource quota units.

[0075] In practical applications, for deep learning model training scenarios, computational resource atoms can be subdivided into dedicated accelerators suitable for different algorithms. For example, for convolutional neural network training, resource atoms with high tensor computation capabilities can be configured; for natural language processing tasks, resource atoms with efficient sequence processing capabilities can be configured. This fine-grained division improves the matching accuracy between resources and tasks.

[0076] The combination of resource atoms can be implemented using heuristic algorithms. Based on the resource requirement characteristics of the training task, a set of candidate resource atoms is selected from the resource pool. Optimization methods such as simulated annealing or genetic algorithms are then used to find the optimal combination scheme while satisfying the combination constraints. The optimization result is then transformed into specific resource quota units. For dynamically changing training environments, the construction of resource quota units can employ an adaptive strategy. By monitoring changes in resource utilization during training in real time, resource bottlenecks are identified, and the combination ratio of resource atoms is adjusted.

[0077] In one optional implementation, a matching function between training tasks and resource quota units is established based on the resource request information and the computing power index to quantify the matching degree between the demand for each training task and the supply of resources, including:

[0078] The computational requirements, storage capacity requirements, and communication bandwidth requirements in the resource request information are normalized to obtain standardized requirement values ​​for each requirement dimension.

[0079] Using the intensity of the obstruction to the progress of training tasks due to insufficient resources in each demand dimension as matrix elements, a demand sensitivity matrix is ​​constructed. Based on the demand sensitivity matrix, the standardized demand values ​​of each demand dimension are weighted and mapped to obtain the sensitivity weighted demand vector of each training task.

[0080] The computing power index is decomposed into supply components of each type according to resource type. The deviation between the sensitivity-weighted demand vector and each supply component is calculated dimension by dimension. The maximum value of the deviation of each dimension is the bottleneck constraint term, and the dispersion of the deviation of each dimension is the equilibrium constraint term. A matching degree function is constructed based on the composite relationship between the bottleneck constraint term and the equilibrium constraint term, so that the matching degree value is negatively correlated with both the bottleneck constraint term and the equilibrium constraint term.

[0081] Substitute the sensitivity-weighted demand vector of each training task and the computational capability index of each resource quota unit into the matching degree function to output the matching degree between each training task and each resource quota unit.

[0082] In this specific embodiment, the computational load requirements, storage capacity requirements, and communication bandwidth requirements in the resource request information are normalized to eliminate the dimensional differences between different resource types. Specifically, for each type of resource requirement, the corresponding requirement values ​​of all currently scheduled training tasks are collected, the maximum and minimum values ​​are identified, and the minimum-maximum normalization method is used to map the requirement values ​​of each task to the range of zero to one. For example, for computational load requirements, assuming that the maximum computational load requirement of all current tasks is one trillion floating-point operations and the minimum is ten billion floating-point operations, then for a task with a requirement of five hundred billion floating-point operations, its normalized computational load requirement is (5000-100) / (10000-100) = 0.494. Similarly, the same normalization process is performed on storage capacity requirements and communication bandwidth requirements to make resource requirements of different dimensions comparable.

[0083] To accurately assess the impact of different resource dimensions on training progress, a demand sensitivity matrix is ​​constructed. The rows of this matrix represent different training tasks, the columns represent different resource types, and the matrix elements represent the blocking intensity on the training task's progress when the corresponding resources are insufficient. The blocking intensity can be obtained by successively decreasing the supply of each resource dimension under controlled conditions, measuring the change in the execution speed of the training task, fitting a curve showing the relationship between resource supply and training speed, and calculating the slope of the curve as a sensitivity index. For example, in deep learning model training, the blocking intensity of computational resources is usually high for feature extraction convolutional networks; while for attention-based models, memory bandwidth will become the main bottleneck, manifesting as a high blocking intensity of storage resources.

[0084] Based on the constructed demand sensitivity matrix, the standardized demand values ​​of each training task across different resource dimensions are weighted. Specifically, the demand sensitivity row vector of each training task is multiplied element-wise with its standardized demand value vector to obtain a sensitivity-weighted demand vector. This vector reflects the actual dependence of the training task on various resources, taking into account the actual impact of resource shortages on training performance. For example, if the standardized demand values ​​for a training task in the three dimensions of computation, storage, and communication are 0.8, 0.5, and 0.3, respectively, and the corresponding sensitivity values ​​are 0.9, 0.4, and 0.2, then the weighted demand vector is [0.72, 0.2, 0.06], indicating that this task's dependence on computational resources is much higher than that on other resources.

[0085] The computing power index of a resource quota unit is decomposed dimensionally, representing it as a supply vector with the same dimension as the sensitivity-weighted demand vector. For example, the comprehensive computing power index of a resource quota unit is decomposed into computing power components, storage capacity components, and communication capacity components. The decomposition process considers the quantity and performance of various resource atoms in the resource quota unit, as well as their synergistic effects. For a resource quota unit containing multiple GPU computing units, its computing power component is not only related to the number of GPUs and the performance of a single GPU, but also affected by the interconnect architecture between GPUs; similarly, the storage capacity component also needs to consider the impact of the cache hierarchy on actual access latency.

[0086] The deviation between the sensitivity-weighted demand vector and the supply vector is calculated dimension by dimension. The deviation calculation adopts the supply-demand ratio method, that is, dividing the supply component by the corresponding demand component to obtain the resource sufficiency rate of each dimension. A resource sufficiency rate greater than one indicates that supply exceeds demand, and less than one indicates that supply exceeds demand. For example, if the weighted demand vector of a certain training task is [0.72, 0.2, 0.06], and the supply vector of a certain resource quota unit is [0.8, 0.3, 0.05], then the resource sufficiency rate of each dimension is [1.11, 1.5, 0.83], indicating that under this configuration, communication resources are slightly insufficient, while computing and storage resources have a certain degree of redundancy.

[0087] Identify the bottleneck constraint and the equilibrium constraint. The bottleneck constraint is defined as the minimum value of the resource sufficiency rate for each dimension, reflecting the supply and demand situation of the scarcest resource. The equilibrium constraint is defined as the coefficient of variation (standard deviation divided by the mean) of the resource sufficiency rate for each dimension, reflecting the degree of equilibrium in resource supply. Continuing the previous example, the bottleneck constraint is 0.83 (communication resource sufficiency rate), and the equilibrium constraint requires calculating the coefficient of variation of [1.11, 1.5, 0.83], reflecting the dispersion of the degree of matching between supply and demand for the three types of resources.

[0088] A matching degree function is constructed such that the matching degree value is negatively correlated with both the bottleneck constraint and the balance constraint. A simple and effective method is to take the reciprocal of the bottleneck constraint and then multiply it by the balance constraint to obtain the final matching degree value. The more severe the bottleneck (lower sufficiency rate) or the more unbalanced the resource allocation (larger coefficient of variation), the larger the matching degree value, indicating a poorer match between the task and the resource quota unit. In practical implementation, different weights can be assigned to the bottleneck constraint and the balance constraint to reflect their importance in resource allocation decisions. For scenarios pursuing high throughput, the weight of the bottleneck constraint can be increased; for scenarios pursuing resource utilization efficiency, the weight of the balance constraint can be increased.

[0089] The sensitivity-weighted demand vector of each training task and the computational capability index of each resource quota unit are substituted into the matching degree function to output a matching degree matrix. The rows of this matrix represent training tasks, the columns represent resource quota units, and the matrix elements represent the degree of matching between the corresponding task and the resource quota unit. The smaller the matrix element value, the better the matching degree. This matching degree matrix provides a quantitative basis for subsequent resource allocation decisions.

[0090] In one optional implementation, based on the matching degree, a set of constraint equations is constructed for the resource competition relationship among multiple tasks in the resource pool. The optimal resource quota unit allocation scheme for each training task under the competition constraint is solved to form an initial resource binding relationship. Then, execution time slices are planned for each training task according to the available time index, including:

[0091] The matching degree between each training task and each resource quota unit is used to form a matching degree feature vector. The cosine similarity between any two matching degree feature vectors is calculated as the competition correlation coefficient, and a competition correlation matrix between all tasks is constructed.

[0092] Using the inter-task competition correlation matrix as the quadratic coefficient matrix, the total amount of each resource quota unit as the linear capacity constraint, and the minimum requirement of each training task as the lower bound constraint of the requirement, a set of constraint equations is constructed with the goal of minimizing the weighted penalty of competition correlation.

[0093] The constraint equations are solved using the interior point obstacle function method. The linear capacity constraint and the lower bound of demand constraint are internalized into the objective function using logarithmic obstacle terms. The resource quota units and dual variables of each training task are iteratively updated until the KKT optimal conditions are met, and the optimal resource quota unit allocation scheme is output.

[0094] According to the optimal resource quota unit allocation scheme, each training task is bound to the corresponding resource quota unit to form an initial resource binding relationship.

[0095] Based on the available time index and the throughput capacity of the resource quota unit bound to each training task, the execution time of each training task is divided into ordered time slices according to the load balancing principle to obtain the execution time slice.

[0096] After determining the matching degree between training tasks and resource quota units, it is necessary to handle resource competition conflicts between multiple tasks. For each training task, extract its matching degree value with all resource quota units, and sort it by resource quota unit number to form a vector. For example, the matching degree feature vector of task i is m. i = [m i1 m i2 , ..., m iN ], where m ijLet represent the matching degree between task i and the j-th resource quota unit. Calculate the cosine similarity of the matching degree feature vectors of any two tasks i and k, using the formula: the dot product of the two vectors divided by the product of their respective norms. This value is used as the competition correlation coefficient c. ik Traverse all task pairs and construct an M×M dimensional competition correlation matrix. The matrix elements reflect the competition intensity between tasks for the same resource quota unit. The higher the similarity, the more intense the competition, indicating that the two tasks have similar preferences for resource quota units.

[0097] Based on the competition correlation matrix, the resource allocation problem is transformed into a constrained optimization model, introducing the decision variable x. ij Let represent the number of resource quota units of type j allocated to task i. The objective function is designed to minimize the competition-related weighted penalty, specifically the weighted sum of the squared competition correlation coefficients between each pair of task allocation vectors, so that highly competitive task pairs obtain differentiated resource quota units. The linear capacity constraint ensures that the total allocation of resource quota units of type j to all tasks does not exceed the total inventory of resource quota units of that type. The demand lower bound constraint ensures that each task obtains at least the number of resource quota units corresponding to its declared minimum computational demand, thus avoiding task starvation.

[0098] The interior-point barrier function method is used to solve the constraint equations. Relaxation variables for capacity constraints and lower bound requirements constraints are introduced separately. A negative logarithmic barrier term is added to the objective function for each inequality constraint. The allocation amounts for each task are initialized to the lower bound of requirements. Initial values ​​for barrier parameters and a contraction coefficient are set. In each iteration, the gradient and Hessian matrix of the objective function with respect to decision and dual variables are calculated while keeping the barrier parameters fixed. Newton's method is used to update the variable values. The original feasibility, dual feasibility, and complementary relaxation conditions are checked. When the gradient norm and dual gap are both less than a set threshold, the KKT optimality condition is satisfied, the iteration terminates, and the current allocation scheme is output. This scheme provides the specific number of various resource quota units that each training task should acquire.

[0099] According to the allocation scheme, a binding record is created for each training task in the system resource management table. The record includes the task identifier, the list of allocated resource quota unit types and their corresponding quantities, and the physical address range of the resource quota units. These binding records are persisted to form an initial resource binding relationship. When a subsequent task starts, it directly reads this relationship to lock its dedicated resource quota unit.

[0100] For time slice division, the available time index of each resource quota unit is extracted, and the throughput capacity, i.e. the amount of data that can be processed per unit time, is calculated in combination with its computing power index. Based on the type and number of resource quota units bound to the task, the total throughput capacity obtained by the task is calculated by summing them up. The estimated total workload of the task is divided by the total throughput capacity to obtain the required execution time. The time axis is divided at a fixed granularity. According to the execution time ratio of each task and the principle of load balancing, a round-robin scheduling strategy is adopted to allocate time slices to each task to ensure that high-throughput tasks get continuous longer time slices and low-throughput tasks get dispersed shorter time slices, so as to maintain a balanced utilization rate of resource quota units.

[0101] In one optional implementation, the constraint equations are solved using the interior-point barrier function method. Logarithmic barrier terms are used to internalize the linear capacity constraint and the lower bound of demand constraint into the objective function. The resource quota units and dual variables for each training task are iteratively updated until the KKT optimality condition is met, and the optimal resource quota unit allocation scheme is output, including:

[0102] Extract the linear capacity constraints and demand lower bound constraints from the constraint equation set, construct a logarithmic barrier function for each inequality constraint, initialize the barrier parameters, and add the product of the logarithmic barrier function and the barrier parameters to the competition-related weighted penalty objective function to form an augmented objective function;

[0103] Set the initial values ​​of the quota allocation for each training task and the dual variable, calculate the gradient vector and Hessian matrix of the augmented objective function, perform diagonal dominance preprocessing on the Hessian matrix, improve the condition number through diagonal scaling transformation, and construct the preconditioned Newton equation.

[0104] The preconditional Newton equation is solved using the conjugate gradient iteration method. The process terminates when the preset number of inner layer iterations is reached or the residual decreases to the inner layer convergence threshold. An approximate Newton direction is output. Based on the approximate Newton direction, a line search algorithm is used to calculate the step size parameter that satisfies the sufficient descent condition. The resource quota unit and dual variable are updated synchronously along the approximate Newton direction and the step size parameter.

[0105] The centrality deviation index is quantified by the degree of violation of the complementary relaxation conditions. The decay rate of the barrier parameters is adaptively adjusted according to the centrality deviation index. The KKT conditional residual is calculated. If it is less than the outer convergence threshold, the current resource quota unit is output as the optimal resource quota unit allocation scheme. Otherwise, the barrier parameters are updated and the augmented objective function formation step is returned.

[0106] After obtaining the competition-related weighted penalty objective function and the constraint equations, extract the form of... Linear capacity constraints and The lower bound constraint of the requirement, for each inequality constraint Constructing a logarithmic barrier function Set initial obstacle parameters Set to 10, and combine all logarithmic obstacle terms with... product Add to the original weighted penalty objective function to form the augmented objective function:

[0107] ,

[0108] Where x represents the resource quota allocation vector for each task.

[0109] Initialize quota allocation for each training task. 0.6 times its requested amount, dual variable Let x be the zero vector, and calculate the first-order partial derivatives of the augmented objective function with respect to x to form the gradient vector. The second-order partial derivatives form the Hessian matrix H. The Hessian matrix H is typically dense and has a large condition number. Diagonal dominance preprocessing is performed on it: diagonal elements of H are extracted to form a diagonal matrix. Calculate the scaling matrix The original Newton equations Convert to preconditional Newton equations After diagonal scaling, the condition number of a matrix can typically be reduced by two orders of magnitude.

[0110] The conjugate gradient iteration method is used to solve the preconditional Newton equations, with the initial search direction set. Calculate in each iteration Update the solution vector and residual Set the upper limit of the number of inner iterations to 50 or reduce the residual norm to 10. -5 As a termination condition, after the iteration terminates, mapping $y$ back to the original space yields the approximate Newtonian direction. The step size is determined using Armijo backtracking search: initial step size α=1, verification. If the condition is not met, multiply α by 0.5 and repeat the test until the sufficient descent condition is met. Then, update the resource quota synchronously according to the determined step size α. With dual variables ,in Exported from the KKT system.

[0111] After each outer layer iteration, the complementary relaxation conditional residuals are calculated:

[0112] ,

[0113] The residual quantifies the degree of deviation between the current iteration point and the center path. When using a conservative attenuation rate ;when When using aggressive decay rate In other cases, the standard attenuation rate is used. Calculate the KKT conditional residuals. and When both are less than the outer convergence threshold of 10 -6 The iteration terminates when the current component of x is reached, and the output is used as the optimal resource quota unit allocation number for each training task; otherwise, the updated value is used. Reconstruct the augmented objective function and execute a new round of inner iterations until the convergence condition is met or the maximum number of outer iterations (100) is reached.

[0114] In one optional implementation, when a non-linear deviation is detected between the resource consumption rate and the training progress speed of each training task, the corresponding resource quota unit is disassembled and a new resource quota unit is reassembled and allocated to the corresponding training task, including:

[0115] After the multi-task parallel training is started, the resource consumption rate and training progress speed of each training task are collected synchronously. Using the deviation of the resource consumption rate and the deviation of the training progress speed as the two axes, a two-dimensional deviation trajectory of each training task is constructed in each monitoring cycle. The trajectory curvature sequence is constructed using the change rate of the direction angle of adjacent segments of the two-dimensional deviation trajectory.

[0116] When the trajectory curvature sequence of each training task shows a continuously monotonically increasing inflection point, it is determined that the resource consumption rate deviates non-linearly from the training progress speed. The angle between the trajectory tangent direction at the inflection point and the dual axes is extracted, and the axis corresponding to the smallest angle is taken as the dominant deviation axis. The dismantling intensity is determined by the curvature increment at the inflection point.

[0117] The dismantling is driven by the deviation from the dominant axis and the dismantling intensity. The dismantling quota of each resource subclass is determined based on the dismantling intensity. The quota is then adjusted differentially by combining the tangent direction component of the trajectory at the inflection point, and reorganized into a new resource quota unit to be allocated to the corresponding training task.

[0118] In this specific embodiment, after parallel training starts, data needs to be collected for all training tasks at a fixed period (e.g., every 10 seconds). For the i-th training task, its current resource consumption rate is collected. This rate is calculated by weighting the change in GPU utilization, the increase in memory usage, and the amount of data transmitted over the network within a unit of time. The training progress speed is collected synchronously. This speed is characterized by a combination of the number of training batches completed per unit of time and the decrease in the loss function.

[0119] The deviation of the collected resource consumption rate is used as the horizontal axis value, and the deviation of the training propulsion speed is used as the vertical axis value. The deviation is calculated as the difference between the current measured value and the expected value when the initial resources were bound. A coordinate point is marked on a two-dimensional plane for each monitoring cycle, and the coordinate points of multiple consecutive cycles are connected sequentially to form a deviation trajectory curve. For this curve, a broken line segment formed by three adjacent monitoring points is selected, and the direction angle of the two segments is calculated. The difference between the two direction angles divided by the time interval yields the direction angle change rate. The direction angle change rates of all monitoring cycles are arranged sequentially to form a trajectory curvature sequence.

[0120] The curvature sequence is continuously monitored. When the curvature value monotonically increases for five consecutive cycles and the increase exceeds a threshold (e.g., an increase of 0.15 radians per second per cycle), an inflection point is identified. The tangent vector of the trajectory is extracted at the inflection point. This tangent is determined by connecting the two coordinate points before and after the inflection point. The angles between the tangent vector and the horizontal and vertical axes are calculated. The coordinate axis corresponding to the smaller angle is taken as the dominant deviation axis. If the horizontal axis angle is 35 degrees and the vertical axis angle is 55 degrees, then the resource consumption rate is determined to be the dominant deviation factor. The dismantling intensity is determined by the curvature increment between two adjacent cycles at the inflection point. A curvature increment between 0.2 and 0.5 corresponds to light dismantling, 0.5 to 0.8 corresponds to moderate dismantling, and an increment exceeding 0.8 results in heavy dismantling.

[0121] Taking moderate decomposition as an example, if the resource quota unit currently bound to the task contains 4 GPU resource atoms, 8GB storage resource atoms, and 2Gbps bandwidth resource atoms, the decomposition operation breaks this unit down into the original set of resource atoms. Based on the deviation from the dominant axis as the resource consumption rate axis, priority is given to adjusting the quota of computing resource atoms. Considering that the tangent at the inflection point has a horizontal axis component of 0.7 and a vertical axis component of 0.3, the number of GPU resource atoms is increased to 6 with a weighting factor of 0.7, the storage resource atoms are fine-tuned to 9GB with a weighting factor of 0.3, and the bandwidth resource atoms remain at 2Gbps. The adjusted 6 GPU atoms, 9GB storage atoms, and 2Gbps bandwidth atoms are then recombined to form a new resource quota unit, replacing the original binding relationship of the training task. Simultaneously, the computing power index and remaining available time index of this unit are recalculated to ensure that the deviation calculation benchmark is updated accordingly in subsequent monitoring periods.

[0122] In one optional implementation, dismantling is jointly driven by the deviation from the dominant axis and the dismantling intensity. The dismantling quota for each resource subclass is determined based on the dismantling intensity, and differentiated adjustments are made by combining the trajectory tangent direction component at the inflection point. The resulting reorganized resource quota units are then allocated to the corresponding training tasks. This includes:

[0123] The axial allocation weight of each resource subclass is determined by the proportion of the absolute value of the component obtained by orthogonally projecting the deviation from the dominant axis to the reference axis of each resource subclass in the sum of the absolute values ​​of all resource subclass projections. The initial dismantling quota of each resource subclass is determined by the product of the axial allocation weight and the dismantling intensity.

[0124] The cosine of the angle between the trajectory tangent vector at the inflection point and the direction deviating from the dominant axis is taken as the tangent dominant coupling coefficient. The ratio of the absolute value of the orthogonal projection component of the tangent vector onto the reference axis of each resource subclass to the magnitude of the tangent vector is taken as the tangent component coefficient of each resource subclass. The product of the tangent dominant coupling coefficient and the tangent component coefficient is normalized to obtain the tangent adjustment weight of each resource subclass. The initial decomposition quota of each resource subclass is proportionally scaled by the tangent adjustment weight to obtain the adjusted quota of each resource subclass.

[0125] Using the dismantling intensity as the upper bound of the total, the difference between the sum of the adjusted quotas of all resource subclasses and the dismantling intensity is used to backfill the quotas according to the axial allocation weight ratio of each resource subclass. The backfilled quotas of each resource subclass are then encapsulated into new resource quota units and allocated to the corresponding training tasks.

[0126] In this specific embodiment, after determining the dismantling intensity, it is necessary to further clarify the specific dismantling quota for each resource subclass and calculate the orthogonal projection components of the deviation from the dominant axis onto the reference axis of each resource subclass. Let the vector of deviation from the dominant axis be denoted as... The reference axes for the computing resource subclass, storage resource subclass, and communication resource subclass are respectively denoted as... , and The absolute values ​​of each projection component are obtained through vector dot product operations, i.e. The calculation method for the remaining subclasses is the same: the absolute values ​​of the three projection components are added together to obtain the total projection. The ratio of the absolute value of each projection component to the total projection is the axial allocation weight of that resource subclass. Multiply the weights allocated to each axis by the dismantling intensity to obtain the initial dismantling quota for each resource subclass. For example, the initial resource dismantling quota can be calculated as follows:

[0127] .

[0128] Based on the initial quota, the trajectory tangent information at the inflection point is introduced for further refinement. The tangent vector of the resource consumption trajectory at the inflection point is extracted. Calculate the angle between the tangent vector and the deviation from the dominant axis, and obtain the tangent dominant coupling coefficient using the cosine function. This coefficient reflects the degree of consistency between the tangent direction and the deviation direction. The closer the value is to 1, the more similar they are. Simultaneously, the absolute values ​​of the orthogonal projection components of the tangent vector on the reference axis of each resource subclass are calculated, and each absolute value of the projection component is divided by the magnitude of the tangent vector. The tangent component coefficients of each resource subclass are obtained. The dominant coupling coefficient of the tangent is multiplied by the tangent component coefficients of each resource subclass, and the product result for all resource subclasses is normalized to ensure that the weights of each tangent are adjusted. The sum is 1. The initial allocation of the corresponding resource subclass is scaled proportionally by adjusting the weights of each tangent line to obtain the adjusted allocation:

[0129] .

[0130] Since the sum of the quotas for each resource subclass after tangent adjustment may not equal the dismantling intensity, it is necessary to normalize the total amount and calculate the sum of the adjusted quotas for all resource subclasses. With disassembly strength The residual quota is obtained by subtraction. The residual quota will be backfilled into each adjusted quota according to the axial allocation weight ratio of each resource subclass. To ensure that the sum of the quotas for each resource subclass is strictly equal to the dismantling intensity, the backfilled quotas for each resource subclass are encapsulated to form a new resource quota unit. The computing power, storage capacity, and communication bandwidth in this unit are reconfigured according to the above calculation results.

[0131] By binding new resource quota units to training tasks that trigger dismantling through the resource pool management interface, and reclaiming the resource amount corresponding to the dismantling intensity from the originally allocated resource quota units, dynamic reallocation of resources is achieved.

[0132] This invention provides an intelligent computing resource allocation system for multi-task parallel training, comprising:

[0133] An information acquisition unit is used to acquire resource request information for multiple tasks to be trained, including computational requirements, storage capacity requirements, and communication bandwidth requirements.

[0134] The resource abstraction unit is used to abstract computing resources into a resource pool composed of multiple resource atoms. The resource atoms are divisible and composable. Multiple resource quota units are formed by combining the resource atoms. Each resource quota unit has a computing power index and an available time index.

[0135] The matching degree building unit is used to establish a matching degree function between training tasks and resource quota units based on the resource request information and the computing power index, and to quantify the matching degree between the needs of each training task and the resource supply.

[0136] The allocation planning unit is used to construct a set of constraint equations for the resource competition relationship among multiple tasks in the resource pool based on the matching degree, solve the optimal resource quota unit allocation scheme for each training task under the competition constraint, form an initial resource binding relationship, and plan an execution time slice for each training task according to the available time index.

[0137] The dynamic adjustment unit is used to initiate multi-task parallel training, monitor the resource consumption rate and training progress speed of each training task in real time, and when it is detected that the resource consumption rate and training progress speed of each training task deviate non-linearly, the corresponding resource quota unit is disassembled and a new resource quota unit is reorganized and allocated to the corresponding training task.

[0138] A third aspect of the present invention provides an electronic device, comprising:

[0139] processor;

[0140] Memory used to store processor-executable instructions;

[0141] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0142] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0143] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent allocation of computing resources for multi-task parallel training, characterized in that, include: Obtain resource request information for multiple tasks to be trained, including computational requirements, storage capacity requirements, and communication bandwidth requirements; Computing resources are abstracted into a resource pool composed of multiple resource atoms. These resource atoms are divisible and composable. Multiple resource quota units are formed by combining these resource atoms. Each resource quota unit has a computing power index and an available duration index. Based on the resource request information and the computing power index, a matching degree function between training tasks and resource quota units is established to quantify the matching degree between the demand of each training task and the supply of resources. Based on the matching degree, a set of constraint equations is constructed for the resource competition relationship among multiple tasks in the resource pool. The optimal resource quota unit allocation scheme of each training task under the competition constraint is solved to form an initial resource binding relationship. Execution time slices are planned for each training task according to the available time index. Initiate multi-task parallel training, monitor the resource consumption rate and training progress speed of each training task in real time, and when a non-linear deviation is detected in the resource consumption rate and training progress speed of each training task, disassemble the corresponding resource quota unit and reorganize the new resource quota unit and allocate it to the corresponding training task.

2. The method according to claim 1, characterized in that, Computing resources are abstracted as a resource pool composed of multiple resource atoms. These resource atoms are divisible and composable. Multiple resource quota units are formed by combining these resource atoms. Each resource quota unit has computing power and available time indicators, including: The computing resources are hierarchically decomposed into computing resource atoms, storage resource atoms, and communication resource atoms according to their functional attributes, and the resource capacity attribute is labeled for each type of resource atom. Establish combination constraint rules for resource atoms. The combination constraint rules define compatibility conditions and combination ratio boundaries between different types of resource atoms. The compatibility conditions determine whether resource atoms can jointly form a resource quota unit. The combination ratio boundaries limit the quantity ratio of each type of resource atoms within the same resource quota unit. Based on the combination constraint rules, select multiple resource atoms that meet the compatibility conditions and aggregate them to form a resource quota unit. For each resource quota unit, based on the resource capacity attributes of the resource atoms participating in the combination and the combination ratio boundary, fusion weights are assigned to the resource capacity attributes of different types of resource atoms, and the computing capability index is calculated by weighted summation of the fusion weights. Based on the expected allocation cycle and release cycle of the resource quota unit, the available duration index is determined, and the resource quota unit and the corresponding computing power index and available duration index are stored in the resource pool.

3. The method according to claim 1, characterized in that, Based on the resource request information and the computing power index, a matching function is established between training tasks and resource quota units to quantify the matching degree between the demand for each training task and the supply of resources, including: The computational requirements, storage capacity requirements, and communication bandwidth requirements in the resource request information are normalized to obtain standardized requirement values ​​for each requirement dimension. Using the intensity of the obstruction to the progress of training tasks due to insufficient resources in each demand dimension as matrix elements, a demand sensitivity matrix is ​​constructed. Based on the demand sensitivity matrix, the standardized demand values ​​of each demand dimension are weighted and mapped to obtain the sensitivity weighted demand vector of each training task. The computing power index is decomposed into supply components of each type according to resource type. The deviation between the sensitivity-weighted demand vector and each supply component is calculated dimension by dimension. The maximum value of the deviation of each dimension is the bottleneck constraint term, and the dispersion of the deviation of each dimension is the equilibrium constraint term. A matching degree function is constructed based on the composite relationship between the bottleneck constraint term and the equilibrium constraint term, so that the matching degree value is negatively correlated with both the bottleneck constraint term and the equilibrium constraint term. Substitute the sensitivity-weighted demand vector of each training task and the computational capability index of each resource quota unit into the matching degree function to output the matching degree between each training task and each resource quota unit.

4. The method according to claim 1, characterized in that, Based on the matching degree, a set of constraint equations is constructed to address the resource competition relationship among multiple tasks in the resource pool. The optimal resource quota unit allocation scheme for each training task under the competition constraint is solved to form an initial resource binding relationship. Furthermore, execution time slices are planned for each training task according to the available time index, including: The matching degree between each training task and each resource quota unit is used to form a matching degree feature vector. The cosine similarity between any two matching degree feature vectors is calculated as the competition correlation coefficient, and a competition correlation matrix between all tasks is constructed. Using the inter-task competition correlation matrix as the quadratic coefficient matrix, the total amount of each resource quota unit as the linear capacity constraint, and the minimum requirement of each training task as the lower bound constraint of the requirement, a set of constraint equations is constructed with the goal of minimizing the weighted penalty of competition correlation. The constraint equations are solved using the interior point obstacle function method. The linear capacity constraint and the lower bound of demand constraint are internalized into the objective function using logarithmic obstacle terms. The resource quota units and dual variables of each training task are iteratively updated until the KKT optimal conditions are met, and the optimal resource quota unit allocation scheme is output. According to the optimal resource quota unit allocation scheme, each training task is bound to the corresponding resource quota unit to form an initial resource binding relationship; based on the available duration index and the throughput capacity of the resource quota unit bound to each training task, the execution duration of each training task is divided into ordered time slices according to the load balancing principle to obtain execution time slices.

5. The method according to claim 4, characterized in that, The constraint equations are solved using the interior-point barrier function method. Logarithmic barrier terms are used to internalize the linear capacity constraint and the lower bound of demand constraint into the objective function. The resource quota units and dual variables for each training task are iteratively updated until the KKT optimality condition is met. The optimal resource quota unit allocation scheme is then output, including: Extract the linear capacity constraints and demand lower bound constraints from the constraint equation set, construct a logarithmic barrier function for each inequality constraint, initialize the barrier parameters, and add the product of the logarithmic barrier function and the barrier parameters to the competition-related weighted penalty objective function to form an augmented objective function; Set the initial values ​​of the quota allocation for each training task and the dual variable, calculate the gradient vector and Hessian matrix of the augmented objective function, perform diagonal dominance preprocessing on the Hessian matrix, improve the condition number through diagonal scaling transformation, and construct the preconditioned Newton equation. The preconditional Newton equation is solved using the conjugate gradient iteration method. The process terminates when the preset number of inner layer iterations is reached or the residual decreases to the inner layer convergence threshold. An approximate Newton direction is output. Based on the approximate Newton direction, a line search algorithm is used to calculate the step size parameter that satisfies the sufficient descent condition. The resource quota unit and dual variable are updated synchronously along the approximate Newton direction and the step size parameter. The centrality deviation index is quantified by the degree of violation of the complementary relaxation conditions. The decay rate of the barrier parameters is adaptively adjusted according to the centrality deviation index. The KKT conditional residual is calculated. If it is less than the outer convergence threshold, the current resource quota unit is output as the optimal resource quota unit allocation scheme. Otherwise, the barrier parameters are updated and the augmented objective function formation step is returned.

6. The method according to claim 1, characterized in that, When a non-linear deviation is detected between the resource consumption rate and the training progress speed of each training task, the corresponding resource quota unit is disassembled and a new resource quota unit is reassembled and allocated to the corresponding training task, including: After the multi-task parallel training is started, the resource consumption rate and training progress speed of each training task are collected synchronously. Using the deviation of the resource consumption rate and the deviation of the training progress speed as the two axes, a two-dimensional deviation trajectory of each training task is constructed in each monitoring cycle. The trajectory curvature sequence is constructed using the change rate of the direction angle of adjacent segments of the two-dimensional deviation trajectory. When the trajectory curvature sequence of each training task shows a continuously monotonically increasing inflection point, it is determined that the resource consumption rate deviates non-linearly from the training progress speed. The angle between the trajectory tangent direction at the inflection point and the dual axes is extracted, and the axis corresponding to the smallest angle is taken as the dominant deviation axis. The dismantling intensity is determined by the curvature increment at the inflection point. The dismantling is driven by the deviation from the dominant axis and the dismantling intensity. The dismantling quota of each resource subclass is determined based on the dismantling intensity. The quota is then adjusted differentially by combining the tangent direction component of the trajectory at the inflection point, and reorganized into a new resource quota unit to be allocated to the corresponding training task.

7. The method according to claim 6, characterized in that, The dismantling is driven by the deviation from the dominant axis and the dismantling intensity. The dismantling quota for each resource subclass is determined based on the dismantling intensity. Differential adjustments are made based on the trajectory tangent direction component at the inflection point, and the resulting units are reorganized into new resource quota units and allocated to the corresponding training tasks. This includes: The axial allocation weight of each resource subclass is determined by the proportion of the absolute value of the component obtained by orthogonally projecting the deviation from the dominant axis to the reference axis of each resource subclass in the sum of the absolute values ​​of all resource subclass projections. The initial dismantling quota of each resource subclass is determined by the product of the axial allocation weight and the dismantling intensity. The cosine of the angle between the trajectory tangent vector at the inflection point and the direction deviating from the dominant axis is taken as the tangent dominant coupling coefficient. The ratio of the absolute value of the orthogonal projection component of the tangent vector onto the reference axis of each resource subclass to the magnitude of the tangent vector is taken as the tangent component coefficient of each resource subclass. The product of the tangent dominant coupling coefficient and the tangent component coefficient is normalized to obtain the tangent adjustment weight of each resource subclass. The initial decomposition quota of each resource subclass is proportionally scaled by the tangent adjustment weight to obtain the adjusted quota of each resource subclass. Using the dismantling intensity as the upper bound of the total, the difference between the sum of the adjusted quotas of all resource subclasses and the dismantling intensity is used to backfill the quotas according to the axial allocation weight ratio of each resource subclass. The backfilled quotas of each resource subclass are then encapsulated into new resource quota units and allocated to the corresponding training tasks.

8. A computational resource intelligent allocation system for multi-task parallel training, used to implement the method as described in any one of claims 1-7, characterized in that, include: An information acquisition unit is used to acquire resource request information for multiple tasks to be trained, including computational requirements, storage capacity requirements, and communication bandwidth requirements. The resource abstraction unit is used to abstract computing resources into a resource pool composed of multiple resource atoms. The resource atoms are divisible and composable. Multiple resource quota units are formed by combining the resource atoms. Each resource quota unit has a computing power index and an available time index. The matching degree building unit is used to establish a matching degree function between training tasks and resource quota units based on the resource request information and the computing power index, and to quantify the matching degree between the needs of each training task and the resource supply. The allocation planning unit is used to construct a set of constraint equations for the resource competition relationship among multiple tasks in the resource pool based on the matching degree, solve the optimal resource quota unit allocation scheme for each training task under the competition constraint, form an initial resource binding relationship, and plan an execution time slice for each training task according to the available time index. The dynamic adjustment unit is used to initiate multi-task parallel training, monitor the resource consumption rate and training progress speed of each training task in real time, and when it is detected that the resource consumption rate and training progress speed of each training task deviate non-linearly, the corresponding resource quota unit is disassembled and a new resource quota unit is reorganized and allocated to the corresponding training task.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images