Intelligent computing resource cooperative allocation method for storage-compute separation architecture of multi-storage cluster

Abstract

The application discloses a method for collaborative allocation of intelligent computing resources for a storage-computing separation architecture of a multi-storage cluster. In each scheduling time slot, a collector obtains the current state of a running task on each GPU server, calculates a cache allocation and a bandwidth allocation scheme for each task with the goal of maximizing the total scheduling welfare, and sends an upgrade or downgrade instruction to a cloud service provider through a bandwidth service to dynamically adjust the transmission bandwidth. The allocation result is simultaneously sent to a cache server to reserve cache space for tasks sharing a data set. The application can eliminate I / O blocking in a multi-storage cluster environment and maximize the total scheduling welfare within a budget constraint.

Description

Technical Field

[0001] This invention belongs to the fields of cloud computing, distributed systems, artificial intelligence infrastructure and resource scheduling technology, specifically a method for collaborative allocation of intelligent computing resources for multi-storage cluster storage-compute separation architecture. Background Technology

[0002] With the continuous expansion of deep learning model scale and the explosive growth of training datasets, cloud computing platforms have become the mainstream infrastructure for deep learning training. Currently, most mainstream cloud service providers adopt a compute-storage separation architecture, where compute clusters and storage clusters are physically isolated and deployed independently. After a training task is submitted, compute nodes need to pull training data from a remote storage cluster in real time via the network to iterate model parameters. This architecture can significantly improve the elastic scalability and operational efficiency of the resource pool. However, as GPU computing power continues to increase, the growth rate of network bandwidth lags far behind the increase in computing power, leading to increasingly serious I / O blocking problems: GPUs idle while waiting for data loading, not only prolonging task completion time but also causing expensive waste of computing resources.

[0003] To address the above problems, existing technologies have proposed resource scheduling and optimization schemes such as computing resource-centric scheduling algorithms, distributed caching systems, and I / O blocking mitigation techniques to solve the I / O blocking problem. However, the following issues still exist: I. The problem of increased computational efficiency caused by fragmented resource dimensions. In existing technologies, the GPU resource scheduler and storage resource manager operate independently. When the system increases GPU quota for a task to accelerate training, its data read requirements increase exponentially. However, due to the lack of cross-resource collaboration, bandwidth and cache often remain unchanged, causing the newly added GPUs to idle for most of the time while waiting for data. The actual training throughput not only fails to improve but may even decrease due to resource contention.

[0004] II. Joint Optimization of Cross-Cluster Data Scheduling and Bandwidth Allocation in Multi-Storage Cluster Scenarios. In real-world production environments, large training datasets are typically redundantly stored across multiple geographically dispersed storage clusters to ensure data disaster recovery and access performance. Each cluster has different bandwidth capacity limits, different bandwidth pricing functions, and the data read ratio for each task across different clusters can be configured independently. Existing scheduling methods either assume data is stored in a single cluster or employ fixed data read strategies, failing to flexibly adjust the read ratio based on dynamic bandwidth pricing and load conditions.

[0005] III. High-precision approximation solutions to NP-hard resource allocation problems. While traditional heuristic algorithms are computationally fast, the quality of the solutions cannot be guaranteed, potentially leading to deviations from the optimal solution and resulting in cost overruns or performance degradation. Furthermore, exact solution methods suffer from combinatorial explosion as the number of tasks and storage clusters increases, rendering them unsuitable for online scheduling.

[0006] IV. The Problem of Maximizing Scheduling Welfare under Heterogeneous Utility Functions. In cloud platforms, different deep learning tasks exhibit significantly different sensitivities to training completion time. For example, real-time fault detection tasks require training to be completed within milliseconds; exceeding the deadline results in zero utility. Offline model tuning tasks, on the other hand, allow for longer training times, with utility decaying linearly with time delay. Furthermore, there are sigmoid-type utility functions that fall between these two extremes. Most existing resource schedulers aim to minimize the average job completion time (JCT) or maximize system throughput, and cannot directly handle user-defined heterogeneous utility functions. Summary of the Invention

[0007] The purpose of this invention is to provide a method for collaborative allocation of intelligent computing resources for a multi-storage cluster compute-in-memory separation architecture. By establishing a joint optimization model of the number of GPUs, cache capacity, bandwidth allocation, and cross-cluster data read ratio, and combining techniques such as polynomial approximation, sparse semidefinite programming relaxation, and multi-stage feasibility repair, a high-quality approximate optimal solution is obtained within an acceptable computational complexity. This addresses the problem, as mentioned in the background art, of how to jointly optimize the number of GPUs, cache capacity, bandwidth allocation of each storage cluster, and data read ratio for each deep learning task in a real compute-in-memory separation environment with multiple storage clusters and dynamically scalable computing resources.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: According to the above technical solution, the method for collaborative allocation of intelligent computing resources for multi-storage cluster storage-compute separation architecture includes the following steps: At the beginning of each time slot, the system status is acquired; the system status includes the status of all tasks to be trained and tasks in operation, the available bandwidth and cache capacity of each storage cluster, the total cost budget, and resource pricing parameters; Establish the objective function and its constraints for multi-resource joint optimization; The objective function and constraints of multi-resource joint optimization are transformed into polynomial form using the polynomial approximation method. Auxiliary variables are then introduced to optimize the objective function and constraints of multi-resource joint optimization in polynomial form, resulting in an objective function and constraints with auxiliary limits. A variable coupling graph is constructed based on the objective function with auxiliary constraints and its constraints, and sparse clique decomposition is performed. Based on sparse cluster decomposition, a sparse semidefinite programming (SDP) relaxation is constructed using the Lasserre relaxation hierarchy to obtain the sparse SDP relaxation problem. Solving sparse SDP relaxation problems using an interior-point method solver; Extract variable estimates from the SDP solution, repair them, and output the optimal resource allocation scheme. The cache manager and bandwidth controller schedule resources according to the optimal resource allocation scheme issued. At the end of each scheduling slot, the system re-collects the latest system state and enters the scheduling cycle for the next slot.

[0009] This invention incorporates the number of GPUs, cache capacity, bandwidth allocation, and data read ratio into a unified mathematical model, and enforces a constraint that the actual data throughput does not exceed the ideal throughput determined by the number of GPUs. This ensures that each GPU expansion is matched with corresponding bandwidth or cache resources. This allows the marginal investment of GPU resources to truly translate into training acceleration, avoiding wasted computing power. The I / O blocking rate remains zero throughout the entire training cycle, and GPU idling is completely eliminated. Simultaneously, the data read ratio... As a decision variable, it is related to bandwidth allocation. Collaborative optimization enables the system to automatically direct data traffic to clusters with lower bandwidth costs and greater capacity. While meeting end-to-end latency requirements, data transmission costs are significantly reduced, and the load across clusters is more balanced, avoiding I / O blocking caused by bandwidth overload of a single cluster.

[0010] According to the above technical solution, the objective function of the multi-resource joint optimization is to maximize the total scheduling welfare of all tasks under the total budget constraint: ; in, ; The constraints of the objective function for the multi-resource joint optimization are as follows: ; In the formula, This represents a user-defined utility function. For the task Completion time, Indicates task Total resource cost This indicates the unit price of the GPU. Indicates assignment to task The number of GPUs, Indicates the unit price of resource caching. Indicates task The allocated data cache capacity, Indicates task Number of training rounds Indicates task The number of datasets, Indicates task From storage cluster The proportion of data read Depend on Exported helper functions, defined This indicates that bandwidth cost increases linearly with bandwidth. In polynomial fitting, its numerical value is used directly. This indicates the total budget for the task. Indicates the cache limit. Represents a cluster The upper limit of bandwidth capacity, Indicates the first auxiliary variable. Indicates task From storage cluster The obtained network bandwidth Indicates task Data set location indicator matrix, This indicates the second auxiliary variable. Indicates task In Adoption Ideal throughput during GPU training Indicates in a given cache and bandwidth Training throughput under certain conditions.

[0011] According to the above technical solution, the non-polynomial function of the objective function of the multi-resource joint optimization is obtained by using the polynomial approximation method. and non-polynomial functions in the constraints Transforming it into polynomial form specifically includes: For each task Measure the number of different GPUs through offline experiments The ideal throughput is obtained to get the set of data points. ;Task In Adoption Ideal throughput during GPU training Fit a using the least squares method polynomial of order : , The fitting coefficients obtained by the least squares method or the minimax method make... Minimize, this approximation method makes the obtained polynomial more uniformly close to the true function in the global scope.

[0012] Using a polynomial To approximate, The fitting coefficients are used to make the polynomial complete in the task time. The possible range of values ​​should be as close as possible to the true utility function.

[0013] In the formula, express The fitting coefficient, yes The fitting coefficient, Indicates task The completion time D raised to the power of k, where k represents the order index of the polynomial fitting. Indicates task The number of GPUs raised to the power of k.

[0014] The invention approximates any user-defined utility function using a polynomial and maximizes total scheduling welfare (utility minus cost) under budget constraints. This allows cloud platforms to simultaneously serve tenants with multiple SLA requirements, and scheduling strategies can automatically allocate resources to high-value, time-sensitive tasks, thereby improving overall revenue and user satisfaction.

[0015] According to the above technical solution, in order to transform the division in the completion time expression into a polynomial constraint, the task... Completion Time By introducing the first auxiliary variable Second auxiliary variable Sub-completion time of task i in storage cluster s Perform auxiliary variable substitution. After auxiliary variable substitution, it is represented as .

[0016] In a specific embodiment, it is assumed that the optimization objective is to minimize the total completion time of all tasks, i.e. ,in Variables include binary assignment variables. (Indicates task) Is it assigned to a resource? ), continuous auxiliary variables Introduce polynomial equality constraints: Division Convert to product form; and , equivalent to ,Right now Each task must be allocated exactly one resource: .resource The total load on it does not exceed its capacity (Assume each task) The load is ): This completely transforms the original fractional programming problem into a mixed-integer nonlinear programming problem (MINLP) under polynomial constraints, making it easier to solve using polynomial optimization or convex relaxation methods.

[0017] At the same time, Completion time on each storage cluster The relationship is transformed from a maximum value operation into a set of inequality constraints. .

[0018] According to the above technical solution, each variable in the objective function containing auxiliary constraints is used as a vertex of the variable coupling graph. If two variables in the objective function containing auxiliary constraints appear simultaneously in the polynomial of the same constraint condition, then an undirected edge is connected between them.

[0019] Sparse clique decomposition details: Based on the problem structure, three types of naturally formed cliques (complete subgraphs) can be identified in the variable coupling graph. The first type is the task clique, which occurs for each task. its internal variable set Any two variables are coupled to each other through training time constraints or cost constraints, forming a system of size The first type is the cache-coupled cluster. The second type is the cache-coupled cluster. All cached variables are subject to total quantity constraints. Interconnected. The third type is bandwidth coupling clusters, which exist for each storage cluster. ,gather By limiting the total bandwidth of the cluster To form a group.

[0020] Therefore, the overall cluster decomposition result is This decomposition transforms the originally globally dense variable relationships into a locally sparse clique structure, with each clique being much smaller than the total number of global variables.

[0021] By leveraging the natural sparse coupling structure between variables (cached variables are only connected to each other through total constraints, and bandwidth variables are only connected to each other within the same cluster), the global problem is decomposed into task clusters, cache clusters, and bandwidth clusters, with each cluster being much smaller than the total number of global variables.

[0022] According to the above technical solution, the sparse SDP relaxation problem The construction steps include: For a given relaxation order ( (The integer is usually 2 or 3, which controls the relaxation precision), for each sparse clique obtained from the decomposition. (A clique is a complete subgraph in a variable coupling graph and belongs to the clique decomposition set) Introduce a set of local moment variables ,in For multiple indicators , Corresponding to monomial Moments under a certain probability measure; constructing a moment matrix based on these moment variables. Its rows and columns are formed by a number of times not exceeding 100. A single-item index, where the element is defined as And apply a positive semidefinite constraint. ; Construct a localized moment matrix for the constraints of the objective function containing auxiliary limits and apply positive semidefinite constraints; For any two sparse clusters and Let their intersection be (That is, a set of variables that simultaneously belong to two groups). Take the intersection. Any of the multiple indicators Its order (That is, the sum of all components) does not exceed in (This is the preset relaxation order). Therefore, sparse clusters are required. and sparse clusters The local moment variables satisfy ,in They are respectively to The result obtained after expansion (padding variables that do not belong to the intersection with zeros) is similar to the sparse clique. and sparse clusters The variables correspond to complete multiple indices. This equation is called the consistency constraint, which ensures that the moment descriptions of shared variables by different cliques are consistent, thereby ensuring that all local moment variables correspond to the same global probability distribution. It is a key step in constructing sparse semidefinite programming relaxation.

[0023] For a set of shared variables, for The above satisfies Multiple indicators and They are respectively to The complete multiple index obtained by padding the non-intersection variables of the two groups with zeros, and the corresponding local moment variables and They must be equal to ensure that different groups have consistent moment descriptions of shared variables. It is a multi-indicator vector, and Similarly, columns used to index a moment matrix have components that correspond to cliques. The exponents of each variable in the equation; This represents the expected value of the monomial corresponding to the local moment variable under a certain probability measure. In the definition of the moment matrix, the row index... and column indicators The corresponding matrix element values ​​are , which is the moment obtained by multiplying two monomials.

[0024] This invention significantly reduces overall computational complexity by independently constructing a moment matrix for each clique and adding consistency constraints. It is also scalable to large-scale task and storage cluster scenarios, making it suitable for practical deployments in large public clouds and supercomputing centers.

[0025] This invention employs polynomial optimization and uses Lasserre relaxation and sparse semidefinite programming for solution. The order can be approximated as the polynomial increases. and relaxation order As the complexity increases, the ratio between the algorithm's output solution and the global optimum asymptotically approaches 1, demonstrating asymptotic optimality. Simultaneously, the algorithm's complexity grows polynomially with the task size, allowing for solution completion within the actual scheduling cycle. This provides cloud platform operators with clear lower bounds on solution quality and optimality gaps, building confidence in the reliability of scheduling decisions. Furthermore, the solution time meets online scheduling requirements, balancing theoretical optimality with engineering feasibility.

[0026] Based on the above technical solution, for each mission team Mission Group It is a collection of sparse clusters A subset, namely From the moment matrix The first moment is extracted as the estimate of each variable.

[0027] Specifically, let For variables The corresponding indicator vector (within this clique) is then estimated to be... For variables that appear in multiple groups simultaneously (such as...) The values ​​can be taken from any clique, as the consistency constraint guarantees their equality. Finally, estimates of all original variables are obtained. , , , .

[0028] Each decision variable can be explicitly extracted from the moment variables of the SDP relaxation, and the bottlenecks of the current system can be determined by analyzing the Lagrange multipliers of the constraints. Operations personnel can intuitively understand the physical meaning of scheduling decisions and quickly diagnose problems. Furthermore, because the algorithm has a definite convergence and optimality gap, performance fluctuations can be quantitatively evaluated rather than attributed to random factors.

[0029] According to the above technical solution, the variable estimate repair includes data scheduling ratio repair, bandwidth capacity repair, cache capacity repair, GPU number integer repair, budget constraint repair, and consistency update; Data scheduling ratio repair: based on data location matrix The proportion of unusable clusters in the variable estimates is forced to zero, i.e., if Then let Then for each task Normalize the remaining proportions ;in, This represents the variable estimates extracted from the SDP relaxation solution. This represents an estimated proportion of data that task i will read from storage cluster s after the repair. Bandwidth capacity restoration: for each storage cluster Check if the total bandwidth allocation exceeds the upper limit; if Then compress proportionally. ;in, This represents the task extracted from the SDP relaxed solution. From storage cluster The obtained network bandwidth Indicates the task after repair. From storage cluster The network bandwidth obtained; Cache capacity repair: for each task Check if the total cache capacity exceeds the cache limit; if Then proportional compression will be performed. ;in, This represents the task extracted from the SDP relaxed solution. The allocated data cache capacity, Indicates the task after repair The allocated network bandwidth; GPU Count Integer Fix: The calculated GPU count is not an integer but a value on a continuous function. Therefore, the continuous estimate is rounded to ensure it is at least 1. ; This refers to the number of GPUs after the repair; This represents the task extracted from the SDP relaxed solution. The number of GPUs; Budget constraint repair: Based on the repaired , , , Recalculate total cost ;like Then scale the adjustable resource cache and bandwidth of all tasks proportionally. ;in, Indicates the task after repair Total resource cost; After scaling, the completion time of each task after repair is recalculated. And check again whether the delay threshold constraint is met; if a certain If the threshold is exceeded, the bandwidth for that task will be locally adjusted and increased, while resources for other less sensitive tasks will be reduced accordingly. Consistent Update: Recalculate tasks based on the final fixed resource allocation. Completion time .

[0030] This invention reduces the number of GPUs during the relaxation phase. Treating data as a continuous variable in the SDP solution, a series of repair operations, including rounding, scaling, and budget recalibration, are used to re-obtain an integer feasible solution. This repair process sequentially handles data scaling, bandwidth capacity, cache capacity, GPU integers, and budget constraints, ensuring that each step does not violate constraints satisfied in the previous step. This allows the final resource allocation scheme to be directly deployed on a physical cluster without manual intervention; simultaneously, because the loss from the repair steps is controlled within a provable range, the approximation ratio still maintains theoretical asymptotic optimality.

[0031] The invention includes a technical solution, an electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the intelligent computing resource collaborative allocation method for a multi-storage cluster storage-compute separation architecture as described in any of the above technical solutions.

[0032] The invention includes a technical solution and a storage medium storing at least one instruction, which is loaded and executed by a processor to implement a method for collaborative allocation of intelligent computing resources for a multi-storage cluster storage-compute separation architecture, as described in any of the above technical solutions.

[0033] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention establishes a joint optimization model of GPU quantity, cache capacity, bandwidth allocation, and cross-cluster data read ratio. It combines techniques such as polynomial approximation, sparse semidefinite programming relaxation, and multi-stage feasibility repair to optimize resource allocation. This allows operations personnel to estimate the solution time and solution quality of the algorithm based on task scale and resource budget before deployment, thereby enabling capacity planning and SLA design. This invention eliminates I / O blocking, improves GPU utilization, and solves the GPU idling problem caused by data transmission bottlenecks in compute-in-memory architectures, ensuring full utilization of expensive GPU computing resources. The cross-cluster data ratio optimization of this invention can guide traffic from congested clusters to idle clusters, thereby alleviating queuing latency and shortening average job completion time.

[0034] Meanwhile, under the same budget constraints and task load, the total scheduling welfare of the method of this invention (i.e., the utility function value of all tasks minus the sum of resource costs) is maximized in terms of economic benefits, bringing higher return on investment to cloud service providers and tenants. Furthermore, in multi-storage cluster scenarios, this invention can significantly reduce the total cost of the storage layer. This cost saving comes from two aspects: first, reducing reliance on expensive bandwidth through cache reuse; and second, directing data traffic to lower-priced storage clusters through cross-cluster optimization. For large-scale cloud platforms, this cost saving directly translates into increased profits and price competitiveness.

[0035] The technical solution of this invention does not rely on specific hardware, nor does it require deep modification of deep learning frameworks. As an independent resource scheduling module, it can interface with cache managers and bandwidth controllers through standard APIs. Furthermore, its thorough consideration of practical constraints such as fixed service entities, cache activation delays, and heterogeneous utility functions allows it to be directly adapted to the existing architectures of mainstream public and private cloud platforms, making it highly valuable for engineering implementation. Attached Figure Description

[0036] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the steps of the intelligent computing resource collaborative allocation method for a multi-storage cluster storage-compute separation architecture according to the present invention; Figure 2 This is a structural deployment diagram of the intelligent computing resource collaborative allocation method for a multi-storage cluster storage-compute separation architecture, as shown in the embodiment. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] This invention provides a technical solution: a method for collaborative allocation of intelligent computing resources for multi-storage cluster storage-compute separation architecture. Figure 1 The application is used in systems whose deployment architecture includes multiple GPU servers and cache servers, as well as collectors, allocators, bandwidth services, and cloud service providers. Figure 2 Resource scheduling is performed in the following steps: S1. At the beginning of each time slot, obtain the status of all tasks to be trained and running tasks, the available bandwidth and cache capacity of each storage cluster, the total resource cost, and resource pricing parameters.

[0039] The storage cluster set is denoted as Each cluster With bandwidth capacity limit and bandwidth cost function Deep learning task set record For each task The size of its dataset is known. Number of training rounds Dataset location indicator matrix (A value of 1 indicates that the dataset is stored in the cluster) (or 0 otherwise), and user-defined utility functions. ,in This refers to the task completion time. Resource pricing parameters include the cache unit price. (RMB / GB / hour), GPU unit price (RMB / GPU / hour) and bandwidth pricing functions for each storage cluster The total system cost budget is .

[0040] Decision variables fall into four categories. First, Indicates assignment to task The number of GPUs, a value that can be dynamically adjusted between upper and lower limits. Second, Represented as a task Allocated data cache capacity (unit: GB). Third, Indicates task From storage cluster The obtained network bandwidth (unit: MB / s). Fourth, Indicates task From storage cluster The proportion of data read meets the requirements. And only if hour It can be non-zero.

[0041] Task The end-to-end completion time is determined by the slowest storage cluster, specifically... , where the denominator Indicates in a given cache and bandwidth The actual achievable training throughput under given conditions. This throughput cannot exceed the ideal throughput determined by the number of GPUs. That is, requirements .Task Total resource cost It consists of three parts: GPU computing cost, cache storage cost, and data transfer cost. Its expression is: ,in It is determined by the bandwidth cost function The exported helper function specifically satisfies This form is derived from the sublinear decomposition of the bandwidth pricing model.

[0042] S2. Establish the objective function and its constraints for multi-resource joint optimization.

[0043] Objective function for multi-resource joint optimization: ; in, ; Constraints of the objective function for multi-resource joint optimization: ; In the formula, This represents a user-defined utility function; For the task Completion time; Indicates task Total resource cost; This indicates the unit price of the GPU; Indicates assignment to task The number of GPUs; Indicates the unit price of resource caching; Indicates task The allocated data cache capacity; Indicates task The number of training rounds; Indicates task The number of datasets; Indicates task From storage cluster The proportion of data read; Indicates by The exported helper function is defined as follows: ; This represents a function that shows the cost of bandwidth increasing linearly with bandwidth. In polynomial fitting, its numerical value is used directly; Indicates the total budget for the task; Indicates the cache limit; Represents a cluster The maximum bandwidth capacity; Indicates the first auxiliary variable; Indicates task From storage cluster Obtained network bandwidth; Indicates task Data set location indicator matrix, This indicates the second auxiliary variable. Indicates that task i is using Ideal throughput during GPU training Indicates in a given cache and bandwidth Training throughput under certain conditions.

[0044] S3. Using the polynomial approximation method, the non-polynomial function of the multi-resource joint optimization objective function is transformed into a polynomial function. and non-polynomial functions in the constraints Transforming it into polynomial form specifically includes: Fit a using the least squares method polynomial of order For each task Measure the number of different GPUs through offline experiments The ideal throughput is obtained to get the set of data points. . Fit a using the least squares method polynomial of order , The fitting coefficients are obtained through the least squares method or the minimax method, such that... Minimize, this approximation method makes the obtained polynomial more uniformly close to the true function in the global scope; For user-defined utility functions Transform into a polynomial To approximate, The fitting coefficients are used to make the polynomial complete in the task time. The possible range of values ​​should be as close as possible to the true utility function.

[0045] And, the task Completion Time By introducing the first auxiliary variable Second auxiliary variable For task i in the storage cluster Sub-completion time Perform auxiliary variable substitution. After auxiliary variable substitution, it is represented as: We obtain the objective function containing auxiliary limits and its constraints.

[0046] S4. Each variable in the objective function containing auxiliary constraints is used as a vertex of the variable coupling graph. If two variables in the objective function containing auxiliary constraints appear simultaneously in the polynomial of the same constraint, an undirected edge is connected between them to construct the variable coupling graph.

[0047] Sparse clique decomposition is performed based on the variable coupling graph to identify task cliques, cache cliques, and bandwidth cliques.

[0048] The first type is the mission team, which is responsible for each mission. its internal variable set Any two variables are coupled to each other through training time constraints or cost constraints, forming a system of size The first type is the cache-coupled cluster. The second type is the cache-coupled cluster. All cached variables are subject to total quantity constraints. Interconnected. The third type is bandwidth coupling clusters, which exist for each storage cluster. ,gather By limiting the total bandwidth of the cluster To form a group.

[0049] Therefore, the overall cluster decomposition result is This decomposition transforms the originally globally dense variable relationships into a locally sparse clique structure, with each clique being much smaller than the total number of global variables.

[0050] S5. Based on sparse clique decomposition, a sparse semidefinite programming (SDP) relaxation is constructed using the Lasserre relaxation hierarchy, resulting in the sparse SDP relaxation problem, specifically: When constructing a sparse semidefinite programming (SDP) relaxation, the relaxation order is first selected. (Usually 2 or 3), then for each sparse cluster (Including task clumps, cache clumps, and bandwidth clumps) Introduce a set of local moment variables ,in It is a multiple indicator (whose components represent the exponents of the variables in the cluster), (Total number of times) Corresponding to monomial Integral over a certain probability measure; then construct the moment matrix. Its rows and columns are formed by a number of times not exceeding 100. A single-item index, where the element is defined as and demand (Semi-positive definite); for each inequality constraint Constructing localized moment matrices , corresponding to the inequality The localized moment matrix, where, It is a pre-selected relaxation order (usually 2 or 3). Representing a polynomial The number of times (the total number of times the highest-order term). yes Round up. group The set of local moment variables (each local moment variable) Corresponding to monomial (The integral under a certain probability measure). The localized moment matrix is ​​constructed by multiplying the monomials in the ordinary moment matrix by... and the relaxation order from Reduce to At the same time, the matrix is ​​required to be positive semi-definite (i.e., all eigenvalues ​​are non-negative).

[0051] Since different groups share variables, consistency constraints must be added to ensure that the moment variables of different groups correspond to the same global probability distribution. Specifically, for any two groups... Let be the set of all groups, and let their intersection be... (i.e., a set of variables that simultaneously belong to two groups). For any multiple indices on the intersection... It is a vector whose components represent the exponents of the variables in the intersection. The sum of all components (i.e., the total degree of the corresponding monomial) is required. and satisfy .in, They are respectively to Expand (padding variables that do not belong to the intersection) to the clique and The complete multiple indicators on the surface and These are the local moment variables in the corresponding clique.

[0052] Finally, add normalization constraints. ,in An arbitrary reference group (usually the first mission group) is selected. ), This represents the local moment variable in the clique corresponding to the zero multiple index (i.e., the constant term), whose value is fixed at 1 to ensure that the total mass of the probability measure is 1.

[0053] S6. The sparse SDP relaxation problem is input into the interior-point solver for solution. The solver returns the optimal value. This value is a lower bound of the optimal value of the original polynomial optimization problem, and When increasing The solution converges monotonically and non-decreasing to the global optimum. Simultaneously, the solver also returns the moment values ​​at the optimal solution. .

[0054] S7, For each mission group From the moment matrix Extract the first moments as estimates of each variable. Specifically, let... For variables The corresponding indicator vector (within this clique) is then estimated to be... For the mission team A variable in (For example ... ), its estimated value Equal to the optimal moment matrix of the clique The first moment extracted from it. It is a unit indicator vector, which is in the variable The components at the corresponding positions are The components at other locations are .therefore, That is, the moment variables corresponding to monomials (i.e., variables) The value of (to the power of 1). Due to the moment variable exist When exactly represents a variable The expected value (first moment) in the probability measure is therefore directly used as a variable. The estimated values ​​are obtained by extracting the moments from any clique containing the variable. If a variable appears in multiple cliques, the consistency constraint ensures that the corresponding moment variables are equal in different cliques. Therefore, extracting the estimated value from any clique containing the variable will yield the same estimated value. In this way, the estimated values ​​of all original decision variables are finally obtained. .

[0055] For variables that appear in multiple groups simultaneously (such as...) The values ​​can be taken from any clique, as the consistency constraint guarantees their equality. Finally, estimates of all original variables are obtained. , , , .

[0056] S8. Extract variable estimates and repair them to output the optimal resource allocation scheme. Specifically, variable estimate repair includes data scheduling ratio repair, bandwidth capacity repair, cache capacity repair, GPU number integer repair, budget constraint repair, and consistency update. Data scheduling ratio repair: based on data location matrix The proportion of unusable clusters in the variable estimates is forced to zero, i.e., if Then let Then for each task Normalize the remaining proportions ; Bandwidth capacity restoration: for each storage cluster Check if the total bandwidth allocation exceeds the upper limit; if Then compress proportionally. ; Cache capacity repair: for each task Check if the total cache capacity exceeds the cache limit; if Then proportional compression will be performed. ; GPU count integer fixation: Round consecutive estimates to ensure they are at least 1. ; Budget constraint repair: based on the repaired , , ,, Recalculate total cost ;like Then scale the adjustable resource cache and bandwidth of all tasks proportionally. ; Recalculate the completion time for each task after scaling. And check again whether the delay threshold constraint is met; if a certain If the threshold is exceeded, the bandwidth for that task will be locally adjusted and increased, while resources for other less sensitive tasks will be reduced accordingly. Consistent Update: Recalculate tasks based on the final fixed resource allocation. Completion time .

[0057] After the repair process, a strictly feasible resource allocation scheme was obtained, including the number of GPUs per task. Cache capacity Bandwidth allocation to each storage cluster and data reading ratio The solution satisfies all hard constraints (total budget, resource capacity, I / O blocking elimination, data integrity, etc.), and the objective function value is... It is a feasible lower bound of the original problem (while SDP relaxation provides a theoretical lower bound). The gap between the two can be used as an assessment of the solution quality.

[0058] S9. The scheduler distributes the optimal resource allocation plan to the cache manager and bandwidth controller. The cache manager then... Reserve cache space for each task (tasks sharing the same dataset can reuse the same cache) and initiate data prefetching. The bandwidth controller sets the upper limit of transmission bandwidth to each storage cluster for each task by calling the cloud service provider's API or configuring SDN policies. The deep learning task begins or continues training on the allocated GPU and is scaled proportionally with the set bandwidth. Read data from each storage cluster.

[0059] At the end of each scheduling slot, the system re-collects the latest task status, bandwidth measurements, cache hit rate, and remaining budget before entering the scheduling cycle for the next slot. For scenarios where computing resources can be dynamically scaled, tasks can add or remove GPUs at slot boundaries, and this method automatically adjusts cache and bandwidth to match the new computing power.

[0060] This scheme achieves a high-quality near-optimal solution within acceptable computational complexity through polynomial approximation, sparse clique decomposition, semidefinite programming relaxation, and multi-stage repair. It completely eliminates I / O blocking and maximizes the total scheduling welfare under consistency constraints.

[0061] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0062] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for collaborative allocation of intelligent computing resources for a multi-storage cluster compute-storage separation architecture, characterized in that, The steps include: At the beginning of each time slot, the system status is acquired; the system status includes the status of all tasks to be trained and tasks in operation, the available bandwidth and cache capacity of each storage cluster, the total resource cost budget, and resource pricing parameters; Establish the objective function and its constraints for multi-resource joint optimization; The objective function and constraints of multi-resource joint optimization are transformed into polynomial form using the polynomial approximation method. Auxiliary variables are then introduced to optimize the objective function and constraints of multi-resource joint optimization in polynomial form, resulting in an objective function and constraints with auxiliary limits. A variable coupling graph is constructed based on the objective function with auxiliary constraints and its constraints, and sparse clique decomposition is performed. Based on sparse cluster decomposition, sparse semidefinite programming relaxation is constructed using the Lasserre relaxation hierarchy to obtain the sparse SDP relaxation problem. Solving sparse SDP relaxation problems using an interior-point method solver; Extract variable estimates from the SDP solution, repair them, and output the optimal resource allocation scheme. The cache manager and bandwidth controller schedule resources according to the optimal resource allocation scheme issued. At the end of each scheduling slot, the system re-collects the latest system state and enters the scheduling cycle for the next slot.

2. The intelligent computing resource collaborative allocation method for a multi-storage cluster compute-storage separation architecture according to claim 1, wherein the objective function for joint optimization of multiple resources is: ; in, ； The constraints of the objective function for the multi-resource joint optimization are as follows: ； In the formula, This represents a user-defined utility function; For the task Completion time; Indicates task Total resource cost; This indicates the unit price of the GPU; Indicates assignment to task The number of GPUs; Indicates the unit price of resource caching; Indicates task The allocated data cache capacity; Indicates task The number of training rounds; Indicates task The number of datasets; Indicates task From storage cluster The proportion of data read; Indicates by The exported helper function is defined as follows: ; This represents a function that shows bandwidth cost increasing linearly with bandwidth. Indicates the total budget for the task; Indicates the cache limit; Represents a cluster The maximum bandwidth capacity; Indicates the first auxiliary variable; Indicates task From storage cluster Obtained network bandwidth; Indicates task Data set location indicator matrix, Indicates the second auxiliary variable. Indicates that task i is using Ideal throughput during GPU training Indicates in a given cache and bandwidth Training throughput under certain conditions.

3. The intelligent computing resource collaborative allocation method for a multi-storage cluster storage-compute separation architecture according to claim 2, characterized in that, The non-polynomial function of the objective function of the multi-resource joint optimization is obtained by employing the polynomial approximation method described above. and non-polynomial functions in the constraints Transforming it into polynomial form specifically includes: For each task Measure the number of different GPUs through offline experiments The ideal throughput is obtained to get the set of data points. Task i in adopting Ideal throughput during GPU training Fitting a using the least squares method polynomial of order ; The user-defined utility function Fit to a polynomial ; In the formula, express The fitting coefficient, yes The fitting coefficient, Indicates task The completion time D raised to the power of k, where k represents the order index of the polynomial fitting. Indicates task The number of GPUs raised to the power of k.

4. The intelligent computing resource collaborative allocation method for a multi-storage cluster compute-storage separation architecture according to claim 3, characterized in that, The task Completion Time By introducing the first auxiliary variable Second auxiliary variable Sub-completion time of task i in storage cluster s Perform auxiliary variable substitution. After auxiliary variable substitution, it is represented as: .

5. The intelligent computing resource collaborative allocation method for a multi-storage cluster compute-storage separation architecture according to claim 4, characterized in that, Each variable in the objective function containing auxiliary constraints is used as a vertex of the variable coupling graph. If two variables in the objective function containing auxiliary constraints appear simultaneously in the polynomial of the same constraint, then an undirected edge is connected between them.

6. The intelligent computing resource collaborative allocation method for a multi-storage cluster compute-storage separation architecture according to claim 4, characterized in that, The sparse SDP relaxation problem The construction steps include: For a given relaxation order For each sparse cluster obtained from the decomposition Introduce a set of local moment transformations ; Constructing the moment matrix based on local moment variables The elements of the moment matrix are And apply a positive semidefinite constraint. The rows and columns of the said moment matrix are formed by a number not exceeding [number missing]. A single-item index; Construct a localized moment matrix for the constraints of the objective function containing auxiliary limits and apply positive semidefinite constraints; For any two sparse graphs and Let their intersection be Take the intersection Any of the multiple indicators Its order No more than And sparse clusters and sparse clusters The local moment variables satisfy ; in, For multiple indicators, For the corresponding monomial Moments under a certain probability measure; sparse cluster Corresponding multiple indicators, sparse cluster Corresponding multiple indicators, Indicators and column indicators The corresponding matrix elements, Indicators of multiple indicators The corresponding matrix elements, Indicates multiple references The corresponding matrix elements.

7. The intelligent computing resource collaborative allocation method for a multi-storage cluster storage-compute separation architecture according to claim 1, characterized in that, For each mission group From the corresponding moment matrix The first moment is extracted as the estimate of each variable; the task force It is a collection of sparse clusters A subset of.

8. The intelligent computing resource collaborative allocation method for a multi-storage cluster storage-compute separation architecture according to claim 1, characterized in that, The variable estimate repair includes data scheduling ratio repair, bandwidth capacity repair, cache capacity repair, GPU number integer repair, budget constraint repair, and consistency update; Data scheduling ratio repair: based on data location matrix The proportion of unusable clusters in the variable estimates is forced to zero, i.e., if Then let For each task Normalize the remaining proportions ,in, This represents the variable estimates extracted from the SDP relaxation solution. This represents an estimated proportion of data that task i will read from storage cluster s after the repair. Bandwidth capacity restoration: for each storage cluster Check if the total bandwidth allocation exceeds the upper limit; if ,but ;in, This represents the task extracted from the SDP relaxed solution. From storage cluster The obtained network bandwidth Indicates the task after repair From storage cluster The network bandwidth obtained; Cache capacity repair: for each task Check if the total cache capacity exceeds the cache limit; if ,but ;in, This represents the task extracted from the SDP relaxed solution. The allocated data cache capacity, Indicates the task after repair The allocated network bandwidth; GPU count integer fixation: Round consecutive estimates to ensure they are at least 1. ,, This refers to the number of GPUs after the repair; This represents the task extracted from the SDP relaxed solution. The number of GPUs; Budget constraint repair: Based on the repaired , , Recalculate task Total resource cost ;like Then the adjustable resource cache and bandwidth of all tasks , ;in, Indicates the task after repair Total resource cost; Recalculate the repaired completion time for each task. And check again whether the delay threshold constraint is met; if a certain If the threshold is exceeded, the bandwidth for that task will be locally increased. Consistent Update: Recalculate tasks based on the final fixed resource allocation. Completion time .

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method according to any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method as described in any one of claims 1-8.

Images