Big model-based intelligent computing center GPU resource dynamic scheduling method and system

Abstract

This invention relates to the field of computing power scheduling technology, specifically to a method and system for dynamic scheduling of GPU resources in intelligent computing centers based on large-scale models. In this invention, by deeply analyzing the gradient data of the weight matrices of each layer during the training of a Transformer large-scale model, the dynamic demand characteristics of different computing tasks on GPU memory resources can be accurately characterized. Simultaneously, by monitoring the generation latency of activation tensors at each layer, the strong and weak dependency chains between tasks are quantified and identified, thereby constructing a multi-dimensional view reflecting the internal computing flow and resource consumption characteristics of the model. Based on this in-depth analysis, differentiated computing power weights are assigned to tasks with different dependency strengths and GPU memory usage characteristics, achieving accurate assessment and priority ranking of GPU computing power requirements. This ensures that computing resources can be tilted towards critical bottleneck tasks based on the inherent importance and resource consumption characteristics of the tasks. Furthermore, by comparing the actual execution time of tasks with the expected execution time in real time, the execution efficiency is dynamically evaluated.

Description

Technical Field

[0001] This invention relates to the field of computing power scheduling technology, and in particular to a method and system for dynamic scheduling of GPU resources in intelligent computing centers based on large models. Background Technology

[0002] The field of computing power scheduling technology involves the efficient management and scheduling of computing resources, especially GPU resources. With the rapid development of technologies such as deep learning, big data analytics, and artificial intelligence, the demand for computing power has increased dramatically, especially in cloud computing and intelligent computing centers. This involves the rational allocation of computing resources based on multi-dimensional information such as the needs of computing tasks, resource status, and network environment.

[0003] Among them, the dynamic scheduling method of GPU resources in intelligent computing centers refers to the allocation and adjustment of GPU resources in intelligent computing centers according to predetermined rules based on task requirements and system load.

[0004] The shortcoming of existing technologies lies in the fact that their dynamic scheduling methods mainly rely on general indicators such as the external requirements of tasks and the macroscopic load of the system, failing to deeply analyze the complex computational characteristics and data dependencies within the training tasks of large Transformer models. This allocation method based on predetermined rules cannot accurately identify the differences in resource requirements and inherent execution relationships among different computational tasks in the model. For example, during training, the scheduling method can only perceive the overall high GPU load, but cannot distinguish whether the load is caused by critical, strongly dependent tasks or by non-critical secondary tasks. This may result in the average distribution of computing resources or their incorrect allocation to non-bottleneck tasks, leading to blind resource allocation. Critical computational tasks cannot be prioritized, resulting in low execution efficiency and thus prolonging the training cycle of the entire large model. Summary of the Invention

[0005] To address the technical problems existing in the prior art, this invention provides a method for dynamic scheduling of GPU resources in intelligent computing centers based on large models, comprising the following steps:

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a dynamic scheduling method for GPU resources in intelligent computing centers based on large models, comprising the following steps:

[0007] S1: Obtain GPU resource requirements data during the training process of the Transformer large model in the intelligent computing center, analyze the dynamic changes in GPU memory usage, and generate GPU memory usage analysis results;

[0008] S2: Based on the memory usage analysis results, combined with the execution order of each task in the Transformer large model, calculate the latency difference between each task, identify the dependency links between tasks based on the latency difference, and generate task dependency link analysis results.

[0009] S3: Referring to the task dependency link analysis results and the memory usage analysis results, evaluate the GPU computing power resource requirements of each task, determine the computing power resource allocation plan for each task, and use it as the GPU computing power resource allocation plan.

[0010] S4: Analyze the difference between the actual execution time and the expected execution time of the tasks during the training process, evaluate the execution efficiency of each task, and determine the task optimization plan;

[0011] S5: Combining the GPU computing power resource allocation plan and the task optimization plan, adjust the GPU memory and computing power resource allocation to generate GPU resource scheduling optimization results.

[0012] As a further aspect of the present invention, the memory usage analysis results include peak memory usage at each layer, average memory usage, and memory usage fluctuation. The task dependency link analysis results include inter-task dependencies, inter-task latency differences, and dependency strength classification. The GPU computing power resource allocation plan specifically includes the GPU computing power required for each task, the proportion of allocated GPU computing power, and the computing power allocation priority. The task optimization plan includes optimizing the execution order, adjusting the computing load, and reducing computing time. The GPU resource scheduling optimization results specifically include memory allocation optimization and computing power allocation optimization.

[0013] As a further aspect of the present invention, step S1 specifically comprises:

[0014] S101: Obtain GPU resource requirement data during the training process of the Transformer large model in the intelligent computing center. The GPU resource requirement data includes the gradient data of the weight matrix of each layer in the self-attention layer, feedforward network layer and output layer of the Transformer large model. Calculate the mean and standard deviation of the gradient data of each layer and establish the layer gradient feature value.

[0015] S102: Based on the hierarchical gradient feature value, calculate the change in the average value and standard deviation of the gradient data within a continuous training period, and then calculate the ratio of the change to the average value and standard deviation of the gradient data at the beginning of the period to obtain the gradient dynamic volatility.

[0016] S103: Compare the gradient dynamic volatility of each layer with the set stable benchmark value of video memory usage, divide the video memory resource demand range for each layer according to the comparison result, and generate video memory usage analysis results.

[0017] As a further embodiment of the present invention, the gradient data of the weight matrix of each layer in the self-attention layer corresponds to the resource requirements of the context relationship modeling task, the gradient data of the weight matrix of each layer in the feedforward network layer corresponds to the resource requirements of the feature transformation task, and the gradient data of the weight matrix of each layer in the output layer corresponds to the resource requirements of the sequence generation task.

[0018] As a further aspect of the present invention, step S2 specifically comprises:

[0019] S201: Based on the memory usage analysis results and the execution order of each task in the Transformer large model, record the generation timestamps of the activation tensors of the self-attention layer, feedforward network layer and output layer, calculate the task latency differences between the self-attention layer and feedforward network layer and between the feedforward network layer and output layer, and obtain the inter-layer task latency.

[0020] S202: Compare the inter-layer task delay with the set strong dependency judgment threshold and secondary dependency identification threshold, identify those with differences exceeding the strong dependency judgment threshold as strong dependency links, perform multi-layer timestamp cross comparison for the inter-layer task delay with differences between the strong dependency judgment threshold and the secondary dependency identification threshold, identify secondary dependency links, and establish a task dependency link hierarchy.

[0021] S203: Based on the task dependency link hierarchy and the memory usage analysis results, associate the strong dependency links and secondary dependency links with the corresponding memory usage data to generate task dependency link analysis results.

[0022] As a further aspect of the present invention, step S3 specifically comprises:

[0023] S301: Referring to the dependency relationship and memory usage of each task in the task dependency link analysis results, assign computing power weight coefficients to strong dependency links and secondary dependency links, and calculate the GPU computing power resource requirements by combining the computing power weight coefficients and the corresponding memory usage values ​​of the tasks to obtain the task computing power requirement evaluation value.

[0024] S302: Sort all the computing power requirement evaluation values ​​of the tasks in descending order, compare the evaluation value of each task with the set priority judgment threshold, mark the tasks with evaluation values ​​higher than the priority judgment threshold as high priority, and mark the tasks with evaluation values ​​lower than the priority judgment threshold as low priority, and establish a computing power resource allocation priority sequence.

[0025] S303: Configure differentiated computing resource quotas for each priority task in the computing resource allocation priority sequence, and determine the GPU computing resource allocation plan for each task.

[0026] As a further aspect of the present invention, the process of allocating computing power weight coefficients for strong dependency links and secondary dependency links specifically involves: configuring a first computing power weight coefficient for each strong dependency link in the task dependency link analysis results according to a preset weight mapping relationship, and configuring a second computing power weight coefficient for each secondary dependency link, wherein the value of the first computing power weight coefficient is greater than the value of the second computing power weight coefficient.

[0027] As a further aspect of the present invention, step S4 specifically comprises:

[0028] S401: Statistically calculate the actual execution time of each task during training, obtain the expected execution time for each task, calculate the difference between the actual execution time and the expected execution time, and generate the task execution time deviation value;

[0029] S402: Compare the task execution time deviation value with the execution efficiency benchmark threshold, and combine the task's computational load data with the memory usage in the memory usage analysis results. Mark tasks with deviation values ​​greater than the efficiency benchmark threshold as low efficiency and tasks with deviation values ​​less than the efficiency benchmark threshold as high efficiency, and establish a task execution efficiency rating.

[0030] S403: Based on the task execution efficiency rating, formulate an optimized arrangement for adjusting the execution timing or load balancing for tasks with a low efficiency rating, maintain the existing arrangement for tasks with a high efficiency rating, and generate a task optimization plan.

[0031] As a further aspect of the present invention, step S5 specifically comprises:

[0032] S501: Combining the computing resource quota of each task in the GPU computing resource allocation plan with the corresponding optimization arrangement or existing arrangement in the task optimization plan, formulate a scheduling strategy to adjust GPU memory and computing resources, and generate a GPU resource scheduling adjustment strategy.

[0033] S502: Execute the GPU resource scheduling and adjustment strategy during training to dynamically allocate GPU memory and computing power resources to form a dynamic resource allocation configuration;

[0034] S503: Summarize the dynamic resource allocation configuration, integrate the GPU resource scheduling situation throughout the training process, and generate GPU resource scheduling optimization results.

[0035] A dynamic scheduling system for GPU resources in intelligent computing centers based on large-scale models includes:

[0036] The GPU memory usage analysis module acquires GPU resource requirement data during the training process of the Transformer large model in the intelligent computing center, analyzes the dynamic changes in GPU memory usage, and generates GPU memory usage analysis results.

[0037] The task dependency analysis module calculates the latency difference between each layer of tasks based on the memory usage analysis results and the execution order of each layer of tasks in the Transformer large model. It then identifies the dependency links between tasks based on the latency differences and generates task dependency link analysis results.

[0038] The computing power allocation plan module, referring to the task dependency link analysis results and the memory usage analysis results, evaluates the GPU computing power resource requirements of each task and determines the computing power resource allocation plan for each task, which serves as the GPU computing power resource allocation plan.

[0039] The task planning optimization module statistically analyzes the difference between the actual execution time and the expected execution time of tasks during training, evaluates the execution efficiency of each task, and determines the task optimization plan.

[0040] The resource scheduling optimization module, in conjunction with the GPU computing power resource allocation plan and the task optimization plan, adjusts the allocation of GPU memory and computing power resources, and generates GPU resource scheduling optimization results.

[0041] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0042] In this invention, by deeply analyzing the gradient data of the weight matrices of each layer during the training of a large Transformer model, the dynamic demand characteristics of different computing tasks on GPU memory resources can be accurately characterized. Simultaneously, by monitoring the generation latency of activation tensors in each layer, the strong and weak dependency chains between tasks are quantified and identified. This constructs a multi-dimensional view reflecting the computational flow and resource consumption characteristics within the model. Based on this in-depth analysis, differentiated computing power weights are assigned to tasks with different dependency strengths and GPU memory usage characteristics, achieving accurate assessment and priority ranking of GPU computing power requirements. This ensures that computing resources are tilted towards critical bottleneck tasks based on their inherent importance and resource consumption characteristics. Furthermore, by comparing the actual execution time with the expected execution time in real time, execution efficiency is dynamically evaluated, and targeted optimization measures such as load balancing or execution timing adjustments are taken for identified inefficient tasks. This forms a closed-loop scheduling process from in-depth analysis and resource planning to dynamic feedback optimization, ultimately significantly improving the utilization efficiency of GPU resources and shortening the overall training cycle of the model. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the steps of the present invention;

[0045] Figure 2 This is a detailed schematic diagram of S1 of the present invention;

[0046] Figure 3 This is a detailed schematic diagram of S2 of the present invention;

[0047] Figure 4 This is a detailed schematic diagram of S3 of the present invention;

[0048] Figure 5 This is a detailed schematic diagram of S4 of the present invention;

[0049] Figure 6 This is a detailed schematic diagram of S5 of the present invention;

[0050] Figure 7 This is a system module diagram of the present invention. Detailed Implementation

[0051] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0052] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0053] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0054] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0055] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0056] Please see Figure 1This invention provides a method for dynamic scheduling of GPU resources in intelligent computing centers based on large models, including the following steps:

[0057] S1: Obtain GPU resource requirements data during the training process of the Transformer large model in the intelligent computing center, analyze the dynamic changes in GPU memory usage, and generate GPU memory usage analysis results;

[0058] S2: Based on the memory usage analysis results and the execution order of each task in the Transformer large model, calculate the latency difference between each task, identify the dependency links between tasks based on the latency difference, and generate task dependency link analysis results.

[0059] S3: Based on the task dependency link analysis results and the memory usage analysis results, evaluate the GPU computing power resource requirements of each task, determine the computing power resource allocation plan for each task, and use it as the GPU computing power resource allocation plan.

[0060] S4: Analyze the difference between the actual execution time and the expected execution time of the tasks during the training process, evaluate the execution efficiency of each task, and determine the task optimization plan;

[0061] S5: Combine the GPU computing power resource allocation plan and task optimization plan to adjust the allocation of GPU memory and computing power resources, and generate GPU resource scheduling optimization results;

[0062] The memory usage analysis results include peak memory usage, average memory usage, and memory usage fluctuation for each layer. The task dependency link analysis results include inter-task dependencies, inter-task latency differences, and dependency strength classification. The GPU computing power resource allocation plan specifies the GPU computing power required for each task, the proportion of GPU computing power allocated, and the computing power allocation priority. The task optimization plan includes optimizing the execution order, adjusting the computing load, and reducing computing time. The GPU resource scheduling optimization results specifically include memory allocation optimization and computing power allocation optimization.

[0063] Please see Figure 2 Step S1 is as follows:

[0064] S101: Obtain GPU resource requirement data during the training process of the Transformer large model in the intelligent computing center. The GPU resource requirement data includes the gradient data of the weight matrix of each layer in the self-attention layer, feedforward network layer and output layer of the Transformer large model. Calculate the mean and standard deviation of the gradient data of each layer and establish the layer gradient feature values. Among them, the gradient data of the weight matrix of each layer in the self-attention layer corresponds to the resource requirement of the context relationship modeling task, the gradient data of the weight matrix of each layer in the feedforward network layer corresponds to the resource requirement of the feature transformation task, and the gradient data of the weight matrix of each layer in the output layer corresponds to the resource requirement of the sequence generation task.

[0065] During the training of a Transformer model with 7 billion parameters for a natural language understanding task at the intelligent computing center, the first step was to acquire GPU resource requirement data. Specifically, within a training cycle, the gradient data generated by the weight matrix of each computational layer during the backpropagation phase was monitored and recorded. The data collection covered all 32 self-attention layers constituting the context relationship modeling task, all 32 feedforward network layers constituting the feature transformation task, and one output layer constituting the sequence generation task. For self-attention layers, the collected data included the gradients of the query weight matrix, key weight matrix, value weight matrix, and output projection matrix; for feedforward network layers, the collected data included the gradients of the weight matrices of the first and second fully connected layers; and for the output layer, the gradient of its linear transformation matrix was collected.

[0066] The collected gradient data consists of a series of floating-point matrices. Taking the query weight matrix of the 10th self-attention layer as an example, its gradient data is a matrix containing a large number of values. Here, we select a small 4x4 sub-matrix for illustration, with the following values: [0.021, -0.015, 0.033, 0.004; -0.018, 0.025, 0.009, -0.022; 0.031, -0.007, -0.014, 0.028; 0.001, 0.019, -0.027, 0.011]. Summing the 16 gradient values ​​in this matrix yields a total of 0.085. Dividing the total by the number of values ​​(16) gives the average gradient data of the query weight matrix as 0.0053125. Next, the difference between each gradient value and the average value is calculated and squared. For example, the square of the difference between the first value 0.021 and the average value is (0.021-0.0053125)^2, resulting in 0.000246... The sum of the squares of all 16 differences is divided by 16, and the square root of the result is then taken, yielding a standard deviation of 0.0205. Similarly, the gradient data submatrix of the weight matrix of the first fully connected layer of the 10th feedforward network layer [0.15, 0.21, -0.18, 0.25; 0.22, -0.19, 0.17, 0.23; -0.24, 0.16, 0.20, -0.17; 0.18, -0.21, 0.23, 0.14] is calculated, yielding a gradient data average of 0.084375 and a standard deviation of 0.197. The gradient data submatrix of the linear transformation matrix of the output layer [-0.05, 0.08, 0.03, -0.06; 0.07, -0.04, 0.09, 0.02; -0.08, 0.01, -0.02, 0.05; 0.06, -0.07, 0.04, -0.01] is calculated, yielding an average gradient value of 0.010625 and a standard deviation of 0.056. The average gradient value and standard deviation of all weight matrices in each layer are then summed and calculated separately. For example, the average gradient value and standard deviation of all weight matrices in the 10th self-attention layer are averaged again to finally establish a layer gradient feature value consisting of two values ​​for that layer. Therefore, a unique hierarchical gradient feature value is established for each of the 65 computational layers. The feature value of the self-attention layer represents the resource requirements of the context relationship modeling task, the feature value of the feedforward network layer represents the resource requirements of the feature transformation task, and the feature value of the output layer represents the resource requirements of the sequence generation task.

[0067] S102: Based on the hierarchical gradient feature value, calculate the change in the average value and standard deviation of the gradient data within a continuous training period, and then calculate the ratio of the change to the average value and standard deviation of the gradient data at the beginning of the period to obtain the gradient dynamic volatility.

[0068] After obtaining the hierarchical gradient feature values ​​for five consecutive training epochs, the gradient dynamic volatility is calculated. Taking the average gradient data of the 10th self-attention layer as an example, in the hierarchical gradient feature values ​​obtained in the first training epoch (initial epoch), the average value is 0.0085 and the standard deviation is 0.0310. In the second training epoch, the average value becomes 0.0092 and the standard deviation becomes 0.0335. Calculating the change in the second epoch relative to the first epoch, the change in the average value is 0.0092 minus 0.0085, resulting in 0.0007; the change in the standard deviation is 0.0335 minus 0.0310, resulting in 0.0025. Subsequently, the ratio of these changes to the initial gradient data of the epoch is calculated. The ratio of the change in the average value is 0.0007 divided by 0.0085, resulting in approximately 0.0824. The ratio of the change in the standard deviation is 0.0025 divided by 0.0310, resulting in approximately 0.0806. Adding these two ratios together and dividing by two, we get (0.0824 + 0.0806) / 2, which gives us the gradient dynamic volatility rate of 0.0815 for the second training cycle.

[0069] Next, the same processing is applied to the data from the third training period. Assume the mean for the third period is 0.0088 and the standard deviation is 0.0320. Calculate the change relative to the second period: the change in mean is 0.0088 minus 0.0092, resulting in -0.0004; the change in standard deviation is 0.0320 minus 0.0335, resulting in -0.0015. Calculate the ratio of this change to the data from the second period: the ratio of the change in mean is -0.0004 divided by 0.0092, resulting in approximately -0.0435; the ratio of the change in standard deviation is -0.0015 divided by 0.0335, resulting in approximately -0.0448. Averaging these absolute values, i.e., (0.0435 + 0.0448) / 2, yields the gradient dynamic volatility of the third training period as 0.04415. This process is repeated for five consecutive training epochs to obtain a time series gradient dynamic volatility value, for example, [0.0815, 0.04415, 0.0620, 0.0950]. The maximum value in this sequence is selected as the final gradient dynamic volatility, i.e., 0.0950. This calculation is performed for each layer in the model, generating a unique gradient dynamic volatility value for each layer.

[0070] S103: Compare the gradient dynamic volatility of each layer with the set stable benchmark value of video memory usage, divide the video memory resource demand range for each layer according to the comparison results, and generate video memory usage analysis results.

[0071] The gradient dynamic volatility corresponding to each computational layer is compared with a pre-defined memory usage stability benchmark. This benchmark is based on regression analysis of memory usage logs and gradient volatility data from over 50 training tasks of large Transformer models of varying sizes. These historical training tasks recorded the percentage of gradient dynamic volatility and actual peak memory usage exceeding the average allocated value. Analysis revealed that when the gradient dynamic volatility is below 0.12, 95% of the training tasks have actual peak memory usage fluctuations not exceeding 5% of their allocated amount; while when the gradient dynamic volatility exceeds 0.12, the probability of drastic and irregular fluctuations in memory usage increases significantly. Based on these statistical results, a memory usage stability benchmark of 0.12 is set as a distinguishing point to differentiate between different levels of memory demand stability.

[0072] The comparison process is as follows: The gradient dynamic volatility of the 10th self-attention layer, 0.0950, is extracted and compared with the stable memory usage benchmark value of 0.12. Since 0.0950 is less than 0.12, the memory resource requirement of the 10th self-attention layer is determined to be stable, and a "stable memory usage interval" is defined for it. Next, the gradient dynamic volatility of the 15th feedforward network layer is extracted, with a calculated value of 0.1580. 0.1580 is compared with 0.12. Since 0.1580 is greater than 0.12, the memory resource requirement of the 15th feedforward network layer is determined to be unstable, and a "fluctuating memory usage interval" is defined for it. This comparison and division operation is performed on all 65 computational layers in the model. After the division of all layers is completed, a memory usage analysis result is generated. This result details each computational layer (e.g., self-attention layers 1 to 32, feedforward network layers 1 to 32, output layer 1) and its corresponding memory resource requirement interval attribute (stable or fluctuating).

[0073] Please see Figure 3 Step S2 is as follows:

[0074] S201: Based on the memory usage analysis results and the execution order of each task in the Transformer large model, record the generation timestamps of the activation tensors of the self-attention layer, feedforward network layer and output layer, calculate the task latency differences between the self-attention layer and feedforward network layer, and between the feedforward network layer and output layer, and obtain the inter-layer task latency.

[0075] Based on the memory usage analysis results and strictly following the inherent computational flow within the Transformer large model—that is, data flows sequentially through the self-attention layer, the feedforward network layer, and finally into the output layer—the completion time of the core computational stages in each layer is recorded. The recorded object here is the generation timestamp of the activation tensor, which is the output data passed from one layer to the next after computation. During a single forward propagation, a high-precision timer captures the completion times of the following events: the timestamp of the 10th self-attention layer completing all its computations and generating its activation tensor is recorded as 215.35 milliseconds; immediately following, the 10th feedforward network layer receives the aforementioned activation tensor, completes its own computation, and generates its own activation tensor, with a timestamp recorded as 288.15 milliseconds; finally, the timestamp of the data flowing through all layers, the output layer completing computation, and generating the final sequence is recorded as 310.65 milliseconds.

[0076] After obtaining these timestamps, the inter-layer task latency is calculated. The task latency between the self-attention layer and the feedforward network layer is calculated by subtracting the activation tensor generation timestamp of the 10th self-attention layer (215.35 milliseconds) from the activation tensor generation timestamp of the 10th feedforward network layer (288.15 milliseconds). The result is 72.80 milliseconds, which represents the time consumed by data transfer and computation between these two consecutive processing stages of the model. Similarly, the task latency between the feedforward network layer and the output layer is calculated by subtracting the activation tensor generation timestamp of the feedforward network layer (here, the 32nd feedforward network layer, assuming its completion timestamp is 305.20 milliseconds) that directly provides input to the output layer from the output layer's activation tensor generation timestamp (310.65 milliseconds). This latency is assumed to be 5.45 milliseconds. This timestamp recording and latency calculation is performed for all adjacent self-attention layers and feedforward network layers with direct data flow dependencies in the model, as well as between feedforward network layers and the output layer. Ultimately, a series of inter-layer task delay values ​​are obtained, which together constitute complete inter-layer task delay data.

[0077] S202: Compare the inter-layer task delay with the set strong dependency judgment threshold and secondary dependency identification threshold. Identify the differences that exceed the strong dependency judgment threshold as strong dependency links. Perform multi-layer timestamp cross comparison for inter-layer task delays with differences between the strong dependency judgment threshold and secondary dependency identification threshold to identify secondary dependency links and establish a task dependency link hierarchy.

[0078] The calculated inter-layer task latency values ​​are compared with the set strong dependency determination threshold and secondary dependency identification threshold to establish a hierarchical structure of task dependency chains. These two thresholds are based on performance profiling of the model after more than 1000 complete training iterations on standard hardware configurations (e.g., NVIDIA A100 GPU clusters). All inter-layer task latencies in the profiling dataset were collected and formed into a distribution histogram. The strong dependency determination threshold was set at the 85th percentile of this distribution, representing a significant bottleneck in the training process, with an experimental value of 60 milliseconds. The secondary dependency identification threshold was set at the 40th percentile of the distribution, representing a general but non-bottleneck processing latency, with an experimental value of 25 milliseconds.

[0079] The comparison process is as follows: The task latency of 72.80 milliseconds between the 10th self-attention layer and the 10th feedforward network layer is extracted. First, 72.80 milliseconds is compared with the strong dependency threshold of 60 milliseconds. Because 72.80 milliseconds is greater than 60 milliseconds, the task link between these two layers is directly identified as a strong dependency link. Next, another pair of layers is extracted, for example, the task latency between the 20th self-attention layer and the 20th feedforward network layer, assuming it is 45.50 milliseconds. 45.50 milliseconds is compared with the strong dependency threshold of 60 milliseconds and the secondary dependency identification threshold of 25 milliseconds. Because 45.50 milliseconds is less than 60 milliseconds but greater than 25 milliseconds, this task latency falls between the two, requiring multi-layer timestamp cross-comparison. Specifically, this cross-comparison checks the task latency between five consecutive self-attention layers from layer 18 to layer 22 and their corresponding feedforward network layers. If most of these latency values ​​(e.g., more than three) stably fall within this range, then the link in layer 20 is confirmed as a secondary dependency link. If the latency value occurs randomly within this range, it is not marked as a secondary dependency. Assuming cross-comparison confirms stability, the link is identified as a secondary dependency link. Links with latency less than 25 milliseconds, such as the 5.45 millisecond latency between the feedforward and output layers, are classified as weakly dependent links. By performing this process on all inter-layer latencies, a complete hierarchy of task dependency links is ultimately established.

[0080] S203: Based on the task dependency link hierarchy and memory usage analysis results, associate strong dependency links and secondary dependency links with their corresponding memory usage data to generate task dependency link analysis results;

[0081] After obtaining the task dependency link hierarchy and memory usage analysis results, a data association operation is performed. This operation pairs the link hierarchy, representing the temporal dependency relationship between tasks, with memory usage data, representing the resource consumption characteristics of each task. Specifically, each link in the task dependency link hierarchy is traversed. Taking the link from the 10th self-attention layer to the 10th feedforward network layer as an example, its dependency level is found to be "strong dependency link" from the task dependency link hierarchy. Simultaneously, the memory usage of these two layers is queried from the memory usage analysis results. The memory usage range of the 10th self-attention layer is found to be "stable memory usage range," and the memory usage range of the 10th feedforward network layer is found to be "fluctuating memory usage range." These three pieces of information—the two endpoints of the link, the dependency level of the link, and the memory usage ranges of the tasks at both ends of the link—are then integrated.

[0082] The same operation is performed on the link from the 20th self-attention layer to the 20th feedforward network layer. The dependency level is found to be "secondary dependency link" from the task dependency link hierarchy. From the memory usage analysis results, the memory usage range of the 20th self-attention layer is found to be "stable memory usage range," and the memory usage range of the 20th feedforward network layer is also "stable memory usage range." This information is integrated. This association process continues until all hierarchical dependency links in the model have been associated with the corresponding layer's memory usage data. Finally, a structured task dependency link analysis result is generated. Each record in this result clearly describes a task dependency relationship, indicating not only the strength of the dependency but also the memory resource requirements of the source and target tasks constituting this dependency relationship, for example: "Link (Self-attention layer 10 -> Feedforward network layer 10): Strong dependency link, Source task memory usage: Stable, Target task memory usage: Fluctuating."

[0083] Please see Figure 4 Step S3 is as follows:

[0084] S301: Referring to the dependency relationship and memory usage of each task in the task dependency link analysis results, assign computing power weight coefficients to strong dependency links and secondary dependency links. Combine the computing power weight coefficients with the memory usage values ​​corresponding to the tasks to calculate the GPU computing power resource requirements and obtain the task computing power requirement evaluation value. Specifically, the process of assigning computing power weight coefficients to strong dependency links and secondary dependency links is as follows: Based on the preset weight mapping relationship, configure a first computing power weight coefficient for each strong dependency link in the task dependency link analysis results and configure a second computing power weight coefficient for each secondary dependency link. The value of the first computing power weight coefficient is greater than the value of the second computing power weight coefficient.

[0085] Based on the dependency relationships and memory usage of each task in the task dependency chain analysis results, computational power weight coefficients were assigned to dependency chains of different levels. The computational power weight coefficients were set according to a preset weight mapping relationship, which was determined through a series of benchmark experiments. In the experiments, computational resources were increased or decreased for tasks in chains of different dependency levels, and the impact on the total training time of the model was observed. The experimental results show that the training acceleration effect of increasing computational power for tasks in strong dependency chains is approximately 2.5 times that of increasing the same amount of computational power for tasks in secondary dependency chains. Therefore, the baseline value of the second computational power weight coefficient for secondary dependency chains was set to 1.0, and the first computational power weight coefficient for strong dependency chains was set to 2.5. For weak dependency chains, the computational power weight coefficient was set to 0.4.

[0086] Next, the GPU computing power resource requirements are calculated by combining the computing power weight coefficient with the corresponding memory usage value of the task. First, the qualitative description of the memory usage range needs to be quantified. By statistically analyzing historical training data, the "stable memory usage range" is quantified as the average memory usage value of the task in a stable state, such as 12 gigabytes; the "fluctuating memory usage range" is quantified as the peak memory usage value of the task during training, such as 18 gigabytes.

[0087] The calculation process is as follows: Taking the 10th self-attention layer task as an example, as the source of the "strong dependency link", its computing power weight coefficient is 2.5. Its memory usage range is the "stable memory usage range", with a quantization value of 12 gigabytes. The computing power requirement evaluation value of this task is obtained by multiplying the computing power weight coefficient by the memory usage value, that is, 2.5 multiplied by 12, resulting in 30.

[0088] Taking the 10th feedforward network layer task as an example, it is the target of the aforementioned "strong dependency link," and also assumed to be the source of another "secondary dependency link," with a computational power weight coefficient of 1.0. Its memory usage range is the "fluctuating memory usage range," with a quantization value of 18 gigabytes. The computational power requirement evaluation value for this task is 1.0 multiplied by 18, resulting in 18.

[0089] This process is performed on all tasks involved in the computation in the model to calculate a task computational power requirement evaluation value for each task.

[0090] S302: Sort all task computing power requirement evaluation values ​​in descending order, compare the evaluation value of each task with the set priority judgment threshold, mark tasks with evaluation values ​​higher than the priority judgment threshold as high priority, and mark tasks with evaluation values ​​lower than the priority judgment threshold as low priority, and establish a computing power resource allocation priority sequence.

[0091] The computational power requirement evaluation values ​​of all tasks calculated in the previous stage are sorted in descending order to form an ordered list. For example, the resulting evaluation value sequence is [30, 28, 25, 18, 17, 11, ...]. Next, each evaluation value in the list is compared with a set priority judgment threshold. This priority judgment threshold is set with reference to the Pareto principle of resource allocation, aiming to concentrate resources on a few key tasks that have the greatest impact on overall performance. By analyzing the computational power requirement evaluation values ​​of multiple historical model training tasks, it was found that setting the threshold in the top 20% after sorting all evaluation values ​​can effectively filter out core bottleneck tasks. In this embodiment, the distribution of evaluation values ​​for all tasks is calculated, and the evaluation value corresponding to the 80th percentile is determined to be 22. Therefore, the priority judgment threshold is set to 22.

[0092] The comparison and labeling process is as follows: Take the first value, 30 (corresponding to the 10th self-attention layer task), from the task computing power requirement evaluation value list. Compare 30 with the priority threshold 22. Since 30 is higher than 22, this task is labeled as high priority. Take the evaluation value 18 (corresponding to the 10th feedforward network layer task) from the list. Compare 18 with 22. Since 18 is lower than 22, this task is labeled as low priority. Perform this comparison operation on all evaluation values ​​in the list one by one. Tasks with evaluation values ​​higher than 22 are labeled as high priority, while tasks with evaluation values ​​equal to or lower than 22 are labeled as low priority. After labeling all tasks, a priority sequence for computing power resource allocation, arranged according to the task computing power requirement evaluation values ​​and labeled with priority, is established. An example of the sequence is as follows: "1. 10th self-attention layer, evaluation value 30, high priority; 2. 5th feedforward network layer, evaluation value 28, high priority; ...; 15. 10th feedforward network layer, evaluation value 18, low priority; ...."

[0093] S303: Configure differentiated computing resource quotas for tasks of each priority in the computing resource allocation priority sequence, and determine the GPU computing resource allocation plan for each task;

[0094] Tasks marked with different priorities in the priority sequence of computing resource allocation are configured with differentiated computing resource quotas. The principle of this quota allocation is to ensure that high-priority tasks receive sufficient computing power to shorten execution time, while reserving necessary computing power for low-priority tasks to maintain the normal progress of the training process. The configuration scheme was derived through multiple rounds of iterative testing in a simulation environment. The goal of the testing was to find the resource allocation ratio that minimizes the training time for each step while ensuring training stability. The final determined scheme is as follows: for all tasks marked as "high priority", 75% of the GPU computing core resources are uniformly allocated as their computing resource quota; for all tasks marked as "low priority", 25% of the GPU computing core resources are allocated as their computing resource quota.

[0095] Based on this configuration scheme, a specific GPU computing resource allocation plan is determined for each task. The priority flag for each task is read from the computing resource allocation priority sequence. For the 10th self-attention layer task marked as "high priority," the allocation plan is: during its execution, the scheduler must ensure that it can use 75% of the streaming multiprocessor (SM) resources on its physical GPU. For the 10th feedforward network layer task marked as "low priority," the allocation plan is: during its execution, the scheduler allocates 25% of the streaming multiprocessor resources to it. This process traverses the entire priority sequence, clarifying the proportion of computing core resources available to each task during execution. Finally, a detailed GPU computing resource allocation plan is formed, which includes the identifier, priority, and corresponding specific percentage of computing resources for each computational task in the model.

[0096] Please see Figure 5 Step S4 is as follows:

[0097] S401: Statistically calculate the actual execution time of each task during training, obtain the expected execution time for each task, calculate the difference between the actual execution time and the expected execution time, and generate the task execution time deviation value;

[0098] During the dynamic execution of model training, the actual execution time of each task is statistically analyzed, and its corresponding expected execution time is obtained. The actual execution time is directly measured by inserting high-precision timestamps at the start and end of each task's computation, and then calculating the difference between the two timestamps. For example, in the Nth training step, the 10th self-attention layer task was observed to have taken 98.5 milliseconds from start to output. The expected execution time is obtained from one or more benchmark performance tests of the entire model on the current hardware platform. In these tests, the model runs under ideal conditions with no interference and no resource contention, and the average execution time of each task is recorded. In this way, the expected execution time of the 10th self-attention layer task is found to be 85.0 milliseconds.

[0099] After obtaining the actual execution time and the expected execution time, the difference between the two is calculated to generate a task execution time deviation value. The calculation method is to subtract the expected execution time from the actual execution time. For the 10th self-attention layer task, its execution time deviation value is 98.5 milliseconds minus 85.0 milliseconds, resulting in 13.5 milliseconds. This positive deviation value indicates that the actual execution speed of the task is slower than expected. This measurement and calculation process is repeated for each computational task involved in the training process. For example, the actual execution time of the 10th feedforward network layer task is 45.2 milliseconds, and its expected execution time is 42.0 milliseconds, so its task execution time deviation value is 3.2 milliseconds. Finally, an execution time deviation value is calculated for all tasks, forming a complete data record.

[0100] S402: Compare the task execution time deviation value with the execution efficiency benchmark threshold, and combine the task's computational load data with the memory usage analysis results to mark tasks with deviation values ​​greater than the efficiency benchmark threshold as low efficiency and tasks with deviation values ​​less than the efficiency benchmark threshold as high efficiency, thus establishing a task execution efficiency rating.

[0101] The execution time deviation of each task is compared with a predefined execution efficiency benchmark threshold. This benchmark threshold is not a fixed value, but a dynamic value related to the expected execution time of the task. It is based on the analysis of a large amount of historical training data, which shows that when the actual execution time of a task exceeds its expected time by more than 10%, it usually indicates a performance bottleneck that needs attention. Therefore, the execution efficiency benchmark threshold is defined as "10% of the expected execution time".

[0102] The comparison process combined the task's computational load data with the previously generated memory usage analysis results. First, an efficiency baseline threshold was calculated for each task. For the 10th self-attention layer task, its expected execution time was 85.0 milliseconds, so its execution efficiency baseline threshold was 10% of 85.0 milliseconds, or 8.5 milliseconds. Then, its task execution time deviation of 13.5 milliseconds was compared with the threshold of 8.5 milliseconds. Because 13.5 milliseconds was greater than 8.5 milliseconds, this task was initially determined to be inefficient. At this point, further confirmation was made by combining other information: it was found that the computational load of the 10th self-attention layer task was at a high level, and its memory usage was within a "stable memory usage range." Based on this information, this task was ultimately marked as inefficient.

[0103] For the 10th feedforward network layer task, its expected execution time is 42.0 milliseconds, and its execution efficiency baseline threshold is 4.2 milliseconds. Its task execution time deviation is 3.2 milliseconds. Because 3.2 milliseconds is less than 4.2 milliseconds, this task is marked as high-efficiency.

[0104] By performing this comparative and combined analysis process on all tasks, a task execution efficiency rating is established, in which each task is assigned a rating label of "high efficiency" or "low efficiency".

[0105] S403: Based on the task execution efficiency rating, formulate optimized arrangements for adjusting execution timing or load balancing for tasks with a low efficiency rating, maintain the existing arrangements for tasks with a high efficiency rating, and generate a task optimization plan;

[0106] Based on the task execution efficiency ratings, differentiated scheduling arrangements are developed for tasks with different ratings. For tasks rated "low efficiency," optimized arrangements aimed at improving their execution efficiency are needed. Taking the 10th self-attention layer task, rated "low efficiency," as an example, its execution time deviation is significant. The reason for this might be due to excessive computational load, preventing timely completion within the allocated computing power. The optimized arrangement is load balancing: the matrix operation of the 10th self-attention layer, originally handled by a single GPU computing unit, is split into two sub-tasks and scheduled for parallel processing on two adjacent GPU computing units. This approach shortens the overall task completion time by increasing parallelism. Another possible optimized arrangement is execution timing adjustment; if a low-efficiency task is not on the critical path, it can be scheduled for execution during a less busy idle window.

[0107] For tasks rated "high efficiency," such as the 10th feedforward network layer, their execution met expectations. Therefore, the arrangement for them was to maintain the existing arrangement, continuing to use the previously planned computing power and scheduling order without any changes. This operation avoided unnecessary scheduling adjustment overhead and maintained system stability. This judgment was performed on each task in the task execution efficiency rating, matching it with an optimization arrangement such as "maintaining the existing arrangement," "adjusting the execution timing," or "load balancing" based on its "high efficiency" or "low efficiency" rating. All these arrangements were then aggregated to generate a final, clear task optimization plan.

[0108] Please see Figure 6 Step S5 is as follows:

[0109] S501: Combine the computing resource quota of each task in the GPU computing resource allocation plan with the corresponding optimization arrangement or existing arrangement in the task optimization plan, formulate a scheduling strategy to adjust GPU memory and computing resources, and generate a GPU resource scheduling adjustment strategy.

[0110] By combining the computing resource allocation quota for each task in the GPU computing resource allocation plan with the optimization or existing arrangements specified for each task in the task optimization plan, a scheduling strategy to adjust GPU memory and computing resources is formulated. This process is an integration and concretization step. First, the task optimization plan is reviewed. Taking the 10th self-attention layer task as an example, its optimization arrangement is "load balancing." Meanwhile, the computing resource allocation quota for this task is found to be 75% in the GPU computing resource allocation plan. The specific adjustment strategy is as follows: modify the computation graph nodes of the 10th self-attention layer task, decomposing them into two parallel child nodes. Map these two child nodes to two different GPUs. Simultaneously, allocate 75% computing resource quota to each of these two child nodes on their respective GPUs. Furthermore, the memory allocation strategy needs to be adjusted to ensure that the required weight data after decomposition is available on both GPUs, and the input activation data is split and transmitted separately to the two GPUs.

[0111] For the 10th feedforward network layer task, the optimization plan is to "maintain the existing arrangement." Its computational resource allocation is 25%. Therefore, the scheduling adjustment strategy for it is to maintain the status quo: allocate 25% of the computational resources on the original GPU during its execution phase, and maintain its original memory allocation scheme. This combined analysis process is performed on all tasks in the task optimization plan one by one, combining the abstract optimization arrangement with the specific resource allocation, and transforming it into a set of instructions that can be executed by the underlying scheduler. The collection of these instruction sets ultimately constitutes the complete GPU resource scheduling adjustment strategy.

[0112] S502: During training, the GPU resource scheduling and adjustment strategy is executed to dynamically allocate GPU memory and computing power resources, forming a dynamic resource allocation configuration.

[0113] In the subsequent training iterations, the GPU resource scheduling adjustment strategy is actually executed. This execution process is completed by the resource scheduler of the intelligent computing center at runtime. When the training program is about to start a new computation step, the scheduler loads the GPU resource scheduling adjustment strategy generated in the previous stage. According to the strategy, the scheduler dynamically allocates the GPU's memory and computing resources. Specifically, for the 10th self-attention layer task that requires load balancing, before starting its computing kernel, the scheduler issues an instruction to the GPU driver to distribute the computation task to two specified GPU devices and sets corresponding resource limits for the computing cores executing on these devices, ensuring that each subtask uses only 75% of the streaming multiprocessors. At the same time, the memory manager copies the required weight data to the target GPU's memory in advance according to the strategy and adjusts the data transfer flow to guide the split input data to the correct devices.

[0114] For the 10th feedforward network layer task, which maintains the existing schedule, the scheduler, as originally planned, allocates 25% of the streaming multiprocessor resources to its designated single GPU core. Throughout the training iterations, the scheduler continuously monitors task execution and adjusts the available computing and storage resources for each task in real time according to the policy. This series of runtime, policy-based resource allocation and adjustment operations together constitute the dynamic resource allocation configuration.

[0115] S503: Summarizes dynamic resource allocation configurations, integrates GPU resource scheduling throughout the entire training process, and generates GPU resource scheduling optimization results;

[0116] During the training process, or after a specified training phase (e.g., several epochs), all executed dynamic resource allocation configurations are summarized. This process aims to integrate and record the GPU resource scheduling throughout the entire training process. Specifically, it involves collecting and processing the dynamic resource allocation configuration logs generated in each training iteration. The logs detail the specific GPU device number, computing power percentage, memory allocation size, and actual start and end times of each task at each time step.

[0117] During the aggregation process, these raw log data undergo aggregator analysis. For example, this includes calculating the number of times each task was dynamically adjusted throughout the training process, analyzing the average computing power fulfillment rate of high-priority tasks, statistically analyzing the cross-GPU communication overhead caused by load balancing strategies, and evaluating the average and peak utilization of the overall GPU cluster. This statistical analysis data, along with the original configuration change records, is integrated into a comprehensive report or structured log file. This file comprehensively reflects the scheduling and optimization trajectory of GPU resources throughout the training process after the introduction of dynamic scheduling strategies. This final summary report represents the GPU resource scheduling optimization results.

[0118] Please see Figure 7 A dynamic scheduling system for GPU resources in intelligent computing centers based on large-scale models includes:

[0119] The GPU memory usage analysis module acquires GPU resource requirement data during the training process of the Transformer large model in the intelligent computing center, analyzes the dynamic changes in GPU memory usage, and generates GPU memory usage analysis results.

[0120] The task dependency analysis module calculates the latency difference between tasks at each layer based on the memory usage analysis results and the execution order of tasks at each layer of the Transformer large model. It then identifies the dependency links between tasks based on the latency differences and generates task dependency link analysis results.

[0121] The computing power allocation plan module, referring to the task dependency link analysis results and the memory usage analysis results, evaluates the GPU computing power resource requirements of each task and determines the computing power resource allocation plan for each task, which serves as the GPU computing power resource allocation plan.

[0122] The task planning optimization module statistically analyzes the difference between the actual execution time and the expected execution time of tasks during training, evaluates the execution efficiency of each task, and determines the task optimization plan.

[0123] The resource scheduling optimization module combines the GPU computing power resource allocation plan and the task optimization plan to adjust the allocation of GPU memory and computing power resources, and generate GPU resource scheduling optimization results.

[0124] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for dynamic scheduling of GPU resources in intelligent computing centers based on large-scale models, characterized in that, Includes the following steps: S1: Obtain GPU resource requirements data during the training process of the Transformer large model in the intelligent computing center, analyze the dynamic changes in GPU memory usage, and generate GPU memory usage analysis results; S2: Based on the memory usage analysis results, combined with the execution order of each task in the Transformer large model, calculate the latency difference between each task, identify the dependency links between tasks based on the latency difference, and generate task dependency link analysis results. S3: Referring to the task dependency link analysis results and the memory usage analysis results, evaluate the GPU computing power resource requirements of each task, determine the computing power resource allocation plan for each task, and use it as the GPU computing power resource allocation plan. S4: Analyze the difference between the actual execution time and the expected execution time of the tasks during the training process, evaluate the execution efficiency of each task, and determine the task optimization plan; S5: Combining the GPU computing power resource allocation plan and the task optimization plan, adjust the GPU memory and computing power resource allocation to generate GPU resource scheduling optimization results; Step S1 is as follows: S101: Obtain GPU resource requirement data during the training process of the Transformer large model in the intelligent computing center. The GPU resource requirement data includes the gradient data of the weight matrix of each layer in the self-attention layer, feedforward network layer and output layer of the Transformer large model. Calculate the mean and standard deviation of the gradient data of each layer and establish the layer gradient feature value. S102: Based on the hierarchical gradient feature value, calculate the change in the average value and standard deviation of the gradient data within a continuous training period, and then calculate the ratio of the change to the average value and standard deviation of the gradient data at the beginning of the period to obtain the gradient dynamic volatility. S103: Compare the gradient dynamic volatility of each layer with the set stable benchmark value of video memory usage, divide the video memory resource demand range for each layer according to the comparison result, and generate video memory usage analysis results. Step S2 is as follows: S201: Based on the memory usage analysis results and the execution order of each task in the Transformer large model, record the generation timestamps of the activation tensors of the self-attention layer, feedforward network layer and output layer, calculate the task latency differences between the self-attention layer and feedforward network layer and between the feedforward network layer and output layer, and obtain the inter-layer task latency. S202: Compare the inter-layer task delay with the set strong dependency judgment threshold and secondary dependency identification threshold, identify those with differences exceeding the strong dependency judgment threshold as strong dependency links, perform multi-layer timestamp cross comparison for the inter-layer task delay with differences between the strong dependency judgment threshold and the secondary dependency identification threshold, identify secondary dependency links, and establish a task dependency link hierarchy. S203: Based on the task dependency link hierarchy and the memory usage analysis results, associate the strong dependency links and secondary dependency links with the corresponding memory usage data to generate task dependency link analysis results.

2. The method for dynamic scheduling of GPU resources in intelligent computing centers based on large models according to claim 1, characterized in that, The memory usage analysis results include peak memory usage, average memory usage, and memory usage fluctuation for each layer. The task dependency link analysis results include inter-task dependencies, inter-task latency differences, and dependency strength classification. The GPU computing power resource allocation plan specifies the GPU computing power required for each task, the proportion of GPU computing power allocated, and the computing power allocation priority. The task optimization plan includes optimizing the execution order, adjusting the computing load, and reducing computing time. The GPU resource scheduling optimization results specifically include memory allocation optimization and computing power allocation optimization.

3. The method for dynamic scheduling of GPU resources in intelligent computing centers based on large models according to claim 1, characterized in that, The gradient data of the weight matrix of each layer in the self-attention layer corresponds to the resource requirements of the context relationship modeling task; the gradient data of the weight matrix of each layer in the feedforward network layer corresponds to the resource requirements of the feature transformation task; and the gradient data of the weight matrix of each layer in the output layer corresponds to the resource requirements of the sequence generation task.

4. The method for dynamic scheduling of GPU resources in intelligent computing centers based on large models according to claim 1, characterized in that, Step S3 is as follows: S301: Referring to the dependency relationship and memory usage of each task in the task dependency link analysis results, assign computing power weight coefficients to strong dependency links and secondary dependency links, and calculate the GPU computing power resource requirements by combining the computing power weight coefficients and the corresponding memory usage values ​​of the tasks to obtain the task computing power requirement evaluation value. S302: Sort all the computing power requirement evaluation values ​​of the tasks in descending order, compare the evaluation value of each task with the set priority judgment threshold, mark the tasks with evaluation values ​​higher than the priority judgment threshold as high priority, and mark the tasks with evaluation values ​​lower than the priority judgment threshold as low priority, and establish a computing power resource allocation priority sequence. S303: Configure differentiated computing resource quotas for each priority task in the computing resource allocation priority sequence, and determine the GPU computing resource allocation plan for each task.

5. The method for dynamic scheduling of GPU resources in intelligent computing centers based on large models according to claim 4, characterized in that, The process of allocating computing power weight coefficients for strong dependency links and secondary dependency links is as follows: based on a preset weight mapping relationship, a first computing power weight coefficient is configured for each strong dependency link in the task dependency link analysis result, and a second computing power weight coefficient is configured for each secondary dependency link, wherein the value of the first computing power weight coefficient is greater than the value of the second computing power weight coefficient.

6. The method for dynamic scheduling of GPU resources in intelligent computing centers based on large models according to claim 1, characterized in that, Step S4 is as follows: S401: Statistically calculate the actual execution time of each task during training, obtain the expected execution time for each task, calculate the difference between the actual execution time and the expected execution time, and generate the task execution time deviation value; S402: Compare the task execution time deviation value with the execution efficiency benchmark threshold, and combine the task's computational load data with the memory usage in the memory usage analysis results. Mark tasks with deviation values ​​greater than the efficiency benchmark threshold as low efficiency and tasks with deviation values ​​less than the efficiency benchmark threshold as high efficiency, and establish a task execution efficiency rating. S403: Based on the task execution efficiency rating, formulate an optimized arrangement for adjusting the execution timing or load balancing for tasks with a low efficiency rating, maintain the existing arrangement for tasks with a high efficiency rating, and generate a task optimization plan.

7. The method for dynamic scheduling of GPU resources in intelligent computing centers based on large models according to claim 1, characterized in that, Step S5 is as follows: S501: Combining the computing resource quota of each task in the GPU computing resource allocation plan with the corresponding optimization arrangement or existing arrangement in the task optimization plan, formulate a scheduling strategy to adjust GPU memory and computing resources, and generate a GPU resource scheduling adjustment strategy. S502: Execute the GPU resource scheduling and adjustment strategy during training to dynamically allocate GPU memory and computing power resources to form a dynamic resource allocation configuration; S503: Summarize the dynamic resource allocation configuration, integrate the GPU resource scheduling situation throughout the training process, and generate GPU resource scheduling optimization results.

8. A dynamic scheduling system for GPU resources in an intelligent computing center based on a large model, characterized in that: The system is used to implement the dynamic scheduling method for GPU resources in intelligent computing centers based on large models as described in any one of claims 1-7, and the system comprises: The GPU memory usage analysis module acquires GPU resource requirement data during the training process of the Transformer large model in the intelligent computing center, analyzes the dynamic changes in GPU memory usage, and generates GPU memory usage analysis results. The task dependency analysis module calculates the latency difference between each layer of tasks based on the memory usage analysis results and the execution order of each layer of tasks in the Transformer large model. It then identifies the dependency links between tasks based on the latency differences and generates task dependency link analysis results. The computing power allocation plan module, referring to the task dependency link analysis results and the memory usage analysis results, evaluates the GPU computing power resource requirements of each task and determines the computing power resource allocation plan for each task, which serves as the GPU computing power resource allocation plan. The task planning optimization module statistically analyzes the difference between the actual execution time and the expected execution time of tasks during training, evaluates the execution efficiency of each task, and determines the task optimization plan. The resource scheduling optimization module, in conjunction with the GPU computing power resource allocation plan and the task optimization plan, adjusts the allocation of GPU memory and computing power resources, and generates GPU resource scheduling optimization results.

Images