A GPU cluster data sharing method for AI model training

Abstract

The present invention relates to the field of data processing technology, specifically a GPU cluster data sharing method for AI model training, comprising the following steps: optimizing the real-time monitoring of gradient information according to the gradient information processed by GPU nodes, adjusting the distribution of data in GPU cluster data sharing between GPU nodes, adjusting the data flow direction, optimizing resource allocation, and generating data flow optimization strategies. In the present invention, by real-time monitoring and adjustment of the gradient information processed by GPU nodes, the efficiency of GPU cluster data sharing and the speed of AI model training are significantly improved, real-time monitoring allows dynamic adjustment of data distribution, effective use of computing resources, reduction of redundancy and processing time, data flow and resource allocation optimization make data transmission more uniform in a multi-GPU environment, reduce the transmission bottleneck between nodes, and enhance the parallel processing capability of the training process. This fine data management supports larger-scale data set processing and improves model accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a GPU cluster data sharing method for AI model training. Background Art

[0002] The field of data processing technology involves methods and tools for collecting, verifying, storing, sorting, converting, analyzing and visualizing data, aiming to extract useful information from large amounts of data and achieve decision support. This field covers everything from simple data management to complex data science applications, including database management, big data analysis, data mining and machine learning. In AI and machine learning projects, high-quality, well-processed data sets are a prerequisite for training effective models. With the development of technology, data processing technology is increasingly able to support real-time data stream processing and advanced analysis, helping enterprises and organizations make fast and accurate decisions in an increasingly competitive environment.

[0003] Among them, the GPU cluster data sharing method involves how to efficiently share and manage data to train artificial intelligence models in a multi-GPU environment. The core of the topic is to improve data access efficiency and optimize resource usage to accelerate the training process of AI models, including reducing model training time, improving the work efficiency of multi-GPU clusters, and ensuring balanced distribution of data among differentiated GPUs. Through effective data sharing strategies, larger-scale data set processing can be supported, thereby training more accurate AI models, which is particularly important for application scenarios that require processing large amounts of data and complex calculations, such as image recognition, natural language processing, and large-scale prediction models.

[0004] Although existing data processing technologies are widely used in multiple fields, their efficiency and response speed still face challenges in processing large-scale, complex computing AI training projects. In a multi-GPU cluster environment, the distribution of data between differentiated GPUs is not flexible enough, resulting in some nodes being overloaded and some nodes being idle. This uneven resource utilization reduces overall work efficiency. Traditional technologies lack effective real-time data stream processing capabilities and cannot quickly respond to dynamic changes in the training process, which limits their practicality in large-scale prediction models and complex application scenarios. The lack of a flexible data synchronization mechanism also leads to data consistency issues, further affecting the reliability and accuracy of training results. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and propose a GPU cluster data sharing method for AI model training.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a GPU cluster data sharing method for AI model training, comprising the following steps:

[0007] S1: By real-time monitoring of the gradient information processed by GPU nodes, analyzing the current load and data requirements of each node, adjusting the data distribution in GPU cluster data sharing, evaluating the data flow between nodes, dynamically adjusting the data flow direction according to the gradient load changes, allocating resource loads, optimizing resource configuration, and generating data flow optimization strategies;

[0008] S2: using the data flow optimization strategy, evaluating the criticality of key data in real time, sorting priority data, analyzing the load balance of data among GPU nodes, adjusting the distribution of key data among GPU nodes, obtaining an optimized data distribution plan, and based on the optimized data distribution plan, reallocating node tasks and reducing delays, generating task reallocation results;

[0009] S3: using the task reallocation result, monitoring the impact of the current data distribution on the gradient update speed, adjusting the gradient synchronization frequency according to the gradient load between nodes, analyzing the bandwidth utilization between nodes, optimizing the gradient synchronization interval, and obtaining the gradient synchronization strategy; analyzing the resource requirements of the nodes during the gradient update process according to the gradient synchronization strategy, dynamically adjusting the virtual GPU resource allocation in the GPU cluster data sharing environment, optimizing the resource utilization during the training process, and generating the virtual resource configuration result;

[0010] S4: Based on the virtual resource configuration results, evaluate the resource load in the GPU cluster, dynamically adjust GPU node allocation according to model training requirements, optimize the computing load and communication requirements of each node, analyze the load balancing of resource distribution in the parallel training process, and dynamically adjust resource scheduling to obtain a resource scheduling optimization plan. Through the resource scheduling optimization plan, optimize the parallel processing capability and efficiency of model training, and generate parallel processing performance enhancement results for the training process.

[0011] As a further solution of the present invention, the step of obtaining the data flow optimization strategy is specifically as follows:

[0012] S111: Based on the gradient information processed by the GPU node, evaluate the gradient change of each node, establish a gradient information monitoring model based on the change trend, and generate a real-time monitoring indicator of the gradient information in combination with the node data;

[0013] S112: Monitor indicators in real time according to the gradient information, analyze the data load of nodes in the GPU cluster, calculate the data imbalance index, and generate data imbalance adjustment requirements through data analysis;

[0014] S113: According to the data imbalance adjustment requirement, the distribution and flow of data between GPU nodes are adjusted, using the formula:

[0015]

[0016] Adjust the data flow, optimize resource allocation, and generate data flow optimization strategies, including: Represents the amount of data transmitted between nodes. Represents the node computing power, Represents the current load of the node, Represents a data flow optimization function.

[0017] As a further solution of the present invention, the steps of obtaining the optimized data distribution solution are specifically as follows:

[0018] S211: using the data flow optimization strategy to monitor the data transmission and processing status between GPU nodes in real time, capturing the operation data of the nodes including the transmission speed and delay time through the network performance monitoring sensor, and generating the node data performance results;

[0019] S212: Based on the node data performance results, the optimization algorithm data items are used to perform criticality rating, using the formula:

[0020]

[0021] The data priority list is calculated, where: Indicates the urgency of the data item, Indicates the size of the data item, and is the adjustment factor, Indicates the data priority score;

[0022] S213: According to the data priority list, adjust the data distribution between GPU nodes, combine the real-time network status and node capabilities, reconfigure the data distribution, and generate an optimized data distribution plan.

[0023] As a further solution of the present invention, the step of obtaining the task reallocation result is specifically as follows:

[0024] S221: by using the optimized data distribution scheme, combined with analyzing the current load and response time of the GPU node, using the monitoring configuration to capture the data processing speed and delay of the node in real time, and extracting performance indicators to generate node performance analysis results;

[0025] S222: According to the node performance analysis results, the processing capacity and response time of multiple nodes are evaluated using the formula:

[0026]

[0027] Generate a resource optimization strategy, where is the node data processing capability, is the average node response time, and are the adjustment coefficients for performance and response time, respectively. Indicates resource optimization strategy;

[0028] S223: Use the resource optimization strategy to implement task reallocation, adjust the distribution of tasks in the GPU cluster, optimize task allocation and reduce latency, determine the optimal state of operation through cyclic dynamic adjustment, and generate task reallocation results.

[0029] As a further solution of the present invention, the steps of obtaining the gradient synchronization strategy are specifically as follows:

[0030] S311: Evaluate the impact of the task reallocation result on the gradient update speed, capture the gradient processing data through the cluster monitoring configuration, calculate the gradient update speed of each node and the data distribution before and after the gradient processing, and obtain the data distribution impact evaluation result;

[0031] S312: According to the data distribution impact assessment result, the data transmission delay and processing capacity difference between nodes are calculated using the gradient update speed and node performance index table, using the formula:

[0032]

[0033] Get the gradient update speed adjustment coefficient, where and is the coefficient adjusted according to the original data, is the gradient update speed, is the transmission delay, Represents the gradient update speed adjustment coefficient;

[0034] S313: adjusting the gradient synchronization frequency in combination with the gradient update speed adjustment coefficient and the data transmission strategy to obtain a gradient synchronization strategy.

[0035] As a further solution of the present invention, the step of obtaining the virtual resource configuration result is specifically:

[0036] S321: Based on the gradient synchronization strategy, evaluate the data processing capabilities and resource usage of multiple GPU units, perform performance and resource consumption analysis, and generate performance evaluation results;

[0037] S322: According to the performance evaluation result, recalculate the resource configuration of the virtual GPU, optimize the resource utilization efficiency, and adopt the resource allocation formula:

[0038]

[0039] Generate an optimized resource allocation proposal, where: Represents the resource adjustment indicator, is the current virtual resource amount of the virtual GPU, represents the efficiency evaluation index, The processing power standard of GPU. represents the resource delay adjustment coefficient between nodes, represents the unit load factor;

[0040] S323: According to the optimized resource configuration proposal, implement dynamic resource management to reconfigure resources, adjust resource allocation of each virtual GPU, monitor the adjusted resource utilization and performance changes, and generate virtual resource configuration results.

[0041] As a further solution of the present invention, the steps of obtaining the resource scheduling optimization solution are specifically as follows:

[0042] S411: Based on the virtual resource configuration result, analyze the GPU usage rate and model training requirements, including the operation efficiency, memory status and processing speed of each node, and generate a resource performance evaluation result;

[0043] S412: Using the resource performance evaluation result, apply a resource optimization formula to dynamically allocate GPU resources, the formula is:

[0044]

[0045] Balance node load and response time, optimize overall operation efficiency, and obtain adjusted resource allocation ratios, including: Represents the adjusted resource allocation ratio, is the data demand, Indicates priority, To calculate the demand, is the amount of available memory;

[0046] S413: According to the adjusted resource allocation ratio, monitor the performance of the nodes before and after the adjustment, record the difference in performance indicators before and after the resource adjustment, analyze the performance change trend of the data, and build a resource scheduling optimization plan.

[0047] As a further solution of the present invention, the steps for obtaining the results of the parallel processing efficiency enhancement during the training process are specifically as follows:

[0048] S421: According to the resource scheduling optimization solution, real-time GPU usage data is captured, including the current task load and resource usage rate of each node, and the data is recorded to generate a resource usage snapshot;

[0049] S422: Based on the resource usage snapshot, perform resource allocation optimization using the formula:

[0050]

[0051] Calculate the new resource allocation ratio and generate an adjusted resource allocation plan, where: Representation Node The new resource allocation result is It's a task At the node The current resource usage of is the task priority, is the data dependency delay of the task, is the total node resources, Indicates the total task volume;

[0052] S423: Apply the adjusted resource allocation scheme, reconfigure GPU resources, monitor the adjusted task efficiency and parallel processing performance, configure new resource parameters, start the model training task, monitor resource utilization and task execution, and obtain the results of the parallel processing efficiency enhancement during the training process.

[0053] Compared with the prior art, the advantages and positive effects of the present invention are:

[0054] In the present invention, by real-time monitoring and adjustment of the gradient information processed by the GPU nodes, the efficiency of GPU cluster data sharing and the speed of AI model training are significantly improved. By real-time monitoring of the gradient information, the distribution of data among the GPU nodes is dynamically adjusted, thereby more effectively utilizing computing resources and reducing redundant operations and data processing time. The optimized data flow and resource allocation make data transmission in a multi-GPU environment more uniform, reduce the data transmission bottleneck between nodes, and enhance the parallel processing capability of the entire training process. Through sophisticated data management, the solution can support larger-scale data set processing, thereby improving the accuracy of the model and training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the main steps of the present invention;

[0056] Figure 2 It is a flow chart of the data flow optimization strategy in the present invention;

[0057] Figure 3 A flow chart of the optimized data distribution scheme in the present invention;

[0058] Figure 4 A flowchart of the task reallocation result in the present invention;

[0059] Figure 5 It is a flow chart of the gradient synchronization strategy in the present invention;

[0060] Figure 6A flowchart of the virtual resource configuration result in the present invention;

[0061] Figure 7 It is a flow chart of the resource scheduling optimization solution in the present invention;

[0062] Figure 8 Flow chart of the performance enhancement results of parallel processing of the training process in the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined. Example

[0065] See also Figure 1 The present invention provides a technical solution: a GPU cluster data sharing method for AI model training, comprising the following steps:

[0066] S1: By real-time monitoring of the gradient information processed by GPU nodes, analyzing the current load and data requirements of each node, adjusting the data distribution in GPU cluster data sharing, evaluating the data flow between nodes, dynamically adjusting the data flow direction according to the gradient load changes, allocating resource loads, optimizing resource configuration, and generating data flow optimization strategies;

[0067] S2: using the data flow optimization strategy, evaluating the criticality of key data in real time, sorting priority data, analyzing the load balance of data among GPU nodes, adjusting the distribution of key data among GPU nodes, obtaining an optimized data distribution plan, and based on the optimized data distribution plan, reallocating node tasks and reducing delays, generating task reallocation results;

[0068] S3: using the task reallocation result, monitoring the impact of the current data distribution on the gradient update speed, adjusting the gradient synchronization frequency according to the gradient load between nodes, analyzing the bandwidth utilization between nodes, optimizing the gradient synchronization interval, and obtaining the gradient synchronization strategy; analyzing the resource requirements of the nodes during the gradient update process according to the gradient synchronization strategy, dynamically adjusting the virtual GPU resource allocation in the GPU cluster data sharing environment, optimizing the resource utilization during the training process, and generating the virtual resource configuration result;

[0069] S4: Based on the virtual resource configuration results, evaluate the resource load in the GPU cluster, dynamically adjust GPU node allocation according to model training requirements, optimize the computing load and communication requirements of each node, analyze the load balancing of resource distribution in the parallel training process, and dynamically adjust resource scheduling to obtain a resource scheduling optimization plan. Through the resource scheduling optimization plan, optimize the parallel processing capability and efficiency of model training, and generate parallel processing performance enhancement results for the training process.

[0070] The data flow optimization strategy includes resource scheduling details and data transmission optimization. The optimized data distribution scheme includes data locality enhancement, load balancing improvement, and communication efficiency improvement. The task redistribution results include response time optimization, task processing efficiency, and delay reduction indicators. The gradient synchronization strategy includes synchronization interval optimization, data integrity assurance, and update frequency adjustment. The virtual resource configuration results include virtualization degree, resource partitioning strategy, and load management adjustment. The resource scheduling optimization scheme includes scheduling logic optimization, execution resource configuration, and queuing mechanism improvement. The parallel processing performance enhancement results of the training process include processing parallelism, speed indicators, and performance improvement rate.

[0071] See also Figure 2 , the specific steps for obtaining the data flow optimization strategy are:

[0072] S111: Based on the gradient information processed by the GPU node, evaluate the gradient change of each node, establish a gradient information monitoring model based on the change trend, and generate a real-time monitoring indicator of the gradient information in combination with the node data;

[0073] The implementation of gradient information monitoring relies on accurately measuring and analyzing the gradient information processed by GPU nodes. This requires first collecting the computing data of each GPU node in real time through sensors and monitoring systems, and then using data analysis algorithms to identify key gradient change trends in the data, quantify the real-time changes in gradient information, and input the quantified data into the gradient monitoring model. The model compares real-time data with historical data patterns to evaluate the dynamic trend of gradient changes, thereby providing a basis for adjusting the data processing and task allocation strategies of GPU nodes. All of this is done automatically through advanced data processing units and algorithms, ensuring real-time updates of data and accurate reflection of the operating status of nodes, and generating real-time monitoring indicators for gradient information.

[0074] S112: monitor indicators in real time according to gradient information, analyze data load conditions of nodes in the GPU cluster, calculate data imbalance index, and generate data imbalance adjustment requirements through data analysis;

[0075] Based on the real-time monitoring indicators of gradient information, the uniformity of data distribution is further analyzed. By monitoring the data processing capacity and current task volume of each node, the data load difference of each node is calculated using statistical methods. The calculation is based on real-time monitoring data and historical performance data, and the efficiency and fairness of data distribution are comprehensively evaluated. Through analysis, nodes with higher loads and nodes where resources are not fully utilized are identified, and adjustment strategies are proposed, including data reallocation and task rescheduling, to reduce delays and increase processing speed. The entire process is carried out dynamically, and the strategy is adjusted according to real-time data to ensure the optimality of system operation and generate data imbalance adjustment needs.

[0076] S113: To adjust the demand for data imbalance, adjust the distribution and flow of data between GPU nodes, using the formula:

[0077]

[0078] Adjust the data flow, optimize resource allocation, and generate data flow optimization strategies, including: Represents the amount of data transmitted between nodes. Represents the node computing power, Represents the current load of the node, represents a data flow optimization function;

[0079] In this step, and Represent the amount of data transmission between nodes, node computing power and node current load respectively. Formula It is designed to optimize the data flow, by adjusting and The value of can effectively balance the load between nodes and optimize resource allocation;

[0080] Set the amount of data transfer between nodes GB, the computing power of the node GFLOPS, the current load of the node ;

[0081] Substitute into the formula: ;

[0082] The results show that by adjusting the data transmission volume and computing power, effective load balancing can be achieved, thereby optimizing the data flow and resource allocation of the entire GPU cluster and generating a data flow optimization strategy.

[0083] See also Figure 3 , the specific steps for obtaining the optimized data distribution plan are:

[0084] S211: Using data flow optimization strategies, real-time monitoring of data transmission and processing status between GPU nodes, capturing node operation data including transmission speed and delay time through network performance monitoring sensors, and generating node data performance results;

[0085] Detailed node performance monitoring is performed using data flow optimization strategies. High-performance sensors deployed on GPU nodes capture data transmission speed and processing delay in real time. The sensors can provide accurate operating data. The data is processed by analysis software to determine the performance status of each node, thereby providing decision support for data allocation. This process ensures the real-time and accuracy of monitoring data and generates node data performance results. The report records the operating efficiency and potential bottlenecks of each GPU node in detail, which is the basis for adjusting data allocation between GPU nodes.

[0086] S212: Based on the node data performance results, apply the optimization algorithm data items to perform criticality rating using the formula:

[0087]

[0088] The data priority list is calculated, where: Indicates the urgency of the data item, Indicates the size of the data item, and is the adjustment factor, Indicates the data priority score;

[0089] The formula is used to calculate the priority of the data, where, Represents the urgency of the data item, defined by the latency of the data request, Represents the size of the data item, obtained directly from the metadata of the data packet. and The weight coefficients are used to adjust the influence of urgency and size in the priority assessment. The coefficients are determined based on historical data analysis to ensure that key data is processed first.

[0090] set up ms delay, MB data size, weight , , the calculation process is:

[0091]

[0092] This result shows that the priority score of this data item is 0.303, and the lower score reflects that despite the large amount of data, the lower urgency reduces its priority.

[0093] S213: adjusting data distribution among GPU nodes according to the data priority list, reconfiguring data distribution in combination with real-time network status and node capabilities, and generating an optimized data distribution plan;

[0094] According to the data priority list, the data distribution is adjusted in combination with the real-time network status and node capabilities. High-priority data is processed first, and the dynamic data routing algorithm is used to achieve effective distribution of data among nodes. The data distribution model takes into account the bandwidth, latency and processing capacity among nodes, and achieves a rapid response to high-priority data. Ultimately, an efficient data distribution scheme is formed. This scheme can be dynamically optimized according to network conditions and node performance, thereby ensuring data processing efficiency while reducing communication overhead between nodes and generating an optimized data distribution scheme.

[0095] See also Figure 4 , the specific steps for obtaining the task redistribution results are:

[0096] S221: Through the optimized data distribution scheme, combined with the analysis of the current load and response time of the GPU node, the monitoring configuration is used to capture the data processing speed and latency of the node in real time, and the performance indicators are extracted to generate the node performance analysis results;

[0097] Through the execution of the data flow optimization solution, an in-depth analysis of the current tasks and data processing delays of the GPU nodes is achieved. High-performance monitoring tools are used to capture the data processing speed and delay of each node in real time, extract key performance indicators from the system log, and generate node performance analysis results. The report records the operating efficiency and potential bottlenecks of each GPU node in detail, providing accurate data support for subsequent task allocation and optimization, thereby ensuring the targetedness and effectiveness of the optimization measures.

[0098] S222: Based on the node performance analysis results, the processing capacity and response time of multiple nodes are evaluated using the formula:

[0099]

[0100] Generate a resource optimization strategy, where is the node data processing capability, is the average node response time, and are the adjustment coefficients for performance and response time, respectively. Indicates resource optimization strategy;

[0101] In the formula, Indicates the data processing capacity of a node. For example, a node can process 1,000 tasks per second. Indicates the average response time of the node. For example, the average response time is 0.1 seconds. and It is an adjustment factor set based on historical performance data. and , substituting the value into the formula to calculate:

[0102]

[0103] This shows that the task reallocation strategy is of medium priority with reference to the processing capacity and response time of the nodes, which helps to balance the equipment load and optimize the performance.

[0104] S223: Use resource optimization strategies to implement task reallocation, adjust the distribution of tasks in the GPU cluster, optimize task allocation and reduce latency, determine the optimal state of operation through cyclic dynamic adjustment, and generate task reallocation results;

[0105] Based on the new resource optimization strategy, the distribution of tasks in the GPU cluster was actually adjusted to optimize the workload between nodes and reduce the overall delay. Through the implementation of efficient task scheduling strategies, the rapid processing of high-priority tasks was ensured. A series of dynamic adjustments improved the operating efficiency. The final task redistribution results effectively increased the processing speed and reduced the response time, thereby achieving overall performance optimization.

[0106] See also Figure 5 , the specific steps for obtaining the gradient synchronization strategy are:

[0107] S311: Evaluate the impact of task redistribution results on gradient update speed, capture gradient processing data through cluster monitoring configuration, calculate the gradient update speed of each node and the data distribution before and after gradient processing, and obtain data distribution impact evaluation results;

[0108] Based on the existing data distribution, the gradient update speed of each node was evaluated in detail. The performance indicators and processing data of each node were collected to evaluate the impact of data distribution on the gradient update speed. By analyzing the performance indicators, the data synchronization requirements between nodes were determined, and basic data was provided for further strategy adjustments.

[0109] S312: According to the data distribution impact assessment results, the data transmission delay and processing capacity difference between nodes are calculated using the gradient update speed and node performance index table, using the formula:

[0110]

[0111] Get the gradient update speed adjustment coefficient, where and is the coefficient adjusted according to the original data, is the gradient update speed, is the transmission delay, Represents the gradient update speed adjustment coefficient;

[0112] In this formula, and are adjustment coefficients based on historical data, affecting the gradient update speed respectively and transmission delay The weight of , through this formula, the gradient update speed adjustment coefficient of each node can be calculated, so as to optimize the data transmission strategy between nodes. This formula helps determine the optimal gradient synchronization frequency between nodes, thereby reducing the overall calculation delay and improving processing efficiency. Through specific data input, such as Updated every millisecond. milliseconds transmission delay, set and ,calculate :

[0113]

[0114] This shows that the node's gradient update speed is adjusted by a factor of 0.5, which helps determine the data synchronization strategy between nodes.

[0115] S313: adjusting the gradient synchronization frequency in combination with the gradient update speed adjustment coefficient and the data transmission strategy to obtain a gradient synchronization strategy;

[0116] The node performance indicators and gradient update speed adjustment coefficients are used to further optimize the gradient synchronization frequency between nodes. Combined with the node processing capabilities and network conditions, real-time monitoring is used to ensure that all nodes can synchronize data at the optimal frequency. An efficient gradient synchronization strategy is determined and implemented, which effectively improves the computing efficiency of the entire cluster and reduces latency.

[0117] See also Figure 6 , the specific steps for obtaining the virtual resource configuration results are:

[0118] S321: Based on the gradient synchronization strategy, evaluate the data processing capabilities and resource usage of multiple GPU units, perform performance and resource consumption analysis, and generate performance evaluation results;

[0119] Based on the existing gradient synchronization strategy, we first evaluate the data processing capabilities and resource usage of each GPU unit. Through performance monitoring tools, we collect real-time performance data of each node, including processing speed, memory usage and energy consumption. Then we analyze the data to identify performance bottlenecks and resource waste areas. According to the analysis results, we formulate specific resource optimization measures, including adjusting GPU power allocation, memory usage and processing priority to achieve optimal resource utilization. Finally, we generate performance evaluation results, which detail the performance of each unit and improvement suggestions, providing a scientific basis for the next step of resource allocation.

[0120] S322: According to the performance evaluation result, recalculate the resource configuration of the virtual GPU, optimize the resource utilization efficiency, and use the resource allocation formula:

[0121]

[0122] Generate an optimized resource allocation proposal, where: Represents the resource adjustment indicator, is the current virtual resource amount of the virtual GPU, represents the efficiency evaluation index, The processing power standard of GPU. represents the resource delay adjustment coefficient between nodes, represents the unit load factor;

[0123] The benefit of the formula is that it realizes a detailed analysis of GPU resource allocation by combining parameters such as virtual resource volume, efficiency evaluation, load factor, and delay adjustment factor, so that the load and resource consumption of each node can be more comprehensively reflected when calculating resource allocation, thereby improving the rationality of resource allocation;

[0124] Represents the current amount of virtual GPU resources, which is obtained by monitoring the resource size allocated to each node and the virtual resource consumption. For example, the value is 800;

[0125] It is an efficiency evaluation indicator, which is obtained by evaluating the task completion rate and resource consumption of the node within a specific time. For example, the value is 1.2;

[0126] is the load factor, which indicates the computing capacity of the node under the current load. It is obtained through statistics of the node load and operating status. For example, the value is 1.1.

[0127] Indicates the processing capacity standard of the GPU, which is used as the benchmark of the node processing capacity. It is normalized and calculated through experimental detection and resource utilization. For example, the value is 500.

[0128] is the resource delay adjustment coefficient between nodes, which is measured based on the network delay and communication efficiency between nodes, for example, the value is 0.05;

[0129] Substitute into the formula:

[0130] Step by step calculation:

[0131] Calculate the square root of the load factor: ;

[0132] Calculate the numerator part: ;

[0133] Calculate the denominator: ;

[0134] Calculate the overall formula: ;

[0135] The results show that resource adjustment indicators , used to further guide resource allocation.

[0136] S323: According to the optimized resource configuration proposal, implement dynamic resource management to reconfigure resources, adjust resource allocation of each virtual GPU, monitor the adjusted resource utilization and performance changes, and generate virtual resource configuration results;

[0137] Based on the resource allocation proposal, dynamic resource management tools are used to implement GPU resource reallocation. Monitoring tools track the effects of resource adjustments in real time to ensure that each GPU node runs according to the new configuration. During the detailed execution process, the performance data of each node before and after the adjustment is recorded in detail, including changes in processing speed, shortened task completion time, and reduced energy consumption. By comparing the data before and after the adjustment, the actual effect of the resource allocation proposal is verified, and finally the effectiveness of the virtual resource configuration results is verified. This result confirms the correctness of the adjustment strategy, ensures the maximum utilization of resources, and improves the efficiency of the overall training process, indicating that the optimization measures have achieved the expected success.

[0138] See also Figure 7 , the specific steps for obtaining the resource scheduling optimization plan are:

[0139] S411: Based on the virtual resource configuration results, analyze the GPU usage and model training requirements, including the operation efficiency, memory status and processing speed of each node, and generate resource performance evaluation results;

[0140] Based on comprehensive data analysis of virtual resource configuration results, the current usage and potential resource bottlenecks of each GPU node are determined. By integrating data on the node's operating efficiency, memory status, and processing speed, the resource performance evaluation results are obtained. This result not only shows the node's immediate performance but also predicts potential resource bottlenecks, providing a scientific basis for the next step of resource scheduling optimization.

[0141] S412: Using the resource performance evaluation result, apply a resource optimization formula to dynamically allocate GPU resources. The formula is:

[0142]

[0143] Balance node load and response time, optimize overall operation efficiency, and obtain adjusted resource allocation ratios, including: Represents the adjusted resource allocation ratio, is the data demand, Indicates priority, To calculate the demand, is the amount of available memory;

[0144] This formula is used to calculate the resource allocation ratio ,in, Indicates the data demand, obtained from the real-time task queue;

[0145] Represents the task priority, which is dynamically adjusted according to the urgency of the task;

[0146] To calculate demand, real-time measurements are made through monitoring tools;

[0147] Indicates the amount of available memory, obtained in real time from hardware monitoring;

[0148] By substituting actual numbers into the formula, e.g. GB, , GHz, GB, then calculate :

[0149]

[0150] The result shows that according to the current resource requirements and priorities, the resource allocation ratio of the node is about 9.8%, indicating that the resource configuration of the node needs to be appropriately improved to meet the computing needs.

[0151] S413: According to the adjusted resource allocation ratio, monitor the performance of the node before and after the adjustment, record the difference in performance indicators before and after the resource adjustment, analyze the performance change trend of the data, and build a resource scheduling optimization plan;

[0152] In the actual operation of implementing the resource allocation plan, the performance changes of each node are continuously monitored to ensure that each node operates optimally according to the new resource allocation strategy, and finally a resource scheduling optimization plan report is compiled. This report records in detail the performance improvement of each node under the new strategy and the optimization of resource utilization, ensuring that the resource scheduling of the overall GPU cluster reaches the optimal state.

[0153] See also Figure 8 , the specific steps for obtaining the results of parallel processing performance enhancement during training are:

[0154] S421: according to the resource scheduling optimization solution, capture real-time GPU usage data, including the current task load and resource usage rate of each node, and record the data to generate a resource usage snapshot;

[0155] The real-time monitoring system collects real-time data of GPU nodes, including the processing power of each node, the current task load, and resource utilization. The performance fluctuations and resource allocation status of each node are analyzed from the monitoring data. The work efficiency of each node is evaluated based on the collected data, and a resource usage snapshot is generated. The snapshot will provide decision support for subsequent resource optimization and adjustment, ensuring real-time update and accuracy of data.

[0156] S422: Optimize resource allocation based on the resource usage snapshot using the formula:

[0157]

[0158] Calculate the new resource allocation ratio and generate an adjusted resource allocation plan, where: Representation Node The new resource allocation result is It's a task At the node The current resource usage of is the task priority, is the data dependency delay of the task, is the total node resources, Indicates the total task volume;

[0159] This formula is used to calculate the new resource allocation ratio, where Representative Node The new resource allocation result is calculated in the formula;

[0160] Representation Task At the node Current resource usage of

[0161] It is the task priority, indicating the importance of the priority, which is obtained through system monitoring;

[0162] Representation Task The data-dependent delay is the sum of network delay and data processing time;

[0163] For Node The total resources are read from the configuration file;

[0164] set up , , , , then the calculation process is:

[0165]

[0166] The result shows that the new resource allocation ratio is 0.12, which means that the node The resource allocation ratio has been adjusted to 12% to increase the accuracy and efficiency of resource allocation.

[0167] S423: Apply the adjusted resource allocation plan, reconfigure GPU resources, monitor the adjusted task efficiency and parallel processing performance, configure new resource parameters, start the model training task, monitor resource utilization and task execution, and obtain the parallel processing performance enhancement result of the training process;

[0168] According to the new resource allocation strategy, the resource allocation of each GPU in the cluster is reconfigured, the effect of resource adjustment is verified through actual model training tasks, the speed of task execution and resource utilization are monitored, and the improvements before and after the adjustment are compared to ensure that each task can obtain the necessary resources to maximize its performance. The ultimate parallel processing performance enhancement result will significantly improve the work efficiency and output quality of the entire cluster. Through adjustment and optimization, the use of resources is more rational, which further promotes the efficient execution of data processing and computing tasks.

[0169] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A GPU cluster data sharing method for AI model training, characterized in that: The following steps are involved: By monitoring the gradient information processed by GPU nodes in real time, analyzing the current load and data requirements of each node, adjusting the data distribution in GPU cluster data sharing, evaluating the data flow between nodes, dynamically adjusting the data flow direction according to the gradient load changes, allocating resource loads, optimizing resource configuration, and generating data flow optimization strategies; By using the data flow optimization strategy, the criticality of key data is evaluated in real time, priority data is sorted, the load balance of data between GPU nodes is analyzed, the distribution of key data between GPU nodes is adjusted, and an optimized data distribution plan is obtained. Based on the optimized data distribution plan, node tasks are reallocated and delays are reduced, and task reallocation results are generated; Using the task reallocation result, monitor the impact of the current data distribution on the gradient update speed, adjust the gradient synchronization frequency according to the gradient load between nodes, analyze the bandwidth utilization between nodes, optimize the gradient synchronization interval, and obtain the gradient synchronization strategy. According to the gradient synchronization strategy, analyze the resource requirements of the nodes during the gradient update process, dynamically adjust the virtual GPU resource allocation in the GPU cluster data sharing environment, optimize the resource utilization during the training process, and generate the virtual resource configuration result; Based on the virtual resource configuration results, the resource load in the GPU cluster is evaluated, the GPU node allocation is dynamically adjusted according to the model training requirements, the computing load and communication requirements of each node are optimized, the load balancing of resource distribution in the parallel training process is analyzed, and the resource scheduling is dynamically adjusted to obtain a resource scheduling optimization plan. Through the resource scheduling optimization plan, the parallel processing capability and efficiency of model training are optimized, and the parallel processing performance enhancement results of the training process are generated.

2. The GPU cluster data sharing method for AI model training according to claim 1, characterized in that: The steps for obtaining the data flow optimization strategy are specifically as follows: Based on the gradient information processed by the GPU node, evaluate the gradient change of each node, establish a gradient information monitoring model based on the change trend, and generate real-time monitoring indicators for gradient information in combination with node data; Monitor indicators in real time based on the gradient information, analyze the data load of nodes in the GPU cluster, calculate the data imbalance index, and generate data imbalance adjustment requirements through data analysis; In order to adjust the imbalanced data, the distribution and flow of data between GPU nodes are adjusted using the formula: ； Adjust the data flow, optimize resource allocation, and generate data flow optimization strategies, including: Represents the amount of data transmitted between nodes. Represents the node computing power, Represents the current load of the node, Represents a data flow optimization function.

3. The GPU cluster data sharing method for AI model training according to claim 2, characterized in that: The steps for obtaining the optimized data distribution solution are specifically as follows: Using the data flow optimization strategy, the data transmission and processing status between GPU nodes is monitored in real time, and the operation data of the nodes, including the transmission speed and delay time, are captured through the network performance monitoring sensor to generate the node data performance results; Based on the node data performance results, the optimization algorithm data items are used to perform criticality rating using the formula: ； The data priority list is calculated, where: Indicates the urgency of the data item, Indicates the size of the data item, and is the adjustment factor, Indicates the data priority score; According to the data priority list, the data distribution between GPU nodes is adjusted, and the data distribution is reconfigured in combination with the real-time network status and node capabilities to generate an optimized data distribution plan.

4. The GPU cluster data sharing method for AI model training according to claim 3 is characterized in that: The steps for obtaining the task reallocation result are specifically as follows: By using the optimized data distribution scheme, combined with analyzing the current load and response time of the GPU node, the monitoring configuration is used to capture the data processing speed and delay of the node in real time, and the performance indicators are extracted to generate the node performance analysis results; According to the node performance analysis results, the processing capacity and response time of multiple nodes are evaluated using the formula: ； Generate a resource optimization strategy, where is the node data processing capability, is the average node response time, and are the adjustment coefficients for performance and response time, respectively. Indicates resource optimization strategy; The resource optimization strategy is used to implement task reallocation, adjust the distribution of tasks in the GPU cluster, optimize task allocation and reduce latency, and through cyclic dynamic adjustment, determine the optimal state of operation and generate task reallocation results.

5. The GPU cluster data sharing method for AI model training according to claim 4, characterized in that: The steps for obtaining the gradient synchronization strategy are specifically as follows: Evaluate the impact of the task redistribution result on the gradient update speed, capture the gradient processing data through the cluster monitoring configuration, calculate the gradient update speed of each node and the data distribution before and after the gradient processing, and obtain the data distribution impact evaluation result; According to the data distribution impact assessment results, the data transmission delay and processing capacity difference between nodes are calculated using the gradient update speed and node performance index table, using the formula: ； Get the gradient update speed adjustment coefficient, where and is the coefficient adjusted according to the original data, is the gradient update speed, is the transmission delay, Represents the gradient update speed adjustment coefficient; The gradient update speed adjustment coefficient and the data transmission strategy are combined to adjust the gradient synchronization frequency and obtain the gradient synchronization strategy.

6. The GPU cluster data sharing method for AI model training according to claim 5, characterized in that: The steps for obtaining the virtual resource configuration result are specifically as follows: Based on the gradient synchronization strategy, the data processing capabilities and resource usage of multiple GPU units are evaluated, performance and resource consumption analysis is performed, and performance evaluation results are generated; According to the performance evaluation results, the resource configuration of the virtual GPU is recalculated to optimize resource utilization efficiency, and the resource allocation formula is adopted: ； Generate an optimized resource allocation proposal, where: Represents the resource adjustment indicator, is the current virtual resource amount of the virtual GPU, represents the efficiency evaluation index, The processing power standard of GPU. represents the resource delay adjustment coefficient between nodes, represents the unit load factor; According to the optimized resource configuration proposal, dynamic resource management is implemented to reconfigure resources, adjust the resource allocation of each virtual GPU, monitor the adjusted resource utilization and performance changes, and generate virtual resource configuration results.

7. The GPU cluster data sharing method for AI model training according to claim 6, characterized in that: The steps for obtaining the resource scheduling optimization solution are specifically as follows: Based on the virtual resource configuration results, analyze GPU usage and model training requirements, including the operating efficiency, memory status and processing speed of each node, and generate resource performance evaluation results; Using the resource performance evaluation results, a resource optimization formula is applied to dynamically allocate GPU resources. The formula is: ； Balance node load and response time, optimize overall operation efficiency, and obtain adjusted resource allocation ratios, including: Represents the adjusted resource allocation ratio, is the data demand, Indicates priority, To calculate the demand, is the amount of available memory; According to the adjusted resource allocation ratio, the performance of the nodes before and after the adjustment is monitored, the difference in performance indicators before and after the resource adjustment is recorded, the performance change trend of the data is analyzed, and a resource scheduling optimization plan is constructed.

8. The GPU cluster data sharing method for AI model training according to claim 7, characterized in that: The steps for obtaining the results of the parallel processing performance enhancement during the training process are specifically as follows: According to the resource scheduling optimization scheme, real-time GPU usage data is captured, including the current task load and resource usage rate of each node, and the data is recorded to generate a resource usage snapshot; Based on the resource usage snapshot, resource allocation optimization is performed using the formula: ； Calculate the new resource allocation ratio and generate an adjusted resource allocation plan, where: Representation Node The new resource allocation result is It's a task At the node The current resource usage of is the task priority, is the data dependency delay of the task, is the total node resources, Indicates the total task volume; Apply the adjusted resource allocation scheme, reconfigure GPU resources, monitor the adjusted task efficiency and parallel processing performance, configure new resource parameters, start the model training task, monitor resource utilization and task execution, and obtain the results of parallel processing efficiency enhancement in the training process.

Images