A method for optimizing GPU programs in a parallel environment

Abstract

The application provides a GPU program optimization method in a parallel environment. A plurality of target models are configured in a GPU cluster for parallel training. The method comprises: performing strategy configuration and memory bottleneck prediction on the parallel training configuration of the target model; using the prediction result, performing throughput optimization through parallel strategy combination and instruction-level performance monitoring; and performing kernel re-optimization on the instruction-level bottleneck occurring in the optimization process, so that the model training and GPU instruction execution efficiency are simultaneously improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for optimizing GPU programs in a parallel environment. Background Technology

[0002] Currently, in the process of training artificial intelligence models using large-scale GPU clusters, efficiently coordinating various parallel strategies such as data parallelism, tensor parallelism, and pipelined parallelism is a key challenge. Existing methods typically rely on empirical rules for parallel configuration, lacking systematic performance analysis and optimization guidance, resulting in the underutilization of GPU hardware computing power and memory bandwidth. Specifically, there is a disconnect between macroscopic parallel strategy decisions and microscopic GPU kernel instruction execution efficiency. This makes the training process highly susceptible to performance degradation due to unreasonable resource allocation or undiscovered instruction-level bottlenecks, affecting overall training throughput and the utilization of computing resources. Summary of the Invention

[0003] This application provides a GPU program optimization method for parallel environments, which combines macro-level parallel strategy configuration with micro-level instruction-level performance analysis to improve GPU operating efficiency.

[0004] In a first aspect, embodiments of this application provide a GPU program optimization method for a parallel environment, wherein multiple target models are configured on a GPU cluster for parallel training, and the method includes:

[0005] The parallel training configuration of the target model is configured with a strategy and memory bottlenecks are predicted.

[0006] Utilizing the prediction results, throughput optimization is achieved through parallel strategy combination and instruction-level performance monitoring;

[0007] Kernel re-optimization was performed to address instruction-level bottlenecks encountered during the optimization process, thereby simultaneously improving the efficiency of model training and GPU instruction execution.

[0008] The strategy configuration and memory bottleneck prediction include: obtaining the parameter scale of the target model, the characteristics of the GPU cluster and the sequence length constraint, generating a memory-feasible parallel configuration set based on the preset performance prediction model, and calculating the memory access efficiency of the parallel configuration set to predict the risk of global memory access bottleneck.

[0009] The parallel strategy combination and instruction-level performance monitoring include: dynamically adjusting the ratio of data parallelism, tensor parallelism and pipeline parallelism under the condition of meeting the preset global batch size, and simultaneously monitoring memory access efficiency and instruction throughput to identify communication bottlenecks and instruction execution bottlenecks.

[0010] The kernel re-optimization of instruction-level bottlenecks includes: reconstructing the shared memory data layout when bank-conflicts are detected in shared memory; and optimizing by increasing thread-level parallelism or instruction-level parallelism when execution dependency bottlenecks are detected in pipeline instructions.

[0011] In some embodiments, the performance prediction model is constructed based on historical benchmark data, using the parameter size, number of nodes, and GPU memory of the target model as input, to predict the theoretical memory usage and communication overhead of different parallel strategy combinations; the formula for calculating the memory access efficiency is:

[0012] ;

[0013] Where β represents the memory access efficiency, B1 represents the actual memory access bandwidth, B2 represents the theoretical peak bandwidth, T1 represents the number of effective memory access transactions, and T2 represents the total number of memory access transactions. and These are weighting coefficients, and .

[0014] In some embodiments, the dynamic adjustment of the ratio of data parallelism, tensor parallelism, and pipelined parallelism includes: when increasing the scale of data parallelism causes the memory access efficiency to fall below a preset threshold, automatically reducing the degree of data parallelism and instead increasing pipelined parallelism or tensor parallelism to rebalance computation and communication.

[0015] In some embodiments, reconstructing the shared memory data layout includes: defining the dimension of the shared memory as the number of warps plus one, and organizing access patterns according to the thread ID in the warp.

[0016] In some embodiments, the number of bank-conflicts access conflicts is obtained through memory transaction statistics in the CUDA program analysis tool.

[0017] In some embodiments, the optimization by increasing thread-level parallelism or instruction-level parallelism includes:

[0018] When matrix operations exist in tensor parallelism, multiple unrelated computational tasks are assigned to independent threads to improve instruction-level parallelism.

[0019] When there are inter-stage dependencies in pipelined parallelism, thread-level parallelism can be improved by increasing the SM occupancy rate.

[0020] In some embodiments, combining the kernel re-optimization with the parallel strategy includes the following steps:

[0021] Determine whether the current configured instruction throughput is lower than the instruction bottleneck threshold;

[0022] If so, the kernel re-optimization process is triggered, and the instruction throughput is updated after optimization;

[0023] If not, proceed directly to the global batch size coordination stage;

[0024] Determine whether the coordinated global batch size has reached the target.

[0025] If not, adjust the gradient accumulation steps and data parallelism ratio, and evaluate the impact on the instruction flow.

[0026] In some embodiments, the evaluation of the impact on the instruction flow includes: analyzing whether the adjustment of the gradient accumulation step number introduces additional branch jumps, and prioritizing implementation schemes that do not introduce additional branch jumps or can be replaced by high-throughput instructions.

[0027] In some embodiments, the kernel re-optimization is used in conjunction with the parallel strategy to optimize shared memory access, eliminate instruction pipeline dependencies, and perform parallel processing of data transfer and computational task allocation in parallel model training, thereby making full use of GPU computing units, storage hierarchy, and interconnect bandwidth resources.

[0028] In some embodiments, the dynamic adjustment of the ratio of data parallelism, tensor parallelism, and pipeline parallelism further includes:

[0029] Based on the real-time monitoring of memory access efficiency and instruction throughput, a dynamic feedback control system is established. When the memory access efficiency continuously falls below a preset efficiency threshold and the instruction throughput decreases synchronously, a parallel strategy reconfiguration is automatically triggered to reallocate the computational load of data parallelism, tensor parallelism, and pipeline parallelism. The constraints of the parallel strategy reconfiguration are as follows:

[0030] Maximize f(TP,PP,DP) = w1×β+ w2×IPC

[0031] TP×PP×DP = Ngpu;

[0032] Mem-U ≤ Mem-A;

[0033] GBS≥GBS-T;

[0034] Where Maximize f(·) is the objective function, TP, PP, and DP represent the parallelism of tensor parallelism, pipeline parallelism, and data parallelism, respectively, Ngpu represents the total number of GPUs, Mem-U represents the model memory usage, Mem-A represents the available video memory, GBS represents the global batch size, GBS-T represents the target global batch size, and w1 and w2 are weight coefficients.

[0035] This application provides a GPU program optimization method in a parallel environment, where multiple target models are configured on a GPU cluster for parallel training. The method includes: configuring strategies and predicting memory bottlenecks in the parallel training configuration of the target models; optimizing throughput through parallel strategy combination and instruction-level performance monitoring using the prediction results; and re-optimizing the kernel for instruction-level bottlenecks encountered during optimization, thereby simultaneously improving model training and GPU instruction execution efficiency. Strategy configuration and memory bottleneck prediction include: obtaining the parameter size of the target models, the characteristics of the GPU cluster, and sequence length constraints; generating a memory-feasible parallel configuration set based on a pre-set performance prediction model; and... The method calculates the memory access efficiency of the parallel configuration set to predict the risk of global memory access bottlenecks; it combines parallel strategies and monitors instruction-level performance, including dynamically adjusting the ratio of data parallelism, tensor parallelism, and pipeline parallelism while meeting the preset global batch size, and simultaneously monitoring memory access efficiency and instruction throughput to identify communication and instruction execution bottlenecks; it performs kernel re-optimization for instruction-level bottlenecks, including reconstructing the shared memory data layout when bank-conflicts are detected in shared memory, and optimizing by increasing thread-level parallelism or instruction-level parallelism when execution dependency bottlenecks are detected in pipeline instructions. By proactively predicting memory bottlenecks and dynamically adjusting the parallel strategy ratio during training, while simultaneously re-optimizing the kernel for detected instruction-level bottlenecks, this method achieves coordinated optimization from the parallel computing dimension to the GPU hardware execution level, improving the utilization of GPU computing and storage resources in complex parallel environments, thereby increasing the overall throughput efficiency of large-scale model training. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A schematic flowchart illustrating a GPU program optimization method in a parallel environment provided in this application embodiment;

[0038] Figure 2 This is a schematic block diagram illustrating a collaborative method combining kernel re-optimization and parallel strategies, provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0041] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0042] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0043] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a GPU program optimization method in a parallel environment provided by an embodiment of this application. Figure 1 As shown, the specific steps of the GPU program optimization method in this parallel environment include: S101-S103.

[0044] S101. Configure strategies and predict memory bottlenecks for the parallel training configuration of the target model. Strategy configuration and memory bottleneck prediction include: obtaining the parameter size of the target model, the characteristics of the GPU cluster and the sequence length constraints; generating a memory-feasible parallel configuration set based on a pre-set performance prediction model; and calculating the memory access efficiency of the parallel configuration set to predict the risk of global memory access bottlenecks.

[0045] For example, the parameter scale of the target model is obtained, including the computation graph structure and operator composition characteristics. Hardware configuration information of the GPU cluster is collected, including the number of nodes, the number of GPUs per node, GPU memory capacity, intra-node NVLink topology, and inter-node network bandwidth. Based on these fundamental parameters, a set of memory-feasible parallel configurations is generated through a pre-defined performance prediction model. This performance prediction model, established by analyzing historical benchmark data, can predict the theoretical memory usage under different parallel strategy combinations based on the ratio between the model parameter scale and GPU memory capacity. Taking a target model with 7 billion parameters as an example, on a GPU computing node equipped with 8 GPUs each with 80GB of memory, the performance prediction model will generate a set of feasible solutions including pure data parallelism, a hybrid of tensor parallelism and data parallelism, and a hybrid of pipelined parallelism and data parallelism. During the generation of the parallel configuration set, the memory access efficiency index of each parallel configuration needs to be calculated. When the estimated memory access efficiency of a certain parallel configuration is lower than a preset threshold, it is marked as having a global memory access bottleneck risk, and such configurations will receive priority attention and processing in subsequent optimization stages.

[0046] In this way, the output set of feasible parallel configurations for memory and its accompanying bottleneck risk markers provide key decision-making basis and optimization direction guidance for throughput optimization.

[0047] S102. Utilize the prediction results to optimize throughput through parallel strategy combination and instruction-level performance monitoring. Parallel strategy combination and instruction-level performance monitoring include: dynamically adjusting the ratio of data parallelism, tensor parallelism, and pipeline parallelism while meeting the preset global batch size, and simultaneously monitoring memory access efficiency and instruction throughput to identify communication bottlenecks and instruction execution bottlenecks.

[0048] For example, under the constraint of a preset global batch size, a dynamic adjustment mechanism is initiated to continuously optimize the ratio of data parallelism, tensor parallelism, and pipelined parallelism. The dynamic adjustment process follows the principle of balancing computation and communication. When increasing the scale of data parallelism causes the actual monitored memory access efficiency to fall below the threshold set in the prediction stage, the degree of data parallelism is automatically reduced, and the proportion of pipelined parallelism or tensor parallelism is correspondingly increased to rebalance the system load. While adjusting the combination of parallel strategies, it is necessary to simultaneously monitor two key performance indicators: memory access efficiency and instruction throughput. CUDA program analysis tools are used to collect low-level hardware data in real time, such as actual memory access bandwidth, effective memory access transactions, number of active thread bundles, and instruction issuance efficiency. Instruction throughput is measured by the number of instructions per clock cycle, reflecting the instruction execution efficiency of the GPU streaming multiprocessor. When the instruction throughput is detected to be consistently below a certain percentage of the theoretical peak, the specific type of communication bottleneck or instruction execution bottleneck is identified in conjunction with the parallel strategy configuration information. For example, in a hybrid configuration of tensor parallelism and data parallelism, when the frequency of data exchange between nodes increases, leading to a decrease in instruction throughput and a simultaneous decrease in memory access efficiency, it can be determined that there is a cross-node communication bottleneck; while when the number of active thread bundles in a streaming multiprocessor is insufficient, resulting in an excessively high proportion of execution dependencies during instruction issue pause cycles, it is identified as an instruction execution bottleneck.

[0049] Through the synergy of parallel strategy combination adjustment and instruction-level performance monitoring, precise bottleneck location and quantitative assessment are provided for kernel re-optimization.

[0050] S103. Kernel re-optimization is performed on instruction-level bottlenecks encountered during the optimization process, thereby simultaneously improving the efficiency of model training and GPU instruction execution. Kernel re-optimization of instruction-level bottlenecks includes: reconstructing the shared memory data layout when bank-conflicts are detected in shared memory; and optimizing by increasing thread-level parallelism or instruction-level parallelism when execution dependency bottlenecks in pipeline instructions are detected.

[0051] For example, kernel re-optimization employs differentiated solutions for specific instruction-level bottleneck types. When performance monitoring data indicates bank-conflicts in shared memory, memory conflicts are eliminated by refactoring the shared memory data layout. This refactoring process involves redesigning the dimensional structure and access patterns of shared memory, defining the shared memory dimension as the number of thread bundles plus one, and reorganizing data access paths according to the logical identifiers of threads within thread bundles, thereby reducing the number of memory conflicts. When pipelined instructions encounter execution dependency bottlenecks, optimization strategies to increase thread-level or instruction-level parallelism are adopted based on the bottleneck characteristics. For matrix operation tasks in tensor parallel computation, instruction-level parallelism is increased by allocating multiple unrelated computational tasks to independent threads. For inter-stage dependencies in pipelined parallelism, the utilization of streaming multiprocessors is increased by adjusting thread block size and grid structure, utilizing more thread-level parallelism to mask instruction pipeline latency. During kernel re-optimization, shared memory access patterns and instruction issue efficiency need to be continuously monitored. The actual effectiveness of optimization measures is verified using memory transaction statistics and instruction stall cause analysis functions in CUDA program analysis tools. The execution bottleneck of pipelined instructions is obtained from the Issue StallsTab of the CUDA program analysis tool, based on the number of eligible warps in Warp Issue Efficiency and the proportion of Execution Dependency in Issue Stall Reasons. The re-optimized computational kernel, while maintaining the correctness of its original functions, significantly improves shared memory access efficiency and instruction execution throughput, enabling a synchronous improvement in model training efficiency and GPU instruction execution efficiency, thereby maximizing overall training throughput.

[0052] The output of kernel re-optimization, together with the parallel strategy configuration information, constitutes a complete optimization scheme, providing performance guarantees for GPU program execution under specific models and hardware environments.

[0053] This application provides a GPU program optimization method in a parallel environment, where multiple target models are configured on a GPU cluster for parallel training. The method includes: configuring strategies and predicting memory bottlenecks in the parallel training configuration of the target models; optimizing throughput through parallel strategy combination and instruction-level performance monitoring using the prediction results; and re-optimizing the kernel for instruction-level bottlenecks encountered during optimization, thereby simultaneously improving model training and GPU instruction execution efficiency. Strategy configuration and memory bottleneck prediction include: obtaining the parameter size of the target models, the characteristics of the GPU cluster, and sequence length constraints; generating a memory-feasible parallel configuration set based on a pre-set performance prediction model; and... The method calculates the memory access efficiency of the parallel configuration set to predict the risk of global memory access bottlenecks; it combines parallel strategies and monitors instruction-level performance, including dynamically adjusting the ratio of data parallelism, tensor parallelism, and pipeline parallelism while meeting the preset global batch size, and simultaneously monitoring memory access efficiency and instruction throughput to identify communication and instruction execution bottlenecks; it performs kernel re-optimization for instruction-level bottlenecks, including reconstructing the shared memory data layout when bank-conflicts are detected in shared memory, and optimizing by increasing thread-level parallelism or instruction-level parallelism when execution dependency bottlenecks are detected in pipeline instructions. By proactively predicting memory bottlenecks and dynamically adjusting the parallel strategy ratio during training, while simultaneously re-optimizing the kernel for detected instruction-level bottlenecks, this method achieves coordinated optimization from the parallel computing dimension to the GPU hardware execution level, improving the utilization of GPU computing and storage resources in complex parallel environments, thereby increasing the overall throughput efficiency of large-scale model training.

[0054] To more clearly illustrate the technical solution of this application, the technical solution of this application will be described below through specific embodiments. It should be noted that the specific embodiments are used to expand the description of the technical solution of this application, and are not intended to limit this application.

[0055] In some embodiments, the performance prediction model is built based on historical benchmark data, taking the parameter size of the target model, the number of nodes, and GPU memory as inputs, to predict the theoretical memory usage and communication overhead of different parallel strategy combinations; the formula for calculating memory access efficiency is:

[0056] ;

[0057] Where β represents memory access efficiency, B1 represents actual memory access bandwidth, B2 represents theoretical peak bandwidth, T1 represents the number of effective memory access transactions, and T2 represents the total number of memory access transactions. and These are weighting coefficients, and .

[0058] For example, in the process of building a performance prediction model, historical benchmark data constitutes the basic dataset for model training. This historical benchmark data comes from actual performance metrics recorded when various combinations of parallel strategies are executed on different hardware configurations.

[0059] The parameter size of the target model serves as input to the performance prediction model, directly impacting the memory footprint and computational intensity of the computation graph. The number of nodes reflects the overall size of the GPU cluster and is used to assess the potential overhead of cross-node communication. GPU memory capacity determines the size of model shards that a single card can accommodate. The performance prediction model learns the complex mapping between these input features and theoretical memory footprint and communication overhead through regression algorithms. Theoretical memory footprint is estimated based on the total number of model parameters, activation values, and optimizer states, combined with the sharding factor under the parallel strategy. Communication overhead is modeled based on the amount of data transmitted between nodes, network topology, and the bandwidth-latency product.

[0060] For example, when the target model has 13 billion parameters and the cluster has 32 nodes, the performance prediction model will output a theoretical memory footprint of 42 gigabytes and a communication overhead of 15% for a combination of tensor parallelism of 4 and pipeline parallelism of 8. The actual memory access bandwidth B1 in the memory access efficiency calculation formula is collected at runtime using GPU performance counters, the theoretical peak bandwidth B2 is taken from the hardware specifications, and the effective memory access transactions T1 and total memory access transactions T2 are obtained from the memory transaction statistics report of the CUDA analysis tool. The weighting coefficients k1 and k2 are empirically set to 0.6 and 0.4 respectively to balance the contributions of bandwidth utilization and transaction efficiency. The output of this performance prediction model provides a quantitative theoretical basis for the subsequent automatic selection of parallel strategies.

[0061] In some embodiments, dynamically adjusting the ratio of data parallelism, tensor parallelism, and pipelined parallelism includes: when increasing the scale of data parallelism causes memory access efficiency to fall below a preset threshold, automatically reducing the degree of data parallelism and instead increasing pipelined parallelism or tensor parallelism to rebalance computation and communication.

[0062] For example, dynamically adjusting the ratio of data parallelism, tensor parallelism, and pipelined parallelism is a closed-loop control process based on real-time performance metrics. This process continuously monitors memory access efficiency values ​​collected by performance counters. When increasing the scale of data parallelism causes memory access efficiency to fall below a preset threshold (e.g., 0.75), the control logic automatically triggers a reallocation of parallel strategies. Reducing the level of data parallelism directly reduces the number of GPUs participating in data parallelism, thereby alleviating the communication pressure caused by global parameter synchronization. Increasing pipelined parallelism, by dividing the model layer sequence into more stages and distributing them to different devices, utilizes pipeline bubble time to hide communication latency. Increasing tensor parallelism further divides model tensor operations within a single computing node, utilizing high-speed interconnects within the node to reduce communication overhead. This dynamic adjustment always aims to rebalance computation and communication, ensuring that the utilization rate of computing units and the occupancy rate of communication bandwidth are maintained at an optimal ratio. In a GPU cluster with sixteen nodes, if the initial configuration is data parallelism of sixteen, tensor parallelism of one, and pipeline parallelism of one, and memory access efficiency is detected to drop below 0.7, the adjustment mechanism may change the configuration to data parallelism of eight, tensor parallelism of two, and pipeline parallelism of one, thereby optimizing resource utilization while keeping the total number of GPUs unchanged.

[0063] In some embodiments, reconfiguring the shared memory data layout includes: defining the dimension of the shared memory as the number of warps plus one, and organizing access patterns according to the thread ID in the warp.

[0064] For example, quantitative data on bank-conflicts can be obtained through the memory transaction statistics report in the CUDA program analysis tool. The shared memory dimension is explicitly defined as the number of currently executing thread bundles plus one. This dimension definition ensures that concurrent access requests within the same thread bundle can be mapped to different memory banks. Access patterns are organized according to the logical ID of the thread within the thread bundle, allowing the access requests of thirty-two threads to be evenly distributed across thirty-three memory banks, fundamentally avoiding serialization waits caused by multiple threads accessing the same memory bank simultaneously. In the matrix transpose kernel optimization case, the unoptimized shared memory access pattern might result in only one memory request being served per memory bank cycle. After optimization, by reconstructing the shared memory data layout, thirty-two concurrent requests can be processed per memory bank cycle, significantly improving the effective bandwidth of the shared memory.

[0065] In some embodiments, the number of bank-conflicts is obtained through memory transaction statistics in the CUDA program analysis tool.

[0066] In some embodiments, optimization is performed by increasing thread-level parallelism or instruction-level parallelism, including: when matrix operations exist in tensor parallelism, assigning multiple unrelated computational tasks to independent threads to improve instruction-level parallelism; when inter-stage dependencies exist in pipelined parallelism, increasing SM occupancy to improve thread-level parallelism.

[0067] For example, when matrix operations exist in tensor parallelism, typical optimization scenarios involve large-scale matrix multiplication operations. Assigning multiple unrelated computational tasks to independent threads can effectively improve instruction-level parallelism. For instance, assigning different block computational tasks of the output matrix to independent thread blocks allows the streaming multiprocessor to schedule more instructions that do not depend on the same data simultaneously. When inter-stage dependencies exist in pipelined parallelism, the forward and backward propagation processes of the model form a natural pipeline. Thread-level parallelism can be improved by increasing the SM occupancy (a core performance metric in CUDA parallel programming, specifically the ratio of the number of active warps to the maximum supported number of warps on each streaming multiprocessor of the GPU). Specifically, this is achieved by adjusting the thread block size and grid dimension so that each streaming multiprocessor can manage more active warps simultaneously, using context switching between warps to mask instruction pipeline stalls caused by inter-stage data dependencies.

[0068] When training large language models containing multiple Transformer blocks, the attention score computation in tensor parallelism improves instruction-level parallelism by decomposing the dot product operation of query vectors and key vectors into multiple subtasks and distributing them to independent threads. Meanwhile, activation value passing in pipelined parallelism increases thread-level parallelism of streaming multiprocessors by increasing the number of micro-batches, effectively utilizing computational resources.

[0069] In some embodiments, such as Figure 2 As shown, the kernel re-optimization and parallel strategy are used in combination, including the following steps: determine whether the currently configured instruction throughput is lower than the instruction bottleneck threshold; if so, trigger the kernel re-optimization process and update the instruction throughput after optimization; if not, directly enter the global batch size coordination stage; determine whether the coordinated global batch size reaches the target; if not, adjust the gradient accumulation step and data parallelism ratio, and evaluate the impact on the instruction flow.

[0070] For example, determining whether the current configured instruction throughput is lower than the instruction bottleneck threshold constitutes the starting decision point of the collaborative optimization process. The instruction throughput is calculated by the ratio of the actual number of instructions executed per clock cycle collected by the CUDA performance counter to the theoretical peak of the hardware. The instruction bottleneck threshold is set to 70% of the theoretical peak instruction throughput based on the characteristics of the GPU architecture.

[0071] When the collected instruction throughput value is lower than the instruction bottleneck threshold, the kernel re-optimization process is triggered. The kernel re-optimization process performs in-depth optimization on the shared memory access mode and instruction execution dependencies. After optimization, the instruction throughput is updated immediately to reflect the actual effect of the optimization measures. If the instruction throughput evaluation result meets or exceeds the instruction bottleneck threshold requirement, the process directly enters the global batch size coordination stage. The core task of the global batch size coordination stage is to ensure that the training scale meets the preset target.

[0072] After the global batch size coordination phase is completed, it is necessary to determine whether the coordinated global batch size has reached the preset global batch size target value. This target value is set to 256 samples based on the model's convergence characteristics and training task requirements. If the coordinated global batch size fails to reach the preset target, a joint adjustment mechanism for gradient accumulation steps and data parallelism ratio is initiated. The gradient accumulation steps are adjusted between one and thirty-two to balance memory usage and training stability, while the data parallelism ratio is adjusted according to the physical constraints of the number of available GPUs in the cluster.

[0073] When evaluating the impact on the instruction flow, the focus is on analyzing whether additional conditional branch instructions are introduced into the forward propagation computation graph after adjusting the gradient accumulation step count. At the same time, the impact of changes in the data parallelism ratio on the parameter synchronization communication mode is evaluated to ensure that the optimized instruction flow can maintain efficient instruction-level parallelism and memory access mode.

[0074] This collaborative optimization process establishes a continuous improvement mechanism by repeatedly executing the above steps until the instruction throughput and global batch size simultaneously meet the preset performance goals and training requirements, forming a close collaboration between macro-parallel strategies and micro-kernel optimization.

[0075] In some embodiments, evaluating the impact on the instruction flow includes: analyzing whether adjusting the gradient accumulation step count introduces additional branch jumps, and prioritizing implementations that do not introduce additional branch jumps or can be replaced by high-throughput instructions.

[0076] For example, assessing the impact on instruction flow requires static and dynamic analysis of the computation graph structure after adjusting the gradient accumulation steps, focusing on detecting whether additional conditional branch jumps are introduced in the forward and backward propagation computation paths. The analysis process traverses the operation nodes of the computation graph through the compiler's intermediate representation layer, identifying conditional judgment instructions added by the gradient accumulation control logic. These conditional branch jumps may disrupt the consistency of instruction execution within a thread bundle, leading to thread bundle divergence and reduced instruction throughput. When a branch structure added due to the gradient accumulation step adjustment is detected, an implementation scheme that does not introduce additional branch jumps is preferred. For example, the gradient accumulation loop is expanded into a linear operation sequence, or conditional judgments are elevated to the thread block level for unified processing. For unavoidable branch jumps, optimization is performed using implementation schemes that can be replaced with high-throughput instructions. For example, vectorized loading instructions are used instead of scalar conditional loading, or predicate execution patterns are used instead of explicit branch instructions, ensuring that all threads within a thread bundle execute the same instruction flow. When training network layers with gating mechanisms, adjusting the gradient accumulation steps from eight to sixteen might increase the step count decision branch in the gradient normalization operation. By replacing the original conditional assignment statement with a combination of mask vectors and fused multiply-accumulate instructions, the differentiation of thread execution paths was effectively eliminated. This evaluation process ensures that adjusting the gradient accumulation steps achieves memory usage optimization without negatively impacting instruction flow efficiency, thus maintaining high utilization of GPU computing units.

[0077] In some embodiments, kernel re-optimization and parallel strategies are used in combination to optimize shared memory access, eliminate instruction pipeline dependencies, and perform parallel processing of data transfer and computational task allocation in parallel model training, making full use of GPU computing units, memory hierarchy, and interconnect bandwidth resources.

[0078] For example, shared memory access optimization restructures the data layout to address access conflicts between model parameters and activation values ​​in memory; instruction pipeline dependency elimination reduces execution stall cycles by increasing streaming multiprocessor utilization and instruction-level parallelism; data transfer in model-parallel training includes gradient synchronization communication flows between parameter servers and worker nodes; and computational task allocation involves computation graph sharding strategies under tensor parallelism and pipeline parallelism. This collaborative mechanism allows global memory access cycles and computational task execution cycles to overlap in the time dimension, and computational task allocation and communication task scheduling to be evenly distributed in the spatial dimension, thereby fully utilizing the arithmetic logic unit of the GPU computing unit, register files and multi-level caches in the memory hierarchy, and NVLink interconnects and InfiniBand networks in interconnect bandwidth resources. In ultra-large-scale model training scenarios, when activation values ​​are transferred between pipeline parallel stages, tensor parallel computing units simultaneously perform matrix multiplication operations in the attention layer, while optimizer state updates are performed in parallel within the data parallel group.

[0079] This multi-level parallel processing allows the utilization of device computing power, memory bandwidth, and network interface to approach their respective physical limits simultaneously, significantly improving the overall throughput efficiency of the training system.

[0080] In some embodiments, dynamically adjusting the ratio of data parallelism, tensor parallelism, and pipelined parallelism further includes: establishing a dynamic feedback control system based on real-time monitored memory access efficiency and instruction throughput; automatically triggering parallel strategy reconfiguration when memory access efficiency continuously falls below a preset efficiency threshold and instruction throughput decreases synchronously, reallocating the computational load of data parallelism, tensor parallelism, and pipelined parallelism; wherein the constraints of parallel strategy reconfiguration are:

[0081] Maximize f(TP,PP,DP) = w1×β+ w2×IPC

[0082] TP×PP×DP = Ngpu;

[0083] Mem-U ≤ Mem-A;

[0084] GBS≥GBS-T;

[0085] Where Maximize f(·) is the objective function, TP, PP, and DP represent the parallelism of tensor parallelism, pipeline parallelism, and data parallelism, respectively, Ngpu represents the total number of GPUs, Mem-U represents the model memory usage, Mem-A represents the available video memory, GBS represents the global batch size, GBS-T represents the target global batch size, and w1 and w2 are weight coefficients.

[0086] For example, the dynamic feedback control system established during the dynamic adjustment of the ratio of data parallelism, tensor parallelism, and pipelined parallelism constitutes a closed-loop optimization mechanism based on multi-sensor input. This dynamic feedback control system continuously collects memory access efficiency readings and instruction throughput samples from the GPU performance counter. When the memory access efficiency remains below a preset efficiency threshold of 0.8 for several consecutive monitoring cycles and the instruction throughput simultaneously decreases by more than 10%, the trigger condition is met, and the parallel strategy reconfiguration process is automatically triggered. Reallocating the computational load of data parallelism, tensor parallelism, and pipelined parallelism requires solving an optimization problem with multiple constraints. The weight coefficients w1 and w2 in the objective function are set to 0.6 and 0.4 respectively, depending on the application characteristics, to balance the contributions of memory access efficiency and instruction throughput to overall performance. The constraints TP×PP×DP = Ngpu ensure that all available GPU computing resources are included in the parallel computing graph and there are no idle devices. The constraint that the model memory usage Mem-U does not exceed the available video memory Mem-A ensures the executability of the training task and avoids interruptions due to insufficient video memory. The constraint that the global batch size GBS is not less than the target global batch size GBS-T maintains the quality and convergence characteristics of model training.

[0087] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A GPU program optimization method for a parallel environment, characterized in that, Multiple target models are configured on a GPU cluster for parallel training, the method comprising: The parallel training configuration of the target model is configured with a strategy and memory bottlenecks are predicted. Utilizing the prediction results, throughput optimization is achieved through parallel strategy combination and instruction-level performance monitoring; Kernel re-optimization was performed to address instruction-level bottlenecks encountered during the optimization process, thereby simultaneously improving the efficiency of model training and GPU instruction execution. The strategy configuration and memory bottleneck prediction include: obtaining the parameter size of the target model, the characteristics of the GPU cluster, and sequence length constraints; generating a memory-feasible parallel configuration set based on a pre-set performance prediction model; and calculating the memory access efficiency of the parallel configuration set to predict the risk of a global memory access bottleneck. The performance prediction model is built based on historical benchmark data, using the parameter size, number of nodes, and GPU memory of the target model as input, to predict the theoretical memory usage and communication overhead of different parallel strategy combinations. The formula for calculating the memory access efficiency is: β = k1 × (B1 / B2) + k2 × (T1 / T2); Where β is the memory access efficiency, B1 represents the actual memory access bandwidth, B2 represents the theoretical peak bandwidth, T1 represents the number of effective memory access transactions, T2 represents the total number of memory access transactions, k1 and k2 are weighting coefficients, and k1 + k2 = 1. The parallel strategy combination and instruction-level performance monitoring include: dynamically adjusting the ratio of data parallelism, tensor parallelism, and pipelined parallelism under the condition of meeting a preset global batch size, and simultaneously monitoring memory access efficiency and instruction throughput to identify communication bottlenecks and instruction execution bottlenecks; the dynamic adjustment of the ratio of data parallelism, tensor parallelism, and pipelined parallelism includes: establishing a dynamic feedback control system based on the real-time monitored memory access efficiency and instruction throughput, and automatically triggering parallel strategy reconfiguration when the memory access efficiency is continuously lower than a preset efficiency threshold and the instruction throughput decreases synchronously, reallocating the computational load of data parallelism, tensor parallelism, and pipelined parallelism, wherein the constraints of the parallel strategy reconfiguration are: Maximize f(TP,PP,DP) = w1×β+ w2×IPC; TP×PP×DP = Ngpu; Mem-U ≤ Mem-A; GBS≥GBS-T; Where Maximize f(·) is the objective function, TP, PP, and DP represent the parallelism of tensor parallelism, pipeline parallelism, and data parallelism, respectively, Ngpu represents the total number of GPUs, Mem-U represents the model memory usage, Mem-A represents the available video memory, GBS represents the global batch size, GBS-T represents the target global batch size, w1 and w2 are weight coefficients, and IPC is the average number of instructions executed per clock cycle. The kernel re-optimization of instruction-level bottlenecks encountered during the optimization process includes: reconstructing the shared memory data layout when bank-conflicts are detected in the shared memory; and optimizing by increasing thread-level parallelism or instruction-level parallelism when execution dependency bottlenecks are detected in the pipeline instructions.

2. The method as described in claim 1, characterized in that, The dynamic adjustment of the ratio of data parallelism, tensor parallelism, and pipelined parallelism also includes: when increasing the scale of data parallelism causes the memory access efficiency to fall below a preset threshold, automatically reducing the degree of data parallelism and instead increasing pipelined parallelism or tensor parallelism to rebalance computation and communication.

3. The method as described in claim 1, characterized in that, The reconstructing of the shared memory data layout includes: defining the dimension of the shared memory as the number of warps plus one, and organizing the access patterns according to the thread ID in the warp, wherein the warp is the basic execution unit in the GPU architecture.

4. The method as described in claim 1, characterized in that, The number of access conflicts in the bank-conflicts is obtained through memory transaction statistics in the CUDA program analysis tool, and the bank-conflicts are memory bank conflicts.

5. The method as described in claim 1, characterized in that, The optimization achieved by increasing thread-level parallelism or instruction-level parallelism includes: When matrix operations exist in tensor parallelism, multiple unrelated computational tasks are assigned to independent threads to improve instruction-level parallelism. When there are inter-stage dependencies in pipelined parallelism, thread-level parallelism can be improved by increasing the SM occupancy rate.

6. The method as described in claim 1, characterized in that, The combined use of the kernel re-optimization and the parallel strategy includes the following steps: Determine whether the current configured instruction throughput is lower than the instruction bottleneck threshold; If so, the kernel re-optimization process is triggered, and the instruction throughput is updated after optimization; If not, proceed directly to the global batch size coordination stage; Determine whether the coordinated global batch size has reached the target. If not, adjust the gradient accumulation steps and data parallelism ratio, and evaluate the impact on the instruction flow.

7. The method as described in claim 6, characterized in that, The evaluation of the impact on the instruction flow includes: analyzing whether adjusting the gradient accumulation step count introduces additional branch jumps, and prioritizing implementation schemes that do not introduce additional branch jumps or can be replaced by high-throughput instructions.

8. The method as described in claim 1, characterized in that, The kernel re-optimization, combined with the parallel strategy, works synergistically to optimize shared memory access, eliminate instruction pipeline dependencies, and enable parallel processing of data transfer and computational task allocation during parallel model training, fully utilizing GPU computing units, storage hierarchy, and interconnect bandwidth resources.

Images