Algorithm of tensor parallel computing large model based on cpu + gpu

Abstract

The invention discloses a cpu + gpu-based tensor parallel computing algorithm for a large model, which comprises the following steps of: S1, combining row parallelism and column parallelism for a linear layer in the model by adopting a mixed-dimension tensor segmentation mode, and dynamically adjusting a segmentation proportion according to a model structure and hardware resources; s2, a CPU and GPU cooperative computing mechanism is constructed, part of tasks which have large video memory requirements and are relatively simple in computation are allocated to the CPU, the GPU is responsible for computing intensive tasks, communication between CPU-GPU is optimized, and overlapping of CPU-GPU communication and GPU computation is achieved; and S3, implementing a dynamic resource allocation and load balancing strategy, monitoring load conditions of the CPU and the GPU in real time, dynamically adjusting task allocation according to calculation requirements of different layers of the model and use conditions of hardware resources, and adopting a self-adaptive batch processing size adjustment strategy. The invention obviously reduces the occupation of the video memory, reduces the communication cost and improves the utilization rate of computing resources.

Description

Technical Field

[0001] The present invention relates to the fields of computer science and artificial intelligence technology, and in particular to an algorithm for a large model of tensor parallel computing based on CPU+GPU. Background Art

[0002] With the continuous expansion of deep learning models, such as the introduction of Transformer and MOE architectures, model parameters have easily exceeded trillions. The traditional single-machine, single-GPU model is no longer sufficient for training very large models. Distributed large-scale model training is now required, based on multiple GPUs on a single machine, or even multiple machines with multiple GPUs. Leveraging AI clusters to efficiently train large, high-performance models is the primary goal of distributed machine learning.

[0003] Common tensor parallel schemes, such as one-dimensional tensor parallelism, partition parameters across multiple processors, but each processor still needs to store the entire intermediate activations, wasting significant memory space when processing large models. Furthermore, because only one-dimensional matrix partitioning is used, communication costs surge with increasing parallelism. To address these issues, while Colossal-AI offers multi-dimensional tensor parallelism (2 / 2.5 / 3 dimensions), it still suffers from inefficient communication and computing resource utilization. Therefore, we propose a CPU-based + GPU-based tensor parallel computing algorithm for large models. Summary of the Invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose an algorithm for large-scale tensor parallel computing based on CPU+GPU.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] An algorithm for parallel computing of large models using tensors based on CPU+GPU, including the following steps:

[0007] S1. Use mixed-dimensional tensor splitting to combine row and column parallelism for the linear layers in the model, and dynamically adjust the split ratio based on the model structure and hardware resources.

[0008] S2. Build a collaborative computing mechanism between the CPU and GPU, assigning some tasks that require large amounts of video memory and are relatively simple to the CPU, while the GPU is responsible for computationally intensive tasks. This optimizes CPU-GPU communication and enables the overlap of CPU-GPU communication and GPU computing.

[0009] S3. Implement dynamic resource allocation and load balancing strategies, monitor the CPU and GPU load in real time, dynamically adjust task allocation based on the computational requirements of different layers of the model and the usage of hardware resources, and adopt an adaptive batch size adjustment strategy;

[0010] S4. Improve the ring algorithm based on communication optimization, dynamically adjust the communication path according to the communication bandwidth and delay between nodes, and introduce a communication cache mechanism.

[0011] Preferably, in step S1, when the memory occupancy rate of the linear layer input data exceeds 70% of the total GPU memory, the column parallel ratio is increased, the weight matrix is ​​divided equally by column, the input matrix is ​​copied to each GPU and the column part of the corresponding weight matrix is ​​calculated, and then the results are spliced ​​through All-Reduce. LSTM is used to predict the memory requirements of the next 5 iterations, and the split ratio is adjusted in advance. Each time the split ratio is adjusted, the weight sharding topology information is automatically saved to the CPU memory. When the GPU memory fragmentation rate is greater than 30%, tasks with high continuous memory requirements are migrated to the CPU, and paths with fewer hops are preferentially selected based on the physical distance between the GPU and the network card.

[0012] Preferably, in step S1, when the bandwidth utilization of the node communication link reaches 90% and the average delay exceeds 50ms, the row parallel ratio is increased, the weight matrix and the input matrix are divided and added after calculation on each GPU.

[0013] Preferably, in step S2, data transmission is compressed using the LZ4 algorithm. When the GPU performs calculations, the CPU completes the next round of data reading and format conversion in parallel, and achieves calculation and communication overlap by using timestamps and event triggering.

[0014] Preferably, in step S3, when the GPU utilization is 75%-85% and the loss convergence is stable, the utilization rate is increased by a factor of 1.1; when it is lower than 70% or fluctuates violently, the utilization rate is decreased by a factor of 0.9.

[0015] Preferably, in step S4, a dedicated communication buffer is set on the GPU, and the data to be transmitted is placed in the buffer in advance. When the communication link is idle, data is transmitted in a timely manner, and block sparse coding is used for the buffer data.

[0016] Preferably, in step S2, at the Embedding layer, the word dimension is segmented, each card stores a partial word vector table, the CPU assists in partial word vector calculation, and then the partial word vector results on each device are aggregated through AllGather, and the segmentation dimension weight is dynamically selected according to the layer type.

[0017] Preferably, in step S3, when a GPU is overloaded, some tasks are transferred to a GPU or CPU with a lower load for execution, and the size of each batch of input data is dynamically adjusted according to the current utilization of hardware resources and the stability of model training. The word vectors with the top 10% word frequency are retained on the GPU, and the rest are allocated to the CPU.

[0018] Preferably, in step S4, when it is detected that the communication delay between two nodes is high, other communication paths are automatically switched to reduce the communication waiting time.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. In the present invention, the video memory usage is significantly reduced. Through mixed-dimensional tensor segmentation and reasonable CPU-GPU task allocation, the storage requirements of each processor for intermediate activations are effectively reduced, and the video memory utilization rate is improved.

[0021] 2. In the present invention, the communication cost is reduced, the improved ring algorithm and the communication cache mechanism reduce the communication waiting time and the data transmission volume, so that the communication cost is effectively controlled as the parallelism increases.

[0022] 3. In the present invention, the utilization rate of computing resources is improved, the dynamic resource allocation and load balancing algorithm, and the CPU-GPU collaborative computing mechanism are used to fully utilize the computing power of the CPU and GPU, avoid idle computing resources, and greatly improve the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a logical block diagram of an algorithm for a large model of tensor parallel computing based on CPU+GPU proposed by the present invention. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0025] Reference Figure 1 , an algorithm for tensor parallel computing of large models based on CPU+GPU, including the following steps:

[0026] S1. Use mixed-dimensional tensor splitting to combine row and column parallelism for the linear layers in the model, and dynamically adjust the split ratio based on the model structure and hardware resources.

[0027] S2. Build a collaborative computing mechanism between the CPU and GPU, assigning some tasks that require large amounts of video memory and are relatively simple to the CPU, while the GPU is responsible for computationally intensive tasks. This optimizes CPU-GPU communication and enables the overlap of CPU-GPU communication and GPU computing.

[0028] S3. Implement dynamic resource allocation and load balancing strategies, monitor the CPU and GPU load in real time, dynamically adjust task allocation based on the computational requirements of different layers of the model and the usage of hardware resources, and adopt an adaptive batch size adjustment strategy;

[0029] S4. Improve the ring algorithm based on communication optimization, dynamically adjust the communication path according to the communication bandwidth and delay between nodes, and introduce a communication cache mechanism.

[0030] In step S1, when the memory occupancy rate of the linear layer input data exceeds 70% of the total GPU memory, the column parallel ratio is increased, the weight matrix is ​​divided equally by column, the input matrix is ​​copied to each GPU and the column part of the corresponding weight matrix is ​​calculated, and the results are then concatenated through All-Reduce. LSTM is used to predict the memory requirements for the next five iterations, and the split ratio is adjusted in advance. Each time the split ratio is adjusted, the weight sharding topology information is automatically saved to the CPU memory. When the GPU memory fragmentation rate is greater than 30%, tasks with high continuous memory requirements are migrated to the CPU, and paths with fewer hops are prioritized based on the physical distance between the GPU and the network card.

[0031] In step S1, when the bandwidth utilization of the node communication link reaches 90% and the average delay exceeds 50ms, the row parallel ratio is increased, the weight matrix and the input matrix are divided and added after calculation on each GPU.

[0032] In step S2, data transmission is compressed using the LZ4 algorithm. When the GPU is performing calculations, the CPU completes the next round of data reading and format conversion in parallel, and uses timestamps and event triggers to achieve computational communication overlap.

[0033] In step S3, when the GPU utilization is 75%-85% and the loss convergence is stable, it is increased by a factor of 1.1. When it is lower than 70% or fluctuates violently, it is reduced by a factor of 0.9.

[0034] In step S4, a dedicated communication buffer is set up on the GPU, and the data to be transmitted is placed in the buffer in advance. When the communication link is idle, data is transmitted in a timely manner, and block sparse coding is used for the buffer data.

[0035] In step S2, at the Embedding layer, the word dimension is split, and each card stores part of the word vector table. The CPU assists in partial word vector calculation, and then AllGather is used to aggregate the partial word vector results on each device. The split dimension weight is dynamically selected according to the layer type.

[0036] In step S3, when the load on a certain GPU is too high, some tasks are transferred to the GPU or CPU with lower load for execution. The size of each batch of input data is dynamically adjusted based on the current hardware resource utilization and the stability of model training. The word vectors of the top 10% of word frequency are retained on the GPU, and the rest are allocated to the CPU.

[0037] In step S4, when it is detected that the communication delay between two nodes is high, the communication path is automatically switched to another one to reduce the communication waiting time.

[0038] Example 1:

[0039] Training based on the Transformer model:

[0040] Model preparation: Take a language model based on the Transformer architecture as an example. The model contains multiple Transformer blocks. Each Transformer block consists of self-attention and MLP, as well as input and output Embedding layers.

[0041] Tensor Splitting and Computation: In the MLP layer, the ratio of row and column parallelism is dynamically determined based on the model's computational requirements and hardware resources. Assume that the initial column parallelism ratio is 60% and the row parallelism ratio is 40%. Column splitting is used for linear layer A, splitting the weight matrix into different parts by column and storing them on different GPUs. The input matrix is ​​copied to each GPU and computed with the corresponding part of linear layer A. Row splitting is used for linear layer B, splitting the weight matrix by row. The results of linear layer A are split by column and then computed on different GPUs with the corresponding rows of linear layer B. In the self-attention layer, the parameter matrix is ​​also split according to a dynamic strategy using mixed dimensions. For example, 70% is split by column and 30% is split by row. The results of the corresponding parts are computed on each GPU and then concatenated.

[0042] CPU-GPU collaborative computing: During embedding layer calculations, the CPU is responsible for partial word vector calculations, and each GPU stores part of the word vector table. After the CPU and GPU collaborate to complete word vector calculations, they aggregate the results through AllGather. While the GPU performs computationally intensive tasks such as matrix multiplication, the CPU performs preprocessing and preparation for the next round of data, such as data loading and format conversion, in parallel.

[0043] Dynamic resource allocation and load balancing: During training, the CPU and GPU loads are monitored in real time. If the load on one GPU exceeds 80% and the load on other GPUs is below 50%, some subsequent computing tasks are transferred from the high-load GPU to the low-load GPU. Furthermore, the batch size is dynamically adjusted based on current hardware resource utilization and model training stability. For example, when GPU utilization stabilizes at 70%-80%, the batch size is appropriately increased to improve computing resource utilization.

[0044] Communication optimization: Utilizing an improved ring algorithm for communication, inter-node communication bandwidth and latency are monitored in real time during training. If the latency between two GPU nodes exceeds a set threshold, the system automatically switches communication paths, selecting a lower-latency path for data transmission. Furthermore, a communication buffer is set up on the GPU to pre-place data to be transmitted. This buffer allows for timely transmission when the communication link is idle, reducing communication latency.

[0045] Example 2:

[0046] Image recognition large model training:

[0047] Model preparation: A large image recognition model with a complex structure, which contains multiple convolutional layers, pooling layers, and fully connected layers.

[0048] Tensor Splitting and Computation: For convolutional layers, the convolution kernel tensor is split into a mixed channel and spatial dimensions, with the split ratio dynamically determined based on the convolutional layer's position in the model and its computational characteristics. For example, in shallow layers of the model, due to the high resolution of the feature maps and the large amount of computation, the spatial dimension split ratio is set to 70% and the channel dimension split ratio is 30%. In deeper layers of the model, the feature map resolution decreases and the number of channels increases, so the spatial dimension split ratio is adjusted to 40% and the channel dimension split ratio is adjusted to 60%. Different GPUs calculate the corresponding parts of the convolution results separately and then merge them. The fully connected layer uses a mixed-dimensional tensor splitting strategy similar to the MLP layer in the Transformer model, dynamically adjusting the row and column parallelism ratios based on hardware resources and computational requirements.

[0049] CPU-GPU collaborative computing: The CPU is responsible for some data pre-processing and post-processing, such as data normalization and image resizing, while the GPU focuses on core computing tasks such as convolution and matrix multiplication. During data transmission, data is compressed and optimized to reduce the amount of data transmitted between the CPU and GPU.

[0050] Dynamic resource allocation and load balancing: Real-time monitoring of hardware resource usage. When a GPU is overloaded while performing convolution calculations, while other GPUs are transferring data or idle, task allocation is dynamically adjusted to allocate some convolution tasks to idle or lightly loaded GPUs. Furthermore, batch size is dynamically adjusted based on model training stability and computing resource utilization to improve overall training efficiency.

[0051] Communication optimization: Utilizing an improved ring algorithm and communication caching mechanism, communication paths are dynamically optimized during convolutional layer calculations based on the transmission requirements of convolution kernels and feature maps. For small-sized convolution kernel data that requires frequent transmission, high-speed communication links are prioritized and data is pre-cached in the GPU's communication cache, reducing communication latency and improving communication efficiency.

[0052] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An algorithm for large-scale tensor parallel computing based on cpu+gpu, characterized in that The following steps are involved: S1. Use mixed-dimensional tensor splitting to combine row and column parallelism for the linear layers in the model, and dynamically adjust the split ratio based on the model structure and hardware resources. S2. Build a collaborative computing mechanism between the CPU and GPU, assigning some tasks that require large amounts of video memory and are relatively simple to the CPU, while the GPU is responsible for computationally intensive tasks. This optimizes CPU-GPU communication and enables the overlap of CPU-GPU communication and GPU computing. S3. Implement dynamic resource allocation and load balancing strategies, monitor the CPU and GPU load in real time, dynamically adjust task allocation based on the computational requirements of different layers of the model and the usage of hardware resources, and adopt an adaptive batch size adjustment strategy; S4. Improve the ring algorithm based on communication optimization, dynamically adjust the communication path according to the communication bandwidth and delay between nodes, and introduce a communication cache mechanism.

2. The algorithm of the large model of tensor parallel computing based on cpu+gpu according to claim 1 is characterized in that: In step S1, when the memory occupancy rate of the linear layer input data exceeds 70% of the total GPU memory, the column parallel ratio is increased, the weight matrix is ​​divided equally by column, the input matrix is ​​copied to each GPU and the column portion of the corresponding weight matrix is ​​calculated, and the results are then spliced ​​together through All-Reduce. The memory requirements for the next five iterations are predicted using LSTM, and the split ratio is adjusted in advance. Each time the split ratio is adjusted, the weight sharding topology information is automatically saved to the CPU memory. When the GPU memory fragmentation rate is greater than 30%, tasks with high continuous memory requirements are migrated to the CPU, and paths with fewer hops are preferentially selected based on the physical distance between the GPU and the network card.

3. The algorithm of the large model of tensor parallel computing based on cpu+gpu according to claim 1 is characterized in that: In step S1, when the bandwidth utilization of the node communication link reaches 90% and the average delay exceeds 50ms, the row parallel ratio is increased, the weight matrix and the input matrix are divided and added after calculation on each GPU.

4. The algorithm of a large model of tensor parallel computing based on cpu+gpu according to claim 1 is characterized in that: In step S2, data transmission is compressed using the LZ4 algorithm. When the GPU is performing calculations, the CPU completes the next round of data reading and format conversion in parallel, and achieves calculation and communication overlap by using timestamps and event triggers.

5. The algorithm of the large model of tensor parallel computing based on cpu+gpu according to claim 1 is characterized in that: In step S3, when the GPU utilization is 75%-85% and the loss convergence is stable, the GPU utilization is increased by a factor of 1.1; when it is lower than 70% or fluctuates violently, the GPU utilization is reduced by a factor of 0.

9.

6. The algorithm of a large model of tensor parallel computing based on cpu+gpu according to claim 1 is characterized in that: In step S4, a dedicated communication buffer is set on the GPU, and data to be transmitted is placed in the buffer in advance. When the communication link is idle, data is transmitted in a timely manner, and block sparse coding is used for the buffer data.

7. The algorithm of a large model of tensor parallel computing based on cpu+gpu according to claim 1 is characterized in that: In step S2, at the Embedding layer, the word dimension is segmented, and each card stores a partial word vector table. The CPU assists in calculating the partial word vectors, and then the partial word vector results on each device are aggregated through AllGather, and the segmentation dimension weight is dynamically selected according to the layer type.

8. The algorithm of a large model of tensor parallel computing based on cpu+gpu according to claim 1 is characterized in that: In step S3, when a GPU is overloaded, some tasks are transferred to the GPU or CPU with lower load for execution. In addition, the size of each batch of input data is dynamically adjusted based on the current hardware resource utilization and the stability of model training. The word vectors with the top 10% word frequency are retained on the GPU, and the rest are allocated to the CPU.

9. The algorithm of a large model of tensor parallel computing based on cpu+gpu according to claim 1, characterized in that: In step S4, when it is detected that the communication delay between two nodes is high, other communication paths are automatically switched to reduce the communication waiting time.

Images