Two-dimensional sequence splitting method and system under large model pipeline parallel training

Abstract

The invention discloses a two-dimensional sequence splitting method and system under large model assembly line parallel training, and belongs to the field of computer science artificial intelligence. The method comprises the steps that a data collection module obtains equipment basic information and model configuration information, wherein the equipment basic information comprises the bandwidth between GPUs, the size of an equipment video memory, the size of an equipment CPU memory, the bandwidth between GPUs and CPUs, model dimensions, the number of model layers and the length of an input data sequence; the decision maker generates an optimal decision according to the acquired data; the decision content comprises a redundant sequence length, a redundant sequence proportion stored in the GPU, a redundant sequence proportion stored in the CPU and a time dimension splitting number; and the deep learning training module integrates the optimal decision into a model training process, so that the overall training performance of the system is improved. According to the method, the free video memory space and the bandwidth between the GPU and the CPU are combined to realize sequence splitting and efficient training in two dimensions of time and space, and meanwhile, the training efficiency of pipeline parallel training is maximized.

Description

Technical Field

[0001] The present invention relates to the field of computer science and artificial intelligence, and in particular to a two-dimensional sequence splitting method and system under large model pipeline parallel training. Background Art

[0002] With the development of artificial intelligence technology, the scale and complexity of models continue to grow, and more powerful performance and more accurate prediction results are obtained. These large models, such as Transformer models, usually contain hundreds of millions or even billions of parameters. For example, one of the latest open source Transformer large models, LLama3.1-405B, contains more than 400 billion parameters. At the same time, the introduction of the latest technologies in the direction of large models (such as RAG) has also led to a significant increase in the length of input samples included in the training process of large models. For the most advanced open source large model Deepseek-V3 model, its context window length has reached 128,000 tokens. The increase in model parameters has brought about a huge demand for video memory capacity, making it very difficult to train these models on a single GPU or CPU. In order to solve this problem, pipeline parallel technology is often introduced in the training process of large models. The core idea of ​​pipeline parallel technology is to split a large model into multiple smaller sub-models. Each sub-model can be trained and calculated independently on different hardware devices, forming a pipeline between each other to efficiently exert the potential of the device.

[0003] The addition of long sequence samples has made video memory capacity a serious problem that limits the training of large Transformer models. This has led to two problems: 1. The video memory usage of activation tensors generated by long sequence samples is high, and 2. The introduction of more device idle time (bubbles) has also led to a decline in the parallel performance of the above pipeline. In order to solve these problems, sequence chunking (parallelism) was introduced into the training of large models. The core idea of ​​sequence chunking is to split a long sequence into several shorter sequences and redistribute these shorter sequences to multiple devices or different time periods. In the process of attention calculation, the attention output of each subsequence is obtained through chunking calculation. There are two types of sequence splitting technologies: one is to split and parallelize in the spatial dimension (such as the RingAttention Attention parallel computing algorithm), splitting long sequences into equal-length subsequences and distributing them to multiple devices for simultaneous calculation. The disadvantage of this type of sequence parallelism is that the degree of parallelism is affected by the number of devices, and it is necessary to ensure that the calculation time can cover the communication time; the other is to split in the time dimension (such as TeraPipe, Seq1F1B technology), splitting long sequences into subsequences of gradually decreasing length according to the uniform amount of calculation and calculating them block by block on a device in sequence. The disadvantage of this type of sequence splitting is that it occupies a high amount of activation tensor memory, and the effect is not as good as sequence splitting in the spatial dimension.

[0004] A simple idea is to combine the two directly. First, use time splitting to split the long sequence into different time periods, and then use space splitting to split the subsequences in the corresponding time period onto multiple devices for parallel computing. However, simply combining the two will cause new problems. Since the Attention mechanism needs to calculate the token in the sequence and all the previous tokens in pairs, the communication volume of each block will increase linearly with the position of the block in the sequence. However, the rate at which the amount of computation of the attention mechanism increases is lower than the rate at which the communication volume increases. In training, the communication time will be greater than the computation time. At this time, additional communication overhead will occur during the training process, reducing the overall training efficiency. Summary of the invention

[0005] In view of the problem that the performance of long sequence samples in the prior art is low under the parallel training of large model pipelines, and the single-dimensional sequence splitting technology cannot fully utilize the performance of cluster equipment, the purpose of the present invention is to provide a two-dimensional sequence splitting parallel acceleration method and system under the parallel training of large model pipelines.

[0006] The object of the present invention is achieved through the following technical solution: a two-dimensional sequence splitting method under large model pipeline parallel training, comprising the following steps:

[0007] S1. Use real RingAttention load test and collect inter-GPU bandwidth, device video memory size, device CPU memory size, GPU-CPU bandwidth, model dimension, number of model layers, and input data sequence length;

[0008] S2. Making intelligent decisions based on the information collected in step S1, wherein the decision content includes the length of the redundant sequence, the proportion of the redundant sequence stored in the GPU, the proportion of the redundant sequence stored in the CPU, and the number of time dimension splits;

[0009] S3. Split the input sequence into first subsequences of decreasing length according to the number of splits in the time dimension and the length of each subsequence, further split the subsequence into M second subsequences of equal length, and distribute the M second subsequences to the GPUs in the M sequence parallel groups; input the second subsequences into the Transformer large model optimized by the two-dimensional sequence parallel operator for training; wherein, during the training process, the redundant sequence length, the proportion of redundant sequences stored in the GPU, and the proportion of redundant sequences stored in the CPU are used as parameters to be input into the two-dimensional sequence parallel operator, and during training, the following are guided by the above parameters: saving redundant sequences to the GPU, transferring redundant sequences from the GPU to the CPU, and transferring redundant sequences from the CPU to the GPU.

[0010] Furthermore, the step S1 is specifically as follows:

[0011] When the user deploys the training framework to the cluster and inputs the model and training configuration, the model dimension, number of model layers, and length of the input data sequence are recorded, and a sequence of arbitrary length and dimension is used to run the RingAttention operator for several rounds in the sequence parallel group. The FP16 precision computing capability of the GPU device and the communication bandwidth between GPU devices are calculated based on the time obtained from the profile when the operator is executed. The above sequence is then used to copy several rounds between the GPU and CPU, and the bandwidth between the GPU and CPU is calculated based on the time obtained from the profile when copying.

[0012] Furthermore, the sequence of arbitrary length and dimension is used to run the RingAttention operator for several rounds in the sequence parallel group, and a sequence with a length of 32k and a dimension of 1024 is selected.

[0013] Furthermore, the step S2 comprises:

[0014] S2.1: Initialize the time dimension split number search interval [N low , N high ) is [1, N max ), where N max The maximum number of time dimension splits set by the user, N lowis the lower bound of the search interval of the current round of binary search, N max The upper bound of the search interval for the current round of binary search;

[0015] S2.2: In the time dimension, the number of splits N is taken as the interval [N low , N high )Intermediate value N mid In the case of , the sequence is split into N segments according to the standard that the amount of calculation in each segment is equal mid subsequences; under the condition that the Attention communication overhead is less than or equal to the Attention computation overhead, linear integer programming is used to solve the shortest redundant sequence length, the corresponding GPU redundant sequence ratio and the CPU redundant sequence ratio;

[0016] S2.3: Calculate the memory overhead of the current decision. If it exceeds the memory budget, update the interval [N low , N high ) is [N low , N mid ), otherwise, update interval [N low , N high ) is [N mid , N high ); If the number of elements in the interval is 1, return the result, otherwise repeat steps S2.2-S2.3.

[0017] Furthermore, the step S2.2 includes: the constraint conditions and optimization objectives for searching the shortest redundant sequence length and the optimal CPU to GPU redundant sequence ratio under the shortest redundant sequence length are shown in the following formula:

[0018] ;

[0019] Among them, L is the total length of the sequence, L local represents the length of redundant sequence, N is the number of time dimension sequence splits, l N is the length of the last subsequence, M is the number of spatial dimension sequence splits, E is the FP16 computing power of a single GPU, and B gpu , B cpu are the inter-GPU communication bandwidth and the GPU-CPU communication bandwidth respectively, d is the model dimension, and F a Attention calculation amount for the last subsequence, p cpu 、p gpu Respectively represent the proportion of redundant sequences stored in CPU memory and GPU memory, M cpu The size of the CPU memory space.

[0020] Furthermore, the step S3 includes: performing the following steps when each layer of the Transformer model performs attention mechanism calculation:

[0021] S3.1: Calculate the attention of the current Q and the current KV; at the same time, receive the next KV block from the GPU numbered i-1 and send the current KV to the GPU numbered i+1. If there is GPU / CPU redundancy, ignore the redundant part of the data when sending / receiving, and only send / receive the non-redundant part; if the CPU redundancy ratio is not 0 at this time, copy the redundant part of the next KV block from the CPU memory to the GPU memory at the same time;

[0022] S3.2: Update the attention calculation results of the local block;

[0023] S3.3: If according to the decision, the KV of the current time period needs to be redundant, the KV involved in the calculation in step S3.1 is saved into a tensor;

[0024] S3.4: Update the current KV to the next KV block, and repeat steps S3.1-S3.4 until the current Q and each KV block have performed attention mechanism calculations;

[0025] S3.5: If, according to the decision, the KV of the current time period needs to be redundant, the tensor storing the KV in step S3.3 is split into two parts, wherein one part is stored in the GPU according to the ratio specified by the decision, and the remaining part of the KV is asynchronously copied to the CPU memory and then the tensor in step S3.3 is released;

[0026] S3.6: If the CPU redundancy ratio is not 0, asynchronously pre-fetch the first CPU redundancy KV block of the next layer.

[0027] The present invention also provides a two-dimensional sequence splitting system under large model pipeline parallel training, the system comprising:

[0028] The data collection module is used to obtain basic device information and model configuration information, including inter-GPU bandwidth, device memory size, GPU-CPU bandwidth, model dimension, number of model layers, and input data sequence length;

[0029] A decision maker, used to generate an optimal decision based on the data acquired by the data collection module; wherein the decision content includes the length of redundant sequences, the proportion of redundant sequences stored in the GPU, the proportion of redundant sequences stored in the CPU, and the number of time dimension splits;

[0030] A deep learning training module is used to integrate the optimal decision into the model training process to improve the overall training performance of the system. The deep learning training module includes a long sequence sample splitting submodule and a two-dimensional sequence parallel operator submodule.

[0031] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the two-dimensional sequence splitting method under the large model pipeline parallel training.

[0032] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the two-dimensional sequence splitting method under the large model pipeline parallel training is implemented.

[0033] The beneficial effect of the present invention is that the traditional single-dimensional sequence splitting cannot maximize the efficiency of large model pipeline parallel training under the condition of limited video memory occupancy, but the simple combination of two-dimensional sequence splitting will introduce new communication overhead. The two-dimensional sequence splitting parallel acceleration scheme proposed by the present invention is different from the traditional single-dimensional or simple combination. The present invention combines the splitting of space and time dimensions and designs corresponding operators, saves redundant Key / Value Tensor copies and Key / Value Tensor memory-video memory transfer on GPU and CPU, and uses linear integer programming algorithm modeling to realize dynamic decision-making of the system, reduces the amount of data transmission between GPU and GPU under limited video memory and memory, and reduces the overall communication overhead, thereby accelerating the overall training efficiency of long sequence samples under large model pipeline parallel training. Compared with not using sequence splitting, two-dimensional sequence splitting can increase the throughput by about 10%, and compared with single-dimensional sequence splitting or simply combining two-dimensional splitting, the present invention can increase the throughput by 2-5%. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a system architecture diagram of the present invention;

[0035] Figure 2 This is a graph showing the experimental results of two-dimensional sequence splitting under pipeline parallel training, comparing no splitting, single dimension splitting, and simple combined splitting. DETAILED DESCRIPTION

[0036] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0038] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0039] like Figure 1 As shown, an embodiment of the present invention provides a two-dimensional sequence splitting system under large model pipeline parallel training, and the system includes:

[0040] The data collection module is used to obtain basic device information and model configuration information, including inter-GPU bandwidth, device video memory size, GPU-CPU bandwidth, model dimension, number of model layers, and input data sequence length.

[0041] A decision maker is used to generate an optimal decision based on the data obtained by the data collection module; wherein the decision content includes the length of redundant sequences, the proportion of redundant sequences stored in the GPU, the proportion of redundant sequences stored in the CPU, and the number of time dimension splits.

[0042] A deep learning training module is used to integrate the optimal decision into the model training process to improve the overall training performance of the system. The deep learning training module includes a long sequence sample splitting submodule and a two-dimensional sequence parallel operator submodule.

[0043] Based on the above system, an embodiment of the present invention further provides a two-dimensional sequence splitting method under large model pipeline parallel training, the method comprising:

[0044] Step 1: When the user deploys the training framework to the cluster and inputs the model and training configuration, the data collection module records the model dimension, number of model layers, and length of the input data sequence. The RingAttention operator is run for several rounds in the sequence parallel group using a sequence with a length of 32k and a dimension of 1024. The FP16 precision computing capability of the GPU device and the communication bandwidth between GPU devices are calculated from the time obtained from the profile when the operator is executed. The above sequence is then copied between the GPU and CPU for several rounds, and the bandwidth between the GPU and CPU is calculated from the time obtained from the profile when copying.

[0045] Step 2: The decision maker module makes dynamic decisions based on the collected information. The specific steps can be divided into the following sub-steps:

[0046] Step 2.1: Initialize the time dimension split number search interval [N low , N high ) is [1, N max ), where N max The maximum number of time dimension splits set by the user, N low is the lower bound of the search interval of the current round of binary search, N max The upper bound of the search interval for the current round of binary search.

[0047] Step 2.2 (Redundancy ratio optimization): Take the interval [N low , N high )Intermediate value N mid In the case of , the sequence is split into N segments according to the standard that each segment has equal computational effort. mid subsequences, and then use linear integer programming to solve the shortest redundant sequence length that needs to be saved and the corresponding GPU redundant sequence ratio and CPU redundant sequence ratio under the condition that the Attention communication overhead is less than or equal to the Attention computation overhead (that is, the communication time can be covered by the computation time).

[0048] The specific constraints and optimization objectives can be expressed by the following formula:

[0049] .

[0050] Among them, L is the total length of the sequence, L local represents the length of redundant sequence, N is the number of time dimension sequence splits, l N is the length of the last subsequence, M is the number of spatial dimension sequence splits (RingAttention parallelism), E is the FP16 computing power of a single GPU (Flops / s), B gpu , B cpu are the inter-GPU communication bandwidth and GPU-CPU communication bandwidth (bytes / s), d is the model dimension, and F a is the Attention calculation amount (Flops) of the last subsequence, p cpu 、p gpu Respectively represent the proportion of redundant sequences stored in CPU memory and GPU memory, M cpu The size of the CPU memory space.

[0051] Step 2.3: Calculate the memory overhead of the current strategy. If it exceeds the memory budget, update the interval [N low , N high ) is [Nlow , N mid ), otherwise, update interval [N low , N high ) is [N mid , N high ). If the number of elements in the interval is 1, return the result, otherwise repeat steps 2.2-2.3.

[0052] Step 3: The deep learning training module first uses the sequence splitting submodule according to the number of time dimension splits N obtained in step 2 to split the input sequence into first subsequences of decreasing length according to the standard that the computational effort of each subsequence is equal, and then further splits each first subsequence into M second subsequences of equal length and distributes the second subsequences to the GPUs in the M sequence parallel groups; the second subsequences obtained after splitting are input into the Transformer large model optimized with a two-dimensional sequence parallel operator for training. When each layer of the Transformer model performs the attention mechanism calculation, the following steps are performed:

[0053] Step 3.1: Calculate the attention of the current Q and the current KV; at the same time, receive the next KV block from the GPU numbered i-1 and send the current KV to the GPU numbered i+1. If there is GPU / CPU redundancy, ignore the redundant part of the data when sending / receiving, and only send / receive the non-redundant part; at this time, if the CPU redundancy ratio is not 0, copy the redundant part of the next KV block from the CPU memory to the GPU video memory at the same time.

[0054] Step 3.2: Update the attention calculation results of the local block.

[0055] Step 3.3: If the KV of the current time period needs to be redundant according to the strategy, save the KV involved in the calculation in 3.1 into a tensor.

[0056] Step 3.4: Update the current KV to the next KV block, and repeat steps 3.1, 3.2, 3.3, and 3.4 until the current Q and each KV block have performed attention mechanism calculations.

[0057] Step 3.5: If the KV of the current time period needs to be redundant according to the policy, the tensor storing the KV in step 3.3 is split into two parts. One part is stored in the GPU according to the ratio specified by the policy, and the remaining KV is asynchronously copied to the CPU memory and the tensor in step 3.3 is released.

[0058] Step 3.6: Finally, if the CPU redundancy ratio is not 0, the first CPU redundancy KV block of the next layer is pre-fetched asynchronously.

[0059] An embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the two-dimensional sequence splitting method under the large model pipeline parallel training.

[0060] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the two-dimensional sequence splitting method under the large model pipeline parallel training is implemented.

[0061] Example 1

[0062] In order to verify the advantages of the present invention compared with the traditional method, the present invention provides the following specific embodiments, wherein Megatron-LM is one of the most widely used pipeline parallel training frameworks (sequence-free splitting optimization technology), RingAttention (sequence splitting in spatial dimension) is one of the most advanced sequence parallel technologies, and Seq1F1B is one of the most advanced technologies for splitting sequences in the time dimension. The specific experiments are as follows:

[0063] Experimental configuration:

[0064] (1) Operating system: Ubuntu 22.04 LTS;

[0065] (2) CPU: 128-core AMD EPYC 7763 CPU with 1TB DRAM;

[0066] (3) GPU: 4*NVIDIA A800 80GB video memory.

[0067] Model configuration:

[0068] (1) Model: llama2-7B;

[0069] (2) Dataset: LongAlpaca, the average length of the samples used in the test is 8k;

[0070] (3) Parallel configuration: PP (pipeline parallelism) = 2; CP (RingAttention sequence parallelism) = 2;

[0071] (4) Training configuration: BatchSize=4, MicrobatchSize=1;

[0072] (5) Memory limit: The CPU memory usage of a single card is limited to 20 GB.

[0073] Test indicators:

[0074] Average throughput per GPU (tokens / s).

[0075] Final test results:

[0076] The final test results are as follows Figure 2 As shown in the figure: the average throughput of Megatron-LM without any sequence parallel method is 1414 tokens / s; the throughput of using only the RingAttention algorithm in the training pipeline is 1528 tokens / s; the throughput of using only Seq1F1B in the training pipeline is 1498 tokens / s; the throughput of the simple combination of the two is 1465 tokens / s; the average throughput of the method of the present invention can reach 1556 tokens / s. Compared with no splitting, the two-dimensional sequence splitting parallelism improves the throughput by about 10%, and improves the throughput by 2-5% compared with other single-dimensional or simple combination schemes.

[0077] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A two-dimensional sequence splitting method under large model pipeline parallel training, characterized in that: The steps include: S1. Use real RingAttention load test and collect inter-GPU bandwidth, device video memory size, device CPU memory size, GPU-CPU bandwidth, model dimension, number of model layers, and input data sequence length; S2. Making intelligent decisions based on the information collected in step S1, wherein the decision content includes the length of the redundant sequence, the proportion of the redundant sequence stored in the GPU, the proportion of the redundant sequence stored in the CPU, and the number of time dimension splits; S3. Split the input sequence into first subsequences of decreasing length according to the number of splits in the time dimension and the length of each subsequence, further split the subsequence into M second subsequences of equal length, and distribute the M second subsequences to the GPUs in the M sequence parallel groups; input the second subsequences into the Transformer large model optimized by the two-dimensional sequence parallel operator for training; wherein, during the training process, the redundant sequence length, the proportion of redundant sequences stored in the GPU, and the proportion of redundant sequences stored in the CPU are used as parameters to be input into the two-dimensional sequence parallel operator, and during training, the following are guided by the above parameters: saving redundant sequences to the GPU, transferring redundant sequences from the GPU to the CPU, and transferring redundant sequences from the CPU to the GPU.

2. According to claim 1, a two-dimensional sequence splitting method under large model pipeline parallel training is characterized in that: The step S1 is specifically as follows: When the user deploys the training framework to the cluster and inputs the model and training configuration, the model dimension, number of model layers, and length of the input data sequence are recorded, and a sequence of arbitrary length and dimension is used to run the RingAttention operator for several rounds in the sequence parallel group. The FP16 precision computing capability of the GPU device and the communication bandwidth between GPU devices are calculated based on the time obtained from the profile when the operator is executed. The above sequence is then used to copy several rounds between the GPU and CPU, and the bandwidth between the GPU and CPU is calculated based on the time obtained from the profile when copying.

3. According to claim 2, a two-dimensional sequence splitting method under large model pipeline parallel training is characterized in that: The RingAttention operator is run for several rounds in the sequence parallel group using sequences of arbitrary length and dimension, and a sequence with a length of 32k and a dimension of 1024 is selected.

4. The two-dimensional sequence splitting method under large model pipeline parallel training according to claim 1 is characterized in that: The step S2 comprises: S2.1: Initialize the time dimension split number search interval [N low , N high ) is [1, N max ), where N max The maximum number of time dimension splits set by the user, N low is the lower bound of the search interval of the current round of binary search, N max The upper bound of the search interval for the current round of binary search; S2.2: In the time dimension, the number of splits N is taken as the interval [N low , N high )Intermediate value N mid In the case of , the sequence is split into N segments according to the standard that the amount of calculation in each segment is equal mid subsequences; under the condition that the Attention communication overhead is less than or equal to the Attention computation overhead, linear integer programming is used to solve the shortest redundant sequence length, the corresponding GPU redundant sequence ratio and the CPU redundant sequence ratio; S2.3: Calculate the memory overhead of the current decision. If it exceeds the memory budget, update the interval [N low , N high ) is [N low , N mid ), otherwise, update interval [N low , N high ) is [N mid , N high ); If the number of elements in the interval is 1, return the result, otherwise repeat steps S2.2-S2.

3.

5. According to claim 4, a two-dimensional sequence splitting method under large model pipeline parallel training is characterized in that: The step S2.2 includes: the constraints and optimization targets for searching the shortest redundant sequence length and the optimal CPU to GPU redundant sequence ratio under the shortest redundant sequence length are shown in the following formula: ； Among them, L is the total length of the sequence, L local represents the length of redundant sequence, N is the number of time dimension sequence splits, l N is the length of the last subsequence, M is the number of spatial dimension sequence splits, E is the FP16 computing power of a single GPU, and B gpu , B cpu are the inter-GPU communication bandwidth and the GPU-CPU communication bandwidth respectively, d is the model dimension, and F a Attention calculation amount for the last subsequence, p cpu 、p gpu Respectively represent the proportion of redundant sequences stored in CPU memory and GPU memory, M cpu The size of the CPU memory space.

6. The two-dimensional sequence splitting method under large model pipeline parallel training according to claim 1 is characterized in that: The step S3 includes: performing the following steps when each layer of the Transformer model performs attention mechanism calculation: S3.1: Calculate the attention of the current Q and the current KV; at the same time, receive the next KV block from the GPU numbered i-1 and send the current KV to the GPU numbered i+1. If there is GPU / CPU redundancy, ignore the redundant part of the data when sending / receiving, and only send / receive the non-redundant part; if the CPU redundancy ratio is not 0 at this time, copy the redundant part of the next KV block from the CPU memory to the GPU memory at the same time; S3.2: Update the attention calculation results of the local block; S3.3: If according to the decision, the KV of the current time period needs to be redundant, the KV involved in the calculation in step S3.1 is saved into a tensor; S3.4: Update the current KV to the next KV block, and repeat steps S3.1-S3.4 until the current Q and each KV block have performed attention mechanism calculations; S3.5: If, according to the decision, the KV of the current time period needs to be redundant, the tensor storing the KV in step S3.3 is split into two parts, wherein one part is stored in the GPU according to the ratio specified by the decision, and the remaining part of the KV is asynchronously copied to the CPU memory and then the tensor in step S3.3 is released; S3.6: If the CPU redundancy ratio is not 0, asynchronously pre-fetch the first CPU redundancy KV block of the next layer.

7. A two-dimensional sequence segmentation system under large model pipeline parallel training, characterized in that: The system comprises: The data collection module is used to obtain basic device information and model configuration information, including inter-GPU bandwidth, device memory size, GPU-CPU bandwidth, model dimension, number of model layers, and input data sequence length; A decision maker, used to generate an optimal decision based on the data acquired by the data collection module; wherein the decision content includes the length of redundant sequences, the proportion of redundant sequences stored in the GPU, the proportion of redundant sequences stored in the CPU, and the number of time dimension splits; A deep learning training module is used to integrate the optimal decision into the model training process to improve the overall training performance of the system. The deep learning training module includes a long sequence sample splitting submodule and a two-dimensional sequence parallel operator submodule.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement a two-dimensional sequence splitting method under large model pipeline parallel training as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a two-dimensional sequence splitting method under large model pipeline parallel training as described in any one of claims 1 to 6 is implemented.

Images