Sequence parallel attention processing method and apparatus

Abstract

This application provides a method, apparatus, computer device, computer-readable storage medium, and computer program product for sequence parallel attention processing, belonging to the field of Internet technology. The method includes: acquiring target tensor data for sequence parallel attention computation to be performed; dividing the target tensor data into multiple target header blocks according to the attention head dimension; using the target header blocks as the scheduling granularity, controlling the input all-to-all communication, attention computation, and output all-to-all communication of each target header block to form a pipelined parallel execution; merging the output blocks obtained from the output all-to-all communication to obtain the target output result. The technical solution of this application embodiment can improve the synergistic utilization of communication and computing resources during sequence parallel attention processing, thereby improving processing efficiency.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a sequential parallel attention processing method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the increasing application of Transformer-type models in natural language processing, image generation, and multimodal processing, sequence-parallel attention processing is widely used in scenarios where multiple computing devices collaboratively perform attention computations. In some all-to-all sequence-parallel attention implementations, multiple computing devices typically need to first perform all-to-all input communication to exchange data corresponding to the current attention computation among the devices before performing attention computation and then performing all-to-all output communication to restore the data layout required for subsequent processing. In existing technologies, in scenarios with large model sizes, long sequence lengths, or collaborative processing by multiple computing devices, computing resources may be in a waiting state, which is not conducive to the coordinated utilization of communication and computing resources and may lead to an increase in the overall latency of sequence-parallel attention processing.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0004] This application provides a sequential parallel attention processing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems mentioned above.

[0005] One aspect of this application provides a sequential parallel attention processing method applied to a target computing device, the method comprising: Obtain the target tensor data for parallel attention computation of the sequence to be executed, wherein the target tensor data includes query tensors, key tensors, and value tensors; The target tensor data is divided into multiple target header blocks according to the attention head dimension; Using the target header block as the scheduling granularity, the input all-to-all communication, attention calculation, and output all-to-all communication of each target header block are controlled to form a pipelined parallel execution. The output blocks obtained from the full-to-full communication are merged to obtain the target output result.

[0006] Optionally, the target tensor data for parallel attention computation of the sequence to be executed is obtained, including: Obtain the input slice; Perform a linear projection operation on the input fragment to obtain the query tensor, the key tensor, and the value tensor.

[0007] Optionally, the method further includes: Position encoding is performed on the query tensor and the key tensor to inject sequence position information into the query tensor and the key tensor, respectively.

[0008] Optionally, the target tensor data is divided into multiple target header blocks according to the attention head dimension, including: Determine the target number of blocks; The query tensor, the key tensor, and the value tensor are divided along the attention head dimension according to the target number of blocks to obtain multiple query sub-blocks, multiple key blocks, and multiple value sub-blocks; The query sub-blocks, key sub-blocks, and value sub-blocks with the same block number are identified as a target header block, wherein the block number is the sorting number of the target header block.

[0009] Optionally, the target number of blocks is determined, including: Obtain pre-acquired or measured communication time parameters, single-block overhead parameters, and communication ratio parameters; The target number of blocks is determined based on the communication time parameter, the single-block overhead parameter, and the communication ratio parameter.

[0010] Optionally, the method further includes: The number of target blocks is constrained based on the number of attention heads corresponding to each computing device.

[0011] Optionally, dividing the target tensor data into multiple target head blocks according to the attention head dimension includes: If the number of attention heads corresponding to each computing device is not divisible by the number of target blocks, the multiple target head blocks are unevenly distributed along the attention head dimension so that the number of attention heads included in the first few target head blocks is one more than that in the remaining target head blocks.

[0012] Optionally, the step of controlling the input all-to-all communication, attention calculation, and output all-to-all communication of each target head block to form a pipelined parallel execution, with the target head block as the scheduling granularity, includes: In the first execution flow, input all-to-all communication and output all-to-all communication are performed for each of the target header blocks; In the second execution flow, attention calculations are performed on each of the target header blocks.

[0013] Optionally, the method further includes: When the input all-to-all communication is completed for the c-th target header block, record the first event corresponding to the c-th target header block; After waiting for the first event corresponding to the c-th target header block in the second execution flow, attention calculation is performed on the c-th target header block; When completing the attention calculation for the c-th target head block, record the second event corresponding to the c-th target head block; After waiting for the second event corresponding to the c-th target header block in the first execution flow, output all-to-all communication is performed on the c-th target header block.

[0014] Optionally, in the presence of adjacent target header blocks, during the attention calculation for the c-th target header block, the first execution flow performs input all-to-all communication for the (c+1)-th target header block and / or output all-to-all communication for the (c-1)-th target header block.

[0015] Optionally, the method further includes: Record the startup event in the main execution flow; After waiting for the start event in the first execution flow and the second execution flow, in the first execution flow, input all-to-all communication and output all-to-all communication are performed for each of the target header blocks respectively, and in the second execution flow, attention calculation is performed for each of the target header blocks respectively.

[0016] Optionally, the output blocks obtained from the full-to-full communication are merged to obtain the target output result, including: Before merging the output blocks, synchronize the first execution flow and the second execution flow; The output blocks are concatenated according to the attention head dimension to obtain the attention output tensor.

[0017] Optionally, the method further includes: Establish block-level cache structures for the key tensors and value tensors respectively according to the target header blocks; When processing the corresponding target header block, the key cache and / or value cache corresponding to that target header block are invoked.

[0018] Optionally, the method further includes: During the processing of multiple target header blocks, the communication buffers corresponding to different target header blocks are reused.

[0019] Optionally, the method further includes: Release the intermediate tensor corresponding to the preceding target header block during subsequent target header block processing.

[0020] Optionally, the method further includes: The sequence parallel processing based on all-to-all is combined with tensor parallel processing, wherein the sequence parallel processing splits the data along the sequence length dimension, and the tensor parallel processing splits the data along the attention head dimension, and the two are orthogonally parallel in different dimensions.

[0021] Another aspect of this application provides a sequential parallel attention processing apparatus, applied to a target computing device, the apparatus comprising: The acquisition module is used to acquire the target tensor data for parallel attention computation of the sequence to be executed, wherein the target tensor data includes query tensors, key tensors, and value tensors; The partitioning module is used to divide the target tensor data into multiple target header blocks according to the attention head dimension; The control module is used to control the input all-to-all communication, attention calculation and output all-to-all communication of each target head block to form a pipelined parallel execution with the target head block as the scheduling granularity; The merging module is used to merge the output blocks obtained from the full-to-full communication to obtain the target output result.

[0022] Another aspect of this application provides a computer device, including: At least one processor; and A memory that is communicatively connected to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0023] Another aspect of this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.

[0024] Another aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method described above.

[0025] The embodiments of this application employing the above-described technical solution may have the following advantages: By acquiring the target tensor data for parallel attention computation of the sequence to be executed, and dividing the target tensor data into multiple target header blocks according to the attention head dimension, the processing process originally performed on the entire tensor data can be refined into a processing process performed on multiple target header blocks, thus providing a basis for the segmented organization of processing corresponding to different target header blocks. Furthermore, by using the target header block as the scheduling granularity, controlling the input all-to-all communication, attention computation, and output all-to-all communication of each target header block to form a pipelined parallel execution, the communication processing and computation processing corresponding to different target header blocks can overlap during execution, reducing resource waiting caused by completely sequential execution of input all-to-all communication, attention computation, and output all-to-all communication. This is beneficial for improving the synergistic utilization of communication and computational resources in the sequential parallel attention processing, increasing processing efficiency, and reducing processing latency caused by sequential execution. Finally, by merging the output blocks obtained from the output all-to-all communication, a target output result can be formed for use in subsequent processing flows. Attached Figure Description

[0026] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0027] Figure 1 The diagram schematically illustrates the operating environment of the sequence parallel attention processing method according to Embodiment 1 of this application; Figure 2 A flowchart illustrating a sequence parallel attention processing method according to Embodiment 1 of this application is shown schematically. Figure 3 Schematic illustration Figure 2 Detailed flowchart of step S200; Figure 4 Schematic illustration Figure 2 Detailed flowchart of step S202; Figure 5 Schematic illustration Figure 2 Detailed flowchart of step S204; Figure 6 The flowchart illustrating the additional steps of the sequence parallel attention processing method according to Embodiment 1 of this application is shown in the schematic diagram. Figure 7 Schematic illustration Figure 2 Detailed flowchart of step S206 in the document; Figure 8 A block diagram of a sequence parallel attention processing apparatus according to Embodiment 2 of this application is schematically shown; and Figure 9 A schematic diagram of the hardware architecture of a computer device according to Embodiment 3 of this application is shown. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0029] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0030] It should be noted that, in any stage of this application involving the collection, storage, use, transmission, and processing of data, each stage strictly adheres to the laws, regulations, industry standards, and regulatory requirements of the data source, usage location, and relevant countries and regions to ensure the legality and compliance of data activities. In the collection stage, the purpose, method, and scope of collection are clearly communicated to the data subject in a prominent manner. Collection is conducted only after obtaining the data subject's legal authorization, ensuring that the collection process follows the "minimum necessary" principle and does not exceed the scope of data collection. In the storage stage, storage periods are limited, and data is promptly deleted or anonymized / encrypted after the storage purpose is achieved. In the usage stage, a strict data security protection mechanism is implemented, using field-level desensitization technology and processing the original data according to preset desensitization rules. For different types of data, multiple desensitization strategies, such as data generalization, data anonymization, and data encryption, are employed to effectively mitigate the risk of sensitive information leakage and ensure that all data used is securely processed and desensitized, comprehensively protecting the rights and interests of data subjects and data security. In the transmission and processing stages, the confidentiality and security of data are ensured during transmission and processing.

[0031] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0032] First, a definition of the terminology used in this application is provided: Input all-to-all communication: Used to convert the data layout of each target header block from a sequence-fragmented layout with a complete header to a sequence-complete layout with fragmented headers.

[0033] Output all-to-all communication: Used to convert the data layout of the attention calculation results corresponding to each target header block from a layout with complete sequence and header fragments to a layout with sequence fragments and complete header.

[0034] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below: With the increasing application of Transformer-type models in natural language processing, image generation, and multimodal processing, sequential parallel attention processing is widely used in scenarios where multiple computing devices collaboratively perform attention computations. In some all-to-all sequential parallel attention implementations, multiple computing devices typically need to first perform input all-to-all communication to exchange data corresponding to the current attention computation among the multiple computing devices before performing attention computation, and then perform output all-to-all communication to restore the data layout required for subsequent processing.

[0035] Taking single-layer attention forward propagation as an example, the entire execution process is as follows: Step 1, QKV projection (calculation); Step 2: Input all-to-all communication: Change the data layout from [B, L / P, H, D] to [B, L, H / P, D]; Step 3, Attention Calculation (Calculation); Step 4: Output all-to-all communication: Layout conversion back to [B, L / P, H, D]; Step 5: Output projection (calculation).

[0036] The above execution process uses a serial mode, which is significantly inefficient in terms of resources. Existing GPU architectures have independent compute engines (SMs) and communication engines, theoretically supporting parallel execution of computation and communication. However, in serial mode, TensorCores / CUDA Cores are idle during all-to-all computations; and the GPU's network communication capabilities are not utilized during attention computations, resulting in wasted hardware resources.

[0037] To address this, this application provides a technical solution for sequential parallel attention processing. In this solution, by acquiring the target tensor data for sequential parallel attention computation and dividing it into multiple target header blocks according to the attention head dimension, the processing originally performed on the entire tensor data can be refined into processing performed on multiple target header blocks, thus providing a basis for segmented organization of processing corresponding to different target header blocks. Furthermore, by using the target header block as the scheduling granularity, controlling the input all-to-all communication, attention computation, and output all-to-all communication of each target header block to form a pipelined parallel execution, the communication and computation processing corresponding to different target header blocks can overlap during execution, reducing resource waiting caused by completely sequential execution of input all-to-all communication, attention computation, and output all-to-all communication. This is beneficial for improving the synergistic utilization of communication and computation resources during sequential parallel attention processing, increasing processing efficiency, and reducing processing latency caused by sequential execution. Finally, by merging the output blocks obtained from the output all-to-all communication, a target output result can be formed for use in subsequent processing flows. See below for details.

[0038] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0039] like Figure 1 As shown, the runtime environment diagram includes: like Figure 1 As shown in the diagram, the environment includes a service platform 2, a network 4, and a client 6, wherein: Service platform 2 can consist of one or more computing devices. These computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as emulations of computer systems, operating systems, servers, etc. The computing devices can load virtual machines based on virtual images and / or other data that define specific software used for emulation (e.g., operating systems, dedicated applications, servers). As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.

[0040] Service platform 2 can be configured to communicate with clients 6, etc., via network 4. Network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar devices. Network 4 may include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, or combinations thereof, or wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0041] Service platform 2 can provide services such as storage, reading, writing, querying, and deleting, such as providing sequential parallel attention processing services to clients.

[0042] Client 6 can be an electronic device running operating systems such as Windows, Android™, or iOS, including smartphones, tablets, laptops, virtual reality devices, gaming devices, set-top boxes, in-vehicle terminals, and smart TVs. Based on these operating systems, various applications can run, such as sequence-parallel attention processing applications.

[0043] Client 6 can provide / configure user access pages, which can be used to control service platform 2 or upload objects, etc.

[0044] It should be noted that the above-mentioned equipment is exemplary, and the number and type of equipment can be adjusted in different scenarios or according to different needs.

[0045] The technical solution of this application will be described below through multiple embodiments, taking the target computing device in service platform 2 as the execution subject. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described herein.

[0046] Example 1 Figure 2 A flowchart illustrating a sequential parallel attention processing method according to Embodiment 1 of this application is shown schematically.

[0047] like Figure 2 As shown, this sequential parallel attention processing method can be applied to target computing devices, which can be hardware such as GPUs, AI acceleration chips, and distributed servers that have distributed parallel computing and distributed communication capabilities.

[0048] The method may include steps S200 to S206, wherein: Step S200: Obtain the target tensor data for parallel attention computation of the sequence to be executed.

[0049] Step S202: Divide the target tensor data into multiple target header blocks according to the attention head dimension.

[0050] Step S204: Using the target header block as the scheduling granularity, control the input all-to-all communication, attention calculation and output all-to-all communication of each target header block to form a pipelined parallel execution.

[0051] Step S206: Merge the output blocks obtained from the full-to-full communication to obtain the target output result.

[0052] The sequence-parallel attention processing method provided in this embodiment acquires the target tensor data for the sequence-parallel attention computation to be performed, and divides the target tensor data into multiple target head blocks according to the attention head dimension. This refines the processing process originally performed on the entire tensor data into a processing process performed on multiple target head blocks, thus providing a basis for the segmented organization of processing corresponding to different target head blocks. Furthermore, by using the target head block as the scheduling granularity, controlling the input all-to-all communication, attention computation, and output all-to-all communication of each target head block to form a pipelined parallel execution, the communication processing and computation processing corresponding to different target head blocks can overlap during execution, reducing resource waiting caused by the completely sequential execution of input all-to-all communication, attention computation, and output all-to-all communication. This is beneficial to improve the synergistic utilization of communication and computation resources in the sequence-parallel attention processing, improve processing efficiency, and reduce processing latency caused by sequential execution. Finally, by merging the output blocks obtained from the output all-to-all communication, a target output result can be formed for use in subsequent processing flows.

[0053] The following combination Figure 2 The steps in steps S200 to S206, as well as other optional steps, are described in detail.

[0054] Step S200 Obtain the target tensor data for parallel attention computation of the sequence to be executed.

[0055] In this embodiment, the target computing device can receive target tensor data that needs to perform sequence parallel attention computation through a preset data interface. The target tensor data is a multidimensional tensor that conforms to the attention operation specification and can contain core parameters such as sequence length, attention head, and feature dimension. It is the basic data for subsequent segmentation, communication, and computation, and there is no need to add extra irrelevant dimensions or data types.

[0056] The target tensor data may include query tensors (Q tensors), key tensors (K tensors), and value tensors (V tensors).

[0057] In this embodiment, the target tensor data refers to the query tensor (Q tensor), key tensor (K tensor), and value tensor (V tensor) obtained after QKV linear projection in the Transformer attention layer. These are the only input data for sequence parallel communication rearrangement and distributed attention computation.

[0058] The query tensor is used to initiate feature matching requests, the key tensor is used to provide indexes for feature matching, and the value tensor is used to provide raw data for feature weighting. The three work together to complete the core process of attention calculation.

[0059] In one implementation, in order to ensure that the three types of tensors can be executed synchronously during subsequent segmentation, communication and computation processes, and to avoid computational errors caused by dimension mismatch, the dimension specifications of the query tensor, key tensor and value tensor need to be completely consistent, all of which contain the same sequence length dimension, attention head dimension and single head feature dimension.

[0060] It should be noted that the shapes of the query tensor (Q tensor), key tensor (K tensor), and value tensor (V tensor) are uniformly [B, L / P, H, D], where B is the batch size (number of independent samples), L / P is the length of the local sequence held by a single GPU, L is the total global sequence length (total number of tokens), P is the total number of GPUs (number of parallel sequence GPUs), H is the total number of attention heads, and D is the dimension of a single attention head.

[0061] In an optional implementation, see [link to relevant documentation]. Figure 3 Step S200 may include: Step S300: Obtain the input fragment; Step S302: Perform a linear projection operation on the input fragment to obtain the query tensor, the key tensor, and the value tensor.

[0062] The target computing device can obtain the input fragment to be processed through the data transmission interface or read the input fragment from local storage; the input fragment contains one or more sequence fragments. Among them, the sequence fragment is a small sub-sequence segment that is uniformly divided along the sequence length L of the global total sequence (all tokens of a single sample) and allocated to a single GPU. It is the original basis for the subsequent generation of the three types of tensors.

[0063] As an example, a global sequence of total length L contains all tokens (0 to L-1). Assuming there are P GPUs participating in the parallel computation of the sequence, the sequence slice will contain a small continuous token slice of length L / P.

[0064] In this embodiment, the target computing device can call the built-in linear projection operator, load the preset projection weights and bias parameters, and perform a linear projection operation on the obtained input slice. Through linear mapping transformation, the feature dimensions of the input slice are converted into the specifications required for attention calculation. Finally, query tensor, key tensor and value tensor are generated synchronously. The generation process of the three types of tensors is completed synchronously, and the dimensions are completely matched.

[0065] This embodiment generates query tensors, key tensors, and value tensors by inputting fragments, which not only simplifies the tensor data generation process but also reduces computational steps and computational overhead.

[0066] In an optional implementation, the method further includes: Position encoding is performed on the query tensor and the key tensor to inject sequence position information into the query tensor and the key tensor, respectively.

[0067] In this embodiment, a preset position encoding algorithm can be invoked to perform position encoding operations on the query tensor and the key tensor respectively, so as to inject the sequence position information into the query tensor and the key tensor respectively, ensuring that the tensor can capture the temporal relationship of the sequence without changing the original dimensional structure and core features of the tensor.

[0068] After completing the position encoding process, the updated query tensor and key tensor are used as input data for subsequent block partitioning, communication, and attention calculation.

[0069] This embodiment performs position encoding only on the query tensor and key tensor, without encoding the value tensor. This avoids redundant calculations, reduces computational overhead, and ensures that the original features of the value tensor are not destroyed. By injecting sequence position information into the query and key tensors through position encoding, the problem of not being able to distinguish sequence positions in attention calculation can be effectively solved, enabling attention weight calculation to accurately reflect the temporal correlation of the sequence and improve the accuracy of attention calculation.

[0070] Step S202 The target tensor data is divided into multiple target header blocks according to the attention head dimension.

[0071] In this embodiment, the attention head dimension is used as the sole dividing criterion, and no division is made in the sequence dimension or feature dimension. According to the preset dividing rules, the complete target tensor data is split into multiple independent target head blocks. Each target head block contains a part of the attention head and retains complete sequence information and feature information, ensuring that a single head block can independently participate in subsequent communication and computation.

[0072] In an optional implementation, see [link to relevant documentation]. Figure 4 Step S202 may include: Step S400: Determine the target number of blocks.

[0073] In this embodiment, the target number of blocks for this segmentation can be determined in advance by combining parameters such as the computing power, communication bandwidth, and total number of attention heads of the target computing device. This target number of blocks serves as the benchmark for subsequent tensor segmentation.

[0074] Step S402: The query tensor, the key tensor, and the value tensor are divided along the attention head dimension according to the target number of blocks to obtain multiple query sub-blocks, multiple key blocks, and multiple value sub-blocks.

[0075] In this embodiment, the query tensor, key tensor, and value tensor are synchronously segmented strictly along the attention head dimension according to the determined target number of blocks. After segmentation, the query tensor obtains N query sub-blocks, the key tensor obtains N key sub-blocks, and the value tensor obtains N value sub-blocks (N is the target number of blocks). The segmentation rules of the three types of tensors are completely consistent, and the attention head indices corresponding to the sub-blocks are one-to-one.

[0076] Step S404: The query sub-blocks, key sub-blocks and value sub-blocks with the same block number are determined as a target header block, wherein the block number is the sorting number of the target header block.

[0077] In this embodiment, the segmented query sub-blocks, key sub-blocks, and value sub-blocks are matched according to their block numbers, and a query sub-block, a key sub-block, and a value sub-block with the same number are bound into an independent target header block; each target header block contains complete sequence information, feature information, and a portion of attention head, and has the ability to independently perform subsequent communication and computation.

[0078] This embodiment divides the target header block partitioning process into three steps: determining the number of target blocks, synchronous tensor splitting, and header block combination. This avoids confusion during the partitioning process, ensures that the query, key, and value tensors in each target header block can be accurately matched, avoids attention calculation errors caused by misalignment of the splitting, and generates independent target header blocks by combining sub-blocks, thus achieving complete decoupling of each header block.

[0079] In an optional implementation, determining the target number of blocks includes: Obtain pre-acquired or measured communication time parameters, single-block overhead parameters, and communication ratio parameters; determine the target number of blocks based on the communication time parameters, the single-block overhead parameters, and the communication ratio parameters.

[0080] In this embodiment, the communication time parameter includes the communication time for input all-to-all communication and the communication time for output all-to-all communication.

[0081] The single-block overhead parameter β is the sum of the fixed small overheads generated by a single head block in the pipeline, which may include fixed delays caused by CUDA event logging and waiting, kernel function startup, multi-stream scheduling, inter-block synchronization, etc.

[0082] The communication ratio parameter ρ represents the proportion of the total global communication time Tcomm in the attention layer to the total execution time Ttotal of that layer. It is used to measure the proportion of communication overhead, and the calculation formula is: ρ=Tcomm / Ttotal.

[0083] In this embodiment, the optimal number of target blocks can be determined by the following formula: ;in, To determine the optimal number of target blocks, Tcomm This represents the total global communication time. β This is the single-block overhead parameter.

[0084] In this embodiment, the communication ratio parameter ρ determines the maximum number of target blocks that can be divided. For example, if ρ < 10%, no blocks are formed (C = 1); if 10% ≤ ρ < 20%, a small number of blocks are formed (C = 2-3); if ρ ≥ 20%, sufficient blocks can be formed (C = 3~6). Therefore, in this embodiment, after calculating the optimal number of target blocks using the above formula, it is necessary to determine the final number of target blocks based on the communication ratio parameter ρ. The final number of target blocks can be the integer closest to C within the allowed range of ρ.

[0085] This embodiment determines the target number of blocks based on the communication time parameter, the single-block overhead parameter, and the communication ratio parameter, avoiding the problems of low parallel efficiency or excessive scheduling overhead caused by manually setting the number of blocks. Through comprehensive judgment of multiple parameters, it can adapt to different device computing power and communication bandwidth scenarios, determine the optimal target number of blocks, and ensure that subsequent pipeline parallelism can maximize its advantages.

[0086] In an optional implementation, the method further includes: The number of target blocks is constrained based on the number of attention heads corresponding to each computing device.

[0087] In this embodiment, the target number of blocks can be constrained as follows: 1≤C≤H / P, where C is the number of target blocks, P is the total number of GPUs (sequential parallel cards) for each computing device, and H is the total number of attention heads.

[0088] In this embodiment, by constraining the target number of blocks, excessive splitting, invalid empty blocks, and additional invalid scheduling overhead can be avoided.

[0089] In an optional implementation, dividing the target tensor data into multiple target head blocks according to the attention head dimension includes: when the number of attention heads corresponding to each computing device is not divisible by the number of target blocks, distributing the multiple target head blocks unevenly along the attention head dimension so that the number of attention heads included in the first few target head blocks is one more than the number of other target head blocks.

[0090] In this embodiment, after determining the target number of blocks, it checks whether the number of attention heads corresponding to each computing device is divisible by the target number of blocks; if it is not divisible, uneven distribution is performed; if it is divisible, the head blocks are divided according to the uniform distribution method.

[0091] The uneven distribution process is as follows: First, divide the number of attention heads by the number of target blocks to obtain the basic allocation number of attention heads and the remainder number of attention heads. Then, allocate each basic allocation number of attention heads to each target head block. Finally, allocate the remainder number of attention heads to the first few target head blocks in sequence, so that the number of attention heads contained in the first few target head blocks is one more than that in the other target head blocks.

[0092] As an example, assuming H=10 (a total of 10 attention heads) and C=4 (to be divided into 4 head blocks), the basic allocation of attention heads is 2, and the remainder of attention heads is also 2. Then, the 2 attention heads will be evenly divided into the 1st, 2nd, 3rd, and 4th target head blocks. Finally, the remainder of attention heads will be allocated to the 1st and 2nd target head blocks, so that the 1st and 2nd target head blocks are ultimately allocated 3 attention heads, and the 3rd and 4th target head blocks are allocated 2 attention heads.

[0093] This embodiment addresses the special scenario where the number of attention heads on each computing device cannot be divided evenly by the number of target blocks. It employs an uneven distribution method, eliminating the need to pad or discard attention heads, thus avoiding invalid calculations and precision deviations, ensuring the integrity of attention calculations. Furthermore, by distributing the remaining attention heads to the first few head blocks, the difference in the number of attention heads among the target head blocks is kept within one level, maximizing the load balance among the head blocks.

[0094] Step S204 Using the target header block as the scheduling granularity, the input all-to-all communication, attention calculation, and output all-to-all communication of each target header block are controlled to form a pipelined parallel execution.

[0095] In this embodiment, a single target header block is used as the smallest scheduling unit. For each target header block, the three core steps of "input full-to-full communication → attention calculation → output full-to-full communication" are executed sequentially. Through a multi-execution flow scheduling mechanism, the above three steps of different target header blocks are controlled to overlap in time and advance in parallel, forming a pipeline execution mode, avoiding the overall efficiency decline caused by the blockage of a single step.

[0096] In an optional implementation, see [link to relevant documentation]. Figure 5 Step S204 may include: Step S500: In the first execution flow, input all-to-all communication and output all-to-all communication are performed for each of the target header blocks.

[0097] In step S502, attention calculation is performed on each of the target header blocks in the second execution flow.

[0098] In this embodiment, two independent execution flows, namely a first execution flow and a second execution flow, can be created in the target computing device. The two execution flows can run asynchronously and in parallel without interfering with each other. These two independent execution flows can be created using an execution flow cache corresponding to the current target computing device.

[0099] In the first execution flow, for each target header block, two operations, input all-to-all communication and output all-to-all communication, are executed sequentially according to the header block number to ensure that the communication operation of each header block is executed completely. In the second execution flow, for each target header block, attention calculation is executed sequentially according to the header block number, which overlaps with the communication operation of the first execution flow in time, thus achieving pipelined parallelism.

[0100] The input all-to-all communication is used to convert the data layout of each target header block from a sequence-fragmented and header-complete layout to a sequence-complete and header-fragmented layout, that is, to change the data layout from [B, L / P, H, D] to [B, L, H / P, D].

[0101] The output full-to-full communication is used to convert the data layout of the attention calculation results corresponding to each target head block from a sequence-complete and head-fragmented layout to a sequence-fragmented and head-complete layout, that is, to change the data layout from [B, L, H / P, D] back to [B, L / P, H, D].

[0102] It should be noted that B (Batch Size) is the number of independent samples processed in parallel at one time, L is the total global sequence length, P is the total number of GPUs per computing device (number of sequence parallel cards), H is the total number of attention heads, and D is the dimension of the model's hidden layers.

[0103] This embodiment separates communication and computation operations by using two independent execution streams, achieving hardware-level parallelism between communication and computation. This completely breaks the problem of communication blocking computation in traditional serial execution. The two execution streams focus on communication and computation respectively, which can improve the execution efficiency of a single operation. At the same time, through the timing overlap design, the communication and computation operations of different header blocks are carried out synchronously, which greatly masks the communication latency and improves the overall computing efficiency.

[0104] In an optional implementation, see [link to relevant documentation]. Figure 6 The method may further include, Step S600: When the input all-to-all communication is completed for the c-th target header block, record the first event corresponding to the c-th target header block.

[0105] In this embodiment, for the c-th target header block (c is the target header block number, which increments from 1), after the input all-to-all communication of the header block is completed in the first execution flow, the first event corresponding to the target header block is immediately recorded to mark that the input communication of the header block has been completed and the data is ready.

[0106] Step S602: After waiting for the first event corresponding to the c-th target header block in the second execution flow, perform attention calculation on the c-th target header block.

[0107] In this embodiment, in the second execution flow, the first event corresponding to the c-th target header block is continuously monitored. After the event is triggered, attention calculation operation is immediately performed on the c-th target header block to ensure that the calculation operation is started only after the input communication is completed, so as to avoid calculation errors caused by data not being ready.

[0108] Step S604: When completing the attention calculation for the c-th target head block, record the second event corresponding to the c-th target head block.

[0109] In this embodiment, after the attention calculation of the c-th target header block is completed in the second execution flow, the second event corresponding to the target header block is immediately recorded to mark that the calculation operation of the header block has been completed.

[0110] Step S606: After waiting for the second event corresponding to the c-th target header block in the first execution flow, perform output all-to-all communication on the c-th target header block.

[0111] In this embodiment, in the first execution flow, the second event corresponding to the c-th target header block is continuously monitored. After the event is triggered, the c-th target header block is immediately subjected to an output all-to-all communication operation to ensure that the output communication is started only after the calculation is completed, thus ensuring the accuracy of the output data.

[0112] This embodiment, through a header-level event marking and waiting mechanism, can ensure the strict serial dependency of "input communication → computation → output communication" within a single target header block, avoiding computational errors caused by out-of-order operations and guaranteeing the correctness of single header block computation. At the same time, this event synchronization mechanism realizes precise coordination between the first execution flow and the second execution flow, without disrupting the computational logic of a single header block or affecting pipeline overlap between different header blocks.

[0113] In an optional implementation, when there are adjacent target header blocks, during the attention calculation for the c-th target header block, the first execution flow performs input all-to-all communication for the (c+1)-th target header block and / or output all-to-all communication for the (c-1)-th target header block.

[0114] In this embodiment, during the second execution flow's attention calculation for the c-th target header block, the first execution flow can simultaneously perform at least one of the following operations: first, perform input all-to-all communication for the (c+1)-th target header block to prepare the input data for the next header block in advance; second, perform output all-to-all communication for the (c-1)-th target header block to transmit the calculation results of the previous header block in a timely manner. This achieves overlapping execution of the three types of tasks—input communication, calculation, and output communication—within the same time window, significantly reducing end-to-end computation latency. At the same time, this multi-task overlapping execution method can fully utilize the computing power and bandwidth resources of the target computing device, improving resource utilization.

[0115] In an optional implementation, the method further includes: The startup event is recorded in the main execution flow; after waiting for the startup event in the first execution flow and the second execution flow, input all-to-all communication and output all-to-all communication are performed for each target header block in the first execution flow, and attention calculation is performed for each target header block in the second execution flow.

[0116] In this embodiment, a main execution flow can be created in the target computing device as a global control flow. After completing all target header initialization, parameter configuration, and resource allocation (such as communication buffers, caches, etc.), the main execution flow will record and trigger a global startup event to mark that all preparations have been completed.

[0117] After the first and second execution flows start, they do not immediately execute tasks, but instead enter a blocked waiting state, continuously listening for the start event issued by the main execution flow. When the first and second execution flows detect the start event, they immediately unblock and execute their respective tasks (the first execution flow performs communication operations, and the second execution flow performs computation operations), ensuring that the two execution flows start synchronously and run collaboratively.

[0118] This embodiment uses the main execution flow to uniformly control the global startup event, which ensures that the first and second execution flows start synchronously only after all preparations (initialization, resource allocation) are completed, thus avoiding logical errors caused by premature execution flows running idly, resource contention, or data not being ready.

[0119] Step S206 The output blocks obtained from the full-to-full communication are merged to obtain the target output result.

[0120] In this embodiment, after all target head blocks have completed full-to-full communication, the output blocks corresponding to each head block are collected. According to the original arrangement order of the attention head dimension, all output blocks are spliced ​​and integrated to restore the complete attention output tensor. Finally, the target output result that meets the requirements of subsequent operations is obtained, ensuring that the dimension of the merged data is consistent with the dimension specification of the original target tensor data.

[0121] In an optional embodiment, see [link to relevant documentation] Figure 7 Step S206 may include: Step S700: Before merging the output blocks, synchronize the first execution flow and the second execution flow; Step S702: Concatenate the output blocks according to the attention head dimension to obtain the attention output tensor.

[0122] In this embodiment, before merging all output blocks, a global synchronization operation is first performed to ensure that the first execution flow has completed all-to-all communication of the output of all target header blocks, the second execution flow has completed the attention calculation of all target header blocks, and there are no unfinished asynchronous tasks, thus avoiding the loss of output blocks due to some header blocks not being processed.

[0123] After completing the execution flow synchronization, all target header blocks are collected and output blocks are obtained through output all-to-all communication. All output blocks are concatenated strictly according to the original arrangement order of the attention head dimensions. The concatenated output tensor is a complete attention output tensor with dimensions consistent with the original target tensor data.

[0124] This embodiment ensures that all communication and computation tasks of the target header blocks are completed by performing a global synchronization step, avoiding merging errors caused by missing output blocks and guaranteeing the integrity of the output results. At the same time, by strictly following the original order of the attention head dimension to assemble the output blocks, it can ensure that the merged attention output tensor restores the dimensional structure of the original attention computation, guaranteeing that the output results are equivalent to the original computation results without any loss of precision.

[0125] In an optional embodiment, the method further includes: The target output result is obtained by performing an output projection on the attention output tensor.

[0126] In this embodiment, the built-in output projection operator of the target computing device can be invoked, and preset projection weights and bias parameters can be loaded to perform a linear projection operation on the attention output tensor. This transforms the feature dimension of the attention output tensor into a dimension that meets the input requirements of the subsequent neural network layer, thereby obtaining the target output result. This result can be directly input into the subsequent neural network layer to complete the entire sequence parallel attention processing flow.

[0127] As an example, the target output result can be obtained by performing output projection on the attention output tensor in the following manner.

[0128] ; in, Y Output the result for the target; O The attention output tensor is defined as Linear(), which is the attention output projection layer (the last linear fully connected layer of Transformer attention) used to fuse and map the results of multi-head attention back to the model's original feature space. B (BatchSize) is the number of independent samples processed in parallel at one time; L This represents the total global sequence length. P Total number of GPUs per computing device (sequential parallel cards); D model This represents the dimension of the model's hidden layers.

[0129] In an optional implementation, the method further includes: A block-level cache structure is established for the key tensor and value tensor respectively according to the target header block; when processing the corresponding target header block, the key cache and / or value cache corresponding to the target header block are called.

[0130] In this embodiment, an independent block-level cache structure can be established at the target header block level, consisting of key tensors and value tensors. Each target header block corresponds to a dedicated key cache and a dedicated value cache, which are isolated from each other and do not share data. Thus, when performing processing (including block segmentation, communication, and attention calculation) on a specific target header block, the key cache and / or value cache corresponding to that target header block can be directly invoked without rereading the original key and value tensor data, achieving cache reuse.

[0131] In one embodiment, the block-level cache structure is a list structure, wherein each element in the list structure corresponds one-to-one with each target header block.

[0132] This embodiment establishes independent block-level caches for key tensors and value tensors at the target header block level, thereby enabling header block-level reuse of key and value tensors. This avoids repeated reading and transmission of the original key and value tensor data, significantly reducing data reading and transmission overhead and improving computational efficiency.

[0133] In an optional implementation, the method further includes: During the processing of multiple target header blocks, the communication buffers corresponding to different target header blocks are reused.

[0134] In this embodiment, before initiating the sequence parallel attention processing, a global communication buffer can be initialized to store the input all-to-all and output all-to-all communication data for all target header blocks. The buffer's capacity is adapted to the data size of the largest header block. Subsequently, during the processing of multiple target header blocks, new communication buffers are not repeatedly created. When the previous target header block completes all communication operations, the data in the buffer for that header block is released, and the buffer is directly reused for the communication data storage and transmission of the next target header block, thus achieving cross-header block reuse of the buffer.

[0135] This embodiment adopts a global communication buffer cross-head block reuse method, which avoids the memory allocation overhead and memory fragmentation caused by frequent creation and destruction of communication buffers, and can effectively reduce the device's memory usage and improve memory utilization.

[0136] In an optional implementation, the method further includes: Release the intermediate tensor corresponding to the preceding target header block during subsequent target header block processing.

[0137] In this embodiment, during the pipelined parallel processing, the processing progress of each target header block is tracked in real time to determine whether the preceding target header block has completed all processing steps (input communication, calculation, output communication) and has no subsequent dependent references.

[0138] During the processing of subsequent target header blocks (such as the (c+1)th and (c+2)th), if the preceding target header block (such as the cth) has completed all processing and has no dependencies, immediately release all intermediate tensors corresponding to the preceding target header block (including temporary data during communication, intermediate results during computation, etc.) and reclaim memory resources.

[0139] This embodiment enables streaming reclamation of intermediate memory by timely releasing the intermediate tensors of the preceding header blocks that have completed all processing and have no dependencies during subsequent target header block processing, thereby significantly reducing the peak memory usage of the entire attention processing flow.

[0140] In an optional implementation, the attention calculation includes: Call the FlashAttention kernel or PyTorch SDPA kernel to perform attention calculations on the corresponding target header block.

[0141] In this embodiment, for each target header block, the corresponding query sub-block, key sub-block, and value sub-block can be input into the selected FlashAttention kernel or PyTorch SDPA kernel. The kernel then performs core operations such as attention weight calculation and feature weighted fusion, and outputs the attention calculation result of the header block.

[0142] This embodiment effectively improves the efficiency and accuracy of attention calculation by calling the FlashAttention kernel or PyTorch SDPA kernel, which have the advantages of high computing power, high memory utilization and low precision loss.

[0143] In an optional implementation, the method further includes: The sequence parallel processing based on all-to-all is combined with tensor parallel processing, wherein the sequence parallel processing splits the data along the sequence length dimension, and the tensor parallel processing splits the data along the attention head dimension, and the two are orthogonally parallel in different dimensions.

[0144] This embodiment solves the single-card memory bottleneck for ultra-long sequences by using sequence parallelism; it solves the single-card storage bottleneck for large model weights by using tensor parallelism; the combination of the two achieves a double reduction in memory overhead, while maintaining the equivalence of computational efficiency and numerical results, thus enabling it to adapt to larger-scale models and longer input sequences.

[0145] In an optional implementation, the method can be used for forward propagation processing in an inference scenario or for forward propagation processing in a training scenario; and the method can be compatible with an automatic differentiation framework and can perform backpropagation processing based on the target output result.

[0146] To make this application easier to understand, an exemplary application is provided below. In this exemplary application, the total number of attention heads H=16, and the number of target head blocks C=4 blocks, wherein, Target head block 1 includes: attention heads 0, 1, 2, and 3 (a total of 4 attention heads); Target head block 2 includes: attention heads 4, 5, 6, and 7 (a total of 4 attention heads); Target head block 3 includes: attention heads 8, 9, 10, and 11 (a total of 4 attention heads); Target Head Block 4: Attention Heads 12, 13, 14, and 15 (a total of 4 attention heads).

[0147] This sequential parallel attention processing method includes a hardware and streaming architecture preparation phase and a data processing phase, wherein: During the hardware and streaming architecture preparation phase, two independent CUDA streams will be created: the S_comm stream (the first execution stream) and the S_comp stream (the second execution stream). The S_comm stream is a dedicated communication stream, running only All-to-All aggregate communication. The S_comp stream is a dedicated computation stream, running only FlashAttention computation.

[0148] In addition, the hardware and streaming architecture preparation phase will also include CUDA Events to perform precise inter-block synchronization and avoid data dependency errors.

[0149] The data processing stage includes the following steps: Step 1: Global preprocessing (done in parallel by all GPUs).

[0150] All GPUs locally complete QKV projection and RoPE position encoding. According to the above header block division, the QKV is divided into 4 independent header block tensors in the header dimension. At this time, the shape of each header block is: [2,8000,4,64].

[0151] Step 2: Pipeline startup (communication-computation overlap begins).

[0152] To achieve parallelism, the scheduling is performed in the order of head block 1 → head block 2 → head block 3 → head block 4. Time step 1 (communication flow only): S_comm communication flow initiates partial all-to-all communication with head block 1; after head block 1 communication is completed, the event E1_comm_done is marked.

[0153] Time step 2 (communication + computation in parallel for the first time): The S_comp computation stream waits for E1_comm_done to be ready, then immediately computes the attention of head block 1 separately. At the same time, the S_comm communication stream runs asynchronously in the background and immediately starts all-to-all communication of head block 2. When head block 2 communication is complete, the event E2_comm_done is marked.

[0154] Time step 3 (fully parallel three-line operation): S_comp finishes the attention calculation for the first block 1; S_comp then waits for E2_comm_done and begins the attention calculation for the first block 2; S_comm simultaneously starts the all-to-all communication for the first block 3.

[0155] Time step 4 will perform the following operations: Computation flow: Calculate head block 2 → Start calculating head block 3; Communication flow: Header block 3 communication ends → Header block 4 communication begins.

[0156] Time step 5 will perform the following operations: Computation flow: Calculate the first block 3 → Start calculating the first block 4 Communication flow: No new communication tasks, waiting for computation events. Time step 6 will perform the following operations: Computation flow: First block 4 is computed; The communication stream waits for the head block 4 calculation to complete before executing the output for full communication.

[0157] Step 3: Result splicing and finishing.

[0158] After all head block computations and communications are synchronized, the outputs of the four independent head blocks will be concatenated in sequence to restore the complete multi-head output. Then, the final output linear projection can be performed to complete the single-layer attention computation.

[0159] Example 2 Figure 8 A block diagram schematically illustrates a sequence-parallel attention processing apparatus according to Embodiment 2 of this application. This apparatus is applied to a target computing device and can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program module referred to in the embodiments of this application refers to a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the function of each program module in this embodiment. For example... Figure 8 As shown, the sequence parallel attention processing device 800 may include: an acquisition module 810, a partitioning module 820, a control module 830, and a merging module 840, wherein: The acquisition module 810 is used to acquire the target tensor data for parallel attention computation of the sequence to be executed, wherein the target tensor data includes query tensors, key tensors and value tensors; The partitioning module 820 is used to partition the target tensor data into multiple target header blocks according to the attention head dimension; The control module 830 is used to control the input all-to-all communication, attention calculation and output all-to-all communication of each target head block to form a pipelined parallel execution with the target head block as the scheduling granularity; The merging module 840 is used to merge the output blocks obtained from the full-to-full communication to obtain the target output result.

[0160] In an optional implementation, obtaining the target tensor data for parallel attention computation of the sequence to be executed includes: obtaining an input slice; performing a linear projection operation on the input slice to obtain the query tensor, the key tensor, and the value tensor.

[0161] In an optional implementation, the sequential parallel attention processing device 800 is further configured to: Position encoding is performed on the query tensor and the key tensor to inject sequence position information into the query tensor and the key tensor, respectively.

[0162] In an optional implementation, the target tensor data is divided into multiple target header blocks according to the attention head dimension, including: determining the number of target blocks; segmenting the query tensor, the key tensor, and the value tensor along the attention head dimension according to the number of target blocks to obtain multiple query sub-blocks, multiple key blocks, and multiple value sub-blocks; and determining the query sub-blocks, key blocks, and value sub-blocks with the same block number as a target header block, wherein the block number is the sorting number of the target header block.

[0163] In an optional implementation, determining the target number of blocks includes: acquiring pre-acquired or measured communication time parameters, single-block overhead parameters, and communication ratio parameters; and determining the target number of blocks based on the communication time parameters, the single-block overhead parameters, and the communication ratio parameters.

[0164] In an optional implementation, the sequential parallel attention processing device 800 is further configured to: The number of target blocks is constrained based on the number of attention heads corresponding to each computing device.

[0165] In an optional implementation, dividing the target tensor data into multiple target head blocks according to the attention head dimension includes: when the number of attention heads corresponding to each computing device is not divisible by the number of target blocks, distributing the multiple target head blocks unevenly along the attention head dimension so that the number of attention heads included in the first few target head blocks is one more than the number of other target head blocks.

[0166] In an optional implementation, the step of controlling the input all-to-all communication, attention calculation, and output all-to-all communication of each target head block to form a pipelined parallel execution with the target head block as the scheduling granularity includes: in the first execution flow, performing input all-to-all communication and output all-to-all communication for each target head block respectively; and in the second execution flow, performing attention calculation for each target head block respectively.

[0167] In an optional implementation, the sequential parallel attention processing device 800 is further configured to: When completing input all-to-all communication for the c-th target header block, record the first event corresponding to the c-th target header block; after waiting for the first event corresponding to the c-th target header block in the second execution flow, perform attention calculation on the c-th target header block; when completing attention calculation for the c-th target header block, record the second event corresponding to the c-th target header block; after waiting for the second event corresponding to the c-th target header block in the first execution flow, perform output all-to-all communication on the c-th target header block.

[0168] In an optional implementation, when there are adjacent target header blocks, during the attention calculation for the c-th target header block, the first execution flow performs input all-to-all communication for the (c+1)-th target header block and / or output all-to-all communication for the (c-1)-th target header block.

[0169] In an optional implementation, the sequential parallel attention processing device 800 is further configured to: The startup event is recorded in the main execution flow; after waiting for the startup event in the first execution flow and the second execution flow, input all-to-all communication and output all-to-all communication are performed for each target header block in the first execution flow, and attention calculation is performed for each target header block in the second execution flow.

[0170] In an optional implementation, the output blocks obtained from the full-to-full communication are merged to obtain the target output result, including: synchronizing the first execution flow and the second execution flow before merging the output blocks; and splicing the output blocks according to the attention head dimension to obtain the attention output tensor.

[0171] In an optional implementation, the sequential parallel attention processing device 800 is further configured to: The target output result is obtained by performing an output projection on the attention output tensor.

[0172] In an optional implementation, the sequential parallel attention processing device 800 is further configured to: A block-level cache structure is established for the key tensor and value tensor respectively according to the target header block; when processing the corresponding target header block, the key cache and / or value cache corresponding to the target header block are called.

[0173] In an optional implementation, the sequential parallel attention processing device 800 is further configured to: During the processing of multiple target header blocks, the communication buffers corresponding to different target header blocks are reused.

[0174] In an optional implementation, the sequential parallel attention processing device 800 is further configured to: Release the intermediate tensor corresponding to the preceding target header block during subsequent target header block processing.

[0175] In an optional implementation, the sequential parallel attention processing device 800 is further configured to: The sequence parallel processing based on all-to-all is combined with tensor parallel processing, wherein the sequence parallel processing splits the data along the sequence length dimension, and the tensor parallel processing splits the data along the attention head dimension, and the two are orthogonally parallel in different dimensions.

[0176] Example 3 Figure 9 This illustration schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a sequential parallel attention processing method according to Embodiment 3 of this application. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual device, workbench, digital assistant, set-top box, robot, etc. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 9 As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus. Wherein: The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as the program code of the method described in the foregoing embodiment. Furthermore, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0177] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0178] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0179] It should be pointed out that, Figure 9 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0180] In this embodiment, the sequence parallel attention processing method stored in memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiments of this application.

[0181] Example 4 This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the foregoing embodiments.

[0182] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium may include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the method described in the foregoing embodiments. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0183] Example 5 This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above embodiments.

[0184] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0185] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A method of processing sequence-parallel attention, the method comprising: The method includes: Obtain the target tensor data for parallel attention computation of the sequence to be executed, wherein the target tensor data includes query tensors, key tensors, and value tensors; The target tensor data is divided into multiple target header blocks according to the attention head dimension; Using the target header block as the scheduling granularity, the input all-to-all communication, attention calculation, and output all-to-all communication of each target header block are controlled to form a pipelined parallel execution. The output blocks obtained from the full-to-full communication are merged to obtain the target output result.

2. The method of claim 1, wherein, Obtain the target tensor data for parallel attention computation of the sequence to be executed, including: Obtain the input slice; Perform a linear projection operation on the input fragment to obtain the query tensor, the key tensor, and the value tensor.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Position encoding is performed on the query tensor and the key tensor to inject sequence position information into the query tensor and the key tensor, respectively.

4. The method according to claim 1 or 2, characterized in that, The target tensor data is divided into multiple target header blocks according to the attention head dimension, including: Determine the target number of blocks; The query tensor, the key tensor, and the value tensor are divided along the attention head dimension according to the target number of blocks to obtain multiple query sub-blocks, multiple key blocks, and multiple value sub-blocks; The query sub-blocks, key sub-blocks, and value sub-blocks with the same block number are identified as a target header block, wherein the block number is the sorting number of the target header block.

5. The method of claim 4, wherein, Determine the target number of blocks, including: Obtain pre-acquired or measured communication time parameters, single-block overhead parameters, and communication ratio parameters; The target number of blocks is determined based on the communication time parameter, the single-block overhead parameter, and the communication ratio parameter.

6. The method according to claim 4 or 5, characterized in that, The method further includes: The number of target blocks is constrained based on the number of attention heads corresponding to each computing device.

7. The method of claim 6, wherein, The step of dividing the target tensor data into multiple target header blocks according to the attention head dimension includes: If the number of attention heads corresponding to each computing device is not divisible by the number of target blocks, the multiple target head blocks are unevenly distributed along the attention head dimension so that the number of attention heads included in the first few target head blocks is one more than that in the remaining target head blocks.

8. The method of claim 1, wherein, The process of controlling the input all-to-all communication, attention calculation, and output all-to-all communication of each target head block to form a pipelined parallel execution, with the target head block as the scheduling granularity, includes: In the first execution flow, input all-to-all communication and output all-to-all communication are performed for each of the target header blocks; In the second execution flow, attention calculations are performed on each of the target header blocks.

9. The method of claim 8, wherein, The method further includes: When the input all-to-all communication is completed for the c-th target header block, record the first event corresponding to the c-th target header block; After waiting for the first event corresponding to the c-th target header block in the second execution flow, attention calculation is performed on the c-th target header block; When completing the attention calculation for the c-th target head block, record the second event corresponding to the c-th target head block; After waiting for the second event corresponding to the c-th target header block in the first execution flow, output all-to-all communication is performed on the c-th target header block.

10. The method of claim 9, wherein, In the presence of adjacent target header blocks, during the attention calculation for the c-th target header block, the first execution flow performs input all-to-all communication for the (c+1)-th target header block and / or output all-to-all communication for the (c-1)-th target header block.

11. The method of claim 8, wherein, The method further includes: Record the startup event in the main execution flow; After waiting for the start event in the first execution flow and the second execution flow, in the first execution flow, input all-to-all communication and output all-to-all communication are performed for each of the target header blocks respectively, and in the second execution flow, attention calculation is performed for each of the target header blocks respectively.

12. The method according to claim 1, characterized in that, The output blocks obtained from the full-to-full communication are merged to obtain the target output result, including: Before merging the output blocks, synchronize the first execution flow and the second execution flow; The output blocks are concatenated according to the attention head dimension to obtain the attention output tensor.

13. The method according to claim 1 or 2, characterized in that, The method further includes: Establish block-level cache structures for the key tensors and value tensors respectively according to the target header blocks; When processing the corresponding target header block, the key cache and / or value cache corresponding to that target header block are invoked.

14. The method according to claim 1, characterized in that, The method further includes: During the processing of multiple target header blocks, the communication buffers corresponding to different target header blocks are reused.

15. The method according to claim 1, characterized in that, The method further includes: Release the intermediate tensor corresponding to the preceding target header block during subsequent target header block processing.

16. The method according to claim 1, characterized in that, The method further includes: The sequence parallel processing based on all-to-all is combined with tensor parallel processing, wherein the sequence parallel processing splits the data along the sequence length dimension, and the tensor parallel processing splits the data along the attention head dimension, and the two are orthogonally parallel in different dimensions.

17. A sequential parallel attention processing device, characterized in that, The device includes: The acquisition module is used to acquire the target tensor data for parallel attention computation of the sequence to be executed. The target tensor data includes query tensors, key tensors, and value tensors. The partitioning module is used to divide the target tensor data into multiple target header blocks according to the attention head dimension; The control module is used to control the input all-to-all communication, attention calculation and output all-to-all communication of each target head block to form a pipelined parallel execution with the target head block as the scheduling granularity; The merging module is used to merge the output blocks obtained from the full-to-full communication to obtain the target output result.

18. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to cause the at least one processor to perform the method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method of any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 16.

Images