Data processing method and apparatus, computing device cluster, and storage medium

Abstract

The present application discloses a data processing method and apparatus, a computing device cluster, and a storage medium. The method comprises: performing feature extraction on data inputted by a user, to obtain query (Q) data, key (K) data, and value (V) data; on the basis of the number N of processing devices, respectively segmenting the Q data, the K data, and the V data to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks; and controlling the N processing devices to perform 2N attention calculation operations on the basis of the 2N Q data blocks, the 2N K data blocks, and the 2N V data blocks, so as to obtain attention calculation results of the N processing devices, wherein each processing device performs two sets of attention calculations on the basis of two Q data blocks, two K data blocks, and two V data blocks respectively during each attention calculation operation. In the present application, attention-based long sequence processing can be implemented on a plurality of processing devices by means of two data streams, thereby improving the efficiency of sequence parallel processing.

Description

Data processing methods, devices, computing equipment clusters, and storage media

[0001] This application claims priority to Chinese Patent Application No. 202411472201.1, filed on October 21, 2024, entitled “Data Processing Method, Apparatus, Computing Device Cluster and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, computing device cluster and storage medium. Background Technology

[0003] In the field of large-scale artificial intelligence models, the length of sequences to be processed is constantly increasing. Since the capabilities of devices supporting model training and inference are typically limited, related technologies often employ long-sequence parallelism to improve processing efficiency. Sequence parallelism in attention mechanisms has received considerable attention in recent years. By segmenting the input query (Q), key (K), and value (V), the data originally input to attention is divided into multiple parts along the sequence dimension, and these parts are distributed to different devices for processing. Common sequence parallel attention mechanisms include ring-attention and striped attention, but the efficiency of sequence parallel processing remains relatively low. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computing device cluster, and storage medium that can achieve long-sequence parallel processing of attention on multiple processing devices with two data streams. Each processing device performs two sets of attention calculations in each iteration, which helps to improve the efficiency of sequence parallel processing.

[0005] Firstly, embodiments of this application provide a data processing method applied to an electronic device or a chip within an electronic device. Taking the application of this method to an electronic device as an example, the method includes:

[0006] Feature extraction is performed on the user input data to obtain query Q data, key K data, and value V data;

[0007] Based on the number of processing devices N, the Q data, K data, and V data are divided into 2N Q data blocks, 2N K data blocks, and 2N V data blocks respectively.

[0008] Control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of the N processing devices;

[0009] In each attention calculation, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively. Each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0010] As can be seen from the embodiments of this application, the electronic device can segment the Q data, K data, and V data based on the number N of parallel processing devices, and finally obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks through direct segmentation or segmentation and recombination. Then, it controls N processing devices to perform 2N attention calculations based on the 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of N processing devices. Each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation. This forms two computational streams on the N processing devices. The dual-stream computation mode makes the computational scale of a single set of attention calculations on any processing device smaller, thus improving the efficiency of attention calculation.

[0011] In one possible implementation, the two Q data blocks used by the i-th processing device among the N processing devices when performing the j-th attention calculation are pre-allocated data blocks Q1 and Q2, and the two K data blocks are data blocks K. 1，j and data block K 2， j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N;

[0012] Where, when j=1, data block K 1，j Data block K 2，j Data block V 1，j and data block V 2，j It is pre-allocated;

[0013] When i = 1 and 1 < j ≤ 2N, data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 Data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，jV is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 1，j-1 Data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ;

[0014] When i = N and 1 < j ≤ 2N, data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 2，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ;

[0015] When 1 < i < N and 1 < j ≤ 2N, data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 .

[0016] In this implementation, for the i-th processing device among N processing devices, the Q data blocks, K data blocks, and V data blocks used in the first two sets of attention calculations are pre-allocated by the electronic device. When i = 1 or N, one set of K data blocks and V data blocks used in the second and third sets of attention calculations after the first set come from adjacent processing devices, and the other set of K data blocks and V data blocks comes from within the i-th processing device. When i ≠ 1 or N, the K data blocks and V data blocks used in the second and third sets of attention calculations after the first set come from adjacent data blocks. This ensures that each K data block and each V data block can perform an attention calculation with each Q data block, thereby guaranteeing the integrity of the attention calculation between Q data and K and V data.

[0017] In one possible implementation, after obtaining 2N Q data blocks, 2N K data blocks, and 2N V data blocks, the method further includes:

[0018] In a symmetrical manner, 2N Q data blocks, 2N K data blocks, and 2N V data blocks are allocated to N processing devices respectively; each processing device is allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks symmetrically.

[0019] In this implementation, the electronic device distributes the 2N Q-blocks, 2N K-blocks, and 2N V-blocks obtained from direct segmentation or recombination to N processing devices in a symmetrical manner. This ensures that each processing device performs two sets of operations based on the Q / K / V data blocks used during the initial attention calculation. Furthermore, this facilitates the transmission of two sets of K / V data blocks in a single communication between subsequent processing devices. This not only fully utilizes the transmission bandwidth but also reduces the computational workload per set, as each attention calculation on each processing device is performed in two groups, thus improving computational efficiency.

[0020] In one possible implementation, when j=1, the preset masks for the score matrices of the Q-block and K-block used for the two sets of attention calculations on the i-th processing device are both the first mask; the first mask is a causal mask.

[0021] The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

[0022] In this implementation, when N processing devices are controlled to perform the first attention calculation in a scenario where a causal mask needs to be applied, since the masks corresponding to the two sets of attention calculations on each processing device are the first mask, there is no need to consider the calculation order of the two sets of attention calculations. The two sets of attention calculations can be completed based on the Q data block, K data block, V data block and the first mask used, respectively, making the calculation relatively flexible.

[0023] In one possible implementation, when 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in any one attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different.

[0024] The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second and third masks, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

[0025] In this implementation, when one set of attention calculations on the i-th processing device corresponds to the second mask and another set of attention calculations corresponds to the third mask, the electronic device needs to control N processing devices to calculate attention using the same mask at the same time. That is, according to the preset execution order of the second and third masks, the attention calculations corresponding to either the second or third mask are executed uniformly first, and then the attention calculations corresponding to the other mask are executed uniformly. By uniformly using the same mask to calculate attention, the consistency of attention calculation can be ensured, avoiding situations where using inconsistent masks may cause some processing devices to see information that should not be seen during calculation, resulting in the disruption of causal relationships and low usability of calculation results.

[0026] In one possible implementation, the second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix.

[0027] When the mask corresponding to any set of attention calculations is the second mask, any set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; the sub-Q data block is the sub-data block from the midpoint of the row direction of the Q data block to the last column of the Q data block;

[0028] When the mask corresponding to any set of attention calculations is the third mask, any set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is the sub-data block from the first column of the K data block to the midline of the row direction of the K data block; the sub-V data block is the sub-data block from the first column of the V data block to the midline of the row direction of the V data block.

[0029] In this implementation, for a set of attention calculations on the i-th processing device where the sub-mask is the second mask, based on the occlusion of the second mask, the electronic device can control the processing device to discard invalid data in the used Q data blocks to obtain sub-Q data blocks, and then perform attention calculations based on the sub-Q data blocks, K data blocks, and V data blocks. Similarly, for another set of attention calculations on the i-th processing device where the sub-mask is the third mask, based on the occlusion of the third mask, the electronic device can control the processing device to discard invalid data in the used K data blocks and V data blocks to obtain sub-K data blocks and sub-V data blocks, and then perform attention calculations based on the Q data blocks, sub-K data blocks, and sub-V data blocks. Discarding invalid data reduces unnecessary computational operations, lowers the amount of data computation, and helps to further improve computational efficiency.

[0030] In one possible implementation, the query Q data, key K data, and value V data are respectively split to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks, including:

[0031] The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1;

[0032] Using a symmetrical approach, respectively (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x Recombining N V data blocks yields 2N Q data blocks, 2N K data blocks, and 2N V data blocks; where each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

[0033] In this implementation, for scenarios requiring causal masking, the Q, K, and V data can be further segmented into finer-grained segments, and then the segmented (2) data can be processed in a symmetrical manner from beginning to end.x )*N Q data blocks, (2 x )*N K data blocks and (2 x N V data blocks are reorganized into 2N Q data blocks, 2N K data blocks, and 2N V data blocks. This symmetrical reorganization method ensures that the amount of data obscured by the mask in the score matrix of each Q data block and its corresponding K data block is closer (or even the same). This facilitates more balanced attention calculation processing across processing devices and improves the problem of uneven computing power.

[0034] In one possible implementation, when controlling the i-th processing device to perform the j-th attention calculation, the method further includes:

[0035] When i=1, control the i-th processing device to transmit data block K. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process;

[0036] When i = N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process;

[0037] When 1 < i < N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation; it controls the transmission of data block K by the i-th processing device. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process.

[0038] In this implementation, when the electronic device controls N processing devices to perform the j-th attention calculation, it also controls the N processing devices to perform the j-th transmission of the K data blocks and V data blocks used in the j-th attention calculation. The K data blocks and V data blocks of the N processing devices rotate in a circular manner, forming an upward flow and a downward flow, thereby ensuring that each processing device has two sets of K data blocks and V data blocks in each attention calculation. This achieves two sets of calculations in one communication, which is beneficial for making full use of communication bandwidth and computing power resources.

[0039] In one possible implementation, before splitting the Q data, K data, and V data respectively based on the number of processing devices N, the method further includes:

[0040] Receive instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation process.

[0041] In this implementation, the electronic device can enable the dual-stream parallel function of the attention sequence based on the user's input instructions, so that each processing device can perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation.

[0042] Secondly, embodiments of this application provide a data processing apparatus applied to an electronic device or a chip in an electronic device; the apparatus includes a feature extraction unit, a data segmentation unit, and a control unit; wherein:

[0043] The feature extraction unit is used to extract features from the user input data to obtain query Q data, key K data, and value V data;

[0044] The data segmentation unit is used to segment the Q data, K data, and V data based on the number of processing devices N, respectively, to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks;

[0045] The control unit is used to control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of the N processing devices;

[0046] In each attention calculation, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively. Each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0047] In one possible implementation, the two Q data blocks used by the i-th processing device among the N processing devices when performing the j-th attention calculation are pre-allocated data blocks Q1 and Q2, and the two K data blocks are data blocks K. 1，j and data block K 2， j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N;

[0048] Where, when j=1, data block K 1，j Data block K 2，j Data block V 1，j and data block V 2，j It is pre-allocated;

[0049] When i = 1 and 1 < j ≤ 2N, data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 Data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 1，j-1 Data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ;

[0050] When i = N and 1 < j ≤ 2N, data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 2，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ;

[0051] When 1 < i < N and 1 < j ≤ 2N, data block K 1，jThe data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 .

[0052] In one possible implementation, the device further includes a data allocation unit; the data allocation unit is used to allocate 2N Q data blocks, 2N K data blocks and 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks and 2 V data blocks symmetrically.

[0053] In one possible implementation, when j=1, the preset masks for the score matrices of the Q-block and K-block used for the two sets of attention calculations on the i-th processing device are both the first mask; the first mask is a causal mask.

[0054] The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

[0055] In one possible implementation, when 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in any one attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different.

[0056] The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second and third masks, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

[0057] In one possible implementation, the second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix.

[0058] When the mask corresponding to any set of attention calculations is the second mask, any set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; the sub-Q data block is the sub-data block from the midpoint of the row direction of the Q data block to the last column of the Q data block;

[0059] When the mask corresponding to any set of attention calculations is the third mask, any set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is the sub-data block from the first column of the K data block to the midline of the row direction of the K data block; the sub-V data block is the sub-data block from the first column of the V data block to the midline of the row direction of the V data block.

[0060] In one possible implementation, the data splitting unit is specifically used to split the query Q data, key K data, and value V data respectively to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks:

[0061] The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1;

[0062] Using a symmetrical approach, respectively (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x Recombining N V data blocks yields 2N Q data blocks, 2N K data blocks, and 2N V data blocks; where each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

[0063] In one possible implementation, when controlling the i-th processing device to perform the j-th attention calculation, the control unit is further configured to:

[0064] When i=1, control the i-th processing device to transmit data block K. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process;

[0065] When i = N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process;

[0066] When 1 < i < N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation; it controls the transmission of data block K by the i-th processing device. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process.

[0067] In one possible implementation, the control unit is also used for:

[0068] Receive instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation process.

[0069] It should be understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the first aspect of the embodiments of this application should be adapted to the second aspect of the embodiments of this application simultaneously, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0070] Thirdly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method as described in any of the embodiments of the first aspect above.

[0071] Fourthly, embodiments of this application provide a chip, including: a processor, configured to call and run a computer program from a memory, causing a device on which the chip is installed to perform the method as described in any of the embodiments of the first aspect above.

[0072] Fifthly, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, implement the method as described in any of the embodiments of the first aspect above.

[0073] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any of the embodiments of the first aspect above. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0075] Figure 1 is a schematic diagram of a transfer key data block provided by related technologies;

[0076] Figure 2 is a schematic diagram of a transmission value data block provided by related technologies;

[0077] Figure 3 is a schematic diagram of a data segmentation and allocation method provided by related technologies;

[0078] Figure 4 is a schematic diagram of another data segmentation and allocation method provided by related technologies;

[0079] Figure 5 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0080] Figure 6 is a schematic diagram of a front-end and back-end processing logic provided in an embodiment of this application;

[0081] Figure 7 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0082] Figure 8A is a schematic diagram of data segmentation and allocation provided in an embodiment of this application;

[0083] Figure 8B is a schematic diagram of data segmentation and recombination provided in an embodiment of this application;

[0084] Figure 8C is a schematic diagram of data segmentation, recombination, and allocation provided in an embodiment of this application;

[0085] Figure 9 is a schematic diagram of data processing and transmission provided in an embodiment of this application;

[0086] Figure 10 is a schematic diagram of a causal mask provided in an embodiment of this application;

[0087] Figure 11 is a schematic diagram of a first mask provided in an embodiment of this application;

[0088] Figure 12 is a schematic diagram of two sets of sub-masks corresponding to attention calculations on a processing device provided in an embodiment of this application;

[0089] Figure 13 is a schematic diagram of a second mask and a third mask provided in an embodiment of this application;

[0090] Figure 14 is a schematic diagram of two sets of sub-masks corresponding to attention calculations on another processing device provided in an embodiment of this application;

[0091] Figure 15 is a schematic diagram of discarding invalid data according to an embodiment of this application;

[0092] Figure 16 is a schematic diagram of discarding invalid data according to an embodiment of this application;

[0093] Figure 17 is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0094] Figure 18 is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0095] Figure 19 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0096] Figure 20 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0097] Figure 21 is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0098] Figure 22 is a schematic diagram of a connection method between computing devices provided in an embodiment of this application. Detailed Implementation

[0099] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0100] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0101] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on electronic devices and electronic devices themselves can be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0102] First, a brief introduction to the relevant terms and technical background used in this application will be provided to facilitate understanding by those skilled in the art.

[0103] (1) Graphics Processing Unit (GPU): A typical artificial intelligence computing device.

[0104] (2) Attention: The attention module in the transformer model is a model structure in artificial intelligence machine learning.

[0105] (3) Neural Processing Unit (NPU): A hardware accelerator specifically designed to accelerate neural network computation.

[0106] (4) Tensor Processing Unit (TPU): A hardware accelerator specifically designed to accelerate machine learning and deep learning computations. The TPU is designed to improve the efficiency of processing large-scale tensor computations and is widely used in deep learning models.

[0107] (5) Rank: The smallest unit of device that performs long-sequence parallelism, which can be one or more GPUs, or one or more devices or machines.

[0108] The current mainstream technique for long-sequence parallelism in attention is ring-self-attention (ring-attention). Its principle is to divide the Q / K / V data into slices, allowing different devices to receive a slice of Q / K / V data, as shown in Figure 1. First, K is iteratively calculated (QK is calculated first in the attention calculation). T Based on QK T The data is divided into Q / K / V segments. After each calculation, the segment K is passed to the next device in a circular manner (or simultaneously). As shown in Figure 2, after the circular calculation of segment K is completed, segment V is calculated iteratively. After each calculation, segment V is passed to the next device in a circular manner. In the above calculation, one way to combine the blockwise attention algorithm is as follows: First, the data Q / K / V is divided into segments. Then, each device is fixed with one segment Q. Segments K and V are passed cyclically between different devices. Each iteration completes one calculation. Each device passes the K / V segment data used for calculation to the next device in a circular manner. Each K / V segment data can be iteratively calculated once on each device. After all iterations are completed, the attention calculation is completed.

[0109] The slicing and allocation method of ring-attention is shown in Figure 3. Assuming there are 4 devices, the Q / K / V data can be divided into 16 parts each, and then every 4 parts of Q / K / V data are sequentially allocated to the same device. For example, parts 0 to 3 are allocated to device 1, parts 4 to 7 to device 2, parts 8 to 11 to device 3, and parts 12 to 15 to device 4. Most ring-attention calculations use causal masks. However, according to the data block allocation method of ring-attention, attention calculations using causal masks suffer from a serious problem of computational imbalance. To solve this problem, striped attention has been proposed. The slicing and allocation method of striped attention is shown in Figure 4. Assuming there are 4 devices, the Q / K / V data can be divided into 16 parts each. Four parts of Q / K / V data are permuted and allocated to the same device using a stepping method. For example, parts 0, 4, 8, and 12 are allocated to device 1; parts 1, 5, 9, and 13 to device 2; parts 2, 6, 10, and 14 to device 3; and parts 3, 7, 11, and 15 to device 4. While striped attention can improve the problem of computational imbalance through stepping, it fails to effectively eliminate the computational imbalance when applying a causal mask, and the improvement depends on the granularity of the Q / K / V data segmentation. Like ring-attention, striped attention still uses a single data stream to calculate attention. Serial computation of a single data stream across multiple devices results in low computational efficiency. Furthermore, both ring-attention and striped attention transmit only one K / V data block for computation per communication, leading to insufficient utilization of transmission bandwidth.

[0110] To overcome the shortcomings of existing technologies, this application provides a data processing method, apparatus, electronic device, and storage medium. This data processing method can be implemented based on the system architecture shown in Figure 5. As shown in Figure 5, the system architecture includes a front-end device, a back-end device, and a processing device cluster. The processing devices in the processing device cluster include, but are not limited to, a Central Processing Unit (CPU), GPU, NPU, and TPU. One or more processing devices can also form a Rank. The front-end device can be used for user input of image data, text data, or audio data to be processed. Further, as shown in Figure 6, the user can select whether to enable a function (such as attention sequence dual-stream parallel function) on the user interface of the front-end device. If the function is enabled, the number of parallel attention sequences (such as the number of processing devices) must be input, and the correction method for the attention calculation results must be selected. Based on the user's operation, an attention sequence parallel task or project for the image data, text data, or audio data to be processed is generated. The sequential parallel task enters the resource pool of the backend device. The scheduling module of the backend device parses the user parameters (such as the number of parallel attention sequences, the correction method of attention calculation results, etc.) through a custom function or tool. The backend device can encode the image data, text data, or audio data to obtain the corresponding Q data, K data, and V data, and further divide the Q data, K data, and V data to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks (N is the number of parallel attention sequences). The scheduling module allocates the 2N Q data blocks, 2N K data blocks, and 2N V data blocks in a symmetrical manner, and distributes them to a preset number of processing device clusters in the form of user Pods (which are container instances or container groups defined and run by the user in a distributed cluster). The backend device controls the processing device allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks to perform attention dual-stream calculation according to the data processing method provided in this application, and corrects the attention calculation results output by each processing device according to the correction method selected by the user. In another implementation, the user does not need to input the relevant parameters for dual-stream parallel processing on the front end. The attention dual-stream calculation algorithm can be deployed on the back end in code. When the front end device inputs image data, text data or audio data to be processed, the back end schedules and recognizes the code, and executes the attention dual-stream algorithm by default.

[0111] For example, the correction method for attention calculation results provided in this application embodiment includes: 1) performing a correction once for each attention calculation on each processing device; 2) if there is no correction during the calculation process, saving the two attention results calculated each time on each processing device, and performing a unified correction after the algorithm is completed.

[0112] For example, the backend device in Figure 5 can be a service device that provides cloud services, and the frontend device can be a user device that uses cloud services.

[0113] The technical solution provided in this application will be described in detail below with reference to specific implementation methods.

[0114] Please refer to Figure 7, which is a flowchart illustrating a data processing method provided in an embodiment of this application. This method can be executed by an electronic device, such as the backend device shown in Figure 5. As shown in Figure 7, the method includes steps 701-703:

[0115] 701: Perform feature extraction on the user-input data to obtain query Q data, key K data, and value V data.

[0116] 702: Based on the number of processing devices N, the Q data, K data and V data are divided into 2N Q data blocks, 2N K data blocks and 2N V data blocks respectively.

[0117] In this embodiment, Q data, K data, and V data can be obtained by feature extraction from user-input image data, text data, or audio data. The feature extraction process can refer to the process of generating Q / K / V matrices in Transformer. For scenarios where causal masking is not required, if the number of processing devices (i.e., the number of parallel processing devices) is N, the Q data, K data, and V data can be divided into 2N Q data blocks, 2N K data blocks, and 2N V data blocks, respectively. Then, in a symmetrical manner, the 2N Q data blocks, 2N K data blocks, and 2N V data blocks are allocated to N processing devices; wherein each processing device is allocated symmetrically 2 Q data blocks, 2 K data blocks, and 2 V data blocks. As shown in Figure 8A, assuming N=4, the Q data, K data, and V data can be divided into 8 parts respectively. The symmetrical 2 parts of Q data, 2 parts of K data, and 2 parts of V data are then allocated to 4 processing devices in a symmetrical manner. For example, the data is marked starting from part 0; parts 0 and 7 are allocated to processing device 1, parts 1 and 6 to processing device 2, parts 2 and 5 to processing device 3, and parts 3 and 4 to processing device 4. The phrase "each processing device is allocated 2 symmetrical Q data blocks, 2 K data blocks, and 2 V data blocks" can be understood as "each processing device is allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks that are symmetrical about the axis of symmetry of the row directions of the 2N Q data blocks, 2N K data blocks, and 2N V data blocks respectively."

[0118] For scenarios requiring causal masking, since the improvement in computing power imbalance depends on the granularity of data segmentation, this embodiment can perform finer-grained segmentation compared to scenarios not requiring causal masking. If the number of processing devices is N, then the Q data, K data, and V data are respectively segmented into (2... x )*N Q data blocks, (2 x )*N K data blocks and (2 x ()*N V data blocks; where x is an integer greater than 1. For example, the Q data, K data, and V data can be divided into 4N Q data blocks, 4N K data blocks, and 4N V data blocks, or into 8N Q data blocks, 8N K data blocks, and 8N V data blocks, and so on. Furthermore, in a symmetrical manner, (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x Recombining N V data blocks yields 2N Q data blocks, 2N K data blocks, and 2N V data blocks. Each symmetrical 2... x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reassembled into one V data block. As shown in Figure 8B, the Q data, K data, and V data are divided into 4N Q data blocks, 4N K data blocks, and 4N V data blocks, respectively. Starting from the 0th block, the 0th data block and the (4N-1)th data block are reassembled into the 0th data block, the 1st data block and the (4N-2)th data block are reassembled into the 1st data block, and so on, until the (2N-1)th data block and the 2Nth data block are reassembled into the (2N-1)th data block, thus obtaining 2N Q data blocks, 2N K data blocks, and 2N V data blocks. Wherein, "each symmetrical 2 x-1 "Reorganizing 2 Q data blocks into one Q data block" can be understood as "using (2) x The axis of symmetry in the row direction of N Q data blocks is 2. x-1 "Reorganizing Q data blocks into one Q data block": Assuming the Q data is divided into 4N Q data blocks, the two axis-symmetric data blocks are the 0th data block and the (4N-1)th data block. Assuming the Q data is divided into 8N Q data blocks, the four axis-symmetric data blocks are the 0th data block, the 1st data block, the (4N-2)th data block, and the (4N-1)th data block. The reorganization method for K data and V data is the same as that for Q data.

[0119] In this implementation, for scenarios requiring causal masking, the Q, K, and V data can be further segmented into finer-grained segments, and then the segmented (2) data can be processed in a symmetrical manner from beginning to end. x )*N Q data blocks, (2 x )*N K data blocks and (2 x N V data blocks are reorganized into 2N Q data blocks, 2N K data blocks, and 2N V data blocks. This symmetrical reorganization method ensures that the amount of data obscured by the mask in the score matrix of each Q data block and its corresponding K data block is closer (or even the same). This facilitates more balanced attention calculation processing across processing devices and improves the problem of uneven computing power.

[0120] Consistent with the scenario where no causal mask is required, after obtaining 2N Q data blocks, 2N K data blocks, and 2N V data blocks, the scenario requiring the application of a causal mask also uses a symmetrical approach, allocating the 2N Q data blocks, 2N K data blocks, and 2N V data blocks to N processing devices respectively. Each processing device is allocated symmetrically 2 Q data blocks, 2 K data blocks, and 2 V data blocks. As shown in Figure 8C, for Q / K / V data, the 0th and (2N-1)th data blocks obtained from the recombination are allocated to processing device 1, the 1st and (2N-2)th data blocks obtained from the recombination are allocated to processing device 2, ..., and the (N-1)th and Nth data blocks obtained from the recombination are allocated to processing device N.

[0121] In this implementation, the electronic device distributes the 2N Q-blocks, 2N K-blocks, and 2N V-blocks obtained from direct segmentation or recombination to N processing devices in a symmetrical manner. This ensures that each processing device performs two sets of operations based on the Q / K / V data blocks used during the initial attention calculation. Furthermore, this facilitates the transmission of two sets of K / V data blocks in a single communication between subsequent processing devices. This not only fully utilizes the transmission bandwidth but also reduces the computational workload per set, as each attention calculation on each processing device is performed in two groups, thus improving computational efficiency.

[0122] 703: Control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of the N processing devices; wherein, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation, and each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0123] In this embodiment, an electronic device controls N processing devices to perform the first attention calculation based on two pre-allocated Q data blocks, two K data blocks, and two V data blocks. Specifically, for the i-th processing device among the N processing devices, the two Q data blocks used during the j-th attention calculation are denoted as pre-allocated data block Q1 and data block Q2, and the two K data blocks are denoted as data block K... 1，j and data block K 2，j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N. When j=1, data block K 1，j Data block K 2，j Data block V 1，j and data block V2，j It is pre-allocated. As shown in Figure 9(a), when j=1, the two Q data blocks allocated to each processing device, along with the K data blocks and V data blocks with the same index, perform attention calculation processing respectively. For example, data block Q1 is the 0th Q data block, and data block K... 1，j For the 0th data block K, data block V 1，j For the 0th V data block, processing device 1 performs a set of attention calculations based on the 0th Q data block, the 0th K data block, and the 0th V data block; data block Q2 is the (2N-1)th Q data block, and data block K... 2，j For the (2N-1)th data block K, data block V 2，j For the (2N-1)th V data block, processing device 1 performs another set of attention calculations based on the (2N-1)th Q data block, the (2N-1)th K data block, and the (2N-1)th V data block.

[0124] When i = 1 and 1 < j ≤ 2N, data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 Data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 1，j-1 Data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ;

[0125] When i = N and 1 < j ≤ 2N, data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 2，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ;

[0126] When 1 < i < N and 1 < j ≤ 2N, data block K 1，jThe data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 .

[0127] Specifically, as shown in Figure 9(a), when the electronic device controls N processing devices to perform the first attention calculation based on 2 pre-allocated Q data blocks, 2 K data blocks, and 2 V data blocks, it also controls the N processing devices to perform the first point-to-point transmission of K / V data blocks. For the second attention calculation of the i-th processing device, as shown in Figure 9(b), if the i-th processing device is processing device 1, then data block K... 2，2 For processing device 1, the 0th data block K is used during the first attention calculation process. 1，2 This is the first K data block used by processing device 2 during the first attention calculation process; data block V 2，2 This refers to the 0th data block V used by processing device 1 during the first attention calculation process. 1，2 This is the first V data block used by processing device 2 during the first attention calculation. If the i-th processing device is processing device N, then data block K... 1，2 Let K be the Nth data block K used by processing device N during the first attention calculation process. 2，2 This refers to the (N+1)th data block K used by the processing device (N-1) during the first attention calculation; data block V 1，2 Let V be the Nth data block V used by processing device N during the first attention calculation process. 2，2 Let K be the (N+1)th V data block used by processing device (N-1) during the first attention calculation. If the i-th processing device is processing device 2, then data block K... 1，2 This is the first K data block used by processing device 3 during the first attention calculation process. 2，2 This refers to the (2N-1)th data block K used by processing device 1 during the first attention calculation process; data block V 1，2This is the first V data block used by processing device 3 during the first attention calculation process. 2，2 This refers to the (2N-1)th V data block used by processing device 1 during the first attention calculation process. Processing device 1 performs one set of attention calculations in the second attention calculation process based on the 0th Q data block, the 1st K data block, and the 1st V data block, and performs another set of attention calculations in the th attention calculation process based on the (2N-1)th Q data block, the 0th K data block, and the 0th V data block. Processing device 2 performs one set of attention calculations in the second attention calculation process based on the first Q data block, the second K data block, and the second V data block, and performs another set of attention calculations in the second attention calculation process based on the (2N-2)th Q data block, the (2N-1)th K data block, and the (2N-1)th V data block; ...; Processing device N performs one set of attention calculations in the second attention calculation process based on the (N-1)th Q data block, the Nth K data block, and the Nth V data block, and performs another set of attention calculations in the second attention calculation process based on the Nth Q data block, the (N+1)th K data block, and the (N+1)th V data block.

[0128] In this implementation, for the i-th processing device among N processing devices, the Q data blocks, K data blocks, and V data blocks used in the first two sets of attention calculations are pre-allocated by the electronic device. When i = 1 or N, one set of K data blocks and V data blocks used in the second and third sets of attention calculations after the first set come from adjacent processing devices, and the other set of K data blocks and V data blocks comes from within the i-th processing device. When i ≠ 1 or N, the K data blocks and V data blocks used in the second and third sets of attention calculations after the first set come from adjacent data blocks. This ensures that each K data block and each V data block can perform an attention calculation with each Q data block, thereby guaranteeing the integrity of the attention calculation between Q data and K and V data.

[0129] Furthermore, when controlling the i-th processing device to perform the j-th attention calculation process, the method also includes:

[0130] When i=1, control the i-th processing device to transmit data block K. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process;

[0131] When i = N, control the i-th processing device to transmit data block K. 1，j and data block V1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process;

[0132] When 1 < i < N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation; it controls the transmission of data block K by the i-th processing device. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process.

[0133] Specifically, as shown in Figure 9(b), when controlling N processing devices to perform the second attention calculation process, control processing device 1 transmits the 0th K data block and V data block to processing device 2, ..., control processing device (N-1) transmits the (N+2)th K data block and V data block to processing device N, control processing device N transmits the Nth K data block and V data block to processing device (N-1), ..., control processing device 2 transmits the 2nd K data block and V data block to processing device 1, so that each processing device can perform the next attention calculation process based on the received K data block and V data block.

[0134] For example, when i=1, the data block K used in its j-th attention calculation process is... 2，j It can be obtained through internal transmission, such as the data block K used in the (j-1)th attention calculation process. 1，j-1 V 1，j-1 The thread in which it is executed is different from the thread used in the j-th attention calculation. When i=N, the data block K used in its j-th attention calculation is... 1，j It can be obtained through internal transmission, such as the data block K used in the (j-1)th attention calculation process. 2，j-1 V 2，j-1 The thread in question is different from the thread executing the j-th attention calculation. Of course, if the execution threads for the two attention calculations are the same, then there is no internal data transfer between processing device 1 and processing device N.

[0135] In this implementation, when the electronic device controls N processing devices to perform the j-th attention calculation, it also controls the N processing devices to perform the j-th transmission of the K data blocks and V data blocks used in the j-th attention calculation. As shown in Figure 9, the K data blocks and V data blocks of the N processing devices rotate in a circular manner, forming an upward flow and a downward flow. This ensures that each processing device has two sets of K data blocks and V data blocks for each attention calculation, realizing two sets of calculations in one communication, which is beneficial for making full use of communication bandwidth and computing power resources.

[0136] Based on the above description, the electronic device controls N processing devices to perform (2N-1) transmissions of K and V data blocks and 2N attention calculations until the (2N-1)th K data block, V data block, and 0th Q data block complete the attention calculation. Depending on the correction method, if correction is performed in each calculation, the attention calculation result of each processing device is two corrected attention calculations; if correction is performed after the calculation is completed, the attention calculation result of each processing device is an intermediate result of 2N attention calculations. The electronic device can then use the result of the j-th calculation to correct the result of the (j+1)-th calculation, obtaining two corrected attention calculations for each processing device. For example, the attention calculation results of the N processing devices are used to predict the image data, text data, or audio data to be processed. For instance, this includes classifying image data, predicting the next sentence of dialogue text data, and matching speakers in audio data.

[0137] For example, in scenarios where causal masking is not required, the two sets of attention calculations executed on each processing device each time can be parallelized, thereby improving computational efficiency. In scenarios where causal masking is required, the two sets of attention calculations executed on each processing device each time also need to consider the masking conditions corresponding to the score matrices of the K and Q data blocks used. That is, attention calculations using FlashAttention2 (an optimized self-attention mechanism) need to be performed based on the masks corresponding to the score matrices. The general steps include calculating the dot product of the Q and K data blocks to obtain the score matrix, applying a mask to the score, and combining the masked score matrix with the V data block to calculate the attention result.

[0138] For example, suppose the Q / K / V data is split into 16 Q data blocks, 16 K data blocks, and 16 V data blocks (labeled with 0-15 respectively). The dot product of each Q data block and each K data block is represented by a rectangular block. Then the preset causal mask of the score matrix of the 16 Q data blocks and the 16 K data blocks can be shown in Figure 10 (where the white area is the masked part and the black area is the unmasked part).

[0139] Based on the causal mask shown in Figure 10, when j=1, the electronic device can determine that the masks corresponding to the score matrices of the Q-data blocks and K-data blocks used for the attention calculations on the i-th processing device are both the first mask. The first mask is also a causal mask, which is a sub-mask with the same shape as the causal mask shown in Figure 10. Taking processing device 1 as an example, its assigned 0th Q-data block is recombined from the split 0th Q-data block and the 15th Q-data block, and the same applies to its assigned 0th K-data block. Using (0,15) as the index of the assigned 0th Q-data block and (0,15) as the index of the assigned 0th K-data block, a rectangular block is taken from the array shown in Figure 10. The corresponding block coordinates are: (0,0), (15,0), (0,15), (15,15). The final rectangular block is shown in Figure 11, which is the sub-mask corresponding to the score matrices of the 0th Q-data block and the 0th K-data block assigned to processing device 1. Based on the same principle, the sub-mask shapes corresponding to the score matrices of the (2N-1)th Q-block and the (2N-1)th K-block allocated on processing device 1 are the same as the sub-mask shapes shown in Figure 11, both being causal masks. That is, during the first attention calculation, the mask situations corresponding to the two sets of attention calculations on each processing device are as shown in Figure 12. In this case, the two sets of attention calculations on the i-th processing device are performed based on the used Q-block, K-block, V-block, and the first mask, respectively. That is, when calculating attention using the attention calculation formula, the value of the first mask needs to be used to occlude the score matrices of the Q-block and K-block.

[0140] In this implementation, when N processing devices are controlled to perform the first attention calculation in a scenario where a causal mask needs to be applied, since the masks corresponding to the two sets of attention calculations on each processing device are the first mask, there is no need to consider the calculation order of the two sets of attention calculations. The two sets of attention calculations can be completed based on the Q data block, K data block, V data block and the first mask used, respectively, making the calculation relatively flexible.

[0141] Based on the causal mask shown in Figure 10, when 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask for the score matrix of the Q-data block and K-data block used in this arbitrary attention calculation is either the second mask or the third mask, and the masks corresponding to the two attention calculations on the i-th processing device are different. Specifically, the second mask is the mask that blocks the midpoint of the first row to column direction of the score matrix, i.e., the upper half of the score matrix; the third mask is the mask that blocks the midpoint of the last column to row direction of the score matrix, i.e., the right half of the score matrix. Taking processing device 1 as an example, its allocated 0th Q-data block is recombined from the split 0th Q-data block and the 15th Q-data block. When j=2, the recombined first K data block and V data block are transmitted to processing device 1. The first K data block is composed of the split first K data block and the 14th K data block. (0,15) is the index of the allocated 0th Q data block, and (1,14) is the index of the transmitted first K data block. A rectangular block is taken from the array shown in Figure 10. The corresponding block coordinates are: (0,1), (15,1), (0,14), (15,14). The finally extracted rectangular block is shown in Figure 13(a), which is the submask corresponding to the score matrix of the 0th Q data block and the 1st K data block. Similarly, the block coordinates corresponding to another set of attention calculations on processing device 1 can be determined as (7,0), (8,0), (7,15), (8,15). The final extracted rectangle is shown in Figure 13(b), which is the sub-mask corresponding to the score matrix of the 0th K data block and the (2N-1)th Q data block. That is to say, in the attention calculation processing after the first time, as the K / V data blocks rotate in a circular manner, the mask situation corresponding to the two sets of attention calculations on each processing device is either as shown in Figure 14(a) or as shown in Figure 14(b). In this case, the two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second mask and the third mask, based on the Q data block, K data block, V data block and the corresponding mask used by each set of attention calculations. For example, when the execution order of the second mask takes precedence over the third mask, each processing device is controlled to first execute the set of attention calculations corresponding to the second mask, and then execute the other set of attention calculations corresponding to the third mask. For example: on processing device 1, the left side of the attention calculation corresponds to the second mask, and the right side of the attention calculation corresponds to the third mask; on processing device 2, the left side of the attention calculation corresponds to the third mask, and the right side of the attention calculation corresponds to the second mask.If the execution order of the third mask takes precedence over the second mask, then the execution order on processing device 1 and processing device 2 is as follows: processing device 1 executes the right-side set of attention calculations, and processing device 2 executes the left-side set of attention calculations; processing device 1 executes the left-side set of attention calculations, and processing device 2 executes the right-side set of attention calculations, thus unifying the mask calculations of N processing devices.

[0142] In this implementation, when one set of attention calculations on the i-th processing device corresponds to the second mask and another set of attention calculations corresponds to the third mask, the electronic device needs to control N processing devices to calculate attention using the same mask at the same time. That is, according to the preset execution order of the second and third masks, the attention calculations corresponding to either the second or third mask are executed uniformly first, and then the attention calculations corresponding to the other mask are executed uniformly. By uniformly using the same mask to calculate attention, the consistency of attention calculation can be ensured, avoiding situations where using inconsistent masks may cause some processing devices to see information that should not be seen during calculation, resulting in the disruption of causal relationships and low usability of calculation results.

[0143] For example, when the mask corresponding to any set of attention calculations is the second mask, the electronic device can control the i-th processing device to discard invalid data. Specifically, as shown in Figure 15, since the second mask occludes the upper half of the score matrix, corresponding to the first half of the Q data block, taking the Q data block as an example where the 0th and 15th parts after splitting are recombined along the sequence dimension, the first half of the Q data block is the 0th part of the split Q data block. This part is discarded, and the 15th part of the split Q data block is retained as the sub-Q data block. That is, the sub-Q data block is the sub-data block from the midpoint of the row direction of the Q data block to the last column of the Q data block. After discarding invalid data, the arbitrary set of attention calculations is performed based on the sub-Q data block, the K data block, and the V data block. That is, the score matrix is ​​calculated using the sub-Q data block and the K data block, and then the attention is calculated in combination with the V data block. There is no need to consider the value of the second mask anymore, and the maskless FlashAttention2 attention calculation is performed. Even when the mask corresponding to any set of attention calculations is the third mask, the electronic device can still control the i-th processing device to discard invalid data. Specifically, as shown in Figure 16, since the third mask occludes the right half of the scoring matrix, corresponding to the latter half of the K / V data block, taking the K / V data block as an example where the 0th and 15th parts of the split data are recombined along the sequence dimension, the latter half of the K data block is the 15th K data block after splitting. This part is discarded, and the 0th K data block after splitting is retained as a sub-K data block, that is, the sub-K data block is the sub-data block from the first column of the K data block to the midpoint of the row direction of the K data block. Similarly, the latter half of the V data block is the 15th V data block after splitting. This part is discarded, and the 0th V data block after splitting is retained as a sub-V data block, that is, the sub-V data block is the sub-data block from the first column of the V data block to the midpoint of the row direction of the V data block. After discarding invalid data, the attention calculation for this arbitrary set is performed based on the Q data block, the sub-K data block, and the sub-V data block. That is, the score matrix is ​​calculated using the Q data block and the sub-K data block, and then the attention is calculated in combination with the sub-V data block. There is no need to consider the value of the third mask. The maskless FlashAttention2 attention calculation is performed.

[0144] In this implementation, for a set of attention calculations on the i-th processing device where the sub-mask is the second mask, based on the occlusion of the second mask, the electronic device can control the processing device to discard invalid data in the used Q data blocks to obtain sub-Q data blocks, and then perform attention calculations based on the sub-Q data blocks, K data blocks, and V data blocks. Similarly, for another set of attention calculations on the i-th processing device where the sub-mask is the third mask, based on the occlusion of the third mask, the electronic device can control the processing device to discard invalid data in the used K data blocks and V data blocks to obtain sub-K data blocks and sub-V data blocks, and then perform attention calculations based on the Q data blocks, sub-K data blocks, and sub-V data blocks. Discarding invalid data reduces unnecessary computational operations, lowers the amount of data computation, and helps to further improve computational efficiency.

[0145] For example, before splitting the Q data, K data, and V data respectively based on the number of processing devices N, the method further includes:

[0146] Receive instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation process.

[0147] In this embodiment, the user can also input the number of parallel attention sequences on the front-end device. For example, when the input instruction is 2, it means that the electronic device needs to enable the dual-stream parallel attention sequence function, that is, control each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively in each attention calculation process. For example, the instruction can also be the user's trigger information for the button to enable the dual-stream parallel attention sequence function. When the user clicks the button to enable the dual-stream parallel attention sequence function, it instructs the electronic device to control each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively in each attention calculation process.

[0148] In this implementation, the electronic device can enable the dual-stream parallel function of the attention sequence based on the user's input instructions, so that each processing device can perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation.

[0149] As can be seen from the embodiments of this application, the electronic device can segment the Q data, K data, and V data based on the number N of parallel processing devices, and finally obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks through direct segmentation or segmentation and recombination. Then, it controls N processing devices to perform 2N attention calculations based on the 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of N processing devices. Each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation. This forms two computational streams on the N processing devices. The dual-stream computation mode makes the computational scale of a single set of attention calculations on any processing device smaller, thus improving the efficiency of attention calculation.

[0150] Please refer to Figure 17, which is a flowchart illustrating another data processing method provided in an embodiment of this application. As shown in Figure 17, the method includes steps 1701-1705:

[0151] 1701: Perform feature extraction on the user input data to obtain query Q data, key K data, and value V data;

[0152] 1702: Based on the number of processing devices N, divide the Q data, K data, and V data into (2... x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1;

[0153] 1703: In a symmetrical manner, respectively (2 x )*N Q data blocks, (2 x )*N K data blocks and (2 x Reorganize N V data blocks to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks;

[0154] Among them, each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

[0155] 1704: Allocate 2N Q data blocks, 2N K data blocks, and 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks symmetrically.

[0156] 1705: Control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of the N processing devices; wherein, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation, and each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0157] The method flow shown in Figure 17 is applicable to scenarios that require the application of a causal mask. In step 1705, the corresponding mask situation needs to be considered each time attention calculation is performed. The specific implementation method has been explained in the embodiment shown in Figure 7, and it can achieve the same or similar beneficial effects.

[0158] Please refer to Figure 18, which is a flowchart illustrating another data processing method provided in an embodiment of this application. As shown in Figure 18, the method includes steps 1801-1804:

[0159] 1801: Perform feature extraction on the user input data to obtain query Q data, key K data, and value V data;

[0160] 1802: Based on the number of processing devices N, divide the Q data, K data, and V data into 2N Q data blocks, 2N K data blocks, and 2N V data blocks respectively;

[0161] 1803: Allocate 2N Q data blocks, 2N K data blocks, and 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks symmetrically.

[0162] 1804: Control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of N processing devices; wherein, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation, and each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0163] The method flow shown in Figure 18 is applicable to scenarios that require the application of a causal mask. In step 1804, the corresponding mask situation does not need to be considered when performing attention calculation each time. The two sets of attention calculations on each processing device can be executed in parallel. The specific implementation method has been described in the embodiment shown in Figure 7, and it can achieve the same or similar beneficial effects.

[0164] Please refer to Figure 19, which is a schematic diagram of a data processing device provided in an embodiment of this application. This device can be applied to an electronic device or a chip in an electronic device. As shown in Figure 19, the device may include a feature extraction unit 1901, a data segmentation unit 1902, and a control unit 1903. Wherein:

[0165] The feature extraction unit 1901 is used to extract features from the user input data to obtain query Q data, key K data, and value V data.

[0166] The data segmentation unit 1902 is used to segment the query Q data, key K data, and value V data based on the number N of processing devices, respectively, to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks;

[0167] The control unit 1903 is used to control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks and 2N V data blocks to obtain the attention calculation results of the N processing devices;

[0168] In each attention calculation, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively. Each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0169] In one possible implementation, the two Q data blocks used by the i-th processing device among the N processing devices when performing the j-th attention calculation are pre-allocated data blocks Q1 and Q2, and the two K data blocks are data blocks K. 1，j and data block K 2， j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N;

[0170] Where, when j=1, data block K 1，j Data block K 2，j Data block V 1，j and data block V 2，j It is pre-allocated;

[0171] When i = 1 and 1 < j ≤ 2N, data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 Data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，jV is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 1，j-1 Data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ;

[0172] When i = N and 1 < j ≤ 2N, data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 2，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ;

[0173] When 1 < i < N and 1 < j ≤ 2N, data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 .

[0174] In one possible implementation, the device further includes a data allocation unit; the data allocation unit is used to allocate 2N Q data blocks, 2N K data blocks and 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks and 2 V data blocks symmetrically.

[0175] In one possible implementation, when j=1, the preset masks for the score matrices of the Q-block and K-block used for the two sets of attention calculations on the i-th processing device are both the first mask; the first mask is a causal mask.

[0176] The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

[0177] In one possible implementation, when 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in any one attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different.

[0178] The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second and third masks, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

[0179] In one possible implementation, the second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix.

[0180] When the mask corresponding to any set of attention calculations is the second mask, any set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; the sub-Q data block is the sub-data block from the midpoint of the row direction of the Q data block to the last column of the Q data block;

[0181] When the mask corresponding to any set of attention calculations is the third mask, any set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is the sub-data block from the first column of the K data block to the midline of the row direction of the K data block; the sub-V data block is the sub-data block from the first column of the V data block to the midline of the row direction of the V data block.

[0182] In one possible implementation, the data segmentation unit 1902 is specifically used to: segment the query Q data, key K data, and value V data respectively to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks.

[0183] The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1;

[0184] Using a symmetrical approach, respectively (2) x )*N Q data blocks, (2 x )*N K data blocks and (2x Recombining N V data blocks yields 2N Q data blocks, 2N K data blocks, and 2N V data blocks; where each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

[0185] In one possible implementation, when controlling the i-th processing device to perform the j-th attention calculation process, the control unit 1903 is further configured to:

[0186] When i=1, control the i-th processing device to transmit data block K. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process;

[0187] When i = N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process;

[0188] When 1 < i < N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation; it controls the transmission of data block K by the i-th processing device. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process.

[0189] In one possible implementation, the control unit 1903 is further configured to: receive instruction information input by the user; the instruction information is configured to enable the function of each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation process.

[0190] It should be noted that the implementation of each unit described in FIG19 can also correspond to the descriptions of the embodiments shown in FIG7 to FIG18. Furthermore, the beneficial effects of the device described in FIG19 can be described in the corresponding descriptions of the embodiments shown in FIG7 to FIG18, and will not be repeated here.

[0191] Based on the description of the above method and device embodiments, this application also provides a computing device 2000. As shown in FIG20, the computing device 2000 includes: a bus 2001, a processor 2002, a memory 2003, and a communication interface 2004. The processor 2002, the memory 2003, and the communication interface 2004 communicate with each other via the bus 2001. The computing device 2000 can be a server or a terminal device; for example, the computing device 2000 can be the electronic device shown in the embodiments of FIG7 to FIG18. It should be understood that this application does not limit the number of processors and memories in the computing device 2000.

[0192] Bus 2001 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 20, but this does not imply that there is only one bus or one type of bus. Bus 2001 can include pathways for transmitting information between various components of computing device 2000 (e.g., memory 2003, processor 2002, communication interface 2004).

[0193] Processor 2002 may include any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device.

[0194] Memory 2003 may include volatile memory, such as random access memory (RAM). Processor 2002 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DR RAM).

[0195] The memory 2003 stores executable program code, which the processor 2002 executes to implement the functions of the aforementioned feature extraction unit, data segmentation unit, and control unit, thereby implementing the methods of the embodiments shown in Figures 7 to 18. That is, the memory 2003 stores instructions for executing the methods of the embodiments shown in Figures 7 to 18.

[0196] Alternatively, the memory 2003 stores executable code, which the processor 2002 executes to implement the functions of the aforementioned data processing device, thereby implementing the methods of the embodiments shown in Figures 7 to 18. That is, the memory 2003 stores instructions for executing the methods of the embodiments shown in Figures 7 to 18.

[0197] The communication interface 2004 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 2000 and other devices or communication networks.

[0198] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone. For example, the computing device can be one of the aforementioned electronic devices.

[0199] As shown in Figure 21, the computing device cluster includes at least one computing device 2000. The memory 2003 of one or more computing devices 2000 in the computing device cluster may store the same instructions for performing the methods of the embodiments shown in Figures 7 to 18.

[0200] In some possible implementations, the memory 2003 of one or more computing devices 2000 in the computing device cluster may also store partial instructions for executing the methods of the embodiments shown in Figures 7 to 18. In other words, a combination of one or more computing devices 2000 can jointly execute the instructions for executing the methods of the embodiments shown in Figures 7 to 18.

[0201] It should be noted that the memory 2003 in different computing devices 2000 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 2003 of different computing devices 2000 can implement the functions of one or more unit modules in the feature extraction unit, data segmentation unit, and control unit.

[0202] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 22 illustrates one possible implementation. As shown in Figure 22, computing device 2000A and computing device 2000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 2003 in computing device 2000A stores instructions for performing the functions of the feature extraction unit. Simultaneously, the memory 2003 in computing device 2000B stores instructions for performing the functions of the data segmentation unit and the control unit.

[0203] The connection method between the computing device clusters shown in Figure 22 can be such that, considering the data processing method provided in this application requires feature extraction, data segmentation and data processing, the functions implemented by the data segmentation unit and the control unit are delegated to computing device 2000B, while the functions implemented by the feature extraction unit are delegated to computing device 2000A.

[0204] It should be understood that the functions of computing device 2000A shown in Figure 22 can also be performed by multiple computing devices 2000. Similarly, the functions of computing device 2000B can also be performed by multiple computing devices 2000.

[0205] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 21 and 22. The difference is that the memory 2003 of one or more computing devices 2000 in this computing device cluster can store the same instructions for executing the data processing method provided in this application.

[0206] In some possible implementations, the memory 2003 of one or more computing devices 2000 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 2000 can jointly execute instructions for executing data processing methods.

[0207] It should be noted that although the computing device shown in Figure 20 only illustrates the bus 2001, processor 2002, memory 2003, and communication interface 2004, those skilled in the art should understand that in specific implementations, the computing device may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the computing device may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in Figure 20.

[0208] This application also provides a chip, including: a processor for calling and running a computer program from a memory, causing a device with the chip installed to perform the method described in any of the embodiments shown in Figures 7 to 18 above. This chip may be a chip in a computing device.

[0209] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, the at least one computing device performs the method described in any of the embodiments of Figures 7 to 18.

[0210] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the method described in any of the embodiments of Figures 7 to 18.

[0211] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0212] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed computing device clusters, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely exemplary. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0216] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0217] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0218] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0219] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: Feature extraction is performed on the user input data to obtain query Q data, key K data, and value V data; Based on the number of processing devices N, the Q data, K data, and V data are divided into 2N Q data blocks, 2N K data blocks, and 2N V data blocks respectively; Control N processing devices to perform 2N attention calculations based on the 2N Q data blocks, the 2N K data blocks, and the 2N V data blocks to obtain the attention calculation results of the N processing devices; Each of the processing devices performs two sets of attention calculations based on two Q data blocks, two K data blocks, and two V data blocks during each attention calculation process. Each set of attention calculations is performed based on one Q data block, one K data block, and one V data block.

2. The method according to claim 1, characterized in that, When the i-th processing device among the N processing devices performs the j-th attention calculation, the two Q data blocks used are pre-allocated data block Q1 and data block Q2, and the two K data blocks are data block K. 1，j and data block K 2，j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N; Wherein, when j=1, the data block K 1，j The data block K 2，j The data block V 1，j and the data block V 2，j It is pre-allocated; When i = 1 and 1 < j ≤ 2N, the data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 The data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block V 2，j The data block V used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 The data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ; When i = N and 1 < j ≤ 2N, the data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 The data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 The data block V 1，j The data block V used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 The data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ; When 1 < i < N and 1 < j ≤ 2N, the data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 The data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 .

3. The method according to claim 1 or 2, characterized in that, After obtaining 2N Q data blocks, 2N K data blocks, and 2N V data blocks, the method further includes: The 2N Q data blocks, 2N K data blocks, and 2N V data blocks are allocated to N processing devices in a symmetrical manner, with each processing device receiving 2 Q data blocks, 2 K data blocks, and 2 V data blocks.

4. The method according to claim 2, characterized in that, When j=1, the preset mask for the score matrix of the Q data block and K data block used for the two sets of attention calculations on the i-th processing device is the first mask; The first mask is a causal mask; The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

5. The method according to claim 2, characterized in that, When 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in the attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different; The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second mask and the third mask, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

6. The method according to claim 5, characterized in that, The second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix. When the mask corresponding to any set of attention calculations is the second mask, the set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; The sub-Q data block is a sub-data block extending from the midpoint of the row direction of the Q data block to the last column of the Q data block; When the mask corresponding to any set of attention calculations is the third mask, the set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is a sub-data block from the first column of the K data block to the midline in the row direction of the K data block; the sub-V data block is a sub-data block from the first column of the V data block to the midline in the row direction of the V data block.

7. The method according to any one of claims 4-6, characterized in that, The step of splitting the query Q data, key K data, and value V data into 2N Q data blocks, 2N K data blocks, and 2N V data blocks includes: The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1; In a symmetrical manner, respectively, the (2) x )*N Q data blocks, the (2 x )*N K data blocks and the (2 x The 2N V data blocks are reassembled to obtain the 2N Q data blocks, the 2N K data blocks, and the 2N V data blocks; wherein, each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

8. The method according to any one of claims 2-7, characterized in that, When controlling the i-th processing device to perform the j-th attention calculation process, the method further includes: When i=1, control the i-th processing device to transmit the data block K. 2，j and the data block V 2，j To the (i+1)th processing device, to process the data block K 2，j and the data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention calculation process; When i = N, control the i-th processing device to transmit the data block K. 1，j and the data block V 1，j To the (i-1)th processing device, to process the data block K 1，j and the data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process; When 1 < i < N, control the i-th processing device to transmit the data block K. 1，j and the data block V 1，j To the (i-1)th processing device, to process the data block K 1，j and the data block V 1，j As a set of K data blocks and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process; controlling the i-th processing device to transmit the data block K 2，j and the data block V 2，j To the (i+1)th processing device, to process the data block K 2，j and the data block V 2，j This is a set of K data blocks and V data blocks used by the (i+1)th processing device during the (j+1)th attention calculation process.

9. The method according to any one of claims 1-8, characterized in that, Before segmenting the Q data, K data, and V data based on the number of processing devices N, the method further includes: The system receives instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively during each attention calculation process.

10. A data processing apparatus, characterized in that, The device includes a feature extraction unit, a data segmentation unit, and a control unit; wherein: The feature extraction unit is used to extract features from the user-input data to obtain query Q data, key K data, and value V data; The data segmentation unit is used to segment the Q data, the K data, and the V data based on the number N of processing devices, respectively, to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks; The control unit is configured to control N processing devices to perform 2N attention calculations based on the 2N Q data blocks, the 2N K data blocks, and the 2N V data blocks, so as to obtain the attention calculation results of the N processing devices; Each of the processing devices performs two sets of attention calculations based on two Q data blocks, two K data blocks, and two V data blocks during each attention calculation process. Each set of attention calculations is performed based on one Q data block, one K data block, and one V data block.

11. The apparatus according to claim 10, characterized in that, When the i-th processing device among the N processing devices performs the j-th attention calculation, the two Q data blocks used are pre-allocated data block Q1 and data block Q2, and the two K data blocks are data block K. 1，j and data block K 2，j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N; Wherein, when j=1, the data block K 1，j The data block K 2，j The data block V 1，j and the data block V 2，j It is pre-allocated; When i = 1 and 1 < j ≤ 2N, the data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 The data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block V 2，j The data block V used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 The data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ; When i = N and 1 < j ≤ 2N, the data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 The data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 The data block V 1，j The data block V used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 The data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ; When 1 < i < N and 1 < j ≤ 2N, the data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 The data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 .

12. The apparatus according to claim 10 or 11, characterized in that, The device further includes a data allocation unit; the data allocation unit is used to: allocate the 2N Q data blocks, the 2N K data blocks and the 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks and 2 V data blocks symmetrically.

13. The apparatus according to claim 11, characterized in that, When j=1, the preset mask for the score matrix of the Q data block and K data block used for the two sets of attention calculations on the i-th processing device is the first mask; The first mask is a causal mask; The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

14. The apparatus according to claim 11, characterized in that, When 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in the attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different; The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second mask and the third mask, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

15. The apparatus according to claim 14, characterized in that, The second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix. When the mask corresponding to any set of attention calculations is the second mask, the set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; The sub-Q data block is a sub-data block extending from the midpoint of the row direction of the Q data block to the last column of the Q data block; When the mask corresponding to any set of attention calculations is the third mask, the set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is a sub-data block from the first column of the K data block to the midline in the row direction of the K data block; the sub-V data block is a sub-data block from the first column of the V data block to the midline in the row direction of the V data block.

16. The apparatus according to any one of claims 13-15, characterized in that, In terms of splitting the query Q data, key K data, and value V data respectively to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks, the data splitting unit is specifically used for: The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1; In a symmetrical manner, respectively, the (2) x )*N Q data blocks, the (2 x )*N K data blocks and the (2 x The 2N V data blocks are reassembled to obtain the 2N Q data blocks, the 2N K data blocks, and the 2N V data blocks; wherein, each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

17. The apparatus according to any one of claims 11-16, characterized in that, When controlling the i-th processing device to perform the j-th attention calculation process, the control unit is further configured to: When i=1, control the i-th processing device to transmit the data block K. 2，j and the data block V 2，j To the (i+1)th processing device, to process the data block K 2，j and the data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention calculation process; When i = N, control the i-th processing device to transmit the data block K. 1，j and the data block V 1，j To the (i-1)th processing device, to process the data block K 1，j and the data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process; When 1 < i < N, control the i-th processing device to transmit the data block K. 1，j and the data block V 1，j To the (i-1)th processing device, to process the data block K 1，j and the data block V 1，j As a set of K data blocks and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process; controlling the i-th processing device to transmit the data block K 2，j and the data block V 2，j To the (i+1)th processing device, to process the data block K 2，j and the data block V 2，j This is a set of K data blocks and V data blocks used by the (i+1)th processing device during the (j+1)th attention calculation process.

18. The apparatus according to any one of claims 10-17, characterized in that, The control unit is also used for: The system receives instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively during each attention calculation process.

19. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-9.

20. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1-9.

21. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-9.

Images