Data processing method and device, computing equipment cluster and storage medium

Abstract

The invention discloses a data processing method and device, a computing device cluster and a storage medium, and the method comprises the steps: carrying out the feature extraction of data inputted by a user, and obtaining query Q data, key K data and value V data; based on the number N of the processing devices, respectively segmenting the query Q data, the key K data and the value V data to obtain 2N Q data blocks, 2N K data blocks and 2N V data blocks; controlling the N processing devices to execute 2N times of attention calculation processing based on the 2N Q data blocks, the 2N K data blocks and the 2N V data blocks so as to obtain attention calculation results of the N processing devices; wherein each processing device executes two groups of attention calculation based on the two Q data blocks, the two K data blocks and the two V data blocks during each time of attention calculation processing. According to the method and the device, attention long sequence processing can be realized on multiple processing devices through two data streams, and the efficiency of sequence parallel processing can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, computing device cluster and storage medium. Background Technology

[0002] In the field of large-scale artificial intelligence models, the length of sequences to be processed is constantly increasing. Since the capabilities of devices supporting model training and inference are typically limited, related technologies often employ long-sequence parallelism to improve processing efficiency. Sequence parallelism in attention mechanisms has received considerable attention in recent years. By segmenting the input query (Q), key (K), and value (V), the data originally input to attention is divided into multiple parts along the sequence dimension, and these parts are distributed to different devices for processing. Common sequence parallel attention mechanisms include ring-attention and striped attention, but the efficiency of sequence parallel processing remains relatively low. Summary of the Invention

[0003] This application provides a data processing method, apparatus, computing device cluster, and storage medium that can achieve long-sequence parallel processing of attention on multiple processing devices with two data streams. Each processing device performs two sets of attention calculations in each iteration, which helps to improve the efficiency of sequence parallel processing.

[0004] Firstly, embodiments of this application provide a data processing method applied to an electronic device or a chip within an electronic device. Taking the application of this method to an electronic device as an example, the method includes:

[0005] Feature extraction is performed on the user input data to obtain query Q data, key K data, and value V data;

[0006] Based on the number of processing devices N, the Q data, K data, and V data are divided into 2N Q data blocks, 2N K data blocks, and 2N V data blocks respectively.

[0007] Control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of the N processing devices;

[0008] In each attention calculation, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively. Each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0009] As can be seen from the embodiments of this application, the electronic device can segment the Q data, K data, and V data based on the number N of parallel processing devices, and finally obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks through direct segmentation or segmentation and recombination. Then, it controls N processing devices to perform 2N attention calculations based on the 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of N processing devices. Each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation. This forms two computational streams on the N processing devices. The dual-stream computation mode makes the computational scale of a single set of attention calculations on any processing device smaller, thus improving the efficiency of attention calculation.

[0010] In one possible implementation, the two Q data blocks used by the i-th processing device among the N processing devices when performing the j-th attention calculation are pre-allocated data blocks Q1 and Q2, and the two K data blocks are data blocks K. 1，j and data block K 2，j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N;

[0011] Where, when j=1, data block K 1，j Data block K 2，j Data block V 1，j and data block V 2，j It is pre-allocated;

[0012] When i = 1 and 1 < j ≤ 2N, data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 Data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 1，j-1 Data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ;

[0013] When i = N and 1 < j ≤ 2N, data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 2，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ;

[0014] When 1 < i < N and 1 < j ≤ 2N, data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2， j -1 .

[0015] In this implementation, for the i-th processing device among N processing devices, the Q data blocks, K data blocks, and V data blocks used in the first two sets of attention calculations are pre-allocated by the electronic device. When i = 1 or N, one set of K data blocks and V data blocks used in the second and third sets of attention calculations after the first set come from adjacent processing devices, and the other set of K data blocks and V data blocks comes from within the i-th processing device. When i ≠ 1 or N, the K data blocks and V data blocks used in the second and third sets of attention calculations after the first set come from adjacent data blocks. This ensures that each K data block and each V data block can perform an attention calculation with each Q data block, thereby guaranteeing the integrity of the attention calculation between Q data and K and V data.

[0016] In one possible implementation, after obtaining 2N Q data blocks, 2N K data blocks, and 2N V data blocks, the method further includes:

[0017] In a symmetrical manner, 2N Q data blocks, 2N K data blocks, and 2N V data blocks are allocated to N processing devices respectively; each processing device is allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks symmetrically.

[0018] In this implementation, the electronic device distributes the 2N Q-blocks, 2N K-blocks, and 2N V-blocks obtained from direct segmentation or recombination to N processing devices in a symmetrical manner. This ensures that each processing device performs two sets of operations based on the Q / K / V data blocks used during the initial attention calculation. Furthermore, this facilitates the transmission of two sets of K / V data blocks in a single communication between subsequent processing devices. This not only fully utilizes the transmission bandwidth but also reduces the computational workload per set, as each attention calculation on each processing device is performed in two groups, thus improving computational efficiency.

[0019] In one possible implementation, when j=1, the preset masks for the score matrices of the Q-block and K-block used for the two sets of attention calculations on the i-th processing device are both the first mask; the first mask is a causal mask.

[0020] The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

[0021] In this implementation, when N processing devices are controlled to perform the first attention calculation in a scenario where a causal mask needs to be applied, since the masks corresponding to the two sets of attention calculations on each processing device are the first mask, there is no need to consider the calculation order of the two sets of attention calculations. The two sets of attention calculations can be completed based on the Q data block, K data block, V data block and the first mask used, respectively, making the calculation relatively flexible.

[0022] In one possible implementation, when 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in any one attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different.

[0023] The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second and third masks, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

[0024] In this implementation, when one set of attention calculations on the i-th processing device corresponds to the second mask and another set of attention calculations corresponds to the third mask, the electronic device needs to control N processing devices to calculate attention using the same mask at the same time. That is, according to the preset execution order of the second and third masks, the attention calculations corresponding to either the second or third mask are executed uniformly first, and then the attention calculations corresponding to the other mask are executed uniformly. By uniformly using the same mask to calculate attention, the consistency of attention calculation can be ensured, avoiding situations where using inconsistent masks may cause some processing devices to see information that should not be seen during calculation, resulting in the disruption of causal relationships and low usability of calculation results.

[0025] In one possible implementation, the second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix.

[0026] When the mask corresponding to any set of attention calculations is the second mask, any set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; the sub-Q data block is the sub-data block from the midpoint of the row direction of the Q data block to the last column of the Q data block;

[0027] When the mask corresponding to any set of attention calculations is the third mask, any set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is the sub-data block from the first column of the K data block to the midline of the row direction of the K data block; the sub-V data block is the sub-data block from the first column of the V data block to the midline of the row direction of the V data block.

[0028] In this implementation, for a set of attention calculations on the i-th processing device where the sub-mask is the second mask, based on the occlusion of the second mask, the electronic device can control the processing device to discard invalid data in the used Q data blocks to obtain sub-Q data blocks, and then perform attention calculations based on the sub-Q data blocks, K data blocks, and V data blocks. Similarly, for another set of attention calculations on the i-th processing device where the sub-mask is the third mask, based on the occlusion of the third mask, the electronic device can control the processing device to discard invalid data in the used K data blocks and V data blocks to obtain sub-K data blocks and sub-V data blocks, and then perform attention calculations based on the Q data blocks, sub-K data blocks, and sub-V data blocks. Discarding invalid data reduces unnecessary computational operations, lowers the amount of data computation, and helps to further improve computational efficiency.

[0029] In one possible implementation, the query Q data, key K data, and value V data are respectively split to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks, including:

[0030] The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1;

[0031] Using a symmetrical approach, respectively (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x Recombining N V data blocks yields 2N Q data blocks, 2N K data blocks, and 2N V data blocks; where each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

[0032] In this implementation, for scenarios requiring causal masking, the Q, K, and V data can be further segmented into finer-grained segments, and then the segmented (2) data can be processed in a symmetrical manner from beginning to end. x )*N Q data blocks, (2 x )*N K data blocks and (2 x N V data blocks are reorganized into 2N Q data blocks, 2N K data blocks, and 2N V data blocks. This symmetrical reorganization method ensures that the amount of data obscured by the mask in the score matrix of each Q data block and its corresponding K data block is closer (or even the same). This facilitates more balanced attention calculation processing across processing devices and improves the problem of uneven computing power.

[0033] In one possible implementation, when controlling the i-th processing device to perform the j-th attention calculation, the method further includes:

[0034] When i=1, control the i-th processing device to transmit data block K. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process;

[0035] When i = N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process;

[0036] When 1 < i < N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation; it controls the transmission of data block K by the i-th processing device. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process.

[0037] In this implementation, when the electronic device controls N processing devices to perform the j-th attention calculation, it also controls the N processing devices to perform the j-th transmission of the K data blocks and V data blocks used in the j-th attention calculation. The K data blocks and V data blocks of the N processing devices rotate in a circular manner, forming an upward flow and a downward flow, thereby ensuring that each processing device has two sets of K data blocks and V data blocks in each attention calculation. This achieves two sets of calculations in one communication, which is beneficial for making full use of communication bandwidth and computing resources.

[0038] In one possible implementation, before splitting the Q data, K data, and V data respectively based on the number of processing devices N, the method further includes:

[0039] Receive instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation process.

[0040] In this implementation, the electronic device can enable the dual-stream parallel function of the attention sequence based on the user's input instructions, so that each processing device can perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation.

[0041] Secondly, embodiments of this application provide a data processing apparatus applied to an electronic device or a chip in an electronic device; the apparatus includes a feature extraction unit, a data segmentation unit, and a control unit; wherein:

[0042] The feature extraction unit is used to extract features from the user input data to obtain query Q data, key K data, and value V data;

[0043] The data segmentation unit is used to segment the Q data, K data, and V data based on the number of processing devices N, respectively, to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks;

[0044] The control unit is used to control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of the N processing devices;

[0045] In each attention calculation, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively. Each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0046] In one possible implementation, the two Q data blocks used by the i-th processing device among the N processing devices when performing the j-th attention calculation are pre-allocated data blocks Q1 and Q2, and the two K data blocks are data blocks K. 1，j and data block K 2，j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N;

[0047] Where, when j=1, data block K 1，j Data block K 2，j Data block V 1，j and data block V 2，j It is pre-allocated;

[0048] When i = 1 and 1 < j ≤ 2N, data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 Data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 1，j-1 Data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ;

[0049] When i = N and 1 < j ≤ 2N, data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 2，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ;

[0050] When 1 < i < N and 1 < j ≤ 2N, data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2， j -1 .

[0051] In one possible implementation, the device further includes a data allocation unit; the data allocation unit is used to allocate 2N Q data blocks, 2N K data blocks and 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks and 2 V data blocks symmetrically.

[0052] In one possible implementation, when j=1, the preset masks for the score matrices of the Q-block and K-block used for the two sets of attention calculations on the i-th processing device are both the first mask; the first mask is a causal mask.

[0053] The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

[0054] In one possible implementation, when 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in any one attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different.

[0055] The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second and third masks, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

[0056] In one possible implementation, the second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix.

[0057] When the mask corresponding to any set of attention calculations is the second mask, any set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; the sub-Q data block is the sub-data block from the midpoint of the row direction of the Q data block to the last column of the Q data block;

[0058] When the mask corresponding to any set of attention calculations is the third mask, any set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is the sub-data block from the first column of the K data block to the midline of the row direction of the K data block; the sub-V data block is the sub-data block from the first column of the V data block to the midline of the row direction of the V data block.

[0059] In one possible implementation, the data splitting unit is specifically used to split the query Q data, key K data, and value V data respectively to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks:

[0060] The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1;

[0061] Using a symmetrical approach, respectively (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x Recombining N V data blocks yields 2N Q data blocks, 2N K data blocks, and 2N V data blocks; where each symmetrical 2 x-12 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

[0062] In one possible implementation, when controlling the i-th processing device to perform the j-th attention calculation, the control unit is further configured to:

[0063] When i=1, control the i-th processing device to transmit data block K. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process;

[0064] When i = N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process;

[0065] When 1 < i < N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation; it controls the transmission of data block K by the i-th processing device. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process.

[0066] In one possible implementation, the control unit is also used for:

[0067] Receive instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation process.

[0068] It should be understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the first aspect of the embodiments of this application should be adapted to the second aspect of the embodiments of this application simultaneously, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0069] Thirdly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method as described in any of the embodiments of the first aspect above.

[0070] Fourthly, embodiments of this application provide a chip, including: a processor, configured to call and run a computer program from a memory, causing a device on which the chip is installed to perform the method as described in any of the embodiments of the first aspect above.

[0071] Fifthly, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, implement the method as described in any of the embodiments of the first aspect above.

[0072] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any of the embodiments of the first aspect above. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0074] Figure 1 A schematic diagram of a transfer key data block provided for related technologies;

[0075] Figure 2 A schematic diagram of a transmission value data block provided for related technologies;

[0076] Figure 3 A schematic diagram illustrating data segmentation and allocation for related technologies;

[0077] Figure 4 A schematic diagram illustrating another data segmentation and allocation method for related technologies;

[0078] Figure 5 A schematic diagram of a system architecture provided for an embodiment of this application;

[0079] Figure 6A schematic diagram illustrating front-end and back-end processing logic provided in an embodiment of this application;

[0080] Figure 7 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0081] Figure 8A A schematic diagram illustrating data segmentation and allocation provided in an embodiment of this application;

[0082] Figure 8B A schematic diagram illustrating data segmentation and recombination as provided in an embodiment of this application;

[0083] Figure 8C A schematic diagram illustrating data segmentation, recombination, and allocation provided in an embodiment of this application;

[0084] Figure 9 A schematic diagram illustrating data processing and transmission provided in an embodiment of this application;

[0085] Figure 10 A schematic diagram of a causal mask provided in an embodiment of this application;

[0086] Figure 11 A schematic diagram of a first mask provided for an embodiment of this application;

[0087] Figure 12 A schematic diagram of two sets of sub-masks corresponding to attention calculations on a processing device provided in an embodiment of this application;

[0088] Figure 13 A schematic diagram of a second mask and a third mask provided for embodiments of this application;

[0089] Figure 14 A schematic diagram of the sub-masks corresponding to two sets of attention calculations on another processing device provided in an embodiment of this application;

[0090] Figure 15 This is a schematic diagram illustrating the discarding of invalid data as provided in an embodiment of this application.

[0091] Figure 16 This is a schematic diagram illustrating the discarding of invalid data as provided in an embodiment of this application.

[0092] Figure 17 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0093] Figure 18 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0094] Figure 19 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0095] Figure 20 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0096] Figure 21 A schematic diagram of a computing device cluster provided in an embodiment of this application;

[0097] Figure 22 This is a schematic diagram illustrating a connection method between computing devices provided in an embodiment of this application. Detailed Implementation

[0098] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0099] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0100] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on electronic devices and electronic devices themselves can be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0101] First, a brief introduction to the relevant terms and technical background used in this application will be provided to facilitate understanding by those skilled in the art.

[0102] (1) Graphics Processing Unit (GPU): A typical artificial intelligence computing device.

[0103] (2) Attention: The attention module in the transformer model is a model structure in artificial intelligence machine learning.

[0104] (3) Neural Processing Unit (NPU): A hardware accelerator specifically designed to accelerate neural network computation.

[0105] (4) Tensor Processing Unit (TPU): A hardware accelerator specifically designed to accelerate machine learning and deep learning computations. The TPU is designed to improve the efficiency of processing large-scale tensor computations and is widely used in deep learning models.

[0106] (5) Rank: The smallest unit of device that performs long-sequence parallelism, which can be one or more GPUs, or one or more devices or machines.

[0107] The current mainstream technique for long-sequence parallelism in attention is ring attention (ring-self-attention / ring-attention). Its principle is to divide the Q / K / V data into slices, allowing different devices to receive a slice of the Q / K / V data, such as... Figure 1 As shown, let K be iterated and calculated first (QK is calculated first in the attention calculation). T Based on QK T (and V to obtain the output), after each calculation, the fragment K is passed to the next device in a circular manner (or transmitted simultaneously); such as Figure 2 As shown, after the circular calculation of slice K is completed, slice V is iteratively calculated. After each calculation, slice V is passed to the next device in a circular manner. In the above calculation, one way to combine the blockwise attention algorithm is as follows: First, the data Q / K / V is sliced. Then, each device is fixed with one slice Q. Slices K and V are passed cyclically between different devices. Each iteration completes one calculation. Each device passes the K / V slice data used for calculation to the next device in a circular form. Each K / V slice data can be iteratively calculated once on each device. After all iterations are completed, the attention calculation is completed.

[0108] The slicing and allocation methods of ring-attention can be as follows: Figure 3 As shown, assuming there are 4 devices, the Q / K / V data can be divided into 16 parts each, and then each set of 4 Q / K / V data is sequentially assigned to the same device. For example, parts 0 to 3 are assigned to device 1, parts 4 to 7 to device 2, parts 8 to 11 to device 3, and parts 12 to 15 to device 4. In ring-attention computation, causal masks are often used. However, according to the data block allocation method of ring-attention, attention computation using causal masks suffers from a serious problem of computational imbalance. To solve this problem, striped attention has been proposed. The slicing and allocation method of striped attention can be described as follows: Figure 4 As shown, assuming there are 4 devices, the Q / K / V data can be divided into 16 parts each. Four parts of the Q / K / V data are then permuted and assigned to the same device. For example, parts 0, 4, 8, and 12 are assigned to device 1; parts 1, 5, 9, and 13 to device 2; parts 2, 6, 10, and 14 to device 3; and parts 3, 7, 11, and 15 to device 4. While Striped attention can improve the problem of computational imbalance by permuting, it fails to effectively eliminate the computational imbalance when applying a causal mask, and the improvement depends on the granularity of the Q / K / V data segmentation. Like ring-attention, Striped attention still uses a single data stream to calculate attention. Serial computation of a single data stream across multiple devices results in low computational efficiency. Furthermore, both ring-attention and Striped attention transmit only one block of K / V data per communication, leading to insufficient utilization of transmission bandwidth.

[0109] To overcome the shortcomings of the prior art, embodiments of this application provide a data processing method, apparatus, electronic device, and storage medium. The data processing method can be based on... Figure 5 The system architecture implementation is shown below. Figure 5 As shown, the system architecture includes front-end devices, back-end devices, and a processing device cluster. The processing devices in the cluster include, but are not limited to, a Central Processing Unit (CPU), GPU, NPU, and TPU. One or more processing devices can also form a Rank. The front-end devices can be used to input image data, text data, or audio data to be processed by the user. Further, such as... Figure 6As shown, users can choose whether to enable features (such as the dual-stream parallel attention sequence function) on the user interface of the front-end device. If the function is enabled, the number of parallel attention sequences (such as the number of processing devices) must be entered, and the correction method for the attention calculation results must be selected. Based on the user's operation, a parallel attention sequence task or project for image data, text data, or audio data to be processed is generated. The sequential parallel task enters the resource pool of the backend device. The scheduling module of the backend device parses the user parameters (such as the number of parallel attention sequences, the correction method of attention calculation results, etc.) through a custom function or tool. The backend device can encode the image data, text data, or audio data to obtain the corresponding Q data, K data, and V data, and further divide the Q data, K data, and V data to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks (N is the number of parallel attention sequences). The scheduling module allocates the 2N Q data blocks, 2N K data blocks, and 2N V data blocks in a symmetrical manner, and distributes them to a preset number of processing device clusters in the form of user Pods (which are container instances or container groups defined and run by the user in a distributed cluster). The backend device controls the processing device allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks to perform attention dual-stream calculation according to the data processing method provided in this application, and corrects the attention calculation results output by each processing device according to the correction method selected by the user. In another implementation, the user does not need to input the relevant parameters for dual-stream parallel processing on the front end. The attention dual-stream computing algorithm can be deployed on the back end in code. When the front end device inputs image data, text data or audio data to be processed, the back end schedules and recognizes the code, and executes the attention dual-stream algorithm by default.

[0110] For example, the correction method for attention calculation results provided in the embodiments of this application includes: 1) performing a correction once for each attention calculation on each processing device; 2) if there is no correction during the calculation process, saving the two attention results calculated each time on each processing device, and performing a unified correction after the algorithm is completed.

[0111] For example, Figure 5 The backend devices in the cloud service can be service devices that provide cloud services, and the frontend devices can be user devices that use cloud services.

[0112] The technical solution provided in this application will be described in detail below with reference to specific implementation methods.

[0113] Please see Figure 7 , Figure 7This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method can be executed by an electronic device, such as... Figure 5 Backend devices in the system. For example... Figure 7 As shown, the method includes steps 701-703:

[0114] 701: Extract features from the user-input data to obtain query Q data, key K data, and value V data.

[0115] 702: Based on the number of processing devices N, the Q data, K data and V data are divided into 2N Q data blocks, 2N K data blocks and 2N V data blocks respectively.

[0116] In this embodiment, Q data, K data, and V data can be obtained by feature extraction from user-input image data, text data, or audio data. The feature extraction process can refer to the process of generating Q / K / V matrices in Transformer. For scenarios where causal masking is not required, if the number of processing devices (i.e., the number of parallel processing devices) is N, the Q data, K data, and V data can be divided into 2N Q data blocks, 2N K data blocks, and 2N V data blocks, respectively. Then, in a symmetrical manner, the 2N Q data blocks, 2N K data blocks, and 2N V data blocks are allocated to N processing devices; each processing device is allocated symmetrically 2 Q data blocks, 2 K data blocks, and 2 V data blocks. Figure 8A As shown, assuming N=4, the Q data, K data, and V data can be divided into 8 parts respectively. Then, in a symmetrical manner, 2 parts of Q data, 2 parts of K data, and 2 parts of V data are allocated to 4 processing devices. For example, the data is marked starting from part 0; parts 0 and 7 are allocated to processing device 1, parts 1 and 6 to processing device 2, parts 2 and 5 to processing device 3, and parts 3 and 4 to processing device 4. The phrase "each processing device is allocated 2 symmetrical Q data blocks, 2 K data blocks, and 2 V data blocks" can be understood as "each processing device is allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks that are axially symmetrical about the axis of symmetry of the row directions of 2N Q data blocks, 2N K data blocks, and 2N V data blocks."

[0117] For scenarios requiring causal masking, since the improvement in computing power imbalance depends on the granularity of data segmentation, this embodiment can perform finer-grained segmentation compared to scenarios not requiring causal masking. If the number of processing devices is N, then the Q data, K data, and V data are respectively segmented into (2... x )*N Q data blocks, (2 x )*N K data blocks and (2 x()*N V data blocks; where x is an integer greater than 1. For example, the Q data, K data, and V data can be divided into 4N Q data blocks, 4N K data blocks, and 4N V data blocks, or into 8N Q data blocks, 8N K data blocks, and 8N V data blocks, and so on. Furthermore, in a symmetrical manner, (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x Recombining N V data blocks yields 2N Q data blocks, 2N K data blocks, and 2N V data blocks. Each symmetrical 2... x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Individual V data blocks are reassembled into one V data block. For example... Figure 8B As shown, the Q data, K data, and V data are divided into 4N Q data blocks, 4N K data blocks, and 4N V data blocks, respectively. Starting from the 0th block, the 0th data block and the (4N-1)th data block are recombined into the 0th data block; the 1st data block and the (4N-2)th data block are recombined into the 1st data block; and so on, until the (2N-1)th data block and the 2Nth data block are recombined into the (2N-1)th data block. This results in 2N Q data blocks, 2N K data blocks, and 2N V data blocks. Wherein, "each symmetrical 2 x-1 "Reorganizing 2 Q data blocks into one Q data block" can be understood as "using (2) x The axis of symmetry in the row direction of N Q data blocks is 2. x-1 "Reorganizing Q data blocks into one Q data block": Assuming the Q data is divided into 4N Q data blocks, the two axis-symmetric data blocks are the 0th data block and the (4N-1)th data block. Assuming the Q data is divided into 8N Q data blocks, the four axis-symmetric data blocks are the 0th data block, the 1st data block, the (4N-2)th data block, and the (4N-1)th data block. The reorganization method for K data and V data is the same as that for Q data.

[0118] In this implementation, for scenarios requiring causal masking, the Q, K, and V data can be further segmented into finer-grained segments, and then the segmented (2) data can be processed in a symmetrical manner from beginning to end. x )*N Q data blocks, (2 x )*N K data blocks and (2 xN V data blocks are reorganized into 2N Q data blocks, 2N K data blocks, and 2N V data blocks. This symmetrical reorganization method ensures that the amount of data obscured by the mask in the score matrix of each Q data block and its corresponding K data block is closer (or even the same). This facilitates more balanced attention calculation processing across processing devices and improves the problem of uneven computing power.

[0119] Consistent with the scenario where no causal mask is required, after obtaining 2N Q data blocks, 2N K data blocks, and 2N V data blocks, the scenario requiring a causal mask also uses a symmetrical approach, distributing the 2N Q data blocks, 2N K data blocks, and 2N V data blocks to N processing devices respectively. Each processing device is allocated symmetrically 2 Q data blocks, 2 K data blocks, and 2 V data blocks. For example... Figure 8C As shown, for Q / K / V data, the 0th and (2N-1)th data blocks obtained from the reassembly are assigned to processing device 1, the 1st and (2N-2)th data blocks obtained from the reassembly are assigned to processing device 2, ..., and the (N-1)th and Nth data blocks obtained from the reassembly are assigned to processing device N.

[0120] In this implementation, the electronic device distributes the 2N Q-blocks, 2N K-blocks, and 2N V-blocks obtained from direct segmentation or recombination to N processing devices in a symmetrical manner. This ensures that each processing device performs two sets of operations based on the Q / K / V data blocks used during the initial attention calculation. Furthermore, this facilitates the transmission of two sets of K / V data blocks in a single communication between subsequent processing devices. This not only fully utilizes the transmission bandwidth but also reduces the computational workload per set, as each attention calculation on each processing device is performed in two groups, thus improving computational efficiency.

[0121] 703: Control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of the N processing devices; wherein, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation, and each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0122] In this embodiment, an electronic device controls N processing devices to perform the first attention calculation based on two pre-allocated Q data blocks, two K data blocks, and two V data blocks. Specifically, for the i-th processing device among the N processing devices, the two Q data blocks used during the j-th attention calculation are denoted as pre-allocated data block Q1 and data block Q2, and the two K data blocks are denoted as data block K... 1，j and data block K 2，j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N. When j=1, data block K 1，j Data block K 2，j Data block V 1，j and data block V 2，j It is pre-allocated. For example... Figure 9 As shown in (a), when j=1, attention calculation is performed on the two Q data blocks allocated to each processing device, along with the K data block and the V data block with the same index. For example, data block Q1 is the 0th Q data block, and data block K... 1，j For the 0th data block K, data block V 1，j For the 0th V data block, processing device 1 performs a set of attention calculations based on the 0th Q data block, the 0th K data block, and the 0th V data block; data block Q2 is the (2N-1)th Q data block, and data block K... 2，j For the (2N-1)th data block K, data block V 2，j For the (2N-1)th V data block, processing device 1 performs another set of attention calculations based on the (2N-1)th Q data block, the (2N-1)th K data block, and the (2N-1)th V data block.

[0123] When i = 1 and 1 < j ≤ 2N, data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 Data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 1，j-1 Data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ;

[0124] When i = N and 1 < j ≤ 2N, data block K 1，jThe data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 2，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ;

[0125] When 1 < i < N and 1 < j ≤ 2N, data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2， j -1 .

[0126] Specifically, such as Figure 9 As shown in (a), when the electronic device controls N processing devices to perform the first attention calculation based on 2 pre-allocated Q data blocks, 2 K data blocks, and 2 V data blocks, it also controls the N processing devices to perform the first point-to-point transmission of K / V data blocks. Then, for the second attention calculation of the i-th processing device, as shown in (a),... Figure 9 As shown in (b), if the i-th processing device is processing device 1, then data block K 2，2 For processing device 1, the 0th data block K is used during the first attention calculation process. 1，2 This is the first K-block of data used by processing device 2 during the first attention calculation process; data block V 2，2 This refers to the 0th data block V used by processing device 1 during the first attention calculation process. 1，2 This is the first V data block used by processing device 2 during the first attention calculation. If the i-th processing device is processing device N, then data block K... 1，2Let K be the Nth data block K used by processing device N during the first attention calculation process. 2，2 This refers to the (N+1)th data block K used by the processing device (N-1) during the first attention calculation; data block V 1，2 Let V be the Nth data block V used by processing device N during the first attention calculation process. 2，2 Let K be the (N+1)th V data block used by processing device (N-1) during the first attention calculation. If the i-th processing device is processing device 2, then data block K... 1，2 This is the first K data block used by processing device 3 during the first attention calculation process. 2，2 This refers to the (2N-1)th data block K used by processing device 1 during the first attention calculation process; data block V 1，2 This is the first V data block used by processing device 3 during the first attention calculation process. 2，2 This refers to the (2N-1)th V data block used by processing device 1 during the first attention calculation process. Processing device 1 performs one set of attention calculations in the second attention calculation process based on the 0th Q data block, the 1st K data block, and the 1st V data block, and performs another set of attention calculations in the 1st attention calculation process based on the (2N-1)th Q data block, the 0th K data block, and the 0th V data block. Processing device 2 performs one set of attention calculations in the second attention calculation process based on the first Q data block, the second K data block, and the second V data block, and performs another set of attention calculations in the second attention calculation process based on the (2N-2)th Q data block, the (2N-1)th K data block, and the (2N-1)th V data block; ...; Processing device N performs one set of attention calculations in the second attention calculation process based on the (N-1)th Q data block, the Nth K data block, and the Nth V data block, and performs another set of attention calculations in the second attention calculation process based on the Nth Q data block, the (N+1)th K data block, and the (N+1)th V data block.

[0127] In this implementation, for the i-th processing device among N processing devices, the Q data blocks, K data blocks, and V data blocks used in the first two sets of attention calculations are pre-allocated by the electronic device. When i = 1 or N, one set of K data blocks and V data blocks used in the second and third sets of attention calculations after the first set come from adjacent processing devices, and the other set of K data blocks and V data blocks comes from within the i-th processing device. When i ≠ 1 or N, the K data blocks and V data blocks used in the second and third sets of attention calculations after the first set come from adjacent data blocks. This ensures that each K data block and each V data block can perform an attention calculation with each Q data block, thereby guaranteeing the integrity of the attention calculation between Q data and K and V data.

[0128] Furthermore, when controlling the i-th processing device to perform the j-th attention calculation, the method also includes:

[0129] When i=1, control the i-th processing device to transmit data block K. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process;

[0130] When i = N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process;

[0131] When 1 < i < N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation; it controls the transmission of data block K by the i-th processing device. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process.

[0132] Specifically, such as Figure 9 As shown in (b), when controlling N processing devices to perform the second attention calculation process, control processing device 1 transmits the 0th K data block and V data block to processing device 2, ..., control processing device (N-1) transmits the (N+2)th K data block and V data block to processing device N, control processing device N transmits the Nth K data block and V data block to processing device (N-1), ..., control processing device 2 transmits the 2nd K data block and V data block to processing device 1, so that each processing device can perform the next attention calculation process based on the received K data block and V data block.

[0133] For example, when i=1, the data block K used in its j-th attention calculation process is... 2，j It can be obtained through internal transmission, such as the data block K used in the (j-1)th attention calculation process. 1，j-1 V 1，j-1 The thread in which it is executed is different from the thread used in the j-th attention calculation. When i=N, the data block K used in its j-th attention calculation is... 1，j It can be obtained through internal transmission, such as the data block K used in the (j-1)th attention calculation process. 2，j-1 V 2，j-1 The thread in question is different from the thread used in the j-th attention calculation. Of course, if the threads used in the two attention calculations are the same, then there is no internal data transfer between processing device 1 and processing device N.

[0134] In this implementation, when the electronic device controls N processing devices to perform the j-th attention calculation, it also controls the N processing devices to perform the j-th transmission of the K data block and V data block used in the j-th attention calculation, such as... Figure 9 As shown, the K data blocks and V data blocks of N processing devices rotate in a circular manner, forming an upward flow and a downward flow. This ensures that each processing device has two sets of K data blocks and V data blocks for each attention calculation, realizing two sets of calculations with one communication, which is beneficial for making full use of communication bandwidth and computing resources.

[0135] Based on the above description, the electronic device controls N processing devices to perform (2N-1) transmissions of K and V data blocks and 2N attention calculations until the (2N-1)th K data block, V data block, and 0th Q data block complete the attention calculation. Depending on the correction method, if correction is performed in each calculation, the attention calculation result of each processing device is two corrected attention calculations; if correction is performed after the calculation is completed, the attention calculation result of each processing device is an intermediate result of 2N attention calculations. The electronic device can then use the result of the j-th calculation to correct the result of the (j+1)-th calculation, obtaining two corrected attention calculations for each processing device. For example, the attention calculation results of the N processing devices are used to predict the image data, text data, or audio data to be processed. For instance, this includes classifying image data, predicting the next sentence of dialogue text data, and matching speakers in audio data.

[0136] For example, in scenarios where causal masking is not required, the two sets of attention calculations executed on each processing device each time can be parallelized, thereby improving computational efficiency. In scenarios where causal masking is required, the two sets of attention calculations executed on each processing device each time also need to consider the masking conditions corresponding to the score matrices of the K and Q data blocks used. That is, attention calculations using FlashAttention2 (an optimized self-attention mechanism) need to be performed based on the masks corresponding to the score matrices. The general steps include calculating the dot product of the Q and K data blocks to obtain the score matrix, applying a mask to the score, and combining the masked score matrix with the V data block to calculate the attention result.

[0137] For example, suppose the Q / K / V data is split into 16 Q data blocks, 16 K data blocks, and 16 V data blocks (labeled 0-15 respectively). The dot product of each Q data block and each K data block is represented by a rectangular block. Then, the preset causal mask of the score matrix of the 16 Q data blocks and the 16 K data blocks can be as follows: Figure 10 As shown (where the white area is the obscured part and the black area is the unobscured part).

[0138] based on Figure 10 The causal mask shown, when j=1, allows the electronic device to determine that the masks corresponding to the score matrices of the Q-block and K-block used in the two sets of attention calculations on the i-th processing device are both the first mask. The first mask is also a causal mask, and its shape is similar to... Figure 10The submask shown is the same as the causal mask. Taking processing device 1 as an example, its allocated 0th Q data block is recombined from the split 0th Q data block and the 15th Q data block, and the same is true for its allocated 0th K data block. Using (0,15) as the index of the allocated 0th Q data block and (0,15) as the index of the allocated 0th K data block, from... Figure 10 If a rectangular block is selected from the array shown, the corresponding block coordinates are: (0,0), (15,0), (0,15), (15,15). The final selected rectangular block is as follows. Figure 11 As shown, this is the sub-mask corresponding to the score matrix of the 0th Q data block and the 0th K data block allocated on processing device 1. Based on the same principle, the shape of the sub-mask corresponding to the score matrix of the (2N-1)th Q data block and the (2N-1)th K data block allocated on processing device 1 is similar to... Figure 11 The sub-masks shown are identical in shape, both being causal masks. This means that during the first attention calculation, the mask conditions corresponding to the two sets of attention calculations on each processing device are as follows: Figure 12 As shown, in this case, the two sets of attention calculations on the i-th processing device are performed based on the Q-block, K-block, V-block, and the first mask, respectively. That is, when calculating attention using the attention calculation formula, the score matrices of the Q-block and K-block need to be occluded using the value of the first mask.

[0139] In this implementation, when N processing devices are controlled to perform the first attention calculation in a scenario where a causal mask needs to be applied, since the masks corresponding to the two sets of attention calculations on each processing device are the first mask, there is no need to consider the calculation order of the two sets of attention calculations. The two sets of attention calculations can be completed based on the Q data block, K data block, V data block and the first mask used, respectively, making the calculation relatively flexible.

[0140] based on Figure 10The causal mask shown, when 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask for the score matrix of the Q-data block and K-data block used in this arbitrary attention calculation is either the second mask or the third mask, and the masks corresponding to the two attention calculations on the i-th processing device are different. The second mask is the mask that blocks the midpoint of the score matrix from the first row to the column, i.e., the upper half of the score matrix; the third mask is the mask that blocks the midpoint of the score matrix from the last column to the row, i.e., the right half of the score matrix. Taking processing device 1 as an example, its allocated 0th Q-data block is recombined from the split 0th Q-data block and the 15th Q-data block. At j=2, the reassembled first K data block and V data block are transmitted to processing device 1. This first K data block is composed of the split first K data block and the 14th K data block, with (0,15) as the index of the allocated 0th Q data block and (1,14) as the index of the transmitted first K data block. Figure 10 If a rectangular block is selected from the array shown, the corresponding block coordinates are: (0,1), (15,1), (0,14), (15,14). The final selected rectangular block is as follows. Figure 13 As shown in (a), this is the submask corresponding to the score matrix of the 0th Q data block and the 1st K data block. Similarly, the block coordinates corresponding to another set of attention calculations on processing device 1 can be determined as (7,0), (8,0), (7,15), (8,15), and the final extracted rectangle is shown below. Figure 13 As shown in (b), this is the sub-mask corresponding to the score matrix of the 0th K data block and the (2N-1)th Q data block. In other words, in the attention calculation processing after the first iteration, as the K / V data blocks rotate in a circular fashion, the mask situation corresponding to the two sets of attention calculations on each processing device is either... Figure 14 As shown in (a), either is Figure 14As shown in (b), in this case, the two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second and third masks, based on the Q data blocks, K data blocks, V data blocks, and corresponding masks used in each set of attention calculations. For example, when the execution order of the second mask takes precedence over the third mask, each processing device is controlled to first execute the set of attention calculations corresponding to the second mask, and then execute the other set of attention calculations corresponding to the third mask. Further, for example: on processing device 1, the left set of attention calculations corresponds to the second mask, and the right set of attention calculations corresponds to the third mask; on processing device 2, the left set of attention calculations corresponds to the third mask, and the right set of attention calculations corresponds to the second mask. If the execution order of the third mask takes precedence over the second mask, then the execution order on processing device 1 and processing device 2 is: processing device 1 executes the right set of attention calculations, and processing device 2 executes the left set of attention calculations; processing device 1 executes the left set of attention calculations, and processing device 2 executes the right set of attention calculations, thus unifying the mask calculations of the N processing devices.

[0141] In this implementation, when one set of attention calculations on the i-th processing device corresponds to the second mask and another set of attention calculations corresponds to the third mask, the electronic device needs to control N processing devices to calculate attention using the same mask at the same time. That is, according to the preset execution order of the second and third masks, the attention calculations corresponding to either the second or third mask are executed uniformly first, and then the attention calculations corresponding to the other mask are executed uniformly. By uniformly using the same mask to calculate attention, the consistency of attention calculation can be ensured, avoiding situations where using inconsistent masks may cause some processing devices to see information that should not be seen during calculation, resulting in the disruption of causal relationships and low usability of calculation results.

[0142] For example, if the mask corresponding to any set of attention calculations is the second mask, the electronic device can control the i-th processing device to discard invalid data. Specifically, as follows: Figure 15As shown, since the second mask obscures the upper half of the scoring matrix, corresponding to the first half of the Q data block, taking the Q data block as an example obtained by recombining the 0th and 15th parts of the split along the sequence dimension, the first half of the Q data block is the 0th part of the split Q data block. This part is discarded, and the 15th part of the split Q data block is retained as the sub-Q data block. That is, the sub-Q data block is the sub-data block from the midpoint of the row direction of the Q data block to the last column of the Q data block. After discarding invalid data, this arbitrary set of attention calculations is performed based on the sub-Q data block, K data block, and V data block. That is, the scoring matrix is ​​calculated using the sub-Q data block and K data block, and then the attention is calculated in combination with the V data block. There is no need to consider the value of the second mask, and the maskless FlashAttention2 attention calculation is performed. When the mask corresponding to any set of attention calculations is the third mask, the electronic device can also control the i-th processing device to discard invalid data. Specifically, as Figure 16 As shown, since the third mask obscures the right half of the scoring matrix, corresponding to the latter half of the K / V data block, taking the K / V data block as an example where the 0th and 15th parts of the split data are recombined along the sequence dimension, the latter half of the K data block is the 15th split K data block. This part is discarded, and the 0th split K data block is retained as the sub-K data block. That is, the sub-K data block is the sub-data block from the first column of the K data block to the midpoint of the row direction of the K data block. Similarly, the latter half of the V data block is the 15th split V data block. This part is discarded, and the 0th split V data block is retained as the sub-V data block. That is, the sub-V data block is the sub-data block from the first column of the V data block to the midpoint of the row direction of the V data block. After discarding invalid data, the attention calculation for any set of data is performed based on the Q data block, the sub-K data block, and the sub-V data block. That is, the score matrix is ​​calculated using the Q data block and the sub-K data block, and then the attention is calculated in combination with the sub-V data block. There is no need to consider the value of the third mask. The maskless FlashAttention2 attention calculation is performed.

[0143] In this implementation, for a set of attention calculations on the i-th processing device where the sub-mask is the second mask, based on the occlusion of the second mask, the electronic device can control the processing device to discard invalid data in the used Q data blocks to obtain sub-Q data blocks, and then perform attention calculations based on the sub-Q data blocks, K data blocks, and V data blocks. Similarly, for another set of attention calculations on the i-th processing device where the sub-mask is the third mask, based on the occlusion of the third mask, the electronic device can control the processing device to discard invalid data in the used K data blocks and V data blocks to obtain sub-K data blocks and sub-V data blocks, and then perform attention calculations based on the Q data blocks, sub-K data blocks, and sub-V data blocks. Discarding invalid data reduces unnecessary computational operations, lowers the amount of data computation, and helps to further improve computational efficiency.

[0144] For example, before splitting the Q data, K data, and V data respectively based on the number of processing devices N, the method further includes:

[0145] Receive instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation process.

[0146] In this embodiment, the user can also input the number of parallel attention sequences on the front-end device. For example, when the input instruction is 2, it means that the electronic device needs to enable the dual-stream parallel attention sequence function, that is, control each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively in each attention calculation process. For example, the instruction can also be the user's trigger information for the button to enable the dual-stream parallel attention sequence function. When the user clicks the button to enable the dual-stream parallel attention sequence function, it instructs the electronic device to control each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively in each attention calculation process.

[0147] In this implementation, the electronic device can enable the dual-stream parallel function of the attention sequence based on the user's input instructions, so that each processing device can perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation.

[0148] As can be seen from the embodiments of this application, the electronic device can segment the Q data, K data, and V data based on the number N of parallel processing devices, and finally obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks through direct segmentation or segmentation and recombination. Then, it controls N processing devices to perform 2N attention calculations based on the 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of N processing devices. Each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation. This forms two computational streams on the N processing devices. The dual-stream computation mode makes the computational scale of a single set of attention calculations on any processing device smaller, thus improving the efficiency of attention calculation.

[0149] Please see Figure 17 , Figure 17 A flowchart illustrating another data processing method provided in this application embodiment is shown below. Figure 17 As shown, the method includes steps 1701-1705:

[0150] 1701: Perform feature extraction on the user input data to obtain query Q data, key K data, and value V data;

[0151] 1702: Based on the number of processing devices N, divide the Q data, K data, and V data into (2... x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1;

[0152] 1703: In a symmetrical manner, respectively (2 x )*N Q data blocks, (2 x )*N K data blocks and (2 x Reorganize N V data blocks to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks;

[0153] Among them, each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

[0154] 1704: Allocate 2N Q data blocks, 2N K data blocks, and 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks symmetrically.

[0155] 1705: Control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of the N processing devices; wherein, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation, and each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0156] in, Figure 17 The method flow shown is applicable to scenarios requiring the application of causal masks. In step 1705, the corresponding mask situation needs to be considered each time attention calculation is performed. The specific implementation method is detailed in [link to implementation details]. Figure 7 The embodiments shown have been described in detail and can achieve the same or similar beneficial effects.

[0157] Please see Figure 18 , Figure 18 A flowchart illustrating another data processing method provided in this application embodiment is shown below. Figure 18 As shown, the method includes steps 1801-1804:

[0158] 1801: Perform feature extraction on the user input data to obtain query Q data, key K data, and value V data;

[0159] 1802: Based on the number of processing devices N, divide the Q data, K data, and V data into 2N Q data blocks, 2N K data blocks, and 2N V data blocks respectively;

[0160] 1803: Allocate 2N Q data blocks, 2N K data blocks, and 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks, and 2 V data blocks symmetrically.

[0161] 1804: Control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks, and 2N V data blocks to obtain the attention calculation results of N processing devices; wherein, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks in each attention calculation, and each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0162] in, Figure 18The illustrated method is suitable for scenarios requiring the application of causal masks. In step 1804, the corresponding mask situation does not need to be considered when performing attention calculations each time. The two sets of attention calculations on each processing device can be executed in parallel. The specific implementation method is detailed in [link to implementation details]. Figure 7 The embodiments shown have been described in detail and can achieve the same or similar beneficial effects.

[0163] Please see Figure 19 , Figure 19 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. This apparatus can be applied to electronic devices or chips within electronic devices. Figure 19 As shown, the device may include a feature extraction unit 1901, a data segmentation unit 1902, and a control unit 1903. Wherein:

[0164] The feature extraction unit 1901 is used to extract features from the user input data to obtain query Q data, key K data, and value V data.

[0165] The data segmentation unit 1902 is used to segment the query Q data, key K data, and value V data based on the number N of processing devices, respectively, to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks;

[0166] The control unit 1903 is used to control N processing devices to perform 2N attention calculations based on 2N Q data blocks, 2N K data blocks and 2N V data blocks to obtain the attention calculation results of the N processing devices;

[0167] In each attention calculation, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively. Each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

[0168] In one possible implementation, the two Q data blocks used by the i-th processing device among the N processing devices when performing the j-th attention calculation are pre-allocated data blocks Q1 and Q2, and the two K data blocks are data blocks K. 1，j and data block K 2，j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N;

[0169] Where, when j=1, data block K 1，j Data block K 2，j Data block V 1，j and data block V 2，j It is pre-allocated;

[0170] When i = 1 and 1 < j ≤ 2N, data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 Data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 1，j-1 Data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ;

[0171] When i = N and 1 < j ≤ 2N, data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j V is the data block used by the i-th processing device when performing the (j-1)-th attention calculation. 2，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ;

[0172] When 1 < i < N and 1 < j ≤ 2N, data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 Data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 Data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2， j -1 .

[0173] In one possible implementation, the device further includes a data allocation unit; the data allocation unit is used to allocate 2N Q data blocks, 2N K data blocks and 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks and 2 V data blocks symmetrically.

[0174] In one possible implementation, when j=1, the preset masks for the score matrices of the Q-block and K-block used for the two sets of attention calculations on the i-th processing device are both the first mask; the first mask is a causal mask.

[0175] The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

[0176] In one possible implementation, when 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in any one attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different.

[0177] The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second and third masks, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

[0178] In one possible implementation, the second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix.

[0179] When the mask corresponding to any set of attention calculations is the second mask, any set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; the sub-Q data block is the sub-data block from the midpoint of the row direction of the Q data block to the last column of the Q data block;

[0180] When the mask corresponding to any set of attention calculations is the third mask, any set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is the sub-data block from the first column of the K data block to the midline of the row direction of the K data block; the sub-V data block is the sub-data block from the first column of the V data block to the midline of the row direction of the V data block.

[0181] In one possible implementation, the data segmentation unit 1902 is specifically used to: segment the query Q data, key K data, and value V data respectively to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks.

[0182] The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1;

[0183] Using a symmetrical approach, respectively (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x Recombining N V data blocks yields 2N Q data blocks, 2N K data blocks, and 2N V data blocks; where each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

[0184] In one possible implementation, when controlling the i-th processing device to perform the j-th attention calculation process, the control unit 1903 is further configured to:

[0185] When i=1, control the i-th processing device to transmit data block K. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process;

[0186] When i = N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process;

[0187] When 1 < i < N, control the i-th processing device to transmit data block K. 1，j and data block V 1，j To the (i-1)th processing device, to transfer data block K 1，j and data block V 1，jThis serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation; it controls the transmission of data block K by the i-th processing device. 2，j and data block V 2，j To the (i+1)th processing device, to transfer data block K 2，j and data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention computation process.

[0188] In one possible implementation, the control unit 1903 is further configured to: receive instruction information input by the user; the instruction information is configured to enable the function of each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks and 2 V data blocks respectively during each attention calculation process.

[0189] It should be noted that, Figure 19 The implementation of each unit described can also be referenced accordingly. Figures 7 to 18 The corresponding description of the illustrated embodiments. Furthermore, Figure 19 The beneficial effects of the described device can be referred to Figures 7 to 18 The corresponding descriptions of the embodiments shown will not be repeated here.

[0190] Based on the description of the above method and device embodiments, this application also provides a computing device 2000. For example... Figure 20 As shown, the computing device 2000 includes: a bus 2001, a processor 2002, a memory 2003, and a communication interface 2004. The processor 2002, memory 2003, and communication interface 2004 communicate with each other via the bus 2001. The computing device 2000 can be a server or a terminal device; for example, the computing device 2000 can be... Figures 7 to 18 The electronic device in the illustrated embodiment. It should be understood that this application does not limit the number of processors and memories in the computing device 2000.

[0191] The bus 2001 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 20 The bus 2001 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 2001 may include a path for transmitting information between various components of the computing device 2000 (e.g., memory 2003, processor 2002, communication interface 2004).

[0192] Processor 2002 may include any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device.

[0193] Memory 2003 may include volatile memory, such as random access memory (RAM). Processor 2002 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DR RAM).

[0194] The memory 2003 stores executable program code, and the processor 2002 executes this executable program code to implement the functions of the aforementioned feature extraction unit, data segmentation unit, and control unit, thereby achieving... Figures 7 to 18 The method of the illustrated embodiment. That is, the memory 2003 stores information for performing... Figures 7 to 18 Instructions for the method of the illustrated embodiment.

[0195] Alternatively, the memory 2003 stores executable code, and the processor 2002 executes the executable code to implement the functions of the aforementioned data processing device, thereby achieving... Figures 7 to 18 The method of the illustrated embodiment. That is, the memory 2003 stores information for performing... Figures 7 to 18 Instructions for the method of the illustrated embodiment.

[0196] The communication interface 2004 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 2000 and other devices or communication networks.

[0197] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone. For example, the computing device can be one of the aforementioned electronic devices.

[0198] like Figure 21 As shown, the computing device cluster includes at least one computing device 2000. The memory 2003 of one or more computing devices 2000 in the computing device cluster may store the same memory for executing... Figures 7 to 18 Instructions for the method of the illustrated embodiment.

[0199] In some possible implementations, the memory 2003 of one or more computing devices 2000 in the computing device cluster may also store memory for execution. Figures 7 to 18 Partial instructions of the method in the illustrated embodiment. In other words, a combination of one or more computing devices 2000 can jointly execute instructions for performing... Figures 7 to 18 Instructions for the method of the illustrated embodiment.

[0200] It should be noted that the memory 2003 in different computing devices 2000 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 2003 of different computing devices 2000 can implement the functions of one or more unit modules in the feature extraction unit, data segmentation unit, and control unit.

[0201] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 22 One possible implementation is shown. For example... Figure 22As shown, computing devices 2000A and 2000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 2003 in computing device 2000A stores instructions for performing the functions of the feature extraction unit. Meanwhile, the memory 2003 in computing device 2000B stores instructions for performing the functions of the data segmentation unit and the control unit.

[0202] Figure 22 The connection method between the computing device clusters shown can be such that, considering the data processing method provided in this application requires feature extraction, data segmentation and data processing, the functions implemented by the data segmentation unit and the control unit are performed by the computing device 2000B, while the functions implemented by the feature extraction unit are performed by the computing device 2000A.

[0203] It should be understood that Figure 22 The functions of computing device 2000A shown can also be performed by multiple computing devices 2000. Similarly, the functions of computing device 2000B can also be performed by multiple computing devices 2000.

[0204] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 21 and Figure 22 The connection method of the computing device cluster is different in that the memory 2003 of one or more computing devices 2000 in the computing device cluster can store the same instructions for executing the data processing method provided in this application.

[0205] In some possible implementations, the memory 2003 of one or more computing devices 2000 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 2000 can jointly execute instructions for executing data processing methods.

[0206] It should be noted that, although Figure 20 The illustrated computing device only shows the bus 2001, processor 2002, memory 2003, and communication interface 2004. However, in specific implementations, those skilled in the art should understand that the computing device also includes other components necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device may also include hardware components for implementing other additional functions. Moreover, those skilled in the art should understand that the computing device may only include the components necessary for implementing the embodiments of this application, and may not necessarily include... Figure 20 All the devices shown.

[0207] This application also provides a chip, including: a processor, configured to retrieve and run a computer program from memory, causing a device with the chip installed to perform the above-described actions. Figures 7 to 18 The method described in any of the embodiments. The chip may be a chip in a computing device.

[0208] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform... Figures 7 to 18 The method described in any one of the embodiments.

[0209] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute... Figures 7 to 18 The method described in any one of the embodiments.

[0210] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0211] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0212] In the several embodiments provided in this application, it should be understood that the disclosed computing device clusters, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely exemplary. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0213] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0214] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0215] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0216] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0217] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0218] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: Feature extraction is performed on the user input data to obtain query Q data, key K data, and value V data; Based on the number of processing devices N, the Q data, K data, and V data are divided into 2N Q data blocks, 2N K data blocks, and 2N V data blocks respectively; Control N processing devices to perform 2N attention calculations based on the 2N Q data blocks, the 2N K data blocks, and the 2N V data blocks to obtain the attention calculation results of the N processing devices; In each attention calculation process, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks, respectively. Each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

2. The method according to claim 1, characterized in that, When the i-th processing device among the N processing devices performs the j-th attention calculation, the two Q data blocks used are pre-allocated data block Q1 and data block Q2, and the two K data blocks are data block K. 1，j and data block K 2，j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N; Wherein, when j=1, the data block K 1，j The data block K 2，j The data block V 1，j and the data block V 2，j It is pre-allocated; When i = 1 and 1 < j ≤ 2N, the data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 The data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block V 2，j The data block V used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 The data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ; When i = N and 1 < j ≤ 2N, the data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 The data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 The data block V 1，j The data block V used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 The data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ; When 1 < i < N and 1 < j ≤ 2N, the data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 The data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 .

3. The method according to claim 1 or 2, characterized in that, After obtaining 2N Q data blocks, 2N K data blocks, and 2N V data blocks, the method further includes: The 2N Q data blocks, 2N K data blocks, and 2N V data blocks are allocated to N processing devices in a symmetrical manner, with each processing device receiving 2 Q data blocks, 2 K data blocks, and 2 V data blocks.

4. The method according to claim 2, characterized in that, When j=1, the preset mask for the score matrix of the Q data block and K data block used for the two sets of attention calculations on the i-th processing device is the first mask; The first mask is a causal mask; The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

5. The method according to claim 2, characterized in that, When 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in the attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different; The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second mask and the third mask, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

6. The method according to claim 5, characterized in that, The second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix. When the mask corresponding to any set of attention calculations is the second mask, the set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; The sub-Q data block is a sub-data block extending from the midpoint of the row direction of the Q data block to the last column of the Q data block; When the mask corresponding to any set of attention calculations is the third mask, the set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is a sub-data block from the first column of the K data block to the midline in the row direction of the K data block; the sub-V data block is a sub-data block from the first column of the V data block to the midline in the row direction of the V data block.

7. The method according to any one of claims 4-6, characterized in that, The step of splitting the query Q data, key K data, and value V data into 2N Q data blocks, 2N K data blocks, and 2N V data blocks includes: The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1; In a symmetrical manner, respectively, the (2) x )*N Q data blocks, the (2 x )*N K data blocks and the (2 x The 2N V data blocks are reassembled to obtain the 2N Q data blocks, the 2N K data blocks, and the 2N V data blocks; wherein, each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

8. The method according to any one of claims 2-7, characterized in that, When controlling the i-th processing device to perform the j-th attention calculation process, the method further includes: When i=1, control the i-th processing device to transmit the data block K. 2，j and the data block V 2，j To the (i+1)th processing device, to process the data block K 2，j and the data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention calculation process; When i = N, control the i-th processing device to transmit the data block K. 1，j and the data block V 1，j To the (i-1)th processing device, to process the data block K 1，j and the data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process; When 1 < i < N, control the i-th processing device to transmit the data block K. 1，j and the data block V 1，j To the (i-1)th processing device, to process the data block K 1，j and the data block V 1，j As a set of K data blocks and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process; controlling the i-th processing device to transmit the data block K 2，j and the data block V 2，j To the (i+1)th processing device, to process the data block K 2，j and the data block V 2，j This is a set of K data blocks and V data blocks used by the (i+1)th processing device during the (j+1)th attention calculation process.

9. The method according to any one of claims 1-8, characterized in that, Before segmenting the Q data, K data, and V data based on the number of processing devices N, the method further includes: The system receives instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively during each attention calculation process.

10. A data processing apparatus, characterized in that, The device includes a feature extraction unit, a data segmentation unit, and a control unit; wherein: The feature extraction unit is used to extract features from the user-input data to obtain query Q data, key K data, and value V data; The data segmentation unit is used to segment the Q data, the K data, and the V data based on the number N of processing devices, respectively, to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks; The control unit is configured to control N processing devices to perform 2N attention calculations based on the 2N Q data blocks, the 2N K data blocks, and the 2N V data blocks, so as to obtain the attention calculation results of the N processing devices; In each attention calculation process, each processing device performs two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks, respectively. Each set of attention calculations is performed based on 1 Q data block, 1 K data block, and 1 V data block.

11. The apparatus according to claim 10, characterized in that, When the i-th processing device among the N processing devices performs the j-th attention calculation, the two Q data blocks used are pre-allocated data block Q1 and data block Q2, and the two K data blocks are data block K. 1，j and data block K 2，j The two V data blocks are data blocks V. 1，j and data block V 2，j ; 1≤i≤N, 1≤j≤2N; Wherein, when j=1, the data block K 1，j The data block K 2，j The data block V 1，j and the data block V 2，j It is pre-allocated; When i = 1 and 1 < j ≤ 2N, the data block K 2，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 The data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block V 2，j The data block V used by the i-th processing device during the (j-1)-th attention calculation process. 1，j-1 The data block V 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 ; When i = N and 1 < j ≤ 2N, the data block K 1，j The data block K used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 The data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 The data block V 1，j The data block V used by the i-th processing device during the (j-1)-th attention calculation process. 2，j-1 The data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 ; When 1 < i < N and 1 < j ≤ 2N, the data block K 1，j The data block K used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block K 2，j The data block K used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 The data block V 1，j The data block V used by the (i+1)th processing device during the (j-1)th attention calculation process. 1，j-1 The data block V 2，j The data block V used by the (i-1)th processing device during the (j-1)th attention calculation process. 2，j-1 .

12. The apparatus according to claim 10 or 11, characterized in that, The device further includes a data allocation unit; the data allocation unit is used to: allocate the 2N Q data blocks, the 2N K data blocks and the 2N V data blocks to N processing devices in a symmetrical manner; wherein each processing device is allocated 2 Q data blocks, 2 K data blocks and 2 V data blocks symmetrically.

13. The apparatus according to claim 11, characterized in that, When j=1, the preset mask for the score matrix of the Q data block and K data block used for the two sets of attention calculations on the i-th processing device is the first mask; The first mask is a causal mask; The two sets of attention calculations on the i-th processing device are performed based on the Q data block, K data block, V data block and the first mask used, respectively.

14. The apparatus according to claim 11, characterized in that, When 1 < j ≤ 2N, for any one of the two attention calculations on the i-th processing device, the preset mask of the score matrix of the Q data block and K data block used in the attention calculation is the second mask or the third mask; wherein, the masks corresponding to the two attention calculations on the i-th processing device are different; The two sets of attention calculations on the i-th processing device are executed sequentially according to the preset execution order of the second mask and the third mask, based on the Q data block, K data block, V data block and the corresponding mask used in each set of attention calculations.

15. The apparatus according to claim 14, characterized in that, The second mask is a mask that blocks the midpoint between the first row and the column of the scoring matrix; the third mask is a mask that blocks the midpoint between the last column and the row of the scoring matrix. When the mask corresponding to any set of attention calculations is the second mask, the set of attention calculations is performed based on the sub-Q data blocks, K data blocks, and V data blocks of the Q data block used; The sub-Q data block is a sub-data block extending from the midpoint of the row direction of the Q data block to the last column of the Q data block; When the mask corresponding to any set of attention calculations is the third mask, the set of attention calculations is performed based on the Q data block, the sub-K data block of the K data block, and the sub-V data block of the V data block used; the sub-K data block is a sub-data block from the first column of the K data block to the midline in the row direction of the K data block; the sub-V data block is a sub-data block from the first column of the V data block to the midline in the row direction of the V data block.

16. The apparatus according to any one of claims 13-15, characterized in that, In terms of splitting the query Q data, key K data, and value V data respectively to obtain 2N Q data blocks, 2N K data blocks, and 2N V data blocks, the data splitting unit is specifically used for: The Q data, K data, and V data are respectively divided into (2) x )*N Q data blocks, (2 x )*N K data blocks and (2 x N data blocks of value V; where x is an integer greater than 1; In a symmetrical manner, respectively, the (2) x )*N Q data blocks, the (2 x )*N K data blocks and the (2 x The 2N V data blocks are reassembled to obtain the 2N Q data blocks, the 2N K data blocks, and the 2N V data blocks; wherein, each symmetrical 2 x-1 2 Q data blocks are reorganized into one Q data block, with each symmetrical 2 x-1 K data blocks are reassembled into one K data block, and each symmetrical 2 x-1 Each V data block is reorganized into a single V data block.

17. The apparatus according to any one of claims 11-16, characterized in that, When controlling the i-th processing device to perform the j-th attention calculation process, the control unit is further configured to: When i=1, control the i-th processing device to transmit the data block K. 2，j and the data block V 2，j To the (i+1)th processing device, to process the data block K 2，j and the data block V 2，j This serves as a set of K and V data blocks used by the (i+1)th processing device during the (j+1)th attention calculation process; When i = N, control the i-th processing device to transmit the data block K. 1，j and the data block V 1，j To the (i-1)th processing device, to process the data block K 1，j and the data block V 1，j This serves as a set of K and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process; When 1 < i < N, control the i-th processing device to transmit the data block K. 1，j and the data block V 1，j To the (i-1)th processing device, to process the data block K 1，j and the data block V 1，j As a set of K data blocks and V data blocks used by the (i-1)th processing device during the (j+1)th attention calculation process; controlling the i-th processing device to transmit the data block K 2，j and the data block V 2，j To the (i+1)th processing device, to process the data block K 2，j and the data block V 2，j This is a set of K data blocks and V data blocks used by the (i+1)th processing device during the (j+1)th attention calculation process.

18. The apparatus according to any one of claims 10-17, characterized in that, The control unit is also used for: The system receives instruction information input by the user; the instruction information is used to instruct each processing device to perform two sets of attention calculations based on 2 Q data blocks, 2 K data blocks, and 2 V data blocks respectively during each attention calculation process.

19. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-9.

20. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1-9.

21. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-9.

Images