Parallel processing method and apparatus based on sequence model

Abstract

Embodiments of the present specification provide a parallel processing method and device based on a sequence model. The method is performed by an i-th device of any of n devices each deploying a sequence model, and includes: obtaining an i-th sub-sequence in n parts obtained by performing first division on an input sequence; mapping the i-th sub-sequence into a query sequence, a key sequence and a value sequence respectively based on self-attention parameters in the sequence model; performing second division on the query sequence to obtain m micro-sequences; performing m rounds of processing together with other n-1 devices, and any j-th round of processing includes: sending the j-th micro-sequence as a target sequence to the n-1 devices respectively; performing distributed calculation together with the key sequence and the value sequence of the target sequence, the key sequence and the value sequence of each of the n-1 devices and the target sequence based on the target sequence, the key sequence and the value sequence to obtain global attention encoding; and determining an attention encoding sequence corresponding to the i-th sub-sequence based on the m global attention encodings to determine a prediction result of the input sequence.

Description

Technical Field

[0001] This specification relates to the field of distributed computing technology, and in particular to a parallel processing method and apparatus based on a sequence model, a computer-readable storage medium, and a computing device. Background Technology

[0002] Long sequence modeling is crucial for question answering, document classification, and other natural language processing (NLP) tasks involving long texts, as well as for processing high-resolution images and series of video frames in the image modality. The Transformer, an attention-based model originally proposed for NLP, has shown great potential in computer vision (CV) and multimodal applications. Because the Transformer can learn global context-aware sequence representations across many modalities, Transformer long sequence modeling has become an emerging trend.

[0003] However, as the sequence expands, memory consumption increases rapidly due to the quadratic complexity of self-attention in the Transformer, inevitably exceeding the limitations of a single device, such as a Graphics Processing Unit (GPU). Therefore, utilizing long-sequence models to process long-sequence data in parallel has become an emerging trend. Summary of the Invention

[0004] This specification describes a parallel processing method and apparatus based on sequence models, which can reduce memory consumption when using sequence models to process long sequence data in parallel, improve computational efficiency, and efficiently and conveniently adapt to the expansion of long sequences.

[0005] According to a first aspect, a parallel processing method based on a sequence model is provided, wherein the sequence model is simultaneously deployed on n devices, and the method is executed by any i-th device, comprising: obtaining the i-th sub-sequence from the n parts obtained by first partitioning the input sequence; mapping the i-th sub-sequence to a local query sequence, a local key sequence, and a local value sequence based on the self-attention parameter in the sequence model, and performing a second partition on the local query sequence to obtain m micro-sequences; jointly performing m rounds of processing with the other n-1 devices, wherein any j-th round of processing comprises: determining the j-th local micro-sequence as the target sequence and sending it to the n-1 devices respectively; jointly performing distributed computation with the key sequence, value sequence, and target sequence of each of the n-1 devices based on the target sequence, the local key sequence, and the local value sequence to obtain a global attention code corresponding to the target sequence; and determining an attention code sequence corresponding to the i-th sub-sequence based on the m global attention codes corresponding to the m micro-sequences, for determining the prediction result of the input sequence.

[0006] In one embodiment, based on the target sequence, local key sequence, and local value sequence, distributed computation is performed together with the key sequences, value sequences, and target sequence of each of the n-1 devices to obtain a global attention code corresponding to the target sequence. This includes: determining a local attention score matrix based on the target sequence and the local key sequence; performing distributed normalization processing on the local attention score matrix and the n-1 attention score matrices of the n-1 devices to obtain a local attention weight matrix; processing the local value sequence using the local attention weight matrix to obtain a local attention code; and obtaining the global attention code, which is the sum of the local attention code and the n-1 local attention codes sent by the n-1 devices.

[0007] In one specific embodiment, based on the local attention score matrix, a distributed normalization process is performed with the n-1 attention score matrices from the n-1 devices to obtain a local attention weight matrix. This includes: performing power operations with the natural constant as the base and each element in the local attention score matrix as the exponent to obtain a local power operation matrix; obtaining a power operation sum matrix, which is the sum of the local power operation matrix and the n-1 power operation matrices sent by the other n-1 devices; and calculating the local attention weight matrix based on the local power operation matrix and the power operation sum matrix.

[0008] In one embodiment, the method further includes: jointly executing other m*(n-1) rounds of processing with the other n-1 devices, wherein any round of processing includes: receiving the target sequence of the current round from a designated device among the n-1 devices, and performing distributed computation jointly with the key sequences, value sequences and the target sequence of the current round of each of the n-1 devices based on the target sequence of the current round, the local key sequence and the local value sequence, so that the designated device obtains a global attention encoding corresponding to the target sequence of the current round.

[0009] In one embodiment, determining the j-th local micro-sequence as the target sequence and sending it to the n-1 devices includes: based on the target sequence, performing full aggregation communication with the n-1 devices to obtain a spliced ​​sequence, which is formed by splicing the target sequence and the target sequences determined by each of the n-1 devices in this round of processing; wherein, based on the target sequence, the local key sequence, and the local value sequence, performing distributed computation with the key sequences, value sequences, and the target sequence of each of the n-1 devices to obtain a global attention code corresponding to the target sequence includes: based on the spliced ​​sequence, the local key sequence, and the local value sequence, performing distributed computation with the key sequences, value sequences, and the spliced ​​sequence of each of the n-1 devices, so that each of the n devices obtains the global attention code corresponding to its target sequence determined in this round.

[0010] In one specific embodiment, based on the concatenated sequence, local key sequence, and local value sequence, distributed computation is performed together with the key sequences, value sequences, and concatenated sequences of the n-1 devices, so that each of the n devices obtains the global attention code corresponding to the target sequence determined in this round. This includes: determining a local attention score matrix based on the concatenated sequence and the local key sequence; performing distributed normalization processing on the local attention score matrix and the attention score matrices determined by each of the n-1 devices to obtain a local attention weight matrix; processing the local value sequence using the local attention weight matrix to obtain a local attention code, which includes n parts of local attention codes corresponding to the n target sequences in the concatenated sequence; and performing reduction distribution together with the local attention codes determined by each of the n-1 devices based on the local attention codes, so that each of the n devices obtains the global attention code corresponding to the target sequence determined in this round.

[0011] In one embodiment, after determining the attention encoding sequence corresponding to the i-th sub-sequence, the method further includes: calculating a training gradient for updating the self-attention parameters based on the attention encoding sequence and the backpropagation gradient, wherein the backpropagation gradient is determined based on the prediction result and the corresponding sequence label.

[0012] In one embodiment, each sequence element in the input sequence is: the word embedding vector of each word segment in the text, the pixel vector of each region in the image, or the acoustic feature vector of each time interval in the audio.

[0013] According to the second aspect, a parallel processing method based on a sequence model is provided, wherein the sequence model is simultaneously deployed on n devices, and the method is executed by any i-th device, comprising: obtaining the i-th sub-sequence from the n parts obtained by first partitioning the input sequence; mapping the i-th sub-sequence to a local query sequence, a local key sequence, and a local value sequence based on the self-attention parameters in the sequence model, and performing a second partition on the local key sequence to obtain m micro-sequences; jointly performing m rounds of processing with the other n-1 devices, wherein any j-th round of processing comprises: determining the j-th local micro-sequence as the target sequence and sending it to the n-1 devices respectively; jointly performing distributed computation with the query sequences, value sequences, and target sequences of the n-1 devices based on the target sequence, the local query sequence, and the local value sequence to obtain a global attention code corresponding to the target sequence; and determining an attention code sequence corresponding to the i-th sub-sequence based on the m global attention codes corresponding to the m micro-sequences, for determining the prediction result of the input sequence.

[0014] In one embodiment, the method further includes: jointly executing other m*(n-1) rounds of processing with the other n-1 devices, wherein any round of processing includes: receiving the target sequence of the current round from a designated device among the n-1 devices, and performing distributed computation jointly with the respective query sequences, value sequences, and target sequences of the n-1 devices based on the target micro-sequence of the current round, the local query sequence, and the local value sequence, so that the designated device obtains a global attention encoding corresponding to the target sequence of the current round.

[0015] In one embodiment, determining the j-th local micro-sequence as the target sequence and sending it to the n-1 devices includes: based on the target sequence, performing full aggregation communication with the n-1 devices to obtain a spliced ​​sequence, which is formed by splicing the target sequence and the target sequences determined by each of the n-1 devices in this round of processing; wherein, based on the target sequence, the local query sequence, and the local value sequence, performing distributed computation with the query sequences, value sequences, and the target sequence of each of the n-1 devices to obtain a global attention code corresponding to the target sequence includes: based on the spliced ​​sequence, the local query sequence, and the local value sequence, performing distributed computation with the query sequences, value sequences, and the spliced ​​sequence of each of the n-1 devices, so that each of the n devices obtains the global attention code corresponding to its target sequence determined in this round.

[0016] According to a third aspect, a parallel processing device based on a sequence model is provided, wherein the sequence model is simultaneously deployed in n devices, and the device is integrated into any i-th device therein, comprising: a sub-sequence acquisition module configured to acquire the i-th sub-sequence from n parts obtained by a first partitioning of an input sequence; a sub-sequence mapping module configured to map the i-th sub-sequence into a local query sequence, a local key sequence, and a local value sequence based on self-attention parameters in the sequence model; a micro-sequence acquisition module configured to perform a second partitioning of the local query sequence to obtain m micro-sequences; and a micro-sequence processing module configured to perform m rounds of processing together with the other n-1 devices. The micro-sequence processing module performs any j-th round of processing through the following units: a micro-sequence sending unit, configured to determine the j-th local micro-sequence as the target sequence and send it to the n-1 devices respectively; a distributed computing unit, configured to perform distributed computing based on the target sequence, the local key sequence, and the local value sequence, together with the key sequences, value sequences, and the target sequence of each of the n-1 devices, to obtain a global attention code corresponding to the target sequence; and an encoding construction module, configured to determine an attention encoding sequence corresponding to the i-th sub-sequence based on the m global attention codes corresponding to the m micro-sequences, for determining the prediction result of the input sequence.

[0017] According to the fourth aspect, a parallel processing device based on a sequence model is provided, wherein the sequence model is simultaneously deployed in n devices, and the device is integrated into any i-th device therein, comprising: a sub-sequence acquisition module configured to acquire the i-th sub-sequence from n parts obtained by a first partition of an input sequence; a sub-sequence mapping module configured to map the i-th sub-sequence into a local query sequence, a local key sequence, and a local value sequence based on the self-attention parameters in the sequence model; a micro-sequence acquisition module configured to perform a second partition of the local key sequence to obtain m micro-sequences; and a micro-sequence processing module configured to perform m rounds of processing together with the other n-1 devices. The micro-sequence processing module performs any j-th round of processing through the following units: a micro-sequence sending unit, configured to determine the j-th local micro-sequence as the target sequence and send it to the n-1 devices respectively; a distributed computing unit, configured to perform distributed computing based on the target sequence, the local query sequence, and the local value sequence, together with the query sequences, value sequences, and the target sequence of each of the n-1 devices, to obtain a global attention code corresponding to the target sequence; and an encoding construction module, configured to determine an attention encoding sequence corresponding to the i-th sub-sequence based on the m global attention codes corresponding to the m micro-sequences, for determining the prediction result of the input sequence.

[0018] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first or second aspect.

[0019] According to a sixth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of the first aspect or the second aspect.

[0020] The parallel processing method and apparatus based on sequence models provided in the embodiments of this specification can be used to segment the attention matrix from both row and column dimensions to achieve an attention matrix with L / mn complexity. This method consumes less memory, has high computational efficiency, and requires only linear device resources to expand the sequence length. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This diagram illustrates the architecture of a scheme for parallel processing of long sequence data based on a sequence model, as disclosed in the embodiments of this specification.

[0023] Figure 2 A flowchart illustrating a parallel processing method based on a sequence model according to one embodiment is shown.

[0024] Figure 3 This diagram illustrates a parallel processing data flow based on full aggregation communication and protocol distribution, as disclosed in the embodiments of this specification.

[0025] Figure 4 This diagram illustrates broadcast communication in the parallel processing method disclosed in the embodiments of this specification.

[0026] Figure 5 This diagram illustrates the full aggregation communication in the parallel processing method disclosed in the embodiments of this specification.

[0027] Figure 6 This document shows a schematic flowchart illustrating the m-round processing using implementation method A as disclosed in the embodiments of this specification.

[0028] Figure 7 This diagram illustrates a process flow diagram of m rounds of processing using implementation method B as disclosed in the embodiments of this specification.

[0029] Figure 8A schematic diagram of protocol communication in the parallel processing method disclosed in the embodiments of this specification is shown;

[0030] Figure 9 This diagram illustrates full protocol communication in the parallel processing method disclosed in the embodiments of this specification.

[0031] Figure 10 This diagram illustrates the protocol distribution communication in the parallel processing method disclosed in the embodiments of this specification.

[0032] Figure 11 A flowchart illustrating a parallel processing method based on a sequence model according to another embodiment is shown;

[0033] Figure 12 A schematic diagram of the structure of a parallel processing apparatus based on a sequence model according to one embodiment is shown;

[0034] Figure 13 A schematic diagram of the structure of a parallel processing apparatus based on a sequence model according to another embodiment is shown. Detailed Implementation

[0035] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0036] As mentioned earlier, parallel processing of long data sequences based on the Transformer long sequence model has become a research hotspot. However, existing parallel processing methods have some shortcomings and cannot meet the higher requirements of practical applications. For example, some methods cannot handle the huge intermediate activations in the Transformer where self-attention is quadratic with the sequence length, and therefore do not support sequence expansion; another example is that some methods require quadratic device resources to expand the sequence length, while the communication efficiency between devices is limited by the weakest link.

[0037] Based on the above observations and analysis, this specification discloses a parallel processing scheme in its embodiments. Figure 1 This diagram illustrates the architecture of a scheme for parallel processing of long sequence data based on a sequence model, as disclosed in the embodiments of this specification. Figure 1 As shown, the data is partitioned in two stages based on the original long sequence data. Specifically, the first stage partitions the original sequence X, which is then distributed to n devices, and correspondingly, each device D... i Obtain the corresponding subsequence x i The second phase involves dividing the data into various devices (D). i The subsequence x is processed using the self-attention parameter in the sequence model. i The determined query sequence q i Query sequence q i It is divided into m parts, and any j-th part in the m parts is denoted as q. i_jFurthermore, each device D i Based on including q i_j Data, including those from other components, is used for inter-device communication to achieve distributed computing, allowing each device to obtain data specific to its subsequence x. i attention encoding c i Then, optional, Figure 1 The diagram illustrates the encoding of n attention points c. i The sequences are concatenated to obtain the attention-encoded sequence C corresponding to the original sequence X. The attention-encoded sequence C is then processed using the output layer of the sequence model to obtain the prediction result for the original sequence X. In this way, long sequence data can be processed efficiently with low communication costs and low memory overhead.

[0038] The specific implementation steps of the above parallel processing scheme are described below. Figure 2 This diagram illustrates a flowchart of a parallel processing method based on a sequence model disclosed in an embodiment of this specification, wherein the sequence model is deployed simultaneously on n devices, for example, Figure 1 The diagram shows n devices. For example, each of the n devices can be a GPU, a Tensor Processing Unit (TPU), or other types of computing or processing chips.

[0039] Figure 2 The method shown can be executed by any one of the n devices, hereinafter referred to as the i-th device. For example... Figure 2 As shown, the method includes the following steps:

[0040] Step S210: Obtain the i-th subsequence from the n parts obtained by the first partitioning of the input sequence; Step S220: Based on the self-attention parameters in the sequence model, map the i-th subsequence to a local query sequence, a local key sequence, and a local value sequence respectively; Step S230: Perform a second partitioning on the local query sequence to obtain m micro-sequences; Step S240: Perform m rounds of processing together with the other n-1 devices, where any j-th round of processing includes: Step S241: Determine the j-th local micro-sequence as the target sequence and send it to the n-1 devices respectively; Step S242: Based on the target sequence, the local key sequence, and the local value sequence, perform distributed computing together with the key sequences, value sequences, and the target sequence of the n-1 devices to obtain the global attention code corresponding to the target sequence; Step S250: Based on the m global attention codes corresponding to the m micro-sequences, determine the attention code sequence corresponding to the i-th subsequence, which is used to determine the prediction result of the input sequence.

[0041] The steps described above are explained in detail below:

[0042] Step S210: Obtain the i-th subsequence from the n parts obtained by first partitioning the input sequence.

[0043] The above input sequence is the original, complete sequence form input for the sequence model. The input sequence includes multiple sequence elements determined based on the business object sample. In one embodiment, the business object sample is text, and correspondingly, the sequence elements in the text input sequence can be word embedding vectors from word segmentation in the text. It is understood that word segmentation of the text and determination of word embedding vectors can be implemented using existing technologies, and will not be elaborated upon.

[0044] In another embodiment, the business object sample is an image, and correspondingly, the sequence elements in the image input sequence can be pixel vectors of local regions of the image. It should be understood that the method of dividing regions in an image is very flexible; for example, it can be divided by a single row or column, or equally divided by rectangular regions of other sizes. For example, assuming an image is 600*800 pixels in size, it can be divided into 10 regions of 60*80 pixels each. Further, based on each divided region, a corresponding pixel vector is constructed using the pixel values ​​of all pixels in that region.

[0045] In another embodiment, the business object sample is audio. Accordingly, the sequence elements in the audio input sequence can be acoustic feature vectors divided into time intervals of a preset length in the audio. It should be understood that the determination of acoustic feature vectors can be achieved using existing technologies, which will not be elaborated upon here.

[0046] In another embodiment, the business object sample is a sequence of user events. Accordingly, the sequence elements in the event input sequence can be feature vectors of user events. For example, a user event can be a user's network behavior or network operation, such as logging in, clicking, placing an order, making a payment, or making a transaction. The feature vector can express user characteristics, such as the user's registration information on the platform, as well as event characteristics, such as the time of occurrence and the location of occurrence.

[0047] The above mainly introduces sample business objects and the input sequences determined based on them. It should be understood that the sample business objects disclosed in the embodiments of this specification are not limited to the text, images, audio, and user event sequences mentioned above; for example, they can also be videos, etc.

[0048] The determined input sequence is divided into n parts, where n is a natural number greater than 1, representing the number of devices actually participating in parallel processing. It should be understood that the division of the input sequence may be referred to as the "first division" in the text. The terms "first" and "second," etc., are used only for descriptive purposes and do not imply any ordering or other limitations. Furthermore, the first division may or may not be equal. For example, if the input sequence is set to equal division and the number of sequence elements (or sequence length) is an integer multiple of n, the input sequence can be directly divided into n equal parts. If the number of sequence elements is not an integer multiple of n, the input sequence can be padded (e.g., with zero values) before being divided equally.

[0049] The resulting n parts are divided into n input subsequences (or simply n subsequences). These n subsequences are then distributed to n devices. Accordingly, in this step, the i-th device can receive one of the n subsequences. Figure 1 and Figure 3 The diagram illustrates the input sequence X and the i-th subsequence x received by the i-th device. i Furthermore, the text uses r1 and r2 to represent the number of sequence elements in the input sequence and the number of sequence elements in the subsequence, respectively, and d to represent the dimension of the sequence elements. Therefore, or, The following descriptions will use the previous notation.

[0050] Therefore, the i-th device can obtain the i-th subsequence x obtained by performing the first partition on the input sequence X. i .

[0051] Next, in step S220, based on the self-attention parameters in the sequence model, the i-th subsequence x is... i Mapped to local query sequence q respectively i Local key sequence k i and local value sequence v i .

[0052] It should be understood that the sequence model disclosed in the embodiments of this specification is implemented based on the Transformer mechanism. Specifically, it borrows from this mechanism to achieve self-attention encoding of sample features by mapping sample features to query space, key space, and value space respectively. In the embodiments disclosed in this specification, the sequence model can be a Transformer model or a variant thereof.

[0053] In sequence models, the self-attention parameters are optimizable model weight parameters, specifically including the query mapping matrix W. Q Key mapping matrix W KSum value mapping matrix W V Therefore, in this step, matrix W can be used respectively. Q W K and W V For the i-th subsequence x i Perform a linear mapping to obtain the local query sequence q. i Local key sequence k i and local value sequence v i For example, this calculation process can be represented by the following formula:

[0054]

[0055] in,

[0056] From the above, we can define the i-th subsequence x. i Mapping these to the query, key, and value spaces respectively yields the local query sequence q. i Key sequence k i Sum sequence v i .

[0057] Then, in step S230, the local query sequence q is... i A second division is performed, resulting in m micro-sequences.

[0058] It's important to understand that the specific value of m can be set by staff based on actual needs; furthermore, the second division can be equal or unequal. For example, if it's set to equal division, and q... i If the sequence length is not an integer multiple of m, we can first calculate q. i Fill in the blanks (e.g., fill in 0 values) and then divide into equal parts.

[0059] After obtaining m micro-sequences, step S240 is executed, and m rounds of processing are performed together with the other n-1 devices.

[0060] Hereinafter, any one of the m rounds of processing will be referred to as the j-th round of processing. In the j-th round of processing, the following sub-steps S241 and S242 will be executed.

[0061] Step S241, transfer the j-th micro-sequence locally. The target sequence is determined and sent to n-1 devices respectively.

[0062] It should be understood that the i-th device can perform global communication operations with the other n-1 devices to achieve the target sequence. The transmission. There are multiple operation methods for global communication operations.

[0063] In one implementation, the global communication operation is a one-to-many broadcast, with one sender and multiple receivers. In this case, this step only involves transmitting the target sequence in the i-th device. Specifically, see Figure 4 The i-th device D i The target sequence can be ( Figure 4 The superscript j is omitted. The sequence is broadcast to the other n-1 devices. Therefore, all other devices can receive the target sequence.

[0064] In another implementation, the global communication operation is a many-to-many all-gather communication. It should be understood that gathering is many-to-one communication, where data from multiple nodes is collected onto a single node; all-gather communication is many-to-many communication, where each node acts as both sender and receiver, receiving data from all nodes. In this case, this step involves transmitting the target sequence from the i-th device. It also involves transmitting the n-1 target sequences determined in this round of processing for the other n-1 devices.

[0065] In this implementation, see Figure 3 The i-th device can be based on a locally determined target sequence. The target sequence determined by each of the n-1 devices By performing full aggregation communication, each of the n devices can receive information from... and The spliced ​​sequence Q j For more information on this fully aggregated communication, please refer to [link / reference needed]. Figure 5 The superscript j is omitted from the q symbol. Figure 3 and Figure 5 In both cases, the symbol f is used to denote fully aggregated communication.

[0066] The above mainly introduces two implementation methods for this step. In practice, other implementation methods can also be used. For example, when n is greater than 3, 2-to-n global communication can be performed to achieve the target sequence. Data sharing, etc.

[0067] Based on the above, the other n-1 devices can obtain the target sequence determined by the i-th device. Based on this, step S242 is executed, based on the target sequence. Local key sequence k i and local value sequence v i , and the key sequence k of each of the n-1 devices z Value sequence v zand the target sequence They jointly perform distributed computation to obtain the target sequence. Corresponding global attention encoding

[0068] It should be understood that by performing this step, at least the i-th device can obtain the global attention code corresponding to the j-th micro-sequence it has determined locally.

[0069] In one implementation, the communication method used in step S241 is... Figure 4 The illustrated broadcast, in this case, can adopt implementation method A, so in this step only the i-th device obtains the global attention encoding. The following is combined with... Figure 6 Description of implementation method A, such as Figure 6 As shown, in implementation method A, the following steps A61-A64 are performed:

[0070] Step A61, Device D i Based on the target sequence and local key sequence k i Determine the local attention score matrix S i .

[0071] It needs to be understood that the target sequence For local query sequence Q i Part of it. This step requires determining the target sequence. Each query element and local key sequence k i The similarity between each key element in the matrix is ​​used as an attention score to form a local attention score matrix S. i For example, the attention score matrix S can be calculated using the following formula. i :

[0072]

[0073] Thus, the local attention score matrix S can be obtained. i .

[0074] Step A62, based on the local attention score matrix S i , and the attention score matrix S of each of the n-1 devices z≠i Distributed normalization is performed to obtain the local attention weight matrix P. i .

[0075] As can be understood, the "distributed" aspect of distributed normalization processing refers to the fact that the n attention score matrices needed for normalization are distributed across n devices. Normalization can be implemented based on the softmax function or a formula for calculating proportions. Below, we will use the softmax function as an example to illustrate the implementation of this step.

[0076] It should be understood that the softmax function is calculated as follows: Based on this, we first use the natural constant e as the base, and then use the local attention score matrix S... i Each element in the matrix is ​​an exponent, and exponentiation is performed to obtain the local exponentiation matrix S. i ′.

[0077] Next, see Figure 7 Based on the local exponentiation matrix S i It performs reduction communication with the other n-1 devices using accumulation as the reduction operator to obtain the local exponentiation matrix S. i {S'} is an n-1 exponentiation matrix in other n-1 devices. z≠i,z∈[1,n] The summation result between} is used as the power operation sum matrix S′; it should be understood that reduction is a many-to-one communication method, with multiple senders and one receiver. By performing a preset reduction operation on the data from multiple senders, the data is integrated into one node.

[0078] Then, based on the local exponentiation matrix S i The local attention weight matrix P is calculated by performing exponentiation and summation on the sum matrix S'. i Specifically, for matrix S i For any element in matrix S', calculate the ratio between it and the element in matrix S' that is at the same position (or has the same row and column index), and use this ratio as the corresponding attention weight to form the local attention weight matrix P. i .

[0079] Thus, we can obtain the local attention score matrix S. i The corresponding local attention weight matrix P i .

[0080] Step A63, using the local attention weight matrix P i Processing local value sequence v i This yields the local attention encoding.

[0081] For example, the local attention code can be calculated using the following formula.

[0082]

[0083] Thus, local attention encoding can be obtained.

[0084] Step A64, obtain the global attention encoding. It is a local attention encoding and n-1 local attention codes sent by n-1 devices The summation result.

[0085] For example, see Figure 8 Based on local attention encoding Reduce communication with the other n-1 devices, using accumulation as the reduction operator, to obtain local attention encoding. n-1 local attention codes in other n-1 devices The summation result between them is used as the global attention encoding. It should be noted that, Figure 8 The superscript j is omitted from the text.

[0086] Thus, the i-th device can obtain the target sequence. Corresponding global attention encoding

[0087] The above mainly combines Figure 6 This section describes the sub-step S242 included in any of the m rounds of processing described above (see [link to documentation]). Figure 2 In Implementation A, only a single device receives global attention encoding in a single round of processing; based on this, by executing only the m micro-sequences in the i-th device... After m rounds of processing, the i-th device can obtain m global attention codes. Furthermore, Figure 2 The method shown may also include performing (n-1)*m rounds of processing on (n-1)*m micro-sequences in the other n-1 devices. For details, please refer to the relevant content in the foregoing embodiments.

[0088] In another implementation, the communication method used in the preceding step S241 is... Figure 5 The illustrated full-aggregation communication, in this case, can employ implementation method B, so that each of the n devices obtains its own global attention encoding. The following is combined with... Figure 7 Description of implementation method B, such as Figure 7 As shown, in implementation method B, the following steps B71-B74 are performed:

[0089] Step B71, Device D i Based on the spliced ​​sequence Q j and local key sequence k iDetermine the local attention score matrix S i .

[0090] It needs to be understood that the splicing sequence Q j It contains n target sequences from n devices, each target sequence being a part of the query sequence within its corresponding device. This step requires determining each query element and the local key sequence k within each target sequence. i The similarity between each key element in the matrix is ​​used as an attention score to form a local attention score matrix S. i For example, the attention score matrix S can be calculated using the following formula. i :

[0091]

[0092] Thus, the local attention score matrix S can be obtained. i It should be noted that the local attention score matrix S determined in step B71 of this procedure... i Unlike the local attention score matrix determined in step A61 above, the mathematical notation for the two is not shown in the text for easy comparison.

[0093] Step B72, based on the local attention score matrix S i The attention score matrix S determined by each of the other n-1 devices z≠i Distributed normalization is performed to obtain the local attention weight matrix P. i .

[0094] As can be understood, the "distributed" aspect of the distributed normalization process in this step refers to the fact that the n attention score matrices needed for normalization are distributed across n devices. Normalization can be implemented based on the softmax function or a formula for calculating proportions, etc. Below, we will use the softmax function to implement normalization as an example to illustrate the implementation of this step.

[0095] Specifically, first, using the natural constant e as the base, and then using the local attention score matrix S... i Each element in the matrix is ​​an exponent, and exponentiation is performed to obtain the local exponentiation matrix S. i ′.

[0096] Next, see Figure 9 Based on the local exponentiation matrix S i It communicates with the other n-1 devices via All Reduce communication using accumulation as the reduction operator to obtain the local exponentiation matrix S. i {S'} is an n-1 exponentiation matrix in other n-1 devices. z≠i,z∈[1,n]The summation result between} is used as the power operation and value matrix S′; it should be understood that full reduction is a many-to-many communication method, in which each node in the multiple nodes acts as both a sender and a receiver. By performing a preset reduction operation on the data from multiple senders, each node obtains the same integrated data.

[0097] Then, based on the local exponentiation matrix S i The local attention weight matrix P is calculated by performing exponentiation and summation on the sum matrix S'. i Specifically, for matrix S i For any element in matrix S', calculate the ratio between it and the element in matrix S' that is at the same position (or has the same row and column index), and use this ratio as the corresponding attention weight to form the local attention weight matrix P. i .

[0098] Thus, we can obtain the local attention score matrix S. i The corresponding local attention weight matrix P i .

[0099] Step B73, using the local attention weight matrix P i Processing local value sequence v i This yields a local attention encoding, which includes the corresponding concatenation sequence Q. j Local attention encoding of n parts of n target sequences.

[0100] For example, the local attention code can be calculated using the following formula.

[0101]

[0102] Thus, local attention encoding can be obtained.

[0103] Step B74, based on local attention encoding Local attention encoding determined in each of the other n-1 devices The protocol distribution is performed jointly, so that each of the n devices obtains the global attention code corresponding to the target sequence determined in this round.

[0104] For example, see Figure 10 Device D i Based on local attention encoding It communicates with the other n-1 devices via reduce scatter. It's important to understand that the reduce scatter operation first sums the inputs from each device, then splits the data along a preset dimension according to the number of devices, and distributes the resulting n parts to the corresponding devices. Thus, device D...i Global attention encoding can be obtained. Understandable. Figure 10 The superscript j of C and c is omitted. Additionally, Figure 10 and Figure 5 The symbol f is used to mark the reduction distribution.

[0105] It should be noted that the global attention encoding in this step... Compared with the global attention encoding in step A64 above Similarly, the main difference between step B74 and the aforementioned step A64 is that in this step, each of the n devices obtains the global attention encoding corresponding to the target sequence determined in this round, while in step A64, only device D... i The global attention encoding was obtained.

[0106] The above mainly combines Figure 7 This describes the sub-step S242 contained in any round of processing in m rounds (see also...). Figure 2 According to Implementation B, in any j-th round of processing, all n devices can obtain a global attention code corresponding to their j-th micro-sequence. Based on this, after performing m rounds of processing, each device can obtain m global attention codes.

[0107] Back Figure 2 Device D i By executing step S240 above, and jointly performing m rounds of processing with the other n-1 devices, m micro-sequences can be obtained. The corresponding m global attention codes

[0108] Next, step S250 can be executed, based on m global attention codes. Determine the i-th subsequence x i The corresponding attention encoding sequence c i Used to determine the prediction result of the input sequence X.

[0109] Specifically, m global attention codes can be encoded. By sequentially concatenating the sequences, we obtain the attention encoding sequence c. i .

[0110] To determine the prediction result for the input sequence X, it should be understood that the sequence model can include at least one attention encoding block. In the first attention encoding block, the i-th subsequence x is used as the basis for prediction. i As input, use attention-encoded sequence c i For the output, please refer to [link / reference]. Figure 1Furthermore, in the case of multiple attention coding blocks, each subsequent attention coding block can use the attention coding sequence output by the previous attention coding block as its input sequence to obtain the attention coding sequence output by the current attention coding block.

[0111] Furthermore, the sequence model also includes an output layer, which processes the attention-encoded sequence c. i To obtain the prediction result corresponding to the input sequence X. The method is quite flexible. For example, it can be done by device D. i The attention encoding sequence c output by the last attention encoding module i and n-1 attention-encoded sequences {c} received from other n-1 devices. z≠i,z∈[1,n] The sequences are concatenated, and then the attention-encoded sequence C corresponding to the original sequence X is processed by the output layer to obtain the prediction result. For example, the concatenation of attention-encoded sequences and processing using the output layer can also be accomplished by devices other than the n devices.

[0112] It should be noted that the parallel processing method using sequence models described above can be implemented during the training phase of the sequence model, or during the usage phase of the sequence model. In one implementation, if implemented during the training phase, after step S240, the method may further include: device D i Based on attention encoding sequence c i The backpropagation gradient is used to calculate the training gradient for updating the self-attention parameters, where the backpropagation gradient is determined based on the prediction result and the corresponding sequence label. It should be noted that the calculation of the training gradient and the updating method of the self-attention parameters are adapted to the aforementioned parallel processing method and will not be elaborated further.

[0113] In summary, the parallel processing method based on sequence models disclosed in the embodiments of this specification achieves a self-attention mechanism through efficient aggregate communication, such as the full aggregate communication and reduction distribution for micro-sequences, by distributing the attention matrix column-wise. The local query sequence (or local key sequence) is further divided into micro-sequences, and computation is split and reused in GPU memory, distributing the attention matrix row-wise. Thus, the attention matrix is ​​segmented from both row and column dimensions, achieving an attention matrix with L / mn complexity, low GPU memory consumption, and requiring only linear device resources to expand the sequence length.

[0114] Furthermore, the aggregated communication of full aggregation and reduction distribution benefits from phased aggregated communication in heterogeneous network environments, is not affected by the weakest link, and achieves higher communication efficiency by first communicating within a group (e.g., 8 nodes on 1 machine) and then between groups (e.g., 2 groups). The method disclosed in the embodiments of this specification does not divide according to the model dimension, but from the sequence dimension, which effectively reduces the huge intermediate activations in self-attention and can linearly expand the sequence length.

[0115] It should be noted that, in Figure 1 In the parallel processing method based on the sequence model shown, the second stage involves partitioning the query sequence locally. Considering that the query sequence and key sequence are symmetrical in computation during self-attention processing, it is also possible to partition the key sequence instead of the query sequence. Therefore, Figure 11 A flowchart illustrating a parallel processing method based on a sequence model according to another embodiment is shown, wherein the sequence model is deployed simultaneously on n devices.

[0116] Figure 11 The method shown can be executed by any one of the n devices, hereinafter referred to as the i-th device. For example... Figure 11 As shown, the method includes the following steps:

[0117] Step S1110: Obtain the i-th subsequence from the n parts obtained by first partitioning the input sequence.

[0118] In one embodiment, each sequence element in the input sequence is: the word embedding vector of each word segment in the text, the pixel vector of each region in the image, or the acoustic feature vector of each time interval in the audio.

[0119] Step S1120: Based on the self-attention parameters in the sequence model, the i-th subsequence is mapped to a local query sequence, a local key sequence, and a local value sequence, respectively.

[0120] Step S1130: Perform a second division on the local key sequence to obtain m micro-sequences.

[0121] Step S1140: Perform m rounds of processing together with the other n-1 devices, where any j-th round of processing includes:

[0122] Step S1141: The j-th micro-sequence locally is determined as the target sequence and sent to the n-1 devices respectively.

[0123] In one embodiment, based on the target sequence, full aggregation communication is performed together with the n-1 devices to obtain a spliced ​​sequence, which is formed by splicing the target sequence and the target sequences determined by the n-1 devices in this round of processing.

[0124] Step S1142: Based on the target sequence, local query sequence, and local value sequence, perform distributed computation together with the query sequences, value sequences, and target sequence of each of the n-1 devices to obtain the global attention code corresponding to the target sequence.

[0125] In one embodiment, this step may include: determining a local attention score matrix based on the target sequence and the local query sequence; performing distributed normalization processing on the local attention score matrix and the n-1 attention score matrices from the n-1 devices to obtain a local attention weight matrix; processing the local value sequence using the local attention weight matrix to obtain a local attention code; and obtaining the global attention code, which is the sum of the local attention code and the n-1 local attention codes sent by the n-1 devices.

[0126] Further, in a specific embodiment, the above-mentioned distributed normalization processing of the local attention score matrix with the n-1 attention score matrices from the n-1 devices to obtain the local attention weight matrix specifically includes: performing power operations with the natural constant as the base and each element in the local attention score matrix as the exponent to obtain a local power operation matrix; obtaining a power operation sum matrix, which is the sum of the local power operation matrix and the n-1 power operation matrices sent by the other n-1 devices; and calculating the local attention weight matrix based on the local power operation matrix and the power operation sum matrix.

[0127] On the other hand, in a specific embodiment, the method further includes: jointly executing other m*(n-1) rounds of processing with the other n-1 devices, wherein any round of processing includes: receiving the target sequence of the current round from a designated device among the n-1 devices, and performing distributed computation jointly with the query sequences, value sequences and the target sequence of the current round of each of the n-1 devices based on the target sequence of the current round, the local query sequence and the local value sequence, so that the designated device obtains a global attention encoding corresponding to the target sequence of the current round.

[0128] In another embodiment, this step may include: performing distributed computation based on the concatenated sequence, local query sequence, and local value sequence, together with the query sequences, value sequences, and concatenated sequences of the n-1 devices, so that each of the n devices obtains the global attention encoding corresponding to the target sequence determined in this round.

[0129] Further, in one embodiment, the above-mentioned distributed computation is performed jointly with the concatenated sequence, local query sequence, and local value sequence of the n-1 devices, along with their respective query sequences, value sequences, and the concatenated sequence, so that each of the n devices obtains the global attention encoding corresponding to its target sequence determined in this round, specifically including:

[0130] Based on the concatenated sequence and the local query sequence, a local attention score matrix is ​​determined; based on the local attention score matrix, distributed normalization processing is performed with the attention score matrices determined by each of the n-1 devices to obtain a local attention weight matrix; the local value sequence is processed using the local attention weight matrix to obtain a local attention code, which includes n parts of local attention codes corresponding to the n target sequences in the concatenated sequence; based on the local attention codes, reduction distribution is performed together with the local attention codes determined by each of the n-1 devices, so that each of the n devices obtains the global attention code corresponding to the target sequence determined in this round.

[0131] Step S1150: Based on the m global attention codes corresponding to the m micro-sequences, determine the attention code sequence corresponding to the i-th sub-sequence, which is used to determine the prediction result of the input sequence.

[0132] In one embodiment, after step S1150, the method may further include: calculating a training gradient for updating the self-attention parameters based on the attention encoding sequence and the backpropagation gradient, wherein the backpropagation gradient is determined based on the prediction result and the corresponding sequence label.

[0133] It should be noted that, for Figure 11 For a description of the steps in the Chinese method, please refer to the section on... Figure 2 Description of the steps in the method.

[0134] In summary, the parallel processing method based on sequence models disclosed in the embodiments of this specification can reduce memory consumption when using sequence models to process long sequence data in parallel, improve computational efficiency, and efficiently and conveniently adapt to the expansion of long sequences.

[0135] Corresponding to the parallel processing method described above, embodiments of this specification also disclose a parallel processing apparatus. Figure 12 A schematic diagram of a parallel processing apparatus based on a sequence model, according to one embodiment, is shown. The sequence model is simultaneously deployed in n devices, and the apparatus is integrated into any i-th device therein. Figure 12 As shown, the device 1200 includes:

[0136] The subsequence acquisition module 1210 is configured to acquire the i-th subsequence from the n parts obtained by first partitioning the input sequence. The subsequence mapping module 1220 is configured to map the i-th subsequence to a local query sequence, a local key sequence, and a local value sequence based on the self-attention parameters in the sequence model. The microsequence acquisition module 1230 is configured to perform a second partitioning on the local query sequence to obtain m microsequences. The microsequence processing module 1240 is configured to perform m rounds of processing together with the other n-1 devices. The microsequence processing module performs any j-th round of processing through the following units: the microsequence sending unit 1241 is configured to determine the j-th local microsequence as the target sequence and send it to the n-1 devices respectively; the distributed computing unit 1242 is configured to perform distributed computing together with the key sequences, value sequences, and target sequences of the n-1 devices based on the target sequence, the local key sequence, and the local value sequence to obtain the global attention encoding corresponding to the target sequence. The encoding construction module 1250 is configured to determine the attention encoding sequence corresponding to the i-th sub-sequence based on the m global attention codes corresponding to the m micro-sequences, and to determine the prediction result of the input sequence.

[0137] In one embodiment, the distributed computing unit 1242 is specifically configured to: determine a local attention score matrix based on the target sequence and the local key sequence; perform distributed normalization processing on the local attention score matrix and the n-1 attention score matrices from the n-1 devices to obtain a local attention weight matrix; process the local value sequence using the local attention weight matrix to obtain a local local attention code; and obtain the global attention code, which is the sum of the local local attention code and the n-1 local attention codes sent by the n-1 devices.

[0138] In a specific embodiment, the distributed computing unit 1242 is configured to perform distributed normalization processing on the local attention score matrix and the n-1 attention score matrices from the n-1 devices to obtain a local attention weight matrix, specifically including:

[0139] Using the natural constant as the base, and performing exponentiation operations with each element in the local attention score matrix as the exponent, a local exponentiation operation matrix is ​​obtained; the exponentiation operation sum matrix is ​​obtained, which is the sum of the local exponentiation operation matrix and the n-1 exponentiation operation matrices sent by the other n-1 devices; based on the local exponentiation operation matrix and the exponentiation operation sum matrix, the local attention weight matrix is ​​calculated.

[0140] In one embodiment, the micro-sequence processing module 1240 is further configured to: jointly execute other m*(n-1) rounds of processing with the other n-1 devices, wherein any round of processing includes: receiving the target sequence of the current round from a designated device among the n-1 devices, and performing distributed computation jointly with the key sequences, value sequences and the target sequence of the current round of each of the n-1 devices based on the target sequence of the current round, the local key sequence and the local value sequence, so that the designated device obtains a global attention encoding corresponding to the target sequence of the current round.

[0141] In one embodiment, the micro-sequence sending unit 1241 is specifically configured to: perform full aggregation communication with the n-1 devices based on the target sequence to obtain a spliced ​​sequence, which is formed by splicing the target sequence and the target sequences determined by the n-1 devices in this round of processing; the distributed computing unit 1242 is specifically configured to: perform distributed computing with the key sequence, value sequence and spliced ​​sequence of the n-1 devices based on the spliced ​​sequence, the local key sequence and the local value sequence, so that the n devices each obtain the global attention encoding corresponding to the target sequence determined by them in this round.

[0142] In a specific embodiment, the distributed computing unit 1242 is further configured to: determine a local attention score matrix based on the concatenated sequence and the local key sequence; perform distributed normalization processing on the local attention score matrix and the attention score matrices determined by each of the n-1 devices to obtain a local attention weight matrix; process the local value sequence using the local attention weight matrix to obtain a local attention code, which includes n parts of local attention codes corresponding to the n target sequences in the concatenated sequence; and perform reduction distribution based on the local attention codes and the local attention codes determined by each of the n-1 devices, so that each of the n devices obtains the global attention code corresponding to the target sequence determined in this round.

[0143] In one embodiment, the apparatus 1200 further includes a gradient calculation unit 1260 configured to calculate a training gradient for updating the self-attention parameters based on the attention encoding sequence and the backpropagation gradient, wherein the backpropagation gradient is determined based on the prediction result and the corresponding sequence label.

[0144] In one embodiment, each sequence element in the input sequence is: the word embedding vector of each word segment in the text, the pixel vector of each region in the image, or the acoustic feature vector of each time interval in the audio.

[0145] Figure 13A schematic diagram of a parallel processing apparatus based on a sequence model according to another embodiment is shown, wherein the sequence model is simultaneously deployed in n devices, and the apparatus is integrated into any i-th device therein. Figure 13 As shown, the device 1300 includes:

[0146] The subsequence acquisition module 1310 is configured to acquire the i-th subsequence from the n parts obtained by first partitioning the input sequence. The subsequence mapping module 1320 is configured to map the i-th subsequence to a local query sequence, a local key sequence, and a local value sequence based on the self-attention parameters in the sequence model. The microsequence acquisition module 1330 is configured to perform a second partitioning on the local query sequence to obtain m microsequences. The microsequence processing module 1340 is configured to perform m rounds of processing together with the other n-1 devices. The microsequence processing module performs any j-th round of processing through the following units: the microsequence sending unit 1341 is configured to determine the j-th local microsequence as the target sequence and send it to the n-1 devices respectively; the distributed computing unit 1342 is configured to perform distributed computing together with the query sequences, value sequences, and target sequences of the n-1 devices based on the target sequence, the local query sequence, and the local value sequence to obtain the global attention code corresponding to the target sequence. The encoding construction module 1350 is configured to determine the attention encoding sequence corresponding to the i-th sub-sequence based on the m global attention codes corresponding to the m micro-sequences, and to determine the prediction result of the input sequence.

[0147] In one embodiment, the distributed computing unit 1342 is specifically configured to: determine a local attention score matrix based on the target sequence and the local query sequence; perform distributed normalization processing on the local attention score matrix and the n-1 attention score matrices from the n-1 devices to obtain a local attention weight matrix; process the local value sequence using the local attention weight matrix to obtain a local local attention code; and obtain the global attention code, which is the sum of the local local attention code and the n-1 local attention codes sent by the n-1 devices.

[0148] In a specific embodiment, the distributed computing unit 1342 is configured to perform distributed normalization processing on the local attention score matrix and the n-1 attention score matrices from the n-1 devices to obtain a local attention weight matrix, specifically including:

[0149] Using the natural constant as the base, and performing exponentiation operations with each element in the local attention score matrix as the exponent, a local exponentiation operation matrix is ​​obtained; the exponentiation operation sum matrix is ​​obtained, which is the sum of the local exponentiation operation matrix and the n-1 exponentiation operation matrices sent by the other n-1 devices; based on the local exponentiation operation matrix and the exponentiation operation sum matrix, the local attention weight matrix is ​​calculated.

[0150] In one embodiment, the micro-sequence processing module 1340 is further configured to: jointly execute other m*(n-1) rounds of processing with the other n-1 devices, wherein any round of processing includes: receiving the target sequence of the current round from a designated device among the n-1 devices, and performing distributed computation jointly with the query sequences, value sequences and the target sequence of the current round of each of the n-1 devices based on the target sequence of the current round, the local query sequence and the local value sequence, so that the designated device obtains a global attention encoding corresponding to the target sequence of the current round.

[0151] In one embodiment, the micro-sequence sending unit 1341 is specifically configured to: perform full aggregation communication with the n-1 devices based on the target sequence to obtain a spliced ​​sequence, which is formed by splicing the target sequence and the target sequences determined by the n-1 devices in this round of processing; the distributed computing unit 1342 is specifically configured to: perform distributed computing with the query sequences, value sequences, and spliced ​​sequences of the n-1 devices based on the spliced ​​sequence, the local query sequence, and the local value sequence, so that the n devices each obtain the global attention encoding corresponding to the target sequence determined by them in this round.

[0152] In a specific embodiment, the distributed computing unit 1342 is further configured to: determine a local attention score matrix based on the concatenated sequence and the local query sequence; perform distributed normalization processing on the local attention score matrix and the attention score matrices determined by each of the n-1 devices to obtain a local attention weight matrix; process the local value sequence using the local attention weight matrix to obtain a local attention code, which includes n parts of local attention codes corresponding to the n target sequences in the concatenated sequence; and perform reduction distribution based on the local attention codes and the local attention codes determined by each of the n-1 devices, so that each of the n devices obtains the global attention code corresponding to the target sequence determined in this round.

[0153] In one embodiment, the apparatus 1300 further includes a gradient calculation unit 1360 configured to calculate a training gradient for updating the self-attention parameters based on the attention encoding sequence and the backpropagation gradient, wherein the backpropagation gradient is determined based on the prediction result and the corresponding sequence label.

[0154] In one embodiment, each sequence element in the input sequence is: the word embedding vector of each word segment in the text, the pixel vector of each region in the image, or the acoustic feature vector of each time interval in the audio.

[0155] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 , Figure 6 or Figure 11 The method described.

[0156] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 , Figure 6 or Figure 11 The methods described herein. Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0157] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A parallel processing method based on a sequence model, wherein the sequence model is simultaneously deployed on n devices, and the method is executed by any i-th device thereon, comprising: Obtain the i-th subsequence from the n parts obtained by first partitioning the input sequence; Based on the self-attention parameter in the sequence model, the i-th subsequence is mapped to a local query sequence, a local key sequence, and a local value sequence, respectively, and the local query sequence is further divided into m micro-sequences. Together with the other n-1 devices, m rounds of processing are performed, where any j-th round of processing includes: determining the local j-th micro-sequence as the target sequence and sending it to the n-1 devices respectively; based on the target sequence, the local key sequence, and the local value sequence, performing distributed computation together with the key sequences, value sequences, and the target sequence of each of the n-1 devices to obtain a global attention code corresponding to the target sequence; Based on the m global attention codes corresponding to the m micro-sequences, an attention code sequence corresponding to the i-th sub-sequence is determined, which is used to determine the prediction result of the input sequence.

2. The method according to claim 1, wherein, Based on the target sequence, local key sequence, and local value sequence, distributed computation is performed together with the key sequences, value sequences, and target sequence of each of the n-1 devices to obtain a global attention encoding corresponding to the target sequence, including: Based on the target sequence and the local key sequence, determine the local attention score matrix; Based on the local attention score matrix, a distributed normalization process is performed on the n-1 attention score matrices of the n-1 devices to obtain the local attention weight matrix; The local value sequence is processed using the local attention weight matrix to obtain the local attention encoding; Obtain the global attention code, which is the sum of the local attention code and the n-1 local attention codes sent by the n-1 devices.

3. The method according to claim 2, wherein, Based on the local attention score matrix, a distributed normalization process is performed between it and the n-1 attention score matrices from the n-1 devices to obtain the local attention weight matrix, including: Using the natural constant as the base, and performing exponentiation operations with each element in the local attention score matrix as the exponent, a local exponentiation operation matrix is ​​obtained; Obtain the power operation sum matrix, which is the sum of the local power operation matrix and the n-1 power operation matrices sent by the other n-1 devices; The local attention weight matrix is ​​calculated based on the local exponentiation matrix and the exponentiation sum matrix.

4. The method according to claim 1, further comprising: Together with the other n-1 devices, perform the remaining m*(n-1) rounds of processing, where any round of processing includes: The designated device receives the target sequence for this round from the n-1 devices. Based on the target sequence, the local key sequence, and the local value sequence, it performs distributed computation together with the key sequences, value sequences, and the target sequence of the n-1 devices, so that the designated device obtains a global attention encoding corresponding to the target sequence for this round.

5. The method according to claim 1, wherein, The local j-th micro-sequence is determined as the target sequence and sent to the n-1 devices respectively, including: Based on the target sequence, full aggregation communication is performed together with the n-1 devices to obtain a spliced ​​sequence, which is formed by splicing the target sequence and the target sequences determined by the n-1 devices in this round of processing; Specifically, based on the target sequence, local key sequence, and local value sequence, distributed computation is performed together with the key sequences, value sequences, and target sequence of each of the n-1 devices to obtain a global attention code corresponding to the target sequence, including: Based on the concatenated sequence, local key sequence, and local value sequence, distributed computation is performed together with the key sequence, value sequence, and concatenated sequence of each of the n-1 devices, so that each of the n devices obtains the global attention encoding corresponding to the target sequence determined in this round.

6. The method according to claim 5, wherein, Based on the concatenated sequence, local key sequence, and local value sequence, distributed computation is performed together with the key sequences, value sequences, and the concatenated sequence of each of the n-1 devices, so that each of the n devices obtains the global attention encoding corresponding to its target sequence determined in this round, including: Based on the spliced ​​sequence and the local key sequence, determine the local attention score matrix; Based on the local attention score matrix, a distributed normalization process is performed on the attention score matrices determined by each of the n-1 devices to obtain the local attention weight matrix; The local value sequence is processed using the local attention weight matrix to obtain local attention encoding, which includes n parts of local attention encoding corresponding to the n target sequences in the concatenated sequence; Based on the local attention encoding, reduction distribution is performed together with the local attention encodings determined by each of the n-1 devices, so that each of the n devices obtains the global attention encoding corresponding to the target sequence determined in this round.

7. The method according to claim 1, wherein, After determining the attention encoding sequence corresponding to the i-th sub-sequence, the method further includes: Based on the attention encoding sequence and the backpropagation gradient, a training gradient for updating the self-attention parameters is calculated, wherein the backpropagation gradient is determined based on the prediction result and the corresponding sequence label.

8. The method according to any one of claims 1-7, wherein, Each sequence element in the input sequence is: the word embedding vector of each word in the text, the pixel vector of each region in the image, or the acoustic feature vector of each time interval in the audio.

9. A parallel processing method based on a sequence model, wherein the sequence model is simultaneously deployed on n devices, and the method is executed by any i-th device thereon, comprising: Obtain the i-th subsequence from the n parts obtained by first partitioning the input sequence; Based on the self-attention parameter in the sequence model, the i-th subsequence is mapped to a local query sequence, a local key sequence, and a local value sequence, respectively, and the local key sequence is further divided to obtain m micro-sequences; Together with the other n-1 devices, m rounds of processing are performed, where any j-th round of processing includes: determining the j-th local micro-sequence as the target sequence and sending it to the n-1 devices respectively; based on the target sequence, the local query sequence, and the local value sequence, performing distributed computation together with the query sequences, value sequences, and the target sequence of each of the n-1 devices to obtain the global attention code corresponding to the target sequence; Based on the m global attention codes corresponding to the m micro-sequences, an attention code sequence corresponding to the i-th sub-sequence is determined, which is used to determine the prediction result of the input sequence.

10. The method of claim 9, further comprising: Together with the other n-1 devices, perform the remaining m*(n-1) rounds of processing, where any round of processing includes: The designated device receives the target sequence for this round from the n-1 devices. Based on the target sequence, the local query sequence, and the local value sequence, it performs distributed computation together with the query sequences, value sequences, and the target sequence of the n-1 devices to enable the designated device to obtain a global attention encoding corresponding to the target sequence for this round.

11. The method according to claim 9, wherein, The local j-th micro-sequence is determined as the target sequence and sent to the n-1 devices respectively, including: Based on the target sequence, full aggregation communication is performed together with the n-1 devices to obtain a spliced ​​sequence, which is formed by splicing the target sequence and the target sequences determined by the n-1 devices in this round of processing; Specifically, based on the target sequence, local query sequence, and local value sequence, distributed computation is performed together with the query sequences, value sequences, and target sequence of each of the n-1 devices to obtain a global attention code corresponding to the target sequence, including: Based on the concatenated sequence, local query sequence, and local value sequence, distributed computation is performed together with the query sequences, value sequences, and concatenated sequences of the n-1 devices, so that each of the n devices obtains the global attention encoding corresponding to the target sequence determined in this round.

12. A parallel processing device based on a sequence model, wherein the sequence model is simultaneously deployed in n devices, and the device is integrated into any i-th device therein, comprising: The subsequence acquisition module is configured to acquire the i-th subsequence from the n parts obtained by first partitioning the input sequence. The subsequence mapping module is configured to map the i-th subsequence into a local query sequence, a local key sequence, and a local value sequence based on the self-attention parameters in the sequence model. The micro-sequence acquisition module is configured to perform a second division on the local query sequence to obtain m micro-sequences; The micro-sequence processing module is configured to perform m rounds of processing together with n-1 other devices. The micro-sequence processing module performs any j-th round of processing through the following units: The micro-sequence sending unit is configured to determine the j-th local micro-sequence as the target sequence and send it to the n-1 devices respectively; A distributed computing unit is configured to perform distributed computing together with the key sequences, value sequences, and target sequences of the n-1 devices based on the target sequence, local key sequence, and local value sequence, to obtain a global attention code corresponding to the target sequence. The encoding construction module is configured to determine the attention encoding sequence corresponding to the i-th sub-sequence based on the m global attention codes corresponding to the m micro-sequences, and is used to determine the prediction result of the input sequence.

13. A parallel processing device based on a sequence model, wherein the sequence model is simultaneously deployed in n devices, and the device is integrated into any i-th device therein, comprising: The subsequence acquisition module is configured to acquire the i-th subsequence from the n parts obtained by first partitioning the input sequence. The subsequence mapping module is configured to map the i-th subsequence into a local query sequence, a local key sequence, and a local value sequence based on the self-attention parameters in the sequence model. The micro-sequence acquisition module is configured to perform a second partitioning on the local key sequence to obtain m micro-sequences; The micro-sequence processing module is configured to perform m rounds of processing together with n-1 other devices. The micro-sequence processing module performs any j-th round of processing through the following units: The micro-sequence sending unit is configured to determine the j-th local micro-sequence as the target sequence and send it to the n-1 devices respectively; A distributed computing unit is configured to perform distributed computing together with the query sequences, value sequences, and target sequences of each of the n-1 devices based on the target sequence, the local query sequence, and the local value sequence, to obtain a global attention code corresponding to the target sequence. The encoding construction module is configured to determine the attention encoding sequence corresponding to the i-th sub-sequence based on the m global attention codes corresponding to the m micro-sequences, and is used to determine the prediction result of the input sequence.

14. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in the computer, it causes the computer to perform the method according to any one of claims 1-11.

15. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-11.

Images