Multi-head attention processing method, related device and medium

Abstract

The invention provides a multi-head attention processing method, a related device and a medium, and the method comprises the steps: dividing an input matrix into a first number of sub-blocks, and carrying out the first linear transformation through a first number of processing units, and obtaining a first number of first transformed sub-blocks; performing sub-block transformation on a first number of first transformed sub-blocks corresponding to the first number of processing units to obtain a second transformed sub-block corresponding to each processing head, and inputting the second transformed sub-block into a processing head corresponding to the second transformed sub-block for attention calculation to obtain a first attention result sub-block; and performing sub-block inverse transformation on the first attention result sub-blocks corresponding to the processing heads to obtain second attention result sub-blocks corresponding to the processing units, and generating a multi-head attention result matrix based on the second attention result sub-blocks. The multi-head attention processing efficiency can be improved, and the multi-head attention processing method and device can be applied to scenes such as artificial intelligence assistants and machine translation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of neural networks, and in particular to a multi-head attention processing method, related apparatus and medium. BACKGROUND

[0002] The multi-head attention mechanism is a commonly used technique in neural networks, which distributes multiple different dimension hidden layer data of an input matrix to multiple processing heads for attention operation, and calculates the attention matrix between different hidden layer data in the input matrix in each processing head, so that the neural network model can accurately capture the features of different levels in the input matrix.

[0003] In order to improve the ability of the neural network model to understand the context, the length of the input matrix will become longer. When the input matrix is longer, the total time required for multi-head attention operation by multiple processing heads of a single device is proportional to the square of the length of the input matrix. In order to improve the calculation efficiency, in the prior art, the input matrix is often divided into blocks and distributed to multiple processing units for pre-linear transformation. This block processing method greatly improves the efficiency of pre-linear transformation processing. However, the real attention calculation after pre-linear transformation is performed by the processing heads in the processing unit (there are multiple processing heads in a processing unit), and the transformation block obtained after the pre-linear transformation is not suitable for processing by the processing heads, so the transformation block is transformed into a new transformation block shared by all processing heads in the processing unit, and the attention calculation is performed by all processing heads in the processing unit. As can be seen from the above, in the prior art, the transformation to the new transformation block shared by all processing heads must be completed before the attention calculation can begin, which makes the multi-head attention processing inefficient. SUMMARY

[0004] The embodiments of the present disclosure provide a multi-head attention processing method, related apparatus and medium, which can improve the efficiency of multi-head attention processing.

[0005] According to an aspect of the present disclosure, a multi-head attention processing method is provided, applied to a processing device, the processing device comprising a first number of processing units, the processing units comprising a second number of processing heads, the method comprising:

[0006] dividing an input matrix into a first number of sub-blocks, and respectively distributing the first number of sub-blocks to the first number of processing units to perform first linear transformation, to obtain a first number of first transformed sub-blocks corresponding to the first number of processing units;

[0007] perform sub-block transformation on the first number of first transformed sub-blocks corresponding to the first number of processing units to sequentially obtain a second transformed sub-block corresponding to each processing head, and input the second transformed sub-block to the processing head corresponding to the second transformed sub-block to perform attention calculation to obtain a first attention result sub-block corresponding to each processing head, wherein the second number of processing heads are in order, and the sub-block transformation of a next processing head of a current processing head is performed while the attention calculation on the second transformed sub-block corresponding to the current processing head is performed;

[0008] perform sub-block inverse transformation on the first attention result sub-block corresponding to each processing head to obtain a second attention result sub-block corresponding to each processing unit, and generate a multi-head attention result matrix based on the second attention result sub-block corresponding to each processing unit.

[0009] According to an aspect of the present disclosure, a multi-head attention processing apparatus is provided, the apparatus comprising:

[0010] a first linear transformation unit configured to divide an input matrix into a first number of sub-blocks, and perform first linear transformation on the first number of sub-blocks by the first number of processing units to obtain a first number of first transformed sub-blocks corresponding to the first number of processing units;

[0011] a first attention calculation unit configured to perform sub-block transformation on the first number of first transformed sub-blocks corresponding to the first number of processing units to sequentially obtain a second transformed sub-block corresponding to each processing head, and input the second transformed sub-block to the processing head corresponding to the second transformed sub-block to perform attention calculation to obtain a first attention result sub-block corresponding to each processing head, wherein the second number of processing heads are in order, and the sub-block transformation of a next processing head of a current processing head is performed while the attention calculation on the second transformed sub-block corresponding to the current processing head is performed;

[0012] a first matrix splicing unit configured to perform sub-block inverse transformation on the first attention result sub-block corresponding to each processing head to obtain a second attention result sub-block corresponding to each processing unit, and generate a multi-head attention result matrix based on the second attention result sub-block corresponding to each processing unit.

[0013] Optionally, the first linear transformation unit is configured to:

[0014] determine a query weight matrix, a key weight matrix and a value weight matrix corresponding to the processing unit;

[0015] The processing unit corresponds to the sub-block and the query weight matrix, the key weight matrix and the value weight matrix corresponding to the processing unit are respectively multiplied by the matrix to obtain the sub-query matrix, the sub-key matrix and the sub-value matrix corresponding to the sub-block, wherein the sub-query matrix, the sub-key matrix and the sub-value matrix constitute the first transformed sub-block corresponding to the processing unit.

[0016] Optionally, the input matrix has a first dimension and a second dimension, and the first linear transformation unit is specifically configured to:

[0017] The input matrix is divided into a first number of sub-blocks in the first dimension;

[0018] The first attention calculation unit is configured to:

[0019] The first transformed sub-block is divided into a block corresponding to each processing head according to the second number in the second dimension;

[0020] The block is divided into a first number of first patches corresponding to the first number of processing units in the second dimension;

[0021] For a single processing head of a single processing unit, the first patch corresponding to the single processing head in each first transformed sub-block is aggregated to obtain the second transformed sub-block corresponding to the single processing head.

[0022] Optionally, the first linear transformation unit is further configured to:

[0023] The communication resource amount of each processing unit is determined;

[0024] Based on the ratio of the communication resource amount of each processing unit, the input matrix is divided into a first number of sub-blocks along the first dimension.

[0025] Optionally, the first linear transformation unit is further configured to:

[0026] The calculation resource amount of each processing unit is determined;

[0027] Based on the ratio of the calculation resource amount of each processing unit, the input matrix is divided into a first number of sub-blocks along the first dimension.

[0028] Optionally, the first attention calculation unit is specifically configured to:

[0029] The second quotient of the fourth number and the first number is determined;

[0030] In the block corresponding to the current processing head, every second quotient of hidden layers is divided into a first patch along the second dimension to obtain a plurality of first patches;

[0031] From the plurality of first patches, a target first patch corresponding to the current processing head is determined, and the plurality of first patches other than the target first patch are determined as to-be-sent first patches.

[0032] Optionally, the first attention calculation unit is specifically configured to:

[0033] determine the amount of computing resources of the processing unit;

[0034] based on the ratio of the amount of computing resources of each processing unit, the sub-block is divided into a first number of first patches corresponding to each processing unit in the second dimension;

[0035] the first patch corresponding to the processing unit of the current processing head is determined as a target first patch, and other first patches other than the target first patch are determined as to-be-sent first patches.

[0036] Optionally, the first attention calculation unit is further configured to:

[0037] in each processing unit, each processing head is sequentially taken as a current processing head;

[0038] data of each to-be-sent first patch in the sub-block corresponding to the current processing head is sent to an interaction processing unit, and a target first patch corresponding to the current processing head is received from each interaction processing unit, wherein the interaction processing unit is other processing unit other than the processing unit corresponding to the current processing head;

[0039] a plurality of target first patches corresponding to the current processing head are spliced along a first dimension to obtain a second transformed sub-block corresponding to the current processing head.

[0040] Optionally, the processing unit includes a sending buffer and a receiving buffer, and the first attention calculation unit is further configured to:

[0041] when each to-be-sent first patch corresponding to the previous processing head of the current processing head in the sending buffer is sent to the corresponding interaction processing unit, each to-be-sent first patch corresponding to the current processing head is loaded into the sending buffer, and the data in the sending buffer is sent to the corresponding interaction processing unit through the communication thread of the processing unit;

[0042] When all data corresponding to a previous processing head of the current processing head in the receiving buffer is fetched, the target first patch corresponding to the current processing head is loaded into the receiving buffer, the target first patch corresponding to the current processing head is received from each interactive processing unit through the communication thread, and is sequentially written into the receiving buffer.

[0043] Optionally, the first attention calculation unit is further configured to:

[0044] After each target first patch corresponding to the current processing head sent by each interactive processing unit is collected in the receiving buffer, and a plurality of target first patches are spliced into the second transformed sub-block, the second transformed sub-block is fetched from the receiving buffer and transmitted to the calculation buffer of the calculation thread of the processing unit;

[0045] After the first attention result sub-block corresponding to the second transformed sub-block of the previous processing head of the current processing head is calculated by the calculation thread of the processing unit, the second transformed sub-block corresponding to the current processing head is read from the calculation buffer.

[0046] Optionally, the first attention calculation unit is further configured to:

[0047] Based on the position of the sub-block corresponding to each processing unit in the input matrix, a first splicing order is determined;

[0048] Based on the first splicing order, the target first patch sent by each interactive processing unit and the target first patch corresponding to the current processing head are spliced to obtain a second transformed patch corresponding to the current processing head.

[0049] Optionally, the second transformed sub-block includes a query matrix block, a key matrix block and a value matrix block, and the first attention calculation unit is further configured to:

[0050] In the current processing head, the key matrix block is transposed to obtain a transposed key matrix block;

[0051] The query matrix block and the transposed key matrix block are multiplied to obtain a product matrix;

[0052] The product matrix is exponentially normalized to obtain an exponentially normalized matrix;

[0053] The exponentially normalized matrix and the value matrix block are multiplied to obtain the first attention result sub-block.

[0054] Optionally, the first attention calculation unit is further configured to:

[0055] taking each element in the product matrix as an exponent of a first base to obtain a power of each element;

[0056] determining, for each matrix row in the product matrix, a first sum of the powers of each element in the matrix row;

[0057] determining, as an element of each matrix row in the exponential normalization matrix, a ratio of the power of each element to the first sum of the corresponding matrix row.

[0058] Optionally, the first matrix splicing unit is further configured to:

[0059] in the first dimension, divide, for the current processing head in each processing unit, the first attention result sub-block corresponding to the current processing head into second patches corresponding to each processing unit according to the first number;

[0060] aggregate the second patches corresponding to a single processing unit to obtain a second attention result sub-block corresponding to the current processing head in each processing unit;

[0061] merge the second attention result sub-blocks corresponding to each processing head in the processing unit to obtain the second attention result sub-block corresponding to the processing unit.

[0062] Optionally, the first matrix splicing unit is further configured to:

[0063] determine the processing unit corresponding to the current processing head as a target processing unit, and determine other processing units in the processing device except the target processing unit as interactive processing units;

[0064] after the data in the sending buffer and the receiving buffer are taken out, load the second patches corresponding to each interactive processing unit into the sending buffer, and load the second patches corresponding to the target processing unit into the receiving buffer;

[0065] send, by a communication thread of the target processing unit, each second patch in the sending buffer to the corresponding interactive processing unit, and receive, from each interactive processing unit, the second patch corresponding to the target processing unit, write into the receiving buffer, and splice each second patch in the receiving buffer into the second attention result sub-block corresponding to the current processing head along the second dimension.

[0066] Optionally, the first matrix splicing unit is further configured to:

[0067] determine a second splicing order based on the hidden layer corresponding to each second patch;

[0068] splicing each of the second patches into the second attention result sub-blocks based on the second splicing order.

[0069] Optionally, the first matrix splicing unit is further configured to:

[0070] aggregate the second attention result sub-blocks corresponding to each processing unit into a to-be-corrected matrix;

[0071] generate the multi-head attention result matrix based on the to-be-corrected matrix and a correction coefficient matrix.

[0072] According to an aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, the memory storing a computer program, and the processor implementing the multi-head attention processing method as described above when executing the computer program.

[0073] According to an aspect of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, implements the multi-head attention processing method as described above.

[0074] According to an aspect of the present disclosure, a computer program product is provided, which includes a computer program, the computer program being read and executed by a processor of a computer device, so that the computer device executes the multi-head attention processing method as described above.

[0075] In the embodiments of the present disclosure, the first transformed sub-block obtained through the pre-linear transformation is not transformed into a new transformed block shared by all processing heads as in the prior art, but is transformed into a second transformed sub-block corresponding to each processing head through sub-block transformation. The processing heads are in order. While attention calculation is performed on the second transformed sub-block corresponding to the current processing head, sub-block transformation is performed on the next processing head of the current processing head to obtain the second transformed sub-block corresponding to the next processing head. While attention calculation is performed on the second transformed sub-block corresponding to the next processing head, sub-block transformation is performed on the next processing head to obtain the second transformed sub-block corresponding to the next processing head. Attention calculation and sub-block transformation of the next processing head are performed simultaneously, which reduces the total time length of processing units for multi-head attention processing and improves the efficiency of multi-head attention processing.

[0076] Other features and advantages of the present disclosure will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present disclosure. The objects and other advantages of the present disclosure can be achieved and obtained by the structure particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0077] The accompanying drawings are used to provide a further understanding of the technical solutions of the present disclosure, and constitute a part of the specification, and are used to explain the technical solutions of the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions of the present disclosure.

[0078] Figure 1 is a framework diagram of a system to which a multi-head attention processing method according to an embodiment of the present disclosure is applied;

[0079] Figures 2A-2F is an interface schematic diagram in an artificial intelligence assistant scenario to which an embodiment of the present disclosure is applied;

[0080] Figures 3A-3B is an interface schematic diagram in a machine translation scenario to which an embodiment of the present disclosure is applied;

[0081] Figure 4 is a flowchart of a multi-head attention processing method according to an embodiment of the present disclosure;

[0082] Figure 5 is a schematic diagram of a multi-head attention processing method according to an embodiment of the present disclosure in a case where an input matrix is subjected to a first linear transformation in a single processing unit;

[0083] Figure 6 is Figure 4 is a flowchart of step 430 in

[0084] Figure 7 is a flowchart according to an embodiment of the present disclosure, in which an input matrix is split along a first dimension and the first transformed sub-block is split along a second dimension, respectively;

[0085] Figure 8 is a schematic diagram of an implementation process according to an embodiment of the present disclosure in a case where an input matrix is split into sub-blocks and the first transformed sub-block is split into sub-blocks;

[0086] Figure 9 is a schematic diagram of an implementation process according to an embodiment of the present disclosure in a case where an input matrix is split into sub-blocks but the first transformed sub-block is not split into sub-blocks;

[0087] Figure 10 is a flowchart of performing a first linear transformation according to an embodiment of the present disclosure;

[0088] Figure 11 is Figure 7 is a flowchart of step 710 in

[0089] Figure 12 is a schematic diagram of a process of splitting an input matrix based on a ratio of a communication resource amount according to an embodiment of the present disclosure;

[0090] Figure 13 is Figure 7 a flowchart of steps 710 in

[0091] Figure 14 is a process diagram of splitting an input matrix based on a ratio of computing resource amounts according to one embodiment of the present disclosure;

[0092] Figure 15 is Figure 7 a flowchart of steps 730 in

[0093] Figure 16 is a process diagram of equally dividing a first transformed sub-block into a second number of sub-blocks according to one embodiment of the present disclosure;

[0094] Figure 17 is Figure 7 a flowchart of steps 740 in

[0095] Figure 18 is a process diagram of equally dividing a sub-block into a first number of first tiles according to one embodiment of the present disclosure;

[0096] Figure 19 is Figure 7 a flowchart of steps 740 in

[0097] Figure 20 is a process diagram of splitting a sub-block into a first number of first tiles based on a ratio of computing resource amounts according to one embodiment of the present disclosure;

[0098] Figure 21 is Figure 7 a flowchart of steps 750 in

[0099] Figure 22 is a process diagram of sub-block transformation and attention computation of a sub-block concurrently performed among a first number of processing units according to one embodiment of the present disclosure;

[0100] Figure 23 is Figure 21 a flowchart of steps 2120 in

[0101] Figure 24 is a flowchart of a multi-head attention processing method according to one embodiment of the present disclosure;

[0102] Figure 25 is a process diagram of concurrent execution of sub-block transformation and attention computation of different processing heads according to one embodiment of the present disclosure;

[0103] Figure 26 is a process diagram of serial execution of sub-block transformation and attention computation of a processing unit according to one embodiment of the present disclosure;

[0104] Figure 27 yes Figure 21 A flowchart of step 2130;

[0105] Figure 28 yes Figure 4 A flowchart of step 420;

[0106] Figure 29 yes Figure 28 A flowchart of step 2830;

[0107] Figure 30 yes Figure 4 A flowchart of step 430;

[0108] Figure 31 yes Figure 30 A flowchart of step 3020;

[0109] Figure 32 This is a schematic diagram illustrating the process of concurrent execution of the sub-block inverse transformation and the attention calculation process, and the serial execution of the sub-block inverse transformation and the sub-block transformation, according to an embodiment of this disclosure.

[0110] Figure 33 yes Figure 30 A flowchart of step 3020;

[0111] Figure 34 This is a process diagram illustrating a specific implementation example of the multi-head attention processing method according to an embodiment of this disclosure;

[0112] Figure 35 This is a block diagram of a multi-head attention processing apparatus according to an embodiment of the present disclosure;

[0113] Figure 36 This is a terminal structure diagram according to an embodiment of the present disclosure;

[0114] Figure 37 This is a server structure diagram according to an embodiment of the present disclosure. Detailed Implementation

[0115] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0116] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0117] Multi-Head Attention (MHA): a technique widely used in natural language processing and other sequence model tasks, which learns different representations of data in different subspaces to capture information in different dimensions of the input data. Specifically, the multi-head attention mechanism first performs a linear transformation on the input sequence to obtain a corresponding query matrix, key matrix, and value matrix. These matrices have the same shape, with each matrix row corresponding to each element of the input sequence and each matrix column corresponding to each hidden layer of the input sequence. For example, if the input sequence has a length of N and the number of hidden layers is d, the shapes of the corresponding query matrix, key matrix, and value matrix are all (N, d). Then, different matrix columns in the query matrix, key matrix, and value matrix are distributed to multiple attention heads for attention calculation. For example, there are hc attention heads, and after the query matrix, key matrix, and value matrix are evenly distributed to each attention head, the shape of the query matrix block, key matrix block, and value matrix block obtained by each attention head is (N, d / hc). Each attention head independently performs attention calculation based on the obtained matrix block to obtain the calculation result of each attention head. Finally, the calculation results of all attention heads are concatenated and linearly transformed to obtain the final output matrix, which reflects the features of the input sequence at multiple different levels.

[0118] Attention head: a component of the multi-head attention mechanism. In the multi-head attention mechanism, the linear transformation result of the input sequence is divided into multiple parts and distributed to each attention head, and each attention head independently performs attention calculation to learn and capture semantic information in different dimensions of the input sequence. By learning different feature associations from different dimensions of the input sequence, the model can focus on different aspects of the input sequence.

[0119] Multi-Head Attention is a commonly used technique in neural networks, often used in neural network models such as natural language processing and CV. It distributes the data of multiple different dimensions of the hidden layers of the input sequence to multiple processing heads for attention operation, calculates the attention matrix of different dimensions of the input sequence in each processing head, and enables the neural network model to accurately capture the features at different levels in the input matrix, enriching the representation of the neural network model.

[0120] The length of the input sequence and the ability of the model to understand the context are positively correlated. The longer the input sequence, the stronger the ability of the model to understand the context. As the length of the input sequence increases, the size of the matrix obtained after thread transformation also increases, and the amount of data to be calculated in the multi-head attention calculation also increases in a large scale, which makes the multi-head attention processing inefficient. In order to improve the efficiency of multi-head attention processing, in the related art, the input sequence is often divided into multiple subsequences, and the subsequences are distributed to multiple processing units for pre-linear transformation, and then the multi-head attention calculation is performed in parallel by multiple processing units. This way improves the efficiency of multi-head attention processing. The steps of attention calculation are performed by processing heads in the processing unit (there are multiple processing heads in one processing unit). The data to be processed by each processing head is actually the data of a part of the hidden layer of the input sequence. At this time, the data in each processing unit is actually a matrix block obtained by pre-linear transformation of the subsequence, which reflects all the hidden layer data of a subsequence instead of a part of the hidden layer dimension of the input sequence. Therefore, before calculating the attention, sub-block transformation needs to be performed between processing units to transform the data in each processing unit into a new transformation block shared by all processing heads in one processing unit, and then the attention calculation is performed by the processing heads in the processing unit. As known from the above, in the prior art, the transformation into a new transformation block shared by all processing heads needs to be completed first, and then the attention calculation can be started, which makes the multi-head attention processing inefficient.

[0121] System architecture and scenario to which the embodiments of the present disclosure are applied

[0122] Figure 1 is a system architecture diagram to which the prompt speech posture generation method according to the embodiments of the present disclosure is applied. It includes an object terminal 110, an Internet 120, a gateway 130, and a server 140.

[0123] The object terminal 110 is a device used by an object to input target content and generate a corresponding input matrix based on the target content, so that the server 140 performs multi-head attention processing on the input matrix to generate. It includes desktop computers, laptops, PDAs (personal digital assistants), mobile phones, vehicle-mounted terminals, home theater terminals, special-purpose terminals, and various forms. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network and share a display device for collaborative work to jointly constitute a terminal. The object terminal 110 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0124] The gateway 130 is also called an internetworking connector or a protocol converter. The gateway 130 implements network interconnection at the transport layer and is a computer system or device that acts as a conversion role. The gateway 130 is a translator between two systems that use different communication protocols, data formats or languages, or even have completely different architectures. At the same time, the gateway 130 can also provide filtering and security functions. The message sent by the object terminal 110 to the server 140 is sent to the corresponding server 140 through the gateway 130. The message sent by the server 140 to the object terminal 110 is also sent to the corresponding object terminal 110 through the gateway 130.

[0125] The server 140 refers to a computer system capable of providing multi-head attention processing services to the object terminal 110. Compared with the object terminal 110, the server 140 has higher requirements in stability, security, performance, etc. The server 140 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc. The server 140 can also communicate with the Internet 120 through wired or wireless means to exchange data.

[0126] The embodiments of the present disclosure can be applied in various scenarios, such as Figures 2A-2F the artificial intelligence assistant scenario shown in Figures 3A-3B the machine translation scenario shown in

[0127] (I) Artificial intelligence assistant scenario.

[0128] The artificial intelligence assistant refers to an application capable of receiving an execution input instruction of an object, processing the execution input instruction of the object through a neural network model, and executing a corresponding task. These tasks can be intelligent question answering, document summarization, etc. In order to enable the artificial intelligence assistant to accurately understand and execute these tasks, the artificial intelligence assistant needs to calculate multi-head attention based on a longer sequence. For example, when executing the task of intelligent question answering, the question input by the object often has strong coherence. If the artificial intelligence assistant only processes the question input by the object at present, it may not be able to accurately understand the question of the object, especially when the question input by the object at present includes some nouns with the same name in different professional fields. If the artificial intelligence assistant only identifies the question input by the object at present, it may produce ambiguity and output an incorrect answer. In order to enable the artificial intelligence assistant to more accurately understand the question input by the object at present, the question input by the object at present and other questions input by the object before, and the answers generated by the artificial intelligence assistant based on the other questions input by the object before need to be processed together. At this time, the sequence processed by the artificial intelligence assistant is relatively long. In the scene of document summarization, etc., the document that needs to be summarized is often a relatively long text. The artificial intelligence assistant needs to process the entire document at one time to accurately generate the summary of the document. The input sequence based on the entire document is also relatively long. At this time, by using the multi-head attention processing method of the present disclosure, the input sequence is processed, the corresponding multi-head attention can be quickly calculated based on the input sequence, and the efficiency of multi-head attention processing is improved.

[0129] With reference to Figure 2A , the object inputs a document named XXXX.pdf to the artificial intelligence assistant, and inputs an instruction to enable the artificial intelligence assistant to generate a summary of the document. Then, with reference to Figure 2B, the artificial intelligence assistant transmits the XXXX.pdf and the instructions of the object to the processing device, the processing device generates a corresponding input matrix based on the text content in the document and the instructions input by the object, then divides the input matrix into multiple sub-blocks and distributes the multiple sub-blocks to different processing units respectively for pre-transformation, then performs data transmission between the multiple processing units in the processing device, transforms the first transformed sub-block in each processing unit, and performs multi-head attention calculation in the multiple processing units in parallel, thereby improving the multi-head attention calculation efficiency. During the transformation process, the part corresponding to each processing head in the first transformed sub-block is transformed by sub-blocks in sequence, and when the part corresponding to a processing head is transformed and a second transformed sub-block corresponding to the processing head is obtained, the second transformed sub-block is calculated by the processing head, and the part corresponding to the next processing head in the first transformed sub-block is transformed by sub-blocks. In this way, in each processing unit, the process of multi-head attention calculation by different processing heads and the process of sub-block transformation are also executed in parallel, thereby reducing the total time required for multi-head attention processing in each processing unit and improving the efficiency of multi-head attention processing. Finally, the first attention result sub-block obtained by the processing head of the multiple processing units is inversely transformed by sub-blocks to obtain the second attention result sub-block corresponding to each processing unit, and the second attention result sub-block is spliced to obtain a multi-head attention result matrix, which is then processed by other parts of the model and converted into a corresponding summary text and transmitted to the terminal interface of the artificial intelligence assistant. Therefore, through the multi-head attention processing method of the present disclosure, the artificial intelligence assistant can quickly complete the generation of the document summary instructions input by the object.

[0130] With reference to Figure 2C The artificial intelligence assistant receives the document summary content output by the processing device from the processing device and displays it in the terminal interface of the object. The object can view the summary of XXXX.pdf output by the artificial intelligence assistant through the terminal interface. The document summary indicates that XXXX.pdf "mainly introduces an algorithm of a multi-head attention mechanism proposed by XX company. The algorithm aims to solve the problem of low calculation efficiency of the traditional multi-head attention mechanism when facing long sequences. The core design of the algorithm is XXXXXX, and the result shows that compared with existing methods, the algorithm proposed by XX company can effectively improve the calculation efficiency of the multi-head attention mechanism."

[0131] With reference to Figure 2D After the document summary of XXXX.pdf is performed, the object wants to know how the algorithm in the document is implemented, so the object inputs the question "how is the algorithm mentioned in the above document implemented?" in the input box of the artificial intelligence assistant and sends it to the artificial intelligence assistant.

[0132] With reference toFigure 2E The artificial intelligence assistant transmits the current question input by the object and the previously input XXXX.pdf as context to the processing device, the processing device generates a corresponding input matrix based on the current question input by the object and the XXXX.pdf as an input sequence, performs multi-head attention calculation based on the input matrix through the multi-head attention processing method proposed in the disclosure to obtain a multi-head attention result matrix, and then inputs the multi-head attention result matrix into other modules in the neural network model for post-processing to convert a corresponding answer text and feed back to the terminal of the object, so that the artificial intelligence assistant can quickly understand the context input by the object and generate a corresponding answer.

[0133] Referring to Figure 2F The artificial intelligence assistant receives the text content returned by the processing device and displays it in the application interface of the artificial intelligence assistant, thereby answering the question input by the object.

[0134] (II) Machine translation scenario.

[0135] The initial machine translation is to convert the input sentence from one language to another language through fixed mapping rules. This translation is to mechanically translate each word in the sentence into another language and reassemble it. When facing long sentences with relatively complex grammar, the translation effect is poor, and the translated sentence is very unnatural. With the development of artificial intelligence technology, neural network models are applied to machine translation, which can achieve good translation effect. The multi-head attention mechanism allows the model to simultaneously capture the corresponding relationship between the source language and the target language of the input sentence from multiple dimensions, improving the quality of translation. However, when the input text is long, the time required for calculating multi-head attention is also long, and the efficiency is low. Through the multi-head attention processing method of the disclosure, the efficiency of multi-head attention processing can be improved, so that the model can quickly translate the input sentence from the source language to the target language.

[0136] Referring to Figure 3A The object inputs a long text of “Mr. Chen was a prominent mathematician renowned for his significant contributions to number theory, ……” in the input box of the machine translation assistant of the terminal, and asks the machine translation assistant to translate this text from English to Chinese.

[0137] Referring to Figure 3B, after the machine translation assistant receives the text input by the object from the input box of the terminal, the machine translation assistant sends the text to the processing device, the processing device generates a corresponding input matrix based on the text, at this time, based on the multi-head attention processing method of the present disclosure, the input matrix is further divided into multiple sub-blocks and input into each processing unit of the processing device, the processing unit performs linear transformation on the received sub-blocks to generate a corresponding first transformed sub-block, the first transformed sub-block includes a part corresponding to each processing head in the processing unit, then, between multiple processing units, the part corresponding to each processing head in the first transformed sub-block is sequentially subjected to sub-block transformation, after completing the sub-block transformation of the part corresponding to the current processing head, a second transformed sub-block corresponding to the current processing head is obtained, and the second transformed sub-block corresponding to the current processing head is subjected to attention calculation, while the part corresponding to the next processing head of the current processing head in the first transformed sub-block is subjected to sub-block transformation. In this way, the processing heads in the processing unit do not need to wait for the completion of the sub-block transformation of the entire first transformed sub-block before starting to calculate the multi-head attention. The process of multi-head attention calculation and the process of sub-block transformation performed by different processing heads are also concurrently executed, thereby shortening the total time length of multi-head attention processing and improving the efficiency of multi-head attention processing. When the processing heads in each processing unit calculate the corresponding first attention result sub-block, the first attention result sub-block is subjected to sub-block inverse transformation, and the results of the inverse transformation of multiple processing heads of the same processing unit are spliced to obtain a second attention result sub-block corresponding to the processing unit, and finally, the second attention result sub-blocks corresponding to each processing unit are spliced to obtain a multi-head attention result matrix. Finally, through the processing of other modules in the model, the multi-head attention result matrix is converted into a corresponding translation text and transmitted to the terminal of the object, and the corresponding translation text is output in the terminal of the object, such as Figure 3B in the interface of the terminal of the object, the translation text "Mr. Chen is an outstanding mathematician known for his significant contributions to the field of number theory, especially …" is displayed.

[0138] It should be understood that the above only illustrates the description of part of the application scenarios of the present disclosure. The business scenarios to which the present disclosure can be applied can include but are not limited to the specific embodiments mentioned above.

[0139] General description of embodiments of the present disclosure

[0140] It should be emphasized that the embodiments of the present disclosure can be applied to various application scenarios, such as Figures 2A-2F the artificial intelligence assistant scenario shown in Figures 3A-3B the machine translation scenario shown in, and can also be applied to other business scenarios involving multi-head attention calculation, such as sentence sentiment analysis, pre-training language models, etc.

[0141] The multi-head attention processing method of the embodiments of the present disclosure can be executed on the object terminal 110, can be executed on the server 140, or can be partially executed on the object terminal 110 and partially executed on the server 140.

[0142] As shown in Figure 4 According to one embodiment of the present disclosure, the multi-head attention processing method includes the following steps:

[0143] In step 410, the input matrix is divided into a first number of sub-blocks, and the first number of sub-blocks are respectively input to the first number of processing units to perform first linear transformation, to obtain a first number of first transformed sub-blocks corresponding to the first number of processing units;

[0144] In step 420, sub-block transformation is performed on the first number of first transformed sub-blocks corresponding to the first number of processing units, to sequentially obtain a second transformed sub-block corresponding to each processing head, and the second transformed sub-block is input to the processing head corresponding to the second transformed sub-block to perform attention calculation, to obtain a first attention result sub-block corresponding to each processing head, wherein the second number of processing heads are in order, and while performing attention calculation on the second transformed sub-block corresponding to the current processing head in the second number of processing heads, sub-block transformation of the next processing head of the current processing head is performed;

[0145] In step 430, sub-block inverse transformation is performed on the first attention result sub-block corresponding to each processing head, to obtain a second attention result sub-block corresponding to each processing unit, and based on the second attention result sub-block corresponding to each processing unit, a multi-head attention result matrix is generated.

[0146] The multi-head attention processing method proposed in the present disclosure will be described below.

[0147] The processing unit is a module used in neural network model training or inference, such as a graphics processing unit (GPU), and each processing unit can work in parallel. The calculation and bandwidth of different processing units are independent of each other. The processing device is a cluster device composed of multiple processing units. It can be understood that the multiple processing units in the processing device can communicate with each other to transmit data, so that the multiple processing units can cooperatively complete the multi-head attention processing method proposed in the present disclosure. In the present disclosure, the processing device is composed of a first number of processing units. The processing head is a basic processing unit in the processing unit for performing multi-head attention calculation. The processing head can also be called an attention head. Multiple processing heads running in one processing unit share the computing resources of the processing unit. Multiple processing heads can independently perform attention calculation to calculate the attention weight of the input sequence in different hidden layer dimensions. In the present disclosure, each processing unit includes a second number of processing heads.

[0148] In step 410, the input matrix is a matrix generated based on an input sequence, and each element in the input sequence is processed by an embedding layer before being input into the multi-head attention mechanism, so as to convert each element in the input sequence into a corresponding vector. After the vectors corresponding to the elements in the input sequence are spliced based on the positions of the elements in the input sequence, an input matrix corresponding to the input sequence is obtained. The row dimension of the input matrix represents the length of the input sequence, and each row of the input matrix corresponds to an element in the input sequence. The column dimension of the input matrix represents the number of hidden layers of the input sequence, and each hidden layer corresponds to a feature of the input sequence at a level. For example, an input sequence with a length of N and d hidden layers is processed by the embedding layer, each element in the input sequence is converted into a d-dimensional vector, and then the vectors of the elements are spliced to obtain a matrix with a shape of (N, d). The matrix is the input matrix corresponding to the input sequence. The sub-block is a matrix block obtained by dividing the input matrix by rows. Specifically, the input matrix is divided by rows to obtain a matrix block. For example, the input matrix is an N-row d-column matrix, i.e., the shape of the input matrix is (N, d). The first number is P, and the sub-block is obtained by dividing the N rows of the input matrix into P parts. Each sub-block has N / P rows and d columns. It can be understood that in other embodiments, the input matrix can be divided into a first number of sub-blocks in other ways. The specific division method will be described later, and will not be described here.

[0149] In this embodiment, after the input matrix is divided into a first number of sub-blocks, each sub-block is input into a processing unit. In each processing unit, the received sub-block is subjected to first linear transformation. By subjecting the sub-blocks to first linear transformation respectively, a sub-query matrix, a sub-key matrix and a sub-value matrix corresponding to the sub-blocks are obtained. The sub-query matrix, the sub-key matrix and the sub-value matrix are matrices required for calculating attention corresponding to the sub-blocks. The sub-query matrix, the sub-key matrix and the sub-value matrix corresponding to each sub-block constitute a first transformed sub-block in the processing unit corresponding to the sub-block.

[0150] Referring to Figure 5 In some related technologies, the entire input matrix is subjected to first linear transformation in a processing unit to obtain a query matrix, a key matrix and a value matrix corresponding to the complete input matrix, and then the three matrices are divided into processing heads for multi-head attention calculation. However, this multi-head attention processing method can only call the computing power of a single processing unit for first linear transformation, and the subsequent multi-head attention calculation is also performed in the processing heads of a single computing unit, so the processing efficiency is not high.

[0151] In the embodiment, the input matrix is divided into a first number of sub-blocks and sent to each processing unit, each processing unit only needs to perform the first linear transformation on one sub-block, and the first number of processing units concurrently perform the first linear transformation on the first number of sub-blocks, thereby improving the efficiency of the first linear transformation on the input matrix and the efficiency of the multi-head attention processing. The detailed process of the first linear transformation will be described later, and will not be described here.

[0152] It should be noted that the sub-query matrix, the sub-key matrix and the sub-value matrix constituting the first transformed sub-block corresponding to the processing unit are independently stored in the processing unit. In the embodiment of the present disclosure, the processing process for the first transformed sub-block is to process the sub-query matrix, the sub-key matrix and the sub-value matrix respectively. For example, in step 420, the sub-block transformation is performed on the first transformed sub-block, and the sub-query matrix, the sub-key matrix and the sub-value matrix are transformed respectively in the process of the sub-block transformation.

[0153] In step 420, the second transformed sub-block is a matrix block corresponding to a plurality of hidden layers of the input sequence, i.e., a matrix block obtained by performing the first linear transformation on a plurality of matrix columns of the input matrix. For example, the input sequence is a sequence with a length of N and d hidden layers, and the corresponding input matrix is a matrix with N rows and d columns. After performing the first linear transformation on the entire input matrix, the query matrix, the key matrix and the value matrix obtained are all matrices with N rows and d columns. A set of matrix blocks composed of a plurality of matrix columns in the query matrix, the key matrix and the value matrix is the second transformed sub-block. For example, a processing head needs to calculate the attention weight of the first four hidden layer dimensions of the input sequence, the first 4 columns in the query matrix, the first 4 columns in the key matrix and the first 4 columns in the value matrix respectively constitute the matrix block of the query matrix, the matrix block of the key matrix and the matrix block of the value matrix, and the set of these matrix blocks is the second transformed sub-block corresponding to the processing head. It should be noted that the matrix block corresponding to the query matrix, the matrix block corresponding to the key matrix and the matrix block corresponding to the value matrix in the second transformed sub-block are independently stored.

[0154] The current processing head is one of the second number of processing heads of the processing unit. Specifically, in the present disclosure, when performing the sub-block transformation processing on the first transformed sub-block, the first transformed sub-block is divided into a plurality of parts, each part corresponding to each processing head in the processing unit, and then the part corresponding to each processing head in the first transformed sub-block is sequentially processed by the sub-block transformation processing. The current processing head is the processing head corresponding to the part in the first transformed sub-block which is currently processed by the sub-block transformation processing.

[0155] The first attention result sub-block is an attention weight matrix corresponding to the partial hidden layer of the input sequence. Specifically, in the multi-head attention mechanism, the weight matrix of the input sequence in multiple hidden layer dimensions needs to be calculated to capture the features of the input sequence at different levels. Each hidden layer of the input sequence is allocated to multiple processing heads for attention calculation. Each processing head calculates the attention weight of the input sequence in several hidden layer dimensions, and these attention weights constitute the first attention result sub-block corresponding to the processing head.

[0156] It can be understood that when calculating multi-head attention, each processing head actually needs to calculate the attention weight of the entire input sequence in the partial hidden layer dimension, that is, each processing head actually needs to calculate the attention weight of the corresponding hidden layer dimension of the input sequence based on the query matrix, the key matrix and the value matrix corresponding to the partial hidden layer of the input sequence. In step 410, the input matrix is divided into multiple sub-blocks and then sent to each processing unit for first linear transformation. The first transformed sub-block obtained in each processing unit actually corresponds to all hidden layers of a sub-sequence of the input sequence, rather than corresponding to the partial hidden layer of the entire input sequence. Therefore, sub-block transformation processing needs to be performed on the first transformed sub-block in each processing unit between the first number of processing units, so as to transform the matrix corresponding to all hidden layers of the sub-sequence in each processing unit into a matrix corresponding to several hidden layers of the entire input sequence. In related technologies, when performing sub-block transformation, the entire first transformed sub-block in the processing unit is subjected to sub-block transformation processing at one time to obtain a matrix block corresponding to all processing heads in each processing unit, and then attention calculation is performed through each processing head in the processing unit. This makes the process of calculating attention wait for the completion of the sub-block transformation of the entire first transformed sub-block in the processing unit before attention calculation is performed through the processing head, which makes the multi-head attention processing inefficient. In the present disclosure, when performing sub-block transformation on each first transformed sub-block between the first number of processing units, the first transformed sub-block is segmented for sub-block transformation. Specifically, the first transformed sub-block in the processing unit is divided into a part corresponding to each processing head in the processing unit, and then the part corresponding to each processing head is subjected to sub-block transformation processing. After the sub-block transformation of the part corresponding to the current processing head is completed, the second transformed sub-block corresponding to the current processing head is obtained. This second transformed sub-block is a matrix block corresponding to several matrixes in the input matrix after the first linear transformation. The current processing head can perform attention calculation based on the corresponding second transformed sub-block to obtain a first attention result sub-block corresponding to the current processing head. While the current processing head performs attention calculation, the part corresponding to the next processing head of the current processing head in the first transformed sub-block is subjected to sub-block transformation processing. In this way, the sub-block transformation process and the attention calculation process corresponding to multiple processing heads in the processing unit are performed in parallel. Before the sub-block transformation processing of the entire first transformed sub-block is completed, the attention calculation process of part of the processing heads can be performed. Therefore, the total time required for multi-head attention processing in each processing unit can be reduced, and the efficiency of multi-head attention processing is improved. The detailed process of sub-block transformation of the first transformed sub-block and multi-head attention calculation of the processing head will be described later, and will not be described here.

[0157] In step 430, the second attention result sub-block is an attention weight matrix block corresponding to all hidden layer dimensions of the sub-block input to the processing unit. For example, if the input sequence has a length of N, there are d hidden layers, and there are P processing units in the processing device, the shape of the corresponding input matrix is (N, d), that is, the input matrix has N matrix rows and d matrix columns. After the input matrix is evenly divided and input to each processing unit, each processing unit obtains a sub-block that is a (N / P, d) matrix, that is, a matrix corresponding to all hidden layers of N / P elements in the input sequence. Correspondingly, the final output result of the processing unit should be an attention weight matrix corresponding to all hidden layer dimensions of the N / P elements of the sub-block. Therefore, the second attention result sub-block corresponding to the processing unit is an attention weight matrix corresponding to all hidden layer dimensions of the N / P elements in the input sequence.

[0158] The multi-head attention result matrix is an attention weight result corresponding to all hidden layer dimensions of the input matrix. The multi-head attention result matrix represents the features of the input sequence at multiple different levels and can capture the importance of each part of the input sequence from multiple different dimensions.

[0159] It can be understood that in step 410, the input matrix is divided into a first number of sub-blocks and then input to each processing unit. When the results calculated by the multiple processing units are spliced, the results of the multiple processing units are spliced based on the positions of the sub-blocks corresponding to the processing units in the input matrix. That is, the final output result of each processing unit should be an attention weight matrix corresponding to all hidden layers of the sub-block input to the processing unit. The first attention result sub-block calculated by each processing head in the processing unit is an attention weight matrix corresponding to several hidden layers of the entire input sequence. Based on this, after the corresponding first attention result sub-block is calculated by each processing head, the first attention result sub-block corresponding to each processing head needs to be sub-block inverse transformed between the first number of processing units. The first attention result sub-block is transformed into a second attention result sub-block, so that the output of each processing unit is changed from an attention weight matrix corresponding to several hidden layers of the input sequence to an attention weight matrix corresponding to all hidden layer dimensions of several elements in the input sequence. The detailed process of sub-block inverse transformation will be described later and will not be described here.

[0160] After the sub-block inverse transformation is completed, the second attention result sub-block corresponding to each processing unit is obtained, and after the second attention result sub-blocks corresponding to the processing units are spliced in sequence, the multi-head attention weight matrix corresponding to all hidden layer dimensions of the entire input matrix is obtained. Specifically, after the sub-block inverse transformation processing, the second attention result sub-block corresponding to each processing unit is a matrix corresponding to the attention weight of all hidden layer dimensions of the sub-block input into the processing unit, and the second attention result sub-blocks corresponding to the processing units are spliced according to the positions of the sub-blocks corresponding to the processing units in the input matrix, so that the multi-head attention result matrix corresponding to all hidden layer dimensions of the entire input matrix is obtained.

[0161] In the embodiments disclosed in steps 410 to 430, by dividing the input matrix into multiple sub-blocks and distributing them into multiple processing units, performing first linear transformation on each sub-block concurrently through multiple processing, then performing sub-block transformation processing on the first transformed sub-blocks between multiple processing units, transforming the first transformed sub-blocks in the processing units into matrix blocks required for attention calculation by each processing head, and performing multi-head attention calculation on the features of the input matrix at different levels in multiple processing units concurrently, the multi-head attention processing efficiency is improved, and at the same time, when performing sub-block transformation, the parts corresponding to each processing head in the first transformed sub-blocks are transformed in sequence, rather than performing sub-block transformation on the entire first transformed sub-block at one time, and after the sub-block transformation of the part corresponding to the current processing head in the first transformed sub-block is completed, attention calculation is performed through the current processing head, while the part corresponding to the next processing head of the current processing head in the first transformed sub-block is transformed, thereby making the sub-block transformation and attention calculation corresponding to different processing heads executed in parallel, thereby reducing the total time length of multi-head attention processing of each processing unit and improving the efficiency of multi-head attention processing.

[0162] In one embodiment, the attention weights of different hidden layer dimensions represent the features of the input sequence captured by the model from different dimensions, and if the second attention result sub-blocks of the processing units are simply spliced and the spliced result is taken as the multi-head attention result matrix, then the importance of the input sequence at multiple different levels of features is the same, however, the features of the input sequence at different levels of features contribute differently to the understanding of the semantic of the input sequence by the model, which highlights the relatively important level of features in the multiple hidden layer dimensions of the input sequence. Based on this, the spliced result of the second attention result sub-blocks needs to be modified, specifically, referring to Figure 6 In step 430, the multi-head attention result matrix is generated based on the second attention result sub-blocks corresponding to the processing units, including:

[0163] Step 610, the second attention result sub-block corresponding to each processing unit is aggregated into a to-be-corrected matrix;

[0164] Step 620, based on the to-be-corrected matrix and the correction coefficient matrix, a multi-head attention result matrix is generated.

[0165] In step 610, the to-be-corrected matrix is a matrix obtained by splicing the second attention result sub-block corresponding to each processing unit based on the position of the sub-block corresponding to each processing unit in the input matrix. Specifically, since the second attention result sub-block corresponding to each processing unit is an attention weight matrix corresponding to a sub-block composed of a plurality of matrix rows of the input matrix, that is, each second attention result sub-block corresponds to a plurality of matrix rows in the input matrix, when splicing the second attention result sub-block, the second attention result sub-block corresponding to each processing unit is spliced along the dimension of the matrix row. For example, in P processing units, the second attention result sub-block corresponding to each processing unit is an N / P row d column matrix. Based on the position of the sub-block corresponding to each processing unit in the input matrix, the splicing order of the second attention result sub-block of the P processing units is determined. For example, the sub-block corresponding to processing unit 1 is the first N / P matrix row in the processing unit, the sub-block corresponding to processing unit 2 is the N / P+1 to 2*N / P matrix row in the input matrix, and so on. The sub-block corresponding to processing unit P is the last N / P matrix row in the input matrix. Then the splicing order is from processing unit 1 to processing unit P, and then the second attention result sub-block corresponding to processing unit 1 to the second attention result sub-block corresponding to processing unit P is spliced along the dimension of the matrix row to obtain a matrix with N matrix rows and d matrix columns, that is, the to-be-corrected matrix.

[0166] In step 620, the correction coefficient matrix is used to correct the to-be-corrected matrix to highlight the coefficients of the features of the relatively important levels in the input sequence. Specifically, the parameters in the correction coefficient matrix are determined in the training process of the neural network model, which will not be described here. By performing matrix multiplication on the to-be-corrected matrix and the correction coefficient matrix, a multi-head attention result matrix can be obtained. By performing linear transformation on the to-be-corrected matrix in sequence, the features of the input sequence at different levels can be better integrated, and the features of the relatively important levels in the neural network model understanding the input sequence can be highlighted, so that the model can more accurately understand the input sequence. It should be noted that the number of matrix rows and matrix columns of the correction coefficient matrix is consistent with the number of hidden layers of the input matrix, so that the shape of the attention result matrix obtained after correction is consistent with that of the to-be-corrected matrix, that is, they have the same number of matrix rows and matrix columns. For example, the number of hidden layers of the input matrix is d, and the correction coefficient matrix is a d row d column matrix.

[0167] In the embodiments disclosed in steps 610 to 620, after the second transformed sub-blocks corresponding to the respective processing units are spliced, the obtained to-be-corrected matrix after splicing is corrected by the correction coefficient matrix, so as to better integrate the features of the input sequence at different levels and highlight the relatively important features at different levels, so that the model can more accurately understand the input sequence.

[0168] The above is a general description of steps 410 to 430. The implementation of steps 410 to 430 will be described in detail below.

[0169] Detailed description of step 410.

[0170] In step 410, the input matrix is divided into a first number of sub-blocks, and the first number of sub-blocks are respectively assigned to the first number of processing units to perform first linear transformation, to obtain a first number of first transformed sub-blocks corresponding to the first number of processing units.

[0171] The input matrix has a first dimension and a second dimension, wherein the first dimension is the matrix row dimension of the input matrix, and the second dimension is the matrix column dimension of the input matrix. It can be understood that each matrix row of the input matrix corresponds to an element in the input sequence, and each data in the same matrix row corresponds to a feature at different levels of a certain element in the input sequence, and the matrix column of the input matrix corresponds to a feature at a level of the input sequence, that is, the data in a matrix column of the input matrix corresponds to the features at a certain level of each element in the input sequence. When performing first linear transformation on the input matrix to generate the query matrix, the key matrix and the value matrix corresponding to the input matrix, the input matrix is multiplied by each corresponding weight matrix in the multi-head attention mechanism, and the data of each matrix row of the input matrix is respectively dot product summed with the data of each matrix column of the corresponding weight matrix. Based on this, when the input matrix is divided into a plurality of sub-blocks, it is necessary to maintain the integrity of each matrix row of the input matrix. In an embodiment, referring to Figure 7 , step 410 includes:

[0172] Step 710, dividing the input matrix into a first number of sub-blocks in the first dimension;

[0173] Step 720, respectively assigning the first number of sub-blocks to the first number of processing units to perform first linear transformation, to obtain a first number of transformed sub-blocks corresponding to the first number of processing units.

[0174] In step 710, the input matrix is split along the first dimension, and several matrix rows of the input matrix are taken as a sub-block, so that the data of each matrix row of the input matrix is complete. Specifically, the splitting manner can be that the input matrix is evenly divided into a first number of sub-blocks along the first dimension. Specifically, refer to Figure 8 The input matrix includes N matrix rows, and the processing device has P processing units, that is, the first number is P. After the N matrix rows of the input matrix are evenly divided into P parts, P sub-blocks with N / P matrix rows are obtained. Then, the P sub-blocks are respectively sent to the respective processing units. Each processing unit only needs to perform the first linear transformation on the sub-block with N / P matrix rows. It can be understood that in some other embodiments, the input matrix can also be split into a first number of sub-blocks along the first dimension based on other rules.

[0175] In step 720, after the input matrix is split into a first number of sub-blocks, each sub-block is respectively sent to a different processing unit. Each processing unit receives a sub-block. Then, the respective processing units concurrently perform the first linear transformation on the received sub-block. Refer to Figure 10 The process of performing the first linear transformation by the processing unit includes:

[0176] In step 1010, the query weight matrix, the key weight matrix, and the value weight matrix corresponding to the processing unit are determined.

[0177] In step 1020, the sub-block corresponding to the processing unit is respectively multiplied by the query weight matrix, the key weight matrix, and the value weight matrix corresponding to the processing unit to obtain a sub-query matrix, a sub-key matrix, and a sub-value matrix corresponding to the sub-block, wherein the sub-query matrix, the sub-key matrix, and the sub-value matrix constitute a first transformed sub-block corresponding to the processing unit.

[0178] In step 1010, the query weight matrix, the key weight matrix, and the value weight matrix are respectively three weight matrices for converting the sub-block into a query matrix, a key matrix, and a value matrix required for calculating attention. The three weight matrices are parameters of the multi-head attention mechanism and are determined in the process of training the neural network model based on the multi-head attention mechanism and are fixed in the respective processing units. Details are not described herein. It can be understood that since the input matrix is split into a first number of sub-blocks and then sent to the respective processing units, and the first linear transformation on the sub-block is independently performed by the respective processing units, since the difference between each sub-block is only that each sub-block corresponds to a different element in the input sequence, the manner of performing the first linear transformation on each element of the input sequence should be consistent. Based on this, the query weight matrix, the key weight matrix, and the value weight matrix corresponding to each processing unit are the same.

[0179] It should be noted that in order to ensure that each data in the transformed result after the first linear transformation of the sub-block can correspond to each data in the sub-block, the shape of the sub-block before and after the transformation should be kept unchanged, and based on this, the query weight matrix, the key weight matrix and the value weight matrix are all d-order square matrices, where d is the number of hidden layers of the sub-block, that is, the number of matrix arrays of the sub-block. For example, each sub-block has 128 matrix arrays, so the query weight matrix, the key weight matrix and the value weight matrix are all square matrices with 128 matrix rows and 128 matrix arrays.

[0180] In step 1020, referring to Figure 8 After the input matrix is divided into a first number of sub-blocks and sent to each processing unit respectively, the processing unit 1 obtains a sub-block with a shape of (N / P, d), and performs matrix multiplication on the sub-block with the query weight matrix, the key weight matrix and the value weight matrix in the processing unit 1 respectively to obtain a sub-query matrix, a sub-key matrix and a sub-value matrix with a shape of (N / P, d), and the three matrices constitute the first transformed sub-block corresponding to the processing unit 1. Specifically, the process of the first linear transformation can refer to the following formula:

[0181] Q=X*W Q ;

[0182] K=X*W K ;

[0183] V=X*W V ;

[0184] Wherein, X is the sub-block corresponding to the processing unit, W Q , W K , W V are the query weight matrix, the key weight matrix and the value weight matrix in turn, and Q, K, V are the sub-query matrix, the sub-key matrix and the sub-value matrix corresponding to the sub-block in turn.

[0185] In the embodiments disclosed in steps 1010 to 1020, by performing matrix multiplication operation on the corresponding sub-block and the query weight matrix, the key weight matrix and the value weight matrix in each processing unit respectively, the first linear transformation of the sub-block is performed to obtain the sub-query matrix, the sub-key matrix and the sub-value matrix corresponding to the sub-block. In this way, each processing unit only needs to process the first linear transformation of one sub-block, and multiple processing units can concurrently perform the first linear transformation on multiple sub-blocks, thereby improving the efficiency of the first linear transformation of the input matrix.

[0186] In one embodiment, referring to Figure 11 , step 710 comprises:

[0187] Step 1110, determining the amount of communication resources of each processing unit;

[0188] At step 1120, the input matrix is split into a first number of sub-blocks along the first dimension based on the ratio of the communication resource amount of each processing unit.

[0189] At step 1110, the communication resource amount refers to the amount of data that can be transmitted and received by the processing unit in a unit of time, and specifically, the communication resource amount of the processing unit can be determined based on the bandwidth of the processing unit.

[0190] At step 1120, it can be understood that after each processing unit performs the first linear transformation on the sub-block to obtain the corresponding first transformed sub-block, since each first transformed sub-block corresponds to all hidden layers of a part of elements in the input sequence, and when performing attention calculation, each processing head actually performs calculation based on a plurality of hidden layers of all elements in the entire input sequence, therefore, before performing multi-head attention calculation, the sub-block transformation processing needs to be performed on each first transformed sub-block between the processing units, the matrix array corresponding to a plurality of hidden layers in the first transformed sub-block of each processing unit is sent to other processing units, and a plurality of hidden layer corresponding matrix arrays of other first transformed sub-blocks are received from other processing units, so as to transform the first transformed sub-block corresponding to all hidden layers of the sub-block in the processing unit into the second transformed sub-block corresponding to part of the hidden layers of the entire input matrix, and this process involves data transmission between processing units. At this time, if the communication resource amount of each processing unit is different, the amount of data that can be transmitted and received in a unit of time is different, and when performing sub-block transformation processing on the first transformed sub-block, the total amount of data that needs to be transmitted and received by the processing unit is proportional to the size of the data amount of the first transformed sub-block, that is, proportional to the size of the sub-block, at this time, if the data matrix is evenly divided into each processing unit, the size of the first transformed sub-block corresponding to each processing unit is also the same, and accordingly, when performing sub-block transformation, the amount of data that needs to be transmitted and received by each processing unit is the same, for the processing unit with less communication resource amount, the time required for completing data transmission and reception is relatively longer, and the processing unit with more communication resource amount also needs to wait for the processing unit with less communication resource amount to send data before receiving the data sent by the processing unit, to obtain the second transformed sub-block corresponding to the processing head, which leads to a longer time required for sub-block transformation, and reduces the efficiency of multi-head attention processing. Based on this, in the present disclosure, the communication resource amount of each processing unit is first determined, and then the input matrix is split into a first number of sub-blocks along the first dimension based on the ratio of the communication resource amount of each processing unit, for example, with reference to Figure 12, the input matrix has 256 matrix rows in the first dimension, the processing device has 4 processing units in total, processing unit 1 to processing unit 4, the communication resource amount of the 4 processing units is CR1, CR2, CR3, CR4 in turn, the ratio of the communication resource amount of the four processing units is CR1: CR2: CR3: CR4 = 14: 15: 17: 18, then, according to the ratio of the communication resource amount, the 256 matrix rows of the input matrix are divided into 4 sub-blocks, which are sub-block 1 to sub-block 4 respectively, sub-block 1 to sub-block 4 include 56 matrix rows, 60 matrix rows, 68 matrix rows and 72 matrix rows in turn, each sub-block obtained by the division corresponds to a processing unit, the sub-block divided based on the proportion of the communication resource amount of a certain processing unit corresponds to the processing unit, for example, sub-block 1 to sub-block 4 correspond to processing unit 1 to processing unit 4 in turn, then, sub-block 1 to sub-block 4 are sent to processing unit 1 to processing unit 4 in turn, the ratio of the size of the sub-block obtained by each processing unit is consistent with the ratio of the communication resource amount of each processing unit, and the ratio of the size of the corresponding first transformed sub-block is also consistent with the ratio of the communication resource amount of each processing unit, so that the first transformed sub-block corresponding to the processing unit with more communication resource amount is larger in size, and the processing unit needs to send relatively more data when performing sub-block transformation on the first transformed sub-block, while the first transformed sub-block corresponding to the processing unit with less communication resource amount is relatively smaller in size, and the processing unit needs to send relatively less data. In this way, the amount of data that each processing unit needs to send is matched with the communication resource amount of the processing unit, and the first number of processing units can relatively synchronously complete sub-block transformation.

[0191] In the embodiments disclosed in steps 1110 to 1120, by obtaining the ratio of the communication resource amount of each processing unit, the input matrix is divided into a first number of sub-blocks along the first dimension based on the ratio of the communication resource amount of each processing unit and sent to the corresponding processing unit respectively, after performing the first linear transformation, the size of the first transformed sub-block corresponding to each processing unit matches the communication resource amount of the processing unit, and the amount of data that each processing unit needs to send matches the communication resource amount when performing sub-block transformation on the first transformed sub-block, so that multiple processing units can relatively synchronously complete sub-block transformation, thereby reducing the time required for sub-block transformation and improving the efficiency of multi-head attention processing.

[0192] In another embodiment, the input matrix can also be divided in other ways, for example, the input matrix can be divided into a first number of sub-blocks along the first dimension based on the ratio of the calculation resource amount of each processing unit, and the first number of sub-blocks are sent to the corresponding processing unit respectively. Figure 13 Step 710 includes:

[0193] Step 1310, determining the calculation resource amount of each processing unit;

[0194] At step 1320, the input matrix is divided into the first number of sub-blocks along the first dimension based on the ratio of the computing resource amount of each processing unit.

[0195] At step 1310, the computing resource amount is the amount of data that a processing unit can complete calculation in a unit of time. Specifically, the computing resource amount of a processing unit can be determined based on the computing power of each processing unit. It can be understood that the more the computing resource amount of a processing unit, the more data calculation is performed in a unit of time.

[0196] At step 1320, it can be understood that after each sub-block is sent to a processing unit, the processing unit needs to perform a first linear transformation on the sub-block. The process of the first linear transformation is that the sub-block respectively performs matrix multiplication with the query weight matrix, the key weight matrix and the value weight matrix. Matrix multiplication itself is a process of dot product summation of data of each matrix row of the sub-block and data of each matrix column of the corresponding weight matrix. The larger the size of the sub-block, the more the number of matrix rows, and the larger the amount of calculation when the first linear transformation is performed on the sub-block. At this time, if the input matrix is evenly divided into the first number of sub-blocks along the first dimension, the number of matrix rows of each sub-block is equal, and the amount of calculation when the first linear transformation is performed is equal. At this time, for the processing unit with more computing resource amount, the first linear transformation of the corresponding sub-block can be quickly completed. However, since the processing unit with less computing resource amount has not completed the first linear transformation of the corresponding sub-block at this time, the processing unit with relatively more computing resource amount needs to wait until the processing unit with less computing resource amount completes the first linear transformation of the corresponding sub-block, and then the sub-block transformation is performed among the processing units. At this time, the computing capability of the processing unit with more computing resource amount is wasted, which leads to low efficiency of the first linear transformation. Based on this, in the present embodiment, after the computing resource amount of each processing unit is determined, the input matrix is divided along the first dimension based on the ratio of the computing resource amount of each processing unit. For example, refer to Figure 14, the input matrix includes 256 matrix rows, the processing device has 4 processing units in sequence, processing unit 1 to processing unit 4, the amount of computing resources of the 4 processing units is R1 to R4 in sequence, R1:R2:R3:R4=8:7:11:6, at this time, the input matrix is split along the first dimension based on R1:R2:R3:R4, to obtain 4 subblocks, subblock 1 to subblock 4, the 4 subblocks include 64 matrix rows, 56 matrix rows, 88 matrix rows and 48 matrix rows in the input matrix in sequence, each subblock is one-to-one corresponding to each processing unit, that is, the subblock split based on the amount of computing resources of each processing unit is corresponding to the processing unit, then subblock 1 to subblock 4 are sent to processing unit 1 to processing unit 4 in sequence, at this time, the ratio of the time required for the first linear transformation of the received subblock by processing unit 1 to processing unit 4 is 64 / 8:56 / 7:88 / 11:48 / 6, that is, 1:1:1:1, at this time, the time required for the first linear transformation of the received subblock by each processing unit is consistent, the multiple processing units can synchronously complete the first linear transformation of the subblock, and the processing unit with more computing resources does not need to wait for the processing unit with less computing resources to complete the first linear transformation of the corresponding subblock. In the first linear transformation, the computing resources of each processing unit can be fully utilized, the total time required for performing the first linear transformation on the first number of subblocks is shortened, and the overall efficiency of the first linear transformation is improved.

[0197] In the embodiments disclosed in steps 1310 to 1320, by determining the amount of computing resources of each processing unit, and then splitting the input matrix into a first number of subblocks along the first dimension based on the ratio of the amount of computing resources of each processing unit, the size of each subblock matches the amount of computing resources of each processing unit. After each subblock is sent to the corresponding processing unit, the time required for the first linear transformation of the received subblock by each processing unit is consistent, and the first number of processing units can synchronously complete the first linear transformation of the corresponding subblock, thereby reducing the total time required for the first linear transformation and improving the overall efficiency of the first linear transformation on the first number of subblocks, thereby improving the efficiency of the multi-head attention processing.

[0198] Detailed description of the subblock transformation in step 420.

[0199] The subblock transformation in step 420 refers to performing subblock transformation on the first number of first transformed subblocks corresponding to the first number of processing units.

[0200] In step 420, sub-block transformation is performed on the first number of first transformed sub-blocks corresponding to the first number of processing units, to obtain second transformed sub-blocks corresponding to each processing head in sequence, wherein the second number of processing heads are in order, and while attention calculation is performed on the second transformed sub-block corresponding to a current processing head in the second number of processing heads, sub-block transformation of a next processing head of the current processing head is performed.

[0201] The input matrix has a first dimension and a second dimension, and the first transformed sub-blocks obtained based on the sub-blocks of the input matrix also have the first dimension and the second dimension. The description of the first dimension and the second dimension can refer to the description of the first dimension and the second dimension in the detailed description of step 410 above, and will not be repeated here. In step 410 of the above embodiment, the input matrix is divided into the first number of sub-blocks along the first dimension. At this time, referring to Figure 7 , in step 420, the first number of first transformed sub-blocks corresponding to the first number of processing units are subjected to sub-block transformation to obtain second transformed sub-blocks corresponding to each processing head in sequence, including:

[0202] In step 730, the first transformed sub-blocks are divided into sub-blocks corresponding to each processing head according to the second number in the second dimension;

[0203] In step 740, the sub-blocks are divided into the first number of first patches corresponding to the first number of processing units in the second dimension;

[0204] In step 750, for a single processing head of a single processing unit, the first patches corresponding to the single processing head in each first transformed sub-block are aggregated to obtain a second transformed sub-block corresponding to the single processing head.

[0205] In step 730, the sub-block refers to a partial matrix block in the first transformed sub-block corresponding to one processing head in the processing unit. It can be understood that the second dimension is the hidden layer dimension of the first transformed sub-block, such as the matrix column dimension of the query matrix and the value matrix. Each sub-block obtained by cutting the first transformed sub-block along the second dimension is a matrix block corresponding to a plurality of hidden layers in the first transformed sub-block, respectively. It can be understood that by cutting each first transformed sub-block into a sub-block corresponding to each processing head in the processing unit along the second dimension, each sub-block corresponding to each processing head can be sequentially transformed in the process of sub-block transformation, instead of the whole first transformed sub-block. It can be understood that each processing unit includes a second number of processing heads, each processing head calculates the attention weight corresponding to different hidden layers, and the attention calculation process of each processing head is independent of each other. Based on this, the first transformed sub-block can be cut into a second number of sub-blocks along the second dimension, and each sub-block corresponds to one processing head in the processing unit. The detailed process of cutting the first transformed sub-block is described below, and is not repeated here.

[0206] It should be noted that since the first transformed sub-block actually includes a sub-query matrix, a sub-key matrix and a sub-value matrix corresponding to the sub-block, the process of cutting the first transformed sub-block along the second dimension is actually cutting the sub-query matrix, the sub-key matrix and the sub-value matrix along the second dimension to obtain a sub-query matrix block, a sub-key matrix block and a sub-value matrix block. The sub-query matrix block, the sub-key matrix block and the sub-value matrix block corresponding to the same plurality of hidden layers are used as the sub-block corresponding to the processing head, that is, each processing head corresponds to the sub-block which actually includes the sub-query matrix block, the sub-key matrix block and the sub-value matrix block.

[0207] In step 740, the first patches are matrix patches obtained by further cutting the sub-block corresponding to each processing head along the second dimension, and the first number of first patches obtained by cutting from the same sub-block correspond to the first number of processing units respectively. It can be understood that after the first transformed sub-block is divided into the second number of sub-blocks along the second dimension, each sub-block corresponds to a processing head, but each sub-block still corresponds to several hidden layers of the sub-block, not several hidden layers of the input matrix, so it is necessary to perform sub-block transformation on the sub-block corresponding to each processing head in each processing unit between the first number of processing units. The process of sub-block transformation is actually to send a part of several hidden layers contained in the sub-block to other processing units, and calculate the attention weight of this part of hidden layers of the input matrix by the processing head of other processing units, while retaining the data of the part of hidden layers of the first transformed sub-block in the processing unit corresponding to the processing head, and receiving the data corresponding to the locally retained part of hidden layers from other processing units. Thus, the attention calculation task of the input matrix on the several hidden layers corresponding to the sub-block is performed by the processing head in the first number of processing units respectively. This process requires mutual transmission of data between the first number of processing units, so it is necessary to cut the sub-block corresponding to the processing head into the first number of first patches along the second dimension in each processing unit, and then transmit the first patches between the first number of processing units.

[0208] In step 750, after the sub-block corresponding to the processing head is cut into the first number of first patches in each processing unit, the processing head of each processing unit has the first number of first patches, and the first number of patches in the same processing head correspond to one processing head in each processing unit respectively. At this time, the first patches corresponding to the same processing head in different processing units are aggregated to the corresponding processing unit, and the first patches corresponding to the processing head of other processing units in the first number of first patches of the same processing head are sent to the corresponding processing unit respectively. Thus, each processing unit obtains a plurality of first patches corresponding to the same several hidden layers in all first transformed sub-blocks, and then the plurality of first patches corresponding to the same several hidden layers are sequentially spliced along the first dimension to obtain the second transformed sub-block corresponding to the processing head. Since the second transformed sub-block is obtained by splicing the first patches corresponding to the same several hidden layers in all first transformed sub-blocks, the second transformed sub-block corresponds to the several hidden layers of the complete input matrix.

[0209] Specifically, referring to Figure 8In the embodiment, after the first linear transformation on the sub-block, the sub-query matrix, the sub-key matrix and the sub-value matrix with the shape of (N / P, d) are obtained, then the first transformed sub-block is split along the second dimension to obtain a plurality of sub-blocks with the shape of (N / P, P*hs), wherein d=m*P*hs, m is the second number, and the sub-block corresponds to P*hs hidden layers of N / P elements in the input sequence. Then, the sub-block transformation processing is performed on the sub-block corresponding to each processing head among the P processing units, and the sub-block with the shape of (N / P, P*hs) corresponding to the processing head is transformed into the second transformed sub-block with the shape of (N, hs), which corresponds to hs hidden layers of N elements in the input sequence, that is, hs hidden layers of the entire input sequence, that is, the processing head obtains the matrix block corresponding to hs hidden layers of the entire input sequence at this time. At this time, the processing head can perform attention calculation without waiting for the sub-block transformation process of other sub-blocks, so that the sub-block transformation and attention calculation corresponding to different processing heads can be executed concurrently, and the efficiency of multi-head attention processing is improved.

[0210] In the related art, with reference to Figure 9 After the first linear transformation on the sub-block to obtain the first transformed sub-block, the sub-block transformation is performed on the first transformed sub-block among the processing units to obtain the transformed block corresponding to the entire first transformed sub-block, and then the transformed block is distributed to each processing head of the processing unit for attention calculation. At this time, each processing head of the processing unit needs to wait until the sub-block transformation process of the entire first transformed sub-block is completed, and then starts to perform attention calculation, so the efficiency of multi-head attention processing is relatively low.

[0211] It can be understood that, in the embodiment, the first number of first patches corresponding to the multiple processing heads in the same processing unit are aggregated, the aggregation process of the first patches corresponding to the multiple processing heads in the same processing unit is independent of each other, based on which, the aggregation tasks of the first patches corresponding to the different processing heads in the same processing unit can be executed in sequence. Specifically, the multiple processing heads in the same unit can be sorted, and then the first patches corresponding to each processing head are aggregated in sequence. After the multiple first patches corresponding to the current processing head are aggregated and the second post-transformation sub-block corresponding to the current processing head is obtained, the aggregation of the first patches corresponding to the next processing head of the current processing head is started. Since the second post-transformation sub-block corresponding to the current processing head has been obtained, the current processing head can perform attention calculation based on the corresponding second post-transformation sub-block at the same time, so that the processes of sub-block transformation of different processing heads and attention calculation can be executed concurrently, and the processing heads do not need to wait for the completion of the sub-block transformation of the entire first post-transformation sub-block and then start the attention calculation, thereby reducing the time required for the attention processing of the processing unit and improving the efficiency of the multi-head attention processing.

[0212] In the embodiments disclosed in steps 730 to 750, in each processing unit, the corresponding first post-transformation sub-block is divided into the second number of sub-blocks corresponding to the second number of processing heads in the processing unit along the second dimension, and then each sub-block corresponding to each processing head is divided into the first number of first patches corresponding to the first number of processing units along the second dimension. Then, between the first number of processing units, the sub-block transformation of each sub-block corresponding to each processing head is performed in sequence, and the multiple first patches corresponding to the same processing head are aggregated into the corresponding processing unit to complete the sub-block transformation of the processing head. Since the multiple processing heads in the same processing unit correspond to different sub-blocks respectively, the sub-block transformation processes of the sub-blocks corresponding to the multiple processing heads in the same processing unit are independent of each other. Therefore, the sub-block transformation processes corresponding to each processing head can be separated and executed in sequence according to a certain order, and after the sub-block transformation corresponding to the current processing head is completed, the current processing head can perform attention calculation. Therefore, the sub-block transformation and attention calculation of different processing heads in the processing unit can be executed concurrently, thereby reducing the total time length of the multi-head attention processing of the processing unit and improving the efficiency of the multi-head attention processing.

[0213] In one embodiment, the second dimension corresponds to each hidden layer of the input matrix, and the first transformed sub-block is obtained by performing a first linear transformation on a sub-block composed of a plurality of matrix rows of the input matrix, the number of hidden layers of the first transformed sub-block is equal to the number of hidden layers of the input matrix, and the number of hidden layers of the input matrix is determined based on the dimension of the vector corresponding to each element in the input sequence, and the dimension of the vector corresponding to each element is determined by the parameter of the embedding layer. In the present disclosure, each element in the input sequence is converted into a vector with a third number of dimensions after being processed by the embedding layer, that is, the first transformed sub-block has a third number of hidden layers in the second dimension. Specifically, the third number as the parameter of the embedding layer can be pre-set, and in the present disclosure, the third number is a common multiple of the first number and the second number, so as to ensure that the third number of hidden layers can be divided into the second number of sub-blocks, and each sub-block can be divided into the first number of first patches. In one embodiment, referring to Figure 15 Step 730 includes step 1510 of determining a first quotient of the third number and the second number.

[0214] Step 1520, in the processing unit, dividing each first quotient of hidden layers into a sub-block along the second dimension to obtain the second number of sub-blocks, wherein each sub-block corresponds to a processing head in the processing unit.

[0215] In step 1510, the first quotient represents the number of corresponding hidden layers of each sub-block after the first transformed sub-block is evenly divided into the second number of sub-blocks along the second dimension. It can be understood that the second number of sub-blocks cut from the same first transformed sub-block correspond to the second number of processing heads of the same processing unit respectively, and the second number of processing heads of the same processing unit share the computing resources and communication resources of the processing unit, that is, the second number of processing heads in the same processing unit are equivalent, and therefore the first transformed sub-block can be evenly divided into the second number of sub-blocks along the second dimension.

[0216] In step 1520, the first quotient is determined, and since each row or each column in the second dimension of the first transformed sub-block corresponds to a hidden layer, based on this, the data corresponding to the first quotient number of hidden layers is divided into a block along the second dimension, that is, the first quotient number of matrix rows or the first quotient number of matrix columns are determined as a block along the second dimension. Specifically, the first transformed sub-block includes a sub-query matrix, a sub-key matrix, and a sub-value matrix corresponding to the sub-block, when the first transformed sub-block is split, the sub-query matrix, the sub-key matrix, and the sub-value matrix are respectively split into matrix blocks, and a block is formed by the matrix blocks of the three matrices. For the sub-query matrix and the sub-value matrix, the second dimension is the dimension of the matrix column, and each matrix column corresponds to a hidden layer. The first quotient number of matrix columns in the sub-query matrix and the sub-value matrix are divided into data in a block. In some embodiments, for the sub-key matrix, since the sub-key matrix needs to be transposed when calculating attention, if the sub-key matrix in the first transformed sub-block is transposed before the sub-block transformation, for the transposed sub-key matrix, the second dimension is the dimension of the matrix row, and each matrix row corresponds to a hidden layer. At this time, the first quotient number of matrix rows in the transposed sub-key matrix can be divided into data in a block along the second dimension. In another embodiment, the sub-key matrix in the second transformed sub-block can also be transposed after the sub-block transformation, that is, the sub-key matrix is not transposed before the sub-block transformation. At this time, for the untransposed sub-key matrix, the second dimension is the dimension of the matrix column, and the first quotient number of matrix columns in the sub-key matrix are divided into data in a block. Referring to Figure 16 , the first transformed sub-block is a matrix including 32 matrix rows and 128 matrix columns, and each processing unit includes 4 processing heads. At this time, the first transformed sub-block is divided into a block every 32 matrix columns, and 4 blocks are obtained. The 4 blocks correspond to the processing head 0 to the processing head 3 in the processing unit in turn.

[0217] In the embodiments disclosed in steps 1510 to 1520, the first quotient is determined based on the number of hidden layers of the first transformed sub-block and the number of processing heads in the processing unit. Then, the data corresponding to each first quotient number of hidden layers in the first transformed sub-block is divided into a block along the second dimension representing the number of hidden layers, thereby obtaining a second number of blocks corresponding to a second number of processing heads respectively. In this way, when the sub-block transformation is performed subsequently, the blocks corresponding to each processing head can be transformed respectively, so that the sub-block transformation and attention calculation of different processing heads are performed concurrently, improving the processing efficiency of multi-head attention.

[0218] In one embodiment, referring to the above description, the third number is a common multiple of the first number and the second number, and the data corresponding to the third number of hidden layers of the first transformed sub-block is evenly divided into the second number of sub-blocks, each of which includes a fourth number of hidden layers, i.e., each sub-block includes a fourth number of matrix rows or matrix columns along the second dimension, wherein the fourth number is the first quotient in the above embodiment, and when the sub-block transformation is performed, the sub-block corresponding to each processing head is divided into a first number of patches, and referring to Figure 17 Step 740 includes:

[0219] Step 1710 determines a second quotient of the fourth number and the first number.

[0220] Step 1720 divides each second quotient of hidden layers into a first patch along the second dimension in the sub-block corresponding to the current processing head to obtain a plurality of first patches.

[0221] Step 1730 determines a target first patch corresponding to the current processing head from the plurality of first patches, and determines the plurality of first patches other than the target first patch as to-be-sent first patches.

[0222] In step 1710, the second quotient represents the number of matrix rows or matrix columns along the second dimension of each first patch after the data corresponding to the fourth number of hidden layers in the sub-block is evenly divided into the first number of first patches.

[0223] In step 1720, after the second quotient is determined, each second quotient of rows or columns corresponding to the hidden layers is divided into a first patch along the second dimension, thereby obtaining the first number of first patches. The details in step 1720 can be referred to the description of step 1520 above, in which each first quotient of hidden layers in the first transformed sub-block is divided into a sub-block, and the first transformed sub-block, the first quotient, and the sub-block are replaced by the sub-block, the second quotient, and the first patch in turn, which will not be described here.

[0224] In step 1730, the target first patch is one of the first number of first patches, and specifically, the first number of first patches can be sequentially assigned to the processing heads of the processing units according to the order of the positions of the sub-blocks corresponding to the processing units in the input matrix, for example, referring to Figure 18, the first number is 4, that is, the number of processing units is 4, the second dimension is the matrix array dimension, and the block includes 32 hidden layers, that is, the block includes 32 matrix arrays, and after the division, every 8 matrix arrays are taken as a first piece block, wherein the sub-block received by the current processing head corresponding to the processing unit of the input matrix is a sub-block composed of a plurality of matrix rows in front of the input matrix, that is, the first sub-block of the input matrix, based on this, the first piece block composed of the first 8 matrix arrays in the block is determined as the first piece block corresponding to the processing unit of the current processing head, and the other three first piece blocks are respectively corresponding to the other three processing units in the processing device. After determining the corresponding relationship between the first number of first piece blocks and the first number of processing units, the first piece block corresponding to the processing unit of the current processing head is determined as the target first piece block, and the plurality of hidden layers corresponding to the target first piece block are the plurality of hidden layers for which the current processing head performs attention calculation, and the other first piece blocks except the target first piece block are respectively corresponding to the other processing units in the processing device. The attention of the hidden layers corresponding to these first piece blocks will be allocated to the processing head in the other processing units for calculation, and based on this, the other first piece blocks except the target first piece block are determined as the to-be-sent first piece blocks.

[0225] In the embodiment of steps 1710 to 1730, based on the second quotient of the number of hidden layers contained in the block and the number of processing units, every second quotient matrix row or matrix array is divided into a first piece block along the second dimension, thereby dividing the block into a first number of first piece blocks, and determining the processing unit corresponding to each first piece block. Then, the first piece block corresponding to the processing unit of the current processing head is determined as the target first piece block, and the first piece block corresponding to the other processing units in the processing device is determined as the to-be-sent first piece block. In this way, each to-be-sent first piece block is subsequently sent to the corresponding processing unit, and the first piece block corresponding to the same hidden layer as the target first piece block is received from the other processing units. Then, the received plurality of first piece blocks corresponding to the same hidden layer are spliced, and the second transformed sub-block corresponding to the current processing head is obtained.

[0226] In one embodiment, the computing resource amount of each processing unit is different, and after the sub-block transformation processing to obtain the second transformed sub-block corresponding to the processing head, attention calculation needs to be performed based on the second transformed sub-block. The number of hidden layers corresponding to the second transformed sub-block determines the amount of calculation in the attention calculation process. At this time, referring to Figure 19 Step 740 includes:

[0227] Step 1910, determining the computing resource amount of the processing unit;

[0228] Step 1920, based on the ratio of the computing resource amount of each processing unit, dividing the block into a first number of first piece blocks corresponding to each processing unit along the second dimension;

[0229] Step 1930, determining the first tile corresponding to the processing unit of the current processing head as a target first tile, and determining other first tiles except the target first tile as to-be-sent first tiles.

[0230] In step 1910, the related description of the computing resource amount can refer to the description of step 1310 described above, which will not be repeated here.

[0231] In step 1920, after determining the computing resource amount of each processing unit, the block is divided along the second dimension based on the ratio of the computing resource amount of each processing unit. It can be understood that in the second dimension, each row or column corresponds to a hidden layer. After the block is divided into first tiles, the more rows or columns of the first tile along the second dimension, the more hidden layers the first tile corresponds to, and the larger the data amount of the first tile. The size of the second transformed sub-block aggregated by multiple first tiles corresponding to the same hidden layer of the first tile is also relatively large. The amount of calculation of the processing head based on the second transformed sub-block for attention calculation is also relatively large. Conversely, the fewer hidden layers the first tile corresponds to, the less calculation the processing head corresponding to the first tile performs for attention calculation. It can be understood that since each processing unit includes a second number of processing heads, and multiple processing heads of the same processing unit are equivalent, it leads to that the calculation efficiency of the processing head in each processing unit is proportional to the computing resource amount of the processing unit. Since the computing resource amounts of each processing unit are different, the calculation efficiency of the processing head in different processing units is different. At this time, if the block is divided into a first number of first tiles, the amount of calculation of the processing head in each processing unit for attention calculation is equal. At this time, for the processing head in the processing unit with less computing resource amount, the time required to complete attention calculation is longer, which makes the total time required to complete attention calculation longer, and the efficiency of multi-head attention processing is not high. In this embodiment, the block is divided into a first number of first tiles corresponding to each processing unit along the second dimension based on the ratio of the computing resource amount of each processing unit. In this way, the first tile corresponding to the processing head in the processing unit with more computing resource amount corresponds to more hidden layers, and the amount of calculation for subsequent attention calculation is relatively large. For the processing head in the processing unit with less computing resource amount, the corresponding first tile corresponds to fewer hidden layers, and the amount of calculation for subsequent attention calculation is relatively small. Thus, the number of hidden layers of the first tile corresponding to the processing head in each processing unit matches the computing resource amount of the processing unit, i.e., the calculation efficiency of the processing head matches each other. For details, refer to Figure 20, the second dimension is the dimension of the matrix array, each sub-block includes 32 matrix arrays, the processing device includes 4 processing units, in turn, processing unit 1 to processing unit 4, the corresponding amount of computing resources is R1, R2, R3, R4 in turn, and R1:R2:R3:R4=8:7:11:6, then one sub-block in processing unit 1 is divided to obtain 4 first sub-blocks, in turn, first sub-block 1 to first sub-block 4, the number of matrix arrays of first sub-block 1 to first sub-block 4 is 8, 7, 11, 6 in turn, and first sub-block 1 to first sub-block 4 correspond to processing unit 1 to processing unit 4 in turn.

[0232] In step 1930, the first sub-block corresponding to the processing unit of the current processing head is determined as the target first sub-block, and the first sub-block corresponding to the processing head of other processing units in the processing device is determined as the to-be-sent first sub-block, for example, referring to Figure 20 , first sub-block 1 corresponding to processing unit 1 is determined as the target first sub-block, and first sub-block 2 to first sub-block 4 are determined as the to-be-sent first sub-block. Then, sub-block transformation is performed between the first number of processing units, each to-be-sent first sub-block is sent to the corresponding processing unit, and the corresponding first sub-block is received from other processing units to splice into the second transformed sub-block corresponding to the processing head, so that the number of hidden layers corresponding to the obtained second transformed sub-block is matched with the amount of computing resources of the processing unit to which the processing head belongs. In this way, the processing head in the processing unit with more computing resources is responsible for more attention calculation of hidden layers, and the processing head in the processing unit with less computing resources only needs to be responsible for attention calculation of a small number of hidden layers. In this way, the processing head in each processing unit can complete attention calculation relatively synchronously, and the computing resources of each processing unit can be fully utilized, thereby reducing the total time required for attention calculation and improving the efficiency of multi-head attention processing.

[0233] In the embodiments disclosed in steps 1910 to 1930, based on the ratio of the amount of computing resources of each processing unit, the sub-block is divided along the second dimension into a first number of first sub-blocks corresponding to each processing unit. In this way, the number of hidden layers of the first sub-block corresponding to the processing head in each processing unit corresponds to the amount of computing resources of the processing unit. After performing sub-block transformation, the number of hidden layers in the second transformed sub-block corresponding to the processing head of each processing unit corresponds to the amount of computing resources of the processing unit. When attention calculation is performed based on the second transformed sub-block, the amount of calculation of the processing head of each processing unit corresponds to the amount of computing resources of the processing unit. The processing heads in different processing units can complete attention calculation synchronously, thereby improving the efficiency of multi-head attention processing.

[0234] After the sub-block corresponding to the processing head is divided into a first number of first sub-blocks, and each first sub-block is determined as a target first sub-block or a to-be-sent first sub-block, for example, referring toFigure 21 In one embodiment, step 750 comprises:

[0235] Step 2110, in each processing unit, sequentially taking each processing head as the current processing head;

[0236] Step 2120, sending the data of each to-be-sent first tile in the sub-block corresponding to the current processing head to the interaction processing unit respectively, and receiving the target first tile corresponding to the current processing head from each interaction processing unit, wherein the interaction processing unit is other than the processing unit corresponding to the current processing head;

[0237] Step 2130, splicing the multiple target first tiles corresponding to the current processing head along the first dimension to obtain the second transformed sub-block corresponding to the current processing head.

[0238] In step 2110, each processing unit comprises a second number of processing heads, at this time, according to a preset order, sequentially taking each processing head as the current processing head, so that the sub-block transformation corresponding to each processing head is sequentially executed. Specifically, after the first transformed sub-block is cut into the second number of sub-blocks in each processing unit, a list can be created in each processing unit, taking the second number of sub-blocks as elements in the list respectively, wherein the i-th element of the list is denoted as list[i], thereby obtaining a list with the second number of elements. Then, through a for loop, the elements in the list are sequentially taken as the object of sub-block transformation to perform sub-block transformation.

[0239] It should be noted that, since the first transformed sub-block comprises the sub-query matrix, the sub-key matrix and the sub-value matrix, when the first transformed sub-block is cut into the second number of sub-blocks, the sub-query matrix, the sub-key matrix and the sub-value matrix are also cut respectively, each sub-block actually comprises three matrix blocks corresponding to the sub-query matrix, the sub-key matrix and the sub-value matrix respectively, and when the list is created based on the sub-block, the list should also be created for the matrix block of the sub-query matrix, the matrix block of the sub-key matrix and the matrix block of the sub-value matrix respectively, that is, actually three lists corresponding to the sub-query matrix, the sub-key matrix and the sub-value matrix, i.e. Q_list, K_list and V_list, are created, then the corresponding matrix blocks are taken as the elements of the list, and then through a for loop, each element in the three lists is sequentially taken as the object of sub-block transformation, thereby sequentially taking each processing head of the processing unit as the current processing head and performing sub-block transformation on the sub-block corresponding to the current processing head

[0240] ​It should be noted that since the sub-block transformation is performed among the first number of processing units, each processing unit needs to simultaneously perform the transceiving of the first patch block to perform the sub-block transformation, at the same time, each processing unit has a current processing head, and the patch of the current processing head of each processing unit corresponds to the same number of hidden layers, so after the sub-block transformation, the patch of the current processing head of each processing unit is transformed into the second transformed sub-block corresponding to the number of hidden layers of the complete input matrix.

[0241] In step 2120, the interaction processing unit refers to the processing unit other than the processing unit corresponding to the current processing head in the processing device, for example, the processing device includes four processing units, processing unit 1 to processing unit 4, and the current processing head is a processing head in processing unit 1, then processing unit 2 to processing unit 4 are interaction processing units, it can be understood that the interaction processing unit is a relative concept, each processing unit in the processing device is an interaction processing unit of other processing units except itself.

[0242] After the patch corresponding to the current processing head is divided into the first number of first patch blocks, and based on the correspondence between each first patch block and the processing unit, the first patch block is determined as the target first patch block or the to-be-sent first patch block, the current processing head needs to calculate the attention weight of the number of hidden layers corresponding to the target first patch block, and the attention weight of the hidden layer corresponding to each to-be-sent first patch block is calculated by the processing head in the corresponding processing unit. Based on this, each to-be-sent first patch block is sent to the corresponding interaction processing unit, and the first patch block corresponding to the same number of hidden layers as the target first patch block of the current processing head is received from each interaction processing unit. In this way, all first patch blocks corresponding to the current processing head can be collected, that is, all first patch blocks corresponding to the same number of hidden layers in the first transformed sub-block obtained by performing the first linear transformation based on each sub-block are obtained. These first patch blocks are all target first patch blocks corresponding to the current processing head.

[0243] Specifically, referring to Figure 22The processing device includes four processing units, i.e., processing unit 1 to processing unit 4, which correspond to sub-block 1 to sub-block 4 in sequence. In each processing unit, after the first transformed sub-block is cut into sub-blocks corresponding to the processing heads, the sub-block transformation is started on the sub-block corresponding to the first processing head of each processing unit. The first processing head of each processing unit corresponds to the first tile 1 in the first transformed sub-block of each processing unit. After the first tile 1 in the first transformed sub-block of each processing unit is cut into a first number of tiles, at this time, the processing unit 1 includes the first tile 1 to the first tile 4 in the first tile 1 of the sub-block 1. The processing unit 2 includes the first tile 1 to the first tile 4 in the first tile 1 of the sub-block 2, and so on. The current processing head of the processing unit 1 to the processing unit 4 is responsible for the attention calculation of the hidden layer corresponding to the first tile 1 to the first tile 4 in sequence. When the sub-block transformation is performed, the processing unit 1 retains the first tile 1 in the first tile 1 of the sub-block 1, and sequentially sends the first tile 2 to the first tile 4 in the first tile 1 of the sub-block 1 to the processing unit 2 to the processing unit 4, and receives the first tile 1 in the first tile 1 of the sub-block 2, the first tile 1 in the first tile 1 of the sub-block 3, and the first tile 1 in the first tile 1 of the sub-block 4 from the processing unit 2 to the processing unit 4. Similarly, the processing unit 2 sends the first tile 1, the first tile 3, and the first tile 4 in the first tile 1 of the sub-block 2 to the processing unit 1, the processing unit 3, and the processing unit 4 respectively, retains the first tile 2 in the first tile 1 of the sub-block 2, and receives the first tile 2 in the first tile 1 of the sub-block 1, the first tile 2 in the first tile 1 of the sub-block 3, and the first tile 2 in the first tile 1 of the sub-block 4 from the processing unit 1, the processing unit 3, and the processing unit 4. In this way, after the sub-block transformation is performed on the sub-block corresponding to the first processing head of each processing unit, the processing unit 1 obtains the first tile 1 in the first tile 1 of the sub-block 1 to the sub-block 4, the processing unit 2 obtains the first tile 2 in the first tile 1 of the sub-block 1 to the sub-block 4, the processing unit 3 obtains the first tile 3 in the first tile 1 of the sub-block 1 to the sub-block 4, and the processing unit 4 obtains the first tile in the first tile 1 of the sub-block 1 to the sub-block 4.

[0244] In step 2130, after all target first tiles corresponding to the current processing head are obtained, each target first tile is sequentially spliced along the first dimension. After splicing, the matrix corresponding to the partial hidden layer of all elements of the input sequence is obtained, which is the second transformed sub-block corresponding to the current processing head. Specifically, for example, referring to Figure 22After the sub-block transformation, the processing unit 1 obtains the first tile 1 in the partition 1 of the sub-block 1 to sub-block 4, and sequentially splices the four first tiles 1 along the first dimension, that is, from top to bottom, the first tile 1 corresponding to the sub-block 1, the first tile 1 corresponding to the sub-block 2, the first tile 1 corresponding to the sub-block 3, and the first tile 1 corresponding to the sub-block 4, to obtain the second transformed sub-block corresponding to the current processing head of the processing unit 1, which corresponds to the data of the front several hidden layers of the entire input sequence.

[0245] It can be understood that after the sub-block transformation corresponding to the current processing head is completed to obtain the second transformed sub-block corresponding to the current processing head, attention calculation can be performed based on the second transformed sub-block corresponding to the current processing head, and the next processing head of the current processing head is taken as a new current processing head, and sub-block transformation processing is performed on the partition corresponding to the new current processing head, so that the attention calculation of the current processing head and the sub-block transformation of the next processing head of the current processing head are concurrently executed, the total time length of the multi-head attention processing of the processing unit is shortened, and the efficiency of the multi-head attention processing is improved.

[0246] In the embodiments disclosed in steps 2110 to 2130, by taking each processing head in the processing unit as a current processing head in turn, and then performing sub-block transformation on the partition corresponding to the current processing head, the sub-block transformation of the first transformed sub-block is performed in segments, and after the sub-block transformation of the partition corresponding to the current processing head is completed, the second transformed sub-block corresponding to the current processing head can be obtained to perform attention calculation, so that the process of attention calculation does not need to wait for the completion of the sub-block transformation of the entire first transformed sub-block, and the process of sub-block transformation and the process of attention calculation can be concurrently executed, thereby improving the efficiency of multi-head attention processing.

[0247] Detailed description of concurrent execution of sub-block transformation and attention calculation in step 420

[0248] In step 420, the sub-block transformations corresponding to different processing heads in the processing unit are executed in series, and the sub-block transformations corresponding to different processing heads and the attention calculation are concurrently executed.

[0249] In an embodiment, the processing unit is provided with a sending buffer and a receiving buffer, the sending buffer is used to temporarily buffer data to be sent, and the receiving buffer is used to temporarily buffer data received from other processing units. In an embodiment, with reference to Figure 23 , step 2120, includes:

[0250] Step 2310, when each of the first piece blocks corresponding to the previous processing head of the current processing head in the sending buffer is sent to the corresponding interactive processing, each of the first piece blocks corresponding to the current processing head corresponding to the current processing head is loaded into the sending buffer, and the data in the sending buffer is sent to the corresponding interactive processing unit through the communication thread of the processing unit;

[0251] Step 2320, when the data corresponding to the previous processing head of the current processing head in the receiving buffer is taken out, the target first piece block corresponding to the current processing head is loaded into the receiving buffer, the target first piece block corresponding to the current processing head is received from each interactive processing unit through the communication thread, and is sequentially written into the receiving buffer.

[0252] In step 2310, when sending the to-be-sent first tile to the corresponding processing unit, the to-be-sent first tile needs to be written into the sending buffer of the processing unit first, and the communication thread of the processing unit reads the data to be sent from the sending buffer. In order to quickly complete the sub-block transformation of a single tile and quickly start attention calculation, the processing unit is configured with only one communication thread, which occupies all communication resources of the processing unit. At this time, the to-be-sent first tiles in the tiles corresponding to all processing heads need to be written into the same sending buffer in turn. At this time, if the to-be-sent first tiles of different tiles are written into the sending buffer at the same time, it may cause the communication thread to randomly send the to-be-sent first tiles in the tiles corresponding to multiple processing heads in the sending buffer to the interaction processing unit. On the one hand, the to-be-sent first tiles corresponding to the subsequent processing head of the current processing head occupy a part of the communication resources, which causes the to-be-sent first tile in the tile corresponding to the current processing head to be unable to be quickly sent to the corresponding processing unit, and the sub-block transformation of the current processing head and the attention calculation cannot be quickly completed. On the other hand, the interaction processing unit may misidentify the first tile corresponding to the subsequent processing head as the first tile corresponding to the current processing head, so as to incorrectly splice the first tile corresponding to the subsequent processing head into the second transformed sub-block corresponding to the current processing head, thereby causing the result of the attention calculation to be incorrect. Based on this, in the present disclosure, the to-be-sent first tiles corresponding to each processing head are written into the sending buffer in turn, and the to-be-sent first tiles corresponding to the previous processing head of the current processing head in the sending buffer need to be sent to the corresponding processing unit, that is, after the to-be-sent first tile corresponding to the previous processing head in the sending buffer is emptied, the to-be-sent first tile corresponding to the current processing head is written into the sending buffer, so as to ensure that the sending buffer writes only the to-be-sent first tile corresponding to one processing head at the same time, and the sub-block transformation processes of different processing heads are isolated from each other. At the same time, in the present disclosure, after the to-be-sent first tile corresponding to the previous processing head of the current processing head is emptied, the to-be-sent first tile corresponding to the current processing head is immediately written into the sending buffer, so that the sub-block transformation processes of the tiles corresponding to multiple processing heads are executed in turn and continuously, forming a pipeline processing, thereby fully utilizing the communication thread of the processing unit.

[0253] In step 2320, the data received from other processing units is temporarily stored in the receiving buffer first, and it can be understood that if multiple first patches corresponding to different processing heads are simultaneously stored in the receiving buffer, when reading data from the receiving buffer, multiple first patches corresponding to different processing heads can be mistakenly identified as multiple first patches corresponding to the same processing head and spliced, resulting in errors in attention calculation results. Based on this, in the present disclosure, the data corresponding to the previous processing head of the current processing head, i.e., the multiple first patches corresponding to the previous processing head, are taken out from the receiving buffer, and then the target first patch corresponding to the current processing head is received from other processing units and written into the receiving buffer, thereby ensuring the isolation between the target first patches corresponding to different processing heads and ensuring that the multiple first patches read from the receiving buffer each time correspond to the same processing head.

[0254] In the embodiments disclosed in steps 2310 to 2320, after the to-be-sent first patches corresponding to the previous processing head of the current processing head in the sending buffer are sent to the corresponding interactive processing unit, the multiple to-be-sent first patches corresponding to the current processing head are immediately written into the sending buffer, and at the same time, after the data corresponding to the previous processing head of the current processing head in the receiving buffer are taken out, the first patch corresponding to the current processing head is received from other processing units and written into the receiving buffer, thereby ensuring the isolation of the sub-block transformation of the patches corresponding to each processing head, thereby ensuring the accuracy of the second transformed sub-block obtained after the sub-block transformation of each patch, and at the same time, the sub-block transformation of multiple processing heads is sequentially and continuously performed, thereby ensuring the accuracy of the sub-block transformation while ensuring the efficiency of the sub-block transformation.

[0255] In one embodiment, referring to Figure 24 , after step 2120, the method comprises:

[0256] Step 2410, after collecting each target first patch corresponding to the current processing head sent by each interactive processing unit in the receiving buffer, and splicing the multiple target first patches into a second transformed sub-block, the second transformed sub-block is taken out from the receiving buffer and transmitted to the calculation buffer of the calculation thread of the processing unit;

[0257] Step 2420, after the calculation thread of the processing unit calculates the second transformed sub-block corresponding to the previous processing head of the current processing head as the corresponding first attention result sub-block, the second transformed sub-block corresponding to the current processing head is read from the calculation buffer.

[0258] In step 2410, when the receiving buffer collects each target first patch corresponding to the current processing head sent by each interaction processing unit and splices into the second transformed sub-block, it means that the sub-block transformation of the partition corresponding to the current processing head has been completed. At this time, attention calculation should be performed based on the second transformed sub-block corresponding to the current processing head. Based on this, the second transformed sub-block is taken out from the receiving buffer and written into the calculation buffer of the calculation thread. After the second transformed sub-block is taken out from the receiving buffer, the receiving buffer is emptied. At this time, the target first patch corresponding to the next processing head of the current processing head can be received from each interaction processing unit through the communication thread and written into the receiving buffer. Thus, while the current processing head performs attention calculation, the sub-block transformation of the partition corresponding to the next processing head of the current processing head is performed, so that the sub-block transformation and attention calculation of different processing heads can be performed concurrently. In the embodiment, when the receiving buffer collects each target first patch corresponding to the current processing head, the second transformed sub-block spliced from each target first patch is taken out from the receiving buffer, and the sub-block transformation of the partition corresponding to the next processing head is started. This ensures that the sub-block transformation processes of adjacent two processing heads are continuous, so that the sub-block transformation and attention calculation of multiple processing heads of each processing unit form a pipelined operation.

[0259] Specifically, since the first number of first patches obtained by splitting the partition corresponding to the current processing head in each processing unit are respectively corresponding to the processing heads in the first number of processing units, that is, each processing head corresponds to a target first patch in each processing unit, and the number of target first patches corresponding to each processing head is the first number. Based on this, whether the receiving buffer has collected multiple target first patches corresponding to the current processing head can be determined by judging the number of first patches in the processing unit. When the number of first patches in the receiving buffer reaches the first number, it is determined that the receiving buffer has collected all target first patches corresponding to the current processing head. At this time, the first number of first patches in the receiving buffer are spliced in turn to obtain the second transformed sub-block corresponding to the current processing head and taken out and written into the calculation buffer of the calculation thread.

[0260] In some related technologies, with reference to Figure 25, the processing unit writes all the data to be sent in the complete first transformed sub-block to the receiving buffer at one time, and performs the sub-block exchange processing on the entire first transformed sub-block at one time. At this time, in order to ensure the accuracy of the sub-block transformation, it is necessary to wait until the sub-block transformation of the entire first transformed sub-block is completed to obtain a transformed block shared by all processing heads, and then write the transformed block to the calculation buffer of the calculation thread for attention calculation by each processing head. In this way, the process of sub-block transformation and the process of attention calculation are executed in series, and cannot be executed in parallel, thereby resulting in low efficiency of multi-head attention processing.

[0261] In the embodiments of the present disclosure, with reference to Figure 26 , the processing unit includes four processing heads, i.e., processing head 0 to processing head 3. The first transformed sub-block of the processing unit is divided into four sub-blocks corresponding to the processing head 0 to the processing head 3, respectively. Then, the data to be sent in the sub-block corresponding to the processing head 0 is written to the sending buffer, and the target first piece block corresponding to the processing head 0 is written to the receiving buffer. The sub-block transformation corresponding to the processing head 0 is started to be executed. When the sub-block transformation corresponding to the processing head 0 is completed, all the data to be sent in the sub-block corresponding to the processing head 0 in the sending buffer is sent, and the target first piece block corresponding to the processing head 0 received in the receiving buffer is also taken out to the calculation thread. The attention calculation corresponding to the processing head 0 in the calculation thread can be executed. At the same time, the data to be sent in the sub-block corresponding to the processing head 1 is written to the sending buffer, the target first piece block corresponding to the processing head 1 is written to the receiving buffer, and the sub-block transformation corresponding to the processing head 1 is executed in parallel. In this way, the process of sub-block transformation and the process of attention calculation are executed concurrently, thereby improving the efficiency of multi-head attention processing.

[0262] In step 2420, after the second transformed sub-block corresponding to the current processing head is written to the calculation buffer, the calculation thread of the processing unit may not complete the attention calculation corresponding to the previous processing head. Based on this, the second transformed sub-block corresponding to the current processing head is read from the calculation buffer for attention calculation after the calculation thread completes the attention calculation corresponding to the previous processing head.

[0263] In the embodiments disclosed in steps 2410 to 2420, after the first number of target first patches corresponding to the current processing head are collected in the receiving buffer and spliced into the corresponding second transformed sub-block, the second transformed sub-block corresponding to the current processing head is taken out from the receiving buffer and written into the calculation buffer, thereby, on the one hand, the attention calculation of the current processing head is prepared, and on the other hand, the receiving buffer is emptied to make the sub-block transformation corresponding to the next processing head of the current processing head start to be executed, so that the attention calculation process of the current processing and the sub-block transformation process of the next processing head of the current processing head are executed concurrently, the total time length required for the processing unit to perform the multi-head attention processing is reduced, and the efficiency of the multi-head attention processing is improved.

[0264] In one embodiment, with reference to Figure 27 , step 2130 comprises:

[0265] Step 2710 determines a first splicing order based on the positions of the sub-blocks corresponding to the respective processing units in the input matrix.

[0266] Step 2720 splices the target first patches sent by the respective interaction processing units and the target first patches corresponding to the current processing head based on the first splicing order to obtain the second transformed sub-block corresponding to the current processing head.

[0267] In step 2710, the first splicing order is the splicing order of the respective target first patches when splicing all the target first patches corresponding to the current processing head into the matrix block corresponding to the plurality of hidden layers of the input sequence. It can be understood that in order to splice the respective first patches into the matrix block corresponding to the plurality of hidden layers of the input sequence, the arrangement order of the elements corresponding to each first patch in the input sequence needs to be determined, and since the first patches sent by the respective processing units correspond to the sub-blocks received by the processing units, and each matrix row of the input matrix corresponds to an element of the input sequence, based on the positions of the sub-blocks received by the processing units in the input matrix, the splicing order of the respective first patches can be determined, for example, the input matrix includes N matrix rows, and each sub-block is composed of N / P non-overlapping matrix rows in the input matrix, then the sub-block composed of the first N / P matrix rows is the first sub-block, the sub-block composed of the N / P+1 matrix row to the 2*N / P matrix row is the second sub-block, and so on, the splicing order of the sub-blocks corresponding to the respective processing units can be determined, and the splicing order of the first patches sent by the respective processing units is the first splicing order.

[0268] In step 2720, after determining the first splicing order, the target first patches sent by each processing unit are spliced along the first dimension based on the first splicing order, and a matrix corresponding to the plurality of hidden layers of the input sequence, i.e., the second transformed sub-block corresponding to the current processing head, is obtained.

[0269] In the embodiments disclosed in steps 2710 to 2720, since the first patches sent by each processing unit are obtained based on the sub-blocks received by the processing unit, the first splicing order of the first patches sent by each processing unit is determined based on the positions of the sub-blocks received by each processing unit in the input matrix, and then the plurality of target first patches corresponding to the same processing head sent by each processing unit are spliced based on the first splicing order, so that the plurality of target first patches corresponding to the processing head can be accurately spliced into the second transformed sub-block corresponding to the plurality of hidden layers of the input sequence, thereby ensuring the accuracy of the attention weight in subsequent calculation.

[0270] Detailed description of attention calculation in step 420

[0271] The attention calculation in step 420 refers to inputting the second transformed sub-block into the processing head corresponding to the second transformed sub-block to perform attention calculation, and obtaining a first attention result sub-block corresponding to each processing head.

[0272] In an embodiment, the second transformed sub-block includes a query matrix block, a key matrix block, and a value matrix block corresponding to the current processing head, and the query matrix block, the key matrix block, and the value matrix block are obtained by performing the first transformation on the sub-block corresponding to the current processing head. Figure 28 In step 420, the second transformed sub-block is input into the processing head corresponding to the second transformed sub-block to perform attention calculation, and a first attention result sub-block corresponding to each processing head is obtained, including:

[0273] In step 2810, the key matrix block is transposed in the current processing head to obtain a transposed key matrix block.

[0274] In step 2820, the query matrix block and the transposed key matrix block are multiplied to obtain a product matrix.

[0275] In step 2830, the product matrix is exponentially normalized to obtain an exponentially normalized matrix.

[0276] In step 2840, the exponentially normalized matrix and the value matrix block are multiplied to obtain the first attention result sub-block.

[0277] In step 2810, the process of transposition is to interchange the matrix rows and the matrix columns of the key matrix block. Specifically, for example, the key matrix block is a 4-row 8-column matrix, i.e., the key matrix block is composed of 4 matrix rows, and each matrix row includes 8 data. The 8 data in the first matrix row of the key matrix block are sequentially taken as the 8 data of the first matrix column of the transposed key matrix block. Similarly, the 8 data in the second matrix row of the key matrix block are taken as the 8 data of the second matrix column of the transposed key matrix block, and so on. Finally, a matrix having 4 matrix columns, each of which includes 8 data, i.e., an 8-row 4-column matrix, is obtained, which is the transposed key matrix block.

[0278] In step 2820, the query matrix block and the transposed key matrix block are subjected to matrix multiplication, i.e., the dot products of the data of each matrix row of the query matrix block and the data of each matrix column of the transposed key matrix block are calculated, to obtain a product matrix of the query matrix block and the transposed key matrix block, which reflects the similarity between each data in the query matrix block and each data in the key matrix block, i.e., the similarity between each pair of elements of the input sequence, which can represent the similarity of the semantics of the two elements in the context of the input sequence. It can be understood that the query matrix block and the key matrix block have the same shape, and after the key matrix block is transposed, the number of matrix columns of the transposed key matrix block is equal to the number of matrix rows of the query matrix block. The product matrix obtained by subjecting the two to matrix multiplication is a square matrix. For example, the query matrix block and the key matrix block are (N, hs) matrices, where N represents the length of the input sequence, i.e., the number of elements of the input sequence, and hs represents the number of hidden layers. After the key matrix block is transposed, a matrix of shape (hs, N) is obtained. After the query matrix block and the transposed key matrix block are subjected to matrix multiplication, a product matrix of shape (N, N), i.e., an N-order square matrix, is obtained. In the N-order square matrix, each matrix row corresponds to an element of the input sequence, and a matrix row includes N data, which respectively represent the similarity between the element corresponding to the matrix row and each element of the input sequence. For example, the N data of the first matrix row of the product matrix are respectively the similarity between the first element of the input sequence and each element of the input sequence in the corresponding hidden layer.

[0279] In step 2830, each data in the product matrix represents an attention score of each element of the input sequence, and the attention score is not fixed in a range, and even a negative value can appear, which is not conducive to normalizing the output by the multi-head attention mechanism. Based on this, in the embodiment, the data in the product matrix needs to be exponentially normalized to map each data in the product matrix to the interval (0, 1), and convert the data in each matrix row in the product matrix into a corresponding probability, so that each matrix row in the product matrix is converted from a vector representation into a corresponding probability distribution, which represents the importance of each element of the input sequence in each hidden layer dimension.

[0280] In an embodiment, since each data in the product matrix is obtained by performing dot product on the matrix row of the query matrix block and the data of one matrix column of the transposed key matrix block, the difference between the values of the data in the same row can be large, and after exponential normalization, the relative difference between each data in the product matrix is exponentially amplified. For example, the two data in a matrix row in the product matrix are 2 and 10 respectively, and after exponential normalization, the two data are converted into e 2 / A and e 10 / A, where A is the sum of the exponential forms of all data in the matrix row, and the relative difference between the two data is changed from 1:5 to 1:e 8 This causes the normalized probability corresponding to the maximum data in each matrix row to be approximately 1 and the normalized probability corresponding to other data to be approximately 0 after the product matrix is exponentially normalized. When the neural network model is trained by the gradient descent method, the gradient vanishing problem occurs. Based on this, in an embodiment, before the product matrix is exponentially normalized, the data in the product matrix of the query matrix block and the transposed key matrix block is scaled. Specifically, the data in the product matrix is divided by the same coefficient. In the present disclosure, the coefficient can be the square root of the order of the product matrix. For example, when the product matrix is an N-order matrix, the scaling is performed by the following formula:

[0281]

[0282] where A is the scaled product matrix, Q is the query matrix block, K T is the transposed key matrix block, and N is the order of the product matrix, i.e., the number of matrix columns of the transposed key matrix block. For example, when N=4, the two data in a matrix row in the product matrix are 2 and 10 respectively, and after scaling, the two data become 1 and 5, and after exponential normalization, the two data are converted into e 1 / A and e 5 / A, and the relative difference between the two is only 1:e 4, compared to the exponential normalization result before scaling, the gap is 1 : e 8 Substantially reduced, so as to avoid the occurrence of gradient disappearance.

[0283] In step 2840, the value matrix block reflects the actual information of each element of the input sequence, and the exponential normalization matrix and the value matrix block are multiplied to obtain the corresponding first attention result sub-block. It can be understood that the exponential normalization matrix and the value matrix block are multiplied, which is actually a process of weighting the actual information of each element in the value matrix block based on the attention weight in the exponential normalization matrix. By weighting the content in the value matrix block, the important content in the value matrix block is highlighted, so that the first attention result sub-block can highlight the relatively important elements in the corresponding hidden layer of the input sequence and ignore some unimportant elements.

[0284] Specifically, referring to the following formula, the above steps 2810 to 2840 can be implemented by the following formula:

[0285]

[0286] Wherein, O represents the first attention result sub-block, Q represents the query matrix block, K T represents the transposed key matrix block, d represents the order of the product matrix, that is, the number of matrix columns of the transposed key matrix block, softmax() represents the exponential normalization function, and V represents the value matrix block.

[0287] In the embodiments disclosed in steps 2810 to 2840, by transposing the key matrix block, then multiplying the query matrix block and the transposed key matrix block to obtain the product matrix, and performing exponential normalization on the product matrix to obtain the exponential normalization matrix, and then multiplying the exponential normalization matrix and the value matrix block to obtain the first attention result block corresponding to the current processing head, in this way, the first attention result sub-block can highlight the relatively important elements in the corresponding hidden layer of the input sequence and ignore the unimportant elements.

[0288] In one embodiment, referring to Figure 29 , step 2830 comprises:

[0289] Step 2910, taking each element in the product matrix as the exponent of the first base to obtain the power of each element;

[0290] Step 2920, for each matrix row in the product matrix, determine the first sum of the powers of each element in the matrix row;

[0291] Step 2930, the ratio of the power of each element to the first sum of the corresponding matrix row is determined as the element of each matrix row in the exponential normalization matrix, and the exponential normalization matrix is obtained.

[0292] In step 2910, the first base number can be the base number e of natural logarithm. It can be understood that the multi-head attention mechanism as a part of the neural network model, in the process of training the neural network model, often needs to optimize the parameters of the neural network model through gradient descent method, which involves the process of derivation, and the original form of each data in the product matrix is not conducive to derivation, while the derivation of the power of e is relatively simple. Based on this, each element in the product matrix is taken as the exponent of the first base number, so as to convert the data of the product matrix into the corresponding power form. In addition, taking each element as the exponent of the first base number can also map the elements with negative values to non-negative values, avoiding the negative attention weight leading to the loss of control of the training process of the neural network model.

[0293] In step 2920, the first sum is the sum of the powers of all elements in the matrix row. As known from the related description of step 2820 above, each matrix row in the product matrix represents the similarity between an element of the input sequence and each element in the input sequence. When performing exponential normalization processing, the similarity scores between the same element and other elements should be normalized. Based on this, in the present disclosure, for each matrix row in the product matrix, the first sum of the powers of each element in the matrix row is determined, so as to normalize each matrix row respectively.

[0294] In step 2930, after determining the first sum of each matrix row, each element in the product matrix is converted into the corresponding attention weight based on the ratio of the power corresponding to each element to the first sum, thereby obtaining the elements of each matrix row in the exponential normalization matrix, and thus obtaining the exponential normalization matrix. Since each element in the exponential normalization matrix is determined based on the ratio of the power of each element in the product matrix to the first sum corresponding to the matrix row where the element is located, the sum of all elements in each matrix row in the exponential normalization matrix is 1. In this way, through exponential normalization processing, the similarity between each element of the input sequence and each element in the input sequence is converted into a standard weight distribution.

[0295] Specifically, the above steps 2910 to 2930 can be implemented by the following formula:

[0296]

[0297] wherein, x i represents the i-th element in a matrix row of the product matrix, e is the base number of natural logarithm, is the power corresponding to the i-th element in the matrix row, n represents the number of elements in a matrix row, represents the j-th element in the matrix row, represents the first sum of the matrix row;

[0298] The embodiments disclosed in steps 2910 to 2930 convert the product matrix into an exponential normalization matrix by converting each element in the product matrix into a corresponding attention weight by taking each element in the product matrix as an exponent of the first base, obtaining the power corresponding to each element, and for each matrix row, determining the first sum corresponding to the matrix row and converting each element in the product matrix into a corresponding attention weight based on the ratio of the power of each element in the matrix row to the first sum corresponding to the matrix row, thereby converting the product matrix into an exponential normalization matrix, and converting the similarity between each element of the input sequence and each element of the input sequence into a standard weight distribution, and thereby subsequent weighting of the value matrix block by the exponential normalization matrix can highlight the relatively important part of the value matrix block with higher weights and ignore the relatively unimportant part with lower weights.

[0299] Detailed description of step 430

[0300] In step 430, the first attention result sub-block corresponding to each processing head is inverse sub-block transformed to obtain a second attention result sub-block corresponding to each processing unit, and a multi-head attention result matrix is generated based on the second attention result sub-block corresponding to each processing unit.

[0301] The first attention result sub-block is an attention weight matrix obtained by attention calculation based on the second transformed sub-block, and the first attention result sub-block also has a first dimension and a second dimension. The details of the first dimension and the second dimension can be referred to the related description in the detailed description of step 410, which will not be described here. In one embodiment, referring to Figure 30 , step 430 includes:

[0302] Step 3010, for each processing head in each processing unit, the first attention result sub-block corresponding to the current processing head is divided into second sub-blocks corresponding to each processing unit according to the first number in the first dimension.

[0303] Step 3020, the plurality of second sub-blocks corresponding to a single processing unit are aggregated to obtain a second attention result sub-block corresponding to the current processing head in each processing unit.

[0304] Step 3030, the second attention result sub-blocks corresponding to each processing head in the processing unit are combined to obtain a second attention result sub-block corresponding to the processing unit.

[0305] In step 3010, the second patches are the parts of the first attention result sub-block corresponding to the sub-blocks of the processing units after the first attention result sub-block is cut along the first dimension. It can be understood that the input to the sub-blocks of the processing units is the sub-blocks corresponding to all hidden layers of a plurality of elements of the input sequence, while the first attention result sub-block calculated by the processing heads is the attention weights corresponding to a plurality of elements of all elements of the input sequence, which leads to the mismatch between the input and the output of each processing unit. Based on this, after obtaining the first attention result sub-block, the sub-block inverse transformation of the first attention result sub-block needs to be performed between the processing units to inverse transform the first attention result sub-block corresponding to each processing head in the processing unit to the output corresponding to the sub-block of the processing unit. Specifically, the first attention result sub-block is an attention weight matrix corresponding to a plurality of hidden layers of all elements of the input sequence, and each matrix row in the first attention result sub-block corresponds to an element of the input sequence, i.e., each matrix row in the first attention result sub-block corresponds to a matrix row of the input matrix respectively. Based on this, the first attention result sub-block can be cut into a first number of second patches in the same way as the input matrix is cut into the first number of sub-blocks. In this way, each second patch corresponds to a processing unit. For example, the input matrix includes 128 matrix rows, and the first attention result sub-block also includes 128 matrix rows. The input matrix is input into four processing units after being cut into four sub-blocks including 32 matrix rows along the first dimension, and at this time, the first attention result is cut into four second patches including 32 matrix rows along the first dimension. The four second patches correspond to the four processing units respectively.

[0306] In step 3020, the attention result sub-block is the attention weight matrix block corresponding to the sub-block of the current processing head. It can be understood that the sub-block of the current processing head refers to the sub-block corresponding to the current processing head obtained by cutting the first transformed sub-block along the second dimension in the step. It can be understood that after the first attention result sub-block corresponding to the current processing head in each processing unit is cut into the first number of second patches, the first number of second patches in the current processing head of each processing unit correspond to one processing unit respectively, and each processing unit corresponds to a second patch in all processing units respectively, for example, the processing device includes 16 processing units, which are processing unit 1 to processing unit 16 in turn, and the 16 second patches in processing unit 1 correspond to processing unit 1 to processing unit 16 respectively, while processing unit 1 to processing unit 16 each include a second patch corresponding to processing unit 1. At this time, each processing unit corresponds to the first number of second patches respectively, and these second patches correspond to the first patches sent to each processing unit by the current processing head one by one. Then, the second patches corresponding to the same processing unit are aggregated into the corresponding processing unit, and then the corresponding multiple second patches in the current processing head of the processing unit are spliced along the second dimension, so that the attention result sub-block corresponding to the sub-block of the current processing head can be obtained, for example, the current processing head corresponding sub-block is composed of the first 32 hidden layers of the sub-block sent to the processing unit, and the attention result sub-block is the attention weight matrix corresponding to the first 32 hidden layers of the sub-block of the processing unit.

[0307] It should be noted that since the attention calculation corresponding to each processing head of the same processing unit is performed in sequence, there will be a situation that the attention calculation corresponding to the previous processing head has been completed, while the attention calculation corresponding to the subsequent processing head has not yet started. Based on this, the sub-block inverse transformation of the first attention result sub-block corresponding to the previous processing head can be performed at the same time as the attention calculation of the subsequent processing head, so that the sub-block inverse transformation and the attention calculation corresponding to different processing heads are performed concurrently, thereby reducing the time cost of the sub-block inverse transformation and improving the efficiency of the multi-head attention processing.

[0308] In step 3030, after obtaining the attention result sub-block corresponding to each processing head of the processing unit, the attention result sub-block corresponding to each processing head is spliced along the second dimension, so that the attention weight matrix corresponding to the sub-block input to the processing unit, i.e., the second attention result sub-block, is obtained. Thus, the output of the processing unit corresponds to the input sub-block.

[0309] In the embodiments disclosed in steps 3010 to 3030, the first attention result sub-block corresponding to the current processing head of each processing unit is divided into the second patches corresponding to the first number of processing units by performing the sub-block inverse transform on the first attention result sub-block, so that each processing unit corresponds to one second patch in each processing unit. At this time, the plurality of second patches corresponding to the processing unit are aggregated to obtain the attention result patch corresponding to the current processing head of the processing unit. Then, the attention result patches corresponding to each processing head in the processing unit are spliced to obtain the second attention result sub-block corresponding to the sub-block input to the processing unit, so that the output of the processing unit matches the input. Since the process of calculating the sub-block inverse transform corresponding to the processing head of the first attention result sub-block is performed concurrently with the subsequent attention calculation corresponding to the processing head, the time overhead caused by the sub-block inverse transform can be reduced, and the efficiency of the multi-head attention processing can be improved.

[0310] With reference to Figure 31 In one embodiment, step 3020 includes:

[0311] Step 3110 determines the processing unit corresponding to the current processing head as the target processing unit, and determines the other processing units in the processing device except the target processing unit as the interactive processing units.

[0312] Step 3120 loads the second patches corresponding to each interactive processing unit into the sending buffer and loads the second patches corresponding to the target processing unit into the receiving buffer after the data in the sending buffer and the receiving buffer are taken out.

[0313] Step 3130 sends each second patch in the sending buffer to the corresponding interactive processing unit and receives the second patch corresponding to the target processing unit from each interactive processing unit by the communication thread of the target processing unit, writes the second patch into the receiving buffer, and splices each second patch in the receiving buffer along the second dimension into the second attention result sub-block corresponding to the current processing head.

[0314] In step 3110, the target processing unit refers to the processing unit to which the current processing head belongs. For example, if the current processing head is in processing unit 1, processing unit 1 is determined as the target processing unit. The description of the interactive processing unit can be referred to the detailed description in the step, which is not repeated here.

[0315] In step 3120, since the sub-block transformation corresponding to different processing heads and the attention calculation are performed in parallel, and the sub-block transformation corresponding to multiple processing heads is performed respectively, the attention calculation corresponding to the previous processing head has been completed, while the sub-block transformation corresponding to the following processing head has not been completed. At this time, if the second patch in the first attention result sub-block calculated by the previous processing head is filled into the sending buffer, the communication thread of the processing unit will be occupied, which causes the sub-block transformation corresponding to the following processing head to be unable to be performed in time, and the process of the sub-block transformation is interrupted, so that the attention calculation of the following processing head cannot be performed in time. Based on this, in the present disclosure, after the sub-block transformation of the sub-block corresponding to all processing heads is completed, the second patch to be sent to the interactive processing unit is written into the sending buffer, and the sub-block inverse transformation is performed on the first attention result sub-block corresponding to each processing head in turn. Specifically, referring to Figure 32 , the processing unit includes four processing heads, which are processing head 0 to processing head 3 in turn. After the sub-block transformation of the sub-block corresponding to the processing head 3 is completed, that is, after the sub-block transformation corresponding to all processing heads is completed, the attention calculation corresponding to the processing head 0 and the processing head 1 has also been completed, and the current calculation thread is performing the attention calculation of the processing head 2. At this time, the multiple second patches corresponding to the processing head 0 are written into the sending buffer, and the sub-block inverse transformation corresponding to the processing head 0 is started.

[0316] In step 3130, since the target processing unit also corresponds to a second patch in each interactive processing unit, the interactive processing unit sends the corresponding second patch to the target processing unit. At this time, since the data in the receiving buffer has been taken out, the received second patches can be temporarily stored in the receiving buffer and spliced into the second attention result sub-block corresponding to the current processing head.

[0317] The embodiments disclosed in steps 3110 to 3130 write the second patch to be sent to each interactive processing unit into the sending buffer, write the second patch corresponding to the target processing unit into the receiving buffer, then perform the sub-block exchange between each processing unit, send the second patch corresponding to the interactive processing unit to the corresponding interactive processing unit, receive the second patch corresponding to the target processing unit from the interactive processing unit, and splice the second patches in the receiving buffer in turn, so as to transform the first attention result sub-block of each processing head into the attention result block corresponding to the block of the processing head. In this way, it is ensured that the sub-block inverse transformation is started after all the sub-block transformations are completed, so that the sub-block transformation corresponding to each processing head is continuously performed, and the sub-block transformation and the attention calculation of the multiple processing heads can form a pipeline process, thereby ensuring that the efficiency of the multi-head attention processing will not be reduced.

[0318] In one embodiment, referring toFigure 33 , step 3020, comprising:

[0319] Step 3310, determining a second splicing order based on the hidden layers corresponding to each second patch;

[0320] Step 3320, splicing each second patch into a second attention result sub-block based on the second splicing order.

[0321] In step 3310, the second splicing order refers to the splicing order when each second patch corresponding to each processing unit is spliced into a second attention result sub-block corresponding to the processing unit. It can be understood that each second patch corresponding to the processing unit is actually an attention weight matrix block corresponding to each hidden layer of the sub-block corresponding to the processing unit. In order to accurately splice multiple second patches into an attention weight matrix corresponding to the sub-block of the processing head, it is necessary to determine the splicing order of the multiple second patches along the second dimension. Specifically, the hidden layer dimension of the second transformed sub-block corresponding to the processing head of each processing unit can be used to determine the hidden layer dimension of the first attention result sub-block corresponding to the processing head, i.e. the hidden layer dimension of each second patch of the processing head. For example, if the second transformed sub-block corresponding to the processing head is composed of the first patch corresponding to the first to eighth columns in each first transformed sub-block, then each second patch calculated by the processing head corresponds to the first to eighth hidden layers. After determining the hidden layers corresponding to each second patch, the second splicing order is determined based on the order from small to large of the corresponding hidden layers. For example, the second patch 1 corresponds to the first 8 hidden layers, the second patch 2 corresponds to the ninth to sixteenth hidden layers, and so on. The second patch P corresponds to the last 8 hidden layers in the sub-block of the processing head. Therefore, the second splicing order is to splice the second patch 1 to the second patch m along the second dimension in turn.

[0322] In step 3320, after determining the second splicing order, each second patch is spliced along the second dimension according to the second splicing order. In this way, the attention weight matrices of multiple different hidden layers corresponding to the same element in the input sequence are spliced, and a second attention result sub-block corresponding to the sub-block of the processing head is obtained.

[0323] The embodiments disclosed in steps 3310 to 3320 determine the second splicing order of each second patch based on the hidden layers corresponding to each second patch corresponding to the same processing unit, and then splice each second patch along the second dimension, thereby obtaining a second attention result sub-block corresponding to the sub-block of the current processing head of the processing unit. In this way, the accuracy of the attention weight corresponding to each hidden layer in the second attention result sub-block can be ensured.

[0324] Specific use process example of the multi-head attention processing method of the embodiments of the present disclosure

[0325] One specific use process of the multi-head attention processing method of the embodiments of the present disclosure will be described in detail below with reference to Figure 34 , which includes but is not limited to the following steps 3401 to 3438. In this process, it will be described that the server 140 independently undertakes all tasks in the multi-head attention processing, but those skilled in the art should understand that this process can also be completed independently by the terminal 110, or the terminal 110 and the server 140 undertake part of the tasks in the multi-head attention processing respectively.

[0326] 3401. Determine the amount of communication resources of each processing unit;

[0327] 3402. Based on the ratio of the amount of communication resources of each processing unit, split the input matrix into a first number of sub-blocks along the first dimension;

[0328] 3403. Determine the amount of computing resources of each processing unit;

[0329] 3404. Based on the ratio of the amount of computing resources of each processing unit, split the input matrix into a first number of sub-blocks along the first dimension;

[0330] 3405. Assign the first number of sub-blocks to the first number of processing units respectively;

[0331] 3406. Determine the query weight matrix, the key weight matrix and the value weight matrix corresponding to the processing unit;

[0332] 3407. Perform matrix multiplication on the sub-block corresponding to the processing unit and the query weight matrix, the key weight matrix and the value weight matrix corresponding to the processing unit respectively, to obtain the sub-query matrix, the sub-key matrix and the sub-value matrix corresponding to the sub-block, wherein the sub-query matrix, the sub-key matrix and the sub-value matrix constitute the first transformed sub-block corresponding to the processing unit;

[0333] 3408. Determine the first quotient of the third number and the second number;

[0334] 3409. In the processing unit, divide each first quotient of hidden layers into a sub-block along the second dimension, to obtain a second number of sub-blocks, wherein each sub-block corresponds to a processing head in the processing unit;

[0335] 3410. Determine the second quotient of the fourth number and the first number;

[0336] 3411. In the sub-block corresponding to the current processing head, divide each second quotient of hidden layers into a first tile along the second dimension, to obtain a plurality of first tiles;

[0337] 3412. From the plurality of first patches, a target first patch corresponding to the current processing head is determined, and the plurality of first patches other than the target first patch are determined as to-be-sent first patches;

[0338] 3413. The amount of computing resources of the processing unit is determined;

[0339] 3414. Based on the ratio of the amount of computing resources of each processing unit, the partition is divided into a first number of first patches corresponding to each processing unit in the second dimension;

[0340] 3415. The first patch corresponding to the processing unit of the current processing head is determined as the target first patch, and the other first patches other than the target first patch are determined as to-be-sent first patches;

[0341] 3416. In each processing unit, each processing head is sequentially taken as the current processing head;

[0342] 3417. When each to-be-sent first patch corresponding to the previous processing head of the current processing head in the sending buffer is sent to the corresponding interactive processing, each to-be-sent first patch corresponding to the current processing head is loaded into the sending buffer, and the data in the sending buffer is sent to the corresponding interactive processing unit through the communication thread of the processing unit;

[0343] 3418. When the data corresponding to the previous processing head of the current processing head in the receiving buffer is taken out, the target first patch corresponding to the current processing head is loaded into the receiving buffer, and the target first patch corresponding to the current processing head is received from each interactive processing unit through the communication thread and sequentially written into the receiving buffer;

[0344] 3419. Based on the position of the sub-block corresponding to each processing unit in the input matrix, a first splicing order is determined;

[0345] 3420. Based on the first splicing order, the target first patch sent by each interactive processing unit and the target first patch corresponding to the current processing head are spliced to obtain a second transformed patch corresponding to the current processing head;

[0346] 3421. After collecting each target first patch corresponding to the current processing head sent by each interactive processing unit in the receiving buffer, and splicing the plurality of target first patches into a second transformed sub-block, the second transformed sub-block is taken out from the receiving buffer and transmitted to the calculation buffer of the calculation thread of the processing unit;

[0347] 3422. After the calculation thread of the processing unit calculates the second transformed sub-block corresponding to the previous processing head of the current processing head as the corresponding first attention result sub-block, the second transformed sub-block corresponding to the current processing head is read from the calculation buffer.

[0348] 3423. The computing thread of the processing unit calculates a second transformed sub-block corresponding to the previous processing head of the current processing head as a corresponding first attention result sub-block, and reads the second transformed sub-block corresponding to the current processing head from the computing buffer after the calculation;

[0349] 3424. In the current processing head, transpose the key matrix block to obtain a transposed key matrix block;

[0350] 3425. Perform matrix multiplication on the query matrix block and the transposed key matrix block to obtain a product matrix;

[0351] 3426. Take each element in the product matrix as an exponent of a first base to obtain a power of each element;

[0352] 3427. For each matrix row in the product matrix, determine a first sum of the powers of the elements in the matrix row;

[0353] 3428. Determine the ratio of the power of each element to the first sum of the corresponding matrix row as an element of each matrix row in the exponent normalization matrix to obtain the exponent normalization matrix;

[0354] 3429. Perform matrix multiplication on the exponent normalization matrix and the value matrix block to obtain the first attention result sub-block;

[0355] 3430. For the current processing head in each processing unit, divide the first attention result sub-block corresponding to the current processing head into second tile blocks corresponding to each processing unit according to a first number in the first dimension;

[0356] 3431. Determine the processing unit corresponding to the current processing head as a target processing unit, and determine other processing units in the processing device except the target processing unit as interactive processing units;

[0357] 3432. After the data in the sending buffer and the receiving buffer are taken out, load the second tile blocks corresponding to each interactive processing unit into the sending buffer, and load the second tile blocks corresponding to the target processing unit into the receiving buffer;

[0358] 3433. Through the communication thread of the target processing unit, send each second tile block in the sending buffer to the corresponding interactive processing unit, and receive the second tile block corresponding to the target processing unit from each interactive processing unit and write it into the receiving buffer;

[0359] 3434. Determine a second splicing order based on the hidden layers corresponding to each second tile block;

[0360] 3435. Splice each second tile block into a second attention result sub-block based on the second splicing order;

[0361] 3436. Merge the second attention results corresponding to each processing head in the processing unit to obtain a second attention result sub-block corresponding to the processing unit;

[0362] 3437. Aggregate the second attention result sub-blocks corresponding to each processing unit into a to-be-corrected matrix;

[0363] 3438. Generate a multi-head attention result matrix based on the to-be-corrected matrix and a correction coefficient matrix.

[0364] It can be understood that, although each step in each of the above flowcharts is displayed in sequence according to the representation of the arrow, these steps are not necessarily executed in the order represented by the arrow. Unless otherwise specified in the embodiments, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowcharts can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0365] It should be noted that, in each specific embodiment of the present application, when it is necessary to perform relevant processing according to target object attribute information or attribute information set and other data related to the characteristics of the target object, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain target object attribute information, a separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the target object, the necessary target object related data for enabling the embodiments of the present application to operate normally will be obtained.

[0366] Device and equipment description of the embodiments of the present disclosure

[0367] Reference Figure 35 , Figure 35 The structure schematic diagram of the multi-head attention processing device 3500 provided by the embodiments of the present disclosure includes:

[0368] The first linear transformation unit 3510 is configured to divide the input matrix into a first number of sub-blocks, and respectively send the first number of sub-blocks to the first number of processing units to perform first linear transformation, to obtain a first number of first transformed sub-blocks corresponding to the first number of processing units;

[0369] The first attention calculation unit 3520 is configured to perform sub-block transformation on the first number of first transformed sub-blocks corresponding to the first number of processing units, sequentially obtain a second transformed sub-block corresponding to each processing head, and input the second transformed sub-block into the processing head corresponding to the second transformed sub-block to perform attention calculation, to obtain a first attention result sub-block corresponding to each processing head, wherein the second number of processing heads are in order, and the sub-block transformation of the next processing head of the current processing head is performed while the attention calculation of the second transformed sub-block corresponding to the current processing head in the second number of processing heads is performed.

[0370] The first matrix splicing unit 3530 is configured to perform sub-block inverse transformation on the first attention result sub-blocks corresponding to the processing heads to obtain a second attention result sub-block corresponding to each processing unit, and generate a multi-head attention result matrix based on the second attention result sub-blocks corresponding to the processing units.

[0371] Optionally, the first linear transformation unit 3510 is configured to:

[0372] determine a query weight matrix, a key weight matrix, and a value weight matrix corresponding to the processing unit;

[0373] perform matrix multiplication on the sub-block corresponding to the processing unit and the query weight matrix, the key weight matrix, and the value weight matrix corresponding to the processing unit respectively, to obtain a sub-query matrix, a sub-key matrix, and a sub-value matrix corresponding to the sub-block, wherein the sub-query matrix, the sub-key matrix, and the sub-value matrix constitute the first transformed sub-block corresponding to the processing unit.

[0374] Optionally, the input matrix has a first dimension and a second dimension, and the first linear transformation unit 3510 is specifically configured to:

[0375] divide the input matrix into the first number of sub-blocks in the first dimension;

[0376] The first attention calculation unit 3520 is configured to:

[0377] divide the first transformed sub-blocks into sub-blocks corresponding to the processing heads according to the second number in the second dimension;

[0378] divide the sub-blocks into the first number of first patches corresponding to the first number of processing units in the second dimension;

[0379] for a single processing head of a single processing unit, aggregate the first patches corresponding to the single processing head in each first transformed sub-block to obtain a second transformed sub-block corresponding to the single processing head.

[0380] Optionally, the first linear transformation unit 3510 is further configured to:

[0381] determine a communication resource amount of each processing unit;

[0382] split the input matrix into the first number of sub-blocks along the first dimension based on a ratio of the communication resource amounts of the processing units.

[0383] Optionally, the first linear transformation unit 3510 is further configured to:

[0384] determine the computation resource amounts of the processing units;

[0385] split the input matrix into the first number of sub-blocks along the first dimension based on a ratio of the computation resource amounts of the processing units.

[0386] Optionally, the first attention calculation unit 3520 is specifically configured to:

[0387] determine a second quotient of the fourth number and the first number;

[0388] divide each second quotient number of hidden layers in the sub-block corresponding to the current processing head along the second dimension into a first patch to obtain a plurality of first patches;

[0389] determine a target first patch corresponding to the current processing head from the plurality of first patches, and determine the plurality of first patches other than the target first patch as to-be-sent first patches.

[0390] Optionally, the first attention calculation unit 3520 is specifically configured to:

[0391] determine the computation resource amounts of the processing units;

[0392] split the sub-block into the first number of first patches corresponding to the processing units along the second dimension based on a ratio of the computation resource amounts of the processing units;

[0393] determine the first patch corresponding to the processing unit of the current processing head as the target first patch, and determine the other first patches other than the target first patch as to-be-sent first patches.

[0394] Optionally, the first attention calculation unit 3520 is further configured to:

[0395] in the processing units, sequentially take each processing head as the current processing head;

[0396] send the data of each to-be-sent first patch in the sub-block corresponding to the current processing head to the interaction processing unit respectively, and receive the target first patch corresponding to the current processing head from each interaction processing unit, wherein the interaction processing unit is other than the processing unit corresponding to the current processing head;

[0397] splice the plurality of target first patches corresponding to the current processing head along the first dimension to obtain a second transformed sub-block corresponding to the current processing head.

[0398] Optionally, the processing unit comprises a sending buffer and a receiving buffer, and the first attention calculation unit 3520 is further configured to:

[0399] When each of the target first patch corresponding to the previous processing head of the current processing head in the sending buffer is sent to the corresponding interactive processing, the processing unit loads each of the target first patch corresponding to the current processing head into the sending buffer, and sends the data in the sending buffer to the corresponding interactive processing unit through the communication thread of the processing unit;

[0400] When each of the data corresponding to the previous processing head of the current processing head in the receiving buffer is taken out, the processing unit loads the target first patch corresponding to the current processing head into the receiving buffer, receives the target first patch corresponding to the current processing head from each of the interactive processing units through the communication thread, and writes the target first patch into the receiving buffer in sequence.

[0401] Optionally, the first attention calculation unit 3520 is further configured to:

[0402] After each of the target first patch corresponding to the current processing head sent by each of the interactive processing units is collected in the receiving buffer, and the plurality of target first patches are spliced into a second transformed subpatch, the second transformed subpatch is taken out from the receiving buffer and transmitted to the calculation buffer of the calculation thread of the processing unit;

[0403] After the calculation thread of the processing unit calculates the second transformed subpatch corresponding to the previous processing head of the current processing head as the corresponding first attention result subpatch, the calculation thread reads the second transformed subpatch corresponding to the current processing head from the calculation buffer.

[0404] Optionally, the first attention calculation unit 3520 is further configured to:

[0405] Based on the positions of the subpatches corresponding to each of the processing units in the input matrix, a first splicing order is determined;

[0406] Based on the first splicing order, the target first patch sent by each of the interactive processing units and the target first patch corresponding to the current processing head are spliced to obtain the second transformed patch corresponding to the current processing head.

[0407] Optionally, the second transformed subpatch comprises a query matrix block, a key matrix block and a value matrix block, and the first attention calculation unit 3520 is further configured to:

[0408] In the current processing head, the key matrix block is transposed to obtain a transposed key matrix block;

[0409] The query matrix block and the transposed key matrix block are multiplied to obtain a product matrix;

[0410] The product matrix is subjected to exponential normalization to obtain an exponential normalization matrix;

[0411] The exponential normalization matrix and the value matrix block are subjected to matrix multiplication to obtain a first attention result sub-block.

[0412] Optionally, the first attention calculation unit 3520 is further configured to:

[0413] Each element in the product matrix is taken as an exponent of a first base to obtain a power of each element;

[0414] For each matrix row in the product matrix, a first sum of the powers of the elements in the matrix row is determined;

[0415] A ratio of the power of each element and the first sum of the corresponding matrix row is determined as an element of each matrix row in the exponential normalization matrix to obtain the exponential normalization matrix.

[0416] Optionally, the first matrix splicing unit 3530 is further configured to:

[0417] For a current processing head in each processing unit, the first attention result sub-block corresponding to the current processing head is divided into second patches corresponding to each processing unit according to a first number in the first dimension;

[0418] The second patches corresponding to a single processing unit are aggregated to obtain a second attention result sub-block corresponding to the current processing head in each processing unit;

[0419] The second attention result sub-blocks corresponding to each processing head in the processing unit are combined to obtain a second attention result sub-block corresponding to the processing unit.

[0420] Optionally, the first matrix splicing unit 3530 is further configured to:

[0421] The processing unit corresponding to the current processing head is determined as a target processing unit, and other processing units in the processing device except the target processing unit are determined as interactive processing units;

[0422] After the data in the sending buffer and the receiving buffer are taken out, the second patches corresponding to each interactive processing unit are loaded into the sending buffer, and the second patches corresponding to the target processing unit are loaded into the receiving buffer;

[0423] Through the communication thread of the target processing unit, each second patch in the sending buffer is sent to the corresponding interactive processing unit, and the second patch corresponding to the target processing unit is received from each interactive processing unit and written into the receiving buffer, and each second patch in the receiving buffer is spliced into the second attention result sub-block corresponding to the current processing head along the second dimension.

[0424] Optionally, the first matrix splicing unit 3530 is further configured to:

[0425] determine the second splicing order based on the hidden layers corresponding to the second patches;

[0426] splice the second patches into the second attention result blocks based on the second splicing order.

[0427] Optionally, the first matrix splicing unit 3530 is further configured to:

[0428] aggregate the second attention result sub-blocks corresponding to each processing unit into a to-be-corrected matrix;

[0429] generate the multi-head attention result matrix based on the to-be-corrected matrix and the correction coefficient matrix.

[0430] Referring to Figure 36 , Figure 36 A structure block diagram of a part of a terminal for implementing the multi-head attention processing method of the embodiments of the present disclosure is shown, and the terminal includes a radio frequency (RF) circuit 3610, a memory 3615, an input unit 3630, a display unit 3640, a sensor 3650, an audio circuit 3660, a wireless fidelity (WiFi) module 3670, a processor 3680, and a power supply 3690, and the like. Those skilled in the art can understand that Figure 36 The terminal structure shown does not constitute a limitation on mobile phones or computers, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0431] The RF circuit 3610 can be used for receiving and sending signals in the process of receiving or calling, and in particular, receiving the downlink information of the base station and processing it by the processor 3680; in addition, sending the uplink data designed to the base station.

[0432] The memory 3615 can be used to store software programs and modules, and the processor 3680 can execute various functions and data processing of the content terminal by running the software programs and modules stored in the memory 3615.

[0433] The input unit 3630 can be used to receive input digital or character information, and generate key signal input related to the setting and function control of the content terminal. Specifically, the input unit 3630 can include a touch panel 3631 and other input devices 3632.

[0434] The display unit 3640 can be used to display input information or provided information and various menus of the content terminal. The display unit 3640 can include a display panel 3641.

[0435] The audio circuit 3660, the speaker 3661, and the microphone 3662 can provide an audio interface.

[0436] In this embodiment, the processor 3680 included in the terminal can perform the multi-head attention processing method of the foregoing embodiments.

[0437] The terminal of the embodiments of the present disclosure includes but is not limited to a mobile phone, a computer, a smart voice interactive device, a smart home appliance, a vehicle-mounted terminal, an aircraft, and the like. The embodiments of the present disclosure can be applied to various scenarios, including but not limited to an artificial intelligence assistant, machine translation, and the like.

[0438] Figure 37 A structural block diagram of a part of the server 140 for implementing the multi-head attention processing method of the embodiments of the present disclosure. The server 140 can have a relatively large difference due to different configurations or performances, and can include one or more central processing units (CPUs) 3722 (for example, one or more processors) and a memory 3732, one or more storage media 3730 (for example, one or more mass storage devices) storing application programs 3742 or data 3744. Among them, the memory 3732 and the storage medium 3730 can be temporary storage or persistent storage. The programs stored in the storage medium 3730 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server 140. Further, the central processing unit 3722 can be configured to communicate with the storage medium 3730 and execute a series of instruction operations in the storage medium 3730 on the server 140.

[0439] The server 140 can also include one or more power supplies 3726, one or more wired or wireless network interfaces 3750, one or more input / output interfaces 3758, and / or one or more operating systems 3741, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and the like.

[0440] The central processing unit 3722 in the server 140 can be used to execute the multi-head attention processing method of the embodiments of the present disclosure.

[0441] The embodiments of the present disclosure also provide a computer-readable storage medium for storing program codes, the program codes being used to execute the multi-head attention processing method of the foregoing embodiments.

[0442] The embodiment of the present disclosure further provides a computer program product comprising a computer program. A processor of a computer device reads the computer program and executes, so that the computer device executes the multi-head attention processing method.

[0443] The terms "first", "second", "third", "fourth" and the like in the description of the present disclosure and the above drawings, if any, are used to distinguish similar objects, and do not necessarily have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprise" and "include" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0444] It should be understood that in the present disclosure, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association between the associated contents, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated contents before and after are a "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0445] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, more than, etc. are not included in the number, above, below, etc. are understood to include the number.

[0446] In several embodiments provided by the present disclosure, it should be understood that the disclosed system, apparatus and method can be implemented in other manners. For example, the apparatus embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0447] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0448] In addition, each functional unit in the various embodiments of the present disclosure can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0449] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present disclosure essentially or the part that contributes to the related art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the various embodiments of the present disclosure. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0450] It should also be understood that the various embodiments provided by the present disclosure can be combined in any manner to achieve different technical effects.

[0451] The above is a specific explanation of the embodiments of the present disclosure, but the present disclosure is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present disclosure, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present disclosure.

Claims

1. A multi-head attention processing method, comprising: The method is applied to a processing device including a first number of processing units, each processing unit including a second number of processing heads, and the method includes: dividing an input matrix into a first number of sub-blocks, performing a first linear transformation on the first number of sub-blocks respectively by the first number of processing units to obtain a first number of first transformed sub-blocks corresponding to the first number of processing units; performing sub-block transformation on the first number of first transformed sub-blocks corresponding to the first number of processing units to obtain second transformed sub-blocks corresponding to each processing head in turn, and inputting the second transformed sub-blocks to the processing head corresponding to the second transformed sub-blocks to perform attention calculation to obtain first attention result sub-blocks corresponding to each processing head, wherein the second number of processing heads are in order, and the sub-block transformation of the next processing head of the current processing head is performed while the attention calculation of the second transformed sub-block corresponding to the current processing head in the second number of processing heads is performed; performing sub-block inverse transformation on the first attention result sub-blocks corresponding to each processing head to obtain second attention result sub-blocks corresponding to each processing unit, and generating a multi-head attention result matrix based on the second attention result sub-blocks corresponding to each processing unit.

2. The multi-head attention processing method of claim 1, wherein, The performing of the first linear transformation includes: determining a query weight matrix, a key weight matrix and a value weight matrix corresponding to the processing unit; performing matrix multiplication on the sub-block corresponding to the processing unit and the query weight matrix, the key weight matrix and the value weight matrix corresponding to the processing unit respectively to obtain a sub-query matrix, a sub-key matrix and a sub-value matrix corresponding to the sub-block, wherein the sub-query matrix, the sub-key matrix and the sub-value matrix constitute the first transformed sub-block corresponding to the processing unit.

3. The multi-head attention processing method of claim 1, wherein, The input matrix has a first dimension and a second dimension; The dividing of the input matrix into a first number of sub-blocks includes: dividing the input matrix into a first number of sub-blocks in the first dimension; The performing of the sub-block transformation on the first number of first transformed sub-blocks corresponding to the first number of processing units to obtain second transformed sub-blocks corresponding to each processing head includes: dividing the first transformed sub-blocks into sub-blocks corresponding to each processing head in the second dimension according to the second number; dividing the sub-blocks into a first number of first patches corresponding to the first number of processing units in the second dimension; for a single processing head of a single processing unit, aggregating the first patches corresponding to the single processing head in each first transformed sub-block to obtain the second transformed sub-block corresponding to the single processing head.

4. The multi-head attention processing method of claim 3, wherein, The dividing of the input matrix into a first number of sub-blocks in the first dimension includes: determining the amount of communication resources of each processing unit; based on the ratio of the amount of communication resources of each processing unit, cutting the input matrix into a first number of sub-blocks along the first dimension.

5. The multi-head attention processing method of claim 3, wherein, The dividing of the input matrix into a first number of sub-blocks in the first dimension includes: determining the amount of computing resources of each processing unit; The input matrix is split into a first number of sub-blocks along the first dimension based on a ratio of computing resource amounts of each processing unit. 6.The multi-head attention processing method of claim 3, wherein the second dimension of the first transformed sub-block comprises a third number of hidden layers. The splitting the first transformed sub-blocks into the second number of sub-blocks corresponding to each processing head along the second dimension includes: determining a first quotient of the third number and the second number; in the processing unit, dividing each first quotient of hidden layers into one sub-block along the second dimension in the sub-block corresponding to the current processing head, to obtain a second number of first sub-blocks, wherein each first sub-block corresponds to one processing head in the processing unit.

7. The multi-headed attention processing method of claim 3, the second dimension of the block comprising a fourth number of hidden layers. The splitting the first transformed sub-blocks into the second number of sub-blocks corresponding to each processing head along the second dimension includes: determining a second quotient of the fourth number and the first number; in the sub-block corresponding to the current processing head, dividing each second quotient of hidden layers into one first tile along the second dimension, to obtain a plurality of first tiles; from the plurality of first tiles, determining a target first tile corresponding to the current processing head, and determining a plurality of first tiles other than the target first tile as to-be-sent first tiles.

8. The multi-head attention processing method of claim 7, wherein, The aggregating the first tiles corresponding to the single processing head in each first transformed sub-block to obtain the second transformed sub-block corresponding to the single processing head includes: in each processing unit, sequentially taking each processing head as a current processing head; sending data of each to-be-sent first tile in the sub-block corresponding to the current processing head to an interactive processing unit respectively, and receiving a target first tile corresponding to the current processing head from each interactive processing unit, wherein the interactive processing unit is another processing unit other than the processing unit corresponding to the current processing head; splicing a plurality of target first tiles corresponding to the current processing head along the first dimension to obtain the second transformed sub-block corresponding to the current processing head.

9. The multi-headed attention processing method of claim 8, the processing unit comprising a transmit buffer and a receive buffer, and wherein, The sending data of each to-be-sent first tile in the sub-block corresponding to the current processing head to an interactive processing unit respectively, and receiving a target first tile corresponding to the current processing head from each interactive processing unit includes: when each to-be-sent first tile corresponding to a previous processing head of the current processing head in the sending buffer is sent to a corresponding interactive processing unit, loading each to-be-sent first tile corresponding to the current processing head in the sending buffer to the sending buffer, and sending data in the sending buffer to a corresponding interactive processing unit through a communication thread of the processing unit; when data corresponding to the previous processing head of the current processing head in the receiving buffer is taken out, loading the target first tile corresponding to the current processing head to the receiving buffer, receiving a target first tile corresponding to the current processing head from each interactive processing unit through the communication thread, and sequentially writing to the receiving buffer.

10. The multi-head attention processing method of claim 8, wherein, The first dimension splicing of the target first patches corresponding to the current processing head includes: determining a first splicing order based on the positions of the sub-patches corresponding to each processing unit in the input matrix; splicing the target first patches sent by each interaction processing unit and the target first patches corresponding to the current processing head based on the first splicing order to obtain the second transformed patch corresponding to the current processing head. 11.The multi-head attention processing method of claim 1, wherein the second transformed sub-blocks comprise a query matrix block, a key matrix block, and a value matrix block. The attention calculation of the second transformed sub-patch in the processing head corresponding to the second transformed sub-patch includes: transposing the key matrix block in the current processing head to obtain a transposed key matrix block; performing matrix multiplication on the query matrix block and the transposed key matrix block to obtain a product matrix; performing exponential normalization processing on the product matrix to obtain an exponential normalization matrix; performing matrix multiplication on the exponential normalization matrix and the value matrix block to obtain the first attention result sub-patch.

12. The multi-head attention processing method of claim 11, wherein, The exponential normalization processing of the product matrix to obtain an exponential normalization matrix includes: taking each element in the product matrix as an exponent of a first base to obtain the power of each element; determining the first sum of the powers of the elements in each matrix row in the product matrix; determining the ratio of the power of each element to the first sum of the corresponding matrix row as the element of each matrix row in the exponential normalization matrix to obtain the exponential normalization matrix.

13. The multi-headed attention processing method of claim 1, the first attention result subblock having a first dimension and a second dimension, wherein, The sub-patch inverse transformation of the first attention result sub-patch corresponding to each processing head to obtain the second attention result sub-patch corresponding to each processing unit includes: for the current processing head in each processing unit, dividing the first attention result sub-patch corresponding to the current processing head into second patches corresponding to each processing unit in the first number in the first dimension; aggregating the second patches corresponding to a single processing unit to obtain the second attention result sub-patch corresponding to the current processing head in each processing unit; combining the second attention result sub-patches corresponding to each processing head in the processing unit to obtain the second attention result sub-patch corresponding to the processing unit.

14. The multi-headed attention processing method of claim 13, the processing unit comprising a transmit buffer and a receive buffer, and wherein, The aggregation of the second patches corresponding to a single processing unit to obtain the second attention result sub-patch corresponding to the current processing head in each processing unit includes: determining the processing unit corresponding to the current processing head as a target processing unit, and determining other processing units in the processing device except the target processing unit as interaction processing units; after the data in the sending buffer and the receiving buffer are taken out, loading the second patches corresponding to each interaction processing unit into the sending buffer and loading the second patches corresponding to the target processing unit into the receiving buffer; and performing sub-patch inverse transformation on the first attention result sub-patch corresponding to each processing head to obtain the second attention result sub-patch corresponding to each processing unit. The communication thread of the target processing unit sends each second patch in the sending buffer to the corresponding interactive processing unit, and receives the second patch corresponding to the target processing unit from each interactive processing unit, writes into the receiving buffer, and splices each second patch in the receiving buffer into the second attention result sub-block corresponding to the current processing head along the second dimension.

15. The multi-headed attention processing method of claim 13, wherein, The aggregating of the plurality of second patches corresponding to a single processing unit includes: determining a second splicing order based on the hidden layer corresponding to each second patch; splicing each second patch into the second attention result sub-block based on the second splicing order.

16. The multi-headed attention processing method of claim 1, wherein, The generating of the multi-head attention result matrix based on the second attention result sub-block corresponding to each processing unit includes: aggregating the second attention result sub-block corresponding to each processing unit into a to-be-corrected matrix; generating the multi-head attention result matrix based on the to-be-corrected matrix and a correction coefficient matrix.

17. A multi-headed attention processing device, comprising: The apparatus includes: a first linear transformation unit, configured to divide an input matrix into a first number of sub-blocks, and respectively send the first number of sub-blocks to the first number of processing units to perform first linear transformation to obtain a first number of first transformed sub-blocks corresponding to the first number of processing units; a first attention calculation unit, configured to perform sub-block transformation on the first number of first transformed sub-blocks corresponding to the first number of processing units to sequentially obtain a second transformed sub-block corresponding to each processing head, and input the second transformed sub-block to the processing head corresponding to the second transformed sub-block to perform attention calculation to obtain a first attention result sub-block corresponding to each processing head, wherein the second number of processing heads are in order, and the sub-block transformation of a next processing head of a current processing head is performed at the same time as the attention calculation of the second transformed sub-block corresponding to the current processing head; a first matrix splicing unit, configured to perform sub-block inverse transformation on the first attention result sub-block corresponding to each processing head to obtain a second attention result sub-block corresponding to each processing unit, and generate a multi-head attention result matrix based on the second attention result sub-block corresponding to each processing unit.

18. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the multi-head attention processing method according to any one of claims 1 to 16.

19. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the multi-head attention processing method according to any one of claims 1 to 16.

20. A computer program product, comprising a computer program, which is read and executed by a processor of a computer device, so that the computer device performs the multi-head attention processing method according to any one of claims 1 to 16.

Images