Data processing method and related equipment

Abstract

The embodiment of the invention discloses a data processing method. The method is applied to a decoder of an attention mechanism. In the method, an intermediate parameter stored in the process of decoding a first word vector to an (i-1) th word vector by using an attention mechanism is firstly obtained, then the ith word vector is incrementally decoded according to the intermediate parameter to obtain an (i + 1) th word vector, and the first word vector to the (i + 1) th word vector are used for generating data. The intermediate parameter in the historical decoding process is acquired, and the current word vector is incrementally decoded based on the intermediate parameter, so that repeated decoding of the historical word vector is reduced. For example, in the ith decoding process, intermediate parameters in the historical decoding process are used in an increment mode, the intermediate parameters in the historical decoding process are stored, and the intermediate parameters are directly used in the current decoding process, so that the repeated calculation amount and complexity in the decoding process are reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular, to a data processing method and related equipment. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] At present, transformers are commonly used in the basic architecture of generative AI models. Under this architecture, the decoding process of large language models (LLMs) mainly relies on autoregression. Each decoding step can only produce one output tensor (token), and the historical output content needs to be spliced ​​and re-used as the input of the LLM before the next decoding step. For example, when generating the i-th token, the input information of the previous i-1 tokens is required, resulting in excessive decoding calculations.

[0004] Therefore, how to reduce the amount of calculation in the decoding process is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a data processing method and related devices for reducing the complexity of a decoding process.

[0006] The first aspect of the embodiment of the present application provides a data processing method, which is applied to the decoder of the attention mechanism. The method can be executed by a data processing device, or the method is executed by some components in the data processing device (such as a processor, a chip or a chip system, etc.), or the method can also be implemented by a logic module or software that can realize all or part of the data processing function. In this method, the intermediate parameters stored in the process of decoding the 1st to i-1th word vectors using the attention mechanism are first obtained, and then the i-th word vector is incrementally decoded according to the intermediate parameters to obtain the i+1th word vector, and the 1st word vector to the i+1th word vector are used to generate data.

[0007] In the embodiment of the present application, the intermediate parameters in the historical decoding process are obtained, and the current word vector is decoded incrementally based on the intermediate parameters to reduce repeated decoding of the historical word vector. For example, in the i-th decoding process, the intermediate parameters in the historical decoding process are used incrementally, the intermediate parameters in the historical decoding process are stored, and the intermediate parameters are directly used in the current decoding process, thereby reducing the amount of repeated calculations and complexity in the decoding process.

[0008] Optionally, in a possible implementation of the first aspect, the above-mentioned intermediate parameters include at least one of the following: a first word embedding matrix corresponding to the i-1th word vector, the first word embedding matrix is ​​i-1 rows and d columns; a first matrix Q, a first matrix K, and a first attention matrix corresponding to the 1st to i-1th word vectors, the first attention matrix is ​​obtained based on the first matrix Q and the first matrix K; a score obtained by normalizing the first attention matrix, the normalization of the first attention matrix belongs to the process of decoding the i-1th word vector, the numerator of the fraction is the autocorrelation index value of the i-1th word vector, and the denominator of the fraction is the sum of the autocorrelations of the 1st to i-1th word vectors; an intermediate value matrix obtained by normalizing the first attention matrix, the intermediate value matrix includes a rows and b columns, the last column records the sum of the first b-1 columns of each row, and the intermediate value matrix is ​​obtained based on the first attention matrix; the first product result of the first matrix V corresponding to the 1st to i-1th word vectors and the first weight matrix.

[0009] In this possible implementation, during the decoding process, redundant calculations of the corresponding calculation process can be reduced by caching different intermediate parameters. The calculation process includes at least one of the following: calculating the KQV matrix, calculating the attention matrix, normalizing the attention matrix, calculating the product of the matrix V and the weight matrix, etc.

[0010] Optionally, in a possible implementation of the first aspect, the above-mentioned intermediate parameters include: a first matrix Q, a first matrix K and a first attention matrix; the method also includes: obtaining a second matrix Q, a second matrix K and a second matrix V of the i-th word vector; incrementally decoding the i-th word vector based on the intermediate parameters, including: obtaining a second attention matrix based on the intermediate parameters, the second matrix Q and the second matrix K; decoding the i-th word vector based on the second attention matrix and the second matrix V.

[0011] In this possible implementation, compared to the prior art method of calculating the attention matrix by multiplying the full amount K by the full amount Q in the process of decoding the i-th word vector, the embodiment of the present application splits the multiplication of the full amount K by the full amount Q, caches the intermediate calculation state of historical decoding, realizes the transpose of K and the incremental multiplication of Q, and greatly reduces the repeated calculation amount caused by repeated variable-length matrix multiplication. Or it can be understood that since the QV matrix is ​​fixed for the same input sequence. When the transpose of K is multiplied by Q, the generated attention matrix tends to expand by row / column, and the i-1 row is fixed, so that the historical attention matrix can be cached, and the newly added part can be calculated and spliced ​​with the historical attention matrix accordingly.

[0012] Optionally, in a possible implementation of the first aspect, the above steps: obtaining the second attention matrix based on the intermediate parameters, the second matrix Q and the second matrix K, include: multiplying the transposed matrix of the first matrix K by the second matrix Q to obtain a first matrix block; multiplying the transposed matrix of the second matrix K by the first matrix Q to obtain a second matrix block; multiplying the transposed matrix of the second matrix K by the second matrix Q to obtain a third matrix block; concatenating the first matrix block, the second matrix block, the third matrix block and the first attention matrix to obtain the second attention matrix, the first matrix block serving as the newly added columns of the second attention matrix compared to the first attention matrix, the second matrix block serving as the newly added rows of the second attention matrix compared to the first attention matrix, and the third matrix block serving as the newly added matrix block of the second attention matrix compared to the first attention matrix.

[0013] In this possible implementation, since the historical attention matrix in the overall attention matrix has been calculated before decoding the i-th word vector, the historical attention matrix can be cached and only the newly added part can be calculated, thereby reducing the need to recalculate the historical attention matrix during the decoding of the i-th word vector.

[0014] Optionally, in a possible implementation of the first aspect, the above-mentioned intermediate parameters also include: a score and a first product result; decoding the i-th word vector based on the second attention matrix and the second matrix V, including: adding the autocorrelation index value of the i-th word vector in the denominator to update the score; calculating the second product result of the second matrix V and the first weight matrix; decoding the i-th word vector based on the updated score, the first product result and the second product result.

[0015] In this possible implementation, compared to the prior art method of repeating exponential operations in each step of the attention matrix normalization process, the embodiment of the present application can use the historical intermediate results for subsequent decoding processes by adding the autocorrelation index value of the new word vector to the denominator of the historical fraction. Greatly reduce repeated exponential calculations. In other words, the numerator and denominator in the normalization process have been calculated before decoding the i-th word vector, and the fraction can be cached by only adding the newly added part to the denominator. Thereby reducing the need to recalculate the historical part of the numerator and denominator during the decoding of the i-th word vector.

[0016] Optionally, in a possible implementation of the first aspect, the above step: before decoding the i-th word vector based on the intermediate parameter increment, the method also includes: obtaining a second word embedding matrix corresponding to the i-th word vector, the second word embedding matrix is ​​1 row and d columns; obtaining the second matrix Q, the second matrix K and the second matrix V of the i-th word vector, including: obtaining the second matrix Q, the second matrix K and the second matrix V based on the second word embedding matrix and the second weight matrix; the number of rows of the second matrix Q, the second matrix K and the second matrix V is 1.

[0017] In this possible implementation, the dynamic length sequence is replaced by an incremental sequence (sequence padding is removed). Compared with the prior art that uses multiple rows of word embedding matrices to calculate the overall KQV in the process of decoding the i-th word vector, the decoder in the embodiment of the present application inputs a row of the second word embedding matrix (also referred to as the newly added word embedding matrix), and then multiplies the second word embedding matrix with the second weight matrix to obtain the incremental KQV matrix (i.e., the second matrix Q, the second matrix K, and the second matrix V). In other words, the historical KQV matrix in the overall KQV matrix has been calculated before decoding the i-th word vector, and the historical KQV matrix can be cached to only calculate the newly added part. This reduces the need to recalculate the historical KQV matrix during decoding the i-th word vector.

[0018] A second aspect of an embodiment of the present application provides a data processing device, which is applied to a decoder of an attention mechanism, and the data processing device includes: an acquisition unit, used to obtain intermediate parameters, which are parameters stored in the process of decoding the 1st to i-1th word vectors based on the attention mechanism; a decoding unit, used to incrementally decode the i-th word vector based on the intermediate parameters to obtain the i+1th word vector, and the 1st to i+1th word vectors are used to generate data.

[0019] Optionally, in a possible implementation of the second aspect, the above-mentioned intermediate parameters include at least one of the following: a first word embedding matrix corresponding to the i-1th word vector, the first word embedding matrix is ​​i-1 rows and d columns; a first matrix Q, a first matrix K, and a first attention matrix corresponding to the 1st to i-1th word vectors, the first attention matrix is ​​obtained based on the first matrix Q and the first matrix K; a score obtained by normalizing the first attention matrix, the normalization of the first attention matrix belongs to the process of decoding the i-1th word vector, the numerator of the fraction is the autocorrelation index value of the i-1th word vector, and the denominator of the fraction is the sum of the autocorrelations of the 1st to i-1th word vectors; an intermediate value matrix obtained by normalizing the first attention matrix, the intermediate value matrix includes a rows and b columns, the last column records the sum of the first b-1 columns of each row, and the intermediate value matrix is ​​obtained based on the first attention matrix; the first product result of the first matrix V corresponding to the 1st to i-1th word vectors and the first weight matrix.

[0020] Optionally, in a possible implementation of the second aspect, the above-mentioned intermediate parameters include; a first matrix Q, a first matrix K and a first attention matrix; an acquisition unit, also used to obtain the second matrix Q, the second matrix K and the second matrix V of the i-th word vector; a decoding unit, specifically used to obtain the second attention matrix based on the intermediate parameters, the second matrix Q and the second matrix K; a decoding unit, specifically used to decode the i-th word vector based on the second attention matrix and the second matrix V.

[0021] Optionally, in a possible implementation of the second aspect, the above-mentioned decoding unit is specifically used to multiply the transposed matrix of the first matrix K with the second matrix Q to obtain a first matrix block; the decoding unit is specifically used to multiply the transposed matrix of the second matrix K with the first matrix Q to obtain a second matrix block; the decoding unit is specifically used to multiply the transposed matrix of the second matrix K with the second matrix Q to obtain a third matrix block; the decoding unit is specifically used to concatenate the first matrix block, the second matrix block, the third matrix block and the first attention matrix to obtain a second attention matrix, the first matrix block serves as the newly added columns of the second attention matrix compared to the first attention matrix, the second matrix block serves as the newly added rows of the second attention matrix compared to the first attention matrix, and the third matrix block serves as the newly added matrix block of the second attention matrix compared to the first attention matrix.

[0022] Optionally, in a possible implementation of the second aspect, the above-mentioned intermediate parameters also include: a score and a first product result; a decoding unit, specifically used to add the autocorrelation index value of the i-th word vector in the denominator to update the score; a decoding unit, specifically used to calculate the second product result of the second matrix V and the first weight matrix; a decoding unit, specifically used to decode the i-th word vector based on the updated score, the first product result and the second product result.

[0023] Optionally, in a possible implementation of the second aspect, the above-mentioned acquisition unit is also used to obtain a second word embedding matrix corresponding to the i-th word vector, and the second word embedding matrix is ​​1 row and d columns; the decoding unit is specifically used to obtain a second matrix Q, a second matrix K and a second matrix V based on the second word embedding matrix and the second weight matrix; the number of rows of the second matrix Q, the second matrix K and the second matrix V is 1.

[0024] A third aspect of an embodiment of the present application provides a data processing device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the data processing device implements the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0025] A fourth aspect of an embodiment of the present application provides a computer-readable medium on which a computer program or instruction is stored. When the computer program or instruction is executed on a computer, the computer executes the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0026] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is executed on a computer, it enables the computer to execute the method in the aforementioned first aspect or any possible implementation manner of the first aspect.

[0027] Among them, the technical effects brought about by the second, third, fourth, fifth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.

[0028] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: the present application obtains the intermediate parameters in the historical decoding process and incrementally decodes the current word vector based on the intermediate parameters to reduce repeated decoding of the historical word vector. For example, in the i-th decoding process, the intermediate parameters in the historical decoding process are incrementally used, the intermediate parameters in the historical decoding process are stored, and the intermediate parameters are directly used in the current decoding process, thereby reducing the amount of repeated calculations and complexity in the decoding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1It is a structural schematic diagram of the decoding process in the prior art;

[0031] Figure 2 A schematic diagram of the structure of the system architecture provided in the embodiment of the present application;

[0032] Figure 3 A schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0033] Figure 4A A structural diagram of a data processing system provided in an embodiment of the present application;

[0034] Figure 4B Another structural diagram of the data processing system provided in the embodiment of the present application;

[0035] Figure 5 A flowchart of a data processing method provided in an embodiment of the present application;

[0036] Fig. 6A and Figure 6B An example diagram of a translation text scenario provided in an embodiment of the present application;

[0037] Figure 7 An example diagram of an initial word embedding matrix provided in an embodiment of the present application;

[0038] Figure 8 An example diagram of a first word embedding matrix provided in an embodiment of the present application;

[0039] Fig. 9 An example diagram of the process of incrementally calculating the KQV matrix provided in an embodiment of the present application;

[0040] Fig.10 An example diagram of the process of incrementally calculating the attention matrix provided in an embodiment of the present application;

[0041] Fig.11 An example diagram of an intermediate value matrix provided in an embodiment of the present application;

[0042] Fig.12 An example diagram of the process of offline calculation of the first product result provided in an embodiment of the present application;

[0043] Fig.13 A structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0044] Fig.14 Another structural schematic diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiment of the present application provides a data processing method and related devices, which are used to reduce the complexity of the decoding process.

[0046] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of the present application are first introduced below.

[0047] 1. Neural Networks

[0048] A neural network can be composed of neural units. A neural unit can be a s and intercept b as inputs, the output of the operation unit can be:

[0049]

[0050] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For X s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a Relu function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0051] The work of each layer in the neural network can be described by the mathematical expression y=a(Wx+b): From a physical level, the work of each layer in the neural network can be understood as completing the transformation from the input space to the output space (i.e., the row space to the column space of the matrix) through five operations on the input space (the set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Zoom in / out; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2, and 3 are completed by Wx, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. The vector W determines the spatial transformation from the input space to the output space mentioned above, that is, the weight W of each layer controls how to transform the space. The purpose of training a neural network is to finally obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, or more specifically, learning the weight matrix.

[0052] 2. Loss Function

[0053] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0054] 3. Multilayer perceptron (MLP)

[0055] A multilayer perceptron, also known as a multilayer perceptron, is a feedforward artificial neural network model that maps inputs to a single output.

[0056] 4. Transformer

[0057] The transformer structure is a feature extraction network that includes an encoder and a decoder (similar to a convolutional neural network). Of course, in some cases, the transformer structure may not include an encoder but a decoder.

[0058] Encoder: Performs feature learning in the global receptive field through self-attention, such as pixel features.

[0059] Decoder: Learn the features of the required modules, such as the features of the output box, through self-attention and cross-attention.

[0060] The following is a description of attention (also called attention mechanism):

[0061] The attention mechanism can quickly extract important features of sparse data. The attention mechanism occurs between the encoder and the decoder, or between the input sentence and the generated sentence. The self-attention mechanism in the self-attention model occurs inside the encoding matrix or the output sequence, and can extract the connection between words that are far apart in the same sentence, such as syntactic features (phrase structure). The self-attention mechanism provides an effective modeling method to capture global context information through QKV. Assume that the input is Q (query), and the context is stored in the form of key-value pairs (K, V). Then the attention mechanism is actually a mapping function from query to a series of key-value pairs (key, value). The essence of the attention function can be described as a mapping from a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in the form of (K, V), then attention completes the addressing by calculating the similarity between Q and K. The similarity calculated between Q and K reflects the importance of the extracted V value, that is, the weight, and then the weighted sum is used to obtain the final feature value.

[0062] The calculation of attention is mainly divided into three steps. The first step is to calculate the similarity between the query and each key to obtain the weight. Common similarity functions include dot product, concatenation, perceptron, etc. Then the second step is generally to use a softmax function (on the one hand, it can be normalized to obtain a probability distribution where the sum of all weight coefficients is 1. On the other hand, the characteristics of the softmax function can be used to highlight the weights of important elements) to normalize these weights; finally, the weight and the corresponding key value are weighted and summed to obtain the final eigenvalue. The specific calculation formula can be as follows:

[0063]

[0064] Where d is the dimension of the matrix QK.

[0065] In addition, attention includes self-attention and cross-attention. Self-attention can be understood as a special attention, that is, the input of QKV is consistent. However, the input of QKV in cross-attention is inconsistent. Attention uses the similarity between features (such as inner product) as weight to integrate the queried features as the update value of the current feature. Self-attention is the attention extracted based on the attention of the feature map itself.

[0066] For convolution, the setting of the convolution kernel limits the size of the receptive field, resulting in the network often requiring multiple layers of stacking to focus on the entire feature map. The advantage of self-attention is that its focus is global, and it can obtain the global spatial information of the feature map through simple query and assignment. The special point of self-attention in the query key value (QKV) model is that the input corresponding to QKV is consistent.

[0067] This application mainly focuses on the decoding process in the attention mechanism, which will be described later with reference to the accompanying drawings.

[0068] At present and in the next few years, the basic architecture of generative AI models is still transformers, and the biggest feature of decoder-only and encoder-decoder large model architectures in reasoning calculations is that decoder decoding requires serial generation of tokens. That is, when generating the i-th token, the input information of the previous i-1 tokens is required. In other words, the historical output content needs to be spliced ​​and re-input before the next step of decoding. For example, Figure 1 An example of the decoding process is shown. Taking the second step as an example, in the process of generating "S I come", it is necessary to splice the historical output content and re-use it as input (i.e. "S I"). The amount of calculation for decoding is too large and there is too much data that is repeatedly calculated. Therefore, how to reduce the amount of calculation in the decoding process is a technical problem that needs to be solved urgently.

[0069] In order to solve the above technical problems, we studied the internal calculation process of the decoder and found that each time the decoder generates a new token, it needs to use the historical token as input, but the historical token input is necessary only when calculating the attention score. The context (historical token + the newly generated token in the previous step) is needed. Therefore, the intermediate calculation results of the token can be cached, and there is no need to repeat the calculation.

[0070] Based on the above idea, an embodiment of the present application provides a data processing method, which reduces repeated decoding of historical word vectors by obtaining intermediate parameters in the historical decoding process and incrementally decoding the current word vector based on the intermediate parameters. For example, in the i-th decoding process, the intermediate parameters in the historical decoding process are incrementally used, the intermediate parameters in the historical decoding process are stored, and the intermediate parameters are directly used in the current decoding process, reducing the amount of repeated calculations and complexity in the decoding process.

[0071] In other words, the dimension of data processed during decoding under the transformer architecture in the existing technology is B×N×D, while the dimension of data processed during decoding in this application is B×1×D. That is, the intermediate parameters of historical decoding are stored and decoding is performed in an incremental manner, and not all historical data are calculated every time a token is decoded.

[0072] The following introduces the system architecture provided by the embodiments of the present application.

[0073] See attached Figure 2 , an embodiment of the present invention provides a system architecture 200. As shown in the system architecture 200, the data acquisition device 260 is used to collect training data. The training data in the embodiment of the present application may include at least one of the following: image, text, audio, video, etc. The training data is stored in the database 230, and the training device 220 obtains the target model / rule 201 based on the training data maintained in the database 230. The target model / rule 201 can be used to implement the computer vision task applied by the data processing method provided in the embodiment of the present application. The computer vision task may include: classification task, segmentation task, detection task or image generation task, etc. The target model / rule 201 in the embodiment of the present application may specifically include a self-attention network, a multi-layer perceptron, a pooling layer, etc. It should be noted that in actual applications, the training data maintained in the database 230 may not all come from the collection of the data acquisition device 260, but may also be received from other devices. It should also be noted that the training device 220 does not necessarily train the target model / rule 201 entirely based on the training data maintained by the database 230, and it is also possible to obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation on the embodiments of the present application.

[0074] The target model / rule 201 in the embodiment of the present application may be a transformer structure, which includes an encoder and a decoder. Or the transformer includes a decoder, etc., which is not specifically limited here. It is understandable that in practical applications, the transformer structure may also include other modules, such as a normalization module, a fully connected layer, an MLP, a feedforward neural network (FNN), etc., which are not specifically limited here.

[0075] The target model / rule 201 trained by the training device 220 can be applied to different systems or devices, such as Figure 2The execution device 210 shown in the figure can be a terminal, such as a mobile terminal, a tablet computer, a laptop computer, an augmented reality (AR) device / virtual reality (VR) device, a vehicle terminal, etc. Of course, the execution device 210 can also be a server or a cloud. Figure 2 In the embodiment, the execution device 210 is configured with an I / O interface 212 for data interaction with an external device, and a user can input data to the I / O interface 212 through a client device 240. The input data can be input by the user, or uploaded by the user through a shooting device, or can come from a database, which is not limited here.

[0076] It is understandable that the input data may also be different for different corresponding target models / rules 201. For example, when the target model / rule 201 includes an encoder and a decoder, the input data may include at least one of the following: an image, text, audio, video, etc. to be processed. For another example, when the target model / rule 201 includes a decoder, the input data may be a sequence or word vector corresponding to the above data.

[0077] The preprocessing module 213 is used to preprocess the input data received by the I / O interface 212. In the embodiment of the present application, the preprocessing module 213 can be used to split the input data into sub-data sets. For example, if the input data is an image, the preprocessing module 213 is used to split the image into multiple image blocks.

[0078] When the execution device 210 preprocesses the input data, or when the computing module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0079] Finally, the I / O interface 212 returns the processing result, such as the result corresponding to the above-mentioned target task, to the client device 240 to provide it to the user.

[0080] It is worth noting that the training device 220 can generate corresponding target models / rules 201 based on different training data for different goals or different tasks. The corresponding target models / rules 201 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0081] In the attached Figure 2In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 212. In another case, the client device 240 can automatically send input data to the I / O interface 212. If the client device 240 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 240. The user can view the results output by the execution device 210 on the client device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 212 and the output results of the output I / O interface 212 as new sample data, and store them in the database 230. Of course, it is also possible not to collect through the client device 240, but the I / O interface 212 directly stores the input data of the input I / O interface 212 and the output results of the output I / O interface 212 as new sample data in the database 230.

[0082] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed in the execution device 210.

[0083] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0084] Figure 3 A chip hardware structure provided in an embodiment of the present invention includes a neural network processor 30. The chip can be arranged as follows: Figure 2 The execution device 210 shown in FIG. 1 is used to complete the calculation work of the calculation module 211. The chip can also be set in Figure 2 The training device 220 shown is used to complete the training work of the training device 220 and output the target model / rule 201.

[0085] The neural network processor 30 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU) or any other processor suitable for large-scale XOR processing. Take NPU as an example: the neural network processor 30 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 303, and the controller 304 controls the operation circuit 303 to extract data from the memory (weight memory or input memory) and perform operations.

[0086] In some implementations, the operation circuit 303 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0087] For example, assume there is a coding matrix A, a weight matrix B, and an output matrix C. The operation circuit 303 takes the corresponding data of matrix B from the weight memory 302 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 301 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 308.

[0088] The vector calculation unit 307 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0089] In some implementations, the vector calculation unit 307 stores the vector of processed outputs to the unified buffer 306. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the operation circuit 303, for example, for use in a subsequent layer in a neural network.

[0090] The unified memory 306 is used to store input data and output data.

[0091] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the direct memory access controller 305 (DMAC), the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0092] The bus interface unit (BIU) 310 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 309 through the bus.

[0093] An instruction fetch buffer 309 connected to the controller 304 is used to store instructions used by the controller 304 .

[0094] The controller 304 is used to call the instructions cached in the instruction fetch memory 309 to control the working process of the computing accelerator.

[0095] Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM for short), a high bandwidth memory (HBM) or other readable and writable memory.

[0096] Next, several application scenarios of this application are introduced.

[0097] Figure 4A A structural diagram of a data processing system provided in an embodiment of the present application, the data processing system includes a terminal device ( Figure 4AIn the figure, only the terminal device is a mobile phone as an example) and a data processing device. It is understandable that, in addition to a mobile phone, the terminal device can also be a tablet computer (pad), a portable game console, a personal digital assistant (PDA), a laptop computer, an ultra mobile personal computer (UMPC), a handheld computer, a netbook, a car media player, a wearable electronic device, a virtual reality (VR) terminal device, an augmented reality (AR), a vehicle, a vehicle terminal, an aircraft terminal, an intelligent robot and other terminal devices. The terminal device is the initiator of data processing. As the initiator of the data processing request, the request is usually initiated by the user through the terminal device.

[0098] The above-mentioned data processing device can be a device or server with data processing function such as a cloud server, a network server, an application server and a management server. The data processing device receives data processing requests from the terminal device through an interactive interface, and then performs data processing in the form of machine learning, deep learning, search, reasoning, decision-making, etc. through the memory that stores data and the processor link of data processing. The memory in the data processing device can be a general term, including local storage and databases that store historical data. The database can be on the data processing device or on other network servers.

[0099] exist Figure 4A In the data processing system shown, the terminal device can receive the user's instructions. For example, the terminal device can obtain the data input / selected by the user (for example, images, text, audio, etc. collected by the terminal device through the terminal device), and then initiate a request to the data processing device, so that the data processing device performs a data processing application (for example, computer vision tasks such as classification, segmentation, detection, and image generation) for the multiple data obtained by the terminal device, thereby obtaining a processing result corresponding to the data. Exemplarily, the terminal device can obtain an image input by the user, and then initiate an image detection request to the data processing device, so that the data processing device detects the image, thereby obtaining the detection result of the image, and displays the detection result of the image for the user to view and use.

[0100] exist Figure 4A In the embodiment of the present application, the data processing device can execute the data processing method of the embodiment of the present application.

[0101] Figure 4B Another structural diagram of the data processing system provided in the embodiment of the present application is shown in FIG. Figure 4B In the terminal device ( Figure 4BIn this example, the terminal device is a mobile phone. The terminal device can directly obtain data (for example, the data is an image, text, audio, etc.) and process it directly by the hardware of the terminal device itself. The specific process is similar to Figure 4A Similarly, please refer to the above description and will not repeat them here.

[0102] Optionally, in Figure 4B In the data processing system shown, the terminal device can receive instructions from the user. For example, the terminal device can obtain the data selected by the user in the terminal device, and then the terminal device itself executes a data processing application (for example, computer vision tasks such as classification, segmentation, detection, and image generation) on the data, thereby obtaining the corresponding processing results for the data, and displaying the processing results for the user to view and use.

[0103] Optionally, in Figure 4B In the data processing system shown, the terminal device can collect data in real time or periodically, and then the terminal device itself executes data processing applications (for example, computer vision tasks such as classification, segmentation, detection, image generation, etc.) on the data, thereby obtaining corresponding processing results for the data, and implementing functions (classification function, segmentation function, detection function, image generation function, etc.) based on the processing results.

[0104] exist Figure 4B In the embodiment of the present application, the terminal device itself can execute the data processing method.

[0105] Above Figure 4A and Figure 4B The terminal device in the example can be Figure 2 The client device 240 or the execution device 210 in Figure 4A The data processing device in the embodiment may be Figure 2 The execution device 210 in the embodiment, wherein the data storage system 250 can store the data to be processed by the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or can be set on the cloud or other network servers.

[0106] Figure 4A and Figure 4B The processor in the system can perform data training / machine learning / deep learning through a neural network model or other models (such as an attention model, MLP, etc.), and use the model finally trained or learned from the data to execute data processing applications on the data, thereby obtaining corresponding processing results.

[0107] The data processing method provided in the embodiment of the present application is described below. The method can be executed by a data processing device, or by a component of the data processing device (such as a processor, a chip, or a chip system, etc.). The data processing device can be a cloud device (such as the aforementioned Figure 4A as shown), or a terminal device (such as Figure 4B Of course, the method can also be executed by a system consisting of a cloud device and a terminal device (as described above). Figure 4A Optionally, the method may be processed by a CPU in a data processing device, or may be processed by a CPU and a GPU together, or may use other processors suitable for neural network calculations without using a GPU, and this application does not limit this.

[0108] The application scenarios to which the method provided in the embodiments of the present application is applicable are mainly attention mechanism scenarios, and can be specifically applied to computer vision tasks such as classification scenarios, segmentation scenarios, detection scenarios, and image generation scenarios, or can be applied to semantic segmentation, indoor scene perception, outdoor driving, etc. In addition, the data involved in the embodiments of the present application may refer to text, images, audio and video, etc. For the convenience of description, this article only takes the data as text as an example for exemplary description.

[0109] See also Figure 5 , a flow chart of a data processing method provided by an embodiment of the present application, the method may include step 501 and step 502. Step 501 and step 502 are described in detail below.

[0110] Step 501, obtaining intermediate parameters.

[0111] The data processing device obtains the intermediate parameters. The intermediate parameters are the parameters stored in the process of decoding the 1st to i-1th word vectors based on the attention mechanism.

[0112] The attention mechanism in the embodiment of the present application can be applied to the transformer structure.

[0113] In a possible implementation, the above transformer structure includes an encoder and a decoder. Alternatively, it can be understood that the method provided in the embodiment of the present application can be applied to an encoder-decoder structure.

[0114] In another possible implementation, the transformer structure includes a decoder. Alternatively, it can be understood that the method provided in the embodiment of the present application can be applied to an only-decoder structure.

[0115] Optionally, in the case of an encoder-decoder structure, Fig. 6A As shown, when applied to the only-decoder structure, Figure 6B As shown. Take the translation text scenario as an example, the original text is "I have a dog" and the target text is "I have a dog". The encoding matrix in this example is the encoding matrix of the original text "I have a dog". <bos>I have a dog" is a word embedding matrix, which is used to map each word in the input sequence to the corresponding word vector. The corresponding structure can be as follows Figure 7 As shown in , the word embedding matrix in this example includes 6 rows and 5 columns (i.e., n=6, d=5). Figure 6B As shown, <bos>" is the translation start symbol, which is used to predict the first word "I".

[0116] The dimension of the above word embedding matrix is ​​usually the same as that of the word embedding matrix in the encoder, so as to ensure that the information between the encoder and the decoder can be well aligned. This enables the transformer structure to better learn the correspondence between the original data and the target data. In the decoder, each target data is mapped through the word embedding matrix to obtain the corresponding word vector representation. These word vectors will be used as part of the decoder input to generate the next word of the target sequence. By learning the appropriate word embedding matrix, the model can better understand and express the semantic information in the target language.

[0117] According to different calculation steps in the transformer decoder, the intermediate parameters in the embodiment of the present application include at least one of the following cases, which are described below respectively:

[0118] First, the intermediate parameters include the first word embedding matrix corresponding to the i-1th word vector.

[0119] This situation can also be understood as the process of calculating the matrix QKV in the decoder.

[0120] The first word embedding matrix is ​​i-1 rows and d columns.

[0121] Exemplarily, continuing the above example, the first word embedding matrix is ​​as follows Figure 8 As shown, the first word embedding matrix includes 3 rows, the first row corresponds to the translation start symbol, the second row corresponds to the first word, and the third row corresponds to the second word. In other words, the historical word embedding matrix used in the process of decoding the third word "a" is as follows Figure 8 shown.

[0122] The second type, the intermediate parameters include the first matrix Q, the first matrix K, and the first attention matrix corresponding to the 1st to i-1th word vectors.

[0123] This situation can also be understood as the process of calculating the attention matrix in the decoder.

[0124] The first attention matrix is ​​obtained according to the first matrix Q and the first matrix V. Specifically, it can be the product of the transposed matrix of the first matrix Q and the first matrix V, or the product of the first matrix Q and the transposed matrix of the first matrix V, etc., which is not limited here.

[0125] The third type of intermediate parameters includes the score obtained by normalizing the first attention matrix, and the normalization process belongs to the process of decoding the i-1th word vector.

[0126] This situation can also be understood as the process of normalizing the attention matrix in the decoder.

[0127] The numerator of the above fraction is the autocorrelation index value of the i-1th word vector, and the denominator of the fraction is the sum of the autocorrelations of the 1st to i-1th word vectors.

[0128] In the embodiment of the present application, there is no limitation on the normalization function used in the normalization process. The following is an exemplary description using the normalization function being a normalized exponential function (softmax), and no specific limitation is made here.

[0129] Exemplarily, the normalization method of the first attention matrix is ​​shown in the following formula 1:

[0130] Formula 1:

[0131] in, Represents the autocorrelation index value of the i-1th word vector, Represents the autocorrelation sum of the 1st to i-1th word vectors.

[0132] Fourthly, the intermediate parameters include the first matrix V corresponding to the 1st to i-1th word vectors, and the first product result of the first matrix V and the first weight matrix W.

[0133] This situation can also be understood as the calculation matrix V and weight matrix W in the decoder out process.

[0134] The first weight matrix mentioned above can also be recorded as W out , the W out Obtained through training.

[0135] It is understandable that the above-mentioned several intermediate parameters are only examples. In practical applications, the intermediate parameters may also be intermediate parameters obtained in other historical decoding processes. In addition, the above-mentioned several intermediate parameters may appear alone or in combination, which is not specifically limited here.

[0136] Step 502, incrementally decode the i-th word vector based on the intermediate parameters to obtain the i+1-th word vector.

[0137] After the data processing device obtains the intermediate parameters, it can incrementally decode the i-th word vector based on the intermediate parameters to obtain the i+1-th word vector. The 1st word vector to the i+1-th word vector are used to generate data, which is related to the initial word embedding matrix.

[0138] Exemplarily, continuing the example of the above translation text sentence, decoding " <bos>” gets the first word "I", decodes "I" to get the second word "have", decodes "have" to get the third word "a", decodes "a" to get the fourth word "dog", and decodes "dog" to get the translation end symbol "eos". The decoded words are concatenated to get the result "I have a dog" (that is, the data generated by the above 1st word vector to the i+1th word vector).

[0139] The incremental decoding process in the embodiment of the present application may include: an incremental calculation KQV process, an incremental calculation attention matrix process, an incremental calculation normalization process, an advance and incremental calculation matrix V and a second weight matrix W out The following is a detailed description of each process:

[0140] The first one is the process of incrementally calculating the KQV matrix.

[0141] In this case, the data processing device can first calculate the newly added part, and then concatenate it with the previously cached intermediate parameters to obtain the overall KQV matrix used to decode the i-th word vector. And decode the i+1th word vector based on the overall KQV matrix or the newly added part. The intermediate parameters in this case include: the first word embedding matrix corresponding to the i-1th word vector, the first matrix Q, the first matrix K, and the first matrix V.

[0142] The newly added part of the above calculation includes: first obtaining the second word embedding matrix corresponding to the i-th word vector, and then obtaining the second matrix Q, the second matrix K and the second matrix V according to the second word embedding matrix and the second weight matrix. The second weight matrix includes W q , W k , W v Specifically, the second word embedding matrix is ​​used with W q , W k , W v Multiply them to obtain the second matrix Q, the second matrix K and the second matrix V. In addition, the number of rows of the second word embedding matrix, the second matrix Q, the second matrix K and the second matrix V is 1. q , W k , W v Obtained through training.

[0143] Compared with the prior art, in which multiple rows of word embedding matrices are used to calculate the overall KQV during decoding of the i-th word vector, the decoder in the embodiment of the present application inputs a row of the second word embedding matrix (also referred to as the newly added word embedding matrix), and then multiplies the second word embedding matrix with the second weight matrix to obtain the incremental KQV matrix (i.e., the second matrix Q, the second matrix K, and the second matrix V). The overall KQV matrix in the current decoding process is obtained by splicing according to the cached historical matrix KQV (i.e., the first matrix Q, the first matrix K, and the first matrix V).

[0144] In other words, the historical KQV matrix in the overall KQV matrix has been calculated before decoding the i-th word vector. By caching the historical KQV matrix and only calculating the newly added part, the historical KQV matrix can be reduced to recalculate during the decoding of the i-th word vector.

[0145] Exemplarily, continuing the above example, in the process of calculating the KQV matrix, Fig. 9 As shown in medium gray, the historical matrix is ​​not calculated, and the overall KQV matrix is ​​obtained by splicing the cache and the newly added matrix.

[0146] It can be understood that the concatenation of the above KQV matrices can be understood as taking a row of the newly added matrix as a newly added row of the historical matrix, or it can be understood as the last row of the overall KQV matrix being the newly added KQV matrix, and the historical KQV matrix being the row above the newly added KQV matrix in the overall KQV matrix.

[0147] Optionally, in the Masked-Multi-Heads Self-Attention module, the KQV matrix generated in the previous i-1 steps is cached in the High Bandwidth Memory (HBM), and the cached tensor dimension is (batch, n-1, dim). For the incremental token of the i-th step, the corresponding incremental input tensor is (batch, 1, dim), which is spliced ​​into the corresponding historical cache KQV matrix according to the corresponding index (batch index) to obtain a new KQV matrix. Among them, batch is the number of sentences, n-1 or 1 is the sentence length, and dim is the length of the word vector.

[0148] Exemplary, continued Fig. 9 For example, the historical word embedding matrix is ​​the input matrix (batch, n-1, dim) of the historical i-1 tokens, and is not used as the input of the i-th step. The newly added word embedding matrix is ​​used as the input of the i-th step, and the newly added word embedding matrix is ​​the input matrix (batch, 1, dim) of the i-th token. The historical matrix QKV is the input matrix (batch, n-1, dim) of the historical i-1 tokens, and the newly added matrix QKV is the input matrix (batch, 1, dim) of the i-th token.

[0149] The second is the process of incrementally calculating the attention matrix.

[0150] In this case, the data processing device can first calculate the newly added matrix block, and then obtain a new attention matrix (i.e., the second attention matrix) based on the previously cached first attention matrix and the newly added matrix block. And decode the i+1th word vector based on the new attention matrix. The intermediate parameters in this case include: the first matrix Q, the first matrix K, and the first attention matrix.

[0151] Among them, the above calculation of newly added rows and columns includes: first obtaining the second matrix Q, the second matrix K and the second matrix V corresponding to the i-th word vector. Multiplying the transpose matrix of the first matrix K with the second matrix Q to obtain the first matrix block (which can also be understood as the multiplication of the transpose matrix of the historical matrix K and the newly added matrix Q); multiplying the transpose matrix of the second matrix K with the first matrix Q to obtain the second matrix block (which can also be understood as the multiplication of the transpose matrix of the newly added matrix K and the historical matrix Q); multiplying the transpose matrix of the second matrix K with the second matrix Q to obtain the third matrix block (which can also be understood as the multiplication of the transpose matrix of the newly added matrix K and the newly added matrix Q); splicing the first matrix block, the second matrix block, the third matrix block and the first attention matrix to obtain the second attention matrix, the first matrix block is the newly added column of the second attention matrix compared to the first attention matrix, the second matrix block is the newly added row of the second attention matrix compared to the first attention matrix, and the third matrix block is the newly added matrix block of the second attention matrix compared to the first attention matrix.

[0152] In addition, when decoding the i-th word vector, the first matrix Q can also be called the history matrix Q, the first matrix K can also be called the history matrix K, and the first attention matrix can also be called the history attention matrix.

[0153] Compared to the prior art method of calculating the attention matrix by multiplying the full amount K by the full amount Q during decoding of the i-th word vector, the embodiment of the present application splits the transpose matrix of the full amount K by the multiplication of the full amount Q, caches the intermediate calculation state of historical decoding, realizes the transpose of K and the incremental multiplication of Q, and greatly reduces the repeated calculation amount caused by repeated variable-length matrix multiplication. Or it can be understood that since the QV matrix has a fixed value for the same input sequence. When the transpose of K is multiplied by Q, the generated attention matrix tends to expand by row / column, and the i-1 row is fixed, so that the historical attention matrix can be cached, and the newly added part can be calculated and then spliced ​​with the historical attention matrix accordingly.

[0154] In other words, the historical attention matrix in the overall attention matrix has been calculated before decoding the i-th word vector. By caching the historical attention matrix and only calculating the newly added part, we can reduce the need to recalculate the historical attention matrix during decoding the i-th word vector.

[0155] Exemplarily, the historical attention matrix, the new attention matrix, and each matrix block can be as follows Fig.10 shown.

[0156] Optionally, in the Masked-Multi-Heads Self-Attention module, the historical attention matrix is ​​cached, and the K input tensor (batch, 1, dim) of the i-th step is multiplied by the full Q matrix (batch, n-1, dim) of the historical cache to obtain a temporary matrix temp_atten1 (batch, n-1, 1); the Q tensor (batch, dim) of the i-th step is multiplied by the full K matrix tensor (batch, n-1, dim) of the historical cache to obtain a temporary matrix temp_atten2 (batch, n-1, 1); temp_atten1 is transposed and multiplied with temp_atten2 to obtain a temporary matrix temp_atten3 (batch, 1, 1); finally, temp_atten1 (i.e., the second matrix block), temp_atten2 (i.e., the first matrix block), and temp_atten3 (i.e., the third matrix block) are spliced ​​into the attention historical cache matrix (i.e., the historical attention matrix) of the corresponding batch index to obtain a new attention matrix.

[0157] The third type is the incremental calculation normalization process

[0158] This situation can also be understood as the process of incremental normalization of the attention matrix in the decoder. The intermediate parameters in this case include: the score obtained by normalizing the first attention matrix.

[0159] Among them, the normalization of the first attention matrix belongs to the process of decoding the i-1th word vector. The numerator of the fraction is the autocorrelation index value of the i-1th word vector, and the denominator of the fraction is the sum of the autocorrelations of the 1st to i-1th word vectors. When decoding the i-th word vector, the above score can also be called the historical score.

[0160] In the embodiment of the present application, there is no limitation on the normalization function used in the normalization process. The following is an exemplary description using the normalization function being a normalized exponential function (softmax), and no specific limitation is made here.

[0161] Exemplarily, the normalization method of the first attention matrix is ​​shown in the following formula 2:

[0162] Formula 2:

[0163] Among them, x is the value of each matrix block in the first attention matrix, or it can be understood as x = the product of the transposed matrix of the first matrix Q and the first matrix K.

[0164] Formula 2 can be understood as adding the autocorrelation index value of the i-th word vector to the denominator of the fraction in the aforementioned formula 1 to update the fraction, and performing subsequent decoding calculations based on the updated fraction.

[0165] Compared with the prior art method of repeatedly performing exponential operations in each step of attention matrix normalization, the embodiment of the present application adds the autocorrelation index value of the new word vector to the denominator of the historical fraction, and can use the historical intermediate results for subsequent decoding processes, which greatly reduces repeated exponential calculations.

[0166] In other words, the numerator and denominator in the normalization process have been calculated before decoding the i-th word vector, and by caching the fractions, only the newly added part is added to the denominator. This reduces the need to recalculate the historical part of the numerator and denominator during the decoding of the i-th word vector.

[0167] Optionally, the intermediate parameter can also be an intermediate value matrix T obtained by normalizing the first attention matrix, the intermediate value matrix T includes a rows and b columns, the last column records the sum of the first b-1 columns of each row, and the intermediate value matrix is ​​obtained according to the first attention matrix. Alternatively, it can be understood that the a rows and b-1 columns in the intermediate value matrix are the exponential values ​​of each value in the first attention matrix.

[0168] For example, the intermediate value matrix T is as follows Fig.11 As shown, the last column of the intermediate value matrix T corresponds to the sum of the first b-1 values ​​of each row.

[0169] This method does not require the softmax matrix in the prior art, but caches the historical softmax intermediate value matrix T and updates the intermediate value matrix T according to the trend of row / column expansion to facilitate the subsequent decoding process.

[0170] Fourth, calculate the matrix V and the second weight matrix W in advance and incrementally out .

[0171] In this case, the data processing device may first calculate the second product result, and then obtain the overall V and W according to the first product result and the second product result previously cached. out The product of . And decode the i+1th word vector according to the product and the updated score. The intermediate parameters in this case include: the intermediate value matrix T obtained by normalizing the first attention matrix, the first matrix V corresponding to the 1st to i-1th word vectors, and the first weight matrix W out The first product result of .

[0172] The above calculation of the second product result includes: calculating the second matrix V and the matrix W out The second product result of .

[0173] This situation can be expressed by formula 3:

[0174] Formula 3:

[0175] Where T' represents the a-row and b-1-column intermediate value matrix T in the normalization process of the updated scores, V i-1 Represents the first matrix V, V corresponding to the 1st to i-1th word vectors i represents the second matrix V corresponding to the i-th word vector, V i-1 *W out Represents the first product result. That is, before decoding the i-th word vector, the first product result can be calculated offline, and in the process of decoding the i-th word vector, the final result can be obtained by calculating the second product result and the cached first product result.

[0176] In addition, when decoding the i-th word vector, the first matrix V can also be called the history matrix V, and the first product result can also be called the history product result.

[0177] Exemplarily, the process of obtaining the i+1th word vector based on the updated score, the first product result, and the second product result can be as follows: Fig.12 shown.

[0178] It is understandable that the above-mentioned decoding based on the intermediate parameters is only an example, and in practical applications, decoding can also be performed based on other intermediate parameters or other incremental methods. In addition, the above-mentioned situations can appear alone or in combination, and are not specifically limited here.

[0179] It should be noted that the above-mentioned situations can be used once or multiple times in the transformer architecture. The embodiment of the present application only explains incremental decoding by taking decoding of the i-th word vector as an example.

[0180] Optionally, after obtaining the 1st word vector to the (i+1)th word vector, subsequent processing (eg, multiplication, normalization, etc.) is performed to obtain target data.

[0181] In an embodiment of the present application, the intermediate parameters of historical decoding are used in the calculation of a token in the transformer decoder, and the current word vector is incrementally decoded based on the intermediate parameters to reduce repeated decoding of historical word vectors. For example, in the i-th decoding process, the intermediate parameters of the historical decoding process are incrementally used, the intermediate parameters of the historical decoding process are stored, and the intermediate parameters are directly used in the current decoding process, reducing the amount of repeated calculations and complexity in the decoding process.

[0182] The data processing method in the embodiment of the present application is described above. The data processing device in the embodiment of the present application is described below. Fig.13 , an embodiment of the data processing device in the embodiment of the present application includes:

[0183] An acquisition unit 1301 is used to acquire intermediate parameters, where the intermediate parameters are parameters stored in the process of decoding the 1st to i-1th word vectors based on the attention mechanism;

[0184] The decoding unit 1302 is used to incrementally decode the i-th word vector based on the intermediate parameters to obtain the i+1-th word vector, and the 1st word vector to the i+1-th word vector are used to generate data.

[0185] Optionally, the intermediate parameters include at least one of the following: the first word embedding matrix corresponding to the i-1th word vector, the first word embedding matrix is ​​i-1 rows and d columns; the first matrix Q, the first matrix K, and the first attention matrix corresponding to the 1st to i-1th word vectors, the first attention matrix is ​​obtained based on the first matrix Q and the first matrix K; the score obtained by normalizing the first attention matrix, the normalization of the first attention matrix belongs to the process of decoding the i-1th word vector, the numerator of the fraction is the autocorrelation index value of the i-1th word vector, and the denominator of the fraction is the sum of the autocorrelations of the 1st to i-1th word vectors; the intermediate value matrix obtained by normalizing the first attention matrix, the intermediate value matrix includes a rows and b columns, the last column records the sum of the first b-1 columns of each row, and the intermediate value matrix is ​​obtained based on the first attention matrix; the first product result of the first matrix V corresponding to the 1st to i-1th word vectors and the first weight matrix.

[0186] Optionally, the intermediate parameters include; a first matrix Q, a first matrix K and a first attention matrix; an acquisition unit 1301, also used to obtain the second matrix Q, the second matrix K and the second matrix V of the i-th word vector; a decoding unit 1302, specifically used to obtain the second attention matrix based on the intermediate parameters, the second matrix Q and the second matrix K; the decoding unit 1302, specifically used to decode the i-th word vector based on the second attention matrix and the second matrix V.

[0187] Optionally, the decoding unit 1302 is specifically used to multiply the transposed matrix of the first matrix K with the second matrix Q to obtain a first matrix block; the decoding unit 1302 is specifically used to multiply the transposed matrix of the second matrix K with the first matrix Q to obtain a second matrix block; the decoding unit 1302 is specifically used to multiply the transposed matrix of the second matrix K with the second matrix Q to obtain a third matrix block; the decoding unit 1302 is specifically used to concatenate the first matrix block, the second matrix block, the third matrix block and the first attention matrix to obtain a second attention matrix, the first matrix block serving as the newly added columns of the second attention matrix compared to the first attention matrix, the second matrix block serving as the newly added rows of the second attention matrix compared to the first attention matrix, and the third matrix block serving as the newly added matrix block of the second attention matrix compared to the first attention matrix.

[0188] Optionally, the intermediate parameters also include: a score and a first product result; a decoding unit 1302, specifically used to add the autocorrelation index value of the i-th word vector in the denominator to update the score; a decoding unit 1302, specifically used to calculate the second product result of the second matrix V and the first weight matrix; a decoding unit 1302, specifically used to decode the i-th word vector based on the updated score, the first product result and the second product result.

[0189] Optionally, the acquisition unit 1301 is also used to obtain a second word embedding matrix corresponding to the i-th word vector, and the second word embedding matrix is ​​1 row and d columns; the decoding unit 1302 is specifically used to obtain a second matrix Q, a second matrix K and a second matrix V based on the second word embedding matrix and the second weight matrix; the number of rows of the second matrix Q, the second matrix K and the second matrix V is 1.

[0190] In this embodiment, the operations performed by each unit in the data processing device are the same as those described above. Figures 1 to 12 The description in the illustrated embodiment is similar and will not be repeated here.

[0191] In this embodiment, the intermediate parameters in the historical decoding process are obtained by the acquisition unit 1301, and the decoding unit 1302 then incrementally decodes the current word vector based on the intermediate parameters to reduce repeated decoding of the historical word vector. For example, in the i-th decoding process, the intermediate parameters in the historical decoding process are incrementally used, the intermediate parameters in the historical decoding process are stored, and the intermediate parameters are directly used in the current decoding process, reducing the amount of repeated calculations and complexity in the decoding process.

[0192] See also Fig.14 , a schematic diagram of the structure of another data processing device provided by the present application. The data processing device may include a processor 1401, a memory 1402 and a communication port 1403. The processor 1401, the memory 1402 and the communication port 1403 are interconnected via a line. The memory 1402 stores program instructions and data.

[0193] The memory 1402 stores the aforementioned Figures 1 to 12 In the corresponding embodiment shown, the steps executed by the data processing device correspond to program instructions and data.

[0194] Processor 1401, used to execute the above Figures 1 to 12 The steps shown in any of the illustrated embodiments are performed by a data processing device.

[0195] Communication port 1403 can be used to receive and send data, and to perform the above Figures 1 to 12 The steps related to obtaining, sending and receiving in any of the embodiments shown.

[0196] In one implementation, the data processing device may include Fig.14 More or fewer components are merely illustrative in this application and are not limiting.

[0197] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0198] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0199] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0201] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.< / bos> < / bos> < / bos>

Claims

1. A data processing method, characterized in that: The method is applied to a decoder of an attention mechanism, and the method comprises: Obtaining intermediate parameters, where the intermediate parameters are parameters stored in the process of decoding the 1st to i-1th word vectors based on the attention mechanism; The i-th word vector is incrementally decoded based on the intermediate parameters to obtain the i+1-th word vector, and the 1st word vector to the i+1-th word vector are used to generate data.

2. The method according to claim 1, characterized in that The intermediate parameters include at least one of the following: The first word embedding matrix corresponding to the i-1th word vector, the first word embedding matrix has i-1 rows and d columns; The first matrix Q, the first matrix K, and the first attention matrix corresponding to the 1st to i-1th word vectors, where the first attention matrix is ​​obtained according to the first matrix Q and the first matrix K; a score obtained by normalizing the first attention matrix, wherein the normalization of the first attention matrix belongs to the process of decoding the i-1th word vector, the numerator of the score is the autocorrelation index value of the i-1th word vector, and the denominator of the score is the autocorrelation sum value of the first to the i-1th word vectors; an intermediate value matrix obtained by normalizing the first attention matrix, the intermediate value matrix comprising a rows and b columns, the last column recording the sum of the first b-1 columns of each row, the intermediate value matrix being obtained according to the first attention matrix; The first product results of the first matrix V corresponding to the 1st to i-1th word vectors and the first weight matrix.

3. The method according to claim 2, characterized in that The intermediate parameters include: the first matrix Q, the first matrix K and the first attention matrix; The method further comprises: Obtain the second matrix Q, the second matrix K and the second matrix V of the i-th word vector; The step of incrementally decoding the i-th word vector based on the intermediate parameter comprises: Obtain a second attention matrix based on the intermediate parameters, the second matrix Q and the second matrix K; Decode the i-th word vector based on the second attention matrix and the second matrix V.

4. The method according to claim 3, characterized in that The obtaining a second attention matrix based on the intermediate parameters, the second matrix Q and the second matrix K includes: Multiplying the transposed matrix of the first matrix K by the second matrix Q to obtain a first matrix block; Multiplying the transposed matrix of the second matrix K by the first matrix Q to obtain a second matrix block; Multiplying the transposed matrix of the second matrix K by the second matrix Q to obtain a third matrix block; The first matrix block, the second matrix block, the third matrix block and the first attention matrix are concatenated to obtain the second attention matrix, the first matrix block serves as the newly added columns of the second attention matrix compared to the first attention matrix, the second matrix block serves as the newly added rows of the second attention matrix compared to the first attention matrix, and the third matrix block serves as the newly added matrix block of the second attention matrix compared to the first attention matrix.

5. The method according to claim 3 or 4, characterized in that: The intermediate parameter also includes: the score and the first product result; The decoding of the i-th word vector based on the second attention matrix and the second matrix V includes: Adding the autocorrelation index value of the i-th word vector to the denominator to update the score; Calculate a second product result of the second matrix V and the first weight matrix; Decode the i-th word vector based on the updated score, the first product result, and the second product result.

6. The method according to any one of claims 3 to 5, characterized in that Before the incremental decoding of the i-th word vector based on the intermediate parameter, the method further includes: Obtain a second word embedding matrix corresponding to the i-th word vector, where the second word embedding matrix has 1 row and d columns; The obtaining of the second matrix Q, the second matrix K and the second matrix V of the i-th word vector includes: Based on the second word embedding matrix and the second weight matrix, the second matrix Q, the second matrix K and the second matrix V are obtained; the number of rows of the second matrix Q, the second matrix K and the second matrix V is 1.

7. A data processing device, characterized in that: The data processing device is applied to a decoder of an attention mechanism, and the data processing device includes: An acquisition unit, used to acquire intermediate parameters, where the intermediate parameters are parameters stored in the process of decoding the 1st to i-1th word vectors based on the attention mechanism; A decoding unit is used to incrementally decode the i-th word vector based on the intermediate parameter to obtain the i+1-th word vector, and the first word vector to the i+1-th word vector are used to generate data.

8. The data processing device according to claim 7, characterized in that The intermediate parameters include at least one of the following: The first word embedding matrix corresponding to the i-1th word vector, the first word embedding matrix has i-1 rows and d columns; The first matrix Q, the first matrix K, and the first attention matrix corresponding to the 1st to i-1th word vectors, where the first attention matrix is ​​obtained according to the first matrix Q and the first matrix K; a score obtained by normalizing the first attention matrix, wherein the normalization of the first attention matrix belongs to the process of decoding the i-1th word vector, the numerator of the score is the autocorrelation index value of the i-1th word vector, and the denominator of the score is the autocorrelation sum value of the first to the i-1th word vectors; an intermediate value matrix obtained by normalizing the first attention matrix, the intermediate value matrix comprising a rows and b columns, the last column recording the sum of the first b-1 columns of each row, the intermediate value matrix being obtained according to the first attention matrix; The first product results of the first matrix V corresponding to the 1st to i-1th word vectors and the first weight matrix.

9. The data processing device according to claim 8, characterized in that The intermediate parameters include: the first matrix Q, the first matrix K and the first attention matrix; The acquisition unit is further used to acquire the second matrix Q, the second matrix K and the second matrix V of the i-th word vector; The decoding unit is specifically used to obtain a second attention matrix based on the intermediate parameters, the second matrix Q and the second matrix K; The decoding unit is specifically used to decode the i-th word vector based on the second attention matrix and the second matrix V.

10. The data processing device according to claim 9, characterized in that The decoding unit is specifically configured to multiply the transposed matrix of the first matrix K by the second matrix Q to obtain a first matrix block; The decoding unit is specifically configured to multiply the transposed matrix of the second matrix K by the first matrix Q to obtain a second matrix block; The decoding unit is specifically configured to multiply the transposed matrix of the second matrix K by the second matrix Q to obtain a third matrix block; The decoding unit is specifically used to splice the first matrix block, the second matrix block, the third matrix block and the first attention matrix to obtain the second attention matrix, the first matrix block serves as the newly added columns of the second attention matrix compared to the first attention matrix, the second matrix block serves as the newly added rows of the second attention matrix compared to the first attention matrix, and the third matrix block serves as the newly added matrix block of the second attention matrix compared to the first attention matrix.

11. The data processing device according to claim 9 or 10, characterized in that: The intermediate parameter also includes: the score and the first product result; The decoding unit is specifically configured to add the autocorrelation index value of the i-th word vector to the denominator to update the score; The decoding unit is specifically used to calculate a second product result of the second matrix V and the first weight matrix; The decoding unit is specifically used to decode the i-th word vector based on the updated score, the first product result and the second product result.

12. The data processing device according to any one of claims 9 to 11, characterized in that: The acquisition unit is further used to acquire a second word embedding matrix corresponding to the i-th word vector, where the second word embedding matrix has 1 row and d columns; The decoding unit is specifically used to obtain the second matrix Q, the second matrix K and the second matrix V based on the second word embedding matrix and the second weight matrix; the number of rows of the second matrix Q, the second matrix K and the second matrix V is 1.

13. A data processing device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the data processing device executes the method according to any one of claims 1 to 6.

14. A computer storage medium, characterized in that: The method comprises computer instructions, which, when executed on a control device, cause the control device to execute the method according to any one of claims 1 to 6.

15. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images