Matrix multiplier, chip, device, data processing method, medium and product

Abstract

The invention discloses a matrix multiplier, a chip, electronic equipment, a data processing method, a computer readable storage medium and a computer program product, and belongs to the field of artificial intelligence. The matrix multiplier comprises a multiplier array, a data preprocessing module and an accumulator; the multiplier array is configured to perform multiplication operation on an input first matrix and a second matrix; the data preprocessing module is configured to preprocess the third matrix to obtain a mask matrix; the accumulator is configured to add the multiplication result of the first matrix and the second matrix and the mask matrix; wherein the first matrix is an M * K matrix, the second matrix is a K * N matrix, the third matrix is an M * N matrix, and M, N and K are all integers greater than or equal to 1. According to the method, the mask process is fused into the matrix multiplication operation, so that the matrix multiplication operation and the mask adding operation can be synchronously carried out, and the calculation rate of the Attention (attention) operation is effectively improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and specifically relates to a matrix multiplier, a chip, an electronic device, a data processing method, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Neural networks are used in various fields such as speech recognition, image recognition, and natural language processing. Running neural networks on AI chips requires the support of numerous operators. In the field of deep learning, especially in Natural Language Processing (NLP) tasks, Attention Mask is a key technique for handling variable-length input sequences. For example, when using a Transformer model, input data is often padded to the same length to meet batch processing requirements. However, these padding parts are meaningless to the model, so Attention Mask is needed to mask them out and prevent the model from focusing on invalid information.

[0003] During the current Attention operation, Q×K T It is a matmul (matrix multiplication) calculation, while (Q×K) T +Mask) is a non-matmul computation, so the two computations cannot be performed in parallel, which affects the computation speed of Attention. Summary of the Invention

[0004] Therefore, the purpose of this application is to provide a matrix multiplier, chip, electronic device, data processing method, computer-readable storage medium, and computer program product to improve the problem of low computation speed in the current Attention operation process.

[0005] The embodiments of this application are implemented as follows: In a first aspect, embodiments of this application provide a matrix multiplier, comprising: a multiplier array, a data preprocessing module, and an accumulator; the multiplier array is configured to perform multiplication operations on an input first matrix and a second matrix; the data preprocessing module is configured to preprocess a third matrix to obtain a mask matrix; the accumulator is configured to add the result of the multiplication operation between the first matrix and the second matrix to the mask matrix; wherein the first matrix is ​​an M×K matrix, the second matrix is ​​a K×N matrix, the third matrix and the mask matrix are both M×N matrices, and M, N and K are all integers greater than or equal to 1.

[0006] In the above embodiments, a matrix multiplier that integrates a mask matrix is ​​provided. By improving the existing matrix multiplier, a data preprocessing module is added to obtain the mask matrix, so that the accumulator can add the result of the multiplication of the first matrix and the second matrix to the mask matrix. By integrating the masking process into the matrix multiplication operation, the matrix multiplication operation and the masking operation can be performed synchronously, thereby effectively improving the Attention calculation speed.

[0007] In one possible implementation of the first aspect embodiment, the data preprocessing module is specifically configured to: obtain a mask mapping matrix based on the third matrix, wherein the size of the mask mapping matrix is ​​the same as the size of the third matrix; determine the positions in the third matrix that need to be masked based on the mask mapping matrix and mask parameters; and replace the elements at the corresponding positions in the third matrix according to the positions to obtain the mask matrix.

[0008] In the above embodiments, when preprocessing the third matrix, a mask mapping matrix for determining the parts of the third matrix that need to be masked can be obtained first. Then, by combining the mask parameters, the positions in the third matrix that need to be masked can be quickly determined. After the positions are determined, the required mask matrix can be quickly obtained simply by replacing the elements corresponding to those positions, which helps to improve the speed.

[0009] In one possible implementation of the first aspect embodiment, when the data preprocessing module obtains the mask mapping matrix based on the third matrix, it is specifically configured to: subtract the row index value and column index value of each element position in the third matrix, and obtain the mask mapping matrix based on the subtraction result.

[0010] In the above embodiment, the mask mapping matrix can be obtained by subtracting the row index and column index of each element in the third matrix. The subtraction can be performed by subtracting the column index of the element from its row index, or vice versa. For example, if Cij represents an element in the third matrix and Pij represents an element in the mask mapping matrix, then Pij = ij, or Pij = ji.

[0011] In one possible implementation of the first aspect embodiment, when the data preprocessing module determines the position in the third matrix that needs to be masked based on the mask mapping matrix and the mask parameters, it is specifically configured to: compare each element in the mask mapping matrix with the mask parameters, and determine the position in the third matrix that needs to be masked based on the comparison result; wherein, if the value of each element in the mask mapping matrix is ​​obtained by subtracting the row index value from the column index value of the element position in the third matrix, then the position of the element in the mask mapping matrix whose value is greater than the mask parameters is the position in the third matrix that needs to be masked; if the value of each element in the mask mapping matrix is ​​obtained by subtracting the column index value from the row index value of the element position in the third matrix, then the position of the element in the mask mapping matrix whose value is less than the mask parameters is the position in the third matrix that needs to be masked.

[0012] In the above embodiments, after obtaining the mask mapping matrix, by comparing each element in the mask mapping matrix with the mask parameter, it is possible to quickly determine whether the position of the element in the third matrix is ​​the position that needs to be masked based on the comparison result. For example, let Pij represent the element in the mask mapping matrix. If Pij = ji and Pij is greater than the mask parameter, then the position of Pij is the position that needs to be masked. Conversely, if Pij = ij and Pij is less than the mask parameter, then the position of Pij is the position that needs to be masked.

[0013] In one possible implementation of the first aspect embodiment, when the data preprocessing module replaces the element at the corresponding position in the third matrix according to the position to obtain the mask matrix, it is specifically configured to: replace the value of the element corresponding to the position to be masked in the third matrix with negative infinity to obtain the mask matrix.

[0014] In the above embodiment, when replacing the element at the corresponding position in the third matrix, the value of the element corresponding to the position in the third matrix that needs to be masked is replaced with negative infinity. Then, when the result of the multiplication operation between the first matrix and the second matrix is ​​added, the result is also negative infinity (negative infinity + a smaller value = negative infinity), thereby achieving the masking effect.

[0015] In one possible implementation of the first aspect embodiment, the data preprocessing module includes: a subtractor array, a comparator array, and a processing unit; the subtractor array is configured to subtract the row index value and column index value of each element position in the third matrix; the comparator array is configured to compare the subtraction result of each element with a mask parameter to obtain a comparison result; the processing unit is configured to determine the position in the third matrix that needs to be masked based on the comparison result, and replace the element at the corresponding position in the third matrix according to the position to obtain the mask matrix.

[0016] In the above embodiments, the data preprocessing module with the above structure implements the above functions. Since subtractor arrays and comparator arrays are mature and commonly used devices, using such mature devices can increase the reliability of the circuit and the speed of improvement. In addition, if the chip already has subtractor arrays and comparator arrays, these devices can be directly reused, which can reduce the need for circuit improvements.

[0017] In one possible implementation of the first aspect embodiment, the processing unit includes: a storage array and a read / write controller; the storage array is configured to store the third matrix; the read / write controller is configured to replace the element at the corresponding position in the third matrix according to the position to obtain the mask matrix.

[0018] In the above embodiments, since the storage array and read / write controller are both commonly used circuits in the chip, the purpose of replacing the elements at the corresponding positions in the third matrix is ​​achieved by reusing the existing circuit structure in the chip. This reduces the need for circuit modifications and, while minimizing circuit modifications, achieves the goal of integrating the masking process into the matrix multiplication operation, thereby improving the operation speed.

[0019] Secondly, embodiments of this application also provide a chip including a matrix multiplier provided as performed as described in the first aspect embodiments and / or in combination with the first aspect embodiments.

[0020] In one possible implementation of the second aspect embodiment, the chip includes a tensor core, the tensor core including the matrix multiplier.

[0021] In the above embodiments, since the tensor core has a computational speed an order of magnitude higher than that of the general-purpose computing core (CUDA Core), when the tensor core includes the aforementioned matrix multiplier, the aforementioned Q×K... T +Mask) can also be computed on the tensor core, which can improve the computation speed.

[0022] Thirdly, embodiments of this application also provide an electronic device, including a chip provided as in the second aspect embodiment and / or a possible implementation in conjunction with the second aspect embodiment.

[0023] Fourthly, embodiments of this application also provide a data processing method that can be applied to a tensor core in an electronic device. The method includes: multiplying an input first matrix with a second matrix; preprocessing a third matrix to obtain a mask matrix; and adding the result of the multiplication of the first matrix with the second matrix to the mask matrix. The first matrix is ​​an M×K matrix, the second matrix is ​​a K×N matrix, and the third matrix is ​​an M×N matrix, where M, N, and K are all integers greater than or equal to 1.

[0024] In one possible implementation of the fourth aspect embodiment, the third matrix is ​​preprocessed to obtain a mask matrix, including: obtaining a mask mapping matrix based on the third matrix, wherein the size of the mask mapping matrix is ​​the same as the size of the third matrix; determining the positions in the third matrix that need to be masked based on the mask mapping matrix and mask parameters; and replacing the elements at the corresponding positions in the third matrix according to the positions to obtain the mask matrix.

[0025] In one possible implementation of the fourth aspect embodiment, obtaining the mask mapping matrix based on the third matrix includes: subtracting the row index value and column index value of each element position in the third matrix, and obtaining the mask mapping matrix based on the subtraction result.

[0026] In one possible implementation of the fourth aspect embodiment, determining the position in the third matrix that needs to be masked based on the mask mapping matrix and the mask parameters includes: comparing each element in the mask mapping matrix with the mask parameters, and determining the position in the third matrix that needs to be masked based on the comparison result; wherein, if the value of each element in the mask mapping matrix is ​​obtained by subtracting the row index value from the column index value of the element position in the third matrix, then the position of the element in the mask mapping matrix whose value is greater than the mask parameters is the position in the third matrix that needs to be masked; if the value of each element in the mask mapping matrix is ​​obtained by subtracting the column index value from the row index value of the element position in the third matrix, then the position of the element in the mask mapping matrix whose value is less than the mask parameters is the position in the third matrix that needs to be masked.

[0027] In one possible implementation of the fourth aspect embodiment, replacing the element at the corresponding position in the third matrix according to the position to obtain the mask matrix includes: replacing the value of the element corresponding to the position to be masked in the third matrix with negative infinity to obtain the mask matrix.

[0028] Fifthly, embodiments of this application also provide an electronic device, characterized in that it includes: a memory and a processor, the processor being connected to the memory; the memory being used to store a program; the processor being used to invoke the program stored in the memory to perform a method provided as described in the fourth aspect embodiments and / or in combination with any possible implementation of the fourth aspect embodiments.

[0029] Sixthly, embodiments of this application also provide a computer-readable storage medium, characterized in that a computer program is stored thereon, which, when executed by a processor, performs the method provided by any possible implementation of the embodiments of the fourth aspect and / or in combination with the embodiments of the fourth aspect.

[0030] In a seventh aspect, embodiments of this application also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the methods provided in any possible implementation of the embodiments of the fourth aspect and / or in combination with the embodiments of the fourth aspect.

[0031] Other features and advantages of this application will be set forth in the following description. The objectives and other advantages of this application can be realized and obtained through the structures specifically pointed out in the written description and the accompanying drawings. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings. The above and other objects, features, and advantages of this application will become clearer through the accompanying drawings.

[0033] Figure 1 A schematic diagram of the structure of a matrix multiplier provided in an embodiment of this application is shown.

[0034] Figure 2 A schematic diagram of another matrix multiplier provided in an embodiment of this application is shown.

[0035] Figure 3 This illustration shows a schematic diagram of an 8×8 mask mapping matrix provided in an embodiment of this application.

[0036] Figure 4 The diagram illustrates the principle of a data preprocessing module provided in an embodiment of this application.

[0037] Figure 5 A flowchart illustrating a data processing method provided in an embodiment of this application is shown.

[0038] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The following embodiments are provided as examples to more clearly illustrate the technical solutions of this application, and should not be used to limit the scope of protection of this application. Those skilled in the art will understand that, without conflict, the following embodiments and features can be combined with each other.

[0040] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, relational terms such as "first," "second," etc., in the description of this application are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0041] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0042] In the description of the embodiments of this application, unless otherwise expressly specified and limited, the technical term "connection" can be a direct connection or an indirect connection through an intermediate medium.

[0043] Given the low computation speed of current Attention operations, this application provides a matrix multiplier that incorporates a mask matrix. By integrating the masking process into the matrix multiplication operation, the computation speed of Attention is effectively improved.

[0044] The formula for calculating attention is as follows: Attention(Q, K, V) = softmax((Q×) +Mask) / Mask, Q, K, V, ... in this formula All are matrices. Let K be the transpose of matrix K. D represents the hidden parameters of the model (constant values), which vary depending on the model. The softmax function is the normalized exponential function. Q×K T It is a matmul (matrix multiplication) calculation, while (Q×K) T +Mask) is a non-matmul computation. Matmul operations are typically computed using Tensor Cores, while non-matmul computations are computed using CUDA Cores. Therefore, the two types of computations cannot be performed in parallel, which affects the computation speed of Attention.

[0045] Attention Mask is a core mechanism in the Transformer architecture, primarily used to control the scope of data the model "sees" when processing sequential data. It is commonly used in Natural Language Processing (NLP) and other fields (such as image and speech processing). For example, it can be applied to NLP text generation, NLP machine translation, image segmentation, and speech recognition. Attention Mask can be used in the following situations: 1. Batch processing contains sequences of unequal length; 2. Autoregressive generation (text / image / speech); 3. Preventing information leakage (training / inference, such as masking polar words in sentiment analysis); 4. Reducing the computational complexity of long sequences; 5. Cross-modal alignment control, etc.

[0046] For batch processing with sequences of unequal length: Input data can be padded to the same length to meet batch processing requirements, and invalid information can be masked using an Attention Mask. For autoregressive generation: In generative tasks (such as text continuation, image patch generation, and speech synthesis), to ensure that the model can only rely on tokens generated before it (to the left) when generating the output at the current position, and cannot "peek" at future information, an Attention Mask can be used to mask future information. To prevent information leakage: During model training or evaluation, an Attention Mask can be used to actively mask specific parts of the input sequence to prevent the model from using this information to "cheate" or test the model's robustness to specific information. For cross-modal alignment control: In multimodal tasks (such as visual question answering, image and text retrieval, image caption generation, and speech recognition), the model needs to process information from different modalities (text, image, audio, video) and establish correspondences between them. An Attention Mask can be used to precisely control which parts of another modality the representation of one modality pays attention to when calculating attention.

[0047] This application provides a matrix multiplier that incorporates a mask matrix, which will be described below in conjunction with... Figure 1 The principle behind this matrix multiplier is explained below. It includes a multiplier array, an accumulator, and a data preprocessing module (also known as a mask data preprocessing module). Both the multiplier array and the data preprocessing module are connected to the accumulator. This application improves upon existing matrix multipliers by adding a data preprocessing module to obtain the mask matrix. This allows the accumulator to add the result of multiplying the first and second matrices to the mask matrix. By integrating the masking process into the matrix multiplication operation, the Attention computation speed is effectively improved.

[0048] In some other possible implementations, such as Figure 2 As shown, a matrix multiplier can also include: data input and data output. Data input is used to input the matrices to be operated on, such as inputting the first matrix (which can be called the left matrix, for example, denoted by A), the second matrix (which can be called the right matrix, for example, denoted by B), and the third matrix (which can also be called the accumulation matrix, for example, denoted by C). Data output is used to output the final operation result.

[0049] Understandably, the elements in the first, second, and third matrices have different meanings in different application scenarios. For example, in image processing, the elements in the first matrix can be the original pixel values ​​of the input image patch, the elements in the second matrix can be the weights of the convolution kernel, and the elements in the third matrix can be the biases. The corresponding formula is: Output feature map = Input image patch × Convolution kernel + Bias. As another example, in text processing, the elements in the first matrix can be word vectors, the elements in the second matrix can be inter-word relevance weights, and the elements in the third matrix can be residual inputs. The corresponding formula is: Output word vector = Word embedding × Inter-word relevance weights + Residual. Furthermore, for cross-modal alignment, the elements in the first matrix can be image features, the elements in the second matrix can be text word features, and the elements in the third matrix can be modal biases. The corresponding formula is: Image-text similarity = Image features × Text word features + Modal bias.

[0050] The multiplier array is configured to perform multiplication operations between the input first matrix and the second matrix. For example, the first matrix is ​​an M×K matrix (representing M rows and K columns), and the second matrix is ​​a K×N matrix, where M, N, and K are all integers greater than or equal to 1. In some possible implementations, M can be equal to K, or N can be equal to K and M can be equal to N, or M=N=K. The multiplier array includes multiple multipliers.

[0051] The accumulator is configured to add the result of the multiplication of the first matrix and the second matrix to the mask matrix. Since the first matrix has a dimension (size) of M×K and the second matrix has a dimension of K×N, the dimension of the result of the multiplication of the first and second matrices is M×N.

[0052] Understandably, an M×N matrix D containing all zeros can be created to store the results of multiplying the first and second matrices and adding the mask matrices. For each element Dij in D, the i-th row of A and the j-th column of B are selected, and the corresponding elements of the i-th row of A and the j-th column of B are multiplied (a total of K elements). Then, the result of multiplying the K elements is accumulated with the corresponding elements in matrix C to obtain the value of Dij. In this example, the values ​​of i are 1 to M, and the values ​​of j are 1 to N. For example, assuming that the first, second, and third matrices are all 8×8 matrices, then D11 = A11×B11 + A12×B21 + A13×B31 + A14×B41 + A15×B51 + A16×B61 + A17×B71 + A18×B81 + C11. Similarly, D12 = A11×B12 + A12×B22 + A13×B32 + A14×B42 + A15×B52 + A16×B62 + A17×B72 + A18×B82 + C12. And so on, D22 = A21×B12 + A22×B22 + A23×B32 + A24×B42 + A25×B52 + A26×B62 + A27×B72 + A28×B82 + C22.

[0053] The data preprocessing module is configured to preprocess the third matrix to obtain a mask matrix, where both the third matrix and the mask matrix are M×N matrices. A mask is added to the third matrix to filter out useless information. The mask matrix is ​​a matrix of the same size as the input sequence (third matrix). When calculating self-attention, the weights corresponding to the padding are set to a minimum value (or negative infinity), making the weights output by the softmax function close to zero, thus ignoring the padding. For example, in some scenarios, the attention mask can be a causal mask to ensure that each position only sees previous positions, thereby maintaining the order of processing. The causal mask is usually presented as an upper triangular matrix, where the elements on the diagonal and below are 1 (allowing attention), while the elements above the diagonal are 0 (preventing attention). In practice, this 0 can be converted to negative infinity to achieve masking. For example, for a sequence of length N, its causal mask matrix C satisfies: Cij = 0 if j > i (i.e., future position), otherwise Cij = 1.

[0054] In one possible implementation, the data preprocessing module is specifically configured to: obtain a mask mapping matrix (also called a mapping table) based on the third matrix, wherein the size of the mask mapping matrix is ​​the same as the size of the third matrix; determine the positions in the third matrix that need to be masked based on the mask mapping matrix and mask parameters (e.g., represented by mask); and replace the elements at the corresponding positions in the third matrix to obtain the mask matrix. The mask matrix is ​​the replaced third matrix. Using the mapping table, the positions in the third matrix that need to be masked can be quickly determined, and then the elements at the corresponding positions in the third matrix can be replaced, thus quickly obtaining the mask matrix, which helps to improve the preprocessing speed.

[0055] The mask parameter can be configured in advance or entered in real time, and the specific value can be set as needed.

[0056] In one possible implementation, when the data preprocessing module obtains the mask mapping matrix based on the third matrix, it is specifically configured to: subtract the row index value (number or row coordinate) and column index value (number or column coordinate) of each element position in the third matrix, and obtain the mask mapping matrix based on the subtraction result. This can be done by subtracting the column index value from the row index value of each element position in the third matrix, or conversely, by subtracting the row index value from the column index value of each element position in the third matrix.

[0057] Taking an 8×8 C matrix as an example, if we subtract the row index from the column index of each element, we can obtain the following: Figure 3 The mask mapping matrix is ​​shown. Assuming P represents the mask mapping matrix, we can first create an 8×8 blank or all-zero matrix P, then Pij = ji ( Figure 3 (as shown) or Pij=ij, for example, Figure 3 In the example, P11 = 1 - 1 = 0, P12 = 2 - 1 = 1, ..., P17 = 7 - 1 = 6, P18 = 8 - 1 = 7. It is understandable that the masking matrix is ​​not limited to the 8×8 shown in the example; its size depends on the size of the third matrix.

[0058] In one possible implementation, when the data preprocessing module determines the positions in the third matrix that need to be masked based on the mask mapping matrix and the mask parameters, it is specifically configured to: compare each element in the mask mapping matrix with the mask parameters, and determine the positions in the third matrix that need to be masked based on the comparison results; wherein, if the value of each element in the mask mapping matrix is ​​obtained by subtracting the row index value from the column index value of the element position in the third matrix, then the positions of the elements in the mask mapping matrix whose values ​​are greater than the mask parameters are the positions in the third matrix that need to be masked, otherwise, they do not need to be masked; if the value of each element in the mask mapping matrix is ​​obtained by subtracting the column index value from the row index value of the element position in the third matrix, then the positions of the elements in the mask mapping matrix whose values ​​are less than the mask parameters are the positions in the third matrix that need to be masked, otherwise, they do not need to be masked.

[0059] That is, if Pij = ji and Pij is greater than the mask parameter, then the position of Pij is the position that needs to be masked; conversely, if Pij = ij and Pij is less than the mask parameter, then the position of Pij is the position that needs to be masked. Figure 3 Taking the masking matrix shown as an example, when the input mask parameter is 0, all points in matrix P with values ​​greater than 0 need to be masked, and all data in the upper right corner of matrix C needs to be replaced. When the input mask parameter is 7, no values ​​in matrix P need to be masked, and no data in matrix C needs to be replaced. When the input mask parameter is -8, all values ​​in matrix P need to be masked, and all data in matrix C needs to be replaced.

[0060] In one implementation, when the data preprocessing module replaces elements at corresponding positions in the third matrix to obtain a mask matrix, it is specifically configured to: replace the values ​​of the elements corresponding to the positions in the third matrix that need to be masked with negative infinity (represented by -inf), thus obtaining the mask matrix; that is, replacing the elements corresponding to the positions that need to be masked with -inf. It is understood that when replacing the values ​​of the elements corresponding to the positions in the third matrix that need to be masked, it is also possible not to replace them with -inf, but with other specified values, such as -100, etc.

[0061] In one possible implementation, the principle of the above-described data preprocessing module can be as follows: Figure 4 As shown. The column number j of each element in the input C matrix can be subtracted from the row number i. Then, a comparator is used to compare ji with the input mask parameter mask to obtain the comparison result. If (ji) ≤ mask, the value of the element remains unchanged; if (ji) > mask, the value of the element is replaced with negative infinity. Figure 4In this case, (ji) <= mask means that (ji) is less than or equal to mask.

[0062] In one possible implementation, the data preprocessing module may include a field-programmable gate array (FPGA) or other programmable logic devices (PLDs), such as complex programmable logic devices (CPLDs), programmable logic controllers (PLCs), etc. Alternatively, the data preprocessing module may also be implemented in a manner that includes, for example, a processing core.

[0063] In another possible implementation, the data preprocessing module includes a subtractor array, a comparator array, and a processing unit. The subtractor array is connected to the comparator array, and the comparator array is connected to the processing unit.

[0064] The subtractor array is configured to subtract the row index and column index of each element in the third matrix, either by subtracting the column index from the row index or vice versa. The subtractor array can contain multiple subtractors.

[0065] A comparator array is configured to compare the result of subtracting each element with a mask parameter, comparing ji or ij with the mask parameter mask to obtain the comparison result. A comparator array can include multiple comparators.

[0066] The processing unit is configured to determine the positions in the third matrix that need to be masked based on the comparison results, and then replace the elements at the corresponding positions in the third matrix to obtain the mask matrix. Figure 4 For example, if (ji) > mask, then Cij is determined to be the position that needs to be masked, and then the value of Cij can be replaced, for example, by replacing -inf.

[0067] In some possible implementations, the processing unit may be implemented in other ways that can achieve the above functions. For example, the processing unit may include a field-programmable gate array (FPGA) or other programmable logic devices, or it may include a processing core.

[0068] In some other possible implementations, the processing unit may include a storage array and a read / write controller connected to the storage array. The storage array is configured to store a third matrix. The read / write controller is configured to replace elements at corresponding positions in the third matrix according to their positions to obtain a mask matrix. For example, if (ji) > mask, the read / write controller updates the value of Cij in the storage array to -inf.

[0069] The subtractor array, comparator array, memory array, and read / write controller mentioned above are all commonly used devices in chips. By reusing the existing devices in the chip to preprocess the third matrix to obtain the mask matrix, the improvement of the circuit structure can be reduced, thus achieving the purpose while saving costs.

[0070] The aforementioned storage array can be a register array or a storage array within a memory. Memory includes, but is not limited to: Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0071] This application also provides a data processing method that can be applied to an electronic device. Further, it can be applied to a processor in an electronic device, or to a matrix multiplier in an electronic device. The processes or functions included in this data processing method can be implemented based on hardware or based on software functional modules. The following describes the implementation in conjunction with... Figure 5 The principle behind it will be explained.

[0072] S1: Perform a multiplication operation between the first input matrix and the second input matrix.

[0073] When this data processing method is applied to a matrix multiplier in an electronic device, it can be a multiplier array in the matrix multiplier, which performs multiplication operations on the input first matrix and the second matrix.

[0074] S2: Preprocess the third matrix to obtain the mask matrix.

[0075] When this data processing method is applied to a matrix multiplier in an electronic device, the data preprocessing module in the matrix multiplier can preprocess the third matrix to obtain the mask matrix.

[0076] S3: Add the result of the multiplication of the first matrix and the second matrix to the mask matrix.

[0077] When this data processing method is applied to a matrix multiplier in an electronic device, it can be an accumulator in the matrix multiplier, which adds the result of the multiplication of the first matrix and the second matrix to the mask matrix.

[0078] In some possible implementations, the above-described process of preprocessing the third matrix to obtain the mask matrix may include: obtaining a mask mapping matrix based on the third matrix, wherein the size of the mask mapping matrix is ​​the same as the size of the third matrix; determining the positions in the third matrix that need to be masked based on the mask mapping matrix and mask parameters; and replacing the elements at the corresponding positions in the third matrix to obtain the mask matrix.

[0079] The process of obtaining the mask mapping matrix based on the third matrix may include: subtracting the row index and column index of each element in the third matrix, and obtaining the mask mapping matrix based on the subtraction result.

[0080] When determining the masked positions in the third matrix based on the mask mapping matrix and mask parameters, the process may include: comparing each element in the mask mapping matrix with the mask parameters, and determining the masked positions in the third matrix based on the comparison results; wherein, if the value of each element in the mask mapping matrix is ​​obtained by subtracting the row index from the column index of that element position in the third matrix, then the positions of the elements in the mask mapping matrix whose values ​​are greater than the mask parameters are the masked positions in the third matrix; if the value of each element in the mask mapping matrix is ​​obtained by subtracting the column index from the row index of that element position in the third matrix, then the positions of the elements in the mask mapping matrix whose values ​​are less than the mask parameters are the masked positions in the third matrix.

[0081] The mask matrix is ​​obtained by replacing the elements at the corresponding positions in the third matrix according to their positions. The process may include replacing the values ​​of the elements at the positions to be masked in the third matrix with negative infinity.

[0082] The implementation principle and technical effects of the method embodiment are the same as those of the aforementioned matrix multiplier embodiment. For the sake of brevity, any parts not mentioned in the method embodiment can be referred to the corresponding content in the aforementioned matrix multiplier embodiment.

[0083] This application also provides a chip that includes the matrix multiplier described above. In some possible implementations, the chip may include a tensor core, wherein the tensor chip includes the matrix multiplier. It is understood that the matrix multiplier described above may also be disposed in a non-tensor chip.

[0084] Furthermore, the chip may also include a general-purpose computing core. By improving the existing matrix multiplier in the tensor core and integrating the masking process into the matrix multiplication operation, Q×K... T With (Q×K) T Both +Mask and the attention function can be computed in the tensor core, which can improve the computation speed of attention.

[0085] The chip can be any chip that includes a tensor core; for example, it can be an integrated circuit chip with signal processing capabilities. The aforementioned chip can be a processor chip. A processor can be a general-purpose processor, including a Central Processing Unit (CPU), Network Processor (NP), Graphics Processing Unit (GPU), Accelerated Processing Unit (ASIC), Multimedia Application Processor (MAP), microprocessor, etc.; it can also be a Digital Signal Processor (DSP), Tensor Processing Unit (TPU), Neural Processing Unit (NPU), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0086] The matrix multiplier provided in the chip embodiment has the same implementation principle and technical effect as the aforementioned matrix multiplier embodiment. For the sake of brevity, any parts not mentioned in the chip embodiment can be referred to the corresponding content in the aforementioned matrix multiplier embodiment.

[0087] This application also provides an electronic device that may include the chip described above. When the chip is a processor, in one embodiment, the electronic device includes: a transceiver, a memory, a communication bus, and a processor. The transceiver, memory, and processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected through one or more communication buses or signal lines. The transceiver is used to send and receive data. The memory stores computer programs, including at least one software functional module that can be stored in the memory as software or firmware or embedded in the operating system (OS) of the electronic device. The processor executes the software functional module or computer program stored in the memory. For example, the processor executes the data processing method described above.

[0088] The memory may be, but is not limited to, Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0089] The processor may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), Network Processor (NP), Graphics Processing Unit (GPU), Accelerated Processing Unit (ACCU), Multimedia Application Processor (MAP), microprocessor, etc.; it can also be a Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. Alternatively, the processor can be any conventional processor.

[0090] The aforementioned electronic devices include, but are not limited to, mobile terminals (phones, tablets), computing terminals (such as laptops, desktop computers, servers), and network devices (such as switches, routers).

[0091] This application also provides a non-volatile computer-readable storage medium (hereinafter referred to as the storage medium) storing a computer program, which, when run by a computer such as the electronic device described above, executes the data processing method described above.

[0092] This application also provides a computer program product, which includes a computer program. When the computer program is executed by a computer, it performs the data processing method described above.

[0093] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0095] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0096] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, laptop, server, or electronic device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned computer-readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A matrix multiplier, characterized in that, include: A multiplier array is configured to perform multiplication operations between the first input matrix and the second input matrix; The data preprocessing module is configured to preprocess the third matrix to obtain the mask matrix; An accumulator is configured to add the result of multiplying the first matrix and the second matrix to the mask matrix; Wherein, the first matrix is ​​an M×K matrix, the second matrix is ​​a K×N matrix, the third matrix and the mask matrix are both M×N matrices, and M, N and K are all integers greater than or equal to 1.

2. The matrix multiplier according to claim 1, characterized in that, The data preprocessing module is specifically configured as follows: A mask mapping matrix is ​​obtained based on the third matrix, wherein the size of the mask mapping matrix is ​​the same as the size of the third matrix; The positions in the third matrix that need to be masked are determined based on the mask mapping matrix and the mask parameters. The mask matrix is ​​obtained by replacing the corresponding element in the third matrix according to the position.

3. The matrix multiplier according to claim 2, characterized in that, When the data preprocessing module obtains the mask mapping matrix based on the third matrix, it is specifically configured as follows: Subtract the row index and column index of each element in the third matrix to obtain the mask mapping matrix based on the subtraction result.

4. The matrix multiplier according to claim 2, characterized in that, When the data preprocessing module determines the positions in the third matrix that need to be masked based on the mask mapping matrix and mask parameters, it is specifically configured as follows: Each element in the mask mapping matrix is ​​compared with the mask parameters, and the position in the third matrix that needs to be masked is determined based on the comparison result. Wherein, if the value of each element in the mask mapping matrix is ​​obtained by subtracting the row index value from the column index value of the element position in the third matrix, then the position of the element in the mask mapping matrix whose value is greater than the mask parameter is the position in the third matrix that needs to be masked. If the value of each element in the mask mapping matrix is ​​obtained by subtracting the column index from the row index of the element in the third matrix, then the position of the element in the mask mapping matrix whose value is less than the mask parameter is the position in the third matrix that needs to be masked.

5. The matrix multiplier according to claim 2, characterized in that, When the data preprocessing module replaces the elements at the corresponding positions in the third matrix according to the positions to obtain the mask matrix, it is specifically configured as follows: The values ​​of the elements corresponding to the positions in the third matrix that need to be masked are replaced with negative infinity to obtain the mask matrix.

6. The matrix multiplier according to claim 1, characterized in that, The data preprocessing module includes: The subtractor array is configured to subtract the row index and column index of each element in the third matrix; The comparator array is configured to compare the result of subtracting each element with the mask parameter to obtain the comparison result; The processing unit is configured to determine the position in the third matrix that needs to be masked based on the comparison result, and replace the element at the corresponding position in the third matrix according to the position to obtain the mask matrix.

7. The matrix multiplier according to claim 6, characterized in that, The processing unit includes: A storage array is configured to store the third matrix; The read / write controller is configured to replace the element at the corresponding position in the third matrix according to the position to obtain the mask matrix.

8. A chip, characterized in that, Includes a matrix multiplier as described in any one of claims 1-7.

9. The chip according to claim 8, characterized in that, The chip includes a tensor core, and the tensor core includes the matrix multiplier.

10. An electronic device, characterized in that, Includes the chip as described in claim 8 or 9.

11. A data processing method, characterized in that, Applied to electronic devices, the method includes: Perform a multiplication operation between the first and second input matrices; Preprocess the third matrix to obtain the mask matrix; The result of the multiplication of the first matrix and the second matrix is ​​added to the mask matrix; wherein the first matrix is ​​an M×K matrix, the second matrix is ​​a K×N matrix, and the third matrix is ​​an M×N matrix, where M, N, and K are all integers greater than or equal to 1.

12. The method according to claim 11, characterized in that, The third matrix is ​​preprocessed to obtain the mask matrix, which includes: A mask mapping matrix is ​​obtained based on the third matrix, wherein the size of the mask mapping matrix is ​​the same as the size of the third matrix; The positions in the third matrix that need to be masked are determined based on the mask mapping matrix and the mask parameters. The mask matrix is ​​obtained by replacing the corresponding element in the third matrix according to the position.

13. The method according to claim 12, characterized in that, The mask mapping matrix is ​​obtained based on the third matrix, including: Subtract the row index and column index of each element in the third matrix to obtain the mask mapping matrix based on the subtraction result.

14. The method according to claim 12, characterized in that, Determining the positions in the third matrix that need to be masked based on the mask mapping matrix and mask parameters includes: Each element in the mask mapping matrix is ​​compared with the mask parameters, and the position in the third matrix that needs to be masked is determined based on the comparison result. Wherein, if the value of each element in the mask mapping matrix is ​​obtained by subtracting the row index value from the column index value of the element position in the third matrix, then the position of the element in the mask mapping matrix whose value is greater than the mask parameter is the position in the third matrix that needs to be masked. If the value of each element in the mask mapping matrix is ​​obtained by subtracting the column index from the row index of the element in the third matrix, then the position of the element in the mask mapping matrix whose value is less than the mask parameter is the position in the third matrix that needs to be masked.

15. The method according to claim 12, characterized in that, The mask matrix is ​​obtained by replacing the elements at the corresponding positions in the third matrix according to the stated positions, including: The values ​​of the elements corresponding to the positions in the third matrix that need to be masked are replaced with negative infinity to obtain the mask matrix.

16. An electronic device, characterized in that, include: A memory and a processor, wherein the processor is connected to the memory; The memory is used to store programs; The processor is configured to invoke a program stored in the memory to execute the method as described in any one of claims 11-15.

17. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, performs the method as described in any one of claims 11-15.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 11-15.

Images