Low-complexity Transform attention module prediction method and device

Abstract

The invention relates to a low-complexity Transform attention module prediction method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a first leading zero counting result obtained by performing leading zero counting processing on a first matrix of a neural network model; an attention weight matrix of the neural network model is a first matrix, and a lexical matrix input into the neural network model is a second matrix; or the lexical matrix input into the neural network model is a first matrix, and the attention weight matrix of the neural network model is a second matrix; based on the first leading zero counting result, performing multiplication approximate shift processing on the second matrix to obtain a query matrix and a key matrix; determining a prediction result of the attention matrix based on the query matrix and the key matrix; the prediction result of the attention matrix is used for representing the matching degree of the context lexical elements. By adopting the method, the hardware resource overhead can be reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a low-complexity Transformer attention module prediction method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Thanks to the attention mechanism that can capture contextual relationships, the neural network model based on the Transformer architecture has achieved great success in natural language processing. However, as the sequence of the input model grows, the computational complexity and storage complexity of the attention mechanism also increase dramatically. This is because the computational and storage requirements of the attention module are proportional to the quadratic length of the sequence. Therefore, how to effectively reduce the complexity of the attention module has become a focus of current academic research.

[0003] Thanks to the natural redundancy of human language, recent studies have found that it is possible to use the sparsity between tokens to simplify the computation and storage complexity of the attention module. Specifically, we can first get an estimate of the attention matrix through some relatively low-complexity calculations. Based on this estimated attention matrix, we sort the values ​​of each row, select the largest K values, and assume that these K largest values ​​have a greater impact on attention. In subsequent calculations, we only need to perform calculations corresponding to these K values. In this way, the calculation and storage of attention can be effectively reduced in formal calculations. Therefore, a low-complexity Transformer attention module prediction method that can achieve attention matrix estimation is very necessary.

[0004] The related low-complexity Transformer attention module prediction method is to use low-bit quantization to predict the attention matrix. Specifically, the word matrix corresponding to the text of the input model is quantized to 4 bits with low precision, and the query weight matrix W is Q and key weight matrix W K 4-bit quantization is performed, and 4-bit multiplication is performed after quantization to obtain the corresponding query (Querry, Q) matrix and key (Key, K) matrix; then, the Q matrix and K matrix are further quantized by 4 bits, and after quantization, they are multiplied to obtain the attention matrix. However, the above method requires the use of integer multipliers for hardware implementation, and the hardware overhead of 4-bit integer multipliers is still huge. Therefore, the related low-complexity Transformer attention module prediction method has large hardware resource overhead. Summary of the invention

[0005] Based on this, it is necessary to provide a low-complexity Transformer attention module prediction method, device, computer equipment, computer-readable storage medium and computer program product that can reduce hardware resource overhead in response to the above technical problems.

[0006] In a first aspect, the present application provides a low-complexity Transformer attention module prediction method, comprising:

[0007] Obtaining a first leading zero count result obtained by performing leading zero count processing on a first matrix of a neural network model; the attention weight matrix of the neural network model is the first matrix, and the word unit matrix input to the neural network model is the second matrix; or the word unit matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix;

[0008] Based on the first leading zero count result, performing multiplication approximate shift processing on the second matrix to obtain a query matrix and a key matrix;

[0009] Based on the query matrix and the key matrix, a prediction result of the attention matrix is ​​determined; the prediction result of the attention matrix is ​​used to characterize the matching degree of the context word.

[0010] In one embodiment, determining a prediction result of an attention matrix based on the query matrix and the key matrix includes:

[0011] Performing leading zero counting processing on the third matrix of the neural network model to obtain a second leading zero counting result corresponding to the third matrix; the query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix; or the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix;

[0012] Based on the second leading zero count result, the fourth matrix is ​​subjected to multiplication approximate shift processing to obtain a prediction result of the attention matrix.

[0013] In one embodiment, the first matrix is ​​the attention weight matrix, and the obtaining of a first leading zero count result obtained by performing leading zero count processing on the first matrix of the neural network model includes:

[0014] Performing leading zero counting processing on the attention weight matrix of the neural network model to obtain a first leading zero counting result corresponding to the attention weight matrix;

[0015] storing the first leading zero count result into a target storage unit;

[0016] When estimating the attention matrix, the first leading zero count result is read from the target storage unit.

[0017] In one embodiment, the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix, and the fourth matrix is ​​subjected to multiplication approximation shift processing based on the second leading zero count result to obtain a prediction result of the attention matrix, including:

[0018] For each group of multiplication elements in the transpose multiplication process of the query matrix and the key matrix, determine the shift object element corresponding to the second element in the group of multiplication elements according to the first element in the group of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix;

[0019] Based on the number of leading zeros corresponding to the first element, the shift object element is shifted to obtain a first shift result corresponding to the group multiplication element;

[0020] Performing sign bit extension on the first shift result to obtain a product corresponding to the group multiplication element;

[0021] Based on the products corresponding to the groups of multiplied elements, the prediction results of the attention matrix are determined.

[0022] In one embodiment, determining the shift object element corresponding to the second element in the group of multiplied elements according to the first element in the group of multiplied elements includes:

[0023] If the first element in the group of multiplied elements is a negative number, the original code of the second element in the group of multiplied elements is bitwise inverted to obtain a shift object element corresponding to the second element;

[0024] If the first element in the group of multiplied elements is a positive number, the original code of the second element in the group of multiplied elements is used as the shift object element corresponding to the second element.

[0025] In one embodiment, the shifting process is performed on the shift object element based on the number of leading zeros corresponding to the first element to obtain the first shift result corresponding to the group multiplication element, including:

[0026] Determine, according to the sign bit of the first element and the sign bit of the second element, the sign bit of the first shift result corresponding to the group of multiplied elements;

[0027] Determine the number of shift bits corresponding to the shift object element according to the bit width of the second element and the number of leading zeros corresponding to the first element;

[0028] Shifting the shift object element to the left according to the shift bit number to obtain the valid bit of the first shift result corresponding to the group multiplication element;

[0029] The sign bit of the first shift result and the valid bit of the first shift result constitute the first shift result corresponding to the group of multiplication elements.

[0030] In one embodiment, the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix, and the fourth matrix is ​​subjected to multiplication approximation shift processing based on the second leading zero count result to obtain a prediction result of the attention matrix, including:

[0031] For each group of multiplication elements in the transpose multiplication process of the query matrix and the key matrix, based on the number of leading zeros corresponding to the first element in the group of multiplication elements, a second element in the group of multiplication elements is shifted to obtain a second shift result corresponding to the group of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix;

[0032] Determine the original code corresponding to each attention element in the prediction result of the attention matrix based on the second shift result corresponding to each group of multiplied elements;

[0033] The original code corresponding to each of the attention elements is decoded to obtain the prediction result of the attention matrix.

[0034] In a second aspect, the present application also provides a low-complexity Transformer attention module prediction device, comprising:

[0035] An acquisition module, used for acquiring a first leading zero count result obtained by performing leading zero count processing on a first matrix of a neural network model; the attention weight matrix of the neural network model is the first matrix, and the word unit matrix input to the neural network model is the second matrix; or the word unit matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix;

[0036] A processing module, configured to perform multiplication approximate shift processing on the second matrix based on the first leading zero count result to obtain a query matrix and a key matrix;

[0037] A determination module is used to determine the prediction result of the attention matrix based on the query matrix and the key matrix; the prediction result of the attention matrix is ​​used to characterize the matching degree of the context word.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the first aspect when executing the computer program.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps described in the first aspect above when executed by a processor.

[0040] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps described in the first aspect when executed by a processor.

[0041] The low-complexity Transformer attention module prediction method, device, computer equipment, computer-readable storage medium and computer program product obtain the first leading zero count result obtained by performing leading zero count processing on the first matrix of the neural network model; the attention weight matrix of the neural network model is the first matrix, and the word unit matrix input to the neural network model is the second matrix; or the word unit matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix; based on the first leading zero count result, the second matrix is ​​subjected to multiplication approximation shift processing to obtain a query matrix and a key matrix; based on the query matrix and the key matrix, the prediction result of the attention matrix is ​​determined; the prediction result of the attention matrix is ​​used to characterize the matching degree of the context word unit. In this way, a unilateral leading zero conversion mechanism is proposed, and only one of the two multipliers needs to be detected for leading zeros, and the other multiplier is subjected to multiplication approximation shift processing based on the leading zero detection result, so that multiplication approximation can be realized by shifting, and the estimated attention matrix does not need to be implemented in hardware using integer multipliers, which can effectively reduce hardware resource overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] FIG1 is a schematic diagram of a process flow of a low-complexity Transformer attention module prediction method in one embodiment;

[0044] FIG2 is a schematic flow chart of a process for performing leading zero counting processing on 0011 (+3) and 1101 (-3) respectively in one embodiment;

[0045] FIG3 is a flow chart of a step of determining a prediction result of an attention matrix based on a query matrix and a key matrix in one embodiment;

[0046] FIG4 is a flow chart of a step of obtaining a first leading zero count result obtained by performing leading zero count processing on a first matrix of a neural network model in one embodiment;

[0047] FIG5 is a schematic diagram of an implementation flow of a low-complexity Transformer attention module prediction method in one embodiment;

[0048] FIG6 is a flowchart of steps for performing multiplication approximation shift processing on the fourth matrix based on the second leading zero count result to obtain the prediction result of the attention matrix in one embodiment;

[0049] FIG7 is a flowchart of a step of determining a shift object element corresponding to a second element in the group of multiplied elements according to a first element in the group of multiplied elements in one embodiment;

[0050] FIG8 is a flowchart of a step of performing a shift process on a shift object element based on the number of leading zeros corresponding to the first element to obtain a first shift result corresponding to the group of multiplied elements in one embodiment;

[0051] FIG9 is a schematic diagram of performing multiplication approximate shift processing on a fourth matrix based on a second leading zero count result in one embodiment;

[0052] FIG10 is a flowchart of steps for performing multiplication approximation shift processing on the fourth matrix based on the second leading zero count result to obtain a prediction result of the attention matrix in another embodiment;

[0053] FIG11 is a schematic diagram of an implementation flow of a low-complexity Transformer attention module prediction method in another embodiment;

[0054] FIG12 is a schematic diagram of performing multiplication approximate shift processing on a fourth matrix based on a second leading zero count result in another embodiment;

[0055] FIG13 is a block diagram of a low-complexity Transformer attention module prediction device in one embodiment;

[0056] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] In one embodiment, as shown in FIG1 , a low-complexity Transformer attention module prediction method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. In this embodiment, the method includes the following steps:

[0059] Step 101, obtaining a first leading zero count result obtained by performing leading zero count processing on a first matrix of a neural network model.

[0060] Among them, the attention weight matrix of the neural network model is the first matrix, and the word unit matrix input to the neural network model is the second matrix. Or the word unit matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix.

[0061] In an embodiment of the present application, the neural network model is a neural network model based on a transformer architecture, including an attention mechanism. The token matrix corresponding to the language sequence is input into the attention module of the neural network model to obtain an attention matrix. The attention matrix is ​​used to characterize the measure of the importance of the contextual relationship between each token and the tokens at other positions in the language sequence, that is, the degree of matching of the context tokens. Therefore, the attention matrix can also be called a context token association matrix or a context token matching degree matrix. Specifically, the token matrix X will be respectively compared with the query weight matrix W. Q and key weight matrix W K Multiply them together to get the corresponding query (Querry, Q) matrix and key (Key, K) matrix; then multiply the Q matrix by the transpose of the K matrix to get the Attention matrix. The above specific process can be expressed as:

[0062] Q=X×W Q

[0063] K=X×W K

[0064] Attention = Q × K T

[0065] The attention weight matrix includes the query weight matrix and the key weight matrix. Leading zero: The bit 0 before the first 1 from the left of a binary number. For example, for the binary number (00101110), the number of leading zeros is 2. Leading zero counter (LoC): used for leading zero counting processing. For a binary number, the number of bits 0 before the highest bit 1 is counted, that is, LoC (00101110) = 2.

[0066] In one example, when estimating the attention matrix, the terminal performs leading zero count processing on the first matrix of the neural network model to obtain a first leading zero count result.

[0067] In one example, when the first matrix is ​​an attention weight matrix, the first leading zero count result includes a query weight matrix leading zero count result and a key weight matrix leading zero count result. The terminal performs leading zero count processing on the query weight matrix and the key weight matrix respectively to obtain a query weight matrix leading zero count result and a key weight matrix leading zero count result.

[0068] In one example, when the first matrix is ​​a word-unit matrix, the terminal performs leading zero counting processing on the word-unit matrix to obtain a first leading zero counting result.

[0069] The specific process of performing leading zero counting processing on a matrix includes: the leading zero counting result of the matrix includes the number of leading zeros corresponding to each element in the matrix. For each element in the matrix, if the binary number of the element is a positive number, the terminal counts the leading zeros of the binary number to obtain the number of leading zeros corresponding to the element. If the binary number of the element is a negative number, the terminal converts the binary number into the original code. Then, the terminal ignores the highest sign bit 1 of the original code, counts the leading zeros of the original code, and obtains the number of leading zeros corresponding to the element. For example, 0011 (+3), the number of leading zeros is 2; 1101 (-3), first convert it into the original code 1011 (-3), ignore its highest sign bit, and the number of leading zeros is 2. The flow chart of the leading zero counting processing for 0011 (+3) and 1101 (-3) is shown in Figure 2.

[0070] In one example, the terminal uses a priority encoder or a logic gate to perform leading zero counting processing on the matrix.

[0071] Step 102: Based on the first leading zero count result, perform multiplication approximate shift processing on the second matrix to obtain a query matrix and a key matrix.

[0072] In an embodiment of the present application, the second matrix may be a one-hot sequence matrix or a non-one-hot sequence matrix. The multiplication approximate shift processing is used to obtain the approximate result of binary multiplication by shifting the binary numbers, and may include or may not include one-hot sequence conversion processing. One-hot sequence: a string of 01 binary vectors, in which only one bit is 1, such as 00010000. One-hot sequence converter (OSC): For a binary number, convert it to a one-hot sequence, but retain the highest bit 1, for example, for 00101110->00100000.

[0073] In one example, when the first matrix is ​​an attention weight matrix, the first leading zero count result includes the leading zero count result of the query weight matrix and the leading zero count result of the key weight matrix. The terminal performs multiplication approximate shift processing on the word unit matrix based on the leading zero count result of the query weight matrix to obtain the query matrix. The terminal performs multiplication approximate shift processing on the word unit matrix based on the leading zero count result of the key weight matrix to obtain the key matrix.

[0074] In one example, when the first matrix is ​​a word-unit matrix, the terminal performs a multiplication approximate shift process on the query weight matrix based on the first leading zero count result to obtain a query matrix. The terminal performs a multiplication approximate shift process on the key weight matrix based on the first leading zero count result to obtain a key matrix.

[0075] The principle of multiplication approximation by shift processing is as follows: for an integer (INT) type binary number x, its mathematical representation is x=sign×2 W-LO-1 ×M, where sign represents the sign bit, W represents the quantization bit width, LO represents the number of leading zeros, and M represents the mantissa, whose value is in the interval [1,2]. Taking the value 3 under 4-bit quantization as an example, 3=(+1)×2 4-2-1 ×1.5, its binary representation is 3=(0011) b , the number of leading zeros LO = 2. Therefore, the multiplication of two binary numbers can be expressed as the following mathematical expression:

[0076] x×y=XOR(S x ,S y )×2 (Wx+Wy-(LOx+LOy)-2) ×(M x ×M y )

[0077] Among them, S x and S yRepresents the sign bits of x and y respectively. When two numbers are multiplied, the sign bit of the product is equivalent to the exclusive OR of the sign bits of the two multipliers. x and W y Represents the quantization bit width of x and y respectively, LOx and LOy represent the number of leading zeros of x and y, and M x and M y Represent the mantissa of x and y respectively. Since the mantissa is between [1,2] and is usually small, the leading zero shift of x and y can be used to approximate the multiplication of the two numbers, that is,

[0078] x×y≈XOR(S x ,S y )×2 (Wx+Wy-(LOx+LOy)-2) ×M x

[0079] For example, the calculation of 00011000(24)×00000110(6)=10010000(144) can be approximated by 00010000(16)×00000110(6)=01100000(96). It is not difficult to see that the result 01100000 is equivalent to shifting the bit 1 in 00000110 left by four bits, and the number of bits shifted left is equivalent to the bit width 8 minus the number of leading zeros in 00010000 minus 1.

[0080] Step 103, based on the query matrix and the key matrix, determine the prediction result of the attention matrix.

[0081] Among them, the prediction result of the attention matrix is ​​used to characterize the matching degree of the context word.

[0082] In an embodiment of the present application, the prediction result of the attention matrix is ​​an approximate result of the attention matrix, which can be an approximate result of multiplying the transpose of the query matrix and the key matrix.

[0083] In one example, the terminal performs multiplication approximation processing on the transpose of the query matrix and the key matrix to obtain the prediction result of the attention matrix. The multiplication approximation processing is used to obtain the approximate result of binary number multiplication, and may include or exclude the one-hot sequence conversion processing. The multiplication approximation processing includes multiplication approximation shift processing.

[0084] In one example, the terminal performs leading zero count processing on the third matrix of the neural network model to obtain a second leading zero count result corresponding to the third matrix. Then, the terminal performs a one-hot sequence conversion on the fourth matrix to obtain a converted fourth matrix. Then, based on the second leading zero count result, the terminal performs a shift processing on the converted fourth matrix to obtain a prediction result of the attention matrix. Among them, the query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix. Or the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix.

[0085] In the above low-complexity Transformer attention module prediction method, the first leading zero count result obtained by performing leading zero count processing on the first matrix of the neural network model is obtained; the attention weight matrix of the neural network model is the first matrix, and the word unit matrix input to the neural network model is the second matrix; or the word unit matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix; based on the first leading zero count result, the second matrix is ​​subjected to multiplication approximation shift processing to obtain a query matrix and a key matrix; based on the query matrix and the key matrix, the prediction result of the attention matrix is ​​determined; the prediction result of the attention matrix is ​​used to characterize the matching degree of the context word unit. In this way, a unilateral leading zero conversion mechanism is proposed, and only one of the two multipliers needs to be detected by leading zero, and the other multiplier is subjected to multiplication approximation shift processing based on the leading zero detection result, so that multiplication approximation can be realized by shifting, and the estimated attention matrix does not need to use integer multipliers for hardware implementation, which can effectively reduce hardware resource overhead, reduce chip area, and reduce chip power consumption and chip energy overhead during operation. Moreover, the object of multiplication approximate shift processing in the single-sided leading zero conversion mechanism proposed by the method can be the original data that has not undergone one-hot sequence conversion. There is no need for one-hot sequence conversion and the hardware overhead of the one-hot sequence converter, which can further reduce hardware resource overhead and hardware area, further reduce chip area, and further reduce chip power consumption and chip energy overhead during operation.

[0086] In an exemplary embodiment, as shown in FIG3 , the specific process of determining the prediction result of the attention matrix based on the query matrix and the key matrix includes the following steps:

[0087] Step 301, performing leading zero counting processing on the third matrix of the neural network model to obtain a second leading zero counting result corresponding to the third matrix.

[0088] The query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix. Alternatively, the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix.

[0089] In an embodiment of the present application, the second leading zero count result is a leading zero count result obtained by performing leading zero count processing on the third matrix of the neural network model.

[0090] Step 302: Based on the second leading zero count result, perform multiplication approximate shift processing on the fourth matrix to obtain the prediction result of the attention matrix.

[0091] In the above low-complexity Transformer attention module prediction method, the third matrix of the neural network model is processed with leading zero counts to obtain the second leading zero count result corresponding to the third matrix; based on the second leading zero count result, the fourth matrix is ​​processed with multiplication approximation shift to obtain the prediction result of the attention matrix. In this way, the transpose multiplication of the query matrix and the key matrix also adopts the newly proposed unilateral leading zero conversion mechanism. It only needs to perform leading zero detection on one of the two multipliers, and perform multiplication approximation shift processing on the other multiplier based on the leading zero detection result. Multiplication approximation can be achieved through shifting, ensuring that the estimated attention matrix does not need to use integer multipliers for hardware implementation, which can effectively reduce hardware resource overhead.

[0092] In an exemplary embodiment, as shown in FIG4 , the first matrix is ​​an attention weight matrix, and the specific process of obtaining the first leading zero count result obtained by performing leading zero count processing on the first matrix of the neural network model includes the following steps:

[0093] Step 401, perform leading zero count processing on the attention weight matrix of the neural network model to obtain the first leading zero count result corresponding to the attention weight matrix.

[0094] In an embodiment of the present application, during the network deployment stage of the neural network model, the attention weight matrix of the neural network model is processed with leading zero counts to obtain a first leading zero count result corresponding to the attention weight matrix.

[0095] Step 402, storing the first leading zero counting result into a target storage unit.

[0096] In an embodiment of the present application, the terminal determines the second digit of the element in the first leading zero count result according to the first digit of the element in the attention weight matrix. , where B is the second digit and A is the first digit. Then, for each decimal number element in the first leading zero count result, the terminal converts the decimal number element into a binary number with a quantization bit width of the second digit, and obtains the converted first leading zero count result. Then, the terminal stores the converted first leading zero count result in a target storage unit. The target storage unit may be a dynamic random access memory (DRAM).

[0097] Step 403, when estimating the attention matrix, read the first leading zero count result from the target storage unit.

[0098] In the above-mentioned low-complexity Transformer attention module prediction method, the attention weight matrix of the neural network model is processed with leading zero counts to obtain the first leading zero count result corresponding to the attention weight matrix; the first leading zero count result is stored in the target storage unit; when the attention matrix is ​​estimated, the first leading zero count result is read from the target storage unit. In this way, since in neural network reasoning, once the network training is completed, the weight of the network can be obtained, and the known weight characteristics of the network model when estimating the attention matrix are utilized, the present invention performs leading zero detection on the attention weight matrix in advance and stores it in the storage unit, which can reduce the storage bit width of the data and the bandwidth of the data transmission. For example, for an 8-bit binary number 00011001, the number of leading zeros is 3, which is converted to a binary number expression of 0011. If 00011001 is directly stored, 8 bits need to be stored, and if the number of leading zeros in the memory is stored, only 4 bits need to be stored, which saves storage capacity and also saves bandwidth resources required when reading data from the storage unit to the chip. At the same time, this method eliminates the need to detect leading zeros each time the attention matrix is ​​estimated, thereby reducing the converter overhead and further reducing hardware resource overhead.

[0099] In one embodiment, a schematic diagram of the implementation process of the low-complexity Transformer attention module prediction method is shown in FIG5. In the network deployment stage, the terminal performs two weight matrices W Q and W K Preprocessing is performed, each element is converted to a leading zero count, and the converted result is stored in the target storage unit DRAM. When estimating the attention matrix, that is, in the formal calculation, the terminal directly converts the weight matrix W Q and W K The number of leading zeros is directly read from DRAM, and the input Token matrix X is subjected to multiplication approximation shift processing; after obtaining matrices Q and K, the leading zeros of the Q matrix are counted, and based on the counting result, the K matrix is ​​subjected to multiplication approximation shift processing to obtain the predicted Attention matrix. It can be seen that since this method cleverly utilizes the known weight characteristics during neural network inference and reduces the conversion requirements of the one-hot sequence, the overall calculation steps are only four steps, and the processing steps of the estimated attention matrix are very few, which can save unnecessary one-hot sequence conversion and decoding module overhead, thereby effectively reducing the hardware area overhead.

[0100] In an exemplary embodiment, as shown in FIG6 , the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix. Based on the second leading zero count result, the fourth matrix is ​​subjected to multiplication approximation shift processing, and the specific process of obtaining the prediction result of the attention matrix includes the following steps:

[0101] Step 601: for each group of multiplied elements in the transpose multiplication process of the query matrix and the key matrix, determine the shift object element corresponding to the second element in the group of multiplied elements according to the first element in the group of multiplied elements.

[0102] The first element is an element in the third matrix, and the second element is an element in the fourth matrix.

[0103] In an embodiment of the present application, a group of multiplied elements includes a first element and a second element. When the query matrix is ​​the third matrix and the transpose of the key matrix is ​​the fourth matrix, the number of columns of the first element in a group of multiplied elements is the same as the number of rows of the second element in the group of multiplied elements. When the transpose of the key matrix is ​​the third matrix and the query matrix is ​​the fourth matrix, the number of rows of the first element in a group of multiplied elements is the same as the number of columns of the second element in the group of multiplied elements. The shift object element is an object to be shifted.

[0104] Step 602: Shift the shift object element based on the number of leading zeros corresponding to the first element to obtain a first shift result corresponding to the group of multiplied elements.

[0105] In the embodiment of the present application, the first shift result is the result of shifting the shift object element based on the number of leading zeros corresponding to the first element.

[0106] Step 603, performing sign bit extension on the first shift result to obtain the product corresponding to the group of multiplied elements.

[0107] Step 604, determining the prediction result of the attention matrix based on the products corresponding to each group of multiplied elements.

[0108] In an embodiment of the present application, for each attention element in the prediction result of the attention matrix, the terminal determines the target group multiplication element corresponding to the attention element based on the number of rows and columns of the attention element and the number of rows and columns corresponding to the first element and the second element in each group of multiplication elements. Then, the terminal performs a first accumulation process on the products corresponding to the multiplication elements of each target group to obtain the attention element. Specifically, the terminal performs an accumulation process on the products corresponding to the multiplication elements of each target group according to the binary complement calculation rule to obtain the attention element. Then, the terminal uses each attention element to form the prediction result of the attention matrix. Among them, the first accumulation process is an accumulation process according to the binary complement calculation rule.

[0109] In one example, when the query matrix is ​​the third matrix and the transpose of the key matrix is ​​the fourth matrix, the terminal determines the group multiplication element whose number of rows of the first element is the same as the number of rows of the attention element and whose number of columns of the second element is the same as the number of columns of the attention element as the target group multiplication element corresponding to the attention element. When the transpose of the key matrix is ​​the third matrix and the query matrix is ​​the fourth matrix, the terminal determines the group multiplication element whose number of columns of the first element is the same as the number of columns of the attention element and whose number of rows of the second element is the same as the number of rows of the attention element as the target group multiplication element corresponding to the attention element.

[0110] In the above low-complexity Transformer attention module prediction method, for each group of multiplication elements in the transpose multiplication process of the query matrix and the key matrix, the shift object element corresponding to the second element in the group of multiplication elements is determined according to the first element in the group of multiplication elements; based on the number of leading zeros corresponding to the first element, the shift object element is shifted to obtain the first shift result corresponding to the group of multiplication elements; the first shift result is sign-bit extended to obtain the product corresponding to the group of multiplication elements; based on the product corresponding to each group of multiplication elements, the prediction result of the attention matrix is ​​determined. In this way, based on the number of leading zeros of one multiplier, the shift object element of another multiplier is directly shifted and sign-bit extended, so that multiplication approximation can be achieved without the need for one-hot sequence conversion and decoding, which can further reduce hardware resource overhead and hardware area.

[0111] In an exemplary embodiment, as shown in FIG7 , the specific process of determining the shift object element corresponding to the second element in the group of multiplied elements according to the first element in the group of multiplied elements includes the following steps:

[0112] Step 701: If the first element in the group of multiplied elements is a negative number, the original code of the second element in the group of multiplied elements is bitwise inverted to obtain the shift object element corresponding to the second element.

[0113] In an embodiment of the present application, if the first element in the group of multiplied elements is a negative number, the terminal performs bitwise inversion of the original code of the second element in the group of multiplied elements including the sign bit to obtain the shift object element corresponding to the second element.

[0114] Step 702: If the first element in the group of multiplied elements is a positive number, the original code of the second element in the group of multiplied elements is used as the shift object element corresponding to the second element.

[0115] In the above low-complexity Transformer attention module prediction method, if the first element in the group of multiplied elements is a negative number, the original code of the second element in the group of multiplied elements is bitwise inverted to obtain the shift object element corresponding to the second element; if the first element in the group of multiplied elements is a positive number, the original code of the second element in the group of multiplied elements is used as the shift object element corresponding to the second element. In this way, when performing multiplication approximation, there is no need for one-hot sequence conversion, and only the mantissa of one of the two multipliers is ignored. The error is very small, which can ensure the accuracy of attention matrix estimation.

[0116] In an exemplary embodiment, as shown in FIG8 , the specific process of performing shift processing on the shift object element based on the number of leading zeros corresponding to the first element to obtain the first shift result corresponding to the group of multiplied elements includes the following steps:

[0117] Step 801, determining the sign bit of the first shift result corresponding to the group of multiplied elements according to the sign bit of the first element and the sign bit of the second element.

[0118] In the embodiment of the present application, if the sign bit of the first element is the same as the sign bit of the second element, the terminal determines 0 as the sign bit of the first shift result corresponding to the group of multiplied elements. If the sign bit of the first element is different from the sign bit of the second element, the terminal determines 1 as the sign bit of the first shift result corresponding to the group of multiplied elements.

[0119] Step 802: Determine the number of shift bits corresponding to the shift object element according to the bit width of the second element and the number of leading zeros corresponding to the first element.

[0120] In an embodiment of the present application, the terminal subtracts the number of leading zeros and 1 corresponding to the first element from the bit width of the second element to obtain the number of shift bits corresponding to the shift object element, which can be expressed as: the number of shift bits corresponding to the shift object element = the bit width of the second element - the number of leading zeros corresponding to the first element - 1.

[0121] Step 803: Left-shift the shift object element according to the number of shift bits to obtain the valid bits of the first shift result corresponding to the group of multiplied elements.

[0122] In the embodiment of the present application, the terminal left-shifts the effective bits of the shift object elements by the shift bit number to obtain the effective bits of the first shift result corresponding to the group of multiplied elements.

[0123] Step 804: The sign bit of the first shift result and the valid bit of the first shift result are used to form the first shift result corresponding to the group of multiplied elements.

[0124] In the above low-complexity Transformer attention module prediction method, the sign bit of the first shift result corresponding to the group of multiplied elements is determined according to the sign bit of the first element and the sign bit of the second element; the number of shifts corresponding to the shift object element is determined according to the bit width of the second element and the number of leading zeros corresponding to the first element; the shift object element is left-shifted according to the number of shifts to obtain the valid bit of the first shift result corresponding to the group of multiplied elements; the sign bit of the first shift result and the valid bit of the first shift result constitute the first shift result corresponding to the group of multiplied elements. In this way, by left-shifting the bit 1 of the shift object element corresponding to one of the multipliers, and the number of left shifts is the bit width minus the number of leading zeros of the other multiplier minus 1, to achieve multiplication approximation, only ignoring the mantissa of one of the two multipliers, the error is very small, and the accuracy of attention matrix estimation can be guaranteed.

[0125] In one embodiment, a schematic diagram of performing multiplication approximate shift processing on the fourth matrix based on the second leading zero count result is shown in FIG9 . The transpose multiplication of the query matrix and the key matrix is ​​[x0, x1]×[y0, y1] T=z=[x0×y0+x1×y1], x0=+3, x1=+4, y0=-3, y1=-3, and the transpose of the key matrix is ​​the third matrix, the query matrix is ​​the fourth matrix, the first set of multiplied elements are x0 (second element) and y0 (first element), and the second set of multiplied elements are x1 (second element) and y1 (first element). For the first group of multiplication elements: +3 (x0) is represented by 4-bit binary as 0011. Since -3 (y0) is a negative number, the terminal inverts the original code of x0 bit by bit to obtain the shift object element 1100 corresponding to x0. Since x0 is a positive number and y0 is a negative number, the sign bits of the two are different. The terminal determines 1 as the sign bit of the first shift result corresponding to the group of multiplication elements. The number of leading zeros L0 of -3 (y0) is 2, so the number of shift bits is 4-2-1=1. The terminal shifts 1100 left by 1 bit to obtain the valid bits of the first shift result corresponding to the group of multiplication elements, and the sign bit of the first shift result and the valid bits of the first shift result constitute the first shift result 10001000 corresponding to the group of multiplication elements. The sign bit of 10001000 is sign-extended to obtain the product 11111000 corresponding to the group of multiplication elements. For the second group of multiplied elements: +4 (x1) is represented by 4 bits in binary as 0100. Since -3 (y1) is a negative number, the terminal inverts the original code of x1 bit by bit to obtain the shift object element 1011 corresponding to x1. Since x1 is a positive number and y1 is a negative number, the sign bits of the two are different. The terminal determines 1 as the sign bit of the first shift result corresponding to the group of multiplied elements. The number of leading zeros L0 of -3 (y1) is 2, so the number of shift bits is 4-2-1=1. The terminal shifts 1011 left by 1 bit to obtain the valid bits of the first shift result corresponding to the group of multiplied elements, and the sign bit of the first shift result and the valid bits of the first shift result constitute the first shift result 10000110 corresponding to the group of multiplied elements. The sign bit of 10000110 is sign-extended to obtain the product 11110110 corresponding to the group of multiplied elements. Then, the terminal adds the two products according to the binary complement calculation rule to obtain the attention element 11101110. Then, the terminal uses the attention element to form the prediction result [11101110] of the attention matrix.

[0126] In an exemplary embodiment, as shown in FIG10 , the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix. Based on the second leading zero count result, the fourth matrix is ​​subjected to multiplication approximation shift processing, and the specific process of obtaining the prediction result of the attention matrix includes the following steps:

[0127] Step 1001, for each group of multiplication elements in the transpose multiplication process of the query matrix and the key matrix, based on the number of leading zeros corresponding to the first element in the group of multiplication elements, the second element in the group of multiplication elements is shifted to obtain a second shift result corresponding to the group of multiplication elements.

[0128] The first element is an element in the third matrix, and the second element is an element in the fourth matrix.

[0129] In an embodiment of the present application, the terminal determines the sign bit of the second shift result corresponding to the group of multiplied elements according to the sign bit of the first element and the sign bit of the second element. Then, the terminal determines the number of shifts corresponding to the second element according to the bit width of the second element and the number of leading zeros corresponding to the first element. Then, the terminal shifts the second element to the left according to the number of shifts to obtain the valid bit of the second shift result corresponding to the group of multiplied elements. Then, the terminal uses the sign bit of the second shift result and the valid bit of the second shift result to form the second shift result corresponding to the group of multiplied elements. Among them, the second shift result is the result of shifting the second element based on the number of leading zeros corresponding to the first element. It can be understood that the specific process of the terminal performing shift processing on the second element in the group of multiplied elements based on the number of leading zeros corresponding to the first element in the group of multiplied elements to obtain the second shift result corresponding to the group of multiplied elements is similar to the specific process of the terminal performing shift processing on the shift object element based on the number of leading zeros corresponding to the first element to obtain the first shift result corresponding to the group of multiplied elements, and is similar to the specific process of steps 801-804.

[0130] Step 1002, based on the second shift results corresponding to each group of multiplied elements, determine the original code corresponding to each attention element in the prediction result of the attention matrix.

[0131] In the embodiment of the present application, for each attention element in the prediction result of the attention matrix, the terminal determines the target group multiplication element corresponding to the attention element according to the row and column number of the attention element and the row and column number corresponding to the first element and the second element in each group of multiplication elements. Then, the terminal performs a second accumulation process on the second shift results corresponding to each target group multiplication element to obtain the original code corresponding to the attention element.

[0132] Specifically, for each accumulation object that undergoes the second accumulation processing, if the accumulation object is a negative number, the terminal updates the original values ​​corresponding to each digit of the accumulation object to the negative of the original values ​​corresponding to each digit. For each accumulation object that undergoes the second accumulation processing, if the accumulation object is a positive number, the terminal maintains the original values ​​corresponding to each digit of the accumulation object. Then, the terminal performs bitwise accumulation on each accumulation object to obtain the result of the second accumulation processing.

[0133] Step 1003, decode the original code corresponding to each attention element to obtain the prediction result of the attention matrix.

[0134] In an embodiment of the present application, the terminal converts the original code corresponding to each attention element into a binary complement code to obtain a prediction result of the attention matrix.

[0135] Specifically, for each attention element, the terminal performs weighted summation based on the values ​​of each bit of the original code corresponding to the attention element and the weights corresponding to each bit to obtain the decimal number corresponding to the attention element. Then, the terminal converts the decimal number into binary to obtain the binary number corresponding to the attention element. Then, the terminal converts the binary number into a binary complement to obtain the prediction result of the attention matrix.

[0136] In the above low-complexity Transformer attention module prediction method, for each group of multiplication elements in the transpose multiplication process of the query matrix and the key matrix, based on the number of leading zeros corresponding to the first element in the group of multiplication elements, the second element in the group of multiplication elements is shifted to obtain the second shift result corresponding to the group of multiplication elements; based on the second shift result corresponding to each group of multiplication elements, the original code corresponding to each attention element in the prediction result of the attention matrix is ​​determined; the original code corresponding to each attention element is decoded to obtain the prediction result of the attention matrix. In this way, when two multipliers are multiplied, the bit 1 of one of the multipliers is directly shifted left, and the number of left shifts is the bit width minus the number of leading zeros of the other multiplier minus 1, and then the shifted cumulative result is decoded. Compared with the above low-complexity Transformer attention module prediction method without decoding, this method does not have the error 1 caused by only negating without adding 1, which can further improve the accuracy of attention matrix estimation.

[0137] In another embodiment, a schematic diagram of the implementation process of the low-complexity Transformer attention module prediction method is shown in FIG11. In the network deployment stage, the terminal performs two weight matrices W Q and W K Preprocessing is performed, each element is converted to a leading zero count, and the converted result is stored in the target storage unit DRAM. When estimating the attention matrix, that is, in the formal calculation, the terminal directly converts the weight matrix W Q and W K The number of leading zeros is read directly from DRAM, and the input Token matrix X is shifted and the second accumulation process is performed, that is, shift-accumulation process, and then the result is decoded; after obtaining the matrices Q and K, the leading zeros of the Q matrix are counted, and based on the counting result, the K matrix is ​​shifted and the second accumulation process is performed, that is, shift-accumulation process, and then the result is decoded and converted into binary complement to obtain the predicted Attention matrix.

[0138] In another embodiment, a schematic diagram of performing multiplication approximation shift processing on the fourth matrix based on the second leading zero count result is shown in FIG12. The transpose multiplication of the query matrix and the key matrix is ​​[x0, x1] × [y0, y1] T =z=[x0×y0+x1×y1], x0=+3, x1=+4, y0=-3, y1=-3, and the transpose of the key matrix is ​​the third matrix, the query matrix is ​​the fourth matrix, the first set of multiplied elements are x0 (second element) and y0 (first element), and the second set of multiplied elements are x1 (second element) and y1 (first element). For the first group of multiplied elements: the 4-bit binary representation of +3 (x0) is 0011. Since x0 is a positive number and y0 is a negative number, the sign bits of the two are different. The terminal determines 1 as the sign bit of the second shift result corresponding to this group of multiplied elements; the number of leading zeros L0 of -3 (y0) is 2, and the number of shift bits is 4-2-1=1; the terminal shifts 0011 left by 1 bit to obtain the valid bits of the second shift result corresponding to the group of multiplied elements, 0110, and the sign bit of the second shift result and the valid bits of the second shift result constitute the second shift result 10000110 corresponding to the group of multiplied elements. For the second group of multiplication elements: +4 (x1) is represented by 4-bit binary as 0100. Since x1 is a positive number and y1 is a negative number, the sign bits of the two are different. The terminal determines 1 as the sign bit of the second shift result corresponding to the group of multiplication elements; the number of leading zeros L0 of -3 (y1) is 2, so the number of shift bits is 4-2-1=1; the terminal shifts 0100 left by 1 bit to obtain the valid bit 1000 of the second shift result corresponding to the group of multiplication elements, and the sign bit of the second shift result and the valid bit of the second shift result constitute the second shift result 10001000 corresponding to the group of multiplication elements. Then, the terminal performs a second accumulation process on the two second shift results to obtain the original code corresponding to the attention element [0 0 0 -1 -1 -1 0]. Then, the terminal decodes the original code corresponding to the attention element to obtain the predicted result of the attention matrix 11110010. Then, the terminal uses the attention element to constitute the predicted result of the attention matrix [11110010].

[0139] In one embodiment, the specific process of step 102 may include: the first leading zero count result includes the number of leading zeros corresponding to each third element in the first matrix. For each group of multiplied elements in the multiplication process of the word unit matrix and the query weight matrix and the multiplication process of the word unit matrix and the key weight matrix, the terminal determines the shift object element corresponding to the fourth element in the group of multiplied elements according to the third element in the group of multiplied elements. Among them, the third element is an element in the first matrix, and the fourth element is an element in the second matrix. Then, the terminal performs a shift process on the shift object element based on the number of leading zeros corresponding to the third element to obtain the third shift result corresponding to the group of multiplied elements. Then, the terminal performs a sign bit extension on the third shift result to obtain the product corresponding to the group of multiplied elements. Then, the terminal determines the query matrix and the key matrix based on the product corresponding to each group of multiplied elements. It can be understood that the specific process of the above steps is similar to the specific process of steps 601-604.

[0140] The specific process of the terminal determining the shift object element corresponding to the fourth element in the group of multiplied elements according to the third element in the group of multiplied elements may include the following steps: if the third element in the group of multiplied elements is a negative number, the original code of the fourth element in the group of multiplied elements is bitwise inverted to obtain the shift object element corresponding to the fourth element. If the third element in the group of multiplied elements is a positive number, the original code of the fourth element in the group of multiplied elements is used as the shift object element corresponding to the second element. It can be understood that the specific process of the above steps is similar to the specific process of steps 701-702.

[0141] Among them, the terminal performs shift processing on the shift object element based on the number of leading zeros corresponding to the third element, and the specific process of obtaining the third shift result corresponding to the group of multiplied elements includes the following steps: the terminal determines the sign bit of the third shift result corresponding to the group of multiplied elements according to the sign bit of the third element and the sign bit of the fourth element. Then, the terminal determines the number of shift bits corresponding to the shift object element according to the bit width of the fourth element and the number of leading zeros corresponding to the third element. Then, the terminal performs left shifting on the shift object element according to the number of shift bits to obtain the valid bits of the third shift result corresponding to the group of multiplied elements. Then, the terminal uses the sign bit of the third shift result and the valid bits of the third shift result to form the third shift result corresponding to the group of multiplied elements. It can be understood that the specific process of the above steps is similar to the specific process of steps 801-804.

[0142] Among them, the specific process of the terminal determining the query matrix and the key matrix based on the product corresponding to each group of multiplication elements includes the following steps: for each query element in the query matrix, the terminal determines the target group multiplication element corresponding to the query element in each group of multiplication elements in the process of multiplying the word unit matrix and the query weight matrix according to the number of rows and columns of the query element and the number of rows and columns corresponding to the third element and the fourth element in each group of multiplication elements. Then, the terminal performs a first accumulation process on the product corresponding to each target group multiplication element corresponding to the query element to obtain the query element. Then, the terminal forms a query matrix with each query element. For each key element in the key matrix, the terminal determines the target group multiplication element corresponding to the key element in each group of multiplication elements in the process of multiplying the word unit matrix and the key weight matrix according to the number of rows and columns of the key element and the number of rows and columns corresponding to the third element and the fourth element in each group of multiplication elements. Then, the terminal performs a first accumulation process on the product corresponding to each target group multiplication element corresponding to the key element to obtain the key element. Then, the terminal forms a key matrix with each key element. It can be understood that the specific process of the terminal determining the query matrix based on the products corresponding to each group of multiplied elements and the specific process of the terminal determining the key matrix based on the products corresponding to each group of multiplied elements are similar to the specific process of step 604.

[0143] In another embodiment, the specific process of step 102 may include: the first leading zero count result includes the number of leading zeros corresponding to each third element in the first matrix. For each group of multiplied elements in the multiplication process between the word unit matrix and the query weight matrix and the multiplication process between the word unit matrix and the key weight matrix, based on the number of leading zeros corresponding to the third element in the group of multiplied elements, the fourth element in the group of multiplied elements is shifted to obtain the fourth shift result corresponding to the group of multiplied elements. Among them, the third element is an element in the first matrix, and the fourth element is an element in the second matrix. Then, the terminal determines the original code corresponding to each query element in the query matrix and the original code corresponding to each key element in the key matrix based on the fourth shift result corresponding to each group of multiplied elements. Specifically, for each query element in the query matrix, the terminal determines the target group of multiplied elements corresponding to the query element in each group of multiplied elements in the multiplication process between the word unit matrix and the query weight matrix according to the number of rows and columns of the query element and the number of rows and columns corresponding to the first element and the second element in each group of multiplied elements. Then, the terminal performs a second accumulation process on the fourth shift result corresponding to the target group multiplication element corresponding to each query element to obtain the original code corresponding to the query element. For each key element in the key matrix, the terminal determines the target group multiplication element corresponding to the key element in each group of multiplication elements in the multiplication process of the word element matrix and the key weight matrix according to the number of rows and columns of the key element and the number of rows and columns corresponding to the first element and the second element in each group of multiplication elements. Then, the terminal performs a second accumulation process on the fourth shift result corresponding to the target group multiplication element corresponding to each key element to obtain the original code corresponding to the key element. Then, the terminal performs a decoding process on the original code corresponding to each query element to obtain a query matrix. The terminal performs a decoding process on the original code corresponding to each key element to obtain a key matrix. It can be understood that the specific process of the above steps is similar to the specific process of steps 1001-1003.

[0144] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0145] Based on the same inventive concept, the embodiment of the present application also provides a low-complexity Transformer attention module prediction device for implementing the low-complexity Transformer attention module prediction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more low-complexity Transformer attention module prediction device embodiments provided below can refer to the limitations of the low-complexity Transformer attention module prediction method above, and will not be repeated here.

[0146] In an exemplary embodiment, as shown in FIG13 , a low-complexity Transformer attention module prediction device 1300 is provided, comprising: an acquisition module 1310, a processing module 1320 and a determination module 1330, wherein:

[0147] An acquisition module 1310 is used to obtain a first leading zero count result obtained by performing leading zero count processing on a first matrix of a neural network model; the attention weight matrix of the neural network model is the first matrix, and the word unit matrix input to the neural network model is the second matrix; or the word unit matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix;

[0148] A processing module 1320 is used to perform multiplication approximate shift processing on the second matrix based on the first leading zero count result to obtain a query matrix and a key matrix;

[0149] The determination module 1330 is used to determine the prediction result of the attention matrix based on the query matrix and the key matrix; the prediction result of the attention matrix is ​​used to characterize the matching degree of the context word.

[0150] Optionally, the determining module 1330 is specifically configured to:

[0151] Performing leading zero counting processing on the third matrix of the neural network model to obtain a second leading zero counting result corresponding to the third matrix; the query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix; or the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix;

[0152] Based on the second leading zero count result, the fourth matrix is ​​subjected to multiplication approximate shift processing to obtain a prediction result of the attention matrix.

[0153] Optionally, the first matrix is ​​the attention weight matrix, and the acquisition module 1310 is specifically used for:

[0154] Performing leading zero counting processing on the attention weight matrix of the neural network model to obtain a first leading zero counting result corresponding to the attention weight matrix;

[0155] storing the first leading zero count result into a target storage unit;

[0156] When estimating the attention matrix, the first leading zero count result is read from the target storage unit.

[0157] Optionally, the second leading zero counting result includes the number of leading zeros corresponding to each first element in the third matrix, and the determining module 1330 is specifically configured to:

[0158] For each group of multiplication elements in the transpose multiplication process of the query matrix and the key matrix, determine the shift object element corresponding to the second element in the group of multiplication elements according to the first element in the group of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix;

[0159] Based on the number of leading zeros corresponding to the first element, the shift object element is shifted to obtain a first shift result corresponding to the group multiplication element;

[0160] Performing sign bit extension on the first shift result to obtain a product corresponding to the group multiplication element;

[0161] Based on the products corresponding to the groups of multiplied elements, the prediction results of the attention matrix are determined.

[0162] Optionally, the determining module 1330 is specifically configured to:

[0163] If the first element in the group of multiplied elements is a negative number, the original code of the second element in the group of multiplied elements is bitwise inverted to obtain a shift object element corresponding to the second element;

[0164] If the first element in the group of multiplied elements is a positive number, the original code of the second element in the group of multiplied elements is used as the shift object element corresponding to the second element.

[0165] Optionally, the determining module 1330 is specifically configured to:

[0166] Determine, according to the sign bit of the first element and the sign bit of the second element, the sign bit of the first shift result corresponding to the group of multiplied elements;

[0167] Determine the number of shift bits corresponding to the shift object element according to the bit width of the second element and the number of leading zeros corresponding to the first element;

[0168] Shifting the shift object element to the left according to the shift bit number to obtain the valid bit of the first shift result corresponding to the group multiplication element;

[0169] The sign bit of the first shift result and the valid bit of the first shift result constitute the first shift result corresponding to the group of multiplication elements.

[0170] Optionally, the second leading zero counting result includes the number of leading zeros corresponding to each first element in the third matrix, and the determining module 1330 is specifically configured to:

[0171] For each group of multiplication elements in the transpose multiplication process of the query matrix and the key matrix, based on the number of leading zeros corresponding to the first element in the group of multiplication elements, a second element in the group of multiplication elements is shifted to obtain a second shift result corresponding to the group of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix;

[0172] Determine the original code corresponding to each attention element in the prediction result of the attention matrix based on the second shift result corresponding to each group of multiplied elements;

[0173] The original code corresponding to each of the attention elements is decoded to obtain the prediction result of the attention matrix.

[0174] Each module in the low-complexity Transformer attention module prediction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0175] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.14As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a low-complexity Transformer attention module prediction method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0176] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0177] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0178] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0179] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0181] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0182] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0183] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A low-complexity Transformer attention module prediction method, characterized in that: The method comprises: Obtaining a first leading zero count result obtained by performing leading zero count processing on a first matrix of a neural network model; the attention weight matrix of the neural network model is the first matrix, and the word unit matrix input to the neural network model is the second matrix; or the word unit matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix; Based on the first leading zero count result, performing multiplication approximate shift processing on the second matrix to obtain a query matrix and a key matrix; Based on the query matrix and the key matrix, a prediction result of the attention matrix is ​​determined; the prediction result of the attention matrix is ​​used to characterize the matching degree of the context word.

2. The method according to claim 1, characterized in that The step of determining a prediction result of an attention matrix based on the query matrix and the key matrix includes: Performing leading zero counting processing on the third matrix of the neural network model to obtain a second leading zero counting result corresponding to the third matrix; the query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix; or the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix; Based on the second leading zero count result, the fourth matrix is ​​subjected to multiplication approximate shift processing to obtain a prediction result of the attention matrix.

3. The method according to claim 1, characterized in that The first matrix is ​​the attention weight matrix, and the obtaining of a first leading zero count result obtained by performing leading zero count processing on the first matrix of the neural network model includes: Performing leading zero counting processing on the attention weight matrix of the neural network model to obtain a first leading zero counting result corresponding to the attention weight matrix; storing the first leading zero count result into a target storage unit; When estimating the attention matrix, the first leading zero count result is read from the target storage unit.

4. The method according to claim 2, characterized in that: The second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix, and the fourth matrix is ​​subjected to multiplication approximation shift processing based on the second leading zero count result to obtain a prediction result of the attention matrix, including: For each group of multiplication elements in the transpose multiplication process of the query matrix and the key matrix, determine the shift object element corresponding to the second element in the group of multiplication elements according to the first element in the group of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix; Based on the number of leading zeros corresponding to the first element, the shift object element is shifted to obtain a first shift result corresponding to the group multiplication element; Performing sign bit extension on the first shift result to obtain a product corresponding to the group multiplication element; Based on the products corresponding to the groups of multiplied elements, the prediction results of the attention matrix are determined.

5. The method according to claim 4, characterized in that The step of determining, according to the first element in the group of multiplied elements, the shift object element corresponding to the second element in the group of multiplied elements comprises: If the first element in the group of multiplied elements is a negative number, the original code of the second element in the group of multiplied elements is bitwise inverted to obtain a shift object element corresponding to the second element; If the first element in the group of multiplied elements is a positive number, the original code of the second element in the group of multiplied elements is used as the shift object element corresponding to the second element.

6. The method according to claim 4, characterized in that The performing shift processing on the shift object element based on the number of leading zeros corresponding to the first element to obtain a first shift result corresponding to the group multiplication element includes: Determine, according to the sign bit of the first element and the sign bit of the second element, the sign bit of the first shift result corresponding to the group of multiplied elements; Determine the number of shift bits corresponding to the shift object element according to the bit width of the second element and the number of leading zeros corresponding to the first element; Shifting the shift object element to the left according to the shift bit number to obtain the valid bit of the first shift result corresponding to the group multiplication element; The sign bit of the first shift result and the valid bit of the first shift result constitute the first shift result corresponding to the group of multiplication elements.

7. The method according to claim 2, characterized in that The second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix, and the fourth matrix is ​​subjected to multiplication approximation shift processing based on the second leading zero count result to obtain a prediction result of the attention matrix, including: For each group of multiplication elements in the transpose multiplication process of the query matrix and the key matrix, based on the number of leading zeros corresponding to the first element in the group of multiplication elements, a second element in the group of multiplication elements is shifted to obtain a second shift result corresponding to the group of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix; Determine the original code corresponding to each attention element in the prediction result of the attention matrix based on the second shift result corresponding to each group of multiplied elements; The original code corresponding to each of the attention elements is decoded to obtain the prediction result of the attention matrix.

8. A low-complexity Transformer attention module prediction device, characterized in that: The device comprises: An acquisition module, used for acquiring a first leading zero count result obtained by performing leading zero count processing on a first matrix of a neural network model; the attention weight matrix of the neural network model is the first matrix, and the word unit matrix input to the neural network model is the second matrix; or the word unit matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix; A processing module, configured to perform multiplication approximate shift processing on the second matrix based on the first leading zero count result to obtain a query matrix and a key matrix; A determination module is used to determine the prediction result of the attention matrix based on the query matrix and the key matrix; the prediction result of the attention matrix is ​​used to characterize the matching degree of the context word.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images