Low complexity transformer attention module prediction method and apparatus

Abstract

This application relates to a low-complexity Transformer attention module prediction method, apparatus, computer device, computer-readable storage medium, and computer program product. The method includes: obtaining a first leading zero count result by performing leading zero counting on a first matrix of a neural network model; the attention weight matrix of the neural network model is the first matrix, and the word matrix input to the neural network model is the second matrix; or the word matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix; based on the first leading zero count result, performing a multiplicative approximate shift operation on the second matrix to obtain a query matrix and a key matrix; based on the query matrix and the key matrix, determining the prediction result of the attention matrix; the prediction result of the attention matrix is ​​used to characterize the degree of matching of context words. This method can reduce hardware resource overhead.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a low-complexity Transformer attention module prediction method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] Thanks to the attention mechanism, which captures contextual relationships, Transformer-based neural network models have achieved great success in natural language processing. However, as the sequence of input models grows, the computational and storage complexity of the attention mechanism also increases dramatically. This is because the computational and storage requirements of the attention module are proportional to the square of the sequence length. Therefore, how to effectively reduce the complexity of the attention module has become a focus of current academic research.

[0003] Thanks to the inherent redundancy in human language, recent research has found that the sparsity between tokens can be used to simplify the computational and storage complexity of attention modules. Specifically, a low-complexity estimate of the attention matrix can be obtained first. Based on this estimate, the values ​​in each row of the attention matrix are sorted, and the K largest values ​​are selected. These K largest values ​​are considered to have the greatest impact on attention. In subsequent calculations, only the calculations corresponding to these K values ​​need to be performed. This approach effectively reduces the computation and storage of attention in the actual computation. Therefore, a low-complexity Transformer attention module prediction method that can estimate the attention matrix is ​​essential.

[0004] The relevant low-complexity Transformer attention module prediction method utilizes low-bit quantization for attention matrix prediction. Specifically, the word matrix corresponding to the input text of the model is quantized using 4-bit low precision, while the query weight matrix W is... Q Bond weight matrix W K The process involves 4-bit quantization followed by 4-bit multiplication to obtain the corresponding query (Q) matrix and key (K) matrix. Subsequently, the Q and K matrices are further quantized by 4 bits, and then multiplied together to obtain the attention matrix. However, this method requires an integer multiplier for hardware implementation, and the hardware overhead of a 4-bit integer multiplier is still significant. Therefore, related low-complexity Transformer attention module prediction methods have high hardware resource costs. Summary of the Invention

[0005] Therefore, it is necessary to provide a low-complexity Transformer attention module prediction method, apparatus, computer device, computer-readable storage medium, and computer program product that can reduce hardware resource consumption in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a low-complexity Transformer attention module prediction method, including:

[0007] Obtain the first leading zero count result obtained by performing leading zero counting processing on the first matrix of the neural network model; the attention weight matrix of the neural network model is the first matrix, and the word matrix input to the neural network model is the second matrix; or the word matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix;

[0008] Based on the first leading zero count result, the second matrix is ​​subjected to a multiplication approximation shift process to obtain the query matrix and the key matrix;

[0009] Based on the query matrix and the key matrix, the prediction result of the attention matrix is ​​determined; the prediction result of the attention matrix is ​​used to characterize the degree of matching of context terms.

[0010] In one embodiment, determining the prediction result of the attention matrix based on the query matrix and the key matrix includes:

[0011] The third matrix of the neural network model is subjected to leading zero counting to obtain the second leading zero count result corresponding to the third matrix; the query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix; or the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix.

[0012] Based on the second leading zero count result, the fourth matrix is ​​subjected to a multiplication approximation shift process to obtain the prediction result of the attention matrix.

[0013] In one embodiment, the first matrix is ​​the attention weight matrix, and obtaining the first leading zero count result obtained by performing leading zero counting on the first matrix of the neural network model includes:

[0014] The attention weight matrix of the neural network model is subjected to leading zero counting to obtain the first leading zero count result corresponding to the attention weight matrix;

[0015] Store the first leading zero count result into the target storage unit;

[0016] When estimating the attention matrix, the first leading zero count result is read from the target storage unit.

[0017] In one embodiment, the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix. The step of performing a multiplicative approximation shift on the fourth matrix based on the second leading zero count result to obtain the prediction result of the attention matrix includes:

[0018] For each set of multiplication elements in the process of multiplying the query matrix and the transpose of the key matrix, the shift object element corresponding to the second element in the set of multiplication elements is determined according to the first element in the set of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix;

[0019] Based on the number of leading zeros corresponding to the first element, the shift object element is shifted to obtain the first shift result corresponding to the group multiplication element;

[0020] Extend the sign bit of the first shift result to obtain the product corresponding to the elements of the group being multiplied.

[0021] The prediction result of the attention matrix is ​​determined based on the product of the elements of each group.

[0022] In one embodiment, determining the shift object element corresponding to the second element in the group multiplication elements based on the first element in the group multiplication elements includes:

[0023] If the first element in the multiplied group is negative, then the original code of the second element in the multiplied group is inverted bit by bit to obtain the shifted object element corresponding to the second element.

[0024] If the first element in the multiplied group is a positive number, then the original code of the second element in the multiplied group is used as the shift object element corresponding to the second element.

[0025] In one embodiment, the step of shifting the shifted object element based on the number of leading zeros corresponding to the first element to obtain the first shift result corresponding to the grouped multiplication elements includes:

[0026] Based on the sign bit of the first element and the sign bit of the second element, determine the sign bit of the first shift result corresponding to the multiplied elements of the group;

[0027] The number of shift bits corresponding to the shift target element is determined based on the bit width of the second element and the number of leading zeros corresponding to the first element;

[0028] The elements to be shifted are shifted to the left according to the number of shifts to obtain the effective bits of the first shift result corresponding to the elements of the group multiplication;

[0029] The sign bit and the valid bit of the first shift result are used to form the first shift result corresponding to the group of multiplied elements.

[0030] In one embodiment, the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix. The step of performing a multiplicative approximation shift on the fourth matrix based on the second leading zero count result to obtain the prediction result of the attention matrix includes:

[0031] For each set of multiplication elements in the process of multiplying the query matrix and the transpose of the key matrix, based on the number of leading zeros corresponding to the first element in the set of multiplication elements, the second element in the set of multiplication elements is shifted to obtain the second shift result corresponding to the set of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix;

[0032] Based on the second shift result corresponding to each of the multiplication elements of the group, the original code corresponding to each attention element in the prediction result of the attention matrix is ​​determined;

[0033] The original code corresponding to each attention element is decoded to obtain the prediction result of the attention matrix.

[0034] Secondly, this application also provides a low-complexity Transformer attention module prediction device, comprising:

[0035] The acquisition module is used to acquire the first leading zero count result obtained by performing leading zero counting processing on the first matrix of the neural network model; the attention weight matrix of the neural network model is the first matrix, and the word matrix input to the neural network model is the second matrix; or the word matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix;

[0036] The processing module is used to perform a multiplication approximation shift process on the second matrix based on the first leading zero count result to obtain a query matrix and a key matrix;

[0037] The determination module is used to determine the prediction result of the attention matrix based on the query matrix and the key matrix; the prediction result of the attention matrix is ​​used to characterize the degree of matching of context lexical units.

[0038] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps described in the first aspect above.

[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps described in the first aspect above.

[0040] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps described in the first aspect above.

[0041] The aforementioned low-complexity Transformer attention module prediction method, apparatus, computer device, computer-readable storage medium, and computer program product obtain a first leading zero count result by performing leading zero counting processing on a first matrix of a neural network model; the attention weight matrix of the neural network model is the first matrix, and the word matrix input to the neural network model is the second matrix; or the word matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix; based on the first leading zero count result, a multiplicative approximation shift processing is performed on the second matrix to obtain a query matrix and a key matrix; based on the query matrix and the key matrix, the prediction result of the attention matrix is ​​determined; the prediction result of the attention matrix is ​​used to characterize the matching degree of context words. Thus, a one-sided leading zero transformation mechanism is proposed, which only requires leading zero detection on one of the two multipliers, and a multiplicative approximation shift processing is performed on the other multiplier based on the leading zero detection result. Multiplication approximation can be achieved through shifting, and the estimation of the attention matrix does not require hardware implementation using an integer multiplier, effectively reducing hardware resource overhead. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating a low-complexity Transformer attention module prediction method in one embodiment;

[0044] Figure 2 This is a flowchart illustrating the leading zero counting process for 0011(+3) and 1101(-3) in one embodiment;

[0045] Figure 3 This is a flowchart illustrating the steps for determining the prediction result of the attention matrix based on the query matrix and the key matrix in one embodiment.

[0046] Figure 4 This is a flowchart illustrating the steps of obtaining the first leading zero count result by performing leading zero counting on the first matrix of the neural network model in one embodiment.

[0047] Figure 5 This is a schematic diagram illustrating the implementation flow of a low-complexity Transformer attention module prediction method in one embodiment;

[0048] Figure 6 This is a flowchart illustrating the steps in one embodiment of performing a multiplication approximation shift on the fourth matrix based on the second leading zero count result to obtain the prediction result of the attention matrix.

[0049] Figure 7 This is a flowchart illustrating the step of determining the shift object element corresponding to the second element in the group of multiplied elements based on the first element in the group of multiplied elements in one embodiment.

[0050] Figure 8 This is a flowchart illustrating the steps of shifting the elements of the shift object based on the number of leading zeros corresponding to the first element in one embodiment to obtain the first shift result corresponding to the multiplied elements.

[0051] Figure 9 This is a schematic diagram of performing a multiplication approximation shift on the fourth matrix based on the second leading zero count result in one embodiment;

[0052] Figure 10 This is a flowchart illustrating the steps in another embodiment to perform a multiplication approximation shift on the fourth matrix based on the second leading zero count result to obtain the prediction result of the attention matrix.

[0053] Figure 11 This is a schematic diagram illustrating the implementation process of a low-complexity Transformer attention module prediction method in another embodiment;

[0054] Figure 12 This is a schematic diagram of performing a multiplication approximation shift on the fourth matrix based on the second leading zero count result in another embodiment;

[0055] Figure 13 This is a block diagram of a low-complexity Transformer attention module prediction device in one embodiment;

[0056] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In one embodiment, such as Figure 1 As shown, a low-complexity Transformer attention module prediction method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. In this embodiment, the method includes the following steps:

[0059] Step 101: Obtain the first leading zero count result obtained by performing leading zero counting on the first matrix of the neural network model.

[0060] In this configuration, the attention weight matrix of the neural network model is the first matrix, and the word matrix input to the neural network model is the second matrix. Alternatively, the word matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix.

[0061] In this embodiment, the neural network model is a transformer-based neural network model, including an attention mechanism. The token matrix corresponding to the language sequence is input into the attention module of the neural network model to obtain the attention matrix. The attention matrix is ​​used to characterize the importance of the contextual relationship between each token in the language sequence and tokens at other positions, i.e., the degree of matching of contextual tokens. Therefore, the attention matrix can also be called the contextual token association matrix or the contextual token matching degree matrix. Specifically, the token matrix X is compared with the query weight matrix W. Q Bond weight matrix W KMultiplying these matrices yields the corresponding query (Q) matrix and key (K) matrix; subsequently, multiplying the Q matrix by the transpose of the K matrix yields the attention matrix. The specific process described above can be represented as follows:

[0062] Q=X×W Q

[0063] K=X×W K

[0064] Attention = Q × K T

[0065] The attention weight matrix includes a query weight matrix and a key weight matrix. Leading zero: The 0 bits preceding the first 1 bit in a binary number. For example, for the binary number (00101110), the number of leading zeros is 2. Leading zero counter (LoC): Used to count leading zeros. For a binary number, it counts the number of 0 bits preceding the most significant 1 bit, i.e., LoC(00101110) = 2.

[0066] In one example, when estimating the attention matrix, the terminal performs leading zero counting on the first matrix of the neural network model to obtain the first leading zero count result.

[0067] In one example, when the first matrix is ​​an attention weight matrix, the first leading zero count result includes the leading zero count result of the query weight matrix and the leading zero count result of the key weight matrix. The terminal performs leading zero counting on the query weight matrix and the key weight matrix respectively to obtain the leading zero count result of the query weight matrix and the leading zero count result of the key weight matrix.

[0068] In one example, when the first matrix is ​​a lexical matrix, the terminal performs leading zero counting on the lexical matrix to obtain the first leading zero count result.

[0069] The specific process of leading zero counting for a matrix includes: the leading zero count result includes the number of leading zeros corresponding to each element in the matrix. For each element in the matrix, if the binary number of the element is positive, the terminal performs leading zero counting on the binary number to obtain the number of leading zeros corresponding to the element. If the binary number of the element is negative, the terminal converts the binary number to its original code. Then, the terminal ignores the highest sign bit (1) of the original code and performs leading zero counting on the original code to obtain the number of leading zeros corresponding to the element. For example, 0011(+3) has 2 leading zeros; 1101(-3) is first converted to its original code 1011(-3), ignoring its highest sign bit, and its leading zero count is 2. The flowcharts for performing leading zero counting on 0011(+3) and 1101(-3) are shown below. Figure 2 As shown.

[0070] In one example, the terminal uses a priority encoder or logic gate to perform leading zero counting on the matrix.

[0071] Step 102: Based on the first leading zero count result, perform a multiplication approximation shift on the second matrix to obtain the query matrix and the key matrix.

[0072] In this embodiment, the second matrix can be a one-hot sequence matrix or a non-one-hot sequence matrix. The multiplication approximation shifting process is used to obtain an approximate result of multiplying binary numbers by shifting the binary numbers. It may or may not include one-hot sequence conversion processing. One-hot sequence: A string of 0s and 1s in a binary vector, containing only one 1 bit, such as 00010000. One-hot sequence converter (OSC): For a binary number, converting it to a one-hot sequence while retaining the most significant 1 bit, for example, 00101110 -> 00100000.

[0073] In one example, when the first matrix is ​​the attention weight matrix, the first leading zero count includes the leading zero count of the query weight matrix and the leading zero count of the key weight matrix. Based on the leading zero count of the query weight matrix, the terminal performs a multiplicative approximation shift on the lexical matrix to obtain the query matrix. Based on the leading zero count of the key weight matrix, the terminal performs a multiplicative approximation shift on the lexical matrix to obtain the key matrix.

[0074] In one example, when the first matrix is ​​a token matrix, the terminal performs a multiplicative approximation shift on the query weight matrix based on the first leading zero count to obtain the query matrix. The terminal also performs a multiplicative approximation shift on the key weight matrix based on the first leading zero count to obtain the key matrix.

[0075] The principle behind multiplication approximation using shift processing is as follows: For an integer (INT) type binary number x, its mathematical representation is x = sign × 2. W-LO-1 ×M, where sign represents the sign bit, W represents the quantization bit width, LO represents the number of leading zeros, and M represents the mantissa, whose value lies in the interval [1, 2]. For example, the value 3 under 4-bit quantization is 3 = (+1) × 2. 4-2-1 ×1.5, its binary representation is 3 = (0011) b The number of leading zeros, LO = 2. Therefore, the multiplication of two binary numbers can be expressed mathematically as follows:

[0076] x×y=XOR(S x ,S y )×2 (Wx+Wy-(LOx+LOy)-2) ×(M x ×M y )

[0077] Among them, S x and S y These represent the sign bits of x and y, respectively. When two numbers are multiplied, the sign bit of the product is equivalent to the XOR of the sign bits of the two multipliers. x and W y These represent the quantization bit widths of x and y, respectively; LOx and LOy represent the number of leading zeros in x and y, respectively; M x and M y Let x and y represent the mantissas, respectively. Since the mantissas are between [1, 2], they are usually small. Therefore, the product of the two numbers can be approximated by shifting the leading zeros of x and y, i.e.

[0078] x×y≈XOR(S x ,S y )×2 (Wx+Wy-(LOx+LOy)-2) ×M x

[0079] For example, the calculation of 00011000(24)×00000110(6)=10010000(144) can be approximated by 00010000(16)×00000110(6)=01100000(96). It is easy to see that the result 01100000 is equivalent to shifting the bit 1 in 00000110 to the left by four bits, and the number of bits shifted to the left is equivalent to the bit width of 8 minus the number of leading zeros in 00010000 minus 1.

[0080] Step 103: Based on the query matrix and key matrix, determine the prediction result of the attention matrix.

[0081] The prediction results of the attention matrix are used to characterize the degree of matching of context terms.

[0082] In the embodiments of this application, the prediction result of the attention matrix is ​​an approximate result of the attention matrix, which can be an approximate result of multiplying the query matrix and the transpose of the key matrix.

[0083] In one example, the terminal performs a multiplicative approximation on the transpose of the query matrix and the key matrix to obtain the predicted result of the attention matrix. This multiplicative approximation is used to obtain an approximate result of multiplying binary numbers, and may or may not include one-hot sequence transformation. The multiplicative approximation includes a multiplicative approximation shift operation.

[0084] In one example, the terminal performs leading zero counting on the third matrix of the neural network model to obtain a second leading zero count result for the third matrix. Then, the terminal performs a one-hot sequence transformation on the fourth matrix to obtain the transformed fourth matrix. Next, based on the second leading zero count result, the terminal performs a shift operation on the transformed fourth matrix to obtain the prediction result of the attention matrix. Here, the query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix. Alternatively, the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix.

[0085] In the aforementioned low-complexity Transformer attention module prediction method, a first leading zero count result is obtained by performing leading zero counting on the first matrix of the neural network model; the attention weight matrix of the neural network model is the first matrix, and the word matrix input to the neural network model is the second matrix; or the word matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix; based on the first leading zero count result, a multiplicative approximation shift is performed on the second matrix to obtain the query matrix and the key matrix; based on the query matrix and the key matrix, the prediction result of the attention matrix is ​​determined; the prediction result of the attention matrix is ​​used to characterize the matching degree of context words. Thus, a one-sided leading zero transformation mechanism is proposed, which only requires leading zero detection on one of the two multipliers, and a multiplicative approximation shift is performed on the other multiplier based on the leading zero detection result. Multiplication approximation can be achieved through shifting, and the estimation of the attention matrix does not require the use of an integer multiplier for hardware implementation, effectively reducing hardware resource overhead, chip area, and chip runtime power consumption and energy consumption. Furthermore, the object undergoing multiplication approximation shift processing in the proposed one-sided leading zero conversion mechanism can be the original data that has not undergone one-hot sequence conversion. This eliminates the need for one-hot sequence conversion and the hardware overhead of one-hot sequence converters, thereby further reducing hardware resource consumption and hardware area, further reducing chip area, and further reducing chip power consumption and chip energy consumption during operation.

[0086] In one exemplary embodiment, such as Figure 3 As shown, the specific process of determining the prediction result of the attention matrix based on the query matrix and the key matrix includes the following steps:

[0087] Step 301: Perform leading zero counting on the third matrix of the neural network model to obtain the second leading zero count result corresponding to the third matrix.

[0088] In this case, the query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix. Alternatively, the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix.

[0089] In this embodiment, the second leading zero count result is the leading zero count result obtained by performing leading zero counting processing on the third matrix of the neural network model.

[0090] Step 302: Based on the second leading zero count result, perform a multiplication approximation shift on the fourth matrix to obtain the prediction result of the attention matrix.

[0091] In the aforementioned low-complexity Transformer attention module prediction method, leading zero counting is performed on the third matrix of the neural network model to obtain the second leading zero count result corresponding to the third matrix. Based on the second leading zero count result, a multiplicative approximation shift is performed on the fourth matrix to obtain the predicted attention matrix. Thus, the transpose multiplication of the query matrix and the key matrix also employs the newly proposed one-sided leading zero transformation mechanism. Only one of the two multipliers needs to be zero-detected, and the other multiplier is then subjected to a multiplicative approximation shift based on the zero-detection result. This shifting allows for multiplication approximation, ensuring that the estimation of the attention matrix does not require hardware implementation using integer multipliers, effectively reducing hardware resource overhead.

[0092] In one exemplary embodiment, such as Figure 4 As shown, the first matrix is ​​the attention weight matrix. The specific process of obtaining the first leading zero count result by performing leading zero counting on the first matrix of the neural network model includes the following steps:

[0093] Step 401: Perform leading zero counting on the attention weight matrix of the neural network model to obtain the first leading zero count result corresponding to the attention weight matrix.

[0094] In this embodiment of the application, during the network deployment stage of the neural network model, the attention weight matrix of the neural network model is subjected to leading zero counting to obtain the first leading zero count result corresponding to the attention weight matrix.

[0095] Step 402: Store the first leading zero count result into the target storage unit.

[0096] In this embodiment, the terminal determines the second digit of the element in the first leading zero count result based on the first digit of the element in the attention weight matrix. For example, In this process, B represents the second digit and A represents the first digit. Then, for each decimal element in the first leading zero count result, the terminal converts that decimal element into a binary number with a quantization width of the second digit, obtaining the converted first leading zero count result. The terminal then stores the converted first leading zero count result into the target storage unit. The target storage unit can be Dynamic Random Access Memory (DRAM).

[0097] Step 403: When estimating the attention matrix, read the first leading zero count result from the target storage unit.

[0098] In the aforementioned low-complexity Transformer attention module prediction method, leading zero counting is performed on the attention weight matrix of the neural network model to obtain the first leading zero count result corresponding to the attention weight matrix; the first leading zero count result is stored in the target storage unit; when estimating the attention matrix, the first leading zero count result is read from the target storage unit. Thus, since the network weights are known once the network training is complete during neural network inference, this invention pre-detects leading zeros in the attention weight matrix and stores them in the storage unit, reducing the data storage bit width and data transmission bandwidth. For example, for an 8-bit binary number 00011001, the number of leading zeros is 3, which is represented as 0011 in binary. Directly storing 00011001 requires storing 8 bits, while storing the leading zero count only requires storing 4 bits, saving storage capacity and bandwidth resources required to read data from the storage unit to the chip. Meanwhile, this method eliminates the need for leading zero detection during subsequent attention matrix estimations, thus reducing converter overhead and further minimizing hardware resource consumption.

[0099] In one embodiment, a schematic diagram of the implementation flow of the low-complexity Transformer attention module prediction method is shown below. Figure 5 As shown. During the network deployment phase, the terminal uses two weight matrices W. Q and W K Preprocessing is performed, converting each element by leading zero count, and the converted result is stored in the target memory unit DRAM. When estimating the attention matrix, i.e., in the formal computation, the terminal directly inputs the W of the weight matrix. Q and W KThe number of leading zeros is directly read from DRAM. A multiplicative approximation shift is performed on the input token matrix X to obtain matrices Q and K. Then, leading zeros are counted on matrix Q, and based on this count, a multiplicative approximation shift is performed on matrix K to obtain the predicted attention matrix. It is evident that this method cleverly utilizes the known weight characteristics during neural network inference and reduces the need for one-hot sequence conversion. The overall computation steps are only four, with very few steps involved in estimating the attention matrix. This saves unnecessary one-hot sequence conversion and decoding module overhead, thus effectively reducing hardware area consumption.

[0100] In one exemplary embodiment, such as Figure 6 As shown, the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix. Based on the second leading zero count result, the fourth matrix is ​​subjected to a multiplication approximation shift to obtain the prediction result of the attention matrix. The specific process includes the following steps:

[0101] Step 601: For each set of multiplication elements in the process of multiplying the query matrix and the key matrix by transpose, determine the shift object element corresponding to the second element in the set of multiplication elements based on the first element in the set of multiplication elements.

[0102] The first element is an element in the third matrix, and the second element is an element in the fourth matrix.

[0103] In this embodiment, a set of multiplied elements includes a first element and a second element. When the query matrix is ​​a third matrix and the transpose of the key matrix is ​​a fourth matrix, the number of columns of the first element in the set of multiplied elements is the same as the number of rows of the second element in the set of multiplied elements. When the transpose of the key matrix is ​​a third matrix and the query matrix is ​​a fourth matrix, the number of rows of the first element in the set of multiplied elements is the same as the number of columns of the second element in the set of multiplied elements. The shifted object element is the object to be shifted.

[0104] Step 602: Based on the number of leading zeros corresponding to the first element, perform shift processing on the shifted object elements to obtain the first shift result corresponding to the multiplied elements.

[0105] In this embodiment of the application, the first shift result is the result of shifting the target element based on the number of leading zeros corresponding to the first element.

[0106] Step 603: Extend the sign bit of the first shift result to obtain the product corresponding to the multiplied elements.

[0107] Step 604: Determine the prediction result of the attention matrix based on the product of the elements in each group.

[0108] In this embodiment, for each attention element in the prediction result of the attention matrix, the terminal determines the target group multiplication element corresponding to the attention element based on the number of rows and columns of the attention element and the number of rows and columns corresponding to the first and second elements in each group of multiplication elements. Then, the terminal performs a first accumulation process on the products corresponding to the multiplication elements of each target group to obtain the attention element. Specifically, the terminal accumulates the products corresponding to the multiplication elements of each target group according to the two's complement arithmetic rule to obtain the attention element. Then, the terminal uses each attention element to construct the prediction result of the attention matrix. The first accumulation process is performed according to the two's complement arithmetic rule.

[0109] In one example, when the query matrix is ​​a third matrix and the key matrix is ​​transposed into a fourth matrix, the terminal determines the target group multiplier element corresponding to the attention element by multiplying the group whose first element has the same number of rows as the attention element and whose second element has the same number of columns as the attention element. Similarly, when the key matrix is ​​transposed into a third matrix and the query matrix is ​​a fourth matrix, the terminal determines the target group multiplier element corresponding to the attention element by multiplying the group whose first element has the same number of columns as the attention element and whose second element has the same number of rows as the attention element.

[0110] In the aforementioned low-complexity Transformer attention module prediction method, for each group of multiplying elements in the transpose multiplication process of the query matrix and the key matrix, the shift object element corresponding to the second element in the group of multiplying elements is determined based on the first element in the group of multiplying elements; based on the number of leading zeros corresponding to the first element, the shift object element is shifted to obtain the first shift result corresponding to the group of multiplying elements; the sign bit of the first shift result is extended to obtain the product corresponding to the group of multiplying elements; based on the product corresponding to each group of multiplying elements, the prediction result of the attention matrix is ​​determined. In this way, by directly shifting the shift object element of another multiplier based on the number of leading zeros of one multiplier and extending the sign bit, multiplication approximation can be achieved without one-hot sequence conversion and decoding, which can further reduce hardware resource overhead and hardware area.

[0111] In one exemplary embodiment, such as Figure 7 As shown, the specific process of determining the shift target element corresponding to the second element in the same group of multiplied elements, based on the first element, includes the following steps:

[0112] Step 701: If the first element in the multiplied group is negative, then invert the original code of the second element in the multiplied group bit by bit to obtain the shifted object element corresponding to the second element.

[0113] In this embodiment of the application, if the first element in the group of multiplied elements is negative, the terminal performs a bitwise inversion of the original code of the second element in the group of multiplied elements, including the sign bit, to obtain the shift object element corresponding to the second element.

[0114] Step 702: If the first element in the multiplied group is a positive number, then the original code of the second element in the multiplied group is used as the shift object element corresponding to the second element.

[0115] In the aforementioned low-complexity Transformer attention module prediction method, if the first element in the multiplicative group is negative, the original code of the second element in the multiplicative group is inverted bit by bit to obtain the shift target element corresponding to the second element; if the first element in the multiplicative group is positive, the original code of the second element in the multiplicative group is used as the shift target element corresponding to the second element. In this way, when performing multiplicative approximation, no one-hot sequence transformation is needed; only the mantissa of one of the two multipliers is ignored, resulting in a very small error and ensuring the accuracy of the attention matrix estimation.

[0116] In one exemplary embodiment, such as Figure 8 As shown, the specific process of shifting the elements of the shifted object based on the number of leading zeros corresponding to the first element to obtain the first shift result corresponding to the multiplied elements includes the following steps:

[0117] Step 801: Determine the sign bit of the first shift result corresponding to the multiplied elements based on the sign bit of the first element and the sign bit of the second element.

[0118] In this embodiment, if the sign bit of the first element and the sign bit of the second element are the same, the terminal determines 0 as the sign bit of the first shift result corresponding to the multiplied elements. If the sign bit of the first element and the sign bit of the second element are different, the terminal determines 1 as the sign bit of the first shift result corresponding to the multiplied elements.

[0119] Step 802: Determine the number of shift bits corresponding to the shift target element based on the bit width of the second element and the number of leading zeros corresponding to the first element.

[0120] In this embodiment, the terminal subtracts the number of leading zeros and 1 corresponding to the first element from the bit width of the second element to obtain the number of shift bits corresponding to the shift target element, which can be expressed as: the number of shift bits corresponding to the shift target element = the bit width of the second element - the number of leading zeros corresponding to the first element - 1.

[0121] Step 803: Shift the elements to the left according to the number of shifts to obtain the valid bits of the first shift result corresponding to the multiplied elements.

[0122] In this embodiment, the terminal shifts the effective bits of the shifted object element to the left by the shift number bits to obtain the effective bits of the first shift result corresponding to the multiplied elements.

[0123] Step 804: The sign bit of the first shift result and the valid bit of the first shift result are used to form the first shift result corresponding to the multiplied elements.

[0124] In the aforementioned low-complexity Transformer attention module prediction method, the sign bit of the first shift result corresponding to the multiplied elements is determined based on the sign bit of the first element and the sign bit of the second element; the shift bit corresponding to the shifted element is determined based on the bit width of the second element and the number of leading zeros of the first element; the shifted element is then left-shifted according to the shift bit to obtain the valid bits of the first shift result corresponding to the multiplied elements; the sign bit and the valid bits of the first shift result constitute the first shift result corresponding to the multiplied elements. In this way, by left-shifting bit 1 of the shifted element corresponding to one of the multipliers, and the left shift bit being the bit width minus the number of leading zeros of the other multiplier minus 1, a multiplication approximation is achieved, ignoring only the mantissa of one of the two multipliers, resulting in a very small error and ensuring the accuracy of the attention matrix estimation.

[0125] In one embodiment, a schematic diagram illustrating the multiplication approximation shift processing of the fourth matrix based on the second leading zero count result is shown below. Figure 9 As shown. The product of the query matrix and the transpose of the key matrix is ​​[x0, x1] × [y0, y1]. T=z=[x0×y0+x1×y1], x0=+3, x1=+4, y0=-3, y1=-3, and the transpose of the key matrix is ​​the third matrix, the query matrix is ​​the fourth matrix, the first group of multiplied elements are x0 (second element) and y0 (first element), and the second group of multiplied elements are x1 (second element) and y1 (first element). For the first group of multiplied elements: +3 (x0) is represented by 0011 in 4 bits. Since -3 (y0) is negative, the terminal inverts the original code of x0 bit by bit to obtain the shifted element 1100. Since x0 is positive and y0 is negative, their sign bits are different. The terminal determines 1 as the sign bit of the first shift result corresponding to the multiplied elements. The number of leading zeros L0 of -3 (y0) is 2, so the number of shift bits is 4-2-1=1. The terminal shifts 1100 left by 1 bit to obtain the valid bits of the first shift result corresponding to the multiplied elements. The sign bit and the valid bits of the first shift result are combined to form the first shift result 10001000. The sign bit of 10001000 is extended to obtain the product 11111000 corresponding to the multiplied elements. For the second group of multiplied elements: +4 (x1) is represented by 0100 in 4 bits. Since -3 (y1) is negative, the terminal inverts the original code of x1 bit by bit to obtain the shift object element 1011 corresponding to x1. Since x1 is positive and y1 is negative, their sign bits are different. The terminal determines 1 as the sign bit of the first shift result corresponding to this group of multiplied elements. The number of leading zeros L0 of -3 (y1) is 2, so the number of shift bits is 4-2-1=1. The terminal shifts 1011 left by 1 bit to obtain the valid bit of the first shift result corresponding to this group of multiplied elements. The sign bit and the valid bit of the first shift result are combined to form the first shift result 10000110 corresponding to this group of multiplied elements. The sign bit of 10000110 is extended to obtain the product 11110110 corresponding to this group of multiplied elements. Then, the terminal adds the two products according to the two's complement calculation rule to obtain the attention element 11101110. Then, the terminal uses the attention element to form the prediction result of the attention matrix [11101110].

[0126] In one exemplary embodiment, such as Figure 10 As shown, the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix. Based on the second leading zero count result, the fourth matrix is ​​subjected to a multiplication approximation shift to obtain the prediction result of the attention matrix. The specific process includes the following steps:

[0127] Step 1001: For each group of multiplication elements in the process of multiplying the query matrix and the key matrix by transpose, based on the number of leading zeros corresponding to the first element in the group of multiplication elements, shift the second element in the group of multiplication elements to obtain the second shift result corresponding to the group of multiplication elements.

[0128] The first element is an element in the third matrix, and the second element is an element in the fourth matrix.

[0129] In this embodiment, the terminal determines the sign bit of the second shift result corresponding to the multiplied elements based on the sign bit of the first element and the sign bit of the second element. Then, the terminal determines the shift bit number corresponding to the second element based on the bit width of the second element and the number of leading zeros corresponding to the first element. Next, the terminal shifts the second element to the left according to the shift bit number to obtain the valid bits of the second shift result corresponding to the multiplied elements. Then, the terminal uses the sign bit of the second shift result and the valid bits of the second shift result to construct the second shift result corresponding to the multiplied elements. The second shift result is the result of shifting the second element based on the number of leading zeros corresponding to the first element. It can be understood that the specific process by which the terminal shifts the second element in the multiplied elements based on the number of leading zeros corresponding to the first element to obtain the second shift result is similar to the specific process by which the terminal shifts the shifted element based on the number of leading zeros corresponding to the first element to obtain the first shift result, and is similar to the specific process of steps 801-804.

[0130] Step 1002: Based on the second shift result corresponding to each group of multiplied elements, determine the original code corresponding to each attention element in the prediction result of the attention matrix.

[0131] In this embodiment, for each attention element in the prediction result of the attention matrix, the terminal determines the target group multiplication element corresponding to the attention element based on the number of rows and columns of the attention element and the number of rows and columns corresponding to the first and second elements in each group of multiplication elements. Then, the terminal performs a second accumulation process on the second shift result corresponding to each target group multiplication element to obtain the original code corresponding to the attention element.

[0132] Specifically, for each accumulation object undergoing the second accumulation process, if the accumulation object is negative, the terminal updates the original values ​​corresponding to each bit of the accumulation object to the negative values ​​of those original values. If the accumulation object is positive, the terminal retains the original values ​​corresponding to each bit of the accumulation object. Then, the terminal performs bitwise accumulation on each accumulation object to obtain the result of the second accumulation process.

[0133] Step 1003: Decode the original code corresponding to each attention element to obtain the prediction result of the attention matrix.

[0134] In this embodiment, the terminal converts the original code corresponding to each attention element into binary two's complement to obtain the prediction result of the attention matrix.

[0135] Specifically, for each attention element, the terminal performs a weighted summation based on the values ​​of each bit in the original code corresponding to that attention element and their corresponding weights to obtain the decimal number for that attention element. Then, the terminal converts this decimal number to binary to obtain the binary number corresponding to that attention element. Finally, the terminal converts this binary number to two's complement to obtain the prediction result of the attention matrix.

[0136] In the aforementioned low-complexity Transformer attention module prediction method, for each group of multiplying elements in the process of multiplying the query matrix and the key matrix by transpose, the second element in the group of multiplying elements is shifted based on the number of leading zeros corresponding to the first element in that group of multiplying elements, resulting in a second shifted result. Based on the second shifted result of each group of multiplying elements, the original code corresponding to each attention element in the prediction result of the attention matrix is ​​determined. The original code corresponding to each attention element is then decoded to obtain the prediction result of the attention matrix. In this way, when multiplying two multipliers, one of the multipliers' bits is shifted left by a number equal to the bit width minus the number of leading zeros of the other multiplier minus 1. The result of the shift accumulation is then decoded. Compared to the aforementioned low-complexity Transformer attention module prediction method that does not perform decoding, this method does not have the error of 1 caused by simply inverting without adding 1, which can further improve the accuracy of attention matrix estimation.

[0137] In another embodiment, a schematic diagram of the implementation flow of the low-complexity Transformer attention module prediction method is shown below. Figure 11 As shown. During the network deployment phase, the terminal uses two weight matrices W. Q and W K Preprocessing is performed, converting each element by leading zero count, and the converted result is stored in the target memory unit DRAM. When estimating the attention matrix, i.e., in the formal computation, the terminal directly inputs the W of the weight matrix. Q and W KThe number of leading zeros is read directly from DRAM. The input token matrix X is shifted and then accumulated (shift-accumulation). The result is then decoded to obtain matrices Q and K. Leading zeros are counted on matrix Q. Based on the count, matrix K is shifted and then accumulated (shift-accumulation). The result is then decoded and converted to two's complement to obtain the predicted attention matrix.

[0138] In another embodiment, a schematic diagram of performing a multiplication approximation shift on the fourth matrix based on the second leading zero count result is shown below. Figure 12 As shown. The product of the query matrix and the transpose of the key matrix is ​​[x0, x1] × [y0, y1]. T =z=[x0×y0+x1×y1], x0=+3, x1=+4, y0=-3, y1=-3, and the transpose of the key matrix is ​​the third matrix, the query matrix is ​​the fourth matrix, the first group of multiplied elements are x0 (second element) and y0 (first element), and the second group of multiplied elements are x1 (second element) and y1 (first element). For the first group of multiplied elements: +3 (x0) is represented by 0011 in 4 bits. Since x0 is a positive number and y0 is a negative number, their sign bits are different. The terminal determines 1 as the sign bit of the second shift result corresponding to this group of multiplied elements. The number of leading zeros L0 of -3 (y0) is 2, so the number of shift bits is 4-2-1=1. The terminal shifts 0011 left by 1 bit to obtain the valid bit 0110 of the second shift result corresponding to this group of multiplied elements. The sign bit of the second shift result and the valid bit of the second shift result are combined to form the second shift result 10000110 corresponding to this group of multiplied elements. For the second group of multiplied elements: +4 (x1) is represented by 0100 in 4-bit binary. Since x1 is positive and y1 is negative, their sign bits are different. The terminal determines 1 as the sign bit of the second shift result corresponding to this group of multiplied elements. The number of leading zeros L0 of -3 (y1) is 2, so the number of shift bits is 4-2-1=1. The terminal shifts 0100 left by 1 bit to obtain the effective bit 1000 of the second shift result corresponding to this group of multiplied elements. The terminal then combines the sign bit of the second shift result with the effective bit of the second shift result to form the second shift result 10001000. Then, the terminal performs a second accumulation process on the two second shift results to obtain the original code [0 0 0 -1 -1 -1 0] corresponding to the attention element. Then, the terminal decodes the original code corresponding to the attention element to obtain the prediction result 11110010 of the attention matrix. Finally, the terminal uses this attention element to form the prediction result [11110010] of the attention matrix.

[0139] In one embodiment, step 102 may include the following: the first leading zero count result includes the number of leading zeros corresponding to each third element in the first matrix. For each group of multiplication elements in the process of multiplying the lexical matrix with the query weight matrix and the process of multiplying the lexical matrix with the key weight matrix, the terminal determines the shift object element corresponding to the fourth element in the group of multiplication elements based on the third element in that group. The third element is an element in the first matrix, and the fourth element is an element in the second matrix. Then, the terminal performs shift processing on the shift object element based on the number of leading zeros corresponding to the third element, obtaining the third shift result corresponding to the group of multiplication elements. Then, the terminal performs sign bit extension on the third shift result to obtain the product corresponding to the group of multiplication elements. Then, the terminal determines the query matrix and the key matrix based on the product corresponding to each group of multiplication elements. It is understood that the specific process of the above steps is similar to the specific process of steps 601-604.

[0140] The specific process by which the terminal determines the shift target element corresponding to the fourth element in the multiplied elements based on the third element may include the following steps: If the third element in the multiplied elements is negative, the original code of the fourth element in the multiplied elements is inverted bit by bit to obtain the shift target element corresponding to the fourth element. If the third element in the multiplied elements is positive, the original code of the fourth element in the multiplied elements is used as the shift target element corresponding to the second element. It can be understood that the specific process of the above steps is similar to that of steps 701-702.

[0141] The specific process by which the terminal shifts the elements to be shifted based on the number of leading zeros corresponding to the third element to obtain the third shift result of the multiplied elements includes the following steps: The terminal determines the sign bit of the third shift result of the multiplied elements based on the sign bit of the third element and the sign bit of the fourth element. Then, the terminal determines the number of shift bits corresponding to the elements to be shifted based on the bit width of the fourth element and the number of leading zeros corresponding to the third element. Then, the terminal shifts the elements to the left according to the number of shift bits to obtain the valid bits of the third shift result of the multiplied elements. Finally, the terminal combines the sign bit and the valid bits of the third shift result to form the third shift result of the multiplied elements. It can be understood that the specific process of the above steps is similar to that of steps 801-804.

[0142] The specific process by which the terminal determines the query matrix and key matrix based on the products of the corresponding multiplication elements includes the following steps: For each query element in the query matrix, the terminal determines the target group multiplication element corresponding to the query element from among the multiplication elements in the process of multiplying the lexical matrix and the query weight matrix, based on the number of rows and columns of the query element and the number of rows and columns corresponding to the third and fourth elements in each group of multiplication elements. Then, the terminal performs a first accumulation process on the products of the target group multiplication elements corresponding to the query element to obtain the query element. Then, the terminal constructs the query matrix from the query elements. For each key element in the key matrix, the terminal determines the target group multiplication element corresponding to the key element from among the multiplication elements in the process of multiplying the lexical matrix and the key weight matrix, based on the number of rows and columns of the key element and the number of rows and columns corresponding to the third and fourth elements in each group of multiplication elements. Then, the terminal performs a first accumulation process on the products of the target group multiplication elements corresponding to the key element to obtain the key element. Then, the terminal constructs the key matrix from the key elements. It is understandable that the specific process by which the terminal determines the query matrix based on the product of the corresponding elements of each group and the specific process by which the terminal determines the key matrix based on the product of the corresponding elements of each group are similar to the specific process in step 604.

[0143] In another embodiment, step 102 may include the following: The first leading zero count result includes the number of leading zeros corresponding to each third element in the first matrix. For each group of multiplication elements in the process of multiplying the lexical matrix with the query weight matrix and the process of multiplying the lexical matrix with the key weight matrix, based on the number of leading zeros corresponding to the third element in the group of multiplication elements, the fourth element in the group of multiplication elements is shifted to obtain the fourth shift result corresponding to the group of multiplication elements. Here, the third element is an element in the first matrix, and the fourth element is an element in the second matrix. Then, based on the fourth shift result corresponding to each group of multiplication elements, the terminal determines the original code corresponding to each query element in the query matrix and the original code corresponding to each key element in the key matrix. Specifically, for each query element in the query matrix, the terminal determines the target group of multiplication elements corresponding to the query element in each group of multiplication elements in the process of multiplying the lexical matrix with the query weight matrix, based on the number of rows and columns of the query element and the number of rows and columns corresponding to the first and second elements in each group of multiplication elements. Then, the terminal performs a second accumulation process on the fourth shift result corresponding to the target group multiplication element of each query element to obtain the original code corresponding to the query element. For each key element in the key matrix, the terminal determines the target group multiplication element corresponding to the key element in the multiplication process of the word matrix and the key weight matrix based on the number of rows and columns of the key element and the number of rows and columns corresponding to the first and second elements in each group of multiplication elements. Then, the terminal performs a second accumulation process on the fourth shift result corresponding to the target group multiplication element of each key element to obtain the original code corresponding to the key element. Then, the terminal decodes the original code corresponding to each query element to obtain the query matrix. The terminal decodes the original code corresponding to each key element to obtain the key matrix. It can be understood that the specific process of the above steps is similar to the specific process of steps 1001-1003.

[0144] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0145] Based on the same inventive concept, this application also provides a low-complexity Transformer attention module prediction device for implementing the low-complexity Transformer attention module prediction method described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations in one or more embodiments of the low-complexity Transformer attention module prediction device provided below can be found in the limitations of the low-complexity Transformer attention module prediction method described above, and will not be repeated here.

[0146] In one exemplary embodiment, such as Figure 13 As shown, a low-complexity Transformer attention module prediction device 1300 is provided, including: an acquisition module 1310, a processing module 1320, and a determination module 1330, wherein:

[0147] The acquisition module 1310 is used to acquire the first leading zero count result obtained by performing leading zero counting processing on the first matrix of the neural network model; the attention weight matrix of the neural network model is the first matrix, and the word matrix input to the neural network model is the second matrix; or the word matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix.

[0148] Processing module 1320 is used to perform multiplication approximation shift processing on the second matrix based on the first leading zero count result to obtain a query matrix and a key matrix;

[0149] The determination module 1330 is used to determine the prediction result of the attention matrix based on the query matrix and the key matrix; the prediction result of the attention matrix is ​​used to characterize the degree of matching of context lexical units.

[0150] Optionally, the determining module 1330 is specifically used for:

[0151] The third matrix of the neural network model is subjected to leading zero counting to obtain the second leading zero count result corresponding to the third matrix; the query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix; or the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix.

[0152] Based on the second leading zero count result, the fourth matrix is ​​subjected to a multiplication approximation shift process to obtain the prediction result of the attention matrix.

[0153] Optionally, the first matrix is ​​the attention weight matrix, and the acquisition module 1310 is specifically used for:

[0154] The attention weight matrix of the neural network model is subjected to leading zero counting to obtain the first leading zero count result corresponding to the attention weight matrix;

[0155] Store the first leading zero count result into the target storage unit;

[0156] When estimating the attention matrix, the first leading zero count result is read from the target storage unit.

[0157] Optionally, the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix, and the determining module 1330 is specifically used for:

[0158] For each set of multiplication elements in the process of multiplying the query matrix and the transpose of the key matrix, the shift object element corresponding to the second element in the set of multiplication elements is determined according to the first element in the set of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix;

[0159] Based on the number of leading zeros corresponding to the first element, the shift object element is shifted to obtain the first shift result corresponding to the group multiplication element;

[0160] Extend the sign bit of the first shift result to obtain the product corresponding to the elements of the group being multiplied.

[0161] The prediction result of the attention matrix is ​​determined based on the product of the elements of each group.

[0162] Optionally, the determining module 1330 is specifically used for:

[0163] If the first element in the multiplied group is negative, then the original code of the second element in the multiplied group is inverted bit by bit to obtain the shifted object element corresponding to the second element.

[0164] If the first element in the multiplied group is a positive number, then the original code of the second element in the multiplied group is used as the shift object element corresponding to the second element.

[0165] Optionally, the determining module 1330 is specifically used for:

[0166] Based on the sign bit of the first element and the sign bit of the second element, determine the sign bit of the first shift result corresponding to the multiplied elements of the group;

[0167] The number of shift bits corresponding to the shift target element is determined based on the bit width of the second element and the number of leading zeros corresponding to the first element;

[0168] The elements to be shifted are shifted to the left according to the number of shifts to obtain the effective bits of the first shift result corresponding to the elements of the group multiplication;

[0169] The sign bit and the valid bit of the first shift result are used to form the first shift result corresponding to the group of multiplied elements.

[0170] Optionally, the second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix, and the determining module 1330 is specifically used for:

[0171] For each set of multiplication elements in the process of multiplying the query matrix and the transpose of the key matrix, based on the number of leading zeros corresponding to the first element in the set of multiplication elements, the second element in the set of multiplication elements is shifted to obtain the second shift result corresponding to the set of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix;

[0172] Based on the second shift result corresponding to each of the multiplication elements of the group, the original code corresponding to each attention element in the prediction result of the attention matrix is ​​determined;

[0173] The original code corresponding to each attention element is decoded to obtain the prediction result of the attention matrix.

[0174] The modules in the aforementioned low-complexity Transformer attention module prediction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0175] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 14As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a low-complexity Transformer attention module prediction method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0176] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0177] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0178] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0179] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0181] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0182] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0183] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A low-complexity Transformer attention module prediction method, characterized in that, The method includes: Obtain the first leading zero count result obtained by performing leading zero counting processing on the first matrix of the neural network model; the attention weight matrix of the neural network model is the first matrix, and the word matrix input to the neural network model is the second matrix; or the word matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix; Based on the first leading zero count result, the second matrix is ​​subjected to a multiplicative approximation shift process to obtain the query matrix and the key matrix; wherein, the principle of the multiplicative approximation shift process to achieve multiplication approximation is as follows: for an integer type binary number x, its mathematical representation is x = sign × 2 W-LO-1 ×M, where sign represents the sign bit, W represents the quantization bit width, LO represents the number of leading zeros, and M represents the mantissa, whose value is located in the interval [1,2]. The multiplication of two binary numbers is represented as: x×y=XOR(S x ,S y )×2 (Wx+Wy-(LOx+LOy)-2) ×(M x ×M y ); Among them, S x and S y These represent the sign bits of x and y, respectively. When two numbers are multiplied, the sign bit of the product is equivalent to the XOR of the sign bits of the two multipliers. x and W y These represent the quantization bit widths of x and y, respectively; LOx and LOy represent the number of leading zeros in x and y, respectively; M x and M y Let x and y represent the mantissas, respectively. Since the mantissas are between [1, 2], we approximate the product of the two numbers by shifting the leading zeros of x and y, i.e.: x×y≈XOR(S x ,S y )×2 (Wx+Wy-(LOx+LOy)-2) ×M x ; Based on the query matrix and the key matrix, the prediction result of the attention matrix is ​​determined; the prediction result of the attention matrix is ​​used to characterize the degree of matching of context terms.

2. The method according to claim 1, characterized in that, The step of determining the prediction result of the attention matrix based on the query matrix and the key matrix includes: The third matrix of the neural network model is subjected to leading zero counting to obtain the second leading zero count result corresponding to the third matrix; the query matrix is ​​the third matrix, and the transpose of the key matrix is ​​the fourth matrix; or the transpose of the key matrix is ​​the third matrix, and the query matrix is ​​the fourth matrix. Based on the second leading zero count result, the fourth matrix is ​​subjected to a multiplication approximation shift process to obtain the prediction result of the attention matrix.

3. The method according to claim 1, characterized in that, The first matrix is ​​the attention weight matrix, and obtaining the first leading zero count result obtained by performing leading zero counting on the first matrix of the neural network model includes: The attention weight matrix of the neural network model is subjected to leading zero counting to obtain the first leading zero count result corresponding to the attention weight matrix; Store the first leading zero count result into the target storage unit; When estimating the attention matrix, the first leading zero count result is read from the target storage unit.

4. The method according to claim 2, characterized in that, The second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix. The step of performing a multiplicative approximation shift on the fourth matrix based on the second leading zero count result to obtain the prediction result of the attention matrix includes: For each set of multiplication elements in the process of multiplying the query matrix and the transpose of the key matrix, the shift object element corresponding to the second element in the set of multiplication elements is determined according to the first element in the set of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix; Based on the number of leading zeros corresponding to the first element, the shift object element is shifted to obtain the first shift result corresponding to the group multiplication element; Extend the sign bit of the first shift result to obtain the product corresponding to the elements of the group being multiplied. The prediction result of the attention matrix is ​​determined based on the product of the elements of each group.

5. The method according to claim 4, characterized in that, The step of determining the shift object element corresponding to the second element in the group of multiplied elements based on the first element in the group of multiplied elements includes: If the first element in the multiplied group is negative, then the original code of the second element in the multiplied group is inverted bit by bit to obtain the shifted object element corresponding to the second element. If the first element in the multiplied group is a positive number, then the original code of the second element in the multiplied group is used as the shift object element corresponding to the second element.

6. The method according to claim 4, characterized in that, The step of shifting the shifted object element based on the number of leading zeros corresponding to the first element to obtain the first shift result corresponding to the grouped multiplication elements includes: Based on the sign bit of the first element and the sign bit of the second element, determine the sign bit of the first shift result corresponding to the multiplied elements of the group; The number of shift bits corresponding to the shift target element is determined based on the bit width of the second element and the number of leading zeros corresponding to the first element; The elements to be shifted are shifted to the left according to the number of shifts to obtain the effective bits of the first shift result corresponding to the elements of the group multiplication; The sign bit and the valid bit of the first shift result are used to form the first shift result corresponding to the group of multiplied elements.

7. The method according to claim 2, characterized in that, The second leading zero count result includes the number of leading zeros corresponding to each first element in the third matrix. The step of performing a multiplicative approximation shift on the fourth matrix based on the second leading zero count result to obtain the prediction result of the attention matrix includes: For each set of multiplication elements in the process of multiplying the query matrix and the transpose of the key matrix, based on the number of leading zeros corresponding to the first element in the set of multiplication elements, the second element in the set of multiplication elements is shifted to obtain the second shift result corresponding to the set of multiplication elements; the first element is an element in the third matrix, and the second element is an element in the fourth matrix; Based on the second shift result corresponding to each of the multiplication elements of the group, the original code corresponding to each attention element in the prediction result of the attention matrix is ​​determined; The original code corresponding to each attention element is decoded to obtain the prediction result of the attention matrix.

8. A low-complexity Transformer attention module prediction device, characterized in that, The device includes: The acquisition module is used to acquire the first leading zero count result obtained by performing leading zero counting processing on the first matrix of the neural network model; the attention weight matrix of the neural network model is the first matrix, and the word matrix input to the neural network model is the second matrix; or the word matrix input to the neural network model is the first matrix, and the attention weight matrix of the neural network model is the second matrix; The processing module is used to perform a multiplication approximation shift process on the second matrix based on the first leading zero count result to obtain a query matrix and a key matrix; wherein, the principle of the multiplication approximation shift process to achieve multiplication approximation is: for an integer type binary number x, its mathematical representation is x = sign × 2 W-LO-1 ×M, where sign represents the sign bit, W represents the quantization bit width, LO represents the number of leading zeros, and M represents the mantissa, whose value is located in the interval [1,2]. The multiplication of two binary numbers is represented as: x×y=XOR(S x ,S y )×2 (Wx+Wy-(LOx+LOy)-2) ×(M x ×M y ); Among them, S x and S y These represent the sign bits of x and y, respectively. When two numbers are multiplied, the sign bit of the product is equivalent to the XOR of the sign bits of the two multipliers. x and W y These represent the quantization bit widths of x and y, respectively; LOx and LOy represent the number of leading zeros in x and y, respectively; M x and M y Let x and y represent the mantissas, respectively. Since the mantissas are between [1, 2], we approximate the product of the two numbers by shifting the leading zeros of x and y, i.e.: x×y≈XOR(S x ,S y )×2 (Wx+Wy-(LOx+LOy)-2) ×M x ; The determination module is used to determine the prediction result of the attention matrix based on the query matrix and the key matrix; the prediction result of the attention matrix is ​​used to characterize the degree of matching of context lexical units.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images