Data processing method, processor and computer equipment
By filtering and normalizing the similarity matrix of word vector sequences, the calculation process of the self-attention mechanism is simplified, the problems of large calculation volume and high delay are solved, and efficiency and accuracy are improved.
Patent Information
- Application Number
- CN202411028350.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
AI Technical Summary
The self-attention mechanism has a large amount of computing and high latency in artificial intelligence, resulting in inefficiency.
The similarity matrix is determined by querying matrix and key matrix based on word vector sequence, filtering the target element and zeroing other elements, and normalizing to obtain the attention weight matrix, thereby simplifying the calculation of the self-attention force matrix.
The sparse calculation of the self-attention mechanism is realized, which reduces the amount of effective calculation, reduces the processing delay, and improves the calculation accuracy.
Smart Images

Figure CN118965021A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to but is not limited to the field of artificial intelligence, and in particular to a data processing method, a processor, and a computer device. Background Art
[0002] The Self-Attention Mechanism is an important technology in the field of artificial intelligence. It allows the model to consider the relationship between each element and all other elements in a sequence when processing a sequence. This mechanism helps the model better understand the contextual information in the sequence by calculating the correlation (or weight) between each element in the sequence and other elements, thereby processing sequence data more accurately. However, the processing flow of the self-attention mechanism in related technologies has a large amount of calculation and high processing delay. Summary of the invention
[0003] In view of this, embodiments of the present application at least provide a data processing method, a processor, and a computer device.
[0004] The technical solution of the embodiment of the present application is implemented as follows:
[0005] The present application provides a data processing method, including:
[0006] Determine a first similarity matrix based on the query matrix and the key matrix corresponding to the word vector sequence;
[0007] Filter out target elements whose values meet the conditions from each row of the first similarity matrix, and set other elements in each row except the target elements to zero, to obtain a second similarity matrix;
[0008] Normalizing the second similarity matrix to obtain an attention weight matrix;
[0009] Based on the attention weight matrix and the value matrix corresponding to the word vector sequence, determine the self-attention matrix corresponding to the word vector sequence.
[0010] An embodiment of the present application provides a processor, including:
[0011] A first determination module, configured to determine a first similarity matrix based on a query matrix and a key matrix corresponding to the word vector sequence;
[0012] A selection module, used for selecting target elements whose values meet the conditions from each row of the first similarity matrix, and setting other elements in each row except the target elements to zero, so as to obtain a second similarity matrix;
[0013] A normalization module, used for normalizing the second similarity matrix to obtain an attention weight matrix;
[0014] The second determination module is used to determine the self-attention matrix corresponding to the word vector sequence based on the attention weight matrix and the value matrix corresponding to the word vector sequence.
[0015] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above-mentioned data processing method when executing the program; or, the processor is the processor described in the above-mentioned embodiment.
[0016] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0018] Figure 1 A schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application;
[0019] Figure 2 A schematic diagram of the structure of a processor provided in an embodiment of the present application Figure 1 ;
[0020] Figure 3 A schematic diagram of the composition structure of a selection module provided in an embodiment of the present application Figure 2 ;
[0021] Figure 4 A schematic diagram of the composition structure of a selection module provided in an embodiment of the present application Figure 3 ;
[0022] Figure 5 A schematic diagram of the composition structure of a selection module provided in an embodiment of the present application Figure 4 ;
[0023] Figure 6 A schematic diagram of the composition structure of a selection module provided in an embodiment of the present application Figure 5 ;
[0024] Figure 7 This is a schematic diagram of the calculation process of a self-attention mechanism in related technologies. Figure 1 ;
[0025] Figure 8 This is a schematic diagram of the calculation process of a self-attention mechanism in related technologies. Figure 2 ;
[0026] Fig. 9A schematic diagram of a calculation flow of a self-attention mechanism provided in an embodiment of the present application;
[0027] Fig.10 A schematic diagram of the hardware architecture of a Top_K sel module provided in an embodiment of the present application;
[0028] Fig.11 A schematic diagram of the implementation flow of a Top_K element selection method provided in an embodiment of the present application;
[0029] Fig.12 A schematic diagram of a hardware entity of a computer device in an embodiment of the present application. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0032] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application. In addition, the technical features involved in the different embodiments and different implementations of this application described below can be combined with each other as long as they do not conflict with each other.
[0034] In order to better understand the solution of the embodiment of the present application, the self-attention mechanism in the related art is first explained below.
[0035] In the self-attention mechanism, the query matrix (Query, denoted as Q matrix), the key matrix (Key, denoted as K matrix) and the value matrix (Value, denoted as V matrix) are three key components, which represent the representation of the input sequence in different dimensions respectively.
[0036] The Q matrix is the query information extracted from the input sequence. Each row in the Q matrix represents a word / word vector (Token) in the input sequence, which is used to evaluate the correlation between the word / Token currently being processed and other words / Tokens.
[0037] The K matrix is the key information extracted from the input sequence. The K matrix is a label or description of all words / tokens in the input sequence. It is similar to what is used to match when searching for related words / tokens. In the attention mechanism, the relationship between the Q matrix and the K matrix can be used to determine the correlation between different words / tokens;
[0038] The V matrix is the value information extracted from the input sequence. Each row in the V matrix is a value vector, which is also used to represent a word / word vector (Token) in the input sequence, usually learned through a neural network. After using the Q matrix and K matrix to evaluate the correlation between different words / Tokens, these correlations and the value vector of the current word can be used to calculate the final representation of the current word. The value vectors are weighted to represent the meaning or importance of the current word.
[0039] In the self-attention mechanism, the K matrix, Q matrix, and V matrix are all generated by different linear transformations of the input sequence. For example, the K matrix, Q matrix, and V matrix can be obtained by mapping the input sequence through the neural network layer. These matrices are used to measure the correlation between the elements in the input sequence during the attention calculation process, thereby realizing the weighted summation of the input sequence. The attention mechanism calculates the dot product of the Q matrix and the K matrix to obtain the similarity between each element in the input sequence (this similarity is also called the attention score, or dependency), and then normalizes the similarity through a normalization function (such as the softmax function) to obtain the attention weights between each element in the input sequence. Finally, these attention weights are dot-producted with the V matrix to obtain the weighted output sequence (i.e., the self-attention matrix).
[0040] Take the self-attention mechanism in the Transformer model as an example. In the Transformer model of related technologies, a simple encoder often introduces a delay of about 100 microseconds, which is about 10 times the delay of a typical convolutional neural network (CNN) model. Different Transformer models are usually stacked with multiple encoders and decoders of similar size. Of such a high delay, about 60% of the time consumption is concentrated in the processing flow of the self-attention mechanism.
[0041] In response to the high computational latency problem of the self-attention mechanism mentioned above, there are some research attempts to optimize the self-attention mechanism, such as sparse attention and attention approximation. Among them, Top_K candidate index is a high-precision self-attention mechanism optimization solution.
[0042] The inspiration of Top_K candidate index comes from the high redundancy of human language, that is, unimportant tokens in the text, such as prepositions, articles, adverbs and other structural and meaningless tokens, can be safely deleted with little impact on the results of text understanding.
[0043] For the Q matrix, K matrix and V matrix corresponding to the input sequence of the self-attention mechanism, the attention weight of each token to other tokens can be calculated by the following formula (1-1):
[0044]
[0045] Among them, Attention(Q,K,V) is the attention weight matrix, which is obtained by multiplying the Q matrix by the transpose K of the K matrix. T After the scale transformation, the softmax() normalization is performed to obtain the attention weight matrix Attention(Q,K,V), d k is the number of columns of matrix K. The attention weight matrix Attention(Q,K,V) can include the attention weight of each Token for other Tokens.
[0046] The Top_K candidate index selects the largest values of each Token’s attention weights for other Tokens, that is, it calculates the corresponding row sequences of the K matrix and deletes other values with smaller attention weights, which greatly reduces the effective amount of calculation while ensuring the calculation accuracy.
[0047] There are two implementation schemes for the hardware accelerator design of the Top_K candidate index:
[0048] Implementation plan 1: Low bit width quantization Top_K selection;
[0049] An ultra-low bit width such as 4 bits or 2 bits is used to pre-complete the attention weight matrix calculation in the self-attention mechanism, and the K row sequences with the highest attention weights in the K matrix (i.e., Top-K row sequences) are selected, and the row indices corresponding to the Top-K row sequences (i.e., Top_K candidate indexes) are recorded. Then, a higher precision (e.g., 16 bits or 8 bits) bit width is used to recalculate the row sequence corresponding to the Top_K candidate index in the self-attention mechanism, and other row sequences are set to zero, thereby completing the fast selection and calculation of the Top_K candidate index and realizing the sparse calculation of the self-attention mechanism.
[0050] However, in the above implementation scheme 1, the calculation accuracy of the self-attention mechanism with low bit width is poor, resulting in a large selection error, and after selecting the Top_K sequence, the attention weight matrix needs to be recalculated, and the calculation process is redundant.
[0051] Implementation plan 2:
[0052] The attention weight matrix calculation in the self-attention mechanism is completed in advance, and the Top-K candidate index is selected through the bubble algorithm after the normalization calculation layer. Then, according to the selected Top-K candidate index, the Top-K row sequence in the K matrix is re-addressed, and other row sequences are set to 0, and the attention weight matrix in the self-attention mechanism is recalculated. Compared with implementation scheme 1, the above implementation scheme 2 has less precision loss and simpler logic.
[0053] However, in the above implementation scheme 2, the time complexity of the selection process of Top_K candidate indexes is high, which is O(n 2 ), after selecting the Top_K sequence, the attention weight matrix needs to be recalculated, and the calculation process is redundant.
[0054] On this basis, an embodiment of the present application provides a data processing method, which can be executed by a processor of a computer device. For example, the computer device may include but is not limited to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, and / or a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other devices with data processing capabilities. The processor may include but is not limited to at least one of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA).
[0055] Figure 1 A schematic diagram of the implementation flow of a data processing method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps S101 to S104:
[0056] Step S101, determining a first similarity matrix based on a query matrix and a key matrix corresponding to a word vector sequence.
[0057] Here, the word vector sequence refers to a set of word vector representations to be processed by the self-attention mechanism. In implementation, the word vector sequence can be the input sequence of the self-attention mechanism.
[0058] For example, the input text unit sequence can be encoded to obtain a word vector sequence corresponding to the text unit sequence, and the word vector sequence can include a word vector representation corresponding to each text unit in the text unit sequence. The text unit can include but is not limited to a word, a single character, a phrase, etc.
[0059] For another example, an input image may be encoded to obtain a word vector sequence corresponding to the image, and the word vector sequence may include word vector representations corresponding to at least one region of the image.
[0060] The query matrix and key matrix corresponding to the word vector sequence can be generated by different linear transformations of the word vector sequence. For example, the query matrix and key matrix can be obtained by mapping the word vector sequence through different neural network layers.
[0061] The first similarity matrix can represent the similarity between each word vector in the word vector sequence. In implementation, those skilled in the art can determine the first similarity matrix based on the query matrix and the key matrix in any appropriate manner according to actual conditions, and the embodiments of the present application are not limited to this.
[0062] In some embodiments, each row in the query matrix represents a word vector in the word vector sequence, and each row in the key matrix represents a word vector in the word vector sequence. The key matrix can be transposed to obtain a transposed key matrix, and then the first similarity matrix is determined based on the product of the query matrix and the transposed key matrix. For example, the product of the query matrix and the transposed key matrix can be determined as the first similarity matrix. For another example, the first similarity matrix can be obtained after scaling the product of the query matrix and the transposed key matrix.
[0063] In some embodiments, each row in the query matrix represents a word vector in the word vector sequence, and each column in the key matrix represents a word vector in the word vector sequence. A first similarity matrix can be obtained based on the product of the query matrix and the key matrix. For example, the product of the query matrix and the key matrix can be determined as the first similarity matrix. For another example, the first similarity matrix can be obtained after scaling the product of the query matrix and the key matrix.
[0064] Step S102: Filter out target elements whose values meet conditions from each row of the first similarity matrix, and set other elements in each row except the target elements to zero, so as to obtain a second similarity matrix.
[0065] Here, a row of elements in the first similarity matrix can represent the similarity between a word vector in the word vector sequence and each word vector in the word vector sequence.
[0066] The target element in a row of elements of the first similarity matrix may include elements whose values in the row meet the conditions. During implementation, those skilled in the art may use any appropriate screening condition to screen out at least one target element from each row of the first similarity matrix according to actual conditions, and the embodiments of the present application are not limited to this.
[0067] In some implementations, a Top-K screening method may be used to screen out at least one element with the largest value in each row of the first similarity matrix as a target element.
[0068] In some embodiments, at least one element having a value greater than a preset threshold is selected from each row of the first similarity matrix as a target element. During implementation, the preset threshold may be set by a person skilled in the art according to actual conditions, and the present application embodiment does not limit this.
[0069] Step S103, normalizing the second similarity matrix to obtain an attention weight matrix.
[0070] Here, any suitable normalization method may be used to normalize the second similarity matrix, and the embodiment of the present application is not limited thereto.
[0071] For example, a linear normalization function, a standard score (Z-Score) normalization function, and / or a normalized exponential (Softmax) function may be used to perform normalization processing on the second similarity matrix.
[0072] Step S104, based on the attention weight matrix and the value matrix corresponding to the word vector sequence, determine the self-attention matrix corresponding to the word vector sequence.
[0073] Here, the value matrix corresponding to the word vector sequence can be generated by performing a linear transformation on the word vector sequence. For example, the value matrix can be obtained by mapping the word vector sequence through a neural network layer.
[0074] During implementation, those skilled in the art may determine the self-attention matrix corresponding to the word vector sequence based on the attention weight matrix and the value matrix in any suitable manner according to actual conditions, and the embodiments of the present application are not limited to this.
[0075] In some embodiments, the self-attention matrix corresponding to the word vector sequence can be determined based on the product of the attention weight matrix and the value matrix. For example, the product of the attention weight matrix and the value matrix can be determined as the self-attention matrix corresponding to the word vector sequence. For another example, the product of the attention weight matrix and the value matrix can be scaled to obtain the self-attention matrix corresponding to the word vector sequence.
[0076] In an embodiment of the present application, after determining the first similarity matrix based on the query matrix and the key matrix corresponding to the word vector sequence, the selection logic of the target element is embedded, and the elements other than the target element in each row of the first similarity matrix are set to zero to obtain a second similarity matrix; then the second similarity matrix is normalized to obtain an attention weight matrix, and based on the attention weight matrix and the value matrix corresponding to the word vector sequence, the self-attention matrix corresponding to the word vector sequence is determined. In this way, the selection of the target element and the calculation process of the attention weight matrix can be realized through the calculation process of a single self-attention mechanism, thereby realizing the sparse calculation of the self-attention mechanism, greatly simplifying the calculation process, reducing the effective amount of calculation, and reducing the processing delay of the self-attention mechanism. In addition, the attention weight matrix can satisfy the normalized distribution, and the calculated value corresponding to the target element in the attention weight matrix is not affected by the zeroing of other elements, thereby improving the calculation accuracy of the self-attention matrix.
[0077] In some embodiments, the above step S102 may include the following steps S111 to S112:
[0078] Step S111: for each row in the first similarity matrix, select K elements with the largest values from the row as target elements, where K is a positive integer and is less than the total number of elements in the row;
[0079] Step S112: setting all elements except the target element in each row of the first similarity matrix to zero to obtain a second similarity matrix.
[0080] During implementation, K can be set according to actual conditions, and the embodiments of the present application are not limited to this.
[0081] In this way, in the process of selecting the target element, the selection of Top_K elements can be realized, so that the determined target element is more important than other elements, thereby obtaining a more accurate self-attention matrix.
[0082] In some embodiments, selecting K elements with the largest values from the row as target elements in the above step S111 may include the following steps S121 to S122:
[0083] Step S121, traverse each element in the row in turn, and perform the following operations on the i-th element currently accessed during the traversal process: when i is less than K+1, store the i-th element in the candidate element queue; when i is greater than K, compare the i-th element with the minimum element in the candidate element queue, and when the i-th element is greater than the minimum element in the candidate element queue, replace the minimum element in the candidate element queue with the i-th element;
[0084] Step S122: determine each element in the candidate element queue as a target element.
[0085] In the above embodiment, for the first K elements of each row in the first similarity matrix, the element is directly stored in the candidate element queue, and for the elements after the Kth element in each row, the element is compared with the minimum element in the candidate element queue, and if the element is greater than the minimum element, the minimum element in the candidate element queue is replaced by the element. In this way, the process of selecting the target element can be simplified, the selection efficiency can be improved, and thus the calculation efficiency of the self-attention matrix can be improved.
[0086] In some embodiments, the above method may further include the following step S131:
[0087] Step S131, when the smallest element in the candidate element queue is swapped out, temporarily storing the swapped out smallest element as the latest first temporarily stored element.
[0088] Here, any suitable storage method can be used to temporarily store the swapped out minimum element. The first temporary storage element can be stored in a storage space such as a memory, a register, and / or a disk. For example, the swapped out minimum element can be temporarily stored in a first register as the latest first temporary storage element. For another example, the swapped out minimum element can be temporarily stored in a first address in a memory as the latest first temporary storage element. Both the first register and the first address can be preset.
[0089] It is understandable that, in the process of temporarily storing the replaced minimum element as the latest first temporarily stored element, if the first temporarily stored element has been temporarily stored in the history, the first temporarily stored element temporarily stored in the history may be removed or overwritten by the latest first temporarily stored element.
[0090] The step S121 described above of comparing the i-th element with the minimum element in the candidate element queue may include the following step S132:
[0091] In the case that the i-th element is greater than the first temporarily stored element, the i-th element is compared with the current minimum element in the candidate element queue.
[0092] In some implementations, when the i-th element is less than or equal to the first temporarily stored element, the i-th element is not compared with the current minimum element in the candidate element queue, and the next element in the current row can be directly accessed.
[0093] In the above embodiment, the smallest element replaced in the candidate element queue is temporarily stored as the first temporary element, and when the currently traversed element is larger than the first temporary element, the element is compared with the current smallest element in the candidate element queue to determine whether the current smallest element in the candidate element queue needs to be replaced with the element. In this way, since the first temporary element is the smallest element replaced in the candidate element queue, if the currently accessed element is smaller than or equal to the first temporary element, the element must be smaller than the current smallest element in the candidate element queue. Therefore, only when the currently accessed element is larger than the first temporary element, the element is compared with the current smallest element in the candidate element queue, which can reduce the number of comparisons and improve the efficiency of selecting the target element.
[0094] In some embodiments, when i is less than K+1, storing the i-th element in the candidate element queue in the above step S121 may include the following steps S141 to S143:
[0095] Step S141, when i is greater than 1 and less than K+1, compare the i-th element with the current minimum element in the candidate element queue;
[0096] Step S142, when the i-th element is greater than the current minimum element in the candidate element queue, temporarily storing the current minimum element as the latest first temporary element, and storing the i-th element in the candidate element queue;
[0097] Step S143, when the i-th element is less than or equal to the current minimum element in the candidate element queue, temporarily store the i-th element as the latest first temporary element, and store the i-th element in the candidate element queue.
[0098] In this way, when traversing to the K+1th element, the latest first temporarily stored element used for comparison with the K+1th element is already the current minimum element in the candidate element queue. Therefore, the number of subsequent comparisons of the K+1th element with the current minimum element in the candidate element queue can be further reduced, so as to further improve the efficiency of selecting the target element.
[0099] In some embodiments, the candidate elements in the candidate element queue are arranged in descending order. In the case where the i-th element is greater than the smallest element in the candidate element queue as described in the above step S121, the smallest element in the candidate element queue is replaced with the i-th element, which may include the following steps S151 to S152:
[0100] Step S151, temporarily storing the i-th element as the latest second temporarily stored element.
[0101] Here, any suitable storage method can be used to temporarily store the second temporary storage element. The second temporary storage element can be stored in a storage space such as a memory, a register, and / or a disk. For example, the i-th element currently accessed can be temporarily stored in the second register as the latest second temporary storage element. For another example, the i-th element currently accessed can be temporarily stored in the second address in the memory as the latest second temporary storage element. Both the second register and the second address can be preset.
[0102] It is understandable that, in the process of temporarily storing the currently accessed i-th element as the latest second temporary storage element, if the second temporary storage element has been temporarily stored in the history, the second temporary storage element temporarily stored in the history may be removed or overwritten by the latest second temporary storage element.
[0103] Step S152, sequentially taking out each candidate element in the candidate element queue, comparing the taken out candidate element with the second temporarily stored element, storing the larger one in the candidate element queue, temporarily storing the smaller one as the latest second temporarily stored element, and temporarily storing the smaller one as the latest first temporarily stored element.
[0104] It can be understood that the retrieval and storage of elements in the candidate element queue follow the first-in-first-out principle, that is, the candidate element taken out from the candidate element queue each time is the current first element in the candidate element queue, and the new element stored in the candidate element queue is stored at the tail of the candidate element queue.
[0105] In this way, through two-level temporary storage, on the one hand, the first temporary storage element used for comparison with the currently accessed element each time during the traversal process can be one of the visited elements whose size is second only to the current smallest element in the candidate element queue, thereby minimizing the number of subsequent comparisons with the current smallest element in the candidate element queue and improving the efficiency of selecting the target element; on the other hand, the candidate elements in the candidate element queue can always be arranged in order from large to small, thereby accurately and efficiently determining the smallest element in the candidate element queue.
[0106] In some embodiments, the above step S101 may include the following steps S161 to S162:
[0107] Step S161, performing matrix multiplication operation on the query matrix and the transposed key matrix to obtain an intermediate matrix.
[0108] Step S162: Based on the number of columns of the key matrix, scale the intermediate matrix to obtain the first similarity matrix.
[0109] In some implementations, the number of columns of the key matrix may be used as a transformation scale, and the intermediate matrix may be divided by the transformation scale to obtain a first similarity matrix.
[0110] In some implementations, the square root of the number of columns of the key matrix may be used as a transformation scale, and the intermediate matrix may be divided by the transformation scale to obtain a first similarity matrix.
[0111] In the above embodiment, after performing matrix multiplication on the query matrix and the transposed key matrix, the result of the matrix multiplication is scaled, so that the value of each element in the first similarity matrix can be within an appropriate range, thereby reducing the problem of excessive element values caused by matrix multiplication.
[0112] In some embodiments, before the above step S101, the method may further include the following steps S171 to S172:
[0113] Step S171, obtaining the word vector sequence corresponding to the text to be processed.
[0114] Step S172, mapping the word vector sequence into the query matrix, the key matrix and the value matrix.
[0115] Here, the text to be processed may be any appropriate text to be processed for semantic understanding.
[0116] In some embodiments, the first weight matrix can be used to perform a dot product operation with the word vector sequence, and then a first bias item is added to obtain a query matrix of the word vector sequence; the second weight matrix can be used to perform a dot product operation with the word vector sequence, and then a second bias item is added to obtain a key matrix of the word vector sequence; the third weight matrix can be used to perform a dot product operation with the word vector sequence, and then a third bias item is added to obtain a value matrix of the word vector sequence. Among them, the first weight matrix, the first bias item, the second weight matrix, the second bias item, the third weight matrix, and the third bias item can all be obtained through model training and learning.
[0117] The present application embodiment provides a processor, such as Figure 2 As shown, the processor 20 includes a first determination module 21, a selection module 22, a normalization module 23 and a second determination module 24, wherein:
[0118] A first determination module 21, configured to determine a first similarity matrix based on a query matrix and a key matrix corresponding to a word vector sequence;
[0119] A selection module 22 is used to select a target element whose value satisfies a condition from each row of the first similarity matrix, and set other elements in each row except the target element to zero, so as to obtain a second similarity matrix;
[0120] A normalization module 23, used for normalizing the second similarity matrix to obtain an attention weight matrix;
[0121] The second determination module 24 is used to determine the self-attention matrix corresponding to the word vector sequence based on the attention weight matrix and the value matrix corresponding to the word vector sequence.
[0122] The processor provided in the embodiment of the present application can realize the selection of the target element and the calculation process of the attention weight matrix through the calculation process of a single self-attention mechanism, thereby realizing the sparse calculation of the self-attention mechanism, greatly simplifying the calculation process, reducing the effective amount of calculation, and reducing the processing delay of the self-attention mechanism. In addition, the attention weight matrix can satisfy the normalized distribution, and the calculated value corresponding to the target element in the attention weight matrix is not affected by the zeroing of other elements, thereby improving the calculation accuracy of the self-attention matrix.
[0123] In some embodiments, Figure 3 As shown, the selection module 22 includes a selection submodule 221 and a clearing submodule 222, wherein:
[0124] A selection submodule 221 is used for selecting K elements with the largest values from each row in the first similarity matrix as target elements, where K is a positive integer and is less than the total number of elements in the row;
[0125] The zeroing submodule 222 is configured to set all elements except the target element in each row of the first similarity matrix to zero, so as to obtain a second similarity matrix.
[0126] In some embodiments, after selecting the target element in a row in the first similarity matrix, the selection submodule 221 can send the position of the selected target element in the row to the clearing submodule 222; the clearing submodule 222 can set the elements other than the target element in each row of the first similarity matrix to zero according to the position of the target element in the row, and finally output the second similarity matrix.
[0127] In the above embodiment, the selection module can implement the selection of Top_K elements in the process of selecting target elements, so that the determined target elements are more important than other elements, thereby obtaining a more accurate self-attention matrix.
[0128] In some embodiments, Figure 4 As shown, the selection submodule 221 includes a replacement unit 2211, a bypass circuit 2212 and a storage queue 2213;
[0129] The bypass circuit 2212 is used to bypass the replacement unit 2211 when it is turned on, so as to store the i-th element currently input in the current row into the storage queue 2213;
[0130] The replacement unit 2211 is used to compare the i-th element currently input in the current row with the minimum element in the storage queue 2213, and when the i-th element is greater than the minimum element in the storage queue 2213, replace the minimum element in the storage queue 2213 with the i-th element;
[0131] In which, when i is less than K+1, the bypass circuit 2212 is turned on, and when i is greater than K, the bypass circuit 2212 is turned off to enable the replacement unit 2211.
[0132] During implementation, the bypass circuit may be implemented in any suitable manner, and the embodiments of the present application are not limited thereto.
[0133] In some embodiments, the bypass circuit 2212 may include a switch element connected in parallel with the replacement unit. When the switch element is closed, the bypass circuit 2212 is turned on and the replacement unit 2211 is bypassed; when the switch element is opened, the bypass circuit 2212 is disconnected and the replacement unit 2211 is enabled.
[0134] In the above embodiment, the activation or deactivation of the replacement unit is controlled by turning on and off the bypass circuit, which can simplify the hardware implementation of the selection submodule and improve the selection efficiency, thereby improving the calculation efficiency of the self-attention matrix.
[0135] In some embodiments, Figure 5 As shown, the replacement unit 2211 may include a first register Reg1, a comparator Comp and a replacement subunit Rep_Unit.
[0136] The comparator Comp has a first input terminal In1 , a second input terminal In2 , and a first output terminal O1 .
[0137] The first input terminal In1 is used to receive the i-th element, the second input terminal In2 is used to receive the first temporary element output by the first register Reg1, and the first output terminal O1 is used to output the i-th element to the replacement subunit Rep_Unit when the i-th element is greater than the first temporary element.
[0138] The replacement subunit Rep_Unit is used to compare the i-th element with the minimum element in the storage queue 2213, and when the i-th element is greater than the minimum element in the storage queue 2213, replace the minimum element in the storage queue 2213 with the i-th element, and temporarily store the replaced minimum element in the first register Reg1 as the latest first temporary element.
[0139] It should be noted that the storage queue 2213 may correspond to the candidate element queue in the aforementioned embodiment, and the elements stored in the storage queue 2213 are the candidate elements in the aforementioned embodiment.
[0140] During implementation, the elements stored in the storage queue 2213 may be arranged in a specific order or may not be arranged in a specific order, and this embodiment of the present application is not limited to this.
[0141] In some embodiments, the elements stored in the storage queue 2213 may be arranged in a set order. In the process of replacing the minimum element in the storage queue 2213 with the i-th element, the elements in the storage queue 2213 may also be sorted so that after the minimum element is replaced, the elements stored in the storage queue 2213 may still be arranged in the set order. For example, the set order may include, but is not limited to, an order from large to small or an order from small to large.
[0142] In the above embodiment, the first register is used to temporarily store the smallest element replaced in the candidate element queue, and when the currently traversed element is greater than the first temporarily stored element temporarily stored in the first register, the element is output to the replacement subunit, so that the replacement subunit compares the element with the current smallest element in the candidate element queue to determine whether the current smallest element in the candidate element queue needs to be replaced with the element. In this way, since the first temporarily stored element in the first register is the smallest element replaced in the candidate element queue, if the currently accessed element is less than or equal to the first temporarily stored element, the element must be less than the current smallest element in the candidate element queue. Therefore, the comparator outputs the element to the replacement subunit for comparison with the current smallest element in the candidate element queue only when the currently accessed element is greater than the first temporarily stored element, which can reduce the number of comparisons in the replacement subunit, thereby improving the efficiency of selecting the target element.
[0143] In some embodiments, Figure 6 As shown, the replacement subunit Rep_Unit includes a second register Reg2 and a selector Sel.
[0144] The first output terminal In1 is used to temporarily store the i-th element as the latest second temporary element in the second register Reg2.
[0145] The storage queue 2213 is used to output the stored candidate elements to the selector Sel in sequence.
[0146] The selector Sel is used to receive the candidate elements output by the storage queue 2213, and select the larger one v1 from the candidate elements and the second temporary elements stored in the second register Reg2 to store in the storage queue 2213, select the smaller one v2 as the latest second temporary element to be temporarily stored in the second register Reg2, and store the smaller one v2 as the latest first temporary element in the first register Reg1.
[0147] In this way, by implementing two-level temporary storage through the first register and the second register, the first temporary storage element stored in the first register during the traversal process can be one of the visited elements whose size is second only to the current smallest element in the candidate element queue. Therefore, the number of comparisons in the subsequent replacement sub-unit can be minimized, thereby improving the efficiency of selecting the target element.
[0148] The selector Sel may directly temporarily store the selected smaller one in the first register Reg1 , or may temporarily store the selected smaller one in the first register Reg1 through the second register Reg2 , which is not limited in the embodiment of the present application.
[0149] The following describes the application of the data processing method and processor provided in the embodiments of the present application in actual scenarios.
[0150] The calculation process of the typical self-attention mechanism is as follows Figure 7 As shown in the figure, first, the K matrix is transposed using the transposition module Transp; secondly, the matrix product module MatMul is used to perform matrix multiplication calculation on the Q matrix and the transposed K matrix to obtain the matrix QK T ; Then, use the scale transformation module Scale_Trans to transform the matrix QK T Perform a scale transformation to obtain the matrix Scaling can improve the problem of too large inner product after multiplying the Q matrix and the transposed K matrix; then, the normalization module Softmax is used to transform the matrix Normalization calculation is performed to obtain the attention weight matrix; finally, the matrix product module MatMul is used to perform matrix multiplication calculation on the attention weight matrix and the V matrix to obtain the self-attention matrix. In the calculation process of the self-attention mechanism, after the Q matrix and the transposed K matrix complete the matrix multiplication, the scale transformation and normalization calculation are performed to obtain the attention weight matrix.
[0151] In some related technologies, the self-attention mechanism introduces the process of selecting Top_K candidate indexes (such as the aforementioned implementation scheme 2). Figure 8 As shown in the processing flow of the self-attention mechanism based on Top_K candidate index selection, Figure 7 Based on the calculation process of the self-attention mechanism shown in FIG, after completing the normalization calculation, the Top_K selection module Top_K sel is used to select the largest K candidate indexes, and then the Top-K row sequence in the K matrix is re-addressed, and other row sequences are set to 0, and the attention weight matrix in the self-attention mechanism is recalculated.
[0152] The calculation process of Softmax is shown in the following formula 1-2:
[0153]
[0154] Among them, X i represents the i-th element in the row sequence X in the matrix to be normalized, softmax(X i ) indicates that i The corresponding normalized value, X j represents the jth element in the row sequence X, d k is the total number of elements in the row sequence X, that is, the number of columns of the matrix, i is less than or equal to d k A positive integer, X max is the maximum value in the row sequence X.
[0155] It can be seen that the calculation process of Softmax is a linear calculation process. Therefore, the results of Top_K candidate index selection before and after Softmax calculation are consistent. Using this feature, the embodiment of the present application provides a processing flow of a self-attention mechanism, such as Fig. 9 As shown in the figure, the Top_K sel module is processed before Softmax. Under the premise of not affecting the Top_K candidate results, it also ensures that the Softmax distribution sum after Top_K is still 1. There is no need to recalculate the self-attention weight matrix again, which greatly saves calculation steps. After the Scale_Trans module completes the calculation, the Top_K sel module uses the matrix The matrix rows in the matrix are taken as units, and the largest K values on each row are selected and set as valid values, and the other values are set to zero; after the Top_K sel module, the zeroed matrix is output to the Softmax module to complete other subsequent calculations of the self-attention mechanism.
[0156] Fig.10 A schematic diagram of the hardware architecture of a Top_K sel module provided in an embodiment of the present application is shown in FIG. Fig.10As shown in FIG. 1 , for each row in the matrix, each element in the row can be sequentially input into the Top_K sel module to select the K elements with the largest values (i.e., Top_K elements); The i-th element data_in(i) of the sel module is input and compared with the first temporary element in the first register min_reg by the comparator Comp, and when the input element is greater than the first temporary element, the input element is stored in the second register temp_reg; the comparator Comp can be bypassed by the bypass circuit bypass, and when the bypass circuit is turned on, the comparator Comp is bypassed, and the input element can be directly stored in the second register temp_reg; the selector Sel is used to select a larger value and a smaller value from the second temporary element in the second register temp_reg and the output element of the storage queue FIFO_TOP, the larger value output by the selector Sel continues to be cached in the storage queue FIFO_TOP, and the smaller value is temporarily stored in the second register temp_reg for continued comparison and selection with other elements in the storage queue FIFO_TOP, in addition, the smaller value temporarily stored in the second register temp_reg is also updated to the first register min_reg as the latest first temporary element; the storage queue FIFO_TOP is used to store the value of the Top_K element finally selected during the selection process and its position information in the corresponding row.
[0157] The Top_K sel module can perform sorting and selection according to the following steps S301 to S304:
[0158] Step S301, for the first input element, bypass the comparator Comp and the selector Sel using the bypass circuit, and directly cache the input element into the storage queue FIFO_TOP;
[0159] Step S302, for the second to K-th input elements, use the bypass circuit bypass comparator Comp to cache the input elements in the second register temp_reg, and at the same time take out the first candidate element in the storage queue FIFO_TOP and input it into the selector Sel, and select the larger value and the smaller value between the second temporary element and the input candidate element in the selector Sel, store the selected larger value in the storage queue FIFO_TOP, cache the selected smaller value in the second register temp_reg, and use the selected smaller value to update the first temporary element in the first register min_reg, until the polling of all candidate elements in the storage queue FIFO_TOP is completed, and then continue to input the next input element and sort and select. When each input element is input, it can be guaranteed that the data in the storage queue FIFO_TOP are arranged in order from large to small;
[0160] Step S303, for the K+1th input element and other input elements after the K+1th input element, first compare the input element with the first temporary element in the first register min_reg. If the input element is greater than the first temporary element in the first register min_reg, cache the input element in the second register temp_reg as the latest second temporary element. At the same time, take out the first candidate element in the storage queue FIFO_TOP, and select the larger value and the smaller value between the second temporary element and the input candidate element in the selector Sel, store the selected larger value in the storage queue FIFO_TOP, cache the selected smaller value in the second register temp_reg, and use the selected smaller value to update the first temporary element in the first register min_reg, until the polling of all candidate elements in the storage queue FIFO_TOP is completed, and then continue to input the next input element and perform sorting selection.
[0161] Step S304, after completing the Top_K selection and sorting of a row of elements in the matrix, the Top_K candidate elements cached in the storage queue FIFO_TOP are output to the clearing output module Out. The clearing output module Out sets the elements at other positions in the row to zero according to the position information of each candidate element in the Top_K candidate elements in the corresponding row, and finally outputs the row of elements after being set to zero.
[0162] Fig.11 The following is a schematic diagram of the implementation process of a Top_K element selection method provided in an embodiment of the present application. The method can be implemented by the Top_K sel module. Fig.11 As shown, the Top_K element selection method includes the following steps S401 to S424:
[0163] Step S401, input the element data_in(i);
[0164] Step S402, determine whether i ≤ K;
[0165] If yes, go to Step S403; if no, go to Step S415.
[0166] Where K is a positive integer and K is less than the total number of elements in a row of the matrix.
[0167] Step S403, temporarily store data_in(i) in temp_reg and assign 0 to j;
[0168] Here, the element stored in the second register temp_reg is the second temporarily stored element Tmp. Each time a new element is stored in temp_reg, the new element will replace the original Tmp as the latest Tmp.
[0169] Step S404, take out a candidate element from FIFO_TOP and assign it to A, and increment j by 1;
[0170] Here, the elements in the storage queue FIFO_TOP are taken out and stored following the first-in-first-out principle.
[0171] Step S405, determine whether A < Tmp or A is empty;
[0172] If no, go to Step S406; if yes, go to Step S410.
[0173] Step S406, store A in FIFO_TOP;
[0174] Step S407, determine whether j == i;
[0175] If no, go to Step S404; if yes, go to Step S408.
[0176] Step S408, store Tmp in FIFO_TOP;
[0177] Step S409, store Tmp in min_reg;
[0178] Here, the element stored in the first register min_reg is the first temporarily stored element M. Each time a new element is stored in min_reg, the new element will replace the original M as the latest M.
[0179] Step S410, store Tmp in FIFO_TOP;
[0180] Step S411, determine whether j == i;
[0181] If so, go to step S412; if not, go to step S414.
[0182] Step S412, store A into FIFO_TOP;
[0183] Step S413, store A into min_reg;
[0184] Step S414, temporarily storing A in temp_reg;
[0185] Here, A may be temporarily stored in the second register temp_reg as the latest second temporary element Tmp.
[0186] Step S415, determine whether data_in(i)>M;
[0187] If so, go to step S416; if not, go to step S424.
[0188] Step S416, temporarily store data_in(i) in temp_reg, and assign j to 0;
[0189] Step S417, take a candidate element from FIFO_TOP and assign it to A, and add 1 to j;
[0190] Step S418, determine whether A>Tmp;
[0191] If so, go to step S419; if not, go to step S421.
[0192] Step S419, store A into FIFO_TOP;
[0193] Step S420, temporarily storing A in temp_reg;
[0194] Step S421, store Tmp into FIFO_TOP;
[0195] Step S422, determine whether j==K;
[0196] If so, go to step S423; if not, go to step S420.
[0197] Step S423, store Tmp into min_reg;
[0198] Step S424, end.
[0199] It should be noted that, for each row of elements to be selected as Top_K elements, the process described in steps S401 to S424 can be executed for each element in the row until the polling of each element in the row is completed. Then, each candidate element stored in FIFO_TOP is the Top_K element (corresponding to the target element in the aforementioned embodiment).
[0200] In an embodiment of the present application, firstly, a fast Top_K element selection hardware architecture based on the element stored in the minimum register (i.e., the first register min_reg) as the threshold is designed and applied to the processing flow of the self-attention mechanism. Top_K screening is completed quickly under the premise of satisfying the Top_K selection accuracy; secondly, in the self-attention module, after matrix multiplication of the Q matrix and the K matrix, the Top_K selection logic is embedded. Under the premise of satisfying the normalized distribution of the Softmax processing and the unchanged calculated value corresponding to the target element in the attention weight matrix, the Top_K selection and the final calculation process steps can be merged, which greatly simplifies the calculation process and reduces the processing delay of the self-attention mechanism.
[0201] According to the embodiments of the present application, a faster Top_K search engine can be implemented, reducing the time complexity of Top_K selection to O(n*k); in addition, a unified Top_K selection and Softmax calculation process is adopted, and there is no need to repeat the matrix multiplication processing and Softmax calculation of the query matrix and the key matrix, which simplifies the processing flow and saves processing time.
[0202] The description of the above processor embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the processor provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the processor embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.
[0203] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.
[0204] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above-mentioned data processing method when executing the program; or, the processor is the processor described in the above-mentioned embodiment.
[0205] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.
[0206] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.
[0207] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.
[0208] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above method.
[0209] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.
[0210] It should be noted that Fig.12 A schematic diagram of a hardware entity of a computer device in an embodiment of the present application is shown in FIG. Fig.12 As shown, the hardware entity of the computer device 500 includes: a processor 501, a communication interface 502 and a memory 503, wherein: the processor 501 generally controls the overall operation of the computer device 500. The communication interface 502 can enable the computer device to communicate with other terminals or servers through a network. The memory 503 is configured to store instructions and applications executable by the processor 501, and can also cache data to be processed or processed by the processor 501 and each module in the computer device 500 (for example, image data, audio data, voice communication data and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be carried out between the processor 501, the communication interface 502 and the memory 503 through a bus 504.
[0211] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.
[0212] It should be noted that, in this article, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes one..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0213] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the embodiments of the present application may be all integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0214] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.
[0215] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0216] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A data processing method, comprising: Determine a first similarity matrix based on the query matrix and the key matrix corresponding to the word vector sequence; Filter out target elements whose values meet the conditions from each row of the first similarity matrix, and set other elements in each row except the target elements to zero, to obtain a second similarity matrix; Normalizing the second similarity matrix to obtain an attention weight matrix; Based on the attention weight matrix and the value matrix corresponding to the word vector sequence, determine the self-attention matrix corresponding to the word vector sequence.
2. The method according to claim 1, wherein the step of selecting a target element whose value satisfies a condition from each row of the first similarity matrix and setting other elements in each row except the target element to zero to obtain a second similarity matrix comprises: For each row in the first similarity matrix, select K elements with the largest values from the row as target elements, where K is a positive integer and K is less than the total number of elements in the row; The elements other than the target element in each row of the first similarity matrix are set to zero to obtain a second similarity matrix.
3. The method according to claim 2, wherein the step of selecting K elements with the largest values from the row as target elements comprises: Traverse each element in the row in turn, and perform the following operations on the i-th element currently accessed during the traversal process: when i is less than K+1, store the i-th element in the candidate element queue; when i is greater than K, compare the i-th element with the minimum element in the candidate element queue, and when the i-th element is greater than the minimum element in the candidate element queue, replace the minimum element in the candidate element queue with the i-th element; Each element in the candidate element queue is determined as a target element.
4. The method according to claim 3, further comprising: When the smallest element in the candidate element queue is swapped out, temporarily storing the swapped out smallest element as the latest first temporary storage element; The comparing the i-th element with the minimum element in the candidate element queue comprises: In the case that the i-th element is greater than the first temporarily stored element, the i-th element is compared with the current minimum element in the candidate element queue.
5. The method according to claim 4, wherein the candidate elements in the candidate element queue are arranged in descending order; When the i-th element is greater than the smallest element in the candidate element queue, replacing the smallest element in the candidate element queue with the i-th element comprises: Temporarily storing the i-th element as the latest second temporary storage element; Each candidate element in the candidate element queue is taken out in turn, the taken candidate element is compared with the second temporarily stored element, and the larger one is stored in the candidate element queue, the smaller one is temporarily stored as the latest second temporarily stored element, and the smaller one is temporarily stored as the latest first temporarily stored element.
6. A processor, comprising: A first determination module, configured to determine a first similarity matrix based on a query matrix and a key matrix corresponding to the word vector sequence; A selection module, used for selecting target elements whose values meet the conditions from each row of the first similarity matrix, and setting other elements in each row except the target elements to zero, so as to obtain a second similarity matrix; A normalization module, used for normalizing the second similarity matrix to obtain an attention weight matrix; The second determination module is used to determine the self-attention matrix corresponding to the word vector sequence based on the attention weight matrix and the value matrix corresponding to the word vector sequence.
7. The processor according to claim 6, wherein the selection module comprises: A selection submodule is used to select, for each row in the first similarity matrix, K elements with the largest values from the row as target elements, where K is a positive integer and is less than the total number of elements in the row; The zeroing submodule is used to set all elements except the target element in each row of the first similarity matrix to zero to obtain a second similarity matrix.
8. The processor according to claim 7, wherein the selection submodule comprises a replacement unit, a bypass circuit and a storage queue; The bypass circuit is used to bypass the replacement unit when it is turned on, so as to store the i-th element currently input in the row into the storage queue; The replacement unit is used to compare the i-th element currently input in the row with the minimum element in the storage queue, and replace the minimum element in the storage queue with the i-th element if the i-th element is greater than the minimum element in the storage queue; in, When i is less than K+1, the bypass circuit is turned on, and when i is greater than K, the bypass circuit is turned off to enable the replacement unit.
9. The processor according to claim 8, wherein the replacement unit comprises a first register, a comparator and a replacement subunit; The comparator has a first input terminal, a second input terminal and a first output terminal; The first input end is used to receive the i-th element, the second input end is used to receive the first temporary element output by the first register, and the first output end is used to output the i-th element to the replacement subunit when the i-th element is greater than the first temporary element; The replacement subunit is used to compare the i-th element with the minimum element in the storage queue, and when the i-th element is greater than the minimum element in the storage queue, replace the minimum element in the storage queue with the i-th element, and temporarily store the replaced minimum element in the first register as the latest first temporary element.
10. The processor according to claim 9, wherein the replacement subunit comprises a second register and a selector; The first output terminal is used to temporarily store the i-th element as the latest second temporary storage element in the second register; The storage queue is used to output the stored candidate elements to the selector in sequence; The selector is used to receive the candidate elements output by the storage queue, and select the larger one from the candidate elements and the second temporary storage elements stored in the second register to store in the storage queue, select the smaller one as the latest second temporary storage element to store in the second register, and store the smaller one as the latest first temporary storage element in the first register.
11. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps of the method according to any one of claims 1 to 5 when executing the program; or, the processor is as described in any one of claims 6 to 10.
Citation Information
Cited By
Data processing method and device, electronic equipment and storage medium
CN119992122A
