Large model self-attention optimization method, device, equipment and medium

Abstract

The invention provides a large model self-attention optimization method and device, equipment and a medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining a KQ large matrix obtained after an input sequence passes through a large model self-attention structure, and dividing the KQ large matrix into a plurality of KQ small matrixes; carrying out local attention calculation on each KQ small matrix, and carrying out similarity descending sorting on the KQ small matrixes and other KQ small matrixes to obtain a sorting list; and for each element in the input sequence, according to the calculated sorting list, selecting the local attention corresponding to the KQ small matrix with the similarity ranking in the first r to carry out weighted calculation, and taking the local attention as the attention representation of the current element until all the elements of the input sequence are completed, so as to obtain an attention result after the input sequence is optimized. According to the method, the calculation complexity of the model is reduced, remote context information can be captured, and the accuracy of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large-model self-attention optimization method, device, equipment and medium. Background Art

[0002] Large models have a wide range of applications in the field of deep learning. Large models have a large parameter scale and complex structure, and the training process requires a lot of computing resources and storage space. For example, in the standard self-attention mechanism, for each element (token) of the input sequence, it is necessary to interact with all other elements, so there are n×n computing units. In actual deployment, large models usually rely on NPU (neural processing unit) for reasoning. As a hardware specially optimized for deep learning reasoning tasks, NPU can accelerate the forward calculation of the model and improve the reasoning efficiency. However, due to the large parameter scale of large models, even during the reasoning process, the computing power of NPU may not be enough to meet the needs of efficient reasoning. This may lead to slower embedding calculations, longer reasoning time, and greater hardware resource consumption, thereby increasing the deployment cost of actual applications.

[0003] Existing large models are usually multi-head self-attention structures. In order to reduce the computational complexity of the self-attention mechanism, most of them adopt a local attention mechanism, that is, only consider elements within a fixed range or select elements based on content; or perform singular value decomposition SVD on the model parameter matrix to achieve data dimensionality reduction and compression, thereby optimizing the large model. However, the above methods still have high computational complexity, especially when processing large-scale data sets, which will increase the training time and computational cost of the model; or they are only limited to the local area and lack the ability to capture long-distance contextual information; or they choose to retain some singular values ​​and ignore other smaller singular values, resulting in local information loss, thereby affecting the accuracy and performance of the model.

[0004] In view of this, the applicant filed this application after studying the existing technology. Summary of the invention

[0005] The present invention aims to provide a large-model self-attention optimization method, device, equipment and medium to solve the shortcomings of existing methods such as high computational complexity and time-consuming, which affect the accuracy and performance of the model.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions: A large model self-attention optimization method, comprising: S1, obtain the KQ large matrix obtained after the input sequence passes through the large model self-attention structure; S2, divide the KQ large matrix into several The KQ small matrix; S3, perform local attention calculation on each KQ small matrix; and sort the similarity with other KQ small matrices in descending order to obtain a sorted list; S4, for each element in the input sequence, according to the calculated sorted list, select the local attention corresponding to the top r KQ small matrices in similarity for weighted calculation as the attention representation of the current element, until all the elements of the input sequence are completed, and the optimized attention result of the input sequence is obtained.

[0007] Preferably, the KQ large matrix is ​​a similarity matrix between the key matrix K generated by the large model self-attention structure and the query matrix Q.

[0008] Preferably, the similarity is obtained by calculating the dot product of the elements.

[0009] Preferably, the similarity is obtained by calculating the cosine similarity of the elements.

[0010] Preferably, the S4 is specifically: For each element in the input sequence, select the top r KQ small matrices with similarity according to the calculated sorted list to obtain a high similarity attention set; After performing weighted calculation of attention on each KQ small matrix in the high similarity attention set according to the preset weight, the softmax function is used for normalization to obtain the attention weight vector of the current element; The attention weight vector of the current element is weighted summed with the value matrix V generated by the self-attention structure of the large model to obtain the global attention representation of the current element.

[0011] The present invention also provides a large model self-attention optimization device, comprising: The acquisition unit is used to obtain the KQ large matrix obtained after the input sequence passes through the large model self-attention structure; A splitting unit is used to split the KQ large matrix into several The KQ small matrix; The similarity sorting unit is used to perform local attention calculation on each KQ small matrix and sort the similarity with other KQ small matrices in descending order to obtain a sorted list; The attention calculation unit is used to calculate the attention representation of each element; for each element in the input sequence, according to the calculated sorted list, the local attention corresponding to the top r KQ small matrices with similarity ranking is selected for weighted calculation as the attention representation of the current element, until all the elements of the input sequence are completed, and the optimized attention result of the input sequence is obtained.

[0012] The present invention also provides a large model self-attention optimization device, including a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a large model self-attention optimization method as described above.

[0013] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a large-model self-attention optimization method as described above is implemented.

[0014] In summary, compared with the prior art, the present invention has the following beneficial effects: When calculating the attention of the input sequence, the present invention divides the global large matrix into small matrices to reduce the computational complexity of calculating the global attention of each element, reduces the operations between attention matrices, and optimizes the computational complexity of the large model. At the same time, the present invention selectively performs weighted calculations with the long-distance matrix through the similarity sorting list to capture the long-distance context information, improve the accuracy of the attention output results, and thus improve the accuracy and performance of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A schematic diagram of a large model self-attention optimization method provided in Example 1.

[0017] Figure 2 A schematic diagram of the structure of a large-model self-attention optimization device provided in Example 2.

[0018] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0020] Embodiment 1 Embodiment 1 of the present invention provides a large model self-attention optimization method, which can be implemented by a large model self-attention optimization device (hereinafter referred to as the optimization device), and in particular, executed by one or more processors in the optimization device.

[0021] In this embodiment, the optimization device may be an electronic device equipped with a processor, the processor having a computer program of the large model self-attention optimization method and the computer program can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., which is not limited here.

[0022] In this embodiment, the large model can be used to process different types of data such as text data, image data, audio and video data, etc., to obtain processing results.

[0023] Q (Query): represents the query matrix, which represents the vector representation of the words or data points we want to understand.

[0024] K (Key): represents the key matrix, which represents the vector representation of all words or data points in the input sequence and is used for comparison with Q.

[0025] V (Value): represents the value matrix, which represents the vector representation of all words or data points in the input sequence. However, after the similarity between Q and K is calculated, it will be used for weighted summation to generate the final output.

[0026] In the self-attention mechanism, K, Q, and V are vector matrices obtained by linearly transforming the input sequence (such as the embedding vector of text data). Specifically, we first define three weight matrices: , and , these weight matrices are the parameters learned during the training process. For each element in the input sequence (such as the word embedding vector of each word), calculate its correlation with the weight matrix , and The product of gets each element i , and .

[0027] The working principle of the standard self-attention mechanism is: dynamically assign attention weights by comparing the similarity of Q and K; perform weighted summation of V according to the attention weights to obtain the attention output of the current element. This output is a weighted combination of the most relevant information in the input sequence, thereby improving the flexibility and efficiency of processing complex sequence data.

[0028] The expression of the standard self-attention mechanism is: ; in, represents the result of self-attention, T is the transposition, is the normalization function, Represents the dimension of the key vector K.

[0029] The calculation of K, Q, and V enables the model to capture the dependencies between any two elements in the input sequence, no matter how far apart they are in the sequence. This is very important for tasks such as understanding context-based meaning and handling long-distance dependencies.

[0030] In the standard self-attention mechanism, for each element (token) of the input sequence, it is necessary to interact with all other elements. Its computational complexity includes the following aspects: (1) Calculate Q (query), K (key), and V (value): Assume that the length of the input sequence is n and the representation dimension of each token is d, then: The complexity of generating Q, K, and V is ( ); (2) Calculate the dot product of Q and K to form an n×n similarity matrix. The complexity of this step is ( ); (3) Next, it is usually necessary to calculate the weights through a softmax operation. The complexity of this operation is ( ); (4) Calculate the weighted V: Apply the attention weights to V and get the output. The complexity of this step is ( ).

[0031] Therefore, the computational complexity of the standard self-attention mechanism is ( ), where n is the length of the input sequence and d is the dimension of the token.

[0032] In the standard self-attention mechanism, there are n×n computational units because each sequence element must interact with all other elements. Although this global interaction can capture rich contextual information, it will lead to a significant increase in computational and memory overhead when processing long sequences.

[0033] In order to reduce the computational complexity of the self-attention mechanism, such as Figure 1 As shown, this embodiment provides a large model self-attention optimization method, which includes steps S1 to S4.

[0034] S1, obtain the large KQ matrix obtained after the input sequence passes through the large model self-attention structure.

[0035] In this embodiment, the large model is a trained deep learning model. Depending on different application scenarios, the input sequence can be different types of data such as text data, image data, audio and video data, etc.

[0036] The KQ large matrix is ​​a similarity matrix between the key matrix K generated by the large model self-attention structure and the query matrix Q. The similarity can be calculated by calculating the dot product of K and Q, or cosine similarity, or other measurement methods, which are not limited here.

[0037] S2, divide the KQ large matrix into several The KQ small matrix.

[0038] In this embodiment, block processing can significantly reduce the amount of calculation. For example, a large KQ matrix of n×n (where n is greater than 3) can be divided into several The KQ small matrix can be divided into | | a small KQ matrix.

[0039] S3, perform local attention calculation on each KQ small matrix, and sort it in descending order of similarity with other KQ small matrices to obtain a sorted list.

[0040] In this embodiment, for each KQ small matrix, its corresponding local attention is calculated, that is, the similarity of the K and Q vectors in the KQ small matrix is ​​calculated to obtain an attention score; then based on the attention score, the similarity is calculated with other KQ small matrices, and they are arranged from large to small according to the similarity to obtain a sorted list.

[0041] For example, there are 64 3×3 KQ small matrices. After calculating the attention score of each 3×3 KQ small matrix, the similarity between each KQ small matrix and other KQ small matrices is calculated in turn. For example, the similarity between the first KQ small matrix and the remaining 63 KQ small matrices is calculated; the similarity between the second KQ small matrix and the other 62 KQ small matrices is calculated, and so on, so as to obtain a sorted list S arranged from large to small according to the similarity.

[0042] S4, for each element in the input sequence, according to the calculated sorted list, select the local attention corresponding to the top r KQ small matrices in similarity for weighted calculation as the attention representation of the current element, until all the elements of the input sequence are completed, and the optimized attention result of the input sequence is obtained.

[0043] For each element in the input sequence, only the product of K and Q corresponding to the current element is calculated. Specifically: According to the calculated sorted list S, select the KQ small matrices with the top r similarity rankings (r is less than the number of elements in the input sequence) to obtain a high-similarity attention set; perform weighted calculation on the corresponding KQ small matrix in the high-similarity attention set as the attention representation of the current element.

[0044] For example, suppose we select the top 2 KQ small matrices with similarity, and assume that the top 2 KQ small matrices are ( , )and( , ), that is, a high similarity attention set, then the attention weight of the current element is ( × + × + × + × ); in, , , , Respectively represent the attention of the a, b, c, and d KQ small matrices; ( , ) represents the similarity between the a-th and b-th KQ small matrices; ( , ) represents the similarity between the cth and dth KQ small matrices; , , , Respectively represent the weights of the settings.

[0045] After the attention weight vector of the current element is normalized by the softmax function, the value matrix V is weighted and summed to obtain the global attention representation of the current element, and then the attention representation of each element is obtained.

[0046] The result of the dot product operation can be scaled as needed before normalization, such as by dividing by the square root of the key vector dimension.

[0047] In summary, when calculating the attention of the input sequence, the present invention reduces the computational complexity of calculating the global attention of each element by dividing the global large matrix into small matrices, reduces the operations between attention matrices, and optimizes the computational complexity of the large model. At the same time, the present invention selects small matrices with higher similarity for weighting through a similarity sorting list, and can perform weighted calculations with long-distance matrices, thereby capturing long-distance contextual information, improving the accuracy of the attention output results, and thus improving the accuracy and performance of the large model.

[0048] Embodiment 2 like Figure 2 As shown, the second embodiment of the present invention further provides a large model self-attention optimization device, including: The acquisition unit is used to obtain the KQ large matrix obtained after the input sequence passes through the large model self-attention structure; A splitting unit is used to split the KQ large matrix into several The KQ small matrix; The similarity sorting unit is used to perform local attention calculation on each KQ small matrix and sort the similarity with other KQ small matrices in descending order to obtain a sorted list; The attention calculation unit is used to calculate the attention representation of each element; for each element in the input sequence, according to the calculated sorted list, the local attention corresponding to the top r KQ small matrices with similarity ranking is selected for weighted calculation as the attention representation of the current element, until all the elements of the input sequence are completed, and the optimized attention result of the input sequence is obtained.

[0049] Embodiment 3 The third embodiment of the present invention also provides a large model self-attention optimization device, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the large model self-attention optimization method as described above.

[0050] Embodiment 4 The fourth embodiment of the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the large model self-attention optimization method as described above is implemented.

[0051] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0052] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0053] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0054] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0055] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0056] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0057] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A large model self-attention optimization method, characterized in that: include: S1, obtain the KQ large matrix obtained after the input sequence passes through the large model self-attention structure; S2, divide the KQ large matrix into several The KQ small matrix; S3, perform local attention calculation on each KQ small matrix; and sort the similarity with other KQ small matrices in descending order to obtain a sorted list; S4, for each element in the input sequence, according to the calculated sorted list, select the local attention corresponding to the top r KQ small matrices in similarity for weighted calculation as the attention representation of the current element, until all the elements of the input sequence are completed, and the optimized attention result of the input sequence is obtained.

2. A large model self-attention optimization method according to claim 1, characterized in that ,The KQ large matrix is ​​the similarity matrix between the key matrix K generated by the large model self-attention structure and the query matrix Q.

3. A large model self-attention optimization method according to claim 1, characterized in that ,The similarity is obtained by calculating the dot product of the elements.

4. A large model self-attention optimization method according to claim 1, characterized in that ,The similarity is obtained by calculating the cosine similarity of the elements.

5. A large model self-attention optimization method according to claim 2, characterized in that , the S4 is specifically: For each element in the input sequence, select the top r KQ small matrices with similarity according to the calculated sorted list to obtain a high similarity attention set; After performing weighted calculation of attention on each KQ small matrix in the high similarity attention set according to the preset weight, the softmax function is used for normalization to obtain the attention weight vector of the current element; The attention weight vector of the current element is weighted summed with the value matrix V generated by the self-attention structure of the large model to obtain the global attention representation of the current element.

6. A large model self-attention optimization device, characterized in that: include: The acquisition unit is used to obtain the KQ large matrix obtained after the input sequence passes through the large model self-attention structure; A splitting unit is used to split the KQ large matrix into several The KQ small matrix; The similarity sorting unit is used to perform local attention calculation on each KQ small matrix and sort the similarity with other KQ small matrices in descending order to obtain a sorted list; The attention calculation unit is used to calculate the attention representation of each element; for each element in the input sequence, according to the calculated sorted list, the local attention corresponding to the top r KQ small matrices with similarity ranking is selected for weighted calculation as the attention representation of the current element, until all the elements of the input sequence are completed, and the optimized attention result of the input sequence is obtained.

7. A large model self-attention optimization device, characterized in that: It includes a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a large model self-attention optimization method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor of a device where the computer-readable storage medium is located, implement a large-model self-attention optimization method as described in any one of claims 1 to 5.

Images