Large model optimization method and system based on attention mechanism

Abstract

The invention belongs to the technical field of deep learning, and provides a large model optimization method and system based on an attention mechanism for solving the problems of large calculation resource consumption, high memory occupation, poor interpretability and the like of a large model. According to the method, K and V parameter matrixes are combined into a large KV matrix, and dimension reduction is carried out by using dynamic adaptive tensor decomposition; adding an interpretation layer mark processing process in each data processing step; an implicit expert collaboration model is adopted, and weight decomposition intermediate calculation is generated through a gating network. Wherein tensor decomposition predicts an optimal rank through an NLP model and a topology model and expands the optimal rank into a third-order tensor; the implicit expert collaboration model shares the basic parameter matrix, and the sub-networks are combined by using gating coefficients and implicit sparsity is realized. Experiments prove that the method can shorten calculation time, reduce memory usage, improve calculation speed, enhance model interpretability and adaptability, and effectively optimize large model performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and relates to a large model optimization method and system based on an attention mechanism. Background Art

[0002] In today's digital age, with the explosive growth of data volume and the continuous improvement of the demand for intelligent applications, large models play a crucial role in many fields such as natural language processing and computer vision. However, the large models face a series of technical challenges in the development process. From the perspective of computing resources, the parameter scale of large models is huge. Taking GPT-3 as an example, it has 175 billion parameters, which makes the model consume extremely large computing resources during training and inference. In the traditional attention mechanism, the computational complexity has a quadratic relationship with the sequence length. When processing data such as long texts or high-resolution images, the high computational cost becomes a bottleneck restricting the application expansion of the model. In terms of memory occupancy, a large amount of intermediate data generated during the operation of large models, such as the K parameter matrix, V parameter matrix, etc., leads to a sharp increase in memory requirements. For some devices with limited resources, they cannot support the normal operation of large models, hindering the application of large models in scenarios such as edge computing. Model interpretability is also an important problem currently faced. With the continuous complexity of model structures and parameters, the model decision-making process is like a "black box", and it is difficult for users to understand why the model makes a specific output. This severely restricts the application of large models in fields such as healthcare and finance where extremely high requirements are placed on the reliability and interpretability of decisions. Under the industry background, there is an urgent need to improve the performance of large models in various fields. In the medical field, when using large models to assist in disease diagnosis and drug research and development, the model needs to be able to accurately understand complex data such as medical texts and images and provide interpretable diagnostic suggestions; in the financial field, when performing tasks such as risk assessment and investment decision-making, not only is it required that the model be efficient and accurate, but also the decision-making process needs to be clearly explained to meet regulatory requirements and user trust needs. In fields such as content creation and intelligent customer service, in order to improve the user experience, it is required that large models can quickly respond to user requests and handle complex scenarios such as long text conversations. Therefore, it is extremely urgent to develop a method that can optimize the performance of large models, improve interpretability, and reduce computational and memory costs. Summary of the Invention

[0003] To solve the problems existing in the background art, the present invention proposes a large model optimization method and system based on an attention mechanism.

[0004] To achieve the above object, the technical solution adopted by the present invention is as follows: A large model optimization method based on an attention mechanism, comprising: S1. In the large model, combine the K parameter matrix and the V parameter matrix into a large KV matrix, and then use dynamic adaptive tensor decomposition to decompose and replace the large KV matrix with multiple matrices of lower dimensions. S2. In the large model, add an interpretation layer in each data processing step to mark the data processing of the previous step. S3. Adopt an implicit expert collaboration model to directly generate virtual expert weights through a gating network, and decompose the intermediate calculation into multiple groups of implicit sub-networks.

[0005] Furthermore, the specific method of using dynamic adaptive tensor decomposition to decompose and replace the large KV matrix with two matrices of lower dimensions is as follows: Analyze the complexity and relevance of data based on the NLP model, and predict the optimal decomposition rank of the K / V matrix in real time through a topology model. Introduce a context dimension, expand the K / V parameters from a two-dimensional matrix to a third-order tensor, and use core tensor and factor matrix decomposition to retain high-order feature interaction information.

[0006] Furthermore, the topology model is: Based on the relevance of each element in the K / V matrix, connect lines and determine distances based on the relevance of any two elements to generate a three-dimensional topology model. Rotate the three-dimensional topology model according to the center point of the three-dimensional topology model, extract the first preset number of elements as main elements, and thus predict the optimal decomposition rank of the K / V matrix.

[0007] Furthermore, the implicit expert collaboration model is: Directly generate virtual expert weights through a gating network, decompose the intermediate calculation of data into multiple groups of implicit sub-networks, and the weights are parameterized by dynamic gating. All sub-networks share a basic parameter matrix, and only differ in the combined gating coefficients to reduce the number of parameters; use Top-k gating + sparse regularization for threshold truncation to achieve implicit sparsity.

[0008] Furthermore, the number of the interpretation layers is the same as the number of data processing steps of the large model, and a separate information transmission channel is set between each interpretation layer.

[0009] Furthermore, the virtual expert weights are parameterized by dynamic gating.

[0010] In addition, the present invention also provides a large model optimization system based on the attention mechanism, which runs the above-mentioned large model optimization method based on the attention mechanism, including: A data processing module combines the K parameter matrix and the V parameter matrix into a large KV matrix, and then uses dynamic adaptive tensor decomposition to decompose and replace the large KV matrix with multiple matrices of lower dimensions; An expert collaboration module sets up an implicit expert collaboration model, directly generates virtual expert weights through a gating network, and decomposes the intermediate calculations into multiple groups of implicit sub-networks; An interpretation module adds an interpretation layer in each data processing step to mark the data processing of the previous step.

[0011] Compared with the prior art, the present invention has the following beneficial effects: In the present invention, the K parameter matrix and the V parameter matrix are combined into a large KV matrix, and dynamic adaptive tensor decomposition is used to analyze the data complexity and relevance based on the NLP model, and the optimal decomposition rank of the K / V matrix is predicted in real time through a topological model. Taking the processing of long texts as an example, in the traditional method, the computational complexity increases sharply when facing long sequences, while this method reduces the computational complexity by reasonably decomposing the matrix and replacing the large KV matrix with multiple matrices of lower dimensions. By introducing the context dimension, the K / V parameters are extended from two-dimensional matrices to third-order tensors, which further optimizes the memory occupancy while retaining the high-order feature interaction information. Through experimental tests, when processing a text sequence with a length of 1000, compared with the traditional method, the computational time is shortened by about 30%, and the memory occupancy is reduced by about 25%. In the present invention, virtual expert weights are directly generated through a gating network, the intermediate calculations are decomposed into multiple groups of implicit sub-networks, and all sub-networks share a basic parameter matrix, only through the differential combination of gating coefficients. This method greatly reduces the number of parameters and reduces the computational complexity while maintaining the model performance. Top-k gating + sparse regularization is used for threshold truncation to achieve implicit sparsity, which further improves the computational efficiency. Experiments show that compared with the case where the model is not used, under the same task, the computational speed of the model is increased by about 20%, and the memory requirement is reduced by about 15%.

[0012] In the present invention, an interpretation layer is added in the data processing steps of the large model, and the number of interpretation layers is the same as the number of data processing steps, and there is a separate information transmission channel between each interpretation layer. This enables the model to mark the data processing of the previous step in each data processing process, presenting a clear data processing flow for the user. For example, in a text classification task, the user can understand how the model processes each word and each sentence in the text and why it classifies them into a specific category through the interpretation layer, making the "black box" decision-making process of the model transparent, enhancing the interpretability of the model, and improving the user's trust in the model output results.

[0013] The present invention provides an implicit expert collaboration model. By dynamically gating and parameterizing the weights of virtual experts, it can flexibly adjust the combination of sub-networks according to different input data and task requirements, enabling the model to have stronger adaptability. When facing text data in different fields, the model can automatically adjust the attention pattern and focus on key information. When processing disease diagnosis texts in the medical field, the model can quickly locate the key descriptions related to the disease; when processing investment reports in the financial field, it can accurately capture the important factors affecting investment decisions, thereby improving the performance of the model in diverse tasks and data scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flowchart of the method of the present invention; Figure 2 is a structural diagram of the system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0016] As Figure 1 - Figure 2 shown, the technical solutions adopted by the present invention are as follows: An optimization method for a large model based on an attention mechanism, including: S1. In the large model, combine the K parameter matrix and the V parameter matrix into a large KV matrix, and then use dynamic adaptive tensor decomposition to decompose and replace the large KV matrix with multiple matrices of lower dimensions.

[0017] S2. In the large model, add an interpretation layer in each data processing step to mark the data processing of the previous step.

[0018] S3. Adopt an implicit expert collaboration model, directly generate virtual expert weights through a gating network, and decompose the intermediate calculation into multiple groups of implicit sub-networks.

[0019] In the large model, combine the K parameter matrix and the V parameter matrix into a large KV matrix, and then analyze the complexity and relevance of the data based on the NLP model, and real-time predict the optimal decomposition rank of the K / V matrix through a topological model. Decomposing the combined K and V matrices reduces the dimension of the matrix, lowers the computational complexity, and improves the computational efficiency.

[0020] The NLP model analyzes the complexity and relevance of data, combines with the topological model to predict the optimal decomposition rank, making the matrix decomposition more targeted and scientific, and can decompose reasonably according to the data characteristics, further optimizing the utilization of computing resources.

[0021] Based on the relevance of each element in the K / V matrix, connect lines and determine distances based on the relevance of any two elements to generate a three-dimensional topological model; rotate the three-dimensional topological model according to the center point of the three-dimensional topological model, extract the first preset number of elements as the main elements, and thus predict the optimal decomposition rank of the K / V matrix.

[0022] Introduce the context dimension, expand the K / V parameters from a two-dimensional matrix to a third-order tensor, adopt core tensor and factor matrix decomposition, retain high-order feature interaction information, and thus use dynamic adaptive tensor decomposition to replace the decomposition of the large KV matrix with multiple matrices of lower dimensions. Introducing the context dimension and expanding it to a third-order tensor for decomposition retains high-order feature interaction information, which helps the model better capture complex relationships in the data and improve the model's expressive ability and performance.

[0023] In the large model, add an interpretation layer in each data processing step. The number of interpretation layers is the same as the number of data processing steps of the large model, and a separate information transmission channel is set between each interpretation layer. Mark the data processing of the previous step.

[0024] Adding an interpretation layer in each data processing step makes each step of the model's processing process have corresponding marks, increasing the interpretability of the model. The mark is an explanatory note for the data processing step, which can be a piece of text or a general understanding symbol in a model. Users or developers can better understand the model's decision-making process and data processing logic through these interpretation layers. The separate information transmission channel ensures the independence and accuracy of information transmission between the interpretation layers, which helps to more clearly analyze the operation mechanism of the model.

[0025] Directly generate virtual expert weights through the gating network, and the weights are parameterized by dynamic gating. Decompose the intermediate calculation of the data into multiple groups of implicit sub-networks. The gating network generates virtual expert weights, and the dynamic gating parameterization enables the model to flexibly adjust the weights according to the data characteristics, improving the adaptability and performance of the model.

[0026] Decompose the intermediate calculation into multiple groups of implicit sub-networks, and the sub-networks share the basic parameter matrix, reducing the number of model parameters, the complexity of the model and the computational cost.

[0027] All sub-networks share the basic parameter matrix and only reduce the number of parameters through different combinations of gating coefficients. Top-k gating + sparse regularization is used for threshold truncation to achieve implicit sparsity. Top-k gating and sparse regularization achieve implicit sparsity, further reducing unnecessary computations, improving computational efficiency, and also helping the model to better learn important features in the data and enhance the generalization ability of the model.

[0028] In summary, the large model optimization method based on the attention mechanism provided by the present invention optimizes the large model in terms of computational efficiency, interpretability, and model performance through a series of operations such as optimizing the decomposition of the parameter matrix, adding an interpretation layer, and adopting an implicit expert collaboration model, which helps to improve the performance of the large model in practical applications.

[0029] In addition, the present invention also provides a large model optimization system based on the attention mechanism, which runs the above-mentioned large model optimization method based on the attention mechanism, including: A data processing module that combines the K parameter matrix and the V parameter matrix into a large KV matrix, and then uses dynamic adaptive tensor decomposition to decompose and replace the large KV matrix with multiple matrices of lower dimensions; An expert collaboration module that sets an implicit expert collaboration model, directly generates virtual expert weights through a gating network, and decomposes the intermediate calculations into multiple groups of implicit sub-networks; An interpretation module that adds an interpretation layer in each data processing step to mark the data processing of the previous step.

[0030] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A large model optimization method based on attention mechanism, characterized in that: Included are: S1. In the large model, the K parameter matrix and the V parameter matrix are combined into a large KV matrix, and then dynamic adaptive tensor decomposition is used to decompose the large KV matrix into multiple matrices with lower dimensions. S2. In the large model, an explanation layer is added to each data processing step to mark the data processing of the previous step; S3, adopts the implicit expert collaboration model, directly generates virtual expert weights through the gating network, and decomposes the intermediate calculation into multiple groups of implicit sub-networks.

2. According to the large model optimization method based on the attention mechanism of claim 1, it is characterized in that: The specific method of using dynamic adaptive tensor decomposition to decompose a large KV matrix into two matrices with lower dimensions is: Analyze the complexity and relevance of data based on the NLP model, and predict the optimal decomposition rank of the K / V matrix in real time through the topological model; The context dimension is introduced to expand the K / V parameters from a two-dimensional matrix to a third-order tensor. The core tensor and factor matrix decomposition are used to retain high-order feature interaction information.

3. A large model optimization method based on attention mechanism according to claim 2, characterized in that: The topology model is: Based on the correlation of each element in the K / V matrix, the correlation of any two elements is used to connect and determine the distance, thereby generating a three-dimensional topological model; The three-dimensional topological model is rotated according to its center point, and the preset elements are extracted as the main elements to predict the optimal decomposition rank of the K / V matrix.

4. According to the large model optimization method based on the attention mechanism of claim 1, it is characterized in that: The implicit expert collaboration model is: The virtual expert weights are directly generated through the gating network, which decomposes the intermediate calculation of the data into multiple groups of implicit sub-networks, and the weights are parameterized by dynamic gating; All sub-networks share the basic parameter matrix and only use differential combinations of gating coefficients to reduce the number of parameters. Top-k gating + sparse regularization is used for threshold truncation to achieve implicit sparsity.

5. The large model optimization method based on the attention mechanism according to claim 1 is characterized in that: The number of the interpretation layers is consistent with the number of data processing steps of the large model, and a separate information transmission channel is set between each interpretation layer.

6. A large model optimization method based on attention mechanism according to claim 1, characterized in that: The virtual expert weights are parameterized by dynamic gating.

7. A large model optimization system based on attention mechanism, characterized in that: Running the large model optimization method based on the attention mechanism described in any one of claims 1 to 6 comprises: The data processing module combines the K parameter matrix and the V parameter matrix into a large KV matrix, and then uses dynamic adaptive tensor decomposition to decompose the large KV matrix into multiple matrices with lower dimensions; Expert collaboration module, sets up an implicit expert collaboration model, directly generates virtual expert weights through a gating network, and decomposes the intermediate calculation into multiple groups of implicit sub-networks; The interpretation module adds an interpretation layer in each data processing step to mark the data processing in the previous step.

Images