A large model optimization method and system based on an attention mechanism

Abstract

The application belongs to the technical field of deep learning, and aims at the problems of large model, such as large calculation resource consumption, high memory occupation and poor interpretability, and proposes a large model optimization method and system based on an attention mechanism. On the method, first, jointly K and V parameter matrices are used as a large KV matrix, and dynamic adaptive tensor decomposition is used for dimension reduction; an explanation layer mark processing process is added to each data processing step; an implicit expert collaborative model is used to generate weights in the intermediate calculation of decomposition. Among them, the tensor decomposition predicts the optimal rank through the NLP model and the topological model and expands it to a three-order tensor; the implicit expert collaborative model shares the basic parameter matrix, combines the sub-networks with the gating system and realizes implicit sparsity. Experimental verification shows that the method can shorten the calculation time, reduce the memory occupation, improve the calculation speed, enhance the model interpretability and adaptability, and effectively optimize the performance of the large model.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology and relates to a method and system for optimizing large models based on attention mechanisms. Background Technology

[0002] In today's digital age, with the explosive growth of data and the ever-increasing demand for intelligent applications, large models play a crucial role in many fields such as natural language processing and computer vision. However, large models face a series of technical challenges in their development.

[0003] From a computational resource perspective, large models have a massive number of parameters; for example, GPT-3 has 175 billion parameters, which makes the computational resource consumption during training and inference extremely high. In traditional attention mechanisms, computational complexity is quadratic with sequence length. When processing long texts or high-resolution images, the high computational cost becomes a bottleneck limiting the expansion of model applications.

[0004] In terms of memory usage, the large amount of intermediate data generated during the operation of large models, such as the K-parameter matrix and V-parameter matrix, leads to a sharp increase in memory requirements. For some resource-constrained devices, the normal operation of large models cannot be supported, hindering their application in scenarios such as edge computing.

[0005] Model interpretability is also a significant challenge. As model structures and parameters become increasingly complex, the model's decision-making process resembles a "black box," making it difficult for users to understand why the model produces specific outputs. This severely limits the application of large models in fields such as healthcare and finance, where the reliability and interpretability of decisions are extremely important.

[0006] In this industry context, various sectors have an urgent need to improve the performance of large-scale models. In the medical field, using large models to assist in disease diagnosis and drug development requires models that can accurately understand complex data such as medical text and images, and provide interpretable diagnostic suggestions. In the financial field, tasks such as risk assessment and investment decision-making not only require models to be efficient and accurate, but also to clearly explain the decision-making process to meet regulatory requirements and user trust needs. In content creation, intelligent customer service, and other fields, to improve user experience, large-scale models need to be able to quickly respond to user requests and handle complex scenarios such as long text conversations. Therefore, developing a method to optimize the performance of large-scale models, improve interpretability, and reduce computational and memory costs is urgently needed. Summary of the Invention

[0007] To address the problems existing in the background technology, this invention proposes a large model optimization method and system based on attention mechanism.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A large model optimization method based on attention mechanism includes:

[0010] S1. In the large model, the K-parameter matrix and V-parameter matrix are combined into a large KV matrix, and then dynamic adaptive tensor decomposition is used to decompose and replace the large KV matrix into multiple lower-dimensional matrices.

[0011] S2. In the large model, add an interpretation layer to each data processing step to mark the data processing of the previous step;

[0012] S3. An implicit expert collaboration model is adopted, which directly generates virtual expert weights through a gating network and decomposes the intermediate calculation into multiple implicit sub-networks.

[0013] Furthermore, the specific method for replacing the large KV matrix with two lower-dimensional matrices using dynamic adaptive tensor decomposition is as follows:

[0014] Based on NLP model analysis of data complexity and relevance, the optimal decomposition rank of the K / V matrix is ​​predicted in real time through topological model;

[0015] By introducing a context dimension, the K / V parameters are expanded from a two-dimensional matrix to a third-order tensor. Core tensor and factor matrix decomposition are used to preserve high-order feature interaction information.

[0016] Furthermore, the topological model is as follows:

[0017] Based on the correlation of each element in the K / V matrix, a three-dimensional topology model is generated by connecting any two elements based on their correlation and determining the distance between them.

[0018] The three-dimensional topological model is rotated based on its center point, and the first few elements are extracted as the principal elements to predict the optimal decomposition rank of the K / V matrix.

[0019] Furthermore, the implicit expert collaboration model is as follows:

[0020] Virtual expert weights are generated directly through a gated network, decomposing the intermediate computation of the data into multiple implicit sub-networks, with the weights parameterized by dynamic gating.

[0021] All subnetworks share the basic parameter matrix, and the number of parameters is reduced by differentiating the gating coefficients. Top-k gating and sparse regularization are used for threshold truncation to achieve implicit sparsity.

[0022] Furthermore, the number of interpretation layers is consistent with the number of data processing steps in the large model, and each interpretation layer is connected by a separate information transmission channel.

[0023] Furthermore, the virtual expert weights are parameterized by dynamic gating.

[0024] In addition, this invention also provides a large model optimization system based on an attention mechanism, which runs the above-mentioned large model optimization method based on an attention mechanism, including:

[0025] The data processing module combines the K-parameter matrix and the V-parameter matrix into a large KV matrix, and then uses dynamic adaptive tensor decomposition to decompose and replace the large KV matrix into multiple lower-dimensional matrices.

[0026] The expert collaboration module sets up an implicit expert collaboration model, which directly generates virtual expert weights through a gating network, decomposing the intermediate calculations into multiple implicit sub-networks.

[0027] The interpretation module adds an interpretation layer to each data processing step, marking the data processing of the previous step.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] This invention combines the K-parameter and V-parameter matrices into a large K / V matrix and employs dynamic adaptive tensor decomposition. Based on an NLP model, it analyzes data complexity and relevance, and uses a topological model to predict the optimal decomposition rank of the K / V matrix in real time. Taking long text processing as an example, traditional methods experience a significant increase in computational cost when dealing with long sequences. This method, however, reduces computation by rationally decomposing the matrix and replacing the large K / V matrix with multiple lower-dimensional matrices. Introducing a context dimension expands the K / V parameters from a two-dimensional matrix to a third-order tensor, preserving high-order feature interaction information while further optimizing memory usage. Experimental results show that when processing text sequences of length 1000, the computation time is reduced by approximately 30% and memory usage by about 25% compared to traditional methods.

[0030] This invention directly generates virtual expert weights through a gated network, decomposing intermediate computations into multiple implicit sub-networks. All sub-networks share the same fundamental parameter matrix, differing only in their gating coefficients. This approach significantly reduces the number of parameters, lowering computational complexity while maintaining model performance. Threshold truncation using Top-k gating and sparse regularization achieves implicit sparsity, further improving computational efficiency. Experiments show that compared to the model without this approach, under the same task, the model's computation speed is improved by approximately 20%, and memory requirements are reduced by approximately 15%.

[0031] This invention incorporates interpretation layers into the data processing steps of a large model, with the number of interpretation layers matching the number of data processing steps. Each interpretation layer has a separate information transmission channel. This allows the model to mark the data processing of the previous step at each stage, presenting a clear data processing flow to the user. For example, in text classification tasks, users can understand how the model processes each word and sentence in the text and why it is classified into a specific category through the interpretation layers. This makes the model's "black box" decision-making process transparent, enhancing the model's interpretability and increasing user trust in the model's output.

[0032] This invention establishes an implicit expert collaboration model. By dynamically gating and parameterizing virtual expert weights, the model can flexibly adjust the combination of subnetworks according to different input data and task requirements, making the model more adaptable. When faced with text data from different domains, the model can automatically adjust its attention mode to focus on key information. When processing disease diagnosis texts in the medical field, the model can quickly locate key descriptions related to symptoms; when processing investment reports in the financial field, it can accurately capture important factors affecting investment decisions, thereby improving the model's performance in diverse tasks and data scenarios. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention;

[0034] Figure 2 This is a system structure diagram of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] like Figures 1-2 As shown, the technical solution adopted in this invention is as follows:

[0037] A large model optimization method based on attention mechanism includes:

[0038] S1. In the large model, the K-parameter matrix and V-parameter matrix are combined into a large KV matrix, and then dynamic adaptive tensor decomposition is used to decompose and replace the large KV matrix into multiple lower-dimensional matrices.

[0039] S2. In the large model, add an interpretation layer to each data processing step to mark the data processing of the previous step.

[0040] S3. An implicit expert collaboration model is adopted, which directly generates virtual expert weights through a gating network and decomposes the intermediate calculation into multiple implicit sub-networks.

[0041] In the large model, the K-parameter matrix and the V-parameter matrix are combined into a large KV matrix. Then, based on the NLP model, the complexity and relevance of the data are analyzed, and the optimal decomposition rank of the K / V matrix is ​​predicted in real time through a topological model. Decomposing the K and V matrices after combining them reduces the matrix dimensionality, lowers computational complexity, and improves computational efficiency.

[0042] NLP models analyze the complexity and relevance of data, and combine topological models to predict the optimal decomposition rank, making matrix decomposition more targeted and scientific. This enables reasonable decomposition based on data characteristics and further optimizes the utilization of computing resources.

[0043] Based on the correlation of each element in the K / V matrix, a three-dimensional topological model is generated by connecting any two elements based on their correlation and determining the distance. The three-dimensional topological model is then rotated based on its center point, and the first few preset elements are extracted as principal elements to predict the optimal decomposition rank of the K / V matrix.

[0044] By introducing a context dimension, the key-value (K / V) parameters are expanded from a two-dimensional matrix to a third-order tensor. Core tensor and factor matrix decomposition are used to preserve high-order feature interaction information. This allows for dynamic adaptive tensor decomposition, replacing the large K / V matrix with multiple lower-dimensional matrices. Introducing a context dimension to expand to a third-order tensor and performing decomposition, while preserving high-order feature interaction information, helps the model better capture complex relationships in the data, improving its expressive power and performance.

[0045] In the large model, an interpretation layer is added to each data processing step. The number of interpretation layers is the same as the number of data processing steps in the large model, and each interpretation layer has a separate information transmission channel. The data processing of the previous step is marked.

[0046] An interpretation layer is added to each data processing step, giving each step of the model a corresponding label and increasing the model's interpretability. This label serves as an explanatory note for each data processing step, and can be a piece of text or a universally understood symbol within the model. Users or developers can better understand the model's decision-making process and data processing logic through these interpretation layers. Separate information transmission channels ensure the independence and accuracy of information transfer between interpretation layers, facilitating a clearer analysis of the model's operational mechanism.

[0047] Virtual expert weights are directly generated through a gating network, with the weights dynamically parameterized by gating. The intermediate computation of the data is decomposed into multiple implicit sub-networks. The gating network generates virtual expert weights, and the dynamic gating parameterization allows the model to flexibly adjust the weights according to the characteristics of the data, improving the model's adaptability and performance.

[0048] By decomposing intermediate computations into multiple implicit subnetworks, and having these subnetworks share the basic parameter matrix, the number of parameters in the model is reduced, thus lowering the model's complexity and computational cost.

[0049] All subnetworks share the basic parameter matrix, with differences only in the combination of gating coefficients to reduce the number of parameters. Top-k gating combined with sparse regularization is used for threshold truncation to achieve implicit sparsity. Top-k gating and sparse regularization achieve implicit sparsity, further reducing unnecessary computation and improving computational efficiency. This also helps the model better learn important features in the data, enhancing its generalization ability.

[0050] In summary, the attention-based large model optimization method provided by this invention optimizes large models in terms of computational efficiency, interpretability, and model performance through a series of operations such as optimizing and decomposing the parameter matrix, adding an interpretation layer, and adopting an implicit expert collaborative model. This helps to improve the performance of large models in practical applications.

[0051] In addition, this invention also provides a large model optimization system based on an attention mechanism, which runs the above-mentioned large model optimization method based on an attention mechanism, including:

[0052] The data processing module combines the K-parameter matrix and the V-parameter matrix into a large KV matrix, and then uses dynamic adaptive tensor decomposition to decompose and replace the large KV matrix into multiple lower-dimensional matrices.

[0053] The expert collaboration module sets up an implicit expert collaboration model, which directly generates virtual expert weights through a gating network, decomposing the intermediate calculations into multiple implicit sub-networks.

[0054] The interpretation module adds an interpretation layer to each data processing step, marking the data processing of the previous step.

[0055] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A large model optimization method based on attention mechanism, characterized in that, Including: S1. In the large model, the K-parameter matrix and the V-parameter matrix are combined into a large KV matrix, and then dynamic adaptive tensor decomposition is used to decompose and replace the large KV matrix into multiple lower-dimensional matrices. The dynamic adaptive tensor decomposition specifically includes: Based on a pre-trained natural language processing model, the complexity and correlation between features of input long text or high-resolution image feature data are analyzed, and the optimal decomposition rank of the K / V matrix is ​​predicted in real time through a topological model. The method for constructing the topology model and predicting the optimal rank is as follows: Based on the feature correlation of each element in the K / V matrix, the connection distance between any two elements is set according to the rule that the higher the correlation, the closer the connection distance, to generate a three-dimensional topology model; the three-dimensional topology model is rotated according to the geometric center point of the three-dimensional topology model, and the first preset number of elements are extracted as the principal elements to determine the optimal decomposition rank of the K / V matrix. The context dimension is introduced to expand it into a third-order tensor, and the core tensor and three factor matrices corresponding to the sequence length, feature dimension and context dimension are used for decomposition to retain high-order feature interaction information. S2. In the large model, an interpretation layer is added to each data processing step to mark the data processing of the previous step. The mark is text containing the data processing operation type, feature weight threshold and computing node number, or binary general symbols that the model can parse. S3. An implicit expert collaboration model is adopted, which directly generates virtual expert weights through a gating network and decomposes the intermediate calculation into multiple implicit sub-networks.

2. The large model optimization method based on attention mechanism according to claim 1, characterized in that, The implicit expert collaboration model is as follows: Virtual expert weights are generated directly through a gated network, decomposing the intermediate computation of the data into multiple implicit sub-networks, with the weights parameterized by dynamic gating. All subnetworks share the basic parameter matrix, and the number of parameters is reduced by differentiating the gating coefficients. Topk gating and sparse regularization are used for threshold truncation to achieve implicit sparsity.

3. The large model optimization method based on attention mechanism according to claim 1, characterized in that, The number of interpretation layers is consistent with the number of data processing steps in the large model, and each interpretation layer is connected by a separate information transmission channel.

4. The large model optimization method based on attention mechanism according to claim 1, characterized in that, The virtual expert weights are parameterized by dynamic gating.

5. A large model optimization system based on an attention mechanism, characterized in that, Running the attention-based large model optimization method according to any one of claims 1 to 4 includes: The data processing module combines the K-parameter matrix and the V-parameter matrix into a large KV matrix, and then uses dynamic adaptive tensor decomposition to decompose and replace the large KV matrix into multiple lower-dimensional matrices. The expert collaboration module sets up an implicit expert collaboration model, which directly generates virtual expert weights through a gating network, decomposing the intermediate calculations into multiple implicit sub-networks. The interpretation module adds an interpretation layer to each data processing step, marking the data processing of the previous step.

Images