An optimization method for text embedding models

By optimizing the attention and intermediate computation modules of the text embedding model, the number of parameters and caching requirements are reduced, the inference efficiency and expressive power of the model are improved, the problem of high computational resource consumption in existing text embedding models is solved, and more efficient text embedding processing is achieved.

CN118520845BActive Publication Date: 2024-11-15ZHONGSHU (XIAMEN) INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410974663.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-15
Estimated Expiration
2044-07-19

AI Technical Summary

Technical Problem

Existing text embedding models consume a lot of computational resources in attention and intermediate computation modules, resulting in low caching and inference efficiency, which makes it difficult to meet practical needs.

Method used

By combining the K and V parameter matrices into a large KV matrix and performing low-rank decomposition, the dimensionality is reduced to two lower-dimensional matrices. The Q parameter matrix is ​​also decomposed into two lower-dimensional matrices. At the same time, a sparse hybrid expert model (MoE module) is used to replace the intermediate computation module, and multiple expert modules are combined with the gating module for optimization.

Benefits of technology

It reduces the number of parameters and caching requirements of the attention computation module, improves the model's inference efficiency and the expressive power of output embedding, enhances the model's flexibility and interpretability, and improves the model's ability to parse and generalize complex data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118520845B_ABST
    Figure CN118520845B_ABST
Patent Text Reader

Abstract

The present invention discloses an optimization method for a text embedding model, comprising the following steps: S1, in the text embedding model, combining a K parameter matrix and a V parameter matrix into a large KV matrix, and then using low-rank matrix decomposition to decompose the large KV matrix into two matrices with lower dimensions; S2, in the text embedding model, decomposing a Q parameter matrix into two matrices with lower dimensions; S3, using a sparse mixed expert model, using a combination of multiple expert modules and an expert selection gate, and outputting the embedding dimension with the best selection; based on the basic structure of the original general Embedding model, this method optimizes the main attention calculation module in the encoding layer to improve the reasoning efficiency while retaining the maximum performance of the model; improving the intermediate calculation module, expanding the learning dimension of the model, enhancing the learning ability, and further improving the expression ability of the output embedding for discrete data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of model optimization, and in particular to an optimization method for a text embedding model. Background Art

[0002] In natural language processing, text embedding models encode text data in latent space and convert text data into text embeddings that can express its underlying semantics. Text embedding models are usually encoding structures, and the main encoding module consists of dozens of encoding layers. The encoding layer is generally divided into an attention part and a forward propagation part. Among them, the attention part is responsible for calculating the attention output of the input. The input of the attention calculation module passes through its main calculation modules: query mapping layer, key mapping layer, and value mapping layer to obtain Query, Key, and Value respectively. The three then obtain the attention output through a series of operations. The main calculation modules of this attention part (the Q parameter matrix of the query mapping layer, the K parameter matrix of the key mapping layer, and the V parameter matrix of the value mapping layer) occupy a considerable part of the total parameters of the model, generating a large amount of KV cache and consuming a lot of computing resources during model training or inference, especially in multi-head attention MHA calculation; the forward propagation part is generally divided into an intermediate calculation module and an output calculation part. The intermediate calculation module is usually a simple full-chain linear layer, or composed of an activation layer, a Dropout layer, etc., to further process and calculate the attention output. Therefore, the existing text embedding model has large cache calculation, low reasoning efficiency and reasoning effect, and it is difficult to meet actual usage needs. Summary of the invention

[0003] To solve the above problems, the present invention provides an optimization method for a text embedding model. Based on the basic structure of the original general Embedding model, the method optimizes the main attention calculation module in the encoding layer on the one hand, so as to improve the reasoning efficiency while retaining the maximum performance of the model and reduce the KV cache during reasoning; on the other hand, improvements are made to the intermediate calculation module to expand the learning dimension of the model, enhance the learning ability, and further improve the output embedding's ability to express discrete data.

[0004] The present invention adopts the following technical solutions:

[0005] A method for optimizing a text embedding model comprises the following steps:

[0006] S1. In the text embedding model, the K parameter matrix and the V parameter matrix are combined into a large KV matrix, and then low-rank matrix decomposition is used to decompose the large KV matrix into two matrices with lower dimensions;

[0007] The specific process of step S1 is:

[0008] S11. Combine the K parameter matrix and V parameter matrix originally used to calculate the K parameter and V parameter into a large KV matrix, and decompose the large KV matrix into two matrices with lower dimensions. and matrix , the calculation formula is: , , , where the K parameter matrix is , , Indicates the dimension size is OK The V parameter matrix is ​​a collection of real matrices with columns , , Indicates the dimension size is OK The set of real matrices with columns; the large KV matrix is , , Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; Represents the merged large KV matrix The original column dimension size of ; express After dimensionality reduction , The corresponding intermediate dimension size of ;

[0009] S12, the matrix and matrix Replace the original K parameter matrix and V parameter matrix respectively to calculate the K parameter and V parameter respectively. The calculation formula is: , , ,in, Represents a token The hidden state feature vector of Indicates the dimension size is The set of real vectors of ; Represents a token The merged key and value feature vectors, Indicates the dimension size is The set of real vectors of ; Represents a token The key feature vector of Indicates the dimension size is The set of real vectors of ; Represents a token The value feature vector, Indicates the dimension size is The set of real vectors of ;

[0010] S2. In the text embedding model, the Q parameter matrix decomposition is replaced by two matrices with lower dimensions;

[0011] The specific process of step S2 is:

[0012] S21. Decompose the Q parameter matrix into two matrices with lower dimensions and matrix , the calculation formula is: , , , , where the Q parameter matrix is , , Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; express The original column dimension size of ; express After dimensionality reduction , The corresponding intermediate dimension size of ;

[0013] S22, the matrix and Replace the original Q parameter matrix to calculate the Q parameter. The calculation formula is: , ,in, Represents a token The query feature vector of Indicates the dimension size is The set of real vectors of ;

[0014] S3, adopts a sparse hybrid expert model, which combines multiple expert modules with an expert selection gate to output the optimal embedding dimension.

[0015] Preferably, the Q parameter matrix The dimensionality reduction is lower than that of the large KV matrix , which is used to not affect the expression effect of attention output and lose too much attention information, that is, .

[0016] Preferably, the specific process of step S3 is:

[0017] S31, treating the original intermediate calculation module as an expert module;

[0018] S32, replacing the original intermediate computing module with a MoE module composed of multiple expert modules and a gating module, where each expert module is an independent trainable module for modeling data distribution and patterns of different attention outputs;

[0019] S33, the gating module uses a network module with learnable parameters to dynamically determine which expert modules should be activated based on the characteristics of the input data or attention output; the input data is assigned to the most suitable top-k expert models for processing through the gating module; the gating performs weighted fusion of the outputs based on the outputs of the top-k experts to obtain the final optimal output.

[0020] After adopting the above technical solution, the present invention has the following advantages compared with the background technology:

[0021] A. Beneficial Effects on Attention Computation

[0022] 1. Reduce the number of parameters in the attention calculation module:

[0023] Each attention calculation module, due to , , Replaced by , , , , the parameter quantity is Reduced to ,in , , while losing as little information as possible, reduce the redundant rows or columns in the parameter matrix and reduce the overall parameter volume of the attention calculation module;

[0024] 2. Reduce the KV cache required for inference training:

[0025] The attention cache of each layer is originally Each token of each element is reduced to , especially for the multi-head attention mechanism with high K and V dimensions, the attention cache overhead can be greatly reduced.

[0026] B. Beneficial effects of the intermediate computing modules of the MoE structure

[0027] 1. Improve output embedding expression ability and improve model performance:

[0028] Compared with the original single intermediate computing module with simple structure, direct processing of attention output, and simple forward propagation, the hybrid expert strategy can effectively utilize the advantages of multiple expert models. Different specialized expert modules can have processing structures for different data, which can better capture the patterns and relationships in the attention output; through the weighted combination of expert outputs by the gating module, the final embedded expression ability is optimized, and the overall model's ability to parse large-scale data with complex features is improved, thereby improving the model's accuracy and generalization ability, so that the model can achieve better performance for complex tasks.

[0029] 2. Improve the flexibility, interpretability and efficiency of the model:

[0030] The sparse hybrid expert model has strong flexibility and can select appropriate experts to combine according to the needs of the task. At the same time, since each expert module is relatively independent, the decision-making process of the model can be easier to explain and understand. During calculation, the sparse expert strategy only activates a few expert modules to process the attention input, and most expert modules are in an inactive state. Compared with the dense expert strategy, it reduces the computational overhead while enhancing the model performance, thereby improving the overall computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the present invention;

[0032] Figure 2 This is a schematic diagram of the architecture of the encoding module of the traditional text embedding model;

[0033] Figure 3 This is a schematic diagram of attention calculation optimization of the present invention;

[0034] Figure 4 Schematic diagram of the intermediate computing module of the MoE module of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the following examples are exemplary and cannot be understood as limiting the present invention.

[0036] See also Figures 1 to 4 , an optimization method for a text embedding model, comprising the following steps:

[0037] S1. In the text embedding model, the K parameter matrix and the V parameter matrix are combined into a large KV matrix, and then low-rank matrix decomposition is used to decompose the large KV matrix into two matrices with lower dimensions;

[0038] The specific process of step S1 is:

[0039] S11. Combine the K parameter matrix and V parameter matrix originally used to calculate the K parameter and V parameter into a large KV matrix, and decompose the large KV matrix into two matrices with lower dimensions. and matrix , the calculation formula is: , , , where the K parameter matrix is , , Indicates the dimension size is OK The V parameter matrix is ​​a collection of real matrices with columns , , Indicates the dimension size is OK The set of real matrices with columns; the large KV matrix is , , Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; Represents the merged large KV matrix The original column dimension size of ; express After dimensionality reduction , The corresponding intermediate dimension size of ;

[0040] S12, the matrix and matrix Replace the original K parameter matrix and V parameter matrix respectively to calculate the K parameter and V parameter respectively. The calculation formula is: , , ,in, Represents a token The hidden state feature vector of Indicates the dimension size is The set of real vectors of ; Represents a token The merged key and value feature vectors, Indicates the dimension size is The set of real vectors of ; Represents a token The key feature vector of Indicates the dimension size is The set of real vectors of ; Represents a token The value feature vector, Indicates the dimension size is The set of real vectors of ;

[0041] S2. In the text embedding model, the Q parameter matrix decomposition is replaced by two matrices with lower dimensions;

[0042] The specific process of step S2 is:

[0043] S21. Decompose the Q parameter matrix into two matrices with lower dimensions and matrix , the calculation formula is: , , , , where the Q parameter matrix is , , Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; express The original column dimension size of ; express After dimensionality reduction , The corresponding intermediate dimension size of ;

[0044] S22, the matrix and Replace the original Q parameter matrix to calculate the Q parameter. The calculation formula is: , ,in, Represents a token The query feature vector of Indicates the dimension size is The set of real vectors of , Represents a token The hidden state feature vector of Same as step S12;

[0045] The Q parameter matrix The dimensionality reduction is lower than that of the large KV matrix , which is used to not affect the expression effect of attention output and lose too much attention information, that is, ;

[0046] S3, using a sparse hybrid expert model, using multiple expert modules combined with an expert selection gate to output the optimal selection embedding dimension;

[0047] The specific process of step S3 is:

[0048] S31, treating the original intermediate calculation module as an expert module;

[0049] S32, replacing the original intermediate computing module with a MoE module composed of multiple expert modules and a gating module, where each expert module is an independent trainable module for modeling data distribution and patterns of different attention outputs;

[0050] S33, the gating module uses a network module with learnable parameters to dynamically determine which expert modules should be activated based on the characteristics of the input data or attention output; the input data is assigned to the most suitable top-k expert models for processing through the gating module; the gating performs weighted fusion of the outputs based on the outputs of the top-k experts to obtain the final optimal output.

[0051] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for optimizing a text embedding model, characterized in that: The following steps are involved: S1. In the text embedding model, the K parameter matrix and the V parameter matrix are combined into a large KV matrix, and then low-rank matrix decomposition is used to decompose the large KV matrix into two matrices with lower dimensions; The specific process of step S1 is: S11. Combine the K parameter matrix and V parameter matrix originally used to calculate the K parameter and V parameter into a large KV matrix, and decompose the large KV matrix into two matrices with lower dimensions. and matrix , the calculation formula is: , , , where the K parameter matrix is , , Indicates the dimension size is OK The V parameter matrix is ​​a collection of real matrices with columns , , Indicates the dimension size is OK The set of real matrices with columns; the large KV matrix is , , Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; Represents the merged large KV matrix The original column dimension size of ; express After dimensionality reduction , The corresponding intermediate dimension size of ; S12, the matrix and matrix Replace the original K parameter matrix and V parameter matrix respectively to calculate the K parameter and V parameter respectively. The calculation formula is: , , ,in, Represents a token The hidden state feature vector of Indicates the dimension size is The set of real vectors of ; Represents a token The merged key and value feature vectors, Indicates the dimension size is The set of real vectors of ; Represents a token The key feature vector of Indicates the dimension size is The set of real vectors of ; Represents a token The value feature vector, Indicates the dimension size is The set of real vectors of ; S2. In the text embedding model, the Q parameter matrix decomposition is replaced by two matrices with lower dimensions; The specific process of step S2 is: S21. Decompose the Q parameter matrix into two matrices with lower dimensions and matrix , the calculation formula is: , , , , where the Q parameter matrix is , , Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; Indicates the dimension size is OK A collection of real matrices with columns; express The original column dimension size of ; express After dimensionality reduction , The corresponding intermediate dimension size of ; S22, the matrix and Replace the original Q parameter matrix to calculate the Q parameter. The calculation formula is: , ,in, Represents a token The query feature vector of Indicates the dimension size is The set of real vectors of ; S3, using a sparse hybrid expert model, using multiple expert modules combined with an expert selection gate to output the optimal selection embedding dimension; The Q parameter matrix The dimensionality reduction is lower than that of the large KV matrix , which is used to not affect the expression effect of attention output and lose too much attention information, that is, .

2. The method for optimizing a text embedding model according to claim 1, wherein: The specific process of step S3 is: S31, treating the original intermediate calculation module as an expert module; S32, replacing the original intermediate computing module with a MoE module composed of multiple expert modules and a gating module, where each expert module is an independent trainable module for modeling data distribution and patterns of different attention outputs; S33, the gating module uses a network module with learnable parameters to dynamically determine which expert modules should be activated based on the characteristics of the input data or attention output; the input data is assigned to the most suitable top-k expert models for processing through the gating module; the gating performs weighted fusion of the outputs based on the outputs of the top-k experts to obtain the final optimal output.