Rapid search method for optimal embedding position of large model

Abstract

The invention discloses a method for quickly searching an optimal embedding position of a large model, which is characterized by comprising the following steps of: selecting K representative samples of a data set of a task, and constructing a representative set for searching the optimal embedding position, in which K is not suitable to be too large; inputting all samples of the representative set to a multi-modal large model (MLLM) and a small model of a specific task at the same time, introducing a feature fusion module, and fusing the features of the i Transform module of the MLLM and the features of the final layer of the small model; the parameters of the small model are frozen, k-step efficient parameter fine tuning is carried out on all the parameters of the MLLM, and k is not suitable to be too large; in each step of fine tuning, solving a two-norm of a gradient vector of the parameter with respect to a loss function to obtain k gradient two-norms, and carrying out weighted summation on the k gradient two-norms to obtain fitness; the random number seeds are replaced, the second step to the fourth step are repeated, P times of repeated experiments are carried out, P should not be too large or too small, and it is preferable that P should be 3-30 times; and i is iterated in a value range from 1 to T, 2 to 5 steps are repeated in each iteration step, the average fitness of different embedding positions is obtained, i * with the maximum average fitness is finally selected as the optimal embedding position, and T is the total number of layers of the MLLM. The rapid searching method for the optimal embedding position of the large model has the advantages of being high in searching speed and high in task expansibility, and is particularly suitable for scenes where small model knowledge fusion needs to be rapidly conducted on the large model.

Description

Technical Field

[0001] The present invention relates to the fields of computer science, large models, multimodal large models, artificial intelligence, deep learning, and pattern recognition, and in particular to a fast search method for an optimal embedding position of a large model. Background Art

[0002] Large multimodal models are usually pre-trained on massive general datasets, which makes them very powerful when handling a wide range of general tasks. However, when applied to specific downstream tasks, general pre-trained models often appear to be incapable of doing so. To address this problem, the current AI community has explored a strategy of using pre-trained small models to improve the performance of downstream tasks. These small models are specifically optimized for different downstream tasks and contain valuable task-specific knowledge in the hidden layers. Therefore, incorporating the hidden layer features of these small models into large language models (LLMs) has become an attractive research direction.

[0003] However, most existing knowledge fusion methods simply embed the features of the small model into the first layer of the LLM, which leads to significant feature level mismatch problems. This is because the first layer of the LLM mainly processes the input token-level features, while the hidden layer of the small model outputs high-level task features that have undergone complex processing. Directly aligning the two at the same level will lead to information loss or feature mismatch problems, thereby reducing model performance.

[0004] To solve this problem, it is necessary to find the optimal embedding position within the LLM. The challenge of this task is that large multimodal models are usually composed of hundreds of layers of Transformer modules, each with different feature processing capabilities. Selecting the embedding position through brute force search requires a lot of training and evaluation work, which consumes a lot of time and computing resources.

[0005] The innovation of our method is that we use the bi-norm of the average parameter gradient at the initial step as the search target. This method avoids the full model training process and task evaluation required in traditional search methods, thereby significantly saving search time. By quickly calculating the bi-norm of the gradient of each layer in the pre-fine-tuning stage, we can effectively identify the optimal embedding position without multiple repeated training or model evaluation. In addition, since this method does not rely on specific evaluation metrics, its task independence makes it scalable and adaptable to different downstream tasks.

[0006] This method not only improves the efficiency of finding the optimal embedding position, but also provides a more flexible model architecture adjustment strategy, so that large multimodal models can better integrate knowledge from small models of specific tasks, thereby improving the performance of downstream tasks. Summary of the invention

[0007] In view of this, in order to solve the above technical problems, the present invention provides a fast search method for the optimal embedding position of a large model, which has the characteristics of fast search speed and strong task scalability.

[0008] A fast search method for the optimal embedding position of a large model comprises the following steps:

[0009] Step 1: Select N representative samples of the task dataset and construct a representative set for searching the optimal embedding position, where N should not be too large.

[0010] Step 2: Input all samples of the representative set to the multimodal large model (MLLM) and the small model for the specific task at the same time, and introduce a feature fusion module to fuse the features of the i-th Transformer module of the MLLM with the features of the final layer of the small model;

[0011] Step 3: Freeze the parameters of the small model and perform k-step efficient fine-tuning on all parameters of the MLLM, where k should not be too large.

[0012] Step 4: At each step of fine-tuning, calculate the second norm of the gradient vector of the parameter with respect to the loss function, obtain k gradient second norms, and perform weighted summation of the k gradient second norms to obtain the fitness;

[0013] Step 5, replace the random number seed, repeat steps 2 to 4, and perform P repeated experiments, where P should not be too large or too small, and 3 to 30 times is appropriate;

[0014] Step 6: Iterate i in the range of 1 to T, repeat 2 to 5 steps for each iteration, get the average fitness of different embedding positions, and finally select i with the largest average fitness * as the optimal embedding position, where T is the total number of MLLM layers.

[0015] Specifically, the step of selecting K representative samples includes the following steps:

[0016] Let I represent the input table image and T represent the corresponding table sequence. Use the image encoder ImageEncoder and text encoder TextEncoder in the image and text feature extractor CLIP to extract the features of I and T respectively:

[0017] f I =ImageEncoder(I),f T =ImageEncoder(T),

[0018] Combine image features and text features into a tabular feature representation of a single sample:

[0019] f table =[f I ,f T ],

[0020] Perform feature extraction on the entire data set to obtain a matrix F, in which each row is a tabular feature representation of a sample:

[0021]

[0022] Use the K-means clustering algorithm to cluster the feature matrix F and obtain K cluster centers:

[0023] C={c1,...,c K},

[0024] Take all centers in C as K representative samples;

[0025] Optionally, the K-means clustering method may also adopt a K nearest neighbor clustering method.

[0026] Specifically, the feature fusion module is defined as:

[0027] Assume that the intermediate layer feature representation of the large model is The final layer features of the small model are represented as K and where d Q and d KV are the dimensions of the large model and small model features, respectively;

[0028] Project the intermediate layer features of the large model to the same dimension d as the small model KV :

[0029] Q proj =ReLU(LayerNorm(W1Q+b1)),

[0030] in, is the projection matrix;

[0031] Use the projected Q proj Perform multi-head attention calculation with K and V to obtain the fused key-value features:

[0032] A output =MultiHeadAttention(Q proj ,K,V),

[0033] in,

[0034] Splicing Q proj and A output , and then calculate the gating value through the gating network:

[0035] G=Sigmoid(LayerNorm(W2[Q proj ; A output ]+b2)),

[0036] in,

[0037] A output Expand to the dimension of Q by linear interpolation:

[0038] A interp =Interpolate(A output ,size=d Q ),

[0039] Finally, the fused new query features are generated:

[0040] Q new =G⊙Q+(1-G)⊙A interp ,

[0041] Here, ⊙ represents element-wise multiplication.

[0042] Specifically, the efficient parameter fine-tuning method adopts an efficient parameter fine-tuning technology based on low-rank adaptation.

[0043] Specifically, the fitness E (i) is defined as:

[0044]

[0045] in, L (g,i) is the loss function value of the g-th step when the embedding position is i,

[0046] The i with the largest average fitness * Defined as:

[0047] BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a fast search method for the optimal embedding position of large models; DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] In order to better understand the technical content of this embodiment, the explanation of terms involved in this embodiment is first introduced.

[0051] Representative samples: In machine learning and data analysis, a sample is a part of a dataset that effectively represents the characteristics and distribution of the entire dataset. Selecting representative samples helps reduce computational complexity and improve model training efficiency.

[0052] Multimodal Large Language Model (MLLM): refers to a deep learning model that can simultaneously process and understand multiple types (modalities) of data (such as text, images, audio, etc.). MLLM aims to fuse information from different modalities to achieve more complex and advanced tasks.

[0053] Feature fusion module: In deep learning, the feature fusion module is used to integrate features from different sources or levels in order to use this information for better decision making or prediction. It can be a simple weighted average, concatenation, or a more complex combination.

[0054] Transformer module: A neural network structure for natural language processing and multimodal tasks, which contains multiple layers of self-attention mechanisms and feedforward neural networks. Transformer modules are often used in deep learning to process sequence data and achieve efficient contextual information capture.

[0055] Freezing parameters: In model training, it refers to fixing some or all parameters of the model so that they remain unchanged during the training process. This is usually used in the transfer learning or fine-tuning stage to avoid destroying the characteristics of the existing pre-trained model.

[0056] Efficient parameter fine-tuning: In deep learning, it refers to partial or selective adjustment of model parameters to optimize model performance on new tasks while reducing the demand for computing resources. It is often used in large model applications to speed up training and improve adaptability.

[0057] The second norm of the gradient vector: Indicates the size (or length) of the gradient vector, which is usually used to measure the parameter adjustment step of the model. In calculation, the second norm of the gradient is the square root of the sum of the squares of all elements of the vector.

[0058] Fitness: In machine learning optimization, fitness refers to the performance of a model or a configuration under certain conditions. It is usually measured by the loss function value or other indicators. The higher the fitness, the more suitable the model is for the current task.

[0059] Random seed: used to ensure repeatability in randomized algorithms, that is, running the same code different times will produce the same results. By fixing the random seed, experimental results can be more easily reproduced.

[0060] Repeated experiments: refers to the experiment being run multiple times under different random initialization conditions to ensure the stability and robustness of the experimental results.

[0061] Average fitness: The fitness results of multiple experiments are averaged to obtain the overall performance under the experimental conditions. A high average fitness indicates that the condition performs well in multiple experiments.

[0062] Embedding location: The location where a feature or module is inserted in the model, usually referring to the location where features are input or fused at a specific layer or stage.

[0063] Total number of MLLM layers: refers to the total number of Transformer modules in the multimodal large model, which determines the range of embedding positions that can be selected.

[0064] CLIP (Contrastive Language-Image Pre-Training) is a multimodal (text-image) deep learning model proposed by OpenAI. CLIP can understand and process the relationship between text and images, and demonstrates superior capabilities in a variety of tasks. The core idea of ​​CLIP is to pre-train a large number of image and text pairs so that the model can map text descriptions and image content into a shared embedding space. In this way, CLIP can not only search for images through text, but also generate descriptions based on images.

[0065] ReLU (Rectified Linear Unit): is an activation function commonly used in deep learning.

[0066] Multi-head Attention: It is an attention mechanism widely used in deep learning, especially in the Transformer structure. It enhances the model's ability to capture different semantic relationships and features by computing multiple independent attention heads in parallel.

[0067] Implementation 1:

[0068] Figure 1 A flowchart of Embodiment 1 of the present invention is shown. A method for quickly searching for an optimal embedding position of a large model comprises the following steps:

[0069] Step 1: Select N representative samples of the task dataset and construct a representative set for searching the optimal embedding position, where N should not be too large.

[0070] Step 2: Input all samples of the representative set to the multimodal large model (MLLM) and the small model for the specific task at the same time, and introduce a feature fusion module to fuse the features of the i-th Transformer module of the MLLM with the features of the final layer of the small model;

[0071] Step 3: Freeze the parameters of the small model and perform k-step efficient fine-tuning on all parameters of the MLLM, where k should not be too large.

[0072] Step 4: At each step of fine-tuning, calculate the second norm of the gradient vector of the parameter with respect to the loss function, obtain k gradient second norms, and perform weighted summation of the k gradient second norms to obtain the fitness;

[0073] Step 5, replace the random number seed, repeat steps 2 to 4, and perform P repeated experiments, where P should not be too large or too small, and 3 to 30 times is appropriate;

[0074] Step 6: Iterate i in the range of 1 to T, repeat 2 to 5 steps for each iteration, get the average fitness of different embedding positions, and finally select i with the largest average fitness * as the optimal embedding position, where T is the total number of MLLM layers.

[0075] Specifically, the step of selecting K representative samples includes the following steps:

[0076] Let I represent the input table image and T represent the corresponding table sequence. Use the image encoder ImageEncoder and text encoder TextEncoder in the image and text feature extractor CLIP to extract the features of I and T respectively:

[0077] f I =ImageEncoder(I),fT =ImageEncoder(T),

[0078] Combine image features and text features into a tabular feature representation of a single sample:

[0079] f table =[f I ,f T ],

[0080] Perform feature extraction on the entire data set to obtain a matrix F, in which each row is a tabular feature representation of a sample:

[0081]

[0082] Use the K-means clustering algorithm to cluster the feature matrix F and obtain K cluster centers:

[0083] C={c1,...,c K},

[0084] Take all centers in C as K representative samples;

[0085] Optionally, the K-means clustering method may also adopt a K nearest neighbor clustering method.

[0086] Specifically, the feature fusion module is defined as:

[0087] Assume that the intermediate layer feature representation of the large model is The final layer features of the small model are represented as K and where d Q and d KV are the dimensions of the large model and small model features, respectively;

[0088] Project the intermediate layer features of the large model to the same dimension d as the small model KV :

[0089] Q proj =ReLU(LayerNorm(W1Q+b1)),

[0090] in, is the projection matrix;

[0091] Use the projected Q proj Perform multi-head attention calculation with K and V to obtain the fused key-value features:

[0092] A output =MultiHeadAttention(Q proj ,K,V),

[0093] in,

[0094] Splicing Q proj and A output , and then calculate the gating value through the gating network:

[0095] G=Sigmoid(LayerNorm(W2[Q proj ; A output ]+b2)),

[0096] in,

[0097] A output Expand to the dimension of Q by linear interpolation:

[0098] A interp =Interpolate(A output ,size=d Q ),

[0099] Finally, the fused new query features are generated:

[0100] Q new =G⊙Q+(1-G)⊙A interp ,

[0101] Here, ⊙ represents element-wise multiplication.

[0102] Specifically, the efficient parameter fine-tuning method adopts an efficient parameter fine-tuning technology based on low-rank adaptation.

[0103] Specifically, the fitness E (i) is defined as:

[0104]

[0105] in, L (g,i) is the loss function value of the g-th step when the embedding position is i,

[0106] Optionally, we can also use the 1-norm ||·||1 to define E (i) .

[0107] The i with the largest average fitness * Defined as:

[0108]

[0109] The technical features or steps of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features or steps in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

Claims

1. A fast search method for the optimal embedding position of a large model, characterized in that: The method comprises: A selects K representative samples of the task's data set and constructs a representative set for searching the optimal embedding position, where K should not be too large; B will represent all samples in the set and input them to the multimodal large model (MLLM) and the small model for specific tasks at the same time, and introduce a feature fusion module to fuse the features of the i-th Transformer module of the MLLM with the features of the final layer of the small model; C freezes the parameters of the small model and performs k-step efficient fine-tuning of all parameters of the MLLM, where k should not be too large; At each step of D fine-tuning, the second norm of the gradient vector of the parameter with respect to the loss function is calculated to obtain k gradient second norms, and the weighted sum of the k gradient second norms is obtained to obtain the fitness E (i) ; E. Replace the random number seed, repeat steps 2 to 4, and perform P repeated experiments, where P should not be too large or too small, and 3 to 30 times is appropriate; F iterates i in the range of 1 to T, repeats each iteration for 2 to 5 steps, obtains the average fitness of different embedding positions, and finally selects i with the largest average fitness. * as the optimal embedding position, where T is the total number of MLLM layers.

2. The fast search method for the optimal embedding position of a large model as claimed in claim 1, characterized in that: The A includes: A1 Let I represent the input table image and T represent the corresponding table sequence. Use the image encoder ImageEncoder and text encoder TextEncoder in the image and text feature extractor CLIP to extract the features of I and T respectively: f I =ImageEncoder(I),f T =ImageEncoder(T), A2 combines image features and text features into a tabular feature representation of a single sample: f table =[f I ,f T ], A3 extracts features from the entire data set and obtains a matrix F, in which each row is a tabular feature representation of a sample: A4 uses the K-means clustering algorithm to cluster the feature matrix F and obtain K cluster centers: C={c1,...,c K }, Take all centers in C as K representative samples.

3. The fast search method for the optimal embedding position of a large model as claimed in claim 2, characterized in that: The feature fusion module is defined as follows: Assume that the intermediate layer feature representation of the large model is The final layer features of the small model are represented as K and where d Q and d KV are the dimensions of the large model and small model features, respectively; Project the intermediate layer features of the large model to the same dimension d as the small model KV : Q proj =ReLU(LayerNorm(W1Q+b1)), in, is the projection matrix; Use the projected Q proj Perform multi-head attention calculation with K and V to obtain the fused key-value features: A output =MultiHeadAttention(Q proj ,K,V), in, Splicing Q proj and A output , and then calculate the gating value through the gating network: G=Sigmoid(LayerNorm(W2[Q proj ;A output ]+b2)), in, A output Expand to the dimension of Q by linear interpolation: A interp =Interpolate(A output ,size=d Q ), Finally, the fused new query features are generated: Q new =G⊙Q+(1-G)⊙A interp , Here, ⊙ represents element-wise multiplication.

4. The fast search method for the optimal embedding position of a large model as claimed in claim 3, characterized in that: The C is: A low-rank decomposition method with mixed sparse gradients is used to efficiently fine-tune the model parameters; The goal is to find the optimal parameter vector w* by optimizing the function f(w), so that: Divide all parameters into two parts: W=[W1,W2], Where W1 is the newly added subset of key parameters, and W2 is the original model parameters that remain frozen during training; The low-rank matrix representation is used to efficiently adjust the parameters, and the update direction is expressed as: Δw=UV T Among them, U, V are the parameter matrices after low-rank decomposition; Using sparse gradient estimation, only the subset of parameters with the largest gradient magnitude is updated Then, only the parameters in S are updated: Where η is the learning rate, and d is the dimension of the low-rank decomposition that is preset as needed; In order to avoid overfitting, a regularization term is added: f(w)=L(w)+λ||w||2, Among them, λ is the regularization parameter, which is used to control the degree of regularization. Based on the above, the update formula of the efficient parameter search method is: Among them, I St is a sparse matrix, retaining only the positions with large gradients, ⊙ is element-by-element multiplication, U t , V t is the current estimate of the low-rank matrix. This method combines low-rank representation with sparse gradient updates, which greatly reduces the computational and memory requirements during optimization and is suitable for efficient training and search tasks of large-scale parameter models.

5. The fast search method for the optimal embedding position of a large model as claimed in claim 4, characterized in that: The fitness E (i) is defined as: in, L (g,i) is the loss function value of the g-th step when the embedding position is i, 6. The fast search method for the optimal embedding position of a large model as claimed in claim 5, characterized in that: The i with the largest average fitness * Defined as: Among them, argmax means that E (i) The superscript i when the maximum value is taken.

7. The fast search method for the optimal embedding position of a large model as claimed in claim 6, characterized in that: In step C, the prompt word technology used in fine-tuning is as follows: In order to make full use of the emerging self-awareness of the big model, when searching for the optimal embedding layer, asking the big model about its own optimal layer access position can improve the search effect. The prompt words are designed as follows: "The current goal is to search for the optimal embedding layer of the small model based on gradient. The specific task is: " + [detailed description of the task].

Images