A fast search method for optimal embedding locations in large models

Abstract

This invention discloses a fast search method for the optimal embedding position of a large model. The method includes: selecting K representative samples from the task's dataset to construct a representative set for searching the optimal embedding position, where K should not be too large; simultaneously inputting all samples from the representative set into a multimodal large model (MLLM) and a task-specific small model, and introducing a feature fusion module to fuse the features of the i-th Transformer module of the MLLM and the features of the final layer of the small model; freezing the parameters of the small model and performing k-step efficient parameter fine-tuning on all parameters of the MLLM, where k should not be too large; in each step of fine-tuning, calculating the L2 norm of the gradient vector of the parameters with respect to the loss function to obtain k gradient L2 norms, and then weighted summing the k gradient L2 norms to obtain the fitness; changing the random number seed and repeating steps 2 to 4 for P repeated experiments, where P should not be too large or too small, and 3 to 30 iterations are preferable; iterating over i within the range of 1 to T, repeating each iteration 2 to 5 times to obtain the average fitness of different embedding positions, and finally selecting i with the largest average fitness. * The optimal embedding position is defined as T, where T is the total number of MLLM layers. The fast search method for the optimal embedding position of a large model provided by this invention features high search speed and strong task scalability, making it particularly suitable for scenarios requiring rapid knowledge fusion between small models and large models.

Description

Technical Field

[0001] This invention relates to the fields of computer science, large models, multimodal large models, artificial intelligence, deep learning, and pattern recognition, and particularly to a fast search method for the optimal embedding position of a large model. Background Technology

[0002] Multimodal large models are typically pre-trained on massive general datasets, making them very powerful when handling a wide range of general tasks. However, when applied to specific downstream tasks, general pre-trained models often fall short. To address this issue, the current AI community has explored a strategy of leveraging pre-trained small models to improve downstream task performance. These small models are specifically optimized for different downstream tasks and contain valuable task-specific knowledge in their hidden layers. Therefore, incorporating the hidden layer features of these small models into large language models (LLMs) has become an attractive research direction.

[0003] However, most existing knowledge fusion methods simply embed the features of the smaller model into the first layer of the LLM, which leads to a significant feature level mismatch problem. This is because the first layer of the LLM mainly processes the input token-level features, while the hidden layers of the smaller model output high-level task features that have undergone complex processing. Directly aligning these two at the same level can result in information loss or feature mismatch, thereby reducing model performance.

[0004] To address this issue, finding the optimal embedding location within the LLM becomes essential. The challenge lies in the fact that large multimodal models typically consist of hundreds of Transformer modules, each with different feature processing capabilities. Using brute-force search methods for embedding location selection requires extensive training and evaluation, consuming significant time and computational resources.

[0005] The innovation of our method lies in using the L2 norm of the average parameter gradient in the initial step as the search target. This approach avoids the complete model training and task evaluation processes required in traditional search methods, thus significantly saving search time. By rapidly calculating the L2 norm of the gradient at each layer during the pre-fine-tuning phase, we can effectively identify the optimal embedding position without repeated training or model evaluation. Furthermore, since this method is independent of specific evaluation metrics, its task independence makes it highly scalable and adaptable to various downstream tasks.

[0006] This approach not only improves the efficiency of finding the optimal embedding position, but also provides a more flexible model architecture adjustment strategy, enabling large multimodal models to better integrate knowledge from small models for specific tasks, thereby improving the performance of downstream tasks. Summary of the Invention

[0007] In view of this, and to address the aforementioned technical problems, this invention provides a fast search method for the optimal embedding position of a large model, which features fast search speed and strong task scalability.

[0008] A fast method for finding the optimal embedding position in a large model includes the following steps:

[0009] Step 1, select the dataset for the task. N We construct a representative set using 3 representative samples to search for the optimal embedding position, where N It should not be too large;

[0010] Step 2 involves inputting all samples from the representative set into both the Multimodal Large Model (MLLM) and the task-specific small model, and introducing a feature fusion module to perform the MLLM's first step. The features of each Transformer module are fused with the features of the final layer of the small model;

[0011] Step 3: Freeze the parameters of the small model and perform adjustments on all parameters of the MLLM. k The parameters of the step are efficiently fine-tuned, among which It should not be too large;

[0012] Step 4: For each step of fine-tuning, calculate the L2 norm of the gradient vector of the parameters with respect to the loss function, and obtain... The gradient 2 norm, and for The fitness is obtained by weighted summation of the gradient L2 norms;

[0013] Step 5: Change the random number seed and repeat steps 2 to 4. Repeated experiments, of which The number of repetitions should be neither too large nor too small, and 3 to 30 repetitions are appropriate.

[0014] Step 6, for In 1 to The iteration is performed within the range of values, with each iteration repeated 2 to 5 times to obtain the average fitness for different embedding positions. Finally, the embedding position with the highest average fitness is selected. As the optimal embedding position, where This represents the total number of layers in the MLLM.

[0015] Specifically, the selection A representative sample is obtained, including the following steps:

[0016] Let I represent the input table image, and T represent the corresponding table sequence. The ImageEncoder and TextEncoder in the CLIP image-text feature extractor are used to extract features from I and T respectively:

[0017] ,

[0018] Combining image features and text features into a tabular feature representation of a single sample:

[0019] ,

[0020] Feature extraction is performed on the entire dataset to obtain a matrix. Each row represents a sample's tabular features:

[0021] ,

[0022] The K-means clustering algorithm is used to cluster the feature matrix. Clustering is performed to obtain Cluster centers:

[0023]

[0024] All centers in C are considered as A representative sample;

[0025] Alternatively, the K-means clustering method can also employ the K-nearest neighbor clustering method.

[0026] Specifically, the feature fusion module is defined as:

[0027] Let the intermediate layer features of the large model be represented as follows: The final layer features of the small model are represented as K and ,in and These are the dimensions of the features of the large model and the small model, respectively;

[0028] Project the intermediate layer features of the large model to the same dimension as the small model. :

[0029]

[0030] in, It is the projection matrix;

[0031] After using projection and Multi-head attention computation is performed to obtain fused key-value features.

[0032] ,

[0033] in, ;

[0034] splicing and Then, the gating value is calculated through the gating network:

[0035] ,

[0036] in, ;

[0037] Will Extended by linear interpolation Dimensions:

[0038] ,

[0039] Finally, the fused new query features are generated:

[0040] ,

[0041] in, This represents element-wise multiplication.

[0042] Specifically, the parameter fine-tuning method employs a high-efficiency parameter fine-tuning technique based on low-rank adaptation.

[0043] Specifically, the fitness The definition of is:

[0044]

[0045] in, , For the embedding position is The first time The loss function value of the step. .

[0046] The one with the highest average fitness Defined as:

[0047] . Attached Figure Description

[0048] Figure 1 It is a fast search method for the optimal embedding position in large models; Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. It should be understood that the specific embodiments described herein are only for explaining this invention and are not intended to limit this invention.

[0050] To better understand the technical content of this embodiment, let's first introduce the explanation of the terms involved in this embodiment.

[0051] Representative samples: In machine learning and data analysis, representative samples refer to a subset of samples selected from a dataset that effectively represent the characteristics and distribution of the entire dataset. Selecting representative samples helps reduce computational complexity and improve the training efficiency of the model.

[0052] Multimodal Large Language Model (MLLM): This refers to a deep learning model that can simultaneously process and understand multiple types (modalities) of data (such as text, images, audio, etc.). MLLM aims to fuse information from different modalities to achieve more complex and advanced tasks.

[0053] Feature fusion module: In deep learning, the feature fusion module is used to integrate features from different sources or levels in order to make better decisions or predictions using this information. It can be a simple weighted average, concatenation, or a more complex combination.

[0054] Transformer module: A neural network architecture for natural language processing and multimodal tasks, consisting of multiple layers composed of self-attention mechanisms and feedforward neural networks. In deep learning, Transformer modules are commonly used to process sequential data and achieve efficient capture of contextual information.

[0055] Parameter freezing: In model training, this refers to fixing some or all of the model's parameters so that they remain unchanged during training. This is typically used in transfer learning or fine-tuning phases to avoid breaking the characteristics of the existing pre-trained model.

[0056] Efficient parameter fine-tuning: In deep learning, this refers to partially or selectively adjusting model parameters to optimize model performance on new tasks while reducing computational resource requirements. It is typically used in large-scale model applications to accelerate training and improve adaptability.

[0057] The 2-norm of the gradient vector: represents the magnitude (or length) of the gradient vector, and is often used to measure the step size of the model's parameter adjustments. In calculation, the 2-norm of the gradient is the square root of the sum of the squares of all elements of the vector.

[0058] Fitness: In machine learning optimization, fitness refers to the performance metric of a model or configuration under certain conditions. It is usually measured by the loss function value or other metrics; a higher fitness indicates that the model is more suitable for the current task.

[0059] Random number seed: Used to ensure repeatability in randomized algorithms, meaning that different runs of the same code will produce the same results. By fixing the random seed, experimental results can be more easily reproduced.

[0060] Repeated experiments: refers to running an experiment multiple times under different random initialization conditions to ensure the stability and robustness of the experimental results.

[0061] Average fitness: The average fitness results from multiple experiments are used to obtain the overall performance under the experimental conditions. A high average fitness indicates that the condition performs well in multiple experiments.

[0062] Embedding location: The location where features or modules are inserted into the model, usually referring to the location where features are input or fused at a specific layer or stage.

[0063] Total number of MLLM layers: This refers to the total number of Transformer modules in a large multimodal model, which determines the range of embedding locations that can be selected.

[0064] CLIP (Contrastive Language-Image Pre-Training) is a multimodal (text-image) deep learning model proposed by OpenAI. CLIP can understand and process the relationships between text and images, and demonstrates superior capabilities in various tasks. The core idea of ​​CLIP is to pre-train on a large number of image and text pairs, enabling the model to map text descriptions and image content into a shared embedding space. In this way, CLIP can not only perform image search from text, but also generate descriptions from images.

[0065] ReLU (Rectified Linear Unit) is a commonly used activation function in deep learning.

[0066] Multi-head attention is a widely used attention mechanism in deep learning, especially in the Transformer architecture. It enhances the model's ability to capture different semantic relationships and features by computing multiple independent attention heads in parallel.

[0067] Implementation

[0068] Figure 1 A flowchart illustrating Embodiment 1 of the present invention is shown. A fast search method for the optimal embedding position of a large model includes the following steps:

[0069] Step 1, select the dataset for the task. N We construct a representative set using 3 representative samples to search for the optimal embedding position, where It should not be too large;

[0070] Step 2 involves inputting all samples from the representative set into both the Multimodal Large Model (MLLM) and the task-specific small model, and introducing a feature fusion module to perform the MLLM's first step. The features of each Transformer module are fused with the features of the final layer of the small model;

[0071] Step 3: Freeze the parameters of the small model and perform adjustments on all parameters of the MLLM. The parameters of the step are efficiently fine-tuned, among which It should not be too large;

[0072] Step 4: For each step of fine-tuning, calculate the L2 norm of the gradient vector of the parameters with respect to the loss function, and obtain... k The gradient 2 norm, and for The fitness is obtained by weighted summation of the gradient 2 norms;

[0073] Step 5: Change the random number seed and repeat steps 2 to 4. Repeated experiments, of which The number of repetitions should be neither too large nor too small, and 3 to 30 repetitions are appropriate.

[0074] Step 6, for In 1 to The iteration is performed within the range of values, with each iteration repeated 2 to 5 times to obtain the average fitness for different embedding positions. Finally, the embedding position with the highest average fitness is selected. As the optimal embedding position, where This represents the total number of layers in the MLLM.

[0075] Specifically, the selection K A representative sample is obtained, including the following steps:

[0076] Let I represent the input table image, and T represent the corresponding table sequence. The ImageEncoder and TextEncoder in the CLIP image-text feature extractor are used to extract features from I and T respectively:

[0077] ,

[0078] Combining image features and text features into a tabular feature representation of a single sample:

[0079] ,

[0080] Feature extraction is performed on the entire dataset to obtain a matrix. Each row represents a sample's tabular features:

[0081] ,

[0082] The K-means clustering algorithm is used to cluster the feature matrix. Clustering is performed to obtain Cluster centers:

[0083]

[0084] All centers in C are considered as A representative sample;

[0085] Alternatively, the K-means clustering method can also employ the K-nearest neighbor clustering method.

[0086] Specifically, the feature fusion module is defined as:

[0087] Let the intermediate layer features of the large model be represented as follows: The final layer features of the small model are represented as K and ,in and These are the dimensions of the features of the large model and the small model, respectively;

[0088] Project the intermediate layer features of the large model to the same dimension as the small model. :

[0089] ,

[0090] in, It is the projection matrix;

[0091] After using projection and Multi-head attention computation is performed to obtain the fused key-value features:

[0092] ,

[0093] in, ;

[0094] splicing and Then, the gating value is calculated through the gating network:

[0095] ,

[0096] in, ;

[0097] Will Extended by linear interpolation Dimensions:

[0098] ,

[0099] Finally, the fused new query features are generated:

[0100] ,

[0101] in, This represents element-wise multiplication.

[0102] A low-rank decomposition method using hybrid sparse gradients is employed to efficiently fine-tune the model's parameters.

[0103] The goal is to optimize the function Find the optimal parameter vector , so that:

[0104] ,

[0105] Divide all parameters into two parts:

[0106] ,

[0107] in It is a newly added subset of key parameters. These are the original model parameters that are kept frozen during training;

[0108] Efficient parameter tuning is achieved using low-rank matrix representation, with the update direction expressed as:

[0109]

[0110] Where U and V are the parameter matrices after low-rank decomposition;

[0111] Using the sparse gradient estimation method, only the subset of parameters with the largest gradient magnitude is updated. :

[0112]

[0113] Then, only for S Update the parameters in:

[0114] ,

[0115] in It's the learning rate. It is the dimension of the low-rank decomposition preset according to the needs;

[0116] To avoid overfitting, a regularization term is added:

[0117] ,

[0118] in, This is the regularization parameter, used to control the degree of regularization.

[0119] In summary, the update formula for the efficient parameter search method is:

[0120]

[0121] in, It is a sparse matrix, retaining only the positions with larger gradients. It is element-wise multiplication. , This is the current estimate of the low-rank matrix. This method combines low-rank representation with sparse gradient updates, which significantly reduces the computation and memory requirements during optimization, making it suitable for efficient training and search tasks of large-scale parameter models.

[0122] Specifically, the parameter fine-tuning method employs a high-efficiency parameter fine-tuning technique based on low-rank adaptation.

[0123] Specifically, the fitness The definition of is:

[0124]

[0125] in, , For the embedding position is The first time The loss function value of the step. ;

[0126] Alternatively, the 1-norm can also be used. definition .

[0127] The one with the highest average fitness Defined as:

[0128] .

[0129] The technical features or steps of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features or steps in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A fast search method for the optimal embedding position of a large model, characterized in that, The method includes: A. Selecting the dataset for the task We construct a representative set using 10 representative samples to search for the optimal embedding position. B represents all samples in the set and is simultaneously input into both the Multimodal Large Model (MLLM) and the task-specific small model. A feature fusion module is also introduced to perform feature fusion on the MLLM. The features of each Transformer module are fused with the features of the final layer of the small model; C freezes the parameters of the small model and performs a process on all parameters of the MLLM. Efficiently fine-tuning of parameters in each step; D. For each step of fine-tuning, the L2 norm of the gradient vector of the parameters with respect to the loss function is calculated, resulting in... The gradient 2 norm, and for The fitness is obtained by weighted summation of the gradient 2-norm numbers. ; E. Change the random number seed and repeat steps B to D. Repeat the experiment; F to In 1 to The iteration is performed within the range of values, repeating steps B to E in each iteration to obtain the average fitness for different embedding positions. Finally, the embedding position with the highest average fitness is selected. As the optimal embedding position, where This represents the total number of layers in the MLLM; A1 Let I represent the input table image and T represent the corresponding table sequence. Use the ImageEncoder and TextEncoder in the CLIP feature extractor to extract features from I and T respectively: ； A2 combines image features and text features into a tabular feature representation of a single sample: ； A3 performs feature extraction on the entire dataset to obtain a matrix. Each row represents a sample's tabular features: ； A4 uses the K-means clustering algorithm on the feature matrix. Clustering is performed to obtain Cluster centers: ； All centers in C are considered as A representative sample.

2. The fast search method for the optimal embedding position of a large model as described in claim 1, characterized in that, The feature fusion module is defined as follows: Let the intermediate layer features of the large model be represented as follows: The final layer features of the small model are represented as K and ,in and These are the dimensions of the features of the large model and the small model, respectively; Project the intermediate layer features of the large model to the same dimension as the small model. : ； in, It is a projection matrix; It is the bias vector; After using projection and Multi-head attention computation is performed to obtain the fused key-value features: ， in, ; splicing and Then, the gating value is calculated through the gating network: ， in, ; It is the bias vector; Will Extended by linear interpolation Dimensions: ， Finally, the fused new query features are generated: ， in, This represents element-wise multiplication.

3. The fast search method for the optimal embedding position of a large model as described in claim 2, characterized in that, The C is: A low-rank decomposition method using hybrid sparse gradients is employed to efficiently fine-tune the model's parameters. The goal is to optimize the function Find the optimal parameter vector , so that: ， It is the parameter vector to be optimized; Divide all parameters into two parts: ， in It is a newly added subset of key parameters. These are the original model parameters that are kept frozen during training; Efficient parameter tuning is achieved using low-rank matrix representation, with the update direction expressed as: ； Where U and V are the parameter matrices after low-rank decomposition; Using the sparse gradient estimation method, only the subset of parameters with the largest gradient magnitude is updated. : ； The number of Transformer layers in the MLLM; Then, only for S Update the parameters in: ， in It's the learning rate. It is the dimension of the low-rank decomposition preset according to the needs. It is the first Layer Transformer projection matrix parameters; To avoid overfitting, a regularization term is added: ， in, This is the regularization parameter, used to control the degree of regularization. This represents the loss function substituted into the parameter vector w to be optimized; In summary, the update formula for the efficient parameter search method is: ； in, It is a sparse matrix, retaining only the positions with larger gradients. It is element-wise multiplication. , This is the current estimate of the low-rank matrix. This method combines low-rank representation with sparse gradient updates, which significantly reduces the computation and memory requirements during optimization, making it suitable for efficient training and search tasks of large-scale parameter models.

4. The fast search method for the optimal embedding position of a large model as described in claim 3, characterized in that, The fitness The definition of is: ； in, , For the embedding position is The first time The loss function value of the step. ； loss function L The first in Function parameters.

5. The fast search method for the optimal embedding position of a large model as described in claim 4, characterized in that, The highest average fitness Defined as: ， Where argmax represents the expression that makes Superscript when taking the maximum value .

6. The fast search method for the optimal embedding position of a large model as described in claim 5, characterized in that, In step C, the prompting word technology used during fine-tuning is as follows: To fully leverage the self-awareness of large models, when searching for the optimal embedding layer, the system queries the large model for its own optimal layer access position, thereby improving search performance. The prompt words are designed as follows: The current goal is to find the optimal embedding layer for a small model based on gradient search. The specific task is: + [Detailed description of the task].

Images