Multi-modal large language model task adaptation fine tuning method and related device
By introducing a trainable adapter layer and a lexical classification model into a multimodal large language model, and dynamically processing lexical types, the catastrophic forgetting problem of multimodal large language models when learning new tasks is solved, achieving efficient task adaptation fine-tuning and improving the model's adaptability and generalization performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-10
- Publication Date
- 2026-03-27
AI Technical Summary
Multimodal large language models are prone to catastrophic forgetting when learning new target tasks or data, and existing efficient parameter fine-tuning methods cannot effectively alleviate the problem of limited adaptive learning ability of the model.
Introducing a parameter-trainable adapter layer and a learnable lexical classification model into the pre-defined weight layer of a multimodal large language model, dynamically determining whether each input lexical should be adapted or rejected, and processing different types of lexicals through the multimodal large language model and the adapter layer to achieve fine-grained task adaptation and protect pre-trained knowledge.
It significantly alleviates the catastrophic forgetting problem during fine-tuning, improves the adaptability of multimodal large language models to new tasks, and maintains the performance of the original pre-training tasks, achieving efficient parameter utilization and strong generalization ability.
Smart Images
Figure CN121745159A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of artificial intelligence, and relates to a multi-modal large language model task adaptation fine-tuning method and a related device. BACKGROUND
[0002] A multi-modal large language model performs well in many downstream tasks through large-scale pre-training learning. However, due to the huge parameter scale and complex training data, the model is prone to lose the representation ability of the previously learned knowledge when learning new target tasks or data, which is called catastrophic forgetting.
[0003] To address the problem of catastrophic forgetting of the multi-modal large language model, a parameter-efficient fine-tuning method can increase the computational efficiency while alleviating the problem of catastrophic forgetting to some extent. The parameter-efficient fine-tuning method mainly controls the influence of the learning of new tasks or new data on the original knowledge of the multi-modal large language model by reducing the modified model parameters in the fine-tuning process, so as to avoid or reduce catastrophic forgetting as much as possible.
[0004] The existing parameter-efficient fine-tuning methods include the first type, such as the adapter-based fine-tuning method, which only allows the adapter parameter model to learn and update, and the second type, such as the low-rank adaptation-based fine-tuning method, which introduces two low-rank matrices to approximate the update amount of the weights. However, these existing parameter-efficient fine-tuning methods are limited in model adaptive learning ability due to the static task allocation architecture, and the ability to alleviate catastrophic forgetting is insufficient. SUMMARY
[0005] The application aims to overcome the shortcomings of the prior art and provide a multi-modal large language model task adaptation fine-tuning method and a related device.
[0006] To achieve the above-mentioned purpose, the application adopts the following technical solutions: In a first aspect, the application provides a multi-modal large language model task adaptation fine-tuning method, which includes: obtaining a training sample of a task to be adapted; wherein the training sample is a text sample and / or an image sample; introducing a parameter trainable adapter layer at a preset weight layer of a multi-modal large language model; obtaining a token sequence of the training sample based on the preset weight layer of the multi-modal large language model, and calling a preset token classification model to obtain the types of tokens in the token sequence, and processing the tokens in the token sequence according to the types of the tokens to obtain the output of the training sample, and updating the parameters of the adapter layer and the parameters of the token classification model according to the output of the training sample; wherein the tokens are processed by the multi-modal large language model and the adapter layer when the types of the tokens are adaptation; the tokens are processed by the multi-modal large language model when the types of the tokens are rejection; and a multi-modal large language model for task adaptation fine-tuning is obtained based on the multi-modal large language model, the final adapter layer and the final token classification model.
[0007] Optionally, the introduction of a parameter-trainable adapter layer into the preset weight layer of the multimodal large language model includes: introducing a parameter-trainable adapter layer into the self-attention layer and the feedforward neural network layer of each Transformer layer of the multimodal large language model.
[0008] Optionally, the step of calling a preset lexical classification model to obtain the type of each lexical in the lexical sequence includes: calling the preset lexical classification model and obtaining the rejection score of each lexical in the lexical sequence using the following formula:
[0009] in, This is a sequence of rejection scores for each word within a word sequence. For the probability of rejection, To adapt to the probability, For word sequence, For the parameters of the word classification model, It is additive Gumbel noise. For temperature coefficient, , The attenuation rate constant is To train step count, To achieve the target low temperature, This represents the initial high temperature.
[0010] when When, the lexical type of the lexical is rejection; when At that time, the lexical type of the word is adaptation; among them, For the first The th word sequence of the th word sequence The rejection probability in the rejection score of each word element. For the first The th word sequence of the th word sequence The probability of fit in the rejection score of each word.
[0011] Optionally, the preset word classification model includes: each adapter layer has a preset word classification model, and the parameters of each word classification model are updated separately; or, all adapter layers share a single word classification model; or, every certain number of adapter layers have a preset word classification model.
[0012] Optionally, the parameter matrix of the trainable adapter layer is: :
[0013] in, This is the scaling factor. and It is a low-rank fitting matrix. Let be the rank of the low-rank fitting matrix.
[0014] Optionally, the step of processing each word in the word sequence according to the word type to obtain the output of the training sample includes: obtaining the output of the training sample using the following formula. :
[0015] in, For word sequence, These are the pre-training parameters for a multimodal large language model. For Hadama accumulation, For the mask matrix, , For the first The th word sequence of the th word sequence A mask for each lexical element, when the lexical type is rejection. When the lexical type is matching .
[0016] Optionally, obtaining the task-adapted fine-tuned multimodal large language model based on the multimodal large language model, the final adapter layer, and the final lexical classification model includes: combining the multimodal large language model, the final adapter layer, and the final lexical classification model as the task-adapted fine-tuned multimodal large language model; or, obtaining the parameter matrix of the final lexical classification model corresponding to each final adapter layer, and summing the elements of each column of the parameter matrix to obtain the rejection weight and adaptation weight of each final adapter layer; when the rejection weight of the final adapter layer... When adapting the weights, the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer are modified as follows: ;in, These are the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer. The final adapter layer's weight matrix is used; otherwise, the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer are retained; and the multimodal large language model with modified parameters is used as the multimodal large language model for task adaptation fine-tuning.
[0017] In a second aspect, the present invention provides a multimodal large language model task adaptation fine-tuning system, comprising: a sample acquisition module for acquiring training samples of the task to be adapted; wherein the training samples are text samples and / or image samples; an adapter layer introduction module for introducing a parameter-trainable adapter layer into a preset weight layer of the multimodal large language model; a fine-tuning training module for acquiring the word sequence of the training samples based on the preset weight layer of the multimodal large language model, and calling a preset word classification model to obtain the type of each word in the word sequence, and processing each word in the word sequence according to the type of the word to obtain the output of the training samples, and updating the parameters of the adapter layer and the parameters of the word classification model according to the output of the training samples; wherein, when the type of the word is adaptation, it is processed by the multimodal large language model and the adapter layer; when the type of the word is rejection, it is processed by the multimodal large language model; and a model output module for obtaining a task-adapted fine-tuned multimodal large language model based on the multimodal large language model, the final adapter layer, and the final word classification model.
[0018] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described multimodal large language model task adaptation fine-tuning method.
[0019] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described multimodal large language model task adaptation fine-tuning method.
[0020] Compared with the prior art, the present invention has the following beneficial effects: This invention presents a task adaptation and fine-tuning method for multimodal large language models. By introducing a trainable adapter layer into the pre-defined weight layer of the multimodal large language model and designing a learnable lexical classification model to dynamically determine whether each input lexical should be adapted or rejected, fine-grained, lexical-level task adaptation is achieved during fine-tuning. Adaptable lexicals are processed by both the multimodal large language model and the adapter layer to learn new task knowledge, while rejected lexicals are processed only by the multimodal large language model to preserve pre-trained knowledge. This method significantly alleviates the catastrophic forgetting problem during fine-tuning, improving the multimodal large language model's adaptability to new tasks while effectively maintaining its performance on the original pre-trained tasks. The result is a multimodal large language model with both strong generalization ability and efficient parameter utilization for task adaptation and fine-tuning. Furthermore, by introducing a parameter-trainable adapter layer and a lexical classification model for fine-tuning, only a small number of parameters are introduced, and the pre-trained parameters of the multimodal large language model are frozen during fine-tuning. This results in a short fine-tuning process with low computational resource requirements. Attached Figure Description
[0021] Figure 1 The flowchart illustrates the multimodal large language model task adaptation and fine-tuning method according to an embodiment of the present invention.
[0022] Figure 2 A detailed flowchart of the multimodal large language model task adaptation and fine-tuning method according to an embodiment of the present invention.
[0023] Figure 3 The diagram below shows the structural block of the multimodal large language model task adaptation and fine-tuning system according to an embodiment of the present invention. Detailed Implementation
[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0026] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, a method for adapting and fine-tuning multimodal large language models to tasks is provided, which can accurately and efficiently achieve the performance balance of multimodal large language models between pre-training tasks and target tasks, and provides a new approach to alleviate the catastrophic forgetting problem of multimodal large language models.
[0027] Specifically, the multimodal large language model task adaptation and fine-tuning method of the present invention includes the following steps: S1: Obtain training samples for the task to be adapted; wherein, the training samples are text samples and / or image samples.
[0028] S2: Introduce a parameter-trainable adapter layer into the preset weight layer of the multimodal large language model.
[0029] S3: Obtain the word sequence of the training sample based on the preset weight layer of the multimodal large language model, and call the preset word classification model to obtain the type of each word in the word sequence. Then, process each word in the word sequence according to the type of the word to obtain the output of the training sample, and update the parameters of the adapter layer and the parameters of the word classification model according to the output of the training sample. When the type of the word is "adaptation", it is processed by the multimodal large language model and the adapter layer. When the type of the word is "rejection", it is processed by the multimodal large language model.
[0030] S4: Based on the multimodal large language model, the final adapter layer, and the final lexical classification model, a multimodal large language model with task adaptation fine-tuning is obtained.
[0031] This invention presents a task adaptation and fine-tuning method for multimodal large language models. By introducing a trainable adapter layer into the pre-defined weight layer of the multimodal large language model and designing a learnable lexical classification model to dynamically determine whether each input lexical should be adapted or rejected, fine-grained, lexical-level task adaptation is achieved during fine-tuning. Adaptable lexicals are processed by both the multimodal large language model and the adapter layer to learn new task knowledge, while rejected lexicals are processed only by the multimodal large language model to preserve pre-trained knowledge. This method significantly alleviates the catastrophic forgetting problem during fine-tuning, improving the multimodal large language model's adaptability to new tasks while effectively maintaining its performance on the original pre-trained tasks. The result is a multimodal large language model with both strong generalization ability and efficient parameter utilization for task adaptation and fine-tuning. Furthermore, by introducing a parameter-trainable adapter layer and a lexical classification model for fine-tuning, only a small number of parameters are introduced, and the pre-trained parameters of the multimodal large language model are frozen during fine-tuning. This results in a short fine-tuning process with low computational resource requirements.
[0032] This invention addresses the fine-tuning of multimodal large language models, proposing a method based on a token classification model to effectively mitigate catastrophic forgetting. This method ensures stable performance of the multimodal large language model on the target task while reducing performance degradation on pre-training tasks. The invention utilizes the token sequence of the pre-defined weight layer of the multimodal large language model to calculate the rejection score of each input token to an additional adapter layer, and then makes discrete decisions based on the rejection scores. Input tokens are divided into rejected and adapted categories, and processed differently: rejected tokens are only calculated with the pre-trained model parameters; while adapted tokens are calculated together with both the pre-trained model parameters and the adapter layer parameters. Finally, the tokens with different processing are merged to obtain the output of the weight layer of the multimodal large language model.
[0033] For example, the preset weight layer of a multimodal large language model can be one, several, or all weight layers of the multimodal large language model.
[0034] Interpretive methods can be directly applied to large-scale pre-trained multimodal language models; otherwise, pre-training can be performed using a pre-training dataset. Let the pre-training parameters of the multimodal language model be denoted as... .
[0035] In one possible implementation, introducing a parameter-trainable adapter layer into the preset weight layer of the multimodal large language model includes introducing a parameter-trainable adapter layer into both the self-attention layer and the feedforward neural network layer (MLP layer) of each Transformer layer of the multimodal large language model.
[0036] Interpretive, this refers to the sequence of words in a multimodal large language model after the training samples have passed through the embedding layer and positional encoding. ,in This refers to the number of training samples used in each iteration of a multimodal large language model. It is the number of lexical units contained in the lexical sequence corresponding to each training sample. It is the dimension of the embedding layer in a multimodal large language model.
[0037] For example, for each preset weight layer of a multimodal large language model Introduce a trainable adapter layer with input parameters. The parameter matrix of the adapter layer is... :
[0038] in, This is the scaling factor. and It is a low-rank fitting matrix. Let be the rank of the low-rank fitting matrix.
[0039] Interpretively, the parameter matrix of the adapter layer adopts a low-rank decomposition form, which realizes efficient parameterization update. By compressing the weight update amount of full rank into the product of two low-rank matrices and introducing scaling factors and rank for regulation, the fine-tuning process only requires training a very small number of new parameters. While greatly reducing computational overhead and memory usage, it can still effectively capture task-specific knowledge changes, thereby achieving a performance improvement comparable to full parameter fine-tuning in resource-constrained scenarios, realizing efficient parameter fine-tuning and catastrophic forgetting mitigation.
[0040] In one possible implementation, the step of calling a preset lexical classification model to obtain the type of each lexical in the lexical sequence includes: calling the preset lexical classification model and obtaining the rejection score of each lexical in the lexical sequence using the following formula:
[0041] in, This is a sequence of rejection scores for each word within a word sequence. For the probability of rejection, To adapt to the probability, For word sequence, For the parameters of the word classification model, It is additive Gumbel noise. For temperature coefficient, , The attenuation rate constant is To train step count, To achieve the target low temperature, This represents the initial high temperature.
[0042] when When, the lexical type of the lexical is rejection; when At that time, the lexical type of the word is adaptation; among them, For the first The th word sequence of the th word sequence The rejection probability in the rejection score of each word element. For the first The th word sequence of the th word sequence The probability of fit in the rejection score of each word.
[0043] For example, temperature coefficient From the number of training steps arrive It exhibits an exponential decay.
[0044] For example, the rejection score sequence of each word within a word sequence. .
[0045] Interpretively, additive Gumbel noise is introduced during the training phase, combined with a temperature annealing strategy to calculate the lexical rejection score. The Gumbel-Softmax reparameterization technique transforms the discrete lexical type decision process into a continuously differentiable optimization problem, allowing gradients to be backpropagated to the lexical classification model through random sampling paths, thus achieving end-to-end learning of the routing strategy. At the same time, the exponential decay mechanism of the temperature coefficient τ effectively balances exploration and utilization: high temperature in the early stage of training promotes extensive exploration of the routing network to avoid getting trapped in local optima, while low temperature in the later stage of training makes the routing decisions tend to be stable and deterministic. Ultimately, the model can automatically learn the optimal lexical-level routing strategy for different tasks without the need for manual rule setting.
[0046] In one possible implementation, the preset word classification model includes: each adapter layer has a preset word classification model, and the parameters of each word classification model are updated separately; or, all adapter layers share a single word classification model; or, a word classification model is preset for every certain number of adapter layers.
[0047] Interpretive, allowing for different designs of the pre-defined lexical classification model based on different needs.
[0048] In one possible implementation, the step of processing each word in the word sequence according to the word type to obtain the output of the training sample includes: obtaining the output of the training sample by the following formula. :
[0049] in, For word sequence, These are the pre-training parameters for a multimodal large language model. For Hadama accumulation, For the mask matrix, , For the first The th word sequence of the th word sequence A mask for each lexical element, when the lexical type is rejection. When the lexical type is matching .
[0050] Explanatoryly, to efficiently and automatically implement different processing methods for different types of lexical units, a mask is introduced. , mask Application in input word sequences superior.
[0051] In one possible implementation, obtaining the task-adaptive fine-tuned multimodal large language model based on the multimodal large language model, the final adapter layer, and the final lexical classification model includes: combining the multimodal large language model, the final adapter layer, and the final lexical classification model as the task-adaptive fine-tuned multimodal large language model.
[0052] Interpretive, frozen-task-adaptive fine-tuning of all parameters in a multimodal large language model allows for dynamic model inference. Given the input during the inference phase... Furthermore, since additive Gumbel noise is no longer introduced during inference, the softmax function of the word classification model is calculated as follows:
[0053] The subsequent lexical classification and processing are consistent with the model fine-tuning process described above, and will not be repeated here.
[0054] In one possible implementation, obtaining the task-adaptive fine-tuning multimodal large language model based on the multimodal large language model, the final adapter layer, and the final lexical classification model includes: obtaining the parameter matrix of the final lexical classification model corresponding to each final adapter layer, and summing the elements of each column of the parameter matrix to obtain the rejection weight and adaptation weight of each final adapter layer; when the rejection weight of the final adapter layer... When adapting the weights, the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer are modified as follows: ;in, These are the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer. The final adapter layer's weight matrix is used; otherwise, the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer are retained; and the multimodal large language model with modified parameters is used as the multimodal large language model for task adaptation fine-tuning.
[0055] Explanatory, parameters of the word classification model corresponding to the weight layer. Summing yields the weight vector of the weight layer. ,in, To reject weights, To adapt the weights. When At that time, retain the adapter layer The parameter matrix of the weight layer after merging is: Otherwise, discard the adapter layer corresponding to the weight layer. , The merged model parameters are as follows: It can be used as a multimodal large language model for task adaptation and fine-tuning. Alternatively, it can be used as a multimodal large language model for the next fine-tuning process.
[0056] Explanatory, based on the above operations, a static variant of the multimodal large language model is implemented in the target task, which can be applied to scenarios with higher requirements for inference speed.
[0057] In one possible implementation, see Figure 2 This paper illustrates a feasible and detailed implementation of the multimodal large language model task adaptation fine-tuning method of the present invention.
[0058] In one possible implementation, the multimodal large language model task adaptation and fine-tuning method of the present invention is used to verify its effectiveness. The model is trained on the OKVQA dataset, and the verification scheme is as follows: The open-source pre-trained model LLaVA-1.5 (Vicuna-7B) was used, and this model underwent one round of pre-training on a filtered subset of the CC-595K dataset. After pre-training, the model was fine-tuned once using the OKVQA training dataset of the target task using the multimodal large language model task adaptation fine-tuning method and the low-rank adaptation fine-tuning method of this invention. The fine-tuned model was then evaluated. Furthermore, after fine-tuning using the multimodal large language model task adaptation fine-tuning method of this invention, parameter merging (static variant) was performed before evaluation.
[0059] Model evaluation was performed on eight pre-training tasks (VQAv2, GQA, VizWiz, SQA, TextVQA, POPE, MMBeck-EN, and MMBeck-CN) and one objective task (OKVQA) to obtain the model's accuracy on each task. And calculate its arithmetic mean (Avg) and harmonic mean (H-score):
[0060]
[0061] in, The number of pre-training tasks, The number of target tasks. For the model in the first Accuracy on each pre-training task For the model in the first Accuracy on each target task For all The arithmetic mean, For all The arithmetic mean.
[0062] The results of the validity verification are shown in Table 1.
[0063] Table 1
[0064] It is evident that the task-adapted fine-tuning method for multimodal large language models of this invention has significant advantages over existing low-rank adaptation methods for multimodal large language models.
[0065] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.
[0066] See Figure 3In another embodiment of the present invention, a multimodal large language model task adaptation and fine-tuning system is provided, which can be used to implement the above-mentioned multimodal large language model task adaptation and fine-tuning method. Specifically, the multimodal large language model task adaptation and fine-tuning system includes a sample acquisition module, an adapter layer introduction module, a fine-tuning training module, and a model output module.
[0067] The system comprises the following modules: a sample acquisition module for acquiring training samples for the task to be adapted; the training samples are text samples and / or image samples; an adapter layer introduction module for introducing a trainable adapter layer into the preset weight layer of the multimodal large language model; a fine-tuning training module for acquiring the word sequence of the training samples based on the preset weight layer of the multimodal large language model, calling the preset word classification model to obtain the type of each word in the word sequence, processing each word in the word sequence according to the word type to obtain the output of the training samples, and updating the parameters of the adapter layer and the parameters of the word classification model according to the output of the training samples; where the word type is adaptation, it is processed through the multimodal large language model and the adapter layer; when the word type is rejection, it is processed through the multimodal large language model; and a model output module for obtaining a task-adapted fine-tuned multimodal large language model based on the multimodal large language model, the final adapter layer, and the final word classification model.
[0068] In one possible implementation, introducing a parameter-trainable adapter layer into the preset weight layer of the multimodal large language model includes introducing a parameter-trainable adapter layer into the self-attention layer and the feedforward neural network layer of each Transformer layer of the multimodal large language model.
[0069] In one possible implementation, the step of calling a preset lexical classification model to obtain the type of each lexical in the lexical sequence includes: calling the preset lexical classification model and obtaining the rejection score of each lexical in the lexical sequence using the following formula:
[0070] in, This is a sequence of rejection scores for each word within a word sequence. For the probability of rejection, To adapt to the probability, For word sequence, For the parameters of the word classification model, It is additive Gumbel noise. For temperature coefficient, , The attenuation rate constant is To train step count, To achieve the target low temperature, This represents the initial high temperature.
[0071] when When, the lexical type of the lexical is rejection; when At that time, the lexical type of the word is adaptation; among them, For the first The th word sequence of the th word sequence The rejection probability in the rejection score of each word element. For the first The th word sequence of the th word sequence The probability of fit in the rejection score of each word.
[0072] In one possible implementation, the preset word classification model includes: each adapter layer has a preset word classification model, and the parameters of each word classification model are updated separately; or, all adapter layers share a single word classification model; or, a word classification model is preset for every certain number of adapter layers.
[0073] In one possible implementation, the parameter matrix of the trainable adapter layer is: :
[0074] in, This is the scaling factor. and It is a low-rank fitting matrix. Let be the rank of the low-rank fitting matrix.
[0075] In one possible implementation, the step of processing each word in the word sequence according to the word type to obtain the output of the training sample includes: obtaining the output of the training sample by the following formula. :
[0076] in, For word sequence, These are the pre-training parameters for a multimodal large language model. For Hadama accumulation, For the mask matrix, , For the first The th word sequence of the th word sequence A mask for each lexical element, when the lexical type is rejection. When the lexical type is matching .
[0077] In one possible implementation, obtaining the task-adaptive fine-tuned multimodal large language model based on the multimodal large language model, the final adapter layer, and the final lexical classification model includes: combining the multimodal large language model, the final adapter layer, and the final lexical classification model as the task-adaptive fine-tuned multimodal large language model; or, obtaining the parameter matrix of the final lexical classification model corresponding to each final adapter layer, and summing the elements of each column of the parameter matrix to obtain the rejection weight and adaptation weight of each final adapter layer; when the rejection weight of the final adapter layer... When adapting the weights, the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer are modified as follows: ;in, These are the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer. The final adapter layer's weight matrix is used; otherwise, the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer are retained; and the multimodal large language model with modified parameters is used as the multimodal large language model for task adaptation fine-tuning.
[0078] All relevant content of each step involved in the aforementioned embodiments of the multimodal large language model task adaptation and fine-tuning method can be referenced from the functional description of the corresponding functional module of the multimodal large language model task adaptation and fine-tuning system in the embodiments of the present invention, and will not be repeated here.
[0079] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0080] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a multimodal large language model task adaptation fine-tuning method.
[0081] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multimodal large language model task adaptation fine-tuning method in the above embodiments.
[0082] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for fine-tuning adaptation of a multimodal large language model task, characterized in that, include: Obtain training samples for the task to be adapted; wherein, the training samples are text samples and / or image samples; Introduce a parameter-trainable adapter layer into the pre-defined weight layer of the multimodal large language model; The training samples are obtained based on the word sequence of the preset weight layer of the multimodal large language model. A preset word classification model is then called to obtain the type of each word in the word sequence. The word types are then processed to obtain the output of the training samples. Finally, the parameters of the adapter layer and the word classification model are updated based on the output of the training samples. Specifically, when the word type is "fit," it is processed through the multimodal large language model and the adapter layer; when the word type is "reject," it is processed through the multimodal large language model. Based on the multimodal large language model, the final adapter layer, and the final lexical classification model, a multimodal large language model with task-adaptive fine-tuning is obtained.
2. The multimodal large language model task adaptation and fine-tuning method according to claim 1, characterized in that, The introduction of a parameter-trainable adapter layer into the preset weight layer of the multimodal large language model includes: In the multimodal large language model, a parameter-trainable adapter layer is introduced into the self-attention layer and feedforward neural network layer of each Transformer layer.
3. The multimodal large language model task adaptation and fine-tuning method according to claim 1, characterized in that, The step of calling a preset lexical classification model to obtain the types of each lexical element within the lexical sequence includes: By calling the preset lexical classification model, the rejection score of each lexical in the lexical sequence is obtained using the following formula: in, This is a sequence of rejection scores for each word within a word sequence. For the probability of rejection, To adapt to the probability, For word sequence, For the parameters of the word classification model, It is additive Gumbel noise. For temperature coefficient, , The attenuation rate constant is To train step count, To achieve the target low temperature, Initial high temperature; when At that time, the lexical type of the lexical is rejection; when At that time, the lexical type of the lexical is adaptation; in, For the first The th word sequence of the th word sequence The rejection probability in the rejection score of each word element. For the first The th word sequence of the th word sequence The probability of fit in the rejection score of each word.
4. The method for adapting and fine-tuning multimodal large language model tasks according to claim 1, characterized in that, The preset lexical classification model includes: Each adapter layer has a pre-defined word classification model, and the parameters of each word classification model are updated separately. Alternatively, all adapter layers can share a single lexical classification model; Alternatively, a word classification model can be pre-defined for every certain number of adapter layers.
5. The multimodal large language model task adaptation and fine-tuning method according to claim 1, characterized in that, The parameter matrix of the trainable adapter layer is as follows: : in, This is the scaling factor. and It is a low-rank fitting matrix. Let be the rank of the low-rank fitting matrix.
6. The multimodal large language model task adaptation and fine-tuning method according to claim 5, characterized in that, The step of processing each word in the word sequence according to the word type to obtain the output of the training sample includes: The output of the training samples is obtained using the following formula. : in, For word sequence, These are the pre-training parameters for a multimodal large language model. For Hadama accumulation, For the mask matrix, , For the first The th word sequence of the th word sequence A mask for each lexical element, when the lexical type is rejection. When the lexical type is matching .
7. The multimodal large language model task adaptation and fine-tuning method according to claim 1, characterized in that, The multimodal large language model, based on the multimodal large language model, the final adapter layer, and the final lexical classification model, yields a multimodal large language model that is fine-tuned for task adaptation, including: The multimodal large language model, the final adapter layer, and the final lexical classification model are combined to form a multimodal large language model for task adaptation and fine-tuning. Alternatively, obtain the parameter matrix of the final word classification model corresponding to each final adapter layer, and sum the elements of each column of the parameter matrix to obtain the rejection weight and adaptation weight of each final adapter layer; when the rejection weight of the final adapter layer... When adapting the weights, the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer are modified as follows: ;in, These are the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer. The final adapter layer's weight matrix is used; otherwise, the parameters of the weight layer of the multimodal large language model corresponding to the final adapter layer are retained; and the multimodal large language model with modified parameters is used as the multimodal large language model for task adaptation fine-tuning.
8. A multimodal large language model task adaptation and fine-tuning system, characterized in that, include: The sample acquisition module is used to acquire training samples for the task to be adapted; wherein, the training samples are text samples and / or image samples; The adapter layer introduction module is used to introduce a parameter-trainable adapter layer into the preset weight layer of the multimodal large language model; The fine-tuning training module is used to obtain the word sequence of the training samples based on the preset weight layer of the multimodal large language model, call the preset word classification model to obtain the type of each word in the word sequence, process each word in the word sequence according to the type of the word to obtain the output of the training samples, and update the parameters of the adapter layer and the parameters of the word classification model according to the output of the training samples. Specifically, when the type of the word is "adaptation", it is processed by the multimodal large language model and the adapter layer; when the type of the word is "rejection", it is processed by the multimodal large language model. The model output module is used to obtain a multimodal large language model with task-adaptive fine-tuning based on the multimodal large language model, the final adapter layer, and the final lexical classification model.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal large language model task adaptation fine-tuning method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal large language model task adaptation fine-tuning method as described in any one of claims 1 to 7.