Fine adjustment method of multi-mode large language model and electronic equipment

Abstract

The invention provides a fine tuning method of a multi-modal large language model and electronic equipment, and belongs to the technical field of fine tuning of multi-modal large language models. The method comprises the following steps: respectively determining the sensitivity of each Transform network layer of a large language model and a visual encoder in a multi-modal large language model to a current task according to an adjustment and optimization data set of the current task, and determining the importance degree of each modal in the current task; according to the sensitivity of each Transformer network layer to the current task and the importance degree of each mode in the current task, low-rank adapters are allocated to the Transformer network layers of the large language model and the visual encoder, and a low-rank adapter allocation result corresponding to the current task is determined; and according to the training data set of the current task and the low-rank adapter distribution result corresponding to the current task, performing fine tuning training on the multi-modal large language model. The invention aims to improve the adaptability of the multi-modal large language model to the new task.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of fine-tuning a large multimodal language model, and more specifically, to a fine-tuning method and electronic device for a large multimodal language model. Background Art

[0002] Multimodal Large Language Models (MLLMs) are increasingly used in various fields. Continuous multimodal instruction tuning is crucial to adapt MLLMs to ever-changing tasks in real-world scenarios.

[0003] Current continuous multimodal instruction tuning methods generally adopt a fixed architecture design. However, due to the significant differences in the sensitivity of different tasks to the various network layers of the Transformer, and the dynamic changes in the degree of dependence of tasks on multimodal data, large multimodal language models with fixed architectures have poor adaptability to new tasks. Summary of the Invention

[0004] Embodiments of the present application provide a method and electronic device for fine-tuning a large multimodal language model, aiming to improve the adaptability of the large multimodal language model to new tasks.

[0005] In a first aspect, an embodiment of the present application provides a method for fine-tuning a multimodal large language model, the method comprising: Based on the tuning dataset for the current task, determine the sensitivity of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model to the current task, and determine the importance of each modality in the current task; Allocating low-rank adapters to the Transformer network layers of the large language model and the visual encoder according to the sensitivity of each Transformer network layer to the current task and the importance of each modality in the current task, and determining a low-rank adapter allocation result corresponding to the current task; Fine-tune the multimodal large language model according to the training data set of the current task and the low-rank adapter allocation result corresponding to the current task.

[0006] Optionally, respectively determining the sensitivity of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model to the current task includes: During the process of the multimodal large language model processing the tuning dataset, the gradient norms of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model are respectively determined, and the gradient norms are used as the sensitivity of each Transformer network layer to the current task.

[0007] Optionally, determining the importance of each modality in the current task includes: During processing of the tuning dataset by the multimodal large language model, determining a difficulty score of each modality of the current task based on the gradient norm of each modality in the tuning dataset of the current task; The importance of each modality in the current task is determined according to the difficulty score of each modality of the current task.

[0008] Optionally, the formula for determining the importance of each modality in the current task is:

[0009]

[0010] in, is the importance of the modality M of the current task t; Score the difficulty of the modality M of the current task t; the value of M includes and ; Score the difficulty of the text modality of the current task t; Score the difficulty of the visual modality of the current task t; A tuning dataset for the current task; is the weight of the modality M in the pre-trained multimodal large language model.

[0011] Optionally, allocating low-rank adapters to the Transformer network layers of the large language model and the visual encoder according to the sensitivity of each Transformer network layer to the current task and the importance of each modality in the current task, and determining a low-rank adapter allocation result corresponding to the current task, including: Determining the number of low-rank adapters allocated to the large language model and the visual encoder respectively according to the importance of each modality in the current task and the low-rank adapter allocation budget corresponding to the current task; Sort the Transformer network layers of the large language model and the visual encoder in descending order of sensitivity; According to the number of low-rank adapters allocated to the large language model, one low-rank adapter is allocated to each Transformer network layer of the large language model, and the Transformer network layer with the top N1 sensitivity is respectively allocated one low-rank adapter, and the number of low-rank adapters allocated to the large language model is the same as N1; According to the number of low-rank adapters allocated to the visual encoder, a low-rank adapter is allocated to each Transformer network layer of the visual encoder, and the Transformer network layer with the top N2 sensitivity is respectively allocated a low-rank adapter, and the number of low-rank adapters allocated to the visual encoder is the same as N2; The low-rank adapters assigned to the large language model and the Transformer network layer of the visual encoder are used as the low-rank adapter assignment results corresponding to the current task.

[0012] Optionally, according to the importance of each modality in the current task and the low-rank adapter allocation budget corresponding to the current task, the formula for proportionally determining the number of low-rank adapters allocated to the large language model and the visual encoder is:

[0013] in, allocating a quantity to the low-rank adapter; Allocating a budget for the low-rank adapter corresponding to the current task; is the importance of each modality M in the current task t; the value of M includes and , when M is hour, Assign the number of low-rank adapters to the large language model under the current task, when M is hour, Assign the number of low-rank adapters to the visual encoder for the current task.

[0014] Optionally, fine-tuning the multimodal large language model according to the training dataset of the current task and the low-rank adapter allocation result corresponding to the current task includes: Extracting sample embeddings corresponding to each sample data in the tuning dataset of the current task, so as to train the autoencoder corresponding to the current task based on the sample embeddings corresponding to each sample data in the tuning dataset and minimizing the reconstruction loss function; Embedding the samples corresponding to each sample data in the tuning dataset, inputting the samples into the autoencoders of each historical task respectively, and determining the reconstruction loss corresponding to the autoencoders of each historical task; Determining, based on the reconstruction losses corresponding to the autoencoders of the respective historical tasks, a low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss; According to the low-rank adapter allocation result corresponding to the current task and the low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss, the multimodal large language model is fine-tuned based on the training data set of the current task.

[0015] Optionally, fine-tuning the multimodal large language model based on the training dataset of the current task according to the low-rank adapter allocation result corresponding to the current task and the low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss, including: During fine-tuning training of the multimodal large language model based on the training dataset of the current task, only the low-rank adapters in the low-rank adapter allocation results corresponding to the current task are updated, and the pre-trained multimodal large language model and the low-rank adapters in the low-rank adapter allocation results corresponding to the historical task with the smallest reconstruction loss are frozen.

[0016] Optionally, after fine-tuning the multimodal large language model, the method includes: Get the test dataset for the current task; Extracting text features and visual features of the test sample data in the test dataset through the multimodal large language model, performing maximum pooling on the text features and visual features of the test sample data in the test dataset, and then concatenating them to obtain sample embeddings for the test sample data; Input the sample embedding of the test sample data into the autoencoder corresponding to each task, and obtain the reconstruction loss output by the autoencoder of each task; According to the reconstruction loss output by the autoencoder of each task and the reconstruction loss threshold corresponding to each task, the activated low-rank adapter allocation result is determined, and the test sample data is processed by the multimodal large language model.

[0017] In a second aspect, an embodiment of the present application provides an electronic device comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the fine-tuning method of the multimodal large language model described in the first aspect of the embodiment is executed.

[0018] Beneficial effects: This method determines the sensitivity of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model to the current task and the importance of each modality in the current task based on the tuning data set of the current task; then, based on the sensitivity of each Transformer network layer to the current task and the importance of each modality in the current task, low-rank adapters are allocated to the Transformer network layers of the large language model and the visual encoder, and the low-rank adapter allocation result corresponding to the current task is determined; and based on the training data set of the current task and the low-rank adapter allocation result corresponding to the current task, the multimodal large language model is fine-tuned.

[0019] By dynamically assigning low-rank adapters to the Transformer network layers of the large language model and visual encoder of the multimodal large language model and performing fine-tuning training based on the sensitivity of different Transformer network layers to the current task and the importance of each modality in the current task, the multimodal large language model can be made more adaptable to the new current task, with the effects of efficient task adaptation and modality balance optimization. In addition, the fine-tuned multimodal large language model can produce more accurate processing results for the current task. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is a flowchart of the steps of a fine-tuning method for a multimodal large language model proposed in one embodiment of the present application; Figure 2 1 is a schematic diagram illustrating an implementation of a fine-tuning method for a multimodal large language model according to an embodiment of the present application; Figure 3 This is a functional module diagram of a device for fine-tuning a multimodal large language model provided in one embodiment of the present application; Figure 4 is a schematic diagram of an electronic device provided in one embodiment of the present application; Figure 5 is a schematic diagram of a non-volatile readable storage medium proposed in an embodiment of the present application; Figure 6 It is a schematic diagram of a computer program product proposed in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] Continuous multimodal instruction tuning is crucial for adapting large multimodal language models to ever-changing tasks in real-world scenarios. Current approaches to continuous multimodal instruction tuning primarily focus on efficient parameter fine-tuning frameworks and can be categorized into two typical approaches: The first category is prompt tuning-based methods, such as the query-key prompt pool strategy and adaptation through dynamic allocation of task-specific prompt vectors. However, this method significantly reduces the effective information capacity of large multimodal models because the prompt vector occupies a large amount of context length.

[0024] The second category is to achieve continuous learning through parameter optimization expansion, including applying orthogonal regularization to the low-rank adapter (LoRA) module, designing task-specific multimodal projectors, suppressing gradient interference through residual projection, using exponential moving average, building a dual-instruction embedding space, and proposing a model pruning strategy to address the problem of pre-training knowledge forgetting after fine-tuning.

[0025] However, current continuous multi-modal instruction tuning methods generally adopt fixed architecture designs, which have two core flaws: First, different tasks have significant differences in their sensitivity to each layer of the Transformer. For example, visual question answering relies on underlying feature extraction, while knowledge-based question answering focuses on high-level semantic understanding. Current methods use a static parameter allocation strategy, which can lead to insufficient resources in key layers and redundant calculations in non-key layers, resulting in task architecture conflicts and resource waste.

[0026] Secondly, when the degree of dependence of a task on multimodal data changes dynamically, for example, the text modality dominates knowledge question answering, while the visual modality dominates image description, the current methods lack a dynamic adjustment mechanism for the modality update ratio, resulting in the dominant modality suppressing the learning of the secondary modality, causing modality imbalance problems.

[0027] In addition, current methods mostly adopt fixed low-rank adapter allocation strategies, which make it difficult to achieve efficient task adaptation under parameter budget constraints, limiting the application potential of the model in resource-constrained scenarios.

[0028] Therefore, since most current methods adopt a fixed architecture and the model capacity is static, it is difficult to adapt to new tasks. To address the above defects, the embodiment of the present application proposes a fine-tuning method for a large multimodal language model. Based on the continuous multimodal instruction fine-tuning of architecture evolution, the architecture of the large multimodal model can be automatically evolved under the budget control of the low-rank adapter, solving the task architecture conflict and modal imbalance problems faced by the large multimodal language model in continuous multimodal instruction tuning, so that the large multimodal language model can continuously adapt to new tasks and retain the learned knowledge, thereby achieving efficient task adaptation and modal balance optimization in continuous multimodal instruction fine-tuning.

[0029] Reference Figure 1 , shows a flowchart of the steps of a fine-tuning method for a multimodal large language model in an embodiment of the present application, and the method may specifically include the following steps: S101: Based on the tuning data set of the current task, respectively determine the sensitivity of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model to the current task, and determine the importance of each modality in the current task.

[0030] Specifically, a task refers to a pre-divided multimodal data set of the same type. In the actual implementation process, the multimodal data in the same task can be multimodal data for the same field. A task can be customized according to the visual modality and / or text modality. For example, multiple multimodal data with the same or similar input visual images but different texts can belong to one task, or multiple multimodal data with different input visual images but the same or similar texts can belong to one task, or multiple multimodal data with the same or similar input visual images and texts can belong to one task.

[0031] For example, the images of various multimodal data in a task can be in the same field, such as all medical images, and the texts are all texts explaining the content of the image, such as "what does this scan result show".

[0032] Multimodal data belonging to the same task can be divided according to the needs of actual applications, and this embodiment does not impose any restrictions.

[0033] In the actual implementation process, a task includes multiple multimodal data. In the process of fine-tuning the multimodal large language model based on the task, a data subset can be selected from the data set of the current task as the tuning data set. For example, a 1% subset can be randomly sampled from the current task as the tuning data set of the current task. ,The sampling method and sampling ratio can be selected according to the ,requirements of actual applications.

[0034] Tuning dataset for the current task The multimodal data in is mainly used to dynamically generate the low-rank adapter assignment results corresponding to the current task.

[0035] In a feasible implementation, the process of respectively determining the sensitivity of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model to the current task includes: During the process of the multimodal large language model processing the tuning dataset, the gradient norms of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model are respectively determined, and the gradient norms are used as the sensitivity of each Transformer network layer to the current task.

[0036] Specifically, by calculating the gradient norm of each Transformer network layer of the large language model and visual encoder ,in, l is the unique number corresponding to each Transformer network layer of the large language model and visual encoder, and t is the identifier of the current task.

[0037] According to the gradient norm of each Transformer network layer To quantify the sensitivity of each Transformer network layer to the current task, the larger the gradient norm of a Transformer network layer, the more sensitive the layer is to the input of the current task.

[0038] Since different tasks have significant differences in sensitivity to each Transformer network layer, such as visual question answering relies on underlying feature extraction, while knowledge-based question answering focuses on high-level semantic understanding, in the process of dynamically allocating low-rank adapters, by considering the sensitivity of each Transformer network layer to the current task, the low-rank adapter can be effectively allocated to the key Transformer network layer that is more sensitive to the current task, thereby resolving task architecture conflicts and reducing resource waste of low-rank adapters.

[0039] In a feasible implementation, the process of determining the importance of each modality in the current task may include: In the process of the multimodal large language model processing the tuning data set, the difficulty score of each modality of the current task is determined according to the gradient norm of each modality in the tuning data set of the current task, and then the importance of each modality in the current task is determined according to the difficulty score of each modality of the current task.

[0040] Specifically, the formula for determining the importance of each modality in the current task is:

[0041]

[0042] in, is the importance of the modality M of the current task t; Score the difficulty of the modality M of the current task t; the value of M includes and ; Score the difficulty of the text modality of the current task t; Score the difficulty of the visual modality of the current task t; A tuning dataset for the current task; is the weight of the modality M in the pre-trained multimodal large language model.

[0043] Since the degree of dependence of different tasks on modalities will change dynamically, such as text modality dominates knowledge question answering, while visual modality dominates image description, in the process of dynamically allocating low-rank adapters for the multimodal large language model to process the current task, by introducing the importance of each modality of the current task to the task, the modal balance optimization can be achieved in the process of the multimodal large language model processing the current task.

[0044] S102: According to the sensitivity of each Transformer network layer to the current task and the importance of each modality in the current task, low-rank adapters are allocated to the Transformer network layers of the large language model and the visual encoder to determine the low-rank adapter allocation result corresponding to the current task.

[0045] Specifically, the process of determining the low-rank adapter allocation result corresponding to the current task may include the following steps: A1: Determine the number of low-rank adapters allocated to the large language model and the visual encoder respectively according to the importance of each modality in the current task and the low-rank adapter allocation budget corresponding to the current task.

[0046] During actual implementation, the low-rank adapter allocation budget for each task may be the same, and the number of low-rank adapter allocation budgets may be set according to actual application requirements, which is not limited in this embodiment.

[0047] First, according to the importance of each modality in the current task, the number of low-rank adapters that can be allocated is determined proportionally for the modality of the current task. The formula is:

[0048] in, allocating a quantity to the low-rank adapter; Allocating a budget for the low-rank adapter corresponding to the current task; is the importance of each modality M in the current task t; the value of M includes and , when M is hour, Assign the number of low-rank adapters to the large language model under the current task, when M is hour, Assign the number of low-rank adapters to the visual encoder for the current task.

[0049] That is, in the current task, the higher the importance of a modality, the more low-rank adapters can be allocated to the Transformer network layer that processes the modality. For example, if the importance of the text modality is higher, the larger the number of low-rank adapters allocated to the large language model; if the importance of the visual modality is higher, the larger the number of low-rank adapters allocated to the visual encoder.

[0050] A2: Sort the Transformer network layers of the large language model and the visual encoder in descending order of sensitivity.

[0051] After determining the number of low-rank adapters that can be allocated to the large language model and visual encoder, it is necessary to determine which Transformer network layers of the large language model and visual encoder to add low-rank adapters.

[0052] Specifically, the Transformer network layers of the large language model and the visual encoder are sorted in descending order of sensitivity to obtain sensitivity sorting results corresponding to the Transformer network layers of the large language model and the visual encoder.

[0053] A3: According to the number of low-rank adapters allocated to the large language model, a low-rank adapter is allocated to each Transformer network layer of the large language model, and the number of low-rank adapters allocated to the large language model is the same as N1.

[0054] A4: According to the number of low-rank adapters allocated to the visual encoder, a low-rank adapter is allocated to each Transformer network layer of the visual encoder, and the Transformer network layer with the top N2 sensitivity is respectively allocated a low-rank adapter. The number of low-rank adapters allocated to the visual encoder is the same as N2.

[0055] A5: The low-rank adapters assigned to the large language model and the Transformer network layer of the visual encoder are used as the low-rank adapter assignment results corresponding to the current task.

[0056] Specifically, the low-rank adapter assignment result corresponding to the current task t can be expressed as a binary indicator variable Indicates that, l A unique number used to identify each Transformer network layer of the large language model and visual encoder. l When allocating a low-rank adapter, The value of is 1, when no low-rank adapter is assigned to any Transformer network layer l, The value of is 0.

[0057] S103: Fine-tune the multimodal large language model according to the training data set of the current task and the low-rank adapter allocation result corresponding to the current task.

[0058] After dynamically allocating low-rank adapters based on the tuning dataset for the current task, the multimodal large language model needs to be fine-tuned.

[0059] In a feasible implementation, the process of fine-tuning the multimodal large language model according to the training dataset of the current task and the low-rank adapter allocation result corresponding to the current task may include the following steps: B1: Extract the sample embedding corresponding to each sample data in the tuning dataset of the current task, and train the autoencoder corresponding to the current task based on the sample embedding corresponding to each sample data in the tuning dataset and the minimization of the reconstruction loss function.

[0060] Specifically, each task corresponds to an autoencoder, and the autoencoder of each task is a two-layer MLP structure, consisting of an encoding layer and a decoding layer.

[0061] When fine-tuning a large multimodal language model for the current task, the low-rank adapter is dynamically allocated based on the tuning dataset of the current task, and the tuning dataset of the current task is further extracted. The sample embeddings corresponding to each sample data in , sample embedding Used to train the autoencoder corresponding to the current task.

[0062] The autoencoder corresponding to the current task represents the input sample Perform a low-rank projection and then try to generate its reconstructed representation .

[0063] Based on sample representation The formula for the reconstruction loss function for training the autoencoder is:

[0064] The training objective of the autoencoder for each task is to minimize the reconstruction loss function.

[0065] B2: Embed the samples corresponding to each sample data in the tuning dataset, input them into the autoencoder of each historical task respectively, and determine the reconstruction loss corresponding to the autoencoder of each historical task.

[0066] Furthermore, in order to fine-tune the multimodal large language model based on multiple different tasks, in addition to fine-tuning the multimodal large language model based on the low-rank adapter allocation results corresponding to the current task itself, the low-rank adapter allocation results of the historical tasks most relevant to the current task can also be superimposed to process the multimodal data of the current task, so as to perform knowledge transfer through the low-rank adapter allocation results of the historical tasks.

[0067] Specifically, the tuning dataset The sample embedding corresponding to each sample data in , input the autoencoder of each historical task respectively, and determine the reconstruction loss corresponding to the autoencoder of each historical task.

[0068] B3: According to the reconstruction losses corresponding to the autoencoders of the respective historical tasks, determine the low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss.

[0069] Among the reconstruction losses corresponding to the autoencoders of each historical task, the historical task with the smallest reconstruction loss is the historical task most relevant to the current task. Knowledge transfer can be performed on the low-rank adapter allocation results of the historical task and applied to the processing of the current task by the multimodal large language model.

[0070] B4: According to the low-rank adapter allocation result corresponding to the current task and the low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss, fine-tune the multimodal large language model based on the training dataset of the current task.

[0071] Specifically, during the fine-tuning training of the multimodal large language model based on the training dataset of the current task, only the low-rank adapters in the low-rank adapter allocation results corresponding to the current task are updated, and the pre-trained multimodal large language model and the low-rank adapters in the low-rank adapter allocation results corresponding to the historical task with the smallest reconstruction loss are frozen.

[0072] The training dataset of the current task may be the full amount of multimodal data in the dataset of the current task excluding the tuning dataset.

[0073] In the actual implementation process, the pre-trained multimodal large language model and the low-rank adapter in the low-rank adapter allocation result corresponding to the historical task with the smallest reconstruction loss are frozen, and only the low-rank adapter in the low-rank adapter allocation result corresponding to the current task is trained. The total loss function can adopt the original loss function of the multimodal large language model, which is not limited in this embodiment.

[0074] In a feasible implementation, after fine-tuning the multimodal large language model, the method further includes a process of testing and evaluating the fine-tuned multimodal large language model based on a test dataset of the current task, specifically comprising the following steps: C1: Get the test dataset for the current task.

[0075] The test dataset of the current task can be pre-built. For example, the test dataset of the current task can include multimodal test sample data of the current task, and can also include multimodal test sample data of other tasks, that is, the multimodal large language model fine-tuned based on the current task t is used in different tasks. arrive Evaluate on.

[0076] The labels of each test sample data in the test dataset are unknown to the multimodal large language model, so it is also necessary to activate the low-rank adapter in the low-rank adapter assignment result corresponding to the test sample according to each test sample data in the test dataset.

[0077] C2: Extracting text features and visual features of the test sample data in the test dataset through the multimodal large language model, performing maximum pooling on the text features and visual features of the test sample data in the test dataset, and then splicing them to obtain sample embeddings for the test sample data.

[0078] Specifically, for any test sample data, the text features are extracted through the large language model in the multimodal large model. w , extract visual features through visual encoder v , and then perform maximum pooling on them to determine and .

[0079] The text features after maximum pooling and visual features Splice and get the sample embedding of the test sample data .

[0080] C3: The sample embedding of the test sample data is input into the autoencoder corresponding to each task respectively to obtain the reconstruction loss output by the autoencoder of each task.

[0081] Since the test sample data of the test dataset does not have a label, the sample embedding of the test sample data is input into the autoencoder corresponding to each task respectively to obtain the reconstruction loss output by the autoencoder of each task. The reconstruction loss of each autoencoder is used to determine which task the test sample data is most likely to belong to, and activate the low-rank adapter allocation result of the corresponding task.

[0082] C4: Determine the activated low-rank adapter allocation result based on the reconstruction loss output by the autoencoder of each task and the reconstruction loss threshold corresponding to each task, and process the test sample data through the multimodal large language model.

[0083] In the actual implementation process, a corresponding reconstruction loss threshold can also be determined for each task. When the reconstruction loss of a sample of test sample data embedded in the autoencoder output of each task is less than the reconstruction loss threshold of the task corresponding to the autoencoder, the low-rank adapter allocation result corresponding to the task is activated.

[0084] If the reconstruction losses output by an autoencoder with multiple tasks are all smaller than the reconstruction loss threshold of the task corresponding to itself, the reconstruction losses of the sample embedding outputs of the multiple autoencoders for the test sample are sorted from small to large, and the low-rank adapter allocation results corresponding to the two tasks with the smallest reconstruction losses are activated, thereby realizing task reasoning and knowledge reuse when processing the test sample data.

[0085] In the actual implementation process, if the reconstruction loss of all autoencoders exceeds the reconstruction loss threshold of their corresponding tasks, the test sample data is considered to belong to an unknown task and is only processed using the pre-trained multimodal large model.

[0086] Reference Figure 2 , shows a schematic diagram of the execution of the fine-tuning method of the multimodal large language model provided in an embodiment of the present application. Based on this method, the pre-trained multimodal large language model can be continuously fine-tuned on multiple different tasks.

[0087] Specifically, the multimodal data included in the dataset of Task 1 may be data summarizing, describing, and explaining any image; the multimodal data included in the dataset of Task 2 may be data analyzing medical images... The current Task t is data of a visual question-answering type based on images including objects; the data in the datasets of each task may be classified and constructed according to the needs of the actual application, and this embodiment does not impose any restrictions.

[0088] When fine-tuning the multimodal large language model according to the current task t, a 1% subset can be randomly sampled in the current task as the tuning data set for the current task. To dynamically generate the low-rank adapter allocation result corresponding to the current task.

[0089] In the actual implementation process, the gradient norms of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model, as well as the gradient norms of each modality in the tuning dataset of the current task can be determined through a zero-cost agent.

[0090] Through the designed dynamic hierarchical expert allocator and gradient-based cross-modal curriculum, low-rank adapters are assigned to the Transformer network layers of the large language model and the visual encoder, and the low-rank adapter allocation result corresponding to the current task is determined.

[0091] Among them, the gradient-based cross-modal course is used to determine the difficulty score of each modality of the current task according to the gradient norm of each modality in the tuning dataset of the current task during the process of the multimodal large language model processing the tuning dataset, and then determine the importance of each modality in the current task according to the difficulty score of each modality of the current task.

[0092] The dynamic hierarchical expert allocator is used to allocate a budget for the low-rank adapters corresponding to the current task according to the importance of each modality in the current task and the current task, and determine the low-rank adapters for the high-difficulty, i.e., more important, and low-difficulty, i.e., less important modalities in proportion, and then determine the number of low-rank adapters allocated corresponding to the large language model and the visual encoder respectively. After sorting the various Transformer network layers of the large language model and the visual encoder in order of the sensitivity from large to small, the low-rank adapters allocated to the Transformer network layers of the large language model and the visual encoder are determined to determine the low-rank adapter allocation result corresponding to the current task.

[0093] For example, the low-rank adapter allocation result of task 1 includes: a low-rank adapter is respectively allocated to the 1st and Nth Transformer network layers of the large language model, and the Mth Transformer network layer of the visual encoder; the low-rank adapter allocation result of task 2 includes: a low-rank adapter is respectively allocated to the 1st and nth Transformer network layers of the large language model, and the 1st Transformer network layer of the visual encoder; the low-rank adapter allocation result of task 3 includes: a low-rank adapter is respectively allocated to the nth Transformer network layer of the large language model, and the mth and Mth Transformer network layers of the visual encoder; the low-rank adapter allocation result of the current task t includes: a low-rank adapter is respectively allocated to the 1st and Nth Transformer network layers of the large language model, and the mth Transformer network layer of the visual encoder.

[0094] After determining the low-rank adapter allocation result of the current task t, during the fine-tuning training of the multimodal large language model, the low-rank adapters in the low-rank adapter allocation results of tasks 1, 2, and 3 will be frozen, and only the low-rank adapters in the newly assigned low-rank adapter allocation results of the current task t will be trained.

[0095] This method achieves efficient adaptation to new tasks and effective reuse of historical knowledge, while ensuring the robustness of the model on unknown tasks. Through dynamic architecture evolution and cross-modal optimization, it significantly alleviates the catastrophic forgetting problem in continuous multimodal instruction fine-tuning, while maintaining the ability of efficient parameter utilization and continuous knowledge retention, which can improve the adaptability of large multimodal language models to new tasks.

[0096] Reference Figure 3 , shows a functional module diagram of a fine-tuning device for a multimodal large language model provided in an embodiment of the present application, the device comprising: A sensitivity and importance determination module 100 is configured to determine the sensitivity of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model to the current task based on the tuning dataset of the current task, and to determine the importance of each modality in the current task; An allocation module 200 is configured to allocate low-rank adapters to the Transformer network layers of the large language model and the visual encoder based on the sensitivity of each Transformer network layer to the current task and the importance of each modality in the current task, and determine a low-rank adapter allocation result corresponding to the current task; The training module 300 is used to fine-tune the multimodal large language model based on the training data set of the current task and the low-rank adapter allocation result corresponding to the current task.

[0097] Optionally, the sensitivity and importance determination module includes: a sensitivity determination unit, configured to determine, during the process of the multimodal large language model processing the tuning dataset, the gradient norm of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model, and use the gradient norm as the sensitivity of each Transformer network layer to the current task.

[0098] Optionally, the sensitivity and importance determination module includes an importance determination unit configured to: During processing of the tuning dataset by the multimodal large language model, determining a difficulty score of each modality of the current task based on the gradient norm of each modality in the tuning dataset of the current task; The importance of each modality in the current task is determined according to the difficulty score of each modality of the current task.

[0099] Optionally, the formula for determining the importance of each modality in the current task is:

[0100]

[0101] in, is the importance of the modality M of the current task t; Score the difficulty of the modality M of the current task t; the value of M includes and ; Score the difficulty of the text modality of the current task t; Score the difficulty of the visual modality of the current task t; A tuning dataset for the current task; is the weight of the modality M in the pre-trained multimodal large language model.

[0102] Optionally, the allocation module includes: a proportional allocation unit, configured to determine the number of low-rank adapters allocated to the large language model and the visual encoder respectively according to the importance of each modality in the current task and the low-rank adapter allocation budget corresponding to the current task; A sorting unit, configured to sort the large language model and each Transformer network layer of the visual encoder in descending order of sensitivity; A first allocating unit is configured to allocate a low-rank adapter to each Transformer network layer of the large language model, wherein the Transformer network layer with the sensitivity ranked in the top N1 is allocated according to the number of low-rank adapters allocated corresponding to the large language model, and the number of low-rank adapters allocated corresponding to the large language model is the same as N1; A second allocating unit is configured to allocate a low-rank adapter to each Transformer network layer of the visual encoder, wherein the Transformer network layer with the sensitivity ranked in the top N2 is allocated according to the number of low-rank adapters allocated corresponding to the visual encoder, and the number of low-rank adapters allocated corresponding to the visual encoder is the same as N2; An allocation result determining unit is used to use the low-rank adapters allocated to the large language model and the Transformer network layer of the visual encoder as the low-rank adapter allocation results corresponding to the current task.

[0103] Optionally, according to the importance of each modality in the current task and the low-rank adapter allocation budget corresponding to the current task, the formula for proportionally determining the number of low-rank adapters allocated to the large language model and the visual encoder is:

[0104] in, allocating a quantity to the low-rank adapter; Allocating a budget for the low-rank adapter corresponding to the current task; is the importance of each modality M in the current task t; the value of M includes and , when M is hour, Assign the number of low-rank adapters to the large language model under the current task, when M is hour, Assign the number of low-rank adapters to the visual encoder for the current task.

[0105] Optionally, the training module includes: A first sample embedding processing unit is configured to extract a sample embedding corresponding to each sample data in the tuning dataset of the current task, so as to train an autoencoder corresponding to the current task based on the sample embedding corresponding to each sample data in the tuning dataset and minimizing a reconstruction loss function; A reconstruction loss determination unit is used to embed the samples corresponding to each sample data in the tuning data set, input them into the autoencoder of each historical task respectively, and determine the reconstruction loss corresponding to the autoencoder of each historical task; a reconstruction loss determining unit, configured to determine, based on the reconstruction losses corresponding to the autoencoders of the respective historical tasks, a low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss; A fine-tuning training unit is used to fine-tune the multimodal large language model based on the training data set of the current task according to the low-rank adapter allocation result corresponding to the current task and the low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss.

[0106] Optionally, the fine-tuning training unit includes: The fine-tuning training subunit is used to update only the low-rank adapters in the low-rank adapter allocation results corresponding to the current task during the fine-tuning training of the multimodal large language model based on the training data set of the current task, and freeze the low-rank adapters in the low-rank adapter allocation results corresponding to the pre-trained multimodal large language model and the historical task with the smallest reconstruction loss.

[0107] Optionally, the method includes a testing module for: Get the test dataset for the current task; Extracting text features and visual features of the test sample data in the test dataset through the multimodal large language model, performing maximum pooling on the text features and visual features of the test sample data in the test dataset, and then concatenating them to obtain sample embeddings for the test sample data; Input the sample embedding of the test sample data into the autoencoder corresponding to each task, and obtain the reconstruction loss output by the autoencoder of each task; According to the reconstruction loss output by the autoencoder of each task and the reconstruction loss threshold corresponding to each task, the activated low-rank adapter allocation result is determined, and the test sample data is processed by the multimodal large language model.

[0108] Reference Figure 4 , shows a schematic diagram of an electronic device provided in an embodiment of the present application, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the fine-tuning method of the multimodal large language model described in the embodiment is executed.

[0109] Reference Figure 5, shows a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the fine-tuning method of the multimodal large language model described in the embodiment is executed.

[0110] Reference Figure 6 , shows a schematic diagram of a computer program product provided in an embodiment of the present application, including a computer program / instruction, which, when executed by a processor, implements the fine-tuning method of the multimodal large language model described in the embodiment.

[0111] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0112] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, apparatuses, or computer program products. Therefore, the embodiments of the present application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0116] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0117] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0118] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A fine-tuning method for a multimodal large language model, characterized in that: The method comprises: Based on the tuning dataset for the current task, determine the sensitivity of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model to the current task, and determine the importance of each modality in the current task; Allocating low-rank adapters to the Transformer network layers of the large language model and the visual encoder according to the sensitivity of each Transformer network layer to the current task and the importance of each modality in the current task, and determining a low-rank adapter allocation result corresponding to the current task; Fine-tune the multimodal large language model according to the training data set of the current task and the low-rank adapter allocation result corresponding to the current task.

2. The method according to claim 1, characterized in that Determine the sensitivity of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model to the current task, including: During the process of the multimodal large language model processing the tuning dataset, the gradient norms of each Transformer network layer of the large language model and the visual encoder in the multimodal large language model are respectively determined, and the gradient norms are used as the sensitivity of each Transformer network layer to the current task.

3. The method according to claim 1, characterized in that Determine the importance of each modality in the current task, including: During processing of the tuning dataset by the multimodal large language model, determining a difficulty score of each modality of the current task based on the gradient norm of each modality in the tuning dataset of the current task; The importance of each modality in the current task is determined according to the difficulty score of each modality of the current task.

4. The method according to claim 3, characterized in that The formula for determining the importance of each modality in the current task is: in, is the importance of the modality M of the current task t; Score the difficulty of the modality M of the current task t; the value of M includes and ; Score the difficulty of the text modality of the current task t; Score the difficulty of the visual modality of the current task t; A tuning dataset for the current task; is the weight of the modality M in the pre-trained multimodal large language model.

5. The method according to claim 1, characterized in that According to the sensitivity of each Transformer network layer to the current task and the importance of each modality in the current task, low-rank adapters are allocated to the Transformer network layers of the large language model and the visual encoder, and a low-rank adapter allocation result corresponding to the current task is determined, including: Determining the number of low-rank adapters allocated to the large language model and the visual encoder respectively according to the importance of each modality in the current task and the low-rank adapter allocation budget corresponding to the current task; Sort the Transformer network layers of the large language model and the visual encoder in descending order of sensitivity; According to the number of low-rank adapters allocated to the large language model, one low-rank adapter is allocated to each Transformer network layer of the large language model, and the Transformer network layer with the top N1 sensitivity is respectively allocated one low-rank adapter, and the number of low-rank adapters allocated to the large language model is the same as N1; According to the number of low-rank adapters allocated to the visual encoder, a low-rank adapter is allocated to each Transformer network layer of the visual encoder, and the Transformer network layer with the top N2 sensitivity is respectively allocated a low-rank adapter, and the number of low-rank adapters allocated to the visual encoder is the same as N2; The low-rank adapters assigned to the large language model and the Transformer network layer of the visual encoder are used as the low-rank adapter assignment results corresponding to the current task.

6. The method according to claim 5, characterized in that According to the importance of each modality in the current task and the low-rank adapter allocation budget corresponding to the current task, the formula for proportionally determining the number of low-rank adapters allocated to the large language model and the visual encoder is: in, allocating a quantity to the low-rank adapter; Allocating a budget for the low-rank adapter corresponding to the current task; is the importance of each modality M in the current task t; the value of M includes and , when M is hour, Assign the number of low-rank adapters to the large language model under the current task, when M is hour, Assign the number of low-rank adapters to the visual encoder for the current task.

7. The method according to claim 1, characterized in that Fine-tuning the multimodal large language model according to the training data set of the current task and the low-rank adapter allocation result corresponding to the current task, including: Extracting sample embeddings corresponding to each sample data in the tuning dataset of the current task, so as to train the autoencoder corresponding to the current task based on the sample embeddings corresponding to each sample data in the tuning dataset and minimizing the reconstruction loss function; Embedding the samples corresponding to each sample data in the tuning dataset, inputting them into the autoencoders of each historical task respectively, and determining the reconstruction loss corresponding to the autoencoders of each historical task; Determining, based on the reconstruction losses corresponding to the autoencoders of the respective historical tasks, a low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss; According to the low-rank adapter allocation result corresponding to the current task and the low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss, the multimodal large language model is fine-tuned based on the training data set of the current task.

8. The method according to claim 7, characterized in that Fine-tuning the multimodal large language model based on the training dataset of the current task according to the low-rank adapter allocation result corresponding to the current task and the low-rank adapter allocation result corresponding to the historical task with the minimum reconstruction loss, including: During fine-tuning training of the multimodal large language model based on the training dataset of the current task, only the low-rank adapters in the low-rank adapter allocation results corresponding to the current task are updated, and the pre-trained multimodal large language model and the low-rank adapters in the low-rank adapter allocation results corresponding to the historical task with the smallest reconstruction loss are frozen.

9. The method according to claim 8, characterized in that After fine-tuning the multimodal large language model, the method includes: Get the test dataset for the current task; Extracting text features and visual features of the test sample data in the test dataset through the multimodal large language model, performing maximum pooling on the text features and visual features of the test sample data in the test dataset, and then concatenating them to obtain sample embeddings for the test sample data; Input the sample embedding of the test sample data into the autoencoder corresponding to each task, and obtain the reconstruction loss output by the autoencoder of each task; According to the reconstruction loss output by the autoencoder of each task and the reconstruction loss threshold corresponding to each task, the activated low-rank adapter allocation result is determined, and the test sample data is processed by the multimodal large language model.

10. An electronic device, characterized in that: include: At least one processor and a memory storing a computer program executable on the processor, wherein the processor executes the method for fine-tuning a multimodal large language model according to any one of claims 1 to 9 when executing the computer program.

Images