Parameter fusion method, device and equipment of multi-modal large model and medium

Abstract

The invention discloses a multi-modal large model parameter fusion method, device, equipment and medium, and the method comprises the steps: firstly dividing a universal multi-modal large model into a visual encoder, a language model and a projection layer, and then carrying out the fine adjustment of the universal multi-modal large model through vertical domain data, so as to obtain a vertical domain professional model; then visual encoder parameters, language model parameters and projection layer parameters are extracted from the general multi-modal large model and the vertical domain professional model respectively, and corresponding parameters of the general multi-modal large model and the vertical domain professional model are fused according to a preset weight ratio based on module parameters. And determining an optimal weight ratio combination of each module according to a performance evaluation result of the target data set, and finally generating a fused multi-modal model based on the optimal weight ratio combination. The fused multi-modal model can ensure the universality of the model under the condition of reducing the fine tuning cost of the model, and meanwhile, the performance of the model in the vertical field is remarkably improved, so that the application breadth and depth of the model are improved.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal large models, and more specifically, to a parameter fusion method, apparatus, device, and medium for a multimodal large model. Background Art

[0002] In recent years, large, general-purpose multimodal models, by integrating visual encoders, language models, and cross-modal projection layers and pre-training on massive amounts of general-purpose data, have demonstrated excellent general-purpose performance in multimodal tasks such as image understanding and text generation. However, the effectiveness of such models in vertical fields such as medical image analysis and industrial quality inspection is significantly limited. This is primarily due to the lack of specialized domain data in their training corpus, resulting in insufficient learning of domain-specific features and knowledge.

[0003] To improve performance in specific scenarios, traditional approaches typically employ a full-parameter fine-tuning strategy, retraining model parameters on domain data. However, this approach has a serious flaw: after fine-tuning, the model becomes overly adapted to the specific task, causing a significant loss of general knowledge and resulting in a drastic drop in general-task performance. This "general-vertical capability imbalance" significantly restricts the model's deployment flexibility in real-world scenarios.

[0004] In response to the above contradictions, the industry has tried to reconcile capability conflicts through model fusion technology, such as interpolating parameters or weight averaging the original pre-trained model and the fine-tuned model. Such methods have been verified in large language models (LLMs) to balance universality and domain specificity with a small amount of fine-tuning data. However, multimodal models face unique challenges due to the complexity of modal interaction: first, the parameter coupling of the visual and text modality encoders makes single modality adjustments prone to cross-modal alignment distortion; second, the low-rank nature of the projection layer makes it sensitive to noise during the fusion process, and directly using the language model fusion strategy will lead to a decrease in modal collaboration efficiency. Therefore, existing fusion methods have not been effectively verified in multimodal scenarios, and targeted solutions are urgently needed. Summary of the Invention

[0005] The purpose of this application is to provide a parameter fusion method, device, equipment and medium for a multimodal large model in order to overcome the existing technical defects. The fused multimodal model can ensure the general capabilities of the model while reducing the cost of model fine-tuning, and significantly improve its performance in vertical fields, thereby increasing the breadth and depth of application of the model.

[0006] The purpose of this application is achieved through the following technical solutions:

[0007] In a first aspect, the present application proposes a parameter fusion method for a multimodal large model, the method comprising:

[0008] S1. Divide the general multimodal large model into visual encoder, language model and projection layer;

[0009] S2. Use vertical domain data to fine-tune the general multimodal large model to obtain a vertical domain professional model;

[0010] S3. Extracting module parameters from the general multimodal large model and the vertical domain specialized model, respectively. The module parameters include visual encoder parameters, language model parameters, and projection layer parameters.

[0011] S4. Based on the module parameters, the corresponding parameters of the general multimodal large model and the vertical domain professional model are integrated according to the preset weight ratio;

[0012] S5. Obtain the optimal weight ratio combination of each module based on the performance evaluation results of the target data set;

[0013] S6. Generate a fused multimodal model based on the optimal weight ratio combination.

[0014] In a possible implementation, step S5 includes:

[0015] According to the performance evaluation results of the target dataset, multiple candidate ratios are preset for the weight ratios of the visual encoder, language model, and projection layer. All candidate weight combinations are traversed through grid search, and the weight combination with the minimum loss function value on the target dataset is taken as the optimal weight ratio combination.

[0016] In a possible implementation, the candidate weight ratio of each module is set independently.

[0017] In one possible implementation, the target data set contains mixed data of general knowledge and vertical domain knowledge, and the mixing ratio is set according to business scenario requirements.

[0018] In one possible implementation, the fine-tuned vertical domain specialized model retains the same module division structure as the general multimodal large model.

[0019] In a second aspect, the present application also proposes a parameter fusion device for a multimodal large model, the device comprising:

[0020] A partitioning module that divides a large, general multimodal model into visual encoder, language model, and projection layers;

[0021] The fine-tuning module is used to fine-tune the general multimodal large model using vertical domain data to obtain a vertical domain professional model;

[0022] An extraction module is used to extract module parameters from the general multimodal large model and the vertical domain professional model respectively. The module parameters include visual encoder parameters, language model parameters, and projection layer parameters;

[0023] The parameter fusion module is used to fuse the corresponding parameters of the general multimodal large model and the vertical domain professional model based on the module parameters and the preset weight ratio;

[0024] The first generation module is used to obtain the optimal weight ratio combination of each module based on the performance evaluation results of the target data set;

[0025] The second generation module is used to generate a fused multimodal model based on the optimal weight ratio combination.

[0026] In a possible implementation, the first generating module is further configured to:

[0027] According to the performance evaluation results of the target dataset, multiple candidate ratios are preset for the weight ratios of the visual encoder, language model, and projection layer. All candidate weight combinations are traversed through grid search, and the weight combination with the minimum loss function value on the target dataset is taken as the optimal weight ratio combination.

[0028] In a third aspect, the present application further proposes a computer device comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the parameter fusion method as described in any one of the first aspects.

[0029] In a fourth aspect, the present application further proposes a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the parameter fusion method as described in any one of the first aspects.

[0030] The above-mentioned main scheme of this application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and protected by this application; and in this application, (non-conflicting options) can also be freely combined with each other and with other options. After understanding the scheme of this application, those skilled in the art will understand that there are many combinations based on existing technology and common knowledge, all of which are technical solutions to be protected by this application, and they are not exhaustive here.

[0031] The present application discloses a parameter fusion method, device, equipment and medium for a multimodal large model. First, the general multimodal large model is divided into a visual encoder, a language model and a projection layer. Then, the general multimodal large model is fine-tuned using vertical domain data to obtain a vertical domain professional model. Then, the visual encoder parameters, language model parameters and projection layer parameters are extracted from the general multimodal large model and the vertical domain professional model respectively. Based on the module parameters, the corresponding parameters of the general multimodal large model and the vertical domain professional model are fused according to the preset weight ratio. The optimal weight ratio combination of each module is determined based on the performance evaluation results of the target data set, and finally the fused multimodal model is generated based on this optimal weight ratio combination. The fused multimodal model can ensure the general capabilities of the model while reducing the cost of model fine-tuning, while significantly improving its performance in the vertical field, thereby improving the breadth and depth of application of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flow chart of a parameter fusion method for a multimodal large model proposed in an embodiment of the present application is shown.

[0034] Figure 2 A schematic diagram of a model for multimodal large model parameter fusion proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0036] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0037] Existing model fusion methods combine an original general pre-trained model with a fine-tuned vertical domain model while reducing the amount of fine-tuning data. This allows the fused model to maintain both vertical domain and general capabilities. However, previous work has only applied this approach to large language models and has not attempted to apply it to large multimodal models.

[0038] Because, in order to solve the above-mentioned technical problems, the embodiments of the present application propose a parameter fusion method, device, equipment and medium for a multimodal large model, which can ensure the general capabilities of the model while reducing the cost of model fine-tuning, and significantly improve its performance in vertical fields, thereby improving the application breadth and depth of the model. The method is described in detail below.

[0039] Please refer to Figure 1 , Figure 1 A flow chart of a parameter fusion method for a multimodal large model proposed in an embodiment of the present application is shown, and the method includes:

[0040] S1. Divide the general multimodal large model into visual encoder, language model and projection layer;

[0041] The general multimodal large model is a model that can process data from multiple modalities. To effectively process these multimodal data, the model is divided into a visual encoder, a language model, and a projection layer. The visual encoder processes input data such as images and encodes them into feature representations that the model can handle. It is composed of convolutional neural networks, visual transformers, etc., and can extract spatial, texture, shape and other features from images. The language model processes text data and encodes text into semantic feature representations. Based on the Transformer architecture, it can capture grammatical, semantic, and contextual information in the text and understand the deep meaning of the text. Projection layer: Connects the visual encoder and the language model, maps features from different modalities to the same feature space for effective fusion. Through the projection layer, visual features and language features can interact and combine in a unified representation space.

[0042] S2. Use vertical domain data to fine-tune the general multimodal large model to obtain a vertical domain professional model;

[0043] When applying large multimodal models in business scenarios, pre-trained models are often based on general data and lack vertical domain-specific data, resulting in poor performance in vertical domains. To address this issue, it is necessary to fine-tune the general multimodal large model using vertical domain data to obtain specialized vertical domain models tailored to specific domain needs. This process involves two main steps: building a vertical domain dataset and fine-tuning the model. This allows the model to better understand and process multimodal data within the vertical domain, improving its application effectiveness in specific business scenarios.

[0044] Collect relevant data from the target business scenario and organize it in the format required by the pre-trained model to form a dedicated training dataset. Using this constructed vertical multimodal dataset, fine-tune the pre-trained multimodal large model. Through this process, the model learns domain-specific knowledge and features, thereby improving its performance in the vertical domain task.

[0045] S3. Extracting module parameters from the general multimodal large model and the vertical domain specialized model, respectively. The module parameters include visual encoder parameters, language model parameters, and projection layer parameters.

[0046] The corresponding module parameters are extracted from the general multimodal large model and the vertical domain professional model that has been fine-tuned with vertical domain data. The module parameters mainly include visual encoder parameters, language model parameters, and projection layer parameters. The visual encoder parameters are responsible for processing and encoding visual information such as images, and can capture various features in the image, such as edges, textures, shapes, etc. After being trained with a large amount of diverse image data, it has strong general visual feature extraction capabilities. The language model parameters are used to process text information and capture the grammar, semantics, and contextual relationships of the text. The language parameters of the general model have been trained with extensive text data and can understand general language knowledge. The projection layer parameters map visual and language features to the same feature space to achieve the fusion of data from different modalities.

[0047] S4. Based on the module parameters, the corresponding parameters of the general multimodal large model and the vertical domain professional model are integrated according to the preset weight ratio;

[0048] Typically, fine-tuning a general multimodal large model using only vertical domain data will improve the model's performance in a specific domain, but its general capabilities will often decline. Therefore, to ensure that the model performs well in both general and vertical domains, it is necessary to fuse the parameters of the general multimodal large model and the vertical domain specialized models.

[0049] The fusion operation targets module parameters extracted from the general multimodal large model and the vertical domain specialized model, specifically including visual encoder parameters, language model parameters, and projection layer parameters. These parameters represent the model's capabilities in processing visual information, text information, and performing multimodal fusion.

[0050] According to the preset weight ratio, the corresponding parameters of the general multimodal large model and the vertical domain professional model are integrated. The weight ratio here is pre-set based on the performance requirements of the model in the general field and the vertical field. By assigning different weights to different modules, a balance between the general and vertical capabilities of the model can be achieved. For example, if the performance of a certain module in the general model is more important, then the parameters of this module in the general model can be given a greater weight; conversely, if the performance of a certain module in the vertical domain professional model is more critical, then the weight of the parameters of this module in the vertical domain professional model can be increased.

[0051] Compared to traditional fine-tuning solutions, this method significantly reduces the amount of fine-tuning data required and the workload of constructing fine-tuning data. Furthermore, unlike general model fusion methods, this application modularizes the model and assigns different weights to each module for fusion, thereby more finely controlling the fusion process and improving the fusion effect.

[0052] S5. Obtain the optimal weight ratio combination of each module based on the performance evaluation results of the target data set;

[0053] By evaluating the performance of the fused multimodal model on the target dataset, we can determine the performance of each visual encoder, language model, and projection layer under different weight ratios. Based on the evaluation results, we determine the weight combination that optimizes model performance, i.e., the optimal weight ratio combination for each module.

[0054] The target dataset contains a mixture of general knowledge and vertical domain knowledge, and the mixing ratio is set according to the business scenario requirements.

[0055] The target dataset is designed to evaluate and optimize the comprehensive performance of multimodal models in general and vertical fields. It cleverly combines general knowledge and vertical field knowledge, and the mixing ratio is finely adjusted according to business scenario requirements. It plays a key role in model evaluation and optimization. By testing and verifying on the target dataset, we can accurately understand the performance of the fused multimodal model in different tasks and scenarios, thereby providing a strong basis for determining the optimal parameter fusion weights.

[0056] The steps of S5 include:

[0057] According to the performance evaluation results of the target dataset, multiple candidate ratios are preset for the weight ratios of the visual encoder, language model, and projection layer. All candidate weight combinations are traversed through grid search, and the weight combination with the minimum loss function value on the target dataset is taken as the optimal weight ratio combination.

[0058] Candidate weight ratios are preset for the visual encoder, language model, and projection layer, and the weight ratios for each module are selected independently. The purpose of presetting multiple candidate weight ratios is to provide diverse weight options when fusing the parameters of a general multimodal large model with specialized vertical domain models, increasing the likelihood of finding the optimal weight combination and achieving the optimal balance between the model's general and vertical domain capabilities.

[0059] Afterwards, a grid search method is used to systematically traverse all preset weight combinations. Grid search is a reliable optimization strategy that ensures that every possible weight combination is fully explored within the specified range of candidate weights, without missing out on potential optimal solutions. Inference is performed on the target dataset, and the loss function value of the model is calculated for each weight combination. The loss function value is an important indicator to measure the difference between the model's prediction results and the true value. The smaller the value, the better the model performance. Selecting the weight combination that minimizes the loss function value means that under this combination, the prediction results of the fused model on the target dataset are closest to the true situation, and the overall performance of the model is the best. The weight combination corresponding to this minimum loss function value is used as the optimal weight ratio combination to determine the specific weight distribution of each module in the fusion process, thereby achieving effective fusion of model parameters and improving the overall performance of the model in general and vertical fields.

[0060] The candidate weight ratio of each module is set independently.

[0061] During the parameter fusion process, nine candidate weight ratios were set for the visual encoder, language model, and projection layers: 9:1, 8:2, 7:3, ..., and 1:9. The weight ratios for each module were chosen independently, allowing subsequent optimization to select the most appropriate weight combination for each module based on evaluation results on the target dataset.

[0062] S6. Generate a fused multimodal model based on the optimal weight ratio combination.

[0063] After determining the optimal weight combination of the visual encoder, language model, and projection layer through grid search, this combination is used to fuse the corresponding parameters of the original pre-trained model and the vertical domain specialized model to generate a new multimodal model. This new model inherits the general capabilities of the pre-trained model and the specialized capabilities of the vertical domain model. While maintaining versatility, it significantly improves performance in vertical domains, thereby expanding the model's application scope and enhancing its performance and applicability in real-world business scenarios.

[0064] Please refer to Figure 2 , Figure 2A schematic diagram of a model for the fusion of multimodal large model parameters proposed in an embodiment of the present application is shown. First, a pre-trained general multimodal large model (pre-trained general VLM) is used to fine-tune the general VLM using vertical domain data to obtain a vertical domain professional VLM adapted to a specific domain, and the parameters of the visual encoder, language model, and projection layer are extracted from the pre-trained general VLM. Afterwards, the parameters of the visual encoder, language model, and projection layer are also extracted from the vertical domain professional VLM, and the extracted visual encoder parameters, language model parameters, and projection layer parameters are weighted fused respectively. For each module (visual encoder, language model, and projection layer), different weight ratios are assigned for fusion. The weight ratio is a pre-set candidate ratio, and the optimal weight combination is determined by a grid search method. The fused visual encoder, language model, and projection layer parameters are combined into a new target VLM model.

[0065] This target VLM model retains the general capabilities of the pre-trained general VLM while integrating the domain-specific capabilities of the vertical domain-specific VLM. The fine-tuned vertical domain-specific model retains the same modular structure as the general multimodal large model.

[0066] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0067] First, the fused model maintains the general capabilities of the pre-trained model in general fields, and enhances professional performance in vertical fields by fusion of fine-tuning data, achieving simultaneous optimization of general and vertical capabilities.

[0068] Second, through parameter fusion, while reducing the amount of fine-tuning data, a significant decline in general capabilities is avoided, and the fine-tuning cost and data construction workload are reduced.

[0069] Third, this application is not only suitable for scenarios that require fine-tuning with vertical domain data, but can also be used in other large-model application systems that include the fusion of multiple multimodal models, and has good versatility and scalability.

[0070] A possible implementation of a parameter fusion device for a multimodal large model is given below, which is used to execute the various execution steps and corresponding technical effects of the parameter fusion method shown in the above embodiments and possible implementations.

[0071] In a second aspect, the present application also proposes a parameter fusion device for a multimodal large model, the device comprising:

[0072] A partitioning module that divides a large, general multimodal model into visual encoder, language model, and projection layers;

[0073] The fine-tuning module is used to fine-tune the general multimodal large model using vertical domain data to obtain a vertical domain professional model;

[0074] An extraction module is used to extract module parameters from the general multimodal large model and the vertical domain professional model respectively. The module parameters include visual encoder parameters, language model parameters, and projection layer parameters;

[0075] The parameter fusion module is used to fuse the corresponding parameters of the general multimodal large model and the vertical domain professional model based on the module parameters and the preset weight ratio;

[0076] The first generation module is used to obtain the optimal weight ratio combination of each module based on the performance evaluation results of the target data set;

[0077] The second generation module is used to generate a fused multimodal model based on the optimal weight ratio combination.

[0078] In a possible implementation, the first generating module is further configured to:

[0079] According to the performance evaluation results of the target dataset, multiple candidate ratios are preset for the weight ratios of the visual encoder, language model, and projection layer. All candidate weight combinations are traversed through grid search, and the weight combination with the minimum loss function value on the target dataset is taken as the optimal weight ratio combination.

[0080] This preferred embodiment provides a computer device that can implement the steps in any embodiment of the parameter fusion method for a multimodal large model provided in the embodiments of the present application. Therefore, the beneficial effects of the parameter fusion method for a multimodal large model provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0081] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be accomplished by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any embodiment of the parameter fusion method for a multimodal large model provided in the embodiment of the present application.

[0082] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0083] Since the instructions stored in the storage medium can execute the steps in the embodiment of the parameter fusion method of any multimodal large model provided in the embodiments of the present application, the beneficial effects that can be achieved by the parameter fusion method of any multimodal large model provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0084] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A parameter fusion method for a multimodal large model, characterized in that: The method comprises: S1. Divide the general multimodal large model into visual encoder, language model and projection layer; S2. Use vertical domain data to fine-tune the general multimodal large model to obtain a vertical domain professional model; S3. Extracting module parameters from the general multimodal large model and the vertical domain specialized model, respectively. The module parameters include visual encoder parameters, language model parameters, and projection layer parameters. S4. Based on the module parameters, the corresponding parameters of the general multimodal large model and the vertical domain professional model are integrated according to the preset weight ratio; S5. Obtain the optimal weight ratio combination of each module based on the performance evaluation results of the target data set; S6. Generate a fused multimodal model based on the optimal weight ratio combination.

2. The parameter fusion method according to claim 1, wherein: The steps of S5 include: According to the performance evaluation results of the target dataset, multiple candidate ratios are preset for the weight ratios of the visual encoder, language model, and projection layer. All candidate weight combinations are traversed through grid search, and the weight combination with the minimum loss function value on the target dataset is taken as the optimal weight ratio combination.

3. The parameter fusion method according to claim 2, wherein: The candidate weight ratio of each module is set independently.

4. The parameter fusion method according to claim 1, wherein: The target dataset contains a mixture of general knowledge and vertical domain knowledge, and the mixing ratio is set according to the business scenario requirements.

5. The parameter fusion method according to claim 1, wherein: The fine-tuned vertical domain specialized model retains the same module division structure as the general multimodal large model.

6. A parameter fusion device for a multimodal large model, characterized in that: The device comprises: A partitioning module that divides a large, general multimodal model into visual encoder, language model, and projection layers; The fine-tuning module is used to fine-tune the general multimodal large model using vertical domain data to obtain a vertical domain professional model; An extraction module is used to extract module parameters from the general multimodal large model and the vertical domain professional model respectively. The module parameters include visual encoder parameters, language model parameters, and projection layer parameters; The parameter fusion module is used to fuse the corresponding parameters of the general multimodal large model and the vertical domain professional model based on the module parameters and the preset weight ratio; The first generation module is used to obtain the optimal weight ratio combination of each module based on the performance evaluation results of the target data set; The second generation module is used to generate a fused multimodal model based on the optimal weight ratio combination.

7. The parameter fusion device according to claim 6, characterized in that: The first generation module is also used to: According to the performance evaluation results of the target dataset, multiple candidate ratios are preset for the weight ratios of the visual encoder, language model, and projection layer. All candidate weight combinations are traversed through grid search, and the weight combination with the minimum loss function value on the target dataset is taken as the optimal weight ratio combination.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the parameter fusion method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which is loaded and executed by a processor to implement the parameter fusion method according to any one of claims 1 to 5.

Images