A Multimodal Large Model Transfer Fine-tuning Question Answering Method Based on Active Learning

Abstract

This invention discloses a multimodal large-scale model transfer fine-tuning question answering method based on active learning. The method first groups the unlabeled image dataset using the K-Means clustering algorithm, extracts a predetermined proportion of samples from each cluster for labeling, forming an initial visual question answering dataset for preliminary fine-tuning of the multimodal large-scale model. Then, based on the initial fine-tuned multimodal large-scale model, the uncertainty index of the unlabeled images is calculated, and high-value samples are selected for labeling to expand the dataset. Finally, the model is iteratively fine-tuned and optimized until a predetermined termination condition is met, and the final transferred multimodal large-scale model completes the domain-specific visual question answering task. This invention significantly reduces the data labeling cost in visual question answering tasks, improves data quality and model domain adaptability, and efficiently realizes the transfer application of multimodal large-scale models in professional fields.

Description

Technical Field

[0001] This invention belongs to the field of model processing technology, specifically relating to a multimodal large model transfer fine-tuning question answering method based on active learning. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large models have shown great potential in processing and understanding multimodal information. In practical applications, it is often necessary to transfer pre-trained multimodal large models to specific professional fields, but the transfer fine-tuning process requires a large amount of training data, while the cost of acquiring high-quality data in the field is high.

[0003] In scenarios requiring highly specialized knowledge, such as medical diagnosis, remote sensing analysis, and industrial manufacturing, traditional full-data annotation not only consumes significant expert resources but also severely restricts the efficiency of deploying large multimodal models. Active learning allows models to autonomously select the most valuable data for annotation and learning, offering significant advantages in application scenarios where multimodal data annotation is costly and resources are limited.

[0004] However, while traditional active learning methods can reduce the number of labeled samples in unimodal tasks such as image classification, object detection, and text classification through uncertainty sampling strategies, they have not yet been effectively adapted to complex multimodal tasks such as visual question answering (VQA). Existing active learning methods for complex tasks have two key limitations: First, at the paradigm design level, most methods do not fully recognize that large multimodal models themselves can introduce active learning mechanisms, and still reduce them to static annotation tools rather than dynamic training subjects, failing to leverage the transfer learning potential of large models in feature representation and uncertainty quantification; second, in the annotation cost modeling process, current active learning methods for visual question answering tasks treat the combination of image and question as atomic unlabeled units, ignoring the domain knowledge threshold and annotation difficulty required for question design in professional fields. This simplification significantly increases the overall annotation cost, severely restricting the effectiveness of the methods in practical applications.

[0005] Therefore, there is an urgent need for a method that can adapt to complex multimodal tasks, effectively reduce annotation costs, and improve model transfer efficiency. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a multimodal large model transfer fine-tuning question-answering method based on active learning.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A multimodal large model transfer fine-tuning question answering method based on active learning includes the following steps:

[0009] S1. Input an unlabeled image dataset and group the images using the K-Means clustering algorithm;

[0010] S2. Extract a preset proportion of image samples from each cluster obtained in step S1, and label the question-answer pairs to form an initial visual question-answering dataset;

[0011] S3. Use the initial visual question answering dataset obtained in step S2 to perform preliminary fine-tuning on the pre-trained multimodal large model to obtain the initial fine-tuned multimodal large model;

[0012] S4. Based on the initial fine-tuned multimodal large model obtained in step S3, calculate the uncertainty index of the unlabeled image; the uncertainty index is a weighted sum of entropy value, minimum confidence level and normalized confidence interval value;

[0013] S5. For each cluster obtained in step S1, select the unlabeled image with the largest uncertainty index calculated in step S4 and label it.

[0014] S6. Merge the newly labeled visual question answering data from step S5 with the initial visual question answering dataset obtained in step S2, and then perform a second fine-tuning on the primary fine-tuning multimodal large model obtained in step S3.

[0015] S7. Repeat steps S4 to S6 until the preset iteration termination condition is met to obtain the final transferred multimodal large model, and complete the domain visual question answering task through the final transferred multimodal large model.

[0016] Preferably, the specific process of step S1 is as follows:

[0017] S11, Input size is Using an unlabeled image dataset, the visual features of each unlabeled image are extracted using a pre-trained multimodal large model visual encoder V. Where i is the number of the unlabeled image, and the i-th unlabeled image is represented as... , 0≤i≤ ;

[0018] S12, for sizes of The unlabeled image dataset contains a total of Image visual features Based on domain knowledge and experience, the number of clusters was set to k, and the K-Means clustering algorithm was used to analyze the visual features of the images. Perform clustering and grouping;

[0019] S13. Iteratively optimize the cluster centers until convergence, and output the grouping results. ,in, This represents the results of the first k groups.

[0020] Preferably, in step S3, the specific process of the initial fine-tuning is as follows: during the training process, the initial learning rate is set to 0.00002, the learning rate scheduler is cosine, the learning rate warm-up ratio is 0.03, the global effective batch size for training is 64, and the multimodal large model is trained for one round to obtain the initial fine-tuned multimodal large model.

[0021] Preferably, the specific process of step S4 is as follows:

[0022] S41. The problem of randomly sampling N labeled images. The i-th problem is represented as , 0≤i≤N; the problem of sampling With M unlabeled images Combine them to get the total. Image-Question Pair ;in, , , ;

[0023] S42, will Image-Question Pair The image visual features are extracted by a visual encoder after being fed into a primary fine-tuned multimodal large model. Then, the text encoder extracts the language features of the problem. ;

[0024] S43, Fusion of image visual features With the characteristics of the problem language To obtain multimodal joint features ;

[0025] S44, Based on multimodal joint features Primary fine-tuning of the probability distribution of the output answer token for multimodal large model inference. Where x represents the image problem input pair ;y represents the predicted token;

[0026] S45. Calculate the entropy value of each token using the following formula: ,in, For the first The entropy value of each token; For the index of the token in the vocabulary, ; For vocabulary size; The index for the vocabulary is A specific token; For a given image-question pair Under the condition that the model predicts the first answer Each token is The probability of;

[0027] S46. Calculate the minimum confidence level for each token using the following formula: ,in, For the first Minimum confidence level of a token; (·) indicates taking the maximum value; For a given image-question pair Under the condition that the model predicts the first answer Each token is any candidate from the vocabulary. The probability of;

[0028] S47. Calculate the confidence interval for each token using the following formula: ,in, For the first The confidence interval of each token; and These represent the tokens corresponding to the highest probability value and the second highest probability value, respectively. For a given image-question pair Under the condition that the model predicts the first answer Each token represents the highest probability in the vocabulary. The probability of; For a given image-question pair Under the condition that the model predicts the first answer This token is the second most probable in the vocabulary. The probability of;

[0029] S48. Calculate the overall uncertainty using the following formula: ,in, To account for uncertainties; , and These represent the preset weighting coefficients for the entropy value, minimum confidence level, and confidence interval, respectively; avg(·) represents taking the average value; min(·) represents taking the minimum value;

[0030] S49. Take the minimum value of the combined uncertainty of N problems as the uncertainty index of image I: ,in, This is the uncertainty index for image I.

[0031] Preferably, the specific process of step S5 is as follows:

[0032] S51. For unlabeled images in each cluster, sort them in descending order of their uncertainty index;

[0033] S52. Select the first M images for annotation, where, ; This is the preset sampling ratio coefficient; This is a preset ratio.

[0034] Preferably, in step S6, the specific process of the secondary fine-tuning is as follows: During the training process, the initial learning rate is set to 0.00002, the learning rate scheduler is cosine, the learning rate warm-up ratio is 0.03, the global effective batch size for training is 64, and the primary fine-tuning of the multimodal large model is performed by training... Rounds, training rounds Adjustments should be made based on the specific task scenario.

[0035] Preferably, the iteration termination condition in step S7 includes at least one of the following:

[0036] (a) Reaching the preset maximum number of iterations;

[0037] (b) The total amount of labeled data reaches a preset threshold;

[0038] (c) The accuracy improvement of the multimodal large model on the validation set is less than the preset threshold.

[0039] By adopting the above technical solution, the present invention has the following beneficial effects:

[0040] 1. This invention eliminates the need for pre-labeled images. It ensures sample richness through cluster sampling and guarantees sample labeling value by combining uncertainty sampling. This significantly reduces the number of labeled samples, lowers labeling resource consumption, and can significantly reduce labeling costs.

[0041] 2. This invention selects and labels the most valuable samples for model optimization through an active learning mechanism, which effectively improves the overall quality of training data and provides strong support for model performance optimization.

[0042] 3. The uncertainty index of active learning in this invention (weighted sum of entropy, minimum confidence and normalized confidence interval values) can be adaptively adjusted by preset weight parameters, which can flexibly adapt to the needs of various professional scenarios such as medical diagnosis and remote sensing analysis, and improve the transfer effect of the model in specific fields. Attached Figure Description

[0043] Figure 1 This is a flowchart of the present invention;

[0044] Figure 2 This is a flowchart of the present invention;

[0045] Figure 3 This is a schematic diagram of the image clustering process of the present invention;

[0046] Figure 4 This is a schematic diagram illustrating the calculation of the uncertainty index of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] like Figures 1 to 4 As shown, a multimodal large model transfer fine-tuning question-answering method based on active learning includes the following steps:

[0049] S1. Input an unlabeled image dataset and group the images using the K-Means clustering algorithm;

[0050] The specific process of step S1 is as follows:

[0051] S11, Input size is Using an unlabeled image dataset, the visual features of each unlabeled image are extracted using a pre-trained multimodal large model visual encoder V. Where i is the number of the unlabeled image, and the i-th unlabeled image is represented as... , 0≤i≤ ;

[0052] S12, for sizes of The unlabeled image dataset contains a total of Image visual features Based on domain knowledge and experience, the number of clusters was set to k, and the K-Means clustering algorithm was used to analyze the visual features of the images. Perform clustering and grouping;

[0053] S13. Iteratively optimize the cluster centers until convergence, and output the grouping results. ,in, This represents the results of the first k groups;

[0054] S2. Extract a preset proportion of image samples from each cluster obtained in step S1, and label the question-answer pairs to form an initial visual question-answering dataset;

[0055] S3. Use the initial visual question answering dataset obtained in step S2 to perform preliminary fine-tuning on the pre-trained multimodal large model to obtain the initial fine-tuned multimodal large model;

[0056] In step S3, the specific process of the initial fine-tuning is as follows: During the training process, the initial learning rate is set to 0.00002, the learning rate scheduler is cosine, the learning rate warm-up ratio is 0.03, the global effective batch size for training is 64, and the multimodal large model is trained for one round to obtain the initial fine-tuned multimodal large model.

[0057] S4. Based on the initial fine-tuned multimodal large model obtained in step S3, calculate the uncertainty index of the unlabeled image; the uncertainty index is a weighted sum of entropy value, minimum confidence level and normalized confidence interval value;

[0058] The specific process of step S4 is as follows:

[0059] S41. The problem of randomly sampling N labeled images. The i-th problem is represented as , 0≤i≤N; the problem of sampling With M unlabeled images Combine them to get the total. Image-Question Pair ;in, , , ;

[0060] S42, will Image-Question Pair The image visual features are extracted by a visual encoder after being fed into a primary fine-tuned multimodal large model. Then, the text encoder extracts the language features of the problem. ;

[0061] S43, Fusion of image visual features With the characteristics of the problem language To obtain multimodal joint features ;

[0062] S44, Based on multimodal joint features Primary fine-tuning of the probability distribution of the output answer token for multimodal large model inference. Where x represents the image problem input pair ;y represents the predicted token;

[0063] S45. Calculate the entropy value of each token using the following formula: ,in, For the first The entropy value of each token; For the index of the token in the vocabulary, ; For vocabulary size; The index for the vocabulary is A specific token; For a given image-question pair Under the condition that the model predicts the first answer Each token is The probability of;

[0064] S46. Calculate the minimum confidence level for each token using the following formula: ,in, For the first Minimum confidence level of a token; (·) indicates taking the maximum value; For a given image-question pair Under the condition that the model predicts the first answer Each token is any candidate from the vocabulary. The probability of;

[0065] S47. Calculate the confidence interval for each token using the following formula: ,in, For the first The confidence interval of each token; and These represent the tokens corresponding to the highest probability value and the second highest probability value, respectively. For a given image-question pair Under the condition that the model predicts the first answer Each token represents the highest probability in the vocabulary. The probability of; For a given image-question pair Under the condition that the model predicts the first answer This token is the second most probable in the vocabulary. The probability of;

[0066] S48. Calculate the overall uncertainty using the following formula: ,in, To account for uncertainties; , and These represent the preset weighting coefficients for the entropy value, minimum confidence level, and confidence interval, respectively; avg(·) represents taking the average value; min(·) represents taking the minimum value;

[0067] S49. Take the minimum value of the combined uncertainty of N problems as the uncertainty index of image I: ,in, This is an indicator of the uncertainty of image I;

[0068] S5. For each cluster obtained in step S1, select the unlabeled image with the largest uncertainty index calculated in step S4 and label it.

[0069] The specific process of step S5 is as follows:

[0070] S51. For unlabeled images in each cluster, sort them in descending order of their uncertainty index;

[0071] S52. Select the first M images for annotation, where, ; This is the preset sampling ratio coefficient; This is a preset ratio;

[0072] S6. Merge the newly labeled visual question answering data from step S5 with the initial visual question answering dataset obtained in step S2, and then perform a second fine-tuning on the primary fine-tuning multimodal large model obtained in step S3.

[0073] In step S6, the specific process of the secondary fine-tuning is as follows: During training, the initial learning rate is set to 0.00002, the learning rate scheduler is cosine, the learning rate warm-up ratio is 0.03, the global effective batch size for training is 64, and the primary fine-tuning of the multimodal large model is performed by training... Rounds, training rounds Adjustments should be made according to the specific task scenario;

[0074] S7. Repeat steps S4 to S6 until the preset iteration termination condition is met, obtain the final transferred multimodal large model, and complete the domain visual question answering task through the final transferred multimodal large model.

[0075] The iteration termination condition in step S7 includes at least one of the following:

[0076] (a) Reaching the preset maximum number of iterations;

[0077] (b) The total amount of labeled data reaches a preset threshold;

[0078] (c) The accuracy improvement of the multimodal large model on the validation set is less than the preset threshold.

[0079] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An active learning-based multi-modal large model transfer fine-tuning question and answer method, characterized in that, The method comprises the following steps: S1, inputting an unlabeled image dataset, and grouping the images by using a K-Means clustering algorithm; The specific process of step S1 is as follows: S11、input a unlabeled image dataset with size , extract image visual features of each unlabeled image using a pre-trained multimodal large model visual encoder V ; where i is the number of unlabeled images, the i-th unlabeled image is represented as , 0≤i≤ ; S12, for the size of unlabeled image data set, a total of image visual features , and based on the experience of domain knowledge, the clustering number k is set, and the K-Means clustering algorithm is used to cluster the image visual features grouping; S13, iteratively optimize the clustering centers until convergence, output the grouping result wherein, is the top k grouping result; S2, extracting image samples in a preset proportion from each cluster obtained in step S1, and labeling a question-answer pair to form an initial visual question answering dataset; S3, using the initial visual question answering dataset obtained in step S2 to preliminarily fine-tune a pre-trained multi-modal large model to obtain a primary fine-tuned multi-modal large model; S4, calculating an uncertainty index of the unlabeled image based on the primary fine-tuned multi-modal large model obtained in step S3; the uncertainty index is a weighted sum of an entropy value, a minimum confidence, and a confidence interval normalization value; S5, for each cluster obtained in step S1, selecting an unlabeled image with the maximum uncertainty index calculated in step S4 for labeling; S6, combining the newly labeled visual question answering data in step S5 with the initial visual question answering data obtained in step S2, and further fine-tuning the primary fine-tuned multi-modal large model obtained in step S3; S7, repeating steps S4 to S6 until a preset iteration termination condition is met, obtaining a final transferred multi-modal large model, and completing a visual question answering task in the field through the final transferred multi-modal large model.

2. The multi-modal large model transfer fine-tuning question and answer method based on active learning according to claim 1, wherein, In step S3, the specific process of the preliminary fine-tuning is as follows: in the training process, the initial learning rate is set to 0.00002, the learning rate scheduler is cosine, the learning rate preheating ratio is 0.03, the global effective batch size for training is 64, and the multi-modal large model is trained for one round to obtain the primary fine-tuned multi-modal large model.

3. The multi-modal large model transfer fine-tuning question answering method based on active learning according to claim 1, wherein, The specific process of step S4 is as follows: S41, randomly sampling N labeled images of a question where the ith question is represented as , 0≤i≤N; combining the sampled question with M unlabeled images to obtain a total of image-question pairs ; where , , ; S42、to image-question pairs input to a primary fine-tuned multimodal large model, extracting image visual features through a visual encoder and question linguistic features through a text encoder ; S43, fusing image visual features with problem language features , obtaining multimodal joint features ; S44, based on the multi-modal joint feature , the probability distribution of the primary fine-tuning multi-modal large model inference output answer token ; wherein x represents an image problem input pair ; y represents a predicted token; S45, calculate the entropy value of each token, the calculation formula is: wherein, is the entropy value of the th token; is the index of the token in the vocabulary, ; is the size of the vocabulary; is a specific token with index in the vocabulary; is the probability that the th token of the answer is predicted by the model under the condition of a given image-question pair ; S46, calculate the minimum confidence of each token, the calculation formula is: wherein, is the minimum confidence of the th token; (·) represents taking the maximum value; is the probability that the th token predicted by the model is any one candidate in the vocabulary under the condition of a given image-question pair ; S47, calculate the confidence interval of each token, the calculation formula is: wherein, is the confidence interval of the th token; and respectively represent the token corresponding to the maximum probability value and the second largest probability value; is the probability that the th token of the model's prediction answer is the most probable word in the vocabulary given the image-question pair ; is the probability that the th token of the model's prediction answer is the second most probable word in the vocabulary given the image-question pair ; is the probability that the th token of the model's prediction answer is the second most probable word in the vocabulary given the image-question pair ; S48, calculating the comprehensive uncertainty, the calculation formula is: wherein, is the comprehensive uncertainty; , and respectively represent the preset weight coefficients of the entropy value, the minimum confidence and the confidence interval; avg(·) represents taking the average value; min(·) represents taking the minimum value; S49, take the minimum value of N question comprehensive uncertainty as the uncertainty index of image I: wherein, is the uncertainty index of image I.

4. The multi-modal large model transfer fine-tuning question answering method based on active learning according to claim 1, wherein, The specific process of step S5 is as follows: S51, arranging the unlabeled images in each cluster in descending order according to their uncertainty indexes; S52, selecting the first M images for labeling, wherein, ; is a preset sampling ratio coefficient; is a preset ratio.

5. The multi-modal large model transfer fine-tuning question answering method based on active learning according to claim 1, wherein, The specific process of the secondary fine-tuning in step S6 is as follows: in the training process, the initial learning rate is set to 0.00002, the learning rate scheduler is cosine, the learning rate preheating ratio is 0.03, the global effective batch size of training is 64, the primary fine-tuning multi-modal large model is trained for 1000 epochs training epochs Adjust according to specific task scenarios.

6. The multi-modal large model transfer fine-tuning question answering method based on active learning according to claim 1, wherein, The iteration termination condition in step S7 comprises at least one of the following: (a) reaching a preset maximum number of iterations; (b) the total amount of labeled data reaching a preset threshold; (c) the accuracy of the multi-modal large model on the validation set improving by less than a preset threshold.

Images