Adaptive resolution based multimodal large language model optimization method and system

Abstract

The present application belongs to the field of pattern recognition and artificial intelligence, and specifically provides a multi-modal large language model optimization method and system based on adaptive resolution, comprising: dividing multi-modal question and answer data into a target task training set and a target task test set; using image data under a target scene re-acquired and paired text instructions to construct a resolution selection training set; constructing a multi-modal large language model with a two-stage image encoding mechanism, and using the target task training set and the resolution selection training set to train the multi-modal large language model to obtain a target model; evaluating the prediction accuracy of the target model on the target task test set to complete optimization of the multi-modal large language model. The technical scheme of the present application realizes adaptive selection of the best resolution for inference through the two-stage image encoding mechanism, thereby significantly improving the visual perception ability of the model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of pattern recognition and artificial intelligence, and particularly relates to a multi-modal large language model optimization method and system based on adaptive resolution. BACKGROUND

[0002] Large language models represented by the GPT series continue to iterate and upgrade in the field of natural language processing. Their emerging semantic modeling, logical reasoning, and complex instruction execution capabilities are driving artificial intelligence technology towards more general directions. On this basis, in order to break through the limitations of a single modality, researchers have expanded the model architecture to build multi-modal large language models with visual understanding and dynamic reasoning capabilities, further expanding the application scenarios and capability boundaries of large models.

[0003] Due to the limitations of image-text contrast pre-training image encoders, early multi-modal large language models could only encode images at a fixed and relatively low resolution, resulting in poor performance on tasks requiring fine-grained visual perception. To overcome this problem, some researchers have proposed a subgraph cropping strategy, which has gradually become the mainstream image processing strategy for current multi-modal large language models. Specifically, the input image is cropped into multiple low-resolution subgraphs, then the image encoder is used to independently encode each subgraph and the thumbnail, and finally all image features are concatenated and input into the large language model. By dividing the original image into different numbers of subgraphs, the model can process images of any resolution. With the further development of multi-modal large language models, some researchers have proposed methods based on rotation position encoding and large-scale training to enable image encoders to natively support any resolution.

[0004] Through the above methods, although the problem of how to process images of any resolution has been solved, another problem arises, namely, what resolution should be used to process the input image. Existing models usually process images directly at their original resolution or scale the images to a corresponding resolution by matching a pre-set fixed template according to the aspect ratio of the image. However, fixed resolution is difficult to adapt to diverse real-world scenarios, and the original resolution is often not the optimal choice. For example, when using cameras of different resolutions to take pictures of the same document or taking screenshots of a screen at different zoom levels, the same content will be presented at significantly different resolutions. When inputting images into a multi-modal large language model, if the resolution is too low, the number of visual features extracted by the image encoder will be insufficient, and the model's perception of the image will be severely biased; on the contrary, if the resolution is too high, the number of visual features will increase by several or even tens of times, not only significantly reducing the inference efficiency, but also weakening the model's perception accuracy due to the interference of redundant features. SUMMARY

[0005] To solve the problems in the prior art, the application provides a multi-modal large language model optimization method and system based on adaptive resolution, which realizes adaptive selection of the best resolution for reasoning through a two-stage image coding mechanism, thereby significantly improving the model reasoning efficiency and visual perception ability.

[0006] To achieve the above-mentioned purpose, the application provides the following scheme:

[0007] The multi-modal large language model optimization method based on adaptive resolution comprises:

[0008] Multi-modal question and answer data under a target scene are collected, and the multi-modal question and answer data are divided into a target task training set and a target task test set; wherein the multi-modal question and answer data comprises image data, text instructions and reference output text;

[0009] Image data under the target scene re-collected and paired text instructions are used to construct a resolution selection training set;

[0010] A multi-modal large language model with a two-stage image coding mechanism is constructed, and the multi-modal large language model is trained using the target task training set and the resolution selection training set, to obtain a target model;

[0011] The prediction accuracy of the target model is evaluated on the target task test set, and the optimization of the multi-modal large language model is completed.

[0012] Preferably, the multi-modal large language model comprises an image coding module, a resolution selection module, a visual token mapping module, a text coding module and a large language model decoding module; wherein the image coding module adopts a ViT architecture, the resolution selection module comprises an attention pooling layer, a first multi-layer perceptron and a first linear classifier, the visual token mapping module comprises a second multi-layer perceptron architecture, the text coding module comprises a tokenizer and a text embedding layer, and the large language model decoding module comprises a Transformer decoding layer and a second linear classifier.

[0013] Preferably, the training of the multi-modal large language model comprises three stages: stage one is a target task training stage, stage two is a best resolution estimation stage, and stage three is a resolution selection module training stage.

[0014] Preferably, in the stage one, the multi-modal large language model is trained using the target task training set, and the training process is as follows:

[0015] After the image data in the target task training set is coded by the image coding module, visual tokens are generated by the visual token mapping module;

[0016] The text instruction is encoded by the text encoding module to generate a text token;

[0017] The visual token and the text token are spliced and input into the large language model decoding module, and a first cross-entropy loss is calculated based on the model output and the reference output text;

[0018] According to the first cross-entropy loss, the weights of each module in the multi-modal large language model are updated to obtain an updated multi-modal large language model, and the target task training is completed.

[0019] Preferably, in the second stage, the updated multi-modal large language model in the first stage is used to process the resolution selection training set to estimate the optimal resolution, and the optimal resolution estimation process is as follows:

[0020] The aspect ratio of the image data in the resolution selection training set is calculated, and the absolute values of the differences between the aspect ratios of each resolution in the preset resolution set and the aspect ratio of the image data are sorted in ascending order to obtain a candidate resolution set;

[0021] The image data is respectively interpolated and scaled to each candidate resolution, and the scaled image data and the corresponding text instruction are input into the updated multi-modal large language model to obtain a conditional probability distribution of the output token;

[0022] The average prediction entropy is calculated using the conditional probability distribution of the output token;

[0023] The candidate resolution with the smallest average prediction entropy is taken as the pseudo-label of the image data, and the optimal resolution estimation is completed.

[0024] Preferably, in the third stage, the resolution selection module is trained using the resolution selection training set with pseudo-labels generated in the second stage, and the training process is as follows:

[0025] All image data in the resolution selection training set with pseudo-labels are uniformly scaled to a preset fixed resolution, and image feature extraction is performed using an image encoding module;

[0026] The image features are input into the resolution selection module to obtain original scores of each preset resolution;

[0027] The original scores are input into a Softmax function to obtain a prediction probability distribution, and a second cross-entropy loss is calculated based on the prediction probability distribution and the class index corresponding to the pseudo-label as a supervision target;

[0028] According to the second cross-entropy loss, the weights of the resolution selection module are updated to complete the training of the resolution selection module.

[0029] The application further provides a multimodal large language model optimization system based on adaptive resolution, which is used for implementing the method and comprises the following modules:

[0030] A dataset construction module is configured to collect multimodal question and answer data in a target scene, and divide the multimodal question and answer data into a target task training set and a target task test set; wherein the multimodal question and answer data comprises image data, text instructions and reference output text.

[0031] A resolution selection training set construction module is configured to construct a resolution selection training set by using image data in the target scene and paired text instructions.

[0032] A target model construction module is configured to construct a multimodal large language model with a two-stage image encoding mechanism, and train the multimodal large language model by using the target task training set and the resolution selection training set, to obtain a target model.

[0033] A target model evaluation module is configured to evaluate the prediction accuracy of the target model on the target task test set, to complete optimization of the multimodal large language model.

[0034] Preferably, in the target model construction module, the multimodal large language model comprises an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module and a large language model decoding module; wherein the image encoding module adopts a ViT architecture, the resolution selection module comprises an attention pooling layer, a first multilayer perceptron and a first linear classifier, the visual token mapping module comprises a second multilayer perceptron architecture, the text encoding module comprises a tokenizer and a text embedding layer, and the large language model decoding module comprises a Transformer decoding layer and a second linear classifier.

[0035] Compared with the prior art, the application has the following beneficial effects: the technical scheme of the application processes input images at the best resolution through a two-stage image encoding mechanism, reduces the average inference time delay, and improves the visual perception ability of the model. By sharing the image encoder, the resolution selection module can realize resolution prediction with only a small amount of additional parameters, and the storage overhead introduced is minimal. In addition, the application also designs a multi-stage iterative training strategy, effectively integrates labeled and unlabeled data, and reduces the data labeling cost. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical scheme of the application, the following briefly introduces the drawings needed in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0037] Figure 1 A flowchart of a multi-modal large language model optimization method based on adaptive resolution for an embodiment of the present application is shown in FIG. 1.

[0038] Figure 2 A multi-modal large language model structure diagram for an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0040] In order to make the above-mentioned objects, features and advantages of the present application more apparent, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0041] Embodiment One

[0042] As shown in FIG. 1, the multi-modal large language model optimization method based on adaptive resolution includes the following steps. Figure 1

[0043] S1: Collect multi-modal question and answer data under a target scene, and divide the multi-modal question and answer data into a target task training set and a target task test set; wherein the multi-modal question and answer data includes image data, text instructions and reference output text. In this embodiment, 100,000 pieces of multi-modal question and answer data of document images are collected, and are divided into a target task training set and a target task test set in a ratio of 8:2.

[0044] S2: Use the image data under the target scene re-collected and the paired text instructions to construct a resolution selection training set. Specifically, the text instructions corresponding to each image in the resolution selection training set are randomly sampled from a small-scale text instruction template library.

[0045] S3: Construct a multi-modal large language model with a two-stage image encoding mechanism, and use the target task training set and the resolution selection training set to train the multi-modal large language model to obtain a target model.

[0046] Further embodiments are as follows. Figure 2 ​As shown, the multi-modal large language model comprises an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module, and a large language model decoding module; wherein the image encoding module adopts a ViT architecture, the resolution selection module comprises an attention pooling layer, a first multi-layer perceptron, and a first linear classifier, the visual token mapping module comprises a second multi-layer perceptron architecture, the text encoding module comprises a tokenizer and a text embedding layer, and the large language model decoding module comprises a Transformer decoding layer and a second linear classifier. In this embodiment, the resolution selection module is randomly initialized by the Kaiming method, and the initial weights of the other modules are all loaded from the pre-trained Qwen2.5-VL model.

[0047] Further embodiments are characterized in that the training of the multi-modal large language model comprises three stages: stage one is a target task training stage, stage two is an optimal resolution estimation stage, and stage three is a resolution selection module training stage.

[0048] Further embodiments are characterized in that, in stage one, the multi-modal large language model is trained using a target task training set, and the training process is as follows:

[0049] After the image data in the target task training set is encoded by the image encoding module, image features are generated:

[0050] ,

[0051] wherein, represents dividing the input image into non-overlapping image blocks and linearly mapping them to dimensions, represents the number of image blocks, the feature dimension of each position of the output is . Subsequently, the image features are reorganized and input into the visual token mapping module to generate visual tokens :

[0052] ,

[0053] ,

[0054] wherein, represents concatenating 2x2 image block features adjacent in space in the channel dimension, so that the sequence length changes from to , and the feature dimension changes from to . Subsequently, the multi-layer perceptron aligns the dimensional features to the dimension of the large language model decoding module.. representing the image features after recombination, representing a set of real numbers.

[0055] The text instructions are encoded by the text encoding module to generate text tokens :

[0056] ,

[0057] wherein, represents the text instructions, represents the tokenizer, represents the text embedding layer, represents the number of text tokens, represents the dimension of the large language model decoding module.

[0058] The visual tokens are spliced with the text tokens to input the large language model decoding module, and a first cross-entropy loss is calculated based on the model output and the reference output text.

[0059] According to the first cross-entropy loss, the weights of each module in the multi-modal large language model are updated to obtain an updated multi-modal large language model, and the target task training is completed.

[0060] In this embodiment, the training parameters of stage one are set as follows:

[0061] Number of training rounds: 5; training batch size: 128; optimizer: AdamW; learning rate: 0.00005; learning rate update strategy: cosine decay; Weight decay: 0.0005.

[0062] Further embodiments are that, in stage two, the updated multi-modal large language model in stage one is used to process the resolution selection training set to estimate the optimal resolution, and the optimal resolution estimation process is as follows:

[0063] The aspect ratios of the image data in the resolution selection training set are calculated, and the absolute values of the differences between the aspect ratios of each resolution in the preset resolution set and the aspect ratios of the image data are sorted in ascending order, and the first resolutions are taken as the candidate resolution set . In this embodiment, the value is 8.

[0064] The preset resolution set is constructed by the following method: first, the value range of length and width is set to pixels, then in the interval, with a step of 56 pixels, and The length and width are equidistantly sampled with a step of 112 pixels in the interval, and the Cartesian product of the length and width sampling values is taken to obtain a complete preset resolution set.

[0065] The image data is respectively interpolated and scaled to each candidate resolution, and the scaled image data and the corresponding text instruction are input into the updated multi-modal large language model to obtain a conditional probability distribution of output tokens; specifically, for each candidate resolution , let be the scaled image data, be the corresponding text instruction. The input model obtained in the first stage generates the conditional probability distribution of output positions in an autoregressive manner:

[0066] ,

[0067] wherein, is a vocabulary, represents an output random variable of the th position, is a prefix token sequence obtained when the th position is decoded.

[0068] According to the conditional probability distribution of the output token, the average prediction entropy of each candidate resolution is calculated:

[0069] .

[0070] The candidate resolution with the minimum average prediction entropy is taken as the pseudo-label of the image data, and the best resolution estimation is completed. The calculation formula is as follows:

[0071] .

[0072] Further embodiments are that, in the third stage, the resolution selection training set after generating the pseudo-label in the second stage is used to train the resolution selection module, and the training process is as follows:

[0073] All image data in the resolution selection training set after generating the pseudo-label are uniformly scaled to a preset fixed resolution , and then input into the image encoding module to obtain image features; in this embodiment, is set to 224x224.

[0074] The image features are input into the resolution selection module to obtain the original score of each preset resolution:

[0075] ,

[0076] ,

[0077] ,

[0078] wherein, is an image feature, is an image feature vector of the i-th position, is a learnable vector, is an attention weight vector, is a weight of the image feature vector is a global feature vector after attention pooling, is a multi-layer perceptron, is a linear classifier, is a number of preset resolutions, denoted as . The original score is input into a Softmax function to obtain a prediction probability distribution:

[0079]

[0080] ,

[0081] wherein, and represent the original scores of the i-th and j-th classes, respectively. The class index corresponding to the pseudo label is taken as a supervision target to calculate a second cross-entropy loss:

[0082] .

[0083] According to the second cross-entropy loss, the weight of the resolution selection module is updated to complete the training of the resolution selection module.

[0084] In this embodiment, the training parameters of stage three are set as follows: the number of training rounds is 50; the training batch size is 256; the optimizer is AdamW; the learning rate is 0.00005; the learning rate update strategy is cosine decay; and the weight decay is 0.0005.

[0085] In this embodiment, the three stages are iterated multiple times.

[0086] In this embodiment, the three stages are iterated multiple times.

[0087] ​​​During the first to the second last round of iterations, after each of the three stages is completed, the optimal resolution of each image in the target task training set is predicted using the image encoding module and the resolution selection module, and the image is scaled at the resolution for the next round of iteration training. Subsequently, the weights of all modules are reinitialized.

[0088] In the first iteration of phase one, the model is trained using the original resolution image; in subsequent iterations, the model is trained using the scaled image at the optimal resolution predicted by the resolution selection module. The optimal resolution is the resolution corresponding to the highest original score The preset resolution corresponding to the category with the largest median value.

[0089] S4: Evaluate the prediction accuracy of the target model on the target task test set, and complete the optimization of the multi-modal large language model.

[0090] In the test phase, given an input image, it is first scaled to a preset low resolution, i.e., the preset fixed resolution , after the image encoding module extracts the features, it is sent to the resolution selection module to predict the optimal resolution . Then, the original image is scaled to , and the image encoding module is used again to generate visual tokens in the visual token mapping module. Finally, the visual tokens and text tokens are spliced and input into the large language model decoding module to generate the final prediction result through the autoregressive method.

[0091] In summary, the data set of the embodiment contains various document images, from small size logos, seals to double-column test papers, etc., and the image content type and layout structure differ significantly, which is not suitable for uniform fixed input resolution processing. At the same time, due to the different resolutions of the shooting devices and the different scaling ratios of the screenshots, the original resolution of the image cannot be directly used as a basis for determining the optimal input resolution, and it is often higher than the actual needs of the model. In view of the above challenges, the method of the embodiment can accurately select the optimal resolution for inference, while improving the visual perception ability and significantly speeding up the inference speed of the model.

[0092] Embodiment Two

[0093] The application also provides a multi-modal large language model optimization system based on adaptive resolution, which is used to implement the method of embodiment one, comprising:

[0094] A data set construction module is used to collect multi-modal question and answer data in a target scene, and divide the multi-modal question and answer data into a target task training set and a target task test set; wherein the multi-modal question and answer data includes image data, text instructions and reference output text.

[0095] A resolution selection training set construction module is configured to construct a resolution selection training set by using the reacquired image data under the target scene and the paired text instructions.

[0096] A target model construction module is configured to construct a multimodal large language model with a two-stage image encoding mechanism, and train the multimodal large language model by using the target task training set and the resolution selection training set to obtain a target model.

[0097] A target model evaluation module is configured to evaluate the prediction accuracy of the target model on the target task test set, and complete the optimization of the multimodal large language model.

[0098] In a further embodiment, in the target model construction module, the multimodal large language model comprises an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module, and a large language model decoding module; the image encoding module adopts a ViT architecture; the resolution selection module comprises an attention pooling layer, a first multi-layer perceptron, and a first linear classifier; the visual token mapping module comprises a second multi-layer perceptron architecture; the text encoding module comprises a tokenizer and a text embedding layer; and the large language model decoding module comprises a Transformer decoding layer and a second linear classifier.

[0099] The above-described embodiments are merely descriptions of the preferred modes of the present application, and are not intended to limit the scope of the present application. Various modifications and improvements to the technical solutions of the present application made by those of ordinary skill in the art without departing from the design spirit of the present application shall fall within the scope of protection of the present application as defined by the claims.

Claims

1. A multi-modal large language model optimization method based on adaptive resolution, characterized in that, The application relates to a method for training a multimodal large language model. The method comprises the following steps: Collecting multimodal question and answer data in a target scene, and dividing the multimodal question and answer data into a target task training set and a target task test set; wherein the multimodal question and answer data comprises image data, text instructions and reference output text; Using image data in the target scene and paired text instructions to construct a resolution selection training set; Constructing a multimodal large language model with a two-stage image encoding mechanism, and training the multimodal large language model using the target task training set and the resolution selection training set to obtain a target model; Evaluating the prediction accuracy of the target model on the target task test set to optimize the multimodal large language model; The multimodal large language model comprises an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module and a large language model decoding module; wherein the image encoding module adopts a ViT architecture, the resolution selection module comprises an attention pooling layer, a first multilayer perceptron and a first linear classifier, the visual token mapping module comprises a second multilayer perceptron architecture, the text encoding module comprises a tokenizer and a text embedding layer, and the large language model decoding module comprises a Transformer decoding layer and a second linear classifier; 2. The method of claim 1, wherein, The training of the multimodal large language model comprises three stages: stage one is a target task training stage, stage two is a best resolution estimation stage, and stage three is a resolution selection module training stage. In the stage one, the multimodal large language model is trained using the target task training set, and the training process is as follows: After the image data in the target task training set is encoded by the image encoding module, visual tokens are generated by the visual token mapping module; After the text instructions are encoded by the text encoding module, text tokens are generated; The visual tokens and the text tokens are spliced and input into the large language model decoding module, and a first cross-entropy loss is calculated based on the model output and the reference output text; 3. The method of claim 2, wherein, According to the first cross-entropy loss, the weights of each module in the multimodal large language model are updated to obtain an updated multimodal large language model, and the target task training is completed. In the stage two, the resolution selection training set is processed using the updated multimodal large language model in the stage one to estimate the best resolution, and the best resolution estimation process is as follows: The aspect ratio of the image data in the resolution selection training set is calculated, the absolute values of the differences between the aspect ratios of each resolution in a preset resolution set and the aspect ratio of the image data are sorted in ascending order, and a candidate resolution set is obtained; The image data is interpolated and scaled to each candidate resolution, and the scaled image data and the corresponding text instructions are input into the updated multimodal large language model to obtain a conditional probability distribution of output tokens; The average prediction entropy is calculated using the conditional probability distribution of the output tokens; 4. The method of claim 3, wherein, The candidate resolution with the minimum average prediction entropy is taken as the pseudo-label of the image data, and the best resolution estimation is completed. In the stage three, the resolution selection module is trained using the resolution selection training set with the pseudo-label generated in the stage two, and the training process is as follows: Scale all image data in the resolution selection training set after generating pseudo labels to a preset fixed resolution, and use an image encoding module to extract image features; Input the image features into the resolution selection module to obtain original scores for each preset resolution; Input the original scores into a Softmax function to obtain a prediction probability distribution, and use the class index corresponding to the pseudo label as a supervision target to calculate a second cross-entropy loss in combination with the prediction probability distribution; Update the weights of the resolution selection module according to the second cross-entropy loss to complete the training of the resolution selection module.

5. A multi-modal large language model optimization system based on adaptive resolution for implementing the method of any one of claims 1-4, characterized in that, Comprise: A dataset construction module is configured to collect multi-modal question and answer data in a target scene, and divide the multi-modal question and answer data into a target task training set and a target task test set; wherein the multi-modal question and answer data comprises image data, text instructions and reference output text; A resolution selection training set construction module is configured to use image data in the target scene re-collected and paired text instructions to construct a resolution selection training set; A target model construction module is configured to construct a multi-modal large language model with a two-stage image encoding mechanism, and train the multi-modal large language model using the target task training set and the resolution selection training set to obtain a target model; A target model evaluation module is configured to evaluate the prediction accuracy of the target model on the target task test set to complete the optimization of the multi-modal large language model.

6. The system of claim 5, wherein, In the target model construction module, the multi-modal large language model comprises an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module and a large language model decoding module; wherein the image encoding module adopts a ViT architecture, the resolution selection module comprises an attention pooling layer, a first multi-layer perceptron and a first linear classifier, the visual token mapping module comprises a second multi-layer perceptron architecture, the text encoding module comprises a tokenizer and a text embedding layer, and the large language model decoding module comprises a Transformer decoding layer and a second linear classifier.

Images