Multi-modal large language model optimization method and system based on adaptive resolution

Abstract

The invention belongs to pattern recognition and artificial intelligence, and particularly provides a multi-modal large language model optimization method and system based on adaptive resolution, and the method comprises the steps: dividing multi-modal question and answer data into a target task training set and a target task test set; constructing a resolution selection training set by using the recollected image data in the target scene and the paired text instruction; constructing a multi-modal large language model with a two-stage image coding mechanism, and training the multi-modal large language model by using the target task training set and the resolution selection training set to obtain a target model; the prediction accuracy of the target model is evaluated on the target task test set, and optimization of the multi-modal large language model is completed. According to the technical scheme, the optimal resolution is adaptively selected for reasoning through a two-stage image coding mechanism, so that the visual perception capability of the model is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition and artificial intelligence technology, specifically relating to a method and system for optimizing multimodal large language models based on adaptive resolution. Background Technology

[0002] Large language models, represented by the GPT series, have been continuously iterating and upgrading in the field of natural language processing. Their emerging capabilities in semantic modeling, logical reasoning, and complex instruction execution are driving artificial intelligence technology towards a more general direction. Building on this foundation, in order to overcome the limitations of single-modality models, researchers have expanded the model architecture to construct multimodal large language models with visual understanding and dynamic reasoning capabilities, further expanding the application scenarios and capability boundaries of large models.

[0003] Due to limitations of image-text contrastive pre-trained image encoders, early multimodal large language models could only encode images at fixed and low resolutions, resulting in poor performance on tasks requiring fine-grained visual perception. To overcome this problem, some researchers proposed sub-image cropping strategies, which have gradually become the mainstream image processing strategy for current multimodal large language models. Specifically, the input image is cropped into multiple low-resolution sub-images. Subsequently, the image encoder independently encodes each sub-image and its thumbnail, and finally, all image features are concatenated and input into the large language model. By dividing the original image into different numbers of sub-images, the model can process images of arbitrary resolutions. With the further development of multimodal large language models, some researchers have proposed methods based on rotational position encoding and large-scale training to enable image encoders to natively support arbitrary resolutions.

[0004] While the aforementioned methods address the problem of handling images with arbitrary resolution, they also raise another issue: what resolution should be used to process the input image? Existing models typically process images directly at their original resolution or match a pre-defined fixed template based on the image's aspect ratio and then scale the image to the corresponding resolution. However, fixed resolutions are difficult to adapt to diverse real-world scenarios, and the original resolution is often not the optimal choice. For example, when photographing the same document with cameras of different resolutions, or when taking screenshots of the screen at different scaling ratios, the same content will appear at significantly different resolutions. When inputting images into a multimodal large language model, if the resolution is too low, the image encoder will not extract enough visual features to fully capture visual information, leading to serious biases in the model's perception of the image. Conversely, if the resolution is too high, the number of visual features will increase several times or even tens of times, significantly reducing inference efficiency and weakening the model's perception accuracy due to redundant features. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a method and system for optimizing multimodal large language models based on adaptive resolution. Through a two-stage image encoding mechanism, the optimal resolution is adaptively selected for inference, thereby significantly improving the model's inference efficiency and visual perception capabilities.

[0006] To achieve the above objectives, the present invention provides the following solution: The optimization method for multimodal large language models based on adaptive resolution includes: Multimodal question-answering data in the target scene is collected, and the multimodal question-answering data is divided into a target task training set and a target task test set; wherein, the multimodal question-answering data includes image data, text commands and reference output text; A resolution selection training set is constructed using re-acquired image data of the target scene and paired text instructions; A multimodal large language model with a two-stage image coding mechanism is constructed, and the multimodal large language model is trained using the target task training set and the resolution selection training set to obtain the target model; The prediction accuracy of the target model is evaluated on the target task test set to complete the optimization of the multimodal large language model.

[0007] Preferably, the multimodal large language model includes an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module, and a large language model decoding module; wherein, the image encoding module adopts a ViT architecture, the resolution selection module includes an attention pooling layer, a first multilayer perceptron, and a first linear classifier, the visual token mapping module includes a second multilayer perceptron architecture, the text encoding module includes a word segmenter and a text embedding layer, and the large language model decoding module includes a Transformer decoding layer and a second linear classifier.

[0008] Preferably, the training of the multimodal large language model includes three stages: stage one is the target task training stage, stage two is the optimal resolution estimation stage, and stage three is the resolution selection module training stage.

[0009] Preferably, in stage one, a multimodal large language model is trained using the target task training set, and the training process is as follows: The image data in the target task training set is encoded by the image encoding module and then a visual token is generated by the visual token mapping module. After the text instruction is encoded by the text encoding module, a text token is generated; The visual token and the text token are concatenated and input into the large language model decoding module, and the first cross-entropy loss is calculated based on the model output and the reference output text. Based on the first cross-entropy loss, the weights of each module in the multimodal large language model are updated to obtain the updated multimodal large language model, thus completing the training for the target task.

[0010] Preferably, in stage two, the updated multimodal large language model from stage one is used to process the resolution selection training set to estimate the optimal resolution. The optimal resolution estimation process is as follows: The aspect ratio of the image data in the resolution selection training set is calculated, and the absolute value of the difference between the aspect ratio of each resolution in the preset resolution set and the aspect ratio of the image data is sorted in ascending order to obtain the candidate resolution set. The image data is interpolated and scaled to each candidate resolution, and the scaled image data and the corresponding text instructions are input into the updated multimodal large language model to obtain the conditional probability distribution of the output token. Calculate the average prediction entropy using the conditional probability distribution of the output tokens; The candidate resolution with the smallest average prediction entropy is used as the pseudo-label for the image data to complete the optimal resolution estimation.

[0011] Preferably, in stage three, the resolution selection module is trained using the resolution selection training set generated in stage two after generating pseudo-labels. The training process is as follows: After generating pseudo-labels, the resolution of all image data in the training set is uniformly scaled to a preset fixed resolution, and image feature extraction is performed using the image encoding module. The image features are input into the resolution selection module to obtain the original scores for each preset resolution; The original score is input into the Softmax function to obtain the predicted probability distribution. The second cross-entropy loss is calculated by taking the category index corresponding to the pseudo-label as the supervision target and combining it with the predicted probability distribution. The weights of the resolution selection module are updated based on the second cross-entropy loss, thus completing the training of the resolution selection module.

[0012] This invention also provides a multimodal large language model optimization system based on adaptive resolution for implementing the method, comprising: The dataset construction module is used to collect multimodal question-and-answer data in the target scene and divide the multimodal question-and-answer data into a target task training set and a target task test set; wherein, the multimodal question-and-answer data includes image data, text instructions and reference output text; The resolution selection training set construction module is used to construct a resolution selection training set using reacquired image data of the target scene and paired text instructions. The target model construction module is used to construct a multimodal large language model with a two-stage image encoding mechanism, and to train the multimodal large language model using the target task training set and the resolution selection training set to obtain the target model; The target model evaluation module is used to evaluate the prediction accuracy of the target model on the target task test set, thereby optimizing the multimodal large language model.

[0013] Preferably, in the target model construction module, the multimodal large language model includes an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module, and a large language model decoding module; wherein, the image encoding module adopts a ViT architecture, the resolution selection module includes an attention pooling layer, a first multilayer perceptron, and a first linear classifier, the visual token mapping module includes a second multilayer perceptron architecture, the text encoding module includes a word segmenter and a text embedding layer, and the large language model decoding module includes a Transformer decoding layer and a second linear classifier.

[0014] Compared with existing technologies, the advantages of this invention are as follows: The technical solution of this invention uses a two-stage image encoding mechanism to process the input image at the optimal resolution, thereby reducing the average inference latency while improving the model's visual perception capability. By sharing an image encoder, the resolution selection module only requires a small number of additional parameters to achieve resolution prediction, introducing negligible storage overhead. Furthermore, this invention also designs a multi-stage iterative training strategy, effectively integrating labeled and unlabeled data, thereby reducing data labeling costs. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the multimodal large language model optimization method based on adaptive resolution according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the multimodal large language model structure according to an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Example 1 like Figure 1 As shown, the optimization method for multimodal large language models based on adaptive resolution includes: S1: Collect multimodal question-and-answer data in the target scene and divide the multimodal question-and-answer data into a target task training set and a target task test set; wherein, the multimodal question-and-answer data includes image data, text instructions, and reference output text. In this embodiment, multimodal question-and-answer data of 100,000 document images are collected and divided into a target task training set and a target task test set in an 8:2 ratio.

[0020] S2: Construct a resolution selection training set using re-acquired image data of the target scene and paired text instructions. Specifically, the text instructions corresponding to each image in the resolution selection training set are randomly sampled from a small-scale text instruction template library.

[0021] S3: Construct a multimodal large language model with a two-stage image encoding mechanism, and train the multimodal large language model using the target task training set and the resolution selection training set to obtain the target model.

[0022] A further implementation method is, such as Figure 2 As shown, the multimodal large language model includes an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module, and a large language model decoding module. The image encoding module adopts the ViT architecture; the resolution selection module includes an attention pooling layer, a first multilayer perceptron, and a first linear classifier; the visual token mapping module uses a second multilayer perceptron architecture; the text encoding module includes a word segmenter and a text embedding layer; and the large language model decoding module includes a Transformer decoding layer and a second linear classifier. In this embodiment, the resolution selection module uses the Kaiming method for random initialization, and the initial weights of all other modules are loaded from the pre-trained Qwen2.5-VL model.

[0023] A further implementation method involves training the multimodal large language model in three stages: stage one is the target task training stage, stage two is the optimal resolution estimation stage, and stage three is the resolution selection module training stage.

[0024] A further implementation method involves, in stage one, training a multimodal large language model using the target task training set, as follows: Image features are generated after the image data in the training set of the target task is encoded by the image encoding module. , in, This indicates that the input image will be displayed. Divide the image into non-overlapping patches and linearly map them to dimension, Indicates the number of image patches. The feature dimension of each position in the output is Subsequently, image features Reorganize and input into the visual token mapping module to generate a visual token. : , , in, This means concatenating the features of spatially adjacent 2×2 image patches in the channel dimension, making the sequence length smaller than that of the previous generation. Become , feature dimension Become Subsequently, the multilayer perceptron... Will Dimensional features aligned to the dimensions of the large language model decoding module . Represents the features of the reconstructed image. It represents the set of real numbers.

[0025] Text commands are encoded by the text encoding module to generate text tokens. : , in, Indicates text instructions. Indicates a word segmenter. Represents a text embedding layer. Indicates the number of text tokens. This indicates the dimension of the large language model decoding module.

[0026] Visual tokens With text tokens The concatenated text is input into the large language model decoding module, and the first cross-entropy loss is calculated based on the model output and the reference output text.

[0027] Based on the first cross-entropy loss, the weights of each module in the multimodal large language model are updated to obtain the updated multimodal large language model, thus completing the training for the target task.

[0028] In this embodiment, the training parameters for Phase 1 are set as follows: Training rounds: 5; Training batch size: 128; Optimizer: AdamW; Learning rate: 0.00005; Learning rate update strategy: cosine decay; Weight decay: 0.0005.

[0029] A further implementation involves, in stage two, using the updated multimodal large language model from stage one to process the resolution selection training set to estimate the optimal resolution. The optimal resolution estimation process is as follows: Calculate the aspect ratio of the image data in the training set to select the resolution, and apply the preset resolution set. Sort the absolute values ​​of the differences between the aspect ratio of each resolution and the aspect ratio of the image data in ascending order, and take the top values. A set of preset resolutions as candidate resolutions In this embodiment, The value is 8.

[0030] The preset resolution set is constructed as follows: First, the range of length and width values ​​is set to... Pixels, then Within the interval, with a step size of 56 pixels, Within the interval, the length and width are sampled at equal intervals with a step size of 112 pixels. Finally, the Cartesian product of the length and width sample values ​​is taken to obtain the complete set of preset resolutions.

[0031] The image data is interpolated and scaled to each candidate resolution, and the scaled image data and corresponding text instructions are input into the updated multimodal large language model to obtain the conditional probability distribution of the output tokens; specifically, for each candidate resolution... ,remember For scaled image data, This corresponds to the text instruction. The model obtained from input phase one is generated through an autoregressive approach. Conditional probability distribution of the output positions: , in, For vocabulary list, Representing the The output random variable at each position In decoding the first The prefix token sequence obtained at position n.

[0032] Calculate the candidate resolution based on the conditional probability distribution of the output tokens. Corresponding average prediction entropy : .

[0033] The candidate resolution with the smallest average prediction entropy As pseudo-labels for image data, optimal resolution estimation is achieved. The calculation formula is as follows: .

[0034] A further implementation method involves, in stage three, using the resolution selection training set generated in stage two after pseudo-labeling, to train the resolution selection module. The training process is as follows: After generating pseudo-labels, scale all image data in the training set to a preset fixed resolution. The image features are then input into the image encoding module to obtain the image features; in this embodiment, Set to 224×224.

[0035] The image features are input into the resolution selection module to obtain the raw scores for each preset resolution. : , , , in, For image features, For the first Image feature vectors at each location, For learnable vectors, For attention weight vectors, Image feature vector The weight, This is the global feature vector after attention pooling. It is a multilayer perceptron. For linear classifiers, The number of preset resolutions, denoted as .

[0036] Original score Input the Softmax function to obtain the predicted probability distribution: , in, and They represent the first and the The original scores for each category.

[0037] Category index corresponding to pseudo-labels To supervise the target, calculate the second cross-entropy loss: .

[0038] The weights of the resolution selection module are updated based on the second cross-entropy loss, thus completing the training of the resolution selection module.

[0039] In this embodiment, the training parameters for stage three are set as follows: number of training rounds: 50; training batch size: 256; optimizer: AdamW; learning rate: 0.00005; learning rate update strategy: cosine decay; weight decay: 0.0005.

[0040] In this embodiment, the three stages will be iterated multiple times.

[0041] During the first to penultimate iterations, after each of the three phases, the optimal resolution for each image in the target task training set is predicted using the image encoding module and the resolution selection module, and the image is scaled to that resolution for the next iteration. Subsequently, the weights of all modules are reinitialized.

[0042] In the first iteration of Phase 1, the model is trained using the original resolution image; in subsequent iterations, it is trained using images scaled to the optimal resolution predicted by the resolution selection module. The optimal resolution is the original score. The preset resolution corresponding to the category with the highest median value.

[0043] S4: Evaluate the prediction accuracy of the target model on the target task test set and complete the optimization of the multimodal large language model.

[0044] During the testing phase, given an input image, it is first scaled down to a preset low resolution, i.e., the preset fixed resolution mentioned above. After features are extracted by the image encoding module, the image is sent to the resolution selection module to predict the optimal resolution. Then, scale the original image to... The text token is then processed again by the image encoding module and then by the visual token mapping module to generate a visual token. Finally, the visual token and the text token are concatenated and input into the large language model decoding module, where an autoregressive method is used to generate the final prediction result.

[0045] In summary, the dataset in this embodiment contains a variety of document images, ranging from small logos and stamps to two-column exam papers. The content types and layout structures of the images vary significantly, making it unsuitable to process them using a uniform, fixed input resolution. Furthermore, due to the varying resolutions of the capturing devices and the different scaling ratios of the screenshots, the original resolution of the images is difficult to directly use as the basis for determining the optimal input resolution, and is often higher than the actual requirements of the model. To address these challenges, the method in this embodiment can accurately select the optimal resolution for inference, significantly accelerating the model's inference speed while improving visual perception capabilities.

[0046] Example 2 This invention also provides a multimodal large language model optimization system based on adaptive resolution, for implementing the method of Embodiment 1, including: The dataset construction module is used to collect multimodal question-and-answer data in the target scene and divide the multimodal question-and-answer data into a target task training set and a target task test set; among them, the multimodal question-and-answer data includes image data, text instructions and reference output text.

[0047] The resolution selection training set construction module is used to construct a resolution selection training set using reacquired image data of the target scene and paired text instructions.

[0048] The target model building module is used to construct a multimodal large language model with a two-stage image encoding mechanism, and to train the multimodal large language model using the target task training set and the resolution selection training set to obtain the target model.

[0049] The target model evaluation module is used to evaluate the prediction accuracy of the target model on the target task test set and to optimize the multimodal large language model.

[0050] A further implementation involves a target model construction module where the multimodal large language model includes an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module, and a large language model decoding module. The image encoding module employs a ViT architecture; the resolution selection module includes an attention pooling layer, a first multilayer perceptron, and a first linear classifier; the visual token mapping module includes a second multilayer perceptron architecture; the text encoding module includes a word segmenter and a text embedding layer; and the large language model decoding module includes a Transformer decoding layer and a second linear classifier.

[0051] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for optimizing a multimodal large language model based on adaptive resolution, characterized in that, include: Multimodal question-answering data in the target scene is collected, and the multimodal question-answering data is divided into a target task training set and a target task test set; wherein, the multimodal question-answering data includes image data, text commands and reference output text; A resolution selection training set is constructed using re-acquired image data of the target scene and paired text instructions; A multimodal large language model with a two-stage image coding mechanism is constructed, and the multimodal large language model is trained using the target task training set and the resolution selection training set to obtain the target model; The prediction accuracy of the target model is evaluated on the target task test set to complete the optimization of the multimodal large language model.

2. The method according to claim 1, characterized in that, The multimodal large language model includes an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module, and a large language model decoding module. The image encoding module employs a ViT architecture; the resolution selection module includes an attention pooling layer, a first multilayer perceptron, and a first linear classifier; the visual token mapping module includes a second multilayer perceptron architecture; the text encoding module includes a word segmenter and a text embedding layer; and the large language model decoding module includes a Transformer decoding layer and a second linear classifier.

3. The method according to claim 2, characterized in that, The training of the multimodal large language model includes three stages: stage one is the target task training stage, stage two is the optimal resolution estimation stage, and stage three is the resolution selection module training stage.

4. The method according to claim 3, characterized in that, In Phase One, a multimodal large language model is trained using the target task training set. The training process is as follows: The image data in the target task training set is encoded by the image encoding module and then a visual token is generated by the visual token mapping module. After the text instruction is encoded by the text encoding module, a text token is generated; The visual token and the text token are concatenated and input into the large language model decoding module, and the first cross-entropy loss is calculated based on the model output and the reference output text. Based on the first cross-entropy loss, the weights of each module in the multimodal large language model are updated to obtain the updated multimodal large language model, thus completing the training for the target task.

5. The method according to claim 4, characterized in that, In Phase Two, the updated multimodal large language model from Phase One is used to process the resolution selection training set to estimate the optimal resolution. The optimal resolution estimation process is as follows: The aspect ratio of the image data in the resolution selection training set is calculated, and the absolute value of the difference between the aspect ratio of each resolution in the preset resolution set and the aspect ratio of the image data is sorted in ascending order to obtain a candidate resolution set. The image data is interpolated and scaled to each candidate resolution, and the scaled image data and the corresponding text instructions are input into the updated multimodal large language model to obtain the conditional probability distribution of the output token. Calculate the average prediction entropy using the conditional probability distribution of the output tokens; The candidate resolution with the smallest average prediction entropy is used as the pseudo-label for the image data to complete the optimal resolution estimation.

6. The method according to claim 5, characterized in that, In stage three, the resolution selection module is trained using the resolution selection training set generated in stage two after generating pseudo-labels. The training process is as follows: After generating pseudo-labels, the resolution of all image data in the training set is uniformly scaled to a preset fixed resolution, and image feature extraction is performed using the image encoding module. The image features are input into the resolution selection module to obtain the original scores for each preset resolution; The original score is input into the Softmax function to obtain the predicted probability distribution. The second cross-entropy loss is calculated by taking the category index corresponding to the pseudo-label as the supervision target and combining it with the predicted probability distribution. The weights of the resolution selection module are updated based on the second cross-entropy loss, thus completing the training of the resolution selection module.

7. A multimodal large language model optimization system based on adaptive resolution, used to implement the method described in any one of claims 1-6, characterized in that, include: The dataset construction module is used to collect multimodal question-and-answer data in the target scene and divide the multimodal question-and-answer data into a target task training set and a target task test set; wherein, the multimodal question-and-answer data includes image data, text instructions and reference output text; The resolution selection training set construction module is used to construct a resolution selection training set using reacquired image data of the target scene and paired text instructions. The target model construction module is used to construct a multimodal large language model with a two-stage image encoding mechanism, and to train the multimodal large language model using the target task training set and the resolution selection training set to obtain the target model; The target model evaluation module is used to evaluate the prediction accuracy of the target model on the target task test set, thereby completing the optimization of the multimodal large language model.

8. The system according to claim 7, characterized in that, The target model construction module includes a multimodal large language model comprising an image encoding module, a resolution selection module, a visual token mapping module, a text encoding module, and a large language model decoding module. The image encoding module employs a ViT architecture; the resolution selection module includes an attention pooling layer, a first multilayer perceptron, and a first linear classifier; the visual token mapping module includes a second multilayer perceptron architecture; the text encoding module includes a word segmenter and a text embedding layer; and the large language model decoding module includes a Transformer decoding layer and a second linear classifier.

Images