A method and system for multi-modal large model inference acceleration and application thereof

Abstract

The application discloses a method for multi-modal large model inference acceleration, and belongs to the technical field of artificial intelligence, aiming to solve the technical problems of long time consumption in the pre-filling stage caused by large image quantization loss and too many image coding tokens during the inference of a multi-modal large model. The method first extracts the overall features of an image through a mask auto-encoding network, then acquires the candidate regions of the image by using a detection network and completes the alignment and extraction of the regional features, and obtains the image sequence feature vector after the overall features and the local features are fused; meanwhile, the input text is segmented and embedded to obtain the text word vector, and the two types of vectors are spliced and fused according to the sequence dimension to input a large language decoding model to complete decoding. The application can greatly reduce the image token calculation amount, significantly reduce the time consumption in the inference pre-filling stage under the premise of ensuring the inference effect, and improve the overall inference efficiency of the multi-modal large model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for processing input images in optimizing the inference latency of multimodal large models, specifically a method, system, and application for accelerating multimodal large model inference. Background Technology

[0002] In existing technologies, to improve the inference efficiency of large multimodal inference models, related solutions mostly focus on model weight quantization or distributed pipeline optimization of the inference system. For example, the patent "Quantization Inference Method, Device, Equipment and Medium Based on Large Multimodal Models" (patent application number: CN202511340466.0) determines the quantization targets of input activation, scaling factor and bias term, quantizes and compresses the weights, then iteratively constructs a parameter set and establishes a loss function to optimize the quantization parameters and weights, and finally completes the inference calculation based on the quantization parameters and weights. Another example is the patent "A High-Performance Large Multimodal Model Inference System and Method" (patent application number: CN202311814694.8), which integrates core components such as a multimodal accelerated inference unit, search unit, caching unit and database, supports flexible combination of multimodal components and accelerated inference, and has caching mechanism and distributed deployment capability, thereby reducing inference latency and saving computing resources.

[0003] However, existing multimodal large-scale model inference efficiency optimization schemes have significant shortcomings, neglecting the differences between image and text optimization: on the one hand, text information is far less sensitive to quantization than image quantization, and image quantization easily leads to significant feature loss; on the other hand, images contain far more information than text, and image encoding often generates a large number of tokens, causing a computational bottleneck in the inference pre-filling stage, and consequently leading to severe inference delays. For example, a 2048*1365 resolution image in the qwen2.5vl multimodal model needs to be represented by 13408 tokens, which is usually far greater than the number of tokens in the user input text. This makes the computational bottleneck in the pre-filling stage concentrated on the encoding calculation of image tokens, seriously affecting the user experience. Summary of the Invention

[0004] To address the shortcomings of the aforementioned technologies, the present invention aims to provide a method for accelerating inference in multimodal large models. This method can reduce the number of image lexical calculations in the inference pre-filling stage and shorten the image feature encoding time, thereby reducing the overall inference time, while ensuring inference performance.

[0005] The method for accelerating inference of multimodal large models according to the present invention includes the following steps:

[0006] Step 1, Overall Image Feature Extraction: The input image data is processed through a masked autoencoder network to obtain the overall image feature map. The feature map size is W×H, where W and H represent the width and height of the feature map pixels, respectively. If the original image is divided into 16*16 blocks, then its size is 1 / 16 of the original image size. By default, a masking rate of 75% is used to reduce the computational cost of image words.

[0007] Step 2, Image Candidate Region Extraction: The input image data is processed using a detection network to obtain candidate regions for the image;

[0008] Step 3, Candidate Region Feature Extraction: Extract the corresponding feature map from the candidate regions obtained in Step 2 using region alignment operation. The size of the obtained feature map is w×h.

[0009] Step 4, Image Feature Fusion: First, scale the feature map of the candidate region to a size of W×H using scaling methods such as bilinear interpolation, area interpolation or bicubic interpolation. Then, fuse the regional features with the overall feature map by adding them point by point according to their spatial positions. Finally, flatten the fused feature map into a sequence feature vector by rows.

[0010] Step 5, Image Feature Regularization: The sequence feature vectors obtained in Step 4 are processed by root mean square regularization to stabilize the feature distribution.

[0011] Step 6, Image Feature Enhancement and Alignment: The sequence feature vectors obtained in Step 5 are used to construct a feedforward neural network for testing and comparing performance indicators through a two-layer perceptron network to perform feature enhancement and feature alignment adaptation.

[0012] Step 7, Text word segmentation: Obtain the corresponding word segments from the text input using a word segmenter;

[0013] Step 8, Text word vector generation: Obtain text word vectors by embedding the lexical units obtained in Step 7;

[0014] Step 9, Multimodal Feature Fusion: The image sequence feature vector obtained in Step 6 and the text word vector obtained in Step 8 are concatenated and fused according to their respective sequence dimensions;

[0015] Step 10, Decoding Calculation: Input the fused features obtained in Step 9 into the large language decoding model for decoding to obtain the probability of each word being generated;

[0016] Step 11, Output generation: Based on the word generation probabilities obtained in Step 10, generate the corresponding text output.

[0017] Based on the above methods, the present invention also proposes a system for accelerating inference of multimodal large models, comprising: a memory and a processor; the memory stores a computer program, and when the computer program is executed by the processor, the above methods are implemented.

[0018] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0019] The method for accelerating inference in multimodal large models disclosed in this invention has the following beneficial effects:

[0020] In existing multimodal large-scale models, quantizing the entire model weights during edge inference leads to significant loss of model accuracy, while quantizing only the language model weights increases computational resource requirements, both resulting in lengthy inference pre-filling stages. This invention addresses this issue by encoding the input image using mask autoencoder technology, significantly reducing the number of image terms processed. For example, in the qwen2.5vl multimodal model, a 2048 × 1365 resolution image that would normally require 13408 terms of data computation only requires 3577 terms of data computation with a 75% masking rate. Simultaneously, it combines object detection networks to extract candidate regions and utilizes candidate region alignment technology to obtain local features. The overall and local features are then fused, ensuring a thorough understanding of local image details while avoiding feature loss due to masking. Compared to direct image encoding, this invention significantly reduces the time spent in the model's inference pre-filling stage, improving the overall inference efficiency of multimodal large-scale models and enhancing the user experience. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the technical solution for the multimodal large model inference acceleration method of the present invention.

[0023] Figures 2-5 The performance indicators of this invention are tested and compared. Detailed Implementation

[0024] The invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the invention are all common knowledge and general knowledge in the art, and the invention does not have any particular limitations.

[0025] This invention discloses a method for accelerating inference in multimodal large-scale models, belonging to the field of artificial intelligence technology. It aims to solve the technical problems of large image quantization loss and excessive image encoding terms leading to time-consuming pre-filling stages during inference in multimodal large-scale models. The method first extracts overall image features using a masked autoencoder network, then uses a detection network to obtain candidate regions of the image and completes region feature alignment extraction. The overall and local features are fused to obtain image sequence feature vectors. Simultaneously, the input text is segmented and embedded to obtain text word vectors. The two types of vectors are concatenated and fused along the sequence dimension before being input into a large language decoding model for decoding. This invention can significantly reduce the computational load of image terms, significantly reduce the time consumption of the inference pre-filling stage while ensuring inference performance, and improve the overall inference efficiency of multimodal large-scale models.

[0026] Example

[0027] This embodiment addresses the real-time requirements of multimodal large model inference in autonomous driving scenarios. It employs the model inference acceleration method of this invention, significantly reducing model inference time. This method effectively meets the real-time inference needs of perception tasks, open reasoning, and dynamic scene understanding in autonomous driving scenarios. The implementation steps of the multimodal large model inference acceleration method are as follows:

[0028] Step 1, Overall Image Feature Extraction: The input image data is fed into a mask autoencoder network. In this embodiment, the Vision Transformer (ViT) model is used. The masking operation is performed at a default masking rate of 75%, and the output size is 14×14 (this is for an input image of 224*224. For other input images, the feature map is obtained according to the proportional size. For example, an image of size 448×448 will obtain a feature map of size 28×28). This step uses the mask autoencoder technique.

[0029] Step 2, Image Candidate Region Extraction: Input the same input image data into the detection network, preferably the YOLO series. In this embodiment, the Nano version of the YOLOv11 model is used. The detection network identifies and outputs candidate regions in the image.

[0030] Step 3, Candidate Region Feature Extraction: For the candidate regions output in Step 2, perform a region alignment operation to complete the extraction of the corresponding feature maps, resulting in a candidate region feature map of size 7 × 7;

[0031] Step 4, Image Feature Fusion: First, bilinear interpolation is used to scale the candidate region feature map to match the overall feature map. Figure 1 The standard size is 14 × 14, but other scaling methods such as area interpolation and bicubic interpolation can be selected according to actual needs. Then, the scaled regional features are added to the overall feature map point by point according to spatial position to complete feature fusion. Finally, the fused feature map is flattened row by row and converted into the shape of a sequence feature vector.

[0032] In step 4, the overall combination of feature scaling, point-by-point addition and fusion, and flattening can significantly reduce the computational load of image coding while ensuring the integrity of image features. The time cost of candidate region feature extraction is two orders of magnitude different from that of traditional visual transformer coding methods, and its overall time consumption is negligible.

[0033] The advantage of the image features obtained in steps 1 to 4 compared with the image features obtained directly by the existing image encoder is that the image encoding time is greatly reduced. Taking the visual transformer commonly used in multimodal image encoding as an example, when masking with a default mask rate of 75%, only 25% of the data of the original image needs to be calculated. The entire calculation time decreases exponentially. The time spent on the feature map representation of the selected region is two orders of magnitude different from that of encoding the image using the visual transformer. Therefore, the overall time consumption can be ignored. The features of the candidate region are guaranteed not to cause the understanding of the details of the local area of ​​the image due to the coverage of the mask region.

[0034] Step 5, Image Feature Regularization: Perform root mean square regularization on the sequence feature vector output in Step 4 to stabilize the feature distribution;

[0035] Step 6, Image Feature Enhancement and Alignment: The regularized sequence feature vector is input into a feedforward neural network composed of two layers of perceptrons to complete feature enhancement and alignment adaptation; the image sequence feature vector generation process consisting of the above steps 1 to 6 is a non-publicly known technology.

[0036] Step 7, Text word segmentation: The input text is fed into the word segmenter, which then segments the text and outputs the corresponding words.

[0037] Step 8, Text word vector generation: Perform an embedding mapping operation on the word units output in Step 7 to generate the corresponding text word vectors;

[0038] Step 9, Multimodal Feature Fusion: The image sequence feature vector obtained in Step 6 and the text word vector obtained in Step 8 are concatenated according to their respective sequence dimensions to achieve multimodal feature fusion;

[0039] Step 10, Decoding Calculation: Input the fused multimodal features into the large language decoding model, and the model will complete the decoding calculation and output the generation probability of each word;

[0040] Step 11, Output Generation: Based on the word generation probabilities output in Step 10, generate text output that meets the requirements.

[0041] The core inventive points of this invention include two aspects: first, reducing the number of image lexical calculations through masking self-encoding technology and compensating for the loss of image features caused by masking through region extraction technology; second, the complete process of obtaining image features and fusing them with text lexical features (i.e., the overall process of steps 1 to 9).

[0042] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0043] As used in this invention, the term "comprising" is an open-ended expression, meaning it includes the contents specified in this invention but does not exclude other aspects.

[0044] As used in this invention, the term "and / or" includes any one or more of the related listed items and all combinations thereof.

[0045] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.

Claims

1. A method for accelerating inference in multimodal large models, characterized in that, Includes the following steps: Step 1: Obtain the overall image feature map by passing the input image data through a masked autoencoder network. The overall feature map has a size of S×H. Step 2: Use a detection network to obtain candidate regions of the input image data; Step 3: Extract the corresponding feature map from the candidate regions obtained in Step 2 using the region alignment operation. The size of the obtained feature map is s×h. Step 4: Scale the feature map of the candidate region to a size of W×H, then fuse the region features with the overall feature map by adding them point by point according to their spatial positions. Finally, flatten the fused feature map into a sequence feature vector by rows; W and H represent the width and height of the feature map pixels, respectively. Step 5: The sequence feature vectors obtained in Step 4 are subjected to feature regularization to stabilize the feature distribution; Step 6: Perform feature enhancement and feature alignment adaptation on the sequence feature vectors obtained in Step 5 using a feedforward neural network; Step 7: Obtain the corresponding word units from the text input using a word segmenter; Step 8: Obtain text word vectors by embedding the lexical units obtained in Step 7; Step 9: Concatenate and fuse the image sequence feature vector obtained in Step 6 with the text word vector obtained in Step 8 according to their respective sequence dimensions; Step 10: Input the fused features obtained in Step 9 into the large language decoding model for decoding to obtain the probability of each word being generated; Step 11: Generate the corresponding text output based on the word generation probabilities obtained in Step 10.

2. The method for accelerating inference in multimodal large models according to claim 1, characterized in that, In step 1, the masking rate of the masking autoencoder network is 75%.

3. The method for accelerating inference in multimodal large models according to claim 1, characterized in that, In step 4, the method for scaling the candidate region feature map to size W×H is any one of bilinear interpolation, area interpolation, or bicubic interpolation.

4. The method for accelerating inference in multimodal large models according to claim 1, characterized in that, In step 5, the feature regularization method is root mean square regularization.

5. The method for accelerating inference in multimodal large models according to claim 1, characterized in that, In step 6, the feedforward neural network is a two-layer perceptron network.

6. The method for accelerating inference in multimodal large models according to claim 1, characterized in that, The overall feature map of the image obtained in step 1 has an actual image metadata size that is 25% of the original image metadata size.

7. The method for accelerating inference in multimodal large models according to claim 1, characterized in that, In step 9, the multimodal feature fusion is performed by splicing and fusing along the sequence dimension.

8. A system for accelerating inference in multimodal large models, characterized in that, Including memory and processor; The memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

10. The application of a method for accelerating multimodal large model inference as described in any one of claims 1-7, or a system for accelerating multimodal large model inference as described in claim 8, in autonomous driving.

Images