Multimodal large model inference acceleration method integrating image feature compression and adaptive pruning

Abstract

The present invention relates to image processing technology, specifically a multimodal large model reasoning acceleration method that integrates image feature compression and adaptive pruning, comprising the following steps: determining the optimal number of image blocks for the input image based on the aspect ratio of the input image and the resolution of the reference image, so as to dynamically adjust the image resolution; obtaining image feature tokens from the input image after adjusting the image resolution through a visual encoder and an attention module, analyzing the redundancy and semantic information contribution between the image feature tokens, retaining image feature tokens that contribute greatly to the image information volume, and achieving image feature compression; encoding the input text to obtain text feature tokens; and identifying and removing single-modal redundant features and cross-modal redundant features of the visual modality and the text modality based on the importance scores of the image feature tokens and the text feature tokens, thereby pruning redundant model structures. The present invention improves the model reasoning speed as much as possible while retaining effective information.

Description

Technical Field

[0001] The present invention relates to image processing technology, and specifically to a multimodal large model inference acceleration method integrating image feature compression and adaptive pruning. Background Art

[0002] Vision-Language Multimodal Models (VLMs) have become a hot area of ​​artificial intelligence research in recent years. By combining two information sources, images and text, they demonstrate superior capabilities in zero-shot understanding and reasoning compared to traditional vision models. However, VLMs typically need to process large amounts of image data and long text sequences, leading to computational efficiency and resource consumption challenges in practical applications. Therefore, optimizing model computational efficiency and reducing resource consumption while maintaining model performance are key issues in implementing VLMs.

[0003] Most of the existing model acceleration methods focus on optimizing large language models, mainly including model pruning and speculative decoding. The invention patent CN119047579A published on November 29, 2024, discloses a large language model inference optimization method based on cascade and speculative decoding strategies. First, a small language model is used to quickly generate a draft text, and then the text is input into the large language model for verification through speculative decoding. Since multimodality requires understanding the image first, this method cannot be directly migrated. The invention patent CN118780372A published on October 15, 2024, discloses a method for lossless acceleration of large language model inference by model pruning. The large language model is pruned by selecting skip layers through a genetic algorithm. When selecting layers, this method only optimizes according to the average inference time of each text feature, without considering the importance of text features, and there is a problem of effective information loss. Summary of the Invention

[0004] In order to solve the problems existing in the prior art, the present invention proposes a multimodal large model inference acceleration method that integrates image feature compression and adaptive pruning, which can improve the model inference speed as much as possible while retaining effective information.

[0005] The multimodal large model inference acceleration method integrating image feature compression and adaptive pruning in an embodiment of the present invention includes the following steps:

[0006] S1. Determine the optimal number of image blocks for the input image based on the aspect ratio of the input image and the reference image resolution to dynamically match and adjust the image resolution;

[0007] S2. For the input image after adjusting the image resolution, obtain image feature tokens through the visual encoder and attention module, analyze the redundancy and semantic information contribution between the image feature tokens, retain the image feature tokens that contribute most to the image information volume, and realize image feature compression;

[0008] S3. Encode the input text to obtain text feature tokens; based on the importance scores of image feature tokens and text feature tokens, identify and remove unimodal redundant features and cross-modal redundant features of the visual modality and text modality, and prune redundant model structures.

[0009] The multimodal large-model inference acceleration system integrating image feature compression and adaptive pruning in the embodiment of the present invention is implemented based on the above-mentioned inference acceleration method. The inference acceleration system includes the following modules:

[0010] An image resolution dynamic matching and adjustment module is used to determine the optimal number of image blocks of the input image according to the aspect ratio of the input image and the reference image resolution, so as to dynamically match and adjust the image resolution;

[0011] The image feature compression module processes the input image after adjusting the image resolution through the visual encoder and attention module to obtain image feature tokens, analyzes the redundancy and semantic information contribution between image feature tokens, retains image feature tokens that contribute most to the image information volume, and realizes image feature compression;

[0012] The model adaptive pruning module encodes the input text to obtain text feature tokens; based on the importance scores of image feature tokens and text feature tokens, it identifies and removes unimodal redundant features and cross-modal redundant features of the visual modality and text modality, and prunes redundant model structures.

[0013] Compared with the prior art, the technical effects achieved by the present invention include:

[0014] 1. The present invention removes redundant features and structures from the image and text feature levels, combined with the model structure level. Compared with the existing technology, it can maximize the model's inference speed while retaining effective information. At the same time, there is no need to train the model during the image resolution dynamic matching and feature compression stages.

[0015] 2. The present invention adopts the technical means of dynamic matching and adjustment of image resolution. By constraining the number of effective and redundant image pixels, it dynamically matches the optimal number of image block groups to prevent image magnification distortion and effectively reduce the amount of calculation at the input image level.

[0016] 3. The present invention performs image feature compression based on mutual redundancy and semantic information. By calculating the mutual redundancy and semantic information of image features, the features that contribute more to reasoning are retained, invalid image feature compression is achieved, and the reasoning speed is improved.

[0017] 4. The present invention uses a model adaptive pruning technology based on importance scores to automatically identify and remove redundant features and model structures of single modality and cross-modality through learnable parameters, thereby reducing inference time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an overall flow chart of the multimodal large model reasoning acceleration method according to an embodiment of the present invention;

[0019] Figure 2 Flowchart for dynamic matching and adjustment of image resolution;

[0020] Figure 3 It is a flowchart of image feature compression;

[0021] Figure 4 This is a flowchart of the model adaptive pruning. DETAILED DESCRIPTION

[0022] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0023] Example

[0024] This embodiment proposes a multimodal large model inference acceleration method that integrates image feature compression and adaptive pruning. The acceleration process is divided into three stages according to data characteristics, namely, the image resolution dynamic matching adjustment stage, the image feature compression stage, and the model adaptive pruning stage. It effectively alleviates the problems of high computational efficiency and resource consumption in multimodal large model inference, significantly improves information utilization and inference speed, and is suitable for multimodal large model deployment inference scenarios with limited computing power.

[0025] During the dynamic matching and adjustment stage of image resolution, the optimal image segmentation form is determined according to the image aspect ratio and the reference image size to avoid image distortion caused by inappropriate segmentation and reduce the amount of calculation at the input end.

[0026] In the image feature compression stage, by analyzing the redundancy and semantic information contribution between image feature tokens, the image feature tokens that contribute more to the image information are retained to prevent excessive compression from having a negative impact on the completion of downstream tasks.

[0027] During the model adaptive pruning stage, automatic recognition of learnable parameters is introduced, and cross-modal redundant image feature tokens and attention heads are removed to further accelerate model reasoning.

[0028] See also Figure 1-Figure 4 The multimodal large model inference acceleration method integrating image feature compression and adaptive pruning in this embodiment specifically includes the following steps:

[0029] S1. Dynamic matching and adjustment of image resolution: For the input image (i.e., the image to be inferred), the optimal number of image blocks for the input image is determined based on the input image aspect ratio and the reference image resolution to dynamically match and adjust the image resolution.

[0030] The main purpose of this step is to perform data compression at the image input level. The specific steps are as follows:

[0031] S11, obtain input image I m , the original resolution of the input image is recorded as ,in Indicates the original width of the input image, Indicates the original height of the input image.

[0032] S12, set the baseline image resolution to ,Determine the optimal number of image blocks for the input image according to the input image aspect ratio and the reference image resolution.

[0033] The specific steps include:

[0034] S121, set the number of image blocks , is the number of blocks along the width of the input image, is the number of blocks along the height direction of the input image, The number of groups to divide the image into.

[0035] S122, Circulate Image Block Quantity Group The number of image blocks in each group Calculate the distance between the aspect ratio of the block image and the original aspect ratio of the input image, and sort the distances corresponding to all image block quantities in ascending order from small to large, and select the image block quantities corresponding to the first three distances after ascending order as the candidate block quantity group. .

[0036] The distance is calculated using the following proportional distance formula:

[0037] ;

[0038] S123, in the candidate block number group The number of image blocks is selected in a loop, and the optimal number of image blocks is determined by calculating the number of valid pixels and the number of redundant pixels.

[0039] The specific calculation process includes the following steps:

[0040] S1231. Calculate the minimum scaling scale according to the original resolution of the input image, the reference image resolution, and the number of cyclically selected image blocks:

[0041] ;

[0042] in, Indicates the width of the base image resolution, Indicates the height of the base image resolution.

[0043] S1232. Calculate the number of effective pixels based on the minimum zoom scale and the number of redundant pixels :

[0044] ;

[0045] ;

[0046] S1233, set the effective pixel number threshold to , the threshold value of the number of redundant pixels is , and set the initial value of the effective pixel number threshold to 0 and the initial value of the redundant pixel number threshold to (infinity); if the number of redundant pixels is less than the redundant pixel number threshold, and the absolute value of the difference between the number of valid pixels and the valid pixel number threshold is greater than the product of the valid pixel number threshold and the scale factor, that is, and , then let 、 , the threshold value of the number of valid pixels , redundant pixel number threshold Perform dynamic adjustments and updates, and record the number of image blocks at this time , and loop to select the next number of image blocks; otherwise, directly loop to select the next number of image blocks.

[0047] in, and It will be dynamically adjusted during the calculation process, so that the new number of valid pixels and the number of redundant pixels are used as new thresholds to limit the values ​​of the two indicators of the number of valid pixels and the number of redundant pixels to an appropriate range; To adjust the proportional factor of the effective pixel number threshold, the value in the actual calculation process is .

[0048] Repeat steps S1231, S1232 and S1233 to obtain the number of image blocks. is the optimal number of image blocks.

[0049] When dynamically matching and adjusting the image resolution through this step, the optimal number of image blocks can be found based on the aspect ratio of the input image itself and the reference image size, preventing inappropriate blocking from causing image distortion and affecting the utilization of image information. At the same time, it also reduces the amount of computation at the input level to a certain extent.

[0050] S2. Image feature compression: For the input image after adjusting the image resolution, the image feature tokens are obtained through the visual encoder and attention module. The redundancy and semantic information contribution between the image feature tokens are analyzed, and the image feature tokens that contribute most to the image information are retained to achieve image feature compression.

[0051] After obtaining the optimal number of image blocks in stage 1 (i.e., step S1) to reduce the amount of computation at the input level, the image features can be further compressed. Specifically, after the input image passes through the visual encoder and attention module, multiple image feature tokens are obtained, which are recorded as , to Representing n feature components of the image feature token respectively, and then analyzing and calculating the image feature information degree through steps S21-S23, and then compressing the redundant image features according to the calculated image feature information degree.

[0052] S21. Calculate the mutual redundancy R between image feature tokens. Mutual redundancy indicates the degree of redundancy between different feature components and can be used as a basis for subsequent merging of image feature tokens.

[0053] ;

[0054] in, Indicates that cosine similarity is used to calculate the similarity of different feature components. 、 Represents different feature components, and max represents the operation of taking the maximum value among the similarities of different feature components.

[0055] S22. Calculate the semantic information degree I contained in the image feature token. The semantic information degree indicates the importance of a corresponding image feature token for image understanding and can be used as a basis for retaining subsequent image feature tokens.

[0056] In this embodiment, the calculation process of semantic information degree specifically includes the following steps:

[0057] S221. Calculate A, where A is the attention map of the image feature token corresponding to the category feature token:

[0058] ;

[0059] in, Indicates the category feature token corresponding to the image feature token, Represents the key vector of the category feature token, the superscript T indicates the transpose of the key vector, and D indicates the dimension of the key vector; softmax represents an activation function based on the normalized exponential function, which compresses any real vector element to between (0,1), and the sum of all elements is 1.

[0060] In this embodiment, the category feature token and key vector Image feature tokens The product of the trainable weight matrix is ​​obtained as follows:

[0061] ;

[0062] ;

[0063] in, is the trainable weight matrix of the category feature token, is the trainable weight matrix of the key vector.

[0064] S222. Calculate the value vector V of the image feature token:

[0065] ;

[0066] in, Represents an affine transformation matrix.

[0067] S223. Calculate the semantic information I contained in the image feature token based on the attention map and the value vector:

[0068] ;

[0069] S23, image feature tokens after each layer of attention module ,According to the mutual redundancy and semantic information, the overall information is calculated, and the image feature tokens are sorted, fused and compressed according to the overall information.

[0070] S231, calculate the overall information degree of the image feature token based on the mutual redundancy and semantic information degree calculated in steps S21 and S22 :

[0071] ;

[0072] S232, sort the overall information degrees corresponding to all image feature tokens in descending order from large to small, and merge the image feature tokens with low overall information degrees according to a preset ratio to obtain the final compressed image feature tokens :

[0073] ;

[0074] in, Represents the image feature token Calculate the overall information degree; Represents the sorting operation based on overall information degree; In order to merge multiple image feature tokens according to the sorted overall information degree and the preset ratio, the specific merging method is weighted averaging; Indicates the compression ratio based on the overall information degree after sorting. The actual calculation process takes a value of 0.3.

[0075] In this embodiment, the weighted average is to fuse the image feature tokens ranked in the top 0.7 after sorting, that is: ;

[0076] In the formula represents the weighted weight, Indicates the number of image feature tokens ranked in the top 0.7. Indicates the image feature tokens.

[0077] When image feature compression is performed using this method, the image feature tokens that contribute more to the image information can be retained based on the mutual redundancy and semantic information between the image feature tokens, preventing excessive compression from affecting the final completion effect of downstream tasks.

[0078] S3, model adaptive pruning: Encode the input text to obtain text feature tokens (i.e., text feature tokens); based on the importance scores of image feature tokens and text feature tokens, identify and remove unimodal redundant features and cross-modal redundant features of the visual modality and text modality, and prune redundant model structures.

[0079] Both phases 1 and 2 (i.e., steps S1 and S2) directly compress the input image or encoded features during the inference phase, and only compress the image feature modal data without considering the text feature modality. This may lead to the problem of poor generalization ability in scene migration due to the mismatch of modal redundant data. Therefore, the model adaptive pruning in phase 3 (i.e., this step) is introduced to further improve the model's effective recognition and removal of multimodal redundant data or structures by adding learnable parameters, thereby effectively reducing inference time. The specific steps are as follows:

[0080] S31. Calculate the importance scores of image feature tokens and text feature tokens for the current layer and the cross-modal layer, and remove redundant image feature tokens and text feature tokens during the inference process according to the calculated importance scores.

[0081] S311. Calculate the importance scores of image feature tokens and text feature tokens for the current layer.

[0082] Image feature tokens and text feature tokens are respectively passed through their respective encoders to obtain corresponding sequences ,in , Indicates the visual modality, Indicates text mode, i.e. Represents the sequence of image feature tokens obtained by the encoder, Represents the sequence of text feature tokens obtained by the encoder. Two MLP (multi-layer perceptron) structures are introduced to respectively and Extract the importance score for the current layer :

[0083] ;

[0084] Among them, s represents single mode, Representation sequence After linear projection, linear operations can be used For calculation, each MLP structure contains two linear layers and one GeLU (Gaussian Error Linear Unit) activation function, that is, the GELU activation function represents the activation function based on the Gaussian Error Linear Unit.

[0085] S312: Calculate the importance scores of the image feature tokens and the text feature tokens for the cross-modal layer.

[0086] S3121, fuse and project the category feature tokens corresponding to the image feature token and the text feature token to obtain the fused feature :

[0087] ;

[0088] in, Represents a sequence of image feature tokens Deletion operation of corresponding category feature token; Represents a sequence of text feature tokens The corresponding category feature token deletion operation; M represents the operation of fusing the category feature tokens of the two modalities (i.e., the category feature token of the image feature token and the category feature token of the text feature token), and the fusion method is weighted averaging.

[0089] ;

[0090] In the formula 、 are weights respectively.

[0091] S3122. Calculate the importance score of the cross-modal layer :

[0092] ;

[0093] Among them, c represents cross-modality, norm represents normalization operation, Represents the linear projection obtained by step S311 The transpose of a vector.

[0094] S313. Calculate the overall importance score :

[0095] ;

[0096] S314, according to the overall importance score, through the function Sampling is performed, where a sampling value of 1 indicates that the corresponding image feature token and text feature token are retained, and a sampling value of 0 indicates that the corresponding image feature token and text feature token are removed.

[0097] S32. Calculate the attention head importance scores of the category feature tokens corresponding to the image feature tokens and the text feature tokens, and remove redundant attention heads during the inference process based on the calculated attention head importance scores.

[0098] S321, introduce the MLP structure to calculate the importance score of the single modal attention head and the importance score of the cross-modal attention head :

[0099] ;

[0100] ;

[0101] Among them, s represents unimodality and c represents cross-modality; Indicates the sequence obtained The deletion operation of the corresponding category feature token, , Indicates the visual modality, Represents a text modal.

[0102] S322, similar to step S314, according to the importance score of the single modal attention head and the importance score of the cross-modal attention head, through the function Sampling is performed, where a sampling value of 1 indicates retaining the unimodal attention head and the cross-modal attention head, and a sampling value of 0 indicates removing the unimodal attention head and the cross-modal attention head.

[0103] By adding learnable parameters to the model structure through the above steps, it is possible to automatically identify redundant feature tokens and redundant attention heads of single modality and cross-modality, further accelerating model reasoning.

[0104] Based on the same inventive concept, this embodiment also provides a multimodal large model inference acceleration system that integrates image feature compression and adaptive pruning, which is implemented based on the above-mentioned multimodal large model inference acceleration method. The inference acceleration system of this embodiment includes the following modules:

[0105] An image resolution dynamic matching and adjustment module is used to determine the optimal number of image blocks of the input image according to the aspect ratio of the input image and the reference image resolution, so as to dynamically match and adjust the image resolution;

[0106] The image feature compression module processes the input image after adjusting the image resolution through the visual encoder and attention module to obtain image feature tokens, analyzes the redundancy and semantic information contribution between image feature tokens, retains image feature tokens that contribute most to the image information volume, and realizes image feature compression;

[0107] The model adaptive pruning module encodes the input text to obtain text feature tokens; based on the importance scores of image feature tokens and text feature tokens, it identifies and removes unimodal redundant features and cross-modal redundant features of the visual modality and text modality, and prunes redundant model structures.

[0108] The above modules are used to implement steps S1-S3 respectively. For detailed implementation process, please refer to the above specific implementation steps.

[0109] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A multimodal large model inference acceleration method integrating image feature compression and adaptive pruning, characterized in that: The following steps are involved: S1. Determine the optimal number of image blocks for the input image based on the aspect ratio of the input image and the reference image resolution to dynamically match and adjust the image resolution; S2. For the input image after adjusting the image resolution, obtain image feature tokens through the visual encoder and attention module, analyze the redundancy and semantic information contribution between the image feature tokens, retain the image feature tokens that contribute most to the image information volume, and realize image feature compression; S3. Encode the input text to obtain text feature tokens; based on the importance scores of image feature tokens and text feature tokens, identify and remove unimodal redundant features and cross-modal redundant features of the visual modality and text modality, and prune redundant model structures.

2. The multimodal large model reasoning acceleration method according to claim 1, characterized in that: Step S1 includes: S11. Obtain the input image and record the original resolution of the input image as ,in Indicates the original width of the input image, Indicates the original height of the input image; S12. Setting a reference image resolution, and determining an optimal number of image blocks for the input image according to the input image aspect ratio and the reference image resolution.

3. The multimodal large model inference acceleration method according to claim 2, characterized in that: Step S12 includes: S121, set the number of image blocks , is the number of blocks along the width of the input image, is the number of blocks along the height direction of the input image, The number of groups for image segmentation; S122, Circulate Image Block Quantity Group The number of image blocks in each group Calculate the distance between the aspect ratio of the block image and the original aspect ratio of the input image, and sort the distances corresponding to the number of image blocks in ascending order, and select the image block numbers corresponding to the first several distances after ascending order as the candidate block number group ; S123, in the candidate block number group The number of image blocks is selected in a loop, and the optimal number of image blocks is determined by calculating the number of valid pixels and the number of redundant pixels.

4. The multimodal large model inference acceleration method according to claim 3, characterized in that: The distance is calculated using the proportional distance formula: .

5. The multimodal large model reasoning acceleration method according to claim 3, characterized in that: Step S123 includes: S1231. Calculate the minimum scaling scale according to the original resolution of the input image, the reference image resolution, and the number of cyclically selected image blocks: ； in, Indicates the width of the base image resolution, Indicates the height of the base image resolution; S1232. Calculate the number of effective pixels based on the minimum zoom scale and the number of redundant pixels : ； ； S1233, set the effective pixel number threshold to , the threshold value of the number of redundant pixels is , and set the initial value of the effective pixel number threshold to 0 and the initial value of the redundant pixel number threshold to ;like and , then let 、 , dynamically adjust and update the effective pixel number threshold and the redundant pixel number threshold, record the image block number at this time, and select the next image block number in a loop; otherwise, directly select the next image block number in a loop; A scaling factor for adjusting the threshold value of the number of valid pixels.

6. The multimodal large model reasoning acceleration method according to claim 1, characterized in that: The image feature token obtained in step S2 is recorded as , to Represent the n feature components of the image feature token respectively; Step S2 specifically includes: S21. Calculate the mutual redundancy R between image feature tokens, which is used to indicate the degree of redundancy between different feature components: ； in, Indicates that cosine similarity is used to calculate the similarity of different feature components. 、 Represents different feature components, and max represents the operation of taking the maximum value among the similarities of different feature components; S22, calculating the semantic information degree I contained in the image feature token, which is used to indicate the importance of a corresponding image feature token for image understanding; S23, image feature tokens after each layer of attention module ,According to the mutual redundancy and semantic information, the overall information is calculated, and the image feature tokens are sorted, fused and compressed according to the overall information.

7. The multimodal large model reasoning acceleration method according to claim 6, characterized in that: Step S22 includes: S221. Calculate A, where A is the attention map of the image feature token corresponding to the category feature token: ； in, Indicates the category feature token corresponding to the image feature token, Represents the key vector of the category feature token, the superscript T represents the transpose of the key vector, and D represents the dimension of the key vector; softmax represents an activation function based on the normalized exponential function, which compresses any real vector element to between (0,1), and the sum of all elements is 1; Category Feature Token and key vector Image feature tokens The product of the trainable weight matrix is: ； ； in, is the trainable weight matrix of the category feature token, is the trainable weight matrix of the key vector; S222. Calculate the value vector V of the image feature token: ； in, represents the affine transformation matrix; S223. Calculate the semantic information I contained in the image feature token based on the attention map and the value vector: 。 8. The multimodal large model reasoning acceleration method according to claim 6, characterized in that: Step S23 includes: S231, calculate the overall information degree of the image feature token based on mutual redundancy and semantic information degree : ； S232, sort the overall information degrees corresponding to all image feature tokens in descending order, and merge the image feature tokens with low overall information degrees according to a preset ratio to obtain the final compressed image feature tokens : ； in, Represents the image feature token Calculate the overall information degree; Represents the sorting operation based on overall information degree; To perform a weighted average operation on multiple image feature tokens according to the sorted overall information degree and the preset ratio; Indicates the compression ratio based on the overall information degree after sorting.

9. The multimodal large model reasoning acceleration method according to claim 1, characterized in that: Step S3 includes: S31, calculating the importance scores of the image feature tokens and the text feature tokens for the current layer and the cross-modal layer, and removing redundant image feature tokens and text feature tokens during the inference process according to the calculated importance scores; S32, calculating the attention head importance scores of the category feature tokens corresponding to the image feature tokens and the text feature tokens, and removing redundant attention heads during the inference process according to the calculated attention head importance scores; Step S31 includes: S311, calculating the importance scores of image feature tokens and text feature tokens for the current layer; Suppose the image feature token and text feature token are respectively passed through the encoder to obtain the sequence ,in , Indicates the visual modality, Indicates text mode, Represents the sequence of image feature tokens obtained by the encoder, Represents the sequence of text feature tokens obtained by the encoder; two MLP structures are introduced to respectively and sequence Extract the importance score for the current layer : ； Among them, s represents single mode, Representation sequence The linearly projected vector; S312, calculating the importance scores of the image feature tokens and the text feature tokens for the cross-modal layer; S3121, fuse and project the category feature tokens corresponding to the image feature token and the text feature token to obtain the fused feature : ； in, Represents a sequence of image feature tokens Deletion operation of corresponding category feature token; Represents a sequence of text feature tokens The corresponding category feature token deletion operation; M represents the operation of fusing the category feature token of the image feature token and the category feature token of the text feature token; S3122. Calculate the importance score of the cross-modal layer : ； Among them, c represents cross-modality, norm represents normalization operation, Represents the linear projection Transpose of a vector; S313. Calculate the overall importance score : ； S314, according to the overall importance score, through the function Sampling is performed, where a sampling value of 1 indicates that the corresponding image feature token and text feature token are retained, and a sampling value of 0 indicates that the corresponding image feature token and text feature token are removed; Step S32 includes: S321, introduce the MLP structure to calculate the importance score of the single modal attention head and the importance score of the cross-modal attention head : ； ； in, Indicates the sequence obtained Deletion operation of corresponding category feature token; S322, according to the importance score of the single modal attention head and the importance score of the cross-modal attention head, through the function Sampling is performed, where a sampling value of 1 indicates retaining the unimodal attention head and the cross-modal attention head, and a sampling value of 0 indicates removing the unimodal attention head and the cross-modal attention head.

10. A multimodal large model inference acceleration system integrating image feature compression and adaptive pruning, characterized in that: The reasoning acceleration method according to any one of claims 1 to 9 is implemented, and the reasoning acceleration system includes the following modules: An image resolution dynamic matching and adjustment module is used to determine the optimal number of image blocks of the input image according to the aspect ratio of the input image and the reference image resolution, so as to dynamically match and adjust the image resolution; The image feature compression module processes the input image after adjusting the image resolution through the visual encoder and attention module to obtain image feature tokens, analyzes the redundancy and semantic information contribution between image feature tokens, retains image feature tokens that contribute most to the image information volume, and realizes image feature compression; The model adaptive pruning module encodes the input text to obtain text feature tokens; based on the importance scores of image feature tokens and text feature tokens, it identifies and removes unimodal redundant features and cross-modal redundant features of the visual modality and text modality, and prunes redundant model structures.

Images