Visual language model reasoning acceleration method based on input-parameter joint pruning

Abstract

The invention discloses a visual language model reasoning acceleration method based on input-parameter joint pruning, which comprises the following steps of: inputting an image-text sample into a visual language model, constructing a visual token sequence, a text token sequence and comprehensive importance scores after layer embedding and noise disturbance, and jointly inputting the comprehensive importance scores into a meta router to construct a pruning strategy; introducing KL divergence to quantify the influence of a pruning strategy on output distribution of the visual language model, and constructing a preference sample pair; carrying out self-supervised training on the element router by using the preference sample pair and a direct preference optimization method; the trained meta-router is applied to a visual language model and used for generating a corresponding pruning strategy when a newly obtained image-text is input and reasoned; and after the visual language model performs pruning of the visual token and the converter layer according to the pruning strategy, reasoning is performed on image-text input, and a reasoning result is output.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of language image multi-modal fusion, and particularly relates to a visual language model inference acceleration method based on input-parameter joint pruning. BACKGROUND

[0002] In recent years, with the development of large-scale pre-training technology, large visual language models have become an important direction of artificial intelligence research. Such models can exhibit strong performance in image understanding, visual question answering, cross-modal reasoning and other multi-task by combining visual encoders, multi-modal connectors and large language models. However, large visual language models usually contain tens of billions or even hundreds of billions of parameters, and the computation cost is extremely high in the inference process, resulting in serious inference delay, which limits the application of the model in practical scenarios.

[0003] Existing visual language model inference acceleration methods mainly fall into two categories: parameter-dependent methods and token-dependent methods. The former reduces the number of parameters by pruning, sparsifying or redesigning the model architecture, but usually requires retraining the model to restore performance, resulting in high labeling and computing costs. The latter reduces the input length by token pruning based on attention scores, but due to the large token selection space, the attention ranking strategy is difficult to stabilize the optimal token combination, which easily leads to performance fluctuations. In addition, existing methods often separate parameter pruning and token pruning, ignoring the mutual influence of the two. At the same time, there is a lack of adaptive joint optimization scheme, making it difficult to achieve efficient landing while balancing inference efficiency and accuracy. SUMMARY

[0004] The purpose of the present application is to provide a visual language model inference acceleration method based on input-parameter joint pruning to solve the problems existing in the prior art.

[0005] In order to achieve the above-mentioned task, the technical scheme adopted by the present application is as follows: The visual language model inference acceleration method based on input-parameter joint pruning comprises: inputting a picture-text sample into a visual language model to obtain a visual token sequence and a text token sequence; determining the comprehensive importance score of each visual token using a multi-head attention mechanism and adding noise disturbance to it; selecting multiple transformer layers from the language model in the visual language model and encoding the layer index into layer embedding; The visual token sequence, the text token sequence, the layer embedding, and the integrated importance score after noise disturbance are jointly input into the meta-router, and the meta-router outputs a visual token reservation probability distribution and a layer reservation probability distribution; based on a preset threshold parameter, the reservation number of the visual token and the converter layer is determined, thereby obtaining a corresponding pruning strategy; based on the size of the KL divergence corresponding to the pruning strategy, the pruning strategy is divided into an optimal pruning scheme and a non-optimal pruning scheme, and the pruning strategy is selected from the optimal pruning scheme and the non-optimal pruning scheme respectively and applied to the visual language model, and the output corresponding to the visual language model and the image-text sample jointly constitute a preference sample pair; The meta-router is trained in a self-supervised manner by using a direct preference optimization method based on the preference sample pair; The trained meta-router is applied to the visual language model to generate a corresponding pruning strategy when reasoning on a newly obtained image-text input; the visual language model performs pruning on the visual token and the converter layer according to the pruning strategy, reasons on the image-text input, and outputs a reasoning result.

[0006] Further, the integrated importance score of each visual token is determined by using a multi-head attention mechanism and noise disturbance is added to the integrated importance score, including: The integrated importance score is calculated according to the following formula:

[0007] Noise disturbance is added to the integrated importance score, which is represented as:

[0008] wherein, is the number of attention heads; represents the attention score of the i-th attention head to the j-th visual token is the noise-disturbed integrated importance score, is zero-mean Gaussian noise, is a noise standard deviation hyperparameter. Further, the meta-router includes an input feature fusion module, a lightweight encoding module, and a decision mapping module, wherein: The input feature fusion module receives the visual token sequence, the text token sequence, the layer embedding, and the integrated importance score after disturbance, and splices and fuses them into a unified feature representation through linear mapping and dimension alignment to obtain a fusion feature.

[0009] ​​​The lightweight encoding module comprises one or more layers of self-attention networks, each layer of self-attention network comprising a feedforward sub-layer structure; the self-attention network is used to capture the interaction relationship between the multi-modal and the inter-level in the fused features, and the feedforward sub-layer is used to perform nonlinear mapping and dimension compression on the features after the interaction to obtain unified representation features; wherein the perturbed comprehensive importance score is used to adjust the attention head dimension allocation in the feature fusion stage, and the visual tokens with higher corresponding scores will be allocated more attention head dimensions; The decision mapping module is composed of a fully connected network; the unified representation features are subjected to nonlinear mapping of the fully connected network, and then subjected to processing by an activation function, and combined with a self-attention mechanism to realize joint modeling of cross-modal and hierarchical information, thereby forming comprehensive representation features that can be used for pruning decisions.

[0010] Further, the visual token retention probability distribution and the layer retention probability distribution are respectively:

[0011]

[0012] wherein is a Sigmoid activation function, represents a multi-layer perceptron, is a visual feature output vector sequence in the comprehensive representation features , and a layer feature output vector sequence; by sorting and in descending order of probability values, a pruning strategy is obtained; the pruning strategy comprises M visual token retention numbers and K transformer layer retention numbers, corresponding to the first M and K probability values in , after sorting, respectively; (M, K) is a threshold parameter.

[0013] Further, the KL divergence is introduced to quantify the influence of the pruning strategy on the output distribution of the visual language model, and based on the size of the KL divergence corresponding to the pruning strategy, the pruning strategy is divided into an optimal pruning scheme and a non-optimal pruning scheme, comprising:

[0014] wherein, represents the probability distribution output by the visual language model without any pruning operation under the input image-text sample x, represents the probability distribution output by the pruned visual language model under the pruning strategy, is the KL divergence; The meta-router generates multiple pruning strategies for the same image / text sample under different threshold parameters (M, K), and then classifies them into optimal pruning schemes by clustering based on the KL divergence of each pruning strategy. Non-optimal pruning schemes .

[0015] Furthermore, for each image and text sample Choose a pair of pruning strategies from the preferred and non-preferred pruning schemes, and apply the pruning strategies to the visual language model respectively. Then, process the image and text sample. Inputting the pruned sensory language model yields the corresponding model output. and ,in This represents the model output corresponding to a pruning strategy with smaller KL divergence, and... As a priority during training; use image and text samples Model output and Construct a preference sample pair . Furthermore, the decision mapping module of the meta-router is also equipped with a performance evaluation sub-model. This sub-model evaluates the overall performance of the visual language model based on different performance indicators and is used to calculate the score of the corresponding pruning strategy based on the preferred sample pairs during training.

[0016] Furthermore, the meta-router is trained self-supervised using the direct preference optimization method, including: For each pair of preference samples The data is then input into the meta-router, which calculates the importance score based on the visual token sequence, text token sequence, layer embedding, and perturbation of the image and text samples, and combines this with the model output. Utilizing the performance evaluation sub-model to calculate Score corresponding to pruning strategy and This score represents the overall performance evaluation score of the visual language model under the corresponding pruning strategy; according to Given the preference labels of the priority terms, calculate the direct preference optimization loss, and minimize this loss; The corresponding pruning strategy scored higher than The corresponding pruning strategy score is used; the gradient descent algorithm is used to iteratively update the meta-router parameters until the loss converges or the preset number of rounds is reached, at which point training stops.

[0017] A terminal device comprises a processor, a memory and a computer program stored in the memory; when the processor executes the computer program, the input-parameter joint pruning based visual language model inference acceleration method is realized.

[0018] A computer readable storage medium, the medium has a computer program stored therein; when the computer program is executed by a processor, the input-parameter joint pruning based visual language model inference acceleration method is realized.

[0019] Compared with the prior art, the present application has the following technical features: The method can expand the search space of token combination and avoid falling into local optimum by simultaneously performing layer pruning and token pruning through the meta-router and combining the noise disturbance mechanism to diversify sampling of token importance without retraining the model. The method can significantly reduce the model calculation amount while maintaining the model accuracy. The method has the advantages of no retraining, self-supervised optimization, flexible configuration, and applicability to various model architectures. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The figure is a schematic diagram of the interaction process between the visual language model and the meta-router in the method of the present application. Figure 2 The figure is an experimental effect diagram in one embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solution of the present application comprises a pruning strategy generation module and a pruning strategy optimization execution module. The pruning strategy generation module comprises three parts: feature extraction, attention weighting, noise disturbance and meta-router. The first two parts are used to evaluate the importance of visual tokens and model layers, and the meta-router jointly models layer pruning and token pruning on this basis to generate a preliminary pruning strategy. The pruning strategy optimization execution module comprises three units: preference data construction, direct preference optimization (DPO) training and inference dynamic execution. The influence of different strategies on the model output is measured by KL divergence, the meta-router is self-supervised optimized by the DPO algorithm, and layers and tokens are dynamically pruned according to the optimized pruning strategy in the inference stage to realize efficient inference and maintain stable model accuracy. The technical solution of the present application will be further described in detail below with reference to the accompanying drawings.

[0022] Referring to Figure 1 The input-parameter joint pruning based visual language model inference acceleration method provided by the present application comprises the following steps: Step 1, input the image-text sample into the visual language model to obtain a visual token sequence and a text token sequence; use the multi-head attention mechanism to determine the comprehensive importance score of each visual token and add noise disturbance to it; select multiple transformer layers from the language model in the visual language model and encode the layer index into a layer embedding.

[0023] This step belongs to the general phase of meta-router self-supervised training and reasoning, and the execution body includes a visual language model and a meta-router. The visual language model is responsible for extracting multi-modal features from unannotated image-text samples, where each image-text sample is composed of an image and its corresponding text, and the text includes description, question or instruction. The text and image in an image-text sample correspond to each other in semantics and together form the multi-modal input of the visual language model. The visual language model can be an existing large model, such as LLaVA-v1.5-7B, LLaVA-Next-7B and Qwen-VL-Chat-9B, etc.

[0024] The image in the image-text sample is extracted into a visual token sequence by the visual encoder in the visual language model = , where represents the th visual token, is the number of visual tokens; the text in the image-text sample is converted into a text token sequence after being processed by the word embedding and language encoder in the visual language model , where represents the th text token, is the number of text tokens.

[0025] To evaluate the relative contribution of visual tokens in model reasoning, the multi-head attention mechanism in the visual language model is used to calculate the comprehensive importance score of each visual token ; this comprehensive importance score is one of the input features of the meta-router, which generates a joint pruning strategy in combination with the layer embedding information to realize joint modeling of tokens and layers.

[0026] The calculation formula of the comprehensive importance score is as follows:

[0027] , where is the number of attention heads; represents the th attention head to the th visual token the attention score of the visual token; the formula can integrate the attention distribution of different attention heads by averaging the multi-head attention scores, so as to more stably measure the global contribution of each token. A higher integrated importance score indicates that the visual token has higher importance in the inference process of the model and should be retained in subsequent pruning. In order to avoid that the pruning strategy relies too much on the deterministic integrated importance score, a noise perturbation mechanism is introduced:

[0028] wherein, is a zero-mean Gaussian noise, used to slightly perturb the integrated importance score of the attention score of the visual token , so as to expand the candidate combination space of the token; is a noise standard deviation hyperparameter, used to control the perturbation strength, is the integrated importance score after noise perturbation.

[0029] The input of the meta-router is always based on the importance score of the visual token, and the perturbed version of the importance score is used in the training stage to increase diversity and avoid falling into local optimum, and the original version is used in the inference stage to ensure stability and enhance model robustness.

[0030] At the same time, the last N transformer layers of the language model (LLM) part in the visual language model are selected by the meta-router as a candidate set, and the layer index is encoded into layer embedding , so that the subsequent joint pruning can take into account the hierarchical structure and token selection, realizing the joint modeling of tokens and layers.

[0031] The selected last transformer layer is considered to be more redundant in the back part of the model, so the language model is more suitable to select the last N layers (configurable) as a candidate set. In this embodiment, the visual language model uses LLaVA-1.5-7B, and the language model selected in the embodiment is the 17th-32nd layer, i.e. N = 16.

[0032] It should be noted that the last N transformer layers refer to the N transformer layers counted from the output end of the language model, and these layers are all Transformer computing layers / blocks; other functional modules such as visual encoder, multi-modal connector (Q-Former / MLP), word embedding, and output projection (LM head) in the visual language model are not included.

[0033] Step 2, input the visual token sequence, the text token sequence, the layer embedding, and the integrated importance score after noise disturbance into the meta-router, output the visual token reservation probability distribution and the layer reservation probability distribution using the meta-router; based on the preset threshold parameter, determine the reservation number of the visual token and the converter layer, thereby obtaining the corresponding pruning strategy; introduce the KL divergence to quantify the influence of the pruning strategy on the output distribution of the visual language model, based on the size of the KL divergence corresponding to the pruning strategy, divide the pruning strategy into the preferred pruning scheme and the non-preferred pruning scheme, and select the pruning strategy from them respectively to apply to the visual language model, and combine the output corresponding to the visual language model with the image-text sample to form a pair of preference samples.

[0034] The meta-router is combined with pruning and preference data construction; the meta-router is the core decision module of the joint pruning method of the application, which is used to jointly model the interaction relationship of the visual token, the text token, and the layer embedding, and generate a pruning strategy. It adopts a lightweight converter architecture, which consists of the following three parts: (1) Input feature fusion module: receives the visual token sequence V, the text token sequence Q, the layer embedding L, and the integrated importance score after disturbance ; then it is spliced and fused into a unified feature representation through linear mapping and dimension alignment, obtaining the fusion feature.

[0035] (2) Lightweight encoding module: contains 1-3 layers of self-attention networks, each layer contains a feedforward sublayer structure; the self-attention network is used to capture the interaction relationship between the multi-modal and the hierarchical in the fusion feature, and the feedforward sublayer is used to perform nonlinear mapping and dimension compression on the features after interaction, obtaining a unified representation feature, which contains joint importance features at the token level and the layer level, for the subsequent decision mapping module to generate a pruning probability distribution.

[0036] Among them, the integrated importance score after disturbance is used to adjust the attention head dimension allocation in the feature fusion stage, and the visual token with a higher corresponding score will be allocated more attention head dimensions; so that the meta-router "learns" which visual token is more important in the image-text sample.

[0037] It should be noted that in the training stage of the meta-router, the integrated importance score after disturbance is used; while in the inference stage, the integrated importance score is used. is replaced by ).

[0038] (3) Decision mapping module: composed of two fully connected networks; unified representation features are mapped through the fully connected network, then processed by the Sigmoid activation function, and combined with the self-attention mechanism to realize the joint modeling of cross-modal and hierarchical information, forming a comprehensive representation that can be used for pruning decision:

[0039] wherein, represents the parameter set of the meta-router, which includes the mapping weight and bias parameters in the input feature fusion module, the self-attention network and feedforward network parameters in the lightweight encoding module, and the fully connected network parameters in the decision mapping module; represents the splicing result of the visual token sequence, the text token sequence and the layer embedding under the influence of the comprehensive importance score after perturbation.

[0040] The visual token retention probability distribution is obtained by the following formula:

[0041] wherein, is a Sigmoid activation function, represents a multi-layer perceptron, represents a visual feature output vector sequence in . The layer retention probability distribution is obtained by the following formula:

[0042] wherein, represents a layer feature output vector sequence in .

[0043] By sorting and from high to low according to the probability value, the meta-router can dynamically generate a pruning strategy for token pruning and layer pruning through the decision mapping module; in the pruning strategy, "token pruning" and "layer pruning" are combined by adjusting the threshold values of the number of retained visual tokens M (i.e. the visual tokens corresponding to the top M visual token retention probabilities) and the number of retained converter layers K (i.e. the converter layers corresponding to the top K layer retention probabilities), each set of strategy corresponds to a set of different threshold parameters (M, K), which is used to evaluate the performance under different pruning ratios in the subsequent preference optimization stage. The system selects the best performer from these candidate strategies according to the accuracy and inference efficiency, thereby realizing a one-to-many correspondence between the image-text sample and the pruning strategy.

[0044] The decision mapping module is also provided with a performance evaluation sub-model, which can be a pre-trained neural network or a pre-set performance evaluation strategy or algorithm (for example, the model performance is evaluated based on different performance indicators combined with the output of the model), used to calculate the score of the corresponding pruning strategy based on the preferred sample pair during training.

[0045] In order to quantify the influence of pruning strategy on the output distribution of the model, a KL divergence-based evaluation mechanism is introduced:

[0046] Among them, represents the probability distribution output by the visual language model without any pruning operation under the input image-text sample x (image+text), represents the probability distribution output by the pruned visual language model under a set of pruning strategies; the KL divergence measures the difference between the model outputs before and after pruning, and a smaller KL value indicates that the pruning strategy has less impact on the accuracy of the visual language model. Based on the KL divergence index, the meta-router generates multiple sets of pruning strategies for the same image-text sample under different threshold parameters (M, K), and according to the KL divergence size of each pruning , it is divided into preferred pruning scheme and non-preferred pruning scheme ; The specific division method may, for example, use the kmeans clustering method.

[0047] For each image-text sample , a pair of pruning strategies is selected from the preferred pruning scheme and the non-preferred pruning scheme, and the pruning strategies are applied to the visual language model, respectively. After that, the image-text sample is input into the pruned visual language model to obtain the corresponding model output and , wherein represents the model output corresponding to the pruning strategy with smaller KL divergence, and is used as the priority (preferred label) during training, and the image-text sample , the model output and are used to construct a preferred sample pair , which is used to guide the model to prefer the better pruning strategy during the preferred optimization phase; The set composed of the preferred sample pairs constructed by all image-text samples is the preferred data set. Step 3, using the preferred sample pairs in the preferred data set, the meta-router is trained by using the direct preferred optimization method, and the training process is as follows: During the preferred optimization training phase, for each preferred sample pair in the preferred data set ​The input is then fed into the meta-router, which is based on the visual token sequence V, text token sequence Q, layer embedding L, and the perturbed comprehensive importance score of the image / text sample x. and combined with model output Utilizing the performance evaluation sub-model to calculate Score corresponding to pruning strategy and This score is the comprehensive performance evaluation score of the visual language model under the corresponding pruning strategy; according to Given the preference labels of the priority terms, calculate the direct preference optimization loss, and minimize this loss; The corresponding pruning strategy scored higher than The corresponding pruning strategy score; the gradient descent algorithm is used to iteratively update the meta-router parameters until the loss converges or the preset number of rounds is reached and then training stops; during training, the parameters of the large visual language model are kept frozen and only used as a feature extractor for forward inference, without participating in parameter updates.

[0048] The optimization objective is to guide the meta-router to increase the probability of selecting the preferred pruning scheme and decrease the probability of selecting the non-preferred pruning scheme by comparing and learning the preferred pruning scheme and the non-preferred pruning scheme. In this way, the optimal pruning decision that can balance speed and accuracy can be learned step by step under unsupervised conditions.

[0049] Step 4: Apply the trained meta-router to the visual language model to generate corresponding pruning strategies when reasoning on newly acquired image-text input (image + text); after pruning the visual token and converter layers according to the pruning strategy, the visual language model performs reasoning on the image-text input and outputs the reasoning results.

[0050] Once training is complete, the meta-router collaborates with the visual language model during the inference phase as follows: During the inference phase, the meta-router, trained with preference optimization, is embedded as an independent module between the multimodal connector of the visual language model and the language model. During inference, image I and corresponding text Q are processed by the visual encoder and language encoder of the visual language model, respectively, to obtain visual token sequences and text token sequences. The multi-head attention mechanism within the visual language model calculates the comprehensive importance score of the visual tokens. Subsequently... It is input to the meta router and obtained and and respectively sort the probability values contained therein in descending order; select the top M, K probabilities under a given computational budget, that is, obtain the selected M visual tokens and K transformer layers, thereby forming a corresponding pruning strategy; filter the corresponding M visual tokens at the multimodal connector and skip the corresponding K transformer layers in the language model of the visual language model according to the strategy; finally, the visual token sequence and the text token sequence are subjected to subsequent processing by the visual language model to generate an inference result for the image I and the corresponding text Q.

[0051] The threshold parameters (M, K) can be adjusted according to the system budget (such as the upper limit of floating-point operation amount, the inference delay target, or the video memory budget); under a 50% pruning budget, the meta-router can automatically generate a pruning strategy that retains about 100 tokens and 12 layers of transformer layers, significantly reducing the computational load of the model; when the budget is further reduced to 40%, the model automatically adjusts to retain 80 tokens and 10 layers of transformer layers, thereby achieving higher acceleration effect.

[0052] Through this mechanism, the present application can adaptively adjust the pruning amplitude under different hardware conditions and task scenarios, achieving a dynamic balance between inference speed and model accuracy; the optimal pruning strategy is adaptively generated according to the input features and computational resource constraints. The number of tokens of LLaVA-1.5-7B without pruning is 576, and the number of layers is 32, and the model inference calculation amount TFLOPS is 11.05. The test results of the present application in 9 test sets all achieve better results. In particular, the token number of the model of the present application is 144, the layer number is 24, and the inference calculation amount of the model is reduced from 11.05 to 5.9, which is better than the best model FastV3.3%, and the compression ratio is also improved by about 6.8%; and when the token number is further compressed to 128, the layer number is 32, the accuracy is consistent with the best model, and the inference calculation amount is reduced to 5.07, which is 7.8% better than the best model. The experimental results on the POPE and SEED datasets show that, compared with the single token pruning method, the average accuracy is improved by about 1.2% under the same budget; compared with the single layer pruning method, the BLEU score is improved by 1.5 points in the visual question answering task of the SEED dataset; in the comprehensive scenario, the present application can reduce the computational load by about 45% compared with the unpruned model, while maintaining close task performance.

[0053] Most importantly, compared with those training-based methods, the model of the present application can achieve the performance of the best model without updating the parameters of the pruned visual language model.

[0054] Embodiment: In the model training process of one embodiment of the application, 5000 samples in the LLaVA-80k dataset are used to construct a preference dataset, and verification is performed on the POPE and SEED and other multi-modal evaluation sets. The visual language model is LLaVA-v1.5-7B, and also supports LLaVA-Next-7B and Qwen-VL-Chat-9B. The training uses the AdamW optimizer, the batch size is 4, and is performed on 4 NVIDIA 3090 GPUs. To achieve efficient preference learning, the pruning action number K is set to 4 and the maximum token pruning number M is set to 400; during the training process, first, based on the KL divergence, the output difference of different pruning strategies is measured, and the preferred pruning scheme and the non-preferred pruning scheme are divided, then the meta-router is trained using direct preference optimization, so that it can stably generate efficient joint pruning strategies under different computing budgets. This training process does not require labeled data, but only relies on preference data and self-supervised signals to complete optimization, which can effectively reduce the computing cost while maintaining the model accuracy.

[0055] In various experiments, the method shows stable inference acceleration ability on the POPE and SEED datasets, and still maintains the accuracy performance close to the unpruned model under high compression rate. The final effect is shown in Figure 2 , which shows the visual output results of the model under the inference acceleration condition. It can be seen that the method can still generate prediction outputs highly consistent with the original model while significantly reducing the amount of calculation, proving the effectiveness and feasibility of the application in practical application scenarios. In the model application stage, the application can flexibly adjust the pruning ratio according to different computing budgets to adapt to resource-constrained or high-precision scenarios.

[0056] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A visual language model inference acceleration method based on input-parameter joint pruning, characterized in that, The method comprises the following steps: inputting the image-text sample into the visual language model to obtain a visual token sequence and a text token sequence; determining the comprehensive importance score of each visual token by using the multi-head attention mechanism and adding noise disturbance to the visual token; selecting multiple transformer layers from the language model in the visual language model and encoding the layer indexes of the multiple transformer layers into layer embeddings; inputting the visual token sequence, the text token sequence, the layer embeddings, and the comprehensive importance score after the noise disturbance into the meta-router, and outputting a visual token retention probability distribution and a layer retention probability distribution by using the meta-router; determining the retention number of the visual token and the transformer layer based on a preset threshold parameter, thereby obtaining a corresponding pruning strategy; and dividing the pruning strategy into an optimal pruning scheme and a non-optimal pruning scheme based on the size of the KL divergence corresponding to the pruning strategy, and selecting the pruning strategy from the optimal pruning scheme and the non-optimal pruning scheme, respectively, and applying the pruning strategy to the visual language model, and combining the output of the visual language model corresponding to the image-text sample to form a preferred sample pair. using the preferred sample pair to perform self-supervised training on the meta-router by using a direct preference optimization method; applying the trained meta-router to the visual language model to generate a corresponding pruning strategy when the visual language model reasons on a newly obtained image-text input; after pruning the visual token and the transformer layer according to the pruning strategy, the visual language model reasons on the image-text input and outputs a reasoning result.

2. The input-parameter joint pruning based visual language model inference acceleration method according to claim 1, characterized in that, determining the comprehensive importance score of each visual token by using the multi-head attention mechanism and adding noise disturbance to the visual token, comprising: The formula for calculating the composite importance score is as follows: ​ adding noise disturbance to the visual token, which is represented as: wherein, is the number of attention heads; denotes the th attention head; th visual token; is the importance score after noise perturbation, is a zero-mean Gaussian noise, is a noise standard deviation hyperparameter.​ 3. The input-parameter joint pruning based visual language model inference acceleration method according to claim 1, wherein, The meta-router comprises an input feature fusion module, a lightweight encoding module, and a decision mapping module, wherein: The input feature fusion module receives the visual token sequence, the text token sequence, the layer embeddings, and the comprehensive importance score after the noise disturbance, and splices and fuses them into a unified feature representation through linear mapping and dimension alignment, thereby obtaining fused features. The lightweight encoding module comprises one or more self-attention networks, and each self-attention network comprises a feedforward sublayer structure; the self-attention network is used to capture the interaction between the multi-modal and the hierarchical levels in the fused features, and the feedforward sublayer is used to perform nonlinear mapping and dimension compression on the features after the interaction, thereby obtaining a unified representation feature; wherein the comprehensive importance score after the noise disturbance is used to adjust the attention head dimension allocation in the feature fusion stage, and the visual token with a higher corresponding score will be allocated more attention head dimensions; The decision mapping module is composed of a fully connected network; the unified representation feature is subjected to nonlinear mapping by the fully connected network, and then is processed by an activation function, and the cross-modal and hierarchical information is jointly modeled by using the self-attention mechanism, thereby forming a comprehensive representation feature that can be used for pruning decision.

4. The input-parameter joint pruning based visual language model inference acceleration method according to claim 1, characterized in that, Visual token retention probability distribution And layer retention probability distribution Respectively: wherein is a Sigmoid activation function, denotes a multi-layer perceptron, is a comprehensive representation of features a sequence of visual feature output vectors, a sequence of layer feature output vectors; Through the and The visual tokens are sorted from highest to lowest probability values ​​to obtain the pruning strategy. This pruning strategy includes M visual token retention quantities and K converter layer retention quantities, which correspond to the sorted values ​​respectively. , The first M and K probability values ​​in the table; (M,K) is the threshold parameter.

5. The input-parameter joint pruning based visual language model inference acceleration method according to claim 1, characterized in that, The KL divergence is introduced to quantify the influence of the pruning strategy on the output distribution of the visual language model, and the pruning strategy is divided into an optimal pruning scheme and a non-optimal pruning scheme based on the size of the KL divergence corresponding to the pruning strategy, comprising: wherein, denotes the probability distribution output by the visual language model without any pruning operation under the input image-text sample x, denotes the probability distribution output by the pruned visual language model under the pruning strategy, is the KL divergence; The meta-router generates multiple sets of pruning strategies for the same image-text sample under different threshold parameter (M, K) settings, and divides the pruning strategies into preferred pruning schemes and non-preferred pruning schemes according to the size of the KL divergence corresponding to each pruning strategy through clustering and non-preferred pruning schemes .

6. The input-parameter joint pruning based visual language model inference acceleration method according to claim 1, characterized in that, For each image-text sample , a pair of pruning strategies is selected from the preferred pruning scheme and the non-preferred pruning scheme, and the pruning strategies are applied to the visual language model respectively, and then the image-text sample is input into the pruned visual language model to obtain the corresponding model output . , where represents the model output corresponding to the pruning strategy with smaller KL divergence, and is taken as the priority during training; an image-text sample pair , model output and is constructed as a preference sample pair .

7. The input-parameter joint pruning based visual language model inference acceleration method according to claim 1, characterized in that, The decision mapping module of the meta-router is further provided with a performance evaluation submodel, which evaluates the comprehensive performance of the visual language model based on different performance indicators, and is used to calculate the score of the corresponding pruning strategy based on the preferred sample pair during training.

8. The input-parameter joint pruning based visual language model inference acceleration method according to claim 1, characterized in that, The direct preference optimization method is used for self-supervised training of the meta-router, including: For each pair of preference samples , input into the meta-router, the meta-router combines the visual token sequence, the text token sequence, the layer embedding of the graph-text sample, and the disturbed comprehensive importance score, and combines the model output , calculates the score of the corresponding pruning strategy by using the performance evaluation sub-model , and , the score represents the comprehensive performance evaluation score of the visual language model under the corresponding pruning strategy; according to the priority preference label of , calculate the direct preference optimization loss by minimizing the loss; make the score of the corresponding pruning strategy higher than the score of the corresponding pruning strategy; use the gradient descent algorithm to iteratively update the meta-router parameters until the loss converges or the preset number of rounds is reached to stop training.​ 9.A terminal device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that, The processor executes the computer program to implement the visual language model inference acceleration method based on input-parameter joint pruning according to any one of claims 1-8.

10. A computer readable storage medium having stored therein a computer program; characterized in that, The computer program is executed by the processor to implement the visual language model inference acceleration method based on input-parameter joint pruning according to any one of claims 1-8.

Images