Visual language model acceleration method and device, equipment and medium

Abstract

The invention discloses a visual language model acceleration method and device, equipment and a medium, and relates to the field of artificial intelligence in the fields of finance, medical treatment, insurance, banking and the like, and the method comprises the steps: in a pre-trained visual language model, freezing front K layers as a backbone network, and adding a branch module composed of T lightweight transformation layers; only the branch modules are finely adjusted, and other parameters are kept frozen. And executing visual token pruning: after input data sequentially passes through the backbone network and the branch module, screening a key visual token according to the attention map of the last layer of the branch module, and inputting the key visual token to a subsequent network layer. In the text decoding stage, self-speculation decoding is executed, specifically, a candidate token sequence is generated with a branch module as a grass model, the sequence is input into an original model in parallel for single-time forward propagation verification, and the grass sequence is corrected according to a verification result to generate final output. According to the method, the reasoning efficiency of the visual language model can be improved, the model accuracy is kept, and the long sequence generation process is remarkably accelerated.

Description

A method, apparatus, device, and medium for accelerating visual language models Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for accelerating visual language models. Background Technology

[0002] In fields such as finance, healthcare, insurance, and banking, visual language models play a crucial role as a key technology for enabling image-to-natural language interaction in numerous practical applications. However, with the continuous improvement of model size and performance, computational overhead and inference latency also increase, posing challenges for real-time deployment in resource-constrained environments. Therefore, efficiently accelerating visual language models has become an important direction in current research and industrial applications.

[0003] Currently, acceleration methods in this field mainly revolve around the core idea of ​​reducing model computation. One mainstream approach is to compress the input visual information during the model inference stage, specifically by pruning visual tokens. These methods analyze intermediate information such as attention maps generated within the model to identify and remove visual tokens deemed redundant, thereby reducing the sequence length required for subsequent layers and improving inference efficiency. However, while existing techniques relying on visual token pruning achieve acceleration, they still have some inherent limitations.

[0004] First, there are shortcomings in the effectiveness of pruning strategies. Most existing methods rely on attention maps generated by the shallow layers of the model to decide which visual tokens should be retained or removed. Since shallow layers typically focus on extracting low-level or local features of an image, the attention maps they generate have weak perception and discrimination capabilities for high-level semantic information. This attention-guided pruning process may prematurely discard visual information that plays a crucial role in the final task judgment in subsequent deep layers. This mis-deletion of information due to the low semantic level of the pruning basis makes it easy for the model to experience a significant drop in accuracy when performing complex visual language understanding tasks, making it difficult to achieve a good balance between efficiency and accuracy.

[0005] Furthermore, there are limitations in the scope of acceleration during the inference process. Existing acceleration techniques mainly focus on optimizing the pre-filling stage of the model, i.e., the process of encoding the input image and text prompts. However, the complete inference process of visual language models also includes a decoding stage, i.e., the process of autoregressively generating text responses. Especially in tasks that require generating long text sequences, the decoding stage generates a large amount of repetitive computation due to its serial generation characteristics, accounting for a significant proportion of the total inference time. Current visual token pruning methods have not adequately optimized this crucial stage, resulting in a bottleneck in the overall inference speed improvement when generating long responses, with limited efficiency gains.

[0006] In summary, existing technologies have not yet adequately addressed the challenge of balancing accuracy maintenance with overall efficiency improvement during the acceleration of visual language models. Summary of the Invention

[0007] The present invention provides a method, apparatus, device and medium for accelerating visual language models. The technical problem it aims to solve is: how to provide a solution that can improve the inference efficiency of visual language models while effectively maintaining model accuracy and significantly accelerating the long sequence generation process.

[0008] In a first aspect, embodiments of the present invention provide a method for accelerating a visual language model, comprising: freezing the first K layers of a pre-trained visual language model as a backbone network, and appending a branch module consisting of T lightweight transform layers after the backbone network, wherein K is a positive integer and T is a positive integer less than the difference between the total number of layers of the visual language model and K; fine-tuning the branch module using a training dataset, wherein the fine-tuning process only updates the parameters contained in the branch module, and keeps the parameters of all layers in the visual language model except the branch module frozen; and performing branch-guided visual token pruning for input data containing images and text prompts: [The text abruptly ends here, so the translation stops here as well.] The system sequentially passes through the backbone network and the finely tuned branch modules. Based on the attention map output from the last layer of the branch modules, key visual tokens are selected from the visual tokens contained in the input data. These key visual tokens are then input into subsequent network layers in the visual language model after the branch modules for processing. During the decoding stage of the visual language model generating the text sequence, auto-inference decoding is performed: using the finely tuned branch modules as a draft model, multiple candidate tokens are generated in an autoregressive manner to form a draft sequence. The draft sequence is then input into the visual language model in parallel for a single forward propagation verification, and the draft sequence is corrected based on the verification results to generate the final output text.

[0009] A further technical solution is that, after the backbone network, a branch module consisting of T lightweight transformation layers is added, comprising: extracting a continuous T-layer network from the (K+1)th layer to the (K+T)th layer from the network structure of the pre-trained visual language model; copying the weight parameters and bias parameters of each layer in the extracted T-layer network to the T lightweight transformation layers of the branch module respectively; and connecting the branch module after parameter copying to the output of the backbone network to form a complete inference architecture including the backbone network, the branch module, and subsequent layers of the visual language model.

[0010] A further technical solution is that the fine-tuning of the branch modules using the training dataset includes: inputting samples from the training dataset into the complete inference architecture for forward propagation; calculating the autoregressive loss of the text sequence generated by the complete inference architecture for the input samples, wherein the loss is the cross-entropy loss between the model's predicted next token and the real label; calculating only the gradient of the parameters in the branch modules during backpropagation; and updating the parameters of the branch modules according to the gradient using a preset optimization algorithm, while keeping the parameters of the backbone network and subsequent layers of the visual language model unchanged.

[0011] A further technical solution is that, based on the attention map output from the last layer of the branching module, key visual tokens are selected from the visual tokens contained in the input data, including: parsing the attention map, extracting the attention scores corresponding to all visual tokens in the attention map, and forming a score set; sorting the attention scores in the score set in descending order; and selecting the visual tokens corresponding to the top R highest attention scores from the sorted score set according to a preset retention quantity R, as the selected key visual tokens.

[0012] A further technical solution is as follows: the step of correcting the draft sequence based on the verification result to generate the final output text includes: performing a single forward propagation on the parallel input draft sequence through the visual language model, and outputting the probability distribution at each candidate token position in the draft sequence; extracting the confidence score of the corresponding target token from the probability distribution as the verification confidence of each candidate token; sequentially determining whether the verification confidence of each candidate token is greater than or equal to a preset confidence threshold according to the order of the candidate tokens in the draft sequence; accepting the consecutive candidate token sequences with verification confidence scores not lower than the confidence threshold as the valid output of this iteration; when the first candidate token with a verification confidence score lower than the confidence threshold is encountered, stopping the acceptance of subsequent candidate tokens, and concatenating the valid output sequence already accepted in this decoding iteration to the final output text generated in all previous iterations to form the updated complete output text; and using the updated complete output text as the input condition for the fine-tuned draft model to generate a new draft sequence in the next decoding iteration.

[0013] A further technical solution is as follows: the preset confidence threshold is configured as follows: during the verification phase before model deployment, a verification dataset containing multiple sets of image-text pairs is prepared; for each sample in the verification dataset, the self-predictive decoding process is run using multiple different candidate confidence thresholds; for each candidate confidence threshold, the average text generation speed is recorded when it runs on the verification dataset, and the automated evaluation score of the generated text relative to the standard answer is calculated; among the multiple candidate confidence thresholds, a threshold is selected such that the weighted comprehensive performance index of the average text generation speed and the automated evaluation score reaches the optimal level.

[0014] A further technical solution is that, in the decoding stage, the method further includes performing visual token rollback: in the network layers of the visual language model, a network layer located after the branch module and with a layer number greater than a first preset value is selected as the rollback point; when the key visual token propagates to the rollback point during inference, all the key visual tokens are removed from the current processing sequence; after removing the key visual tokens, the subsequent autoregressive text generation process continues only based on the remaining text token sequence.

[0015] Secondly, embodiments of the present invention also provide a visual language model acceleration device, which includes a unit for performing the above-described method.

[0016] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0017] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0018] This invention provides a method, apparatus, device, and medium for accelerating visual language models. The method includes: freezing the first K layers of a pre-trained visual language model as a backbone network, and appending a branch module consisting of T lightweight transform layers after the backbone network, where K is a positive integer and T is a positive integer less than the difference between the total number of layers in the visual language model and K; fine-tuning the branch module using a training dataset, wherein the fine-tuning process only updates the parameters contained in the branch module, while keeping the parameters of all layers in the visual language model frozen except for the branch module; for input data containing images and text prompts, performing branch-guided visual token pruning: sequentially passing the input data through the backbone network and... The fine-tuned branching module, based on the attention map output from the last layer of the branching module, filters key visual tokens from the visual tokens contained in the input data, and inputs these key visual tokens into subsequent network layers in the visual language model after the branching module for processing. During the decoding stage of the text sequence generated by the visual language model, auto-inference decoding is performed: using the fine-tuned branching module as a draft model, multiple candidate tokens are generated in an autoregressive manner to form a draft sequence; the draft sequence is input in parallel into the visual language model for a single forward propagation verification, and the draft sequence is corrected based on the verification result to generate the final output text. This invention constructs a parameter-efficient fine-tuning framework by freezing the first K layers of the base model as the backbone network and adding trainable branching modules. During inference, visual token pruning is performed using the attention map of the last layer of the branching module, accurately retaining key visual tokens based on deep semantic information, reducing computational load while maintaining model accuracy. During the decoding stage, the trained branch modules are used as a lightweight draft model to generate candidate sequences, which are then verified and corrected in parallel by the original model. The serial generation is transformed into batch processing through a self-inferential decoding mechanism, thereby significantly accelerating the generation of long texts. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is a flowchart illustrating a visual language model acceleration method provided in an embodiment of the present invention; Figure 2 is a schematic block diagram illustrating a computer device provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0023] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0026] Please refer to Figure 1. This embodiment of the invention provides a method for accelerating a visual language model. By introducing a collaborative architecture and inference mechanism, it systematically solves the problem of balancing efficiency and accuracy in existing technologies and achieves a significant speed improvement in long sequence generation scenarios. Specifically, the method includes the following steps: S1, in the pre-trained visual language model, the first K layers of the visual language model are frozen as the backbone network, and a branch module consisting of T lightweight transformation layers is added after the backbone network, where K is a positive integer and T is a positive integer less than the difference between the total number of layers of the visual language model and K.

[0027] In practice, a parameter-efficient knowledge transfer and adaptation framework is established by freezing the first K layers of the pre-trained visual language model as the backbone network and adding a lightweight branch module. Freezing the backbone network ensures the stable preservation of basic visual feature extraction capabilities, avoiding the huge computational overhead and potential performance fluctuations caused by retraining. The branch module initializes parameters from the corresponding layers of the visual language model, enabling it to possess preliminary semantic understanding capabilities compatible with the features of the backbone network from the very beginning of training.

[0028] In some preferred embodiments, the step of adding a branch module consisting of T lightweight transformation layers after the backbone network includes: extracting a continuous network of T layers from the (K+1)th layer to the (K+T)th layer from the network structure of the pre-trained visual language model; copying the weight parameters and bias parameters of each layer in the extracted T layers to the T lightweight transformation layers of the branch module; and connecting the branch module with the parameters copied to the output of the backbone network to form a complete inference architecture including the backbone network, the branch module, and subsequent layers of the visual language model.

[0029] In practice, firstly, a continuous T-layer network from layer K+1 to layer K+T is precisely extracted from the pre-trained basic model network structure. This T-layer network is part of the original model, has been pre-trained on a large scale, and contains rich knowledge of feature transformation.

[0030] Furthermore, a parameter copying operation is performed. The extracted weight parameter matrix and bias parameter vector of each network layer are copied one by one and correspondingly to the T lightweight transformation layers of the newly created branch module. This means that the first layer of the branch module receives parameters from the (K+1)th layer of the base model, the second layer receives parameters from the (K+2)th layer, and so on. This initialization process ensures that the branch module has feature processing capabilities similar to the corresponding intermediate layers of the original model at the start of training, laying a good foundation for subsequent efficient fine-tuning.

[0031] Furthermore, this parameter-initialized branch module is logically and structurally connected to the output of the frozen backbone network. This constitutes a complete inference architecture capable of end-to-end forward propagation and training, which sequentially includes: the frozen backbone network (first K layers), trainable branch modules (newly initialized T layers), and the remaining frozen subsequent network layers in the base model.

[0032] This embodiment achieves a key technical effect by initializing the branch modules by copying parameters from the corresponding intermediate layers of the original visual-language model. This initialization strategy is not random assignment, but rather provides the branch modules with a high-starting-point initial state compatible with the feature representation of the backbone network. Since the copied parameters originate from the fully pre-trained model, these parameters encode general visual-language association knowledge, enabling the branch modules to effectively deepen visual features from the outset of training. This significantly reduces the amount of data required and the training difficulty for the branch modules to learn from scratch, accelerating the fine-tuning convergence process. More importantly, this initialization, originating from the backbone network, ensures that the features processed by the branch modules are semantically and statistically more consistent with the expected input feature distribution of the subsequent frozen layers of the original model, reducing interference caused by feature mismatch between modules. Therefore, the reliability of subsequent pruning based on the attention maps output by the branch modules and their use as draft models is guaranteed, providing a fundamental support for the effectiveness and stability of the entire acceleration method.

[0033] S2, fine-tuning the branch module using the training dataset, wherein the fine-tuning process only updates the parameters contained in the branch module, and keeps the parameters of all layers in the visual language model frozen except for the branch module.

[0034] In practice, during fine-tuning, only the parameters of the branch modules are updated, while the parameters of the main model remain frozen. This design limits the number of parameters that need to be optimized to a very small range, significantly reducing the computational memory and time resources required for training, thus achieving high efficiency and low cost during the training phase. This provides feasibility for quickly adapting to different downstream tasks and is a prerequisite for the practical application of the entire acceleration method.

[0035] In some preferred embodiments, the fine-tuning of the branch modules using the training dataset includes: inputting samples from the training dataset into the full inference architecture for forward propagation; calculating the autoregressive loss of the text sequence generated by the full inference architecture for the input samples, wherein the loss is the cross-entropy loss between the model's predicted next token and the true label; calculating only the gradient of the parameters in the branch modules during backpropagation; and updating the parameters of the branch modules according to the gradient using a preset optimization algorithm, while keeping the parameters of the backbone network and subsequent layers of the visual language model unchanged.

[0036] In practice, firstly, the samples from the training dataset, i.e., image-text pairs, are input into the complete inference architecture. After being encoded, the images and text are sequentially passed through the frozen backbone network, the branch modules to be trained, and the frozen subsequent network layers for a complete forward propagation, ultimately outputting the predicted text sequence.

[0037] Furthermore, the autoregressive loss for this prediction is calculated. Specifically, based on the probability distribution of the next token predicted by the model at each token position, it is compared with the next token in the true label. The cross-entropy loss function is used to quantify the prediction error, and the sum or average of the losses at all positions is the total loss for this sample.

[0038] Furthermore, the backpropagation process is initiated. During this process, the loss gradient is propagated backward layer by layer from the output to the input. The system is configured to compute the gradients of only the parameters of the T transform layers in the branch modules, while the gradient computation of the parameters of the backbone network and all subsequent frozen layers is masked or set to zero.

[0039] Furthermore, using a pre-defined optimization algorithm, such as stochastic gradient descent or its variant Adam, these parameters are updated based on the calculated gradients of the branch module parameters. Simultaneously, the parameters of the backbone network and subsequent layers of the visual-language model are explicitly kept unchanged. This process is iterated across the entire training dataset until the parameters of the branch modules converge.

[0040] The specific training process defined in this embodiment directly results in high efficiency and low cost. By utilizing the complete inference architecture in forward propagation, it is ensured that the calculation of the loss function during training is performed under the full process of simulating the actual inference task, which keeps the learning objectives of the branch modules consistent with the final goal of accelerating inference. Crucially, only the parameter settings of the branch modules are updated during backpropagation, constituting a parameter-efficient fine-tuning paradigm. Since the main parameters of the visual language model (potentially reaching billions) remain completely frozen, the number of parameters that need to be optimized is limited to a small number of parameters in the branch modules (e.g., parameters corresponding to layer T). This significantly reduces the computational memory requirements and gradient calculation time overhead during training compared to fine-tuning the entire model or even just fine-tuning a portion of the original layers. Therefore, this embodiment can complete the adaptation training of branch modules with significantly fewer hardware resources and shorter training time, achieving high efficiency in the training phase. This efficiency allows this acceleration method to be quickly adapted to different downstream tasks or datasets, improving the method's practicality and deployability.

[0041] S3, For input data containing images and text prompts, perform branch-guided visual token pruning: Pass the input data sequentially through the backbone network and the fine-tuned branch module, and based on the attention map output by the last layer of the branch module, filter out key visual tokens from the visual tokens contained in the input data, and input the key visual tokens into the subsequent network layer in the visual language model located after the branch module for processing.

[0042] In practice, the branch-guided visual token pruning step during the inference phase achieves precise reduction of computational load and retention of key information.

[0043] Traditional methods rely on attention maps in the early layers of the model. The attention mechanism at this stage focuses more on low-level features or local associations and does not fully understand the global semantics. Based on this, pruning can easily lead to the accidental deletion of information that is crucial to the final task.

[0044] This invention innovatively utilizes the attention map of the last layer of a finely tuned branching module as the pruning basis. Since this layer is located after the backbone network and has undergone targeted task-adaptive training, its generated attention map incorporates a deeper level of semantic understanding, enabling it to more accurately identify visual regions highly relevant to the semantics of the text prompts. Sorting and filtering visual tokens based on this high-level semantic attention score essentially performs a task-relevance-based information filtering process. This process can remove a large number of background and redundant visual tokens to compress sequence length, thereby reducing the computational complexity of all subsequent layers, while maximizing the retention of visual cues indispensable for generating accurate descriptions or answers. Therefore, this embodiment effectively alleviates the problem of decreased model accuracy caused by coarse pruning while achieving the primary goal of accelerating inference.

[0045] In some preferred embodiments, the step of selecting key visual tokens from the visual tokens contained in the input data based on the attention map output from the last layer of the branching module includes: parsing the attention map, extracting the attention scores corresponding to all visual tokens in the attention map, and forming a score set; sorting the attention scores in the score set in descending order; and selecting the visual tokens corresponding to the top R highest attention scores from the sorted score set according to a preset retention quantity R, as the selected key visual tokens.

[0046] In practice, during inference, after the input data passes through the backbone network and branch modules, the attention map output by the last lightweight transform layer of the branch module is obtained. This attention map is a data structure that contains the importance weight score of each visual token corresponding to the current context (combined with text prompts), i.e., the attention score. First, the attention map is parsed to extract the attention scores corresponding to all visual tokens in the input image, and these scores are aggregated into a score set.

[0047] Furthermore, a descending sort operation is performed on all attention scores in this score set to obtain a sorted list of visual tokens from most important to least important.

[0048] Furthermore, based on a preset retention number R (e.g., R=64), the top R visual tokens with the highest attention scores are selected from the sorted list.

[0049] Furthermore, these R visual tokens are identified as the key visual tokens required for this inference. In subsequent processing, only these selected key visual tokens will be retained and passed to the subsequent network layers of the visual language model, while the remaining unselected visual tokens will be removed from the processing sequence and will no longer participate in subsequent calculations.

[0050] The high-score-based pruning strategy described in this embodiment achieves precise compression and information fidelity. By sorting in descending order, the system can clearly distinguish the relative ranking of all visual tokens in terms of their contribution to the current task. Selecting based on a preset fixed number R is a stable and controllable method for controlling pruning intensity, ensuring that the number of visual tokens input to subsequent computationally intensive network layers remains constant regardless of the input image content. This is beneficial for reducing prediction inference latency and computational cost. Compared to methods relying on early layer attention or simple thresholds, using the attention scores of the last layer of the branching module after training for sorting and filtering has the advantage of containing deeper semantic understanding. Because the branching module has undergone targeted fine-tuning, its high-level attention mechanism is more focused on visual regions semantically relevant to the text prompts. Therefore, this embodiment essentially retains the most task-relevant visual information after high-level semantic filtering. This filtering method removes a large number of background or redundant visual tokens to improve efficiency while retaining visual cues crucial for generating accurate text responses with the highest probability, thus effectively maintaining model accuracy during the pruning process.

[0051] S4, in the decoding stage of the text sequence generated by the visual language model, self-inferential decoding is performed: using the fine-tuned branch module as a draft model, multiple candidate tokens are generated in an autoregressive manner to form a draft sequence; the draft sequence is input into the visual language model in parallel for single forward propagation verification, and the draft sequence is corrected according to the verification result to generate the final output text.

[0052] In its implementation, this method achieves a breakthrough in efficiency optimization for the computational bottleneck of autoregressive generation in the self-predictive decoding strategy during the decoding phase. Traditional decoding methods are entirely serial, requiring a complete forward propagation to generate each token, leading to a linear increase in latency when generating long texts. This invention creatively reuses a trained lightweight branching module as a drafting model and forms a collaborative workflow with the original visual language model. The drafting model quickly generates a sequence of candidate tokens with a small number of parameters. Subsequently, this candidate sequence is input in parallel into the original model (visual language model) for a one-time forward propagation verification. This design transforms the originally required N serial computations into one N-step rapid drafting by a lightweight model plus one batch verification by a heavyweight model. During the verification process, the original model confirms or corrects the drafting results by analyzing the probability distribution of each position in the candidate sequence and uses a continuous acceptance rule based on a confidence threshold to ensure the reliability of the output text. This iterative cycle of drafting and verification partially shifts the computational burden from multiple calls to an expensive, large model to an efficient lightweight module, while fully utilizing the parallel computing capabilities of the hardware through batch verification. Especially when generating longer text sequences, the number of valid tokens that can be confirmed in a single verification is often greater than one, which makes the number of forward propagations of the large model required to generate each token on average much less than one. This results in a superlinear speedup in the decoding stage, directly overcoming the speed bottleneck of long response generation.

[0053] In some preferred embodiments, the step of correcting the draft sequence based on the verification result to generate the final output text includes: performing a single forward propagation on the parallel input draft sequence through the visual language model to output the probability distribution at each candidate token position in the draft sequence; extracting the confidence score of the corresponding target token from the probability distribution as the verification confidence of each candidate token; sequentially determining whether the verification confidence of each candidate token is greater than or equal to a preset confidence threshold according to the order of the candidate tokens in the draft sequence; accepting the consecutive candidate token sequences with verification confidence scores not lower than the confidence threshold as valid outputs of this iteration; when the first candidate token with a verification confidence score lower than the confidence threshold is encountered, stopping the acceptance of subsequent candidate tokens, and concatenating the valid output sequence already accepted in this decoding iteration to the final output text generated in all previous iterations to form an updated complete output text; and using the updated complete output text as the input condition for the fine-tuned draft model to generate a new draft sequence in the next decoding iteration.

[0054] In practice, after the draft model generates a draft sequence containing L candidate tokens, this sequence is input in parallel into the visual language model (target model). The target model outputs a probability distribution matrix through one forward propagation, where each row corresponds to a position in the draft sequence and each column corresponds to a token in the vocabulary. For each candidate token in the draft sequence, the score of that specific token is extracted from the probability distribution at the corresponding position as its validation confidence. The system then checks these confidences one by one in the order of the draft sequence. It sets a continuous acceptance rule: starting from the first token in the sequence, as long as the validation confidence of a candidate token is greater than or equal to a preset confidence threshold (e.g., 0.6), it is accepted. This acceptance process continues until the first candidate token with a validation confidence lower than the threshold is encountered. At this point, the acceptance process stops immediately, and all the previously consecutively accepted tokens constitute the valid output sequence of this decoding iteration. This valid output sequence is appended to the end of the final output text generated in all previous iterations, thus forming an updated, longer, complete output text. This updated, complete output text will be fed back to the drafting model as the input context for the next decoding iteration to generate new draft sequences, and so on, until a complete answer is generated or the length limit is reached.

[0055] The serialization verification and iterative concatenation mechanism described in this embodiment is the core of achieving efficient and reliable text generation, and its technical effect is reflected in the synergistic improvement of generation speed and text quality. This mechanism, through the "prefix verification" principle, quickly adopts the reliable parts (high-confidence prefixes) of the draft sequence generated in one go as output, avoiding the inefficient operation of the target model performing independent serial verification on each token. In each iteration, the number of tokens that the target model can verify and convert into the final output in a single forward propagation increases from one in traditional methods to multiple (i.e., the number of tokens that pass verification consecutively), which directly improves the output efficiency of a single iteration step. Simultaneously, the setting of concatenating the valid outputs received in each iteration in real time and feeding them back as input for the next round ensures the strict autoregressive characteristics of the generation process, guaranteeing contextual coherence and semantic consistency of the final text. This design cleverly balances the speed advantage brought by parallel verification with the historical dependency necessary for autoregressive generation. By controlling the acceptance threshold with a confidence level, a technique is provided to adjust the trade-off between generation speed and text accuracy: a higher threshold leads to more rigorous verification, and fewer tokens may be accepted each time, but with greater accuracy; a lower threshold may accept longer prefixes in pursuit of speed. The entire process thus ensures, at the algorithmic level, that reliable quality of the generated text is maintained while accelerating decoding.

[0056] In some preferred embodiments, the preset confidence threshold is configured as follows: during the validation phase before model deployment, a validation dataset containing multiple sets of image-text pairs is prepared; for each sample in the validation dataset, the self-predictive decoding process is run using multiple different candidate confidence thresholds; for each candidate confidence threshold, the average text generation speed is recorded when it runs on the validation dataset, and the automated evaluation score of the generated text relative to the standard answer is calculated; among the multiple candidate confidence thresholds, a threshold is selected such that the weighted comprehensive performance index of the average text generation speed and the automated evaluation score reaches the optimal level.

[0057] In practice, before deploying the accelerated model to real-world applications, an offline validation phase is required to determine the optimal threshold. First, a validation dataset, independent of the training set, is prepared. This dataset contains multiple sets of image-text pairs with standard answers, representing the target application scenario. Further, a set of candidate confidence thresholds is preset for this validation dataset, such as [0.5, 0.55, 0.6, 0.65, 0.7, 0.75]. For each sample in the validation dataset, a complete self-predictive decoding process (including draft generation, target model validation, and acceptance judgment based on the threshold) is run sequentially using each candidate threshold. After running the process on the entire validation dataset for each candidate threshold, two key performance indicators are collected: average text generation speed (e.g., number of tokens generated per second) and the automated evaluation score of the generated text relative to the standard answer (e.g., using metrics such as BLEU, ROUGE, or CIDER). Furthermore, a weighted composite performance index P is defined, for example, P = α * (standardized speed score) + (1-α) * (standardized evaluation score), where α is a pre-defined weight (0 < α < 1). Further, among all candidate confidence thresholds for testing, the threshold that maximizes the weighted composite performance index P is selected as the pre-set confidence threshold used when finally deploying the model.

[0058] This embodiment introduces a threshold optimization configuration method based on validation set performance evaluation, significantly improving the adaptability and overall performance of the acceleration method in different application scenarios. Its technical effect lies in transforming the selection of the key hyperparameter, the confidence threshold, from empirical guesswork into a quantifiable and reproducible optimization process. By conducting gridded testing on a representative validation dataset, this method can objectively measure the specific impact of different thresholds on two core and often conflicting objectives—generation speed and text quality. Defining a weighted comprehensive performance index allows users to specify the optimization direction based on actual needs (e.g., applications biased towards real-time performance or accuracy). By selecting the threshold that optimizes this comprehensive index, this method ensures that the final deployed acceleration system achieves its best performance at the specific performance balance point that the user cares about. This avoids the problem of poor performance in specific scenarios that may result from using a fixed threshold, making the entire visual language model acceleration solution more robust and adaptable to different scenarios, thereby enhancing its practical value and technical reliability in diverse real-world tasks.

[0059] In some preferred embodiments, during the decoding stage, the method further includes performing visual token rollback: in the network layers of the visual language model, a network layer located after the branch module and with a layer number greater than a first preset value is selected as a rollback point; when the key visual token propagates to the rollback point during inference, all the key visual tokens are removed from the current processing sequence; after removing the key visual tokens, the subsequent autoregressive text generation process continues only based on the remaining text token sequence.

[0060] In practice, during the model building or configuration phase, a specific layer in the visual language model's network is selected as a pullback point. This pullback point must meet two conditions: first, it must be located after a branch module; second, its layer index must be greater than a first preset value (for example, this value can be set to 2 / 3 of the total number of layers to ensure a sufficiently deep layer). During inference execution, when the selected key visual tokens reach this preset pullback point during forward propagation, the system performs a pullback operation: removing all remaining visual tokens in the current processing sequence, retaining only the text token sequence. Specifically, in the Transformer architecture, this means deleting the feature vectors corresponding to the visual tokens in the sequence from the input tensor. Afterward, all subsequent layers of the model, including all remaining decoding and generation steps, will continue autoregressive text generation based solely on the text token sequence and the accumulated text context information; visual tokens will no longer participate in any subsequent computations.

[0061] The visual token rollback strategy defined in this embodiment achieves a further reduction in computational complexity during the later stages of decoding. Its principle is based on utilizing the diminishing utility of visual information during generation. Visual information is crucial in the initial stages of text response generation by the visual language model. However, as generation progresses, especially after the response has sufficiently described or referenced image content, subsequent text generation may rely more heavily on established text context and prior knowledge of the language model, reducing the direct dependence on the original high-dimensional visual features. By actively removing all visual tokens at deeper levels of the model (rollback points), this step directly and permanently reduces the sequence length of key-value pairs in each subsequent decoding layer, thereby significantly reducing the mathematical computation of core operations such as the attention mechanism. Since the decoding stage is serially iterative and involves deep layers, the computational savings at each layer are amplified multiple times throughout the long sequence generation process, resulting in considerable additional speedup benefits. Meanwhile, since the pullback point is carefully set after a sufficiently deep network layer, the model should have already extracted and fused the necessary semantic information from the visual features into the text context. Therefore, under ideal circumstances, this operation will not have a significant negative impact on the semantic accuracy and coherence of the subsequently generated text, achieving an additional efficiency improvement without losing quality.

[0062] This invention proposes a method to accelerate a visual language model, comprising: freezing the first K layers of a pre-trained visual language model as a backbone network, and appending a branch module consisting of T lightweight transformation layers after the backbone network, where K is a positive integer and T is a positive integer less than the difference between the total number of layers in the visual language model and K; fine-tuning the branch module using a training dataset, wherein the fine-tuning process only updates the parameters contained in the branch module, while keeping the parameters of all layers in the visual language model except the branch module frozen; and performing branch-guided visual token pruning for input data containing images and text prompts: sequentially processing the input data through... The backbone network and fine-tuned branch modules, based on the attention map output from the last layer of the branch modules, filter key visual tokens from the visual tokens contained in the input data, and input these key visual tokens into subsequent network layers in the visual language model after the branch modules for processing. During the decoding stage of the text sequence generated by the visual language model, auto-inference decoding is performed: using the fine-tuned branch modules as a draft model, multiple candidate tokens are generated in an autoregressive manner to form a draft sequence; the draft sequence is input in parallel into the visual language model for a single forward propagation verification, and the draft sequence is corrected based on the verification result to generate the final output text. This invention constructs a parameter-efficient fine-tuning framework by freezing the first K layers of the base model as the backbone network and adding trainable branch modules. During inference, visual token pruning is performed using the attention map of the last layer of the branch modules, accurately retaining key visual tokens based on deep semantic information, reducing computational load while maintaining model accuracy. During the decoding stage, the trained branch modules are used as a lightweight draft model to generate candidate sequences, which are then verified and corrected in parallel by the original model. The serial generation is transformed into batch processing through a self-inferential decoding mechanism, thereby significantly accelerating the generation of long texts.

[0063] This invention can be applied to fields such as finance, healthcare, insurance, and banking. Application examples are as follows: Financial Application: In the financial sector, this technical solution can be applied to automated financial document analysis and customer service scenarios. Specifically, financial institutions need to process a large number of complex financial reports, audit documents, or loan application forms daily, containing charts, seals, handwritten annotations, and structured tables. Traditional processing methods rely on manual review or single-modal models, which are inefficient and prone to errors. After applying this invention, the system can input the financial document images uploaded by the customer along with text queries (such as "Please extract the second quarter net profit data and analyze the trend") into the accelerated visual language model. The model first quickly extracts the general visual features of the document images through a frozen backbone network, and then performs in-depth processing through fine-tuned branch modules. The attention map generated in the last layer of the branch modules can accurately focus on data areas in the document that are semantically related to keywords such as "net profit" and "second quarter," such as specific table cells or line chart inflection points. Based on this, token pruning is performed to filter out a large amount of irrelevant background textures, decorative borders, and other redundant visual information, retaining only the key data area features. During the text decoding stage of generating the analysis report, the branch module, acting as a lightweight drafting model, quickly generates a text draft containing preliminary data descriptions and trend judgments. This draft is then validated and corrected in parallel by the original model, ultimately outputting an accurate and structured analysis summary. This significantly improves the efficiency and accuracy of financial institutions in processing unstructured financial information, providing efficient technical support for risk assessment, compliance review, and robo-advisory.

[0064] In the medical field, this technical solution empowers intelligent medical image interpretation and report generation systems. Specifically, in clinical practice, doctors need to combine medical images (such as X-rays and MRI slices) with patient medical history text descriptions to make a comprehensive diagnosis and generate detailed reports. This process is time-consuming and requires a high level of experience. With this invention, the system can input medical image sequences along with patient complaints or examination requests in text form. The accelerated model utilizes its backbone network to stably extract basic biological features from the images, while fine-tuned branch modules for the medical field further learn deep semantic associations related to specific symptoms in the images. During inference, the attention maps of the branch modules guide the system to accurately locate key region tokens highly correlated with the textual descriptions of symptoms (such as "pulmonary nodules" or "abnormal signal lesions") among numerous image slices, filtering out large areas of redundant information from normal tissues. Subsequently, when generating the diagnostic report text, the branch modules, acting as a drafting model, can quickly draft paragraphs containing preliminary findings and descriptive language, while the original model uses parallel validation to ensure the accuracy of medical terminology and the rigor of diagnostic inferences, rapidly generating a structurally complete preliminary diagnostic report draft for doctor review. This application can significantly shorten image interpretation time, help doctors improve work efficiency, and facilitate the efficient and standardized transfer of diagnostic experience in medical institutions at different levels.

[0065] In the insurance sector, this technical solution can optimize the automated remote damage assessment and claims process in scenarios such as auto insurance and property insurance. Specifically, currently, claims adjusters need to inspect or review photos / videos of the damage uploaded by customers on-site, and then conduct loss assessments and report writing based on the policy terms, a cumbersome process. After applying this invention, customers can upload multi-angle photos of the accident scene and a brief text description (such as "left front door scratched, headlight broken") via mobile devices. The accelerated model first processes the images and text. The branch module, through its fine-tuned high-level semantic understanding of vehicle parts and damage types, guides the attention mechanism to focus on key areas such as "door" and "headlight" in the text description, and removes irrelevant visual tokens such as the surrounding environment and license plates from the images. This allows the model to efficiently extract the most relevant visual evidence for loss determination. Subsequently, when generating damage assessment reports or claims opinions, the branch module, as a draft model, quickly generates preliminary text descriptions of the damage location and extent. The original model then verifies its consistency with the policy terms and damage images in parallel, and corrects and generates claims suggestion text that conforms to business specifications. This application enables rapid remote and automated damage assessment, significantly improving claims processing speed and customer satisfaction. At the same time, it ensures the objectivity and accuracy of assessment results through precise visual information filtering.

[0066] In the banking sector, this technical solution can be applied to enhance intelligent assistance for remote identity verification and complex business processing. Specifically, in high-risk transactions such as remote account opening and large-amount transfers, banks need to strictly verify the consistency of customer ID documents, facial information, and completed forms. Using this invention, the system can simultaneously receive images of the front and back of the customer's ID card, real-time facial capture images, and the business application text filled out by the customer. The accelerated model utilizes its visual language understanding capabilities, first using a branch-guided pruning strategy to accurately locate and extract visual information from key fields such as name, ID number, and photo from the document image, while filtering out interfering elements such as anti-counterfeiting backgrounds. Then, the model cross-validates the extracted key visual information with the application text (e.g., comparing the name on the document with the name entered). When a verification result report or risk warning needs to be generated, the self-inferential decoding mechanism begins: a lightweight branching module drafts preliminary conclusions, the original model performs parallel verification, and generates the final standard-format text output, detailing whether the verification passed or failed and specific anomalies. This application automates the process that originally relied on manual visual comparison and judgment. While ensuring financial security and compliance, it greatly improves the processing efficiency and user experience of remote business. At the same time, it reduces the misjudgment rate of document images under non-ideal shooting conditions such as blurry or tilted images through a high-level semantic attention mechanism.

[0067] Corresponding to the above-described visual language model acceleration method, this invention also provides a visual language model acceleration device. This device includes units for executing the aforementioned visual language model acceleration method and can be configured in a terminal or server. Specifically, the device includes: an appending unit, used to freeze the first K layers of a pre-trained visual language model as a backbone network, and append a branch module consisting of T lightweight transformation layers after the backbone network, where K is a positive integer and T is a positive integer less than the difference between the total number of layers in the visual language model and K; a fine-tuning unit, used to fine-tune the branch module using a training dataset, wherein the fine-tuning process only updates the parameters contained in the branch module and keeps the parameters of all layers in the visual language model except the branch module frozen; and a pruning unit, used to perform branch-guided visual token pruning for input data containing images and text prompts: [The text abruptly ends here, so the translation stops here as well.] The system sequentially passes through the backbone network and the finely tuned branch modules. Based on the attention map output by the last layer of the branch modules, key visual tokens are selected from the visual tokens contained in the input data. These key visual tokens are then input into subsequent network layers in the visual language model after the branch modules for processing. The decoding unit performs auto-speculation decoding during the decoding stage of the text sequence generated by the visual language model. It uses the finely tuned branch modules as a draft model to generate multiple candidate tokens in an autoregressive manner to form a draft sequence. The draft sequence is then input into the visual language model in parallel for a single forward propagation verification. Based on the verification results, the draft sequence is corrected to generate the final output text.

[0068] In some preferred embodiments, the step of adding a branch module consisting of T lightweight transformation layers after the backbone network includes: extracting a continuous network of T layers from the (K+1)th layer to the (K+T)th layer from the network structure of the pre-trained visual language model; copying the weight parameters and bias parameters of each layer in the extracted T layers to the T lightweight transformation layers of the branch module; and connecting the branch module with the parameters copied to the output of the backbone network to form a complete inference architecture including the backbone network, the branch module, and subsequent layers of the visual language model.

[0069] In some preferred embodiments, the fine-tuning of the branch modules using the training dataset includes: inputting samples from the training dataset into the full inference architecture for forward propagation; calculating the autoregressive loss of the text sequence generated by the full inference architecture for the input samples, wherein the loss is the cross-entropy loss between the model's predicted next token and the true label; calculating only the gradient of the parameters in the branch modules during backpropagation; and updating the parameters of the branch modules according to the gradient using a preset optimization algorithm, while keeping the parameters of the backbone network and subsequent layers of the visual language model unchanged.

[0070] In some preferred embodiments, the step of selecting key visual tokens from the visual tokens contained in the input data based on the attention map output from the last layer of the branching module includes: parsing the attention map, extracting the attention scores corresponding to all visual tokens in the attention map, and forming a score set; sorting the attention scores in the score set in descending order; and selecting the visual tokens corresponding to the top R highest attention scores from the sorted score set according to a preset retention quantity R, as the selected key visual tokens.

[0071] In some preferred embodiments, the step of correcting the draft sequence based on the verification result to generate the final output text includes: performing a single forward propagation on the parallel input draft sequence through the visual language model to output the probability distribution at each candidate token position in the draft sequence; extracting the confidence score of the corresponding target token from the probability distribution as the verification confidence of each candidate token; sequentially determining whether the verification confidence of each candidate token is greater than or equal to a preset confidence threshold according to the order of the candidate tokens in the draft sequence; accepting the consecutive candidate token sequences with verification confidence scores not lower than the confidence threshold as valid outputs of this iteration; when the first candidate token with a verification confidence score lower than the confidence threshold is encountered, stopping the acceptance of subsequent candidate tokens, and concatenating the valid output sequence already accepted in this decoding iteration to the final output text generated in all previous iterations to form an updated complete output text; and using the updated complete output text as the input condition for the fine-tuned draft model to generate a new draft sequence in the next decoding iteration.

[0072] In some preferred embodiments, the preset confidence threshold is configured as follows: during the validation phase before model deployment, a validation dataset containing multiple sets of image-text pairs is prepared; for each sample in the validation dataset, the self-predictive decoding process is run using multiple different candidate confidence thresholds; for each candidate confidence threshold, the average text generation speed is recorded when it runs on the validation dataset, and the automated evaluation score of the generated text relative to the standard answer is calculated; among the multiple candidate confidence thresholds, a threshold is selected such that the weighted comprehensive performance index of the average text generation speed and the automated evaluation score reaches the optimal level.

[0073] In some preferred embodiments, during the decoding stage, the method further includes performing visual token rollback: in the network layers of the visual language model, a network layer located after the branch module and with a layer number greater than a first preset value is selected as a rollback point; when the key visual token propagates to the rollback point during inference, all the key visual tokens are removed from the current processing sequence; after removing the key visual tokens, the subsequent autoregressive text generation process continues only based on the remaining text token sequence.

[0074] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned visual language model acceleration device and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0075] The aforementioned visual language model acceleration device can be implemented as a computer program that can run on the computer device shown in Figure 2.

[0076] Please refer to Figure 2, which is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0077] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0078] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a visual language model acceleration method.

[0079] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0080] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a visual language model acceleration method.

[0081] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0082] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of a visual language model acceleration method provided in any of the above embodiments.

[0083] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0084] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0085] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of a visual language model acceleration method provided in any of the above embodiments.

[0086] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0087] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0088] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0089] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0091] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0092] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0093] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for accelerating visual language models, characterized in that, include: In a pre-trained visual language model, the first K layers of the model are frozen as the backbone, and a branch module consisting of T lightweight transform layers is appended after the backbone, where K is a positive integer and T is a positive integer less than the difference between the total number of layers in the visual language model and K. The branch module is fine-tuned using the training dataset, wherein the fine-tuning process only updates the parameters contained in the branch module, while keeping the parameters of all layers in the visual language model frozen except for the branch module. For input data containing images and text prompts, branch-guided visual token pruning is performed: the input data is sequentially passed through the backbone and the fine-tuned branch module. The branch module, based on the attention map output from the last layer of the branch module, filters key visual tokens from the visual tokens contained in the input data, and inputs the key visual tokens into the subsequent network layer in the visual language model after the branch module for processing; in the decoding stage of the visual language model generating the text sequence, auto-inference decoding is performed: using the fine-tuned branch module as a draft model, multiple candidate tokens are generated in an autoregressive manner to form a draft sequence; the draft sequence is input in parallel into the visual language model for a single forward propagation verification, and the draft sequence is corrected according to the verification result to generate the final output text.

2. The visual language model acceleration method according to claim 1, characterized in that, The step of adding a branch module consisting of T lightweight transformation layers after the backbone network includes: extracting T consecutive network layers from layer K+1 to layer K+T from the network structure of the pre-trained visual language model; copying the weight parameters and bias parameters of each layer in the extracted T network layers to the T lightweight transformation layers of the branch module; and connecting the branch module with the parameters copied to the output of the backbone network to form a complete inference architecture including the backbone network, the branch module, and subsequent layers of the visual language model.

3. The visual language model acceleration method according to claim 2, characterized in that, The fine-tuning of the branch modules using the training dataset includes: inputting samples from the training dataset into the complete inference architecture for forward propagation; calculating the autoregressive loss of the text sequence generated by the complete inference architecture for the input samples, wherein the loss is the cross-entropy loss between the model's predicted next token and the true label; calculating only the gradient of the parameters in the branch modules during backpropagation; and updating the parameters of the branch modules according to the gradient using a preset optimization algorithm, while keeping the parameters of the backbone network and subsequent layers of the visual language model unchanged.

4. The visual language model acceleration method according to claim 1, characterized in that, The step of selecting key visual tokens from the visual tokens contained in the input data based on the attention map output from the last layer of the branching module includes: parsing the attention map, extracting the attention scores corresponding to all visual tokens in the attention map, and forming a score set; sorting the attention scores in the score set in descending order; and selecting the visual tokens corresponding to the top R highest attention scores from the sorted score set according to a preset retention quantity R, as the selected key visual tokens.

5. The visual language model acceleration method according to claim 1, characterized in that, The step of correcting the draft sequence based on the verification results to generate the final output text includes: performing a single forward propagation on the parallel input draft sequence through the visual language model to output the probability distribution at each candidate token position in the draft sequence; extracting the confidence score of the corresponding target token from the probability distribution as the verification confidence of each candidate token; sequentially determining whether the verification confidence of each candidate token is greater than or equal to a preset confidence threshold according to the order of the candidate tokens in the draft sequence; accepting the consecutive candidate token sequences with verification confidence scores not lower than the confidence threshold as valid outputs of this iteration; when encountering the first candidate token with a verification confidence score lower than the confidence threshold, stopping the acceptance of subsequent candidate tokens, and concatenating the valid output sequence already accepted in this decoding iteration to the final output text generated in all previous iterations to form an updated complete output text; and using the updated complete output text as the input condition for the fine-tuned draft model to generate a new draft sequence in the next decoding iteration.

6. The visual language model acceleration method according to claim 5, characterized in that, The preset confidence threshold is configured as follows: during the validation phase before model deployment, a validation dataset containing multiple sets of image-text pairs is prepared; for each sample in the validation dataset, the self-inferential decoding process is run using multiple different candidate confidence thresholds. For each candidate confidence threshold, the average text generation speed is recorded when it is run on the validation dataset, and the automated evaluation score of the generated text relative to the standard answer is calculated; among the multiple candidate confidence thresholds, a threshold is selected such that the weighted comprehensive performance index of the average text generation speed and the automated evaluation score is optimal.

7. The visual language model acceleration method according to claim 1, characterized in that, In the decoding stage, the method further includes performing visual token rollback: in the network layers of the visual language model, a network layer located after the branch module and with a layer number greater than a first preset value is selected as the rollback point; when the key visual token propagates to the rollback point during inference, all the key visual tokens are removed from the current processing sequence; after removing the key visual tokens, the subsequent autoregressive text generation process continues only based on the remaining text token sequence.

8. A visual language model acceleration device, characterized in that, Includes a unit for performing the method as described in any one of claims 1-7.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images