Model decoding method and device based on visual token hiding, equipment and medium

Abstract

The application relates to the technical field of intelligent decision-making, and discloses a model decoding method and device based on visual token hiding, equipment and a medium. The method comprises the following steps: obtaining a target input image and a target text token, performing visual analysis by using a visual encoder to obtain visual embedding; performing text feature extraction on the text token by using a text encoder to obtain text embedding; performing cross-modal fusion on the visual embedding and the text embedding by using a decoder to generate visual text features; performing linear transformation on the visual text features and fusing the linearly transformed visual text features with the text embedding to obtain visual-injected text embedding; performing self-recurrence analysis on the visual-injected text embedding by using a drafter to obtain corresponding candidate tokens; and verifying the candidate tokens according to the output probability distribution of a target visual language model, and taking the verified tokens as decoding output results. The application can be applied to a business system platform such as financial technology and medical health, and can improve the efficiency of model decoding.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology, and in particular to a model decoding method, apparatus, device, and medium based on visual token hiding. Background Technology

[0002] With the rapid development of multimodal artificial intelligence technology, visual-language models have shown great potential in image understanding and text generation fusion tasks, and have been widely applied in industries such as automatic generation of medical image reports, intelligent recognition of financial documents, and scene description for autonomous driving. However, speculative decoding, as an effective inference acceleration technique, rapidly generates candidate tokens through a lightweight drafter, which are then validated in parallel by the original target model, achieving significant speedup in large-scale pure text language models.

[0003] In the healthcare field, doctors typically need to analyze chest X-rays, CT images, or pathology slides in real time and quickly generate structured diagnostic reports. However, existing vision-language model-based assisted diagnostic systems require generating a large number of visual tokens when processing high-resolution medical images. The drafter must also process these visual token sequences, which not only causes the computational load of the drafter to be proportional to the number of visual tokens, significantly reducing the speedup effect, but also, because the semantic spaces of visual tokens and text tokens are fundamentally different, directly inputting visual tokens into the drafter will cause contextual signal distortion, resulting in semantic inconsistencies in the generated diagnostic descriptions, such as misplaced lesion locations and misjudged features.

[0004] In the fintech sector, banks, insurance companies, and other institutions need to automate the extraction of information from massive amounts of images of bills, contracts, and invoices. However, existing vision-language model-based recognition systems, when processing invoice images, may generate hundreds of visual tokens from a single high-resolution invoice image, each token representing a local region. During the generation of candidate text tokens, the drafter needs to repeatedly interact with these visual tokens, resulting in wasted computational resources. Furthermore, semantic misalignment between visual and text features often leads to errors in number recognition or the omission of decimal parts, directly impacting business accuracy.

[0005] Most existing speculative decoding techniques directly use plain text frameworks without optimizing for the characteristics of visual tokens. The drafter still needs to process a large number of visual tokens directly, which does not effectively reduce the computational burden. Furthermore, it cannot identify and remove redundant visual tokens during the inference process, causing unnecessary computational burden and resulting in low efficiency when decoding the model. Summary of the Invention

[0006] This invention provides a model decoding method, apparatus, device, and medium based on visual token hiding to solve the problem of low efficiency and accuracy in model decoding.

[0007] Firstly, a model decoding method based on visual token hiding is provided, the method comprising: Obtain the target input image and the target text token, and use the visual encoder in the preset visual language model to perform visual analysis on the target input image to obtain the visual embedding; The text encoder in the visual language model is used to extract text features from the target text token to obtain a text embedding. The visual language model's decoder performs cross-modal fusion of the visual embedding and the text embedding to generate visual text features; The visual text features are linearly transformed, and the linearly transformed visual text features are fused with the text embedding to obtain the visually injected text embedding. The drafter in the visual language model is used to perform autoregressive analysis on the visually injected text embedding to obtain candidate tokens corresponding to the visually injected text embedding. The output probability of the candidate token is verified based on the output probability distribution of the visual language model, and the candidate token that passes the verification is used as the decoding output of the visual language model.

[0008] Secondly, a model decoding device based on visual token hiding is provided, the device comprising: The visual analysis module is used to acquire the target input image and the target text token, and to perform visual analysis on the target input image using the visual encoder in the preset visual language model to obtain the visual embedding. The text feature extraction module is used to extract text features from the target text token using the text encoder in the visual language model to obtain the text embedding. The cross-modal fusion module is used to perform cross-modal fusion of the visual embedding and the text embedding through the decoder of the visual language model to generate visual text features; The feature fusion module is used to perform a linear transformation on the visual text features, and then fuse the linearly transformed visual text features with the text embedding to obtain a visually injected text embedding. The autoregressive analysis module is used to perform autoregressive analysis on the visually injected text embedding using the drafter in the visual language model to obtain the candidate tokens corresponding to the visually injected text embedding. The probability verification module is used to verify the output probability of the candidate tokens according to the output probability distribution of the visual language model, and to use the candidate tokens that pass the verification as the decoding output of the visual language model.

[0009] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described model decoding method based on visual token hiding.

[0010] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described model decoding method based on visual token hiding.

[0011] In this invention, by injecting visual information into text embeddings, the drafter is completely free from processing the original visual tokens, significantly reducing the computational burden. By utilizing the cross-modal capabilities of the target visual-language model itself to generate visual-injected text embeddings, the problem of inconsistency between visual and textual semantics is solved. Through a step-dependent bias correction residual mechanism, the drafter's prediction drift is stabilized in the absence of real-time feedback from the target model. Thus, while ensuring that the output quality is consistent with the target model, efficient speculative decoding acceleration is achieved, which can solve the problem of low efficiency when performing model decoding. It can be widely applied to scenarios such as real-time generation of medical image reports and high-speed automatic recognition of financial documents, significantly improving inference throughput and reducing deployment costs. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for a model decoding method based on visual token hiding in one embodiment of the present invention; Figure 2 This is a flowchart illustrating a model decoding method based on visual token hiding in one embodiment of the present invention; Figure 3 This is a schematic diagram of a model decoding device based on visual token hiding in one embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0014] It should be noted that in the technical solutions disclosed in this invention, the acquisition of user information (personal image data (e.g., facial videos or pictures, facial feature videos or pictures, etc.) and personal privacy information (e.g., name, ID number, occupation, address, etc.)) is all completed with the user's knowledge and consent, and the acquisition of the relevant user information is legal and compliant.

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] The model decoding method based on visual token hiding provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0017] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a model decoding method based on visual token hiding provided in an embodiment of the present invention is shown. The method includes the following steps: S1. Obtain the target input image and the target text token, and use the visual encoder in the preset visual language model to perform visual analysis on the target input image to obtain the visual embedding.

[0018] In this embodiment of the invention, the target input image refers to the original image to be visually understood, such as a chest X-ray in medical imaging, and the target text token refers to the discretized representation of the initial text instruction or prompt associated with the image, such as a word sequence obtained after word segmentation of a descriptive text about a suspected lesion area.

[0019] In detail, the target input image and target text token can be obtained from a pre-stored storage area through computer statements with data scraping capabilities (such as Java statements, Python statements, etc.), where the storage area includes, but is not limited to, databases and blockchains.

[0020] In this embodiment of the invention, the visual embedding refers to converting image content into a continuous vector sequence that can be processed by a computer, with each vector corresponding to a local region in the image.

[0021] In this embodiment of the invention, the step of performing visual analysis on the target input image using a visual encoder in a preset visual language model to obtain a visual embedding includes: The target input image is divided into multiple non-overlapping image blocks, and the pixel values ​​of each image block are flattened into a one-dimensional pixel sequence; Each one-dimensional pixel sequence is mapped to an initial visual feature vector through the linear projection layer of the visual encoder, and the initial visual feature vector is encoded into an initial visual embedding sequence. Position embedding is performed on each feature vector in the initial visual embedding sequence to obtain a visual embedding sequence. The global attention weights between each image patch in the visual embedding sequence are analyzed using the multi-head self-attention layer of the visual encoder. The global attention weights are weighted and summed with the value vectors of all image blocks to obtain the output feature sequence of the multi-head self-attention layer. The output feature sequence is then linearly transformed to obtain the visual embedding.

[0022] In detail, if the target input image is an RGB color image with a resolution of 224×224 pixels, the visual encoder first divides the image into non-overlapping grids according to a fixed block size (e.g., 16×16 pixels). Each image block is originally a three-dimensional tensor (16, 16, 3) containing 3 color channels. Through a reshape operation, this tensor is flattened into a one-dimensional pixel sequence. For example, 196 image blocks correspond to 196 one-dimensional pixel sequences of length 768, forming an initial data matrix with shape [196, 768]. The linear projection layer is a trainable fully connected layer with an input dimension of 768 (or 256, depending on the number of image channels) and an output dimension of a preset hidden dimension. For the one-dimensional pixel sequence p_i (shape

[768] ) of the i-th image block, the linear projection is calculated as follows: ,in The weight matrix is ​​[768, 768]. This is a bias vector of length 768. Output This is the initial visual feature vector (length 768). After performing the same operation on all 196 image patches, the initial visual embedding sequence F=[f_1, f_2, ..., f_196] is obtained, with a shape of [196, 768].

[0023] Specifically, a learnable location embedding vector is assigned to each image patch. (Length 768), its index corresponds to the row and column number of the image patch in the grid. and Element-wise summation yields the visual embedding sequence V = [v_1, ..., v_196], with its shape remaining unchanged. This visual embedding sequence is then input into a multi-head self-attention layer. For each attention head h, the query matrix Q_h, key matrix K_h, and value matrix V_h are calculated, along with the attention weight matrix for each attention head h. , d_k=64, For normalization function, It is the transpose symbol. Let be the square root function, and the output feature sequence of each attention head be... ,Right now The outputs of all attention heads are concatenated along the feature dimension to obtain the concatenated feature O_concat, which is then subjected to a linear transformation to obtain the visual embedding. , To output the weight matrix of the projection layer, As the bias term for the output projection layer, the visual embedding, as the output of the visual encoder, will be sent to the decoder for cross-modal fusion.

[0024] For example, in a medical scenario, a chest X-ray (224×224 resolution, grayscale) is divided into 196 image blocks of 14×14 pixels. Each block is flattened into a 256-dimensional pixel sequence, mapped to a 768-dimensional initial visual feature vector through a linear projection layer. After superimposing learnable positional embeddings, a 12-head self-attention layer calculates the global attention weights between the image blocks. In the output visual embedding sequence, the image block of the lung nodule region can establish strong correlation weights with the surrounding lung texture region, thereby generating a high-quality visual representation rich in spatial context. In a fintech scenario, a VAT invoice image (also 224×224) is divided into 196 image blocks. Each invoice block may contain local information such as the amount, product name, or company seal. The visual encoder uses multi-head self-attention to enable the image block of the total amount region to automatically focus on the numerical region on the right side of the invoice. Even if the image block of the numerical region is far from the text region, the visual embedding can accurately encode this cross-regional correspondence, providing accurate visual features for subsequent recognition of the invoice total amount.

[0025] Furthermore, through multi-head self-attention and position embedding, the visual encoder is able to capture the dependencies between pixels at any distance in the image, providing a rich and global visual representation for subsequent cross-modal alignment, and solving the problem of detail loss caused by the limited receptive field of traditional convolutional networks.

[0026] S2. Use the text encoder in the visual language model to extract text features from the target text token to obtain the text embedding.

[0027] In this embodiment of the invention, the text embedding refers to the feature representation obtained by mapping discrete tokens into a continuous vector sequence and then encoding it with context. Each vector not only contains the meaning of the current word, but also integrates the semantic information of the preceding and following words.

[0028] In this embodiment of the invention, the step of extracting text features from the target text token using the text encoder in the visual language model to obtain the text embedding includes: Each discrete token in the target text tokens is mapped to an initial text feature vector, and the initial text feature vector is encoded into an initial text embedding sequence; Positional encodings are superimposed on each position in the initial text embedding sequence to obtain the text embedding sequence; The text embedding sequence is encoded and transformed using the encoding layer of the text encoder to generate a text embedding.

[0029] In detail, the target text tokens are integer sequences obtained after segmenting natural language sentences. A word embedding matrix E_word is maintained by a text encoder, where the shape of the word embedding matrix is ​​[vocabulary size, 768], and the vocabulary size is typically 50257. For each discrete token id, the integer index of each token is used as the row number to extract E_word[id] from the word embedding matrix, where E_word[id] is all the elements corresponding to the id-th discrete token (indexed as id) in the word embedding matrix E_word, thus obtaining a 768-dimensional initial text feature vector. The vectors of all discrete tokens are stacked in order to obtain the initial text embedding sequence T_init.

[0030] Specifically, since the self-attention mechanism itself does not distinguish sequence order, positional information must be explicitly added. A positional embedding matrix of maximum length is predefined. The initial embedding of the i-th position in the sequence is added to the i-th row vector of this matrix to generate a text embedding sequence carrying order information. The text embedding sequence with stacked positional encoding (shape [number of text tokens, hidden dimension]) is sequentially input into multiple stacked Transformer encoding layers in the text encoder. Each layer performs layer normalization, multi-head self-attention, which allows each token to pay attention to all tokens in the sequence, capturing bidirectional contextual dependencies, residual connections, feedforward networks, expanding and then compressing the feature dimension to enhance non-linear expressive power, and performing a second layer normalization. After all encoding layers, the initial embedding of each token has fused the semantic information on its left and right sides, and the final output is a context-aware feature sequence with the same dimension as the input. This sequence is the final text embedding. The text encoder contains 12 identical Transformer encoding layers, and all trainable parameters are obtained through end-to-end training. After 12 layers of encoding, the text embedding is obtained.

[0031] Furthermore, the text encoder, through multi-layer Transformer encoding and positional encoding, is able to understand word order, context, and negation relationships in natural language instructions, and output high-quality semantic vectors, laying the foundation for accurate visual text alignment.

[0032] S3. The visual embedding and the text embedding are fused across modally using the decoder of the visual language model to generate visual text features.

[0033] In this embodiment of the invention, the decoder is a Transformer decoder used to gradually fuse visual information and textual information to generate intermediate features that are sensitive to both; the visual-textual features refer to the features in the decoder output corresponding to the position of the text token, and each text token already carries semantic information of the relevant visual region.

[0034] In this embodiment of the invention, the step of performing cross-modal fusion of the visual embedding and the text embedding through the decoder of the visual language model to obtain visual text features includes: The visual embedding and the text embedding are concatenated according to the sequence dimension to generate a joint embedding sequence. The text embedding in the joint embedding vector is used as the query vector, and the visual embedding in the joint embedding vector is used as the key vector and value vector. The attention weight of each text token for all visual tokens is calculated based on the query vector, the key vector, and the value vector. The attention weight is then used to perform a weighted summation of the value vector to obtain the preliminary visual text intermediate features corresponding to the position of each text token. The intermediate features of the preliminary visual text are processed by the decoding layer of the decoder to obtain a deep visual text feature sequence. The deep visual text feature sequence is segmented to obtain the feature subsequence corresponding to the text embedding, and the feature subsequence is used as the visual text feature.

[0035] In detail, if the visual embedding sequence length is N_v and the text embedding sequence length is N_t, after concatenation along the sequence dimensions, the total length of the joint embedding sequence is N_v + N_t, which serves as the initial input to the decoder. In the cross-attention sublayer of the decoder, the query vector of this sublayer represents the part of the joint sequence belonging to the text embedding, while the key and value vectors represent the part belonging to the visual embedding. For example, the query matrix has a shape of [N_t, 768], and the key and value matrix has a shape of [N_v, 768]. For each text token position i, its query vector is multiplied by the key vectors of all visual tokens, and the attention weights are obtained after scaling and softmax. Attention weights Let i represent the attention level of the i-th text token to the j-th visual token. The value vectors of all visual tokens are weighted and summed to obtain the preliminary visual text intermediate features of the text tokens. The preliminary visual text intermediate features are the feature vectors obtained by weighting and summing the value vectors of all visual tokens for each text token position after passing through the cross attention sublayer. It initially injects visual information into the text representation.

[0036] Specifically, the decoder contains multiple identical decoding layers, each of which, in addition to a cross-attention sublayer, includes a self-attention sublayer (enabling text tokens to interact with each other) and a feedforward network. The initial visual text intermediate features, along with the visual embeddings from the input sequence, are fed into multiple stacked decoding layers of the decoder. Each decoding layer sequentially performs self-attention operations to enable interaction between text tokens. Causal masks prevent future information leakage, and cross-attention operations further deepen the injection of visual information by using the current representation of the text token as the query and the visual embedding as the key and value. A feedforward network enhances non-linear expressive capabilities through dimensional expansion and contraction. Residual connections and layer normalization are applied after each sub-layer. After multiple layers of such layer transformations, the low-level features in the original visual embeddings and the semantic features in the text embeddings are gradually abstracted, aligned, and fused. The part corresponding to the text position in the output joint feature sequence is no longer just a preliminary visual-text hybrid representation, but a deep feature sequence that integrates multiple rounds of self-attention (internal context of the text) and cross-attention (multiple directional attention of the text to the visual). This sequence is the deep visual-text feature sequence, containing rich cross-modal interaction information, providing high-quality input for subsequent extraction of visual-text features.

[0037] Furthermore, after the decoder completes all multi-layer transformations, the output deep feature sequence is a joint sequence. The first half corresponds to the positions of each image patch in the original visual embedding, and the second half corresponds to the positions of each text token in the original text embedding. The segmentation operation extracts the corresponding vector in the second half of the joint sequence according to the known text token length. That is, it extracts the sub-matrix from the visual token at position i+1 to the last position in the sequence. The number of rows in this sub-matrix is ​​equal to the number of text tokens, and the number of columns is the hidden dimension. Since the cross-attention mechanism in each layer of decoding has fully integrated the related visual semantics into the vector of the text token position, the segmented feature sub-sequence is the text-side feature that integrates visual information, i.e., visual-text features, which are used for subsequent linear transformation and fusion steps. This avoids directly passing a large number of visual tokens to the drafter and significantly reduces the computational complexity.

[0038] For example, in a medical scenario, the text token sequence is "description," "lung," and "abnormality," while the visual embedding consists of 196 image patches. The decoder concatenates the visual and textual elements, using the text token as the query and the visual token as the key and value, to calculate cross-attention weights. For instance, for the text token "abnormality," the attention weights are highly focused on image patches with brightness anomalies (such as nodules or consolidation). After weighted summation, preliminary intermediate visual-text features are obtained. These are then processed through multiple decoding layers, and the final output visual-text features contain vectors corresponding to abnormalities that incorporate the precise location and morphological information of the lesion region, thereby guiding the subsequent generation of accurate diagnostic descriptions. In a fintech scenario, the text token sequence is "amount," "for," and "RMB," while the visual embedding includes various regions on the invoice. The cross-attention mechanism assigns extremely high weight to the numeric area on the right side of the invoice (such as the image block where ¥1280.00 is located) in the query vector of the text token amount, while paying almost no attention to the background or stamp area. In the visual text features obtained after multi-layer fusion, the vector of the amount token internalizes the visual form of the specific value ¥1280.00, providing sufficient visual guidance for the drafter to correctly output the number.

[0039] Furthermore, the cross-attention sublayer in the decoder enables text tokens to focus on the most relevant image regions, achieving precise injection of visual semantics into the text modality, while preventing visual tokens from entering the drafter, thus significantly reducing the computational burden.

[0040] S4. Perform a linear transformation on the visual text features, and fuse the linearly transformed visual text features with the text embedding to obtain a visually injected text embedding.

[0041] In this embodiment of the invention, the linear transformation refers to a trainable fully connected layer used to adjust the numerical distribution of visual text features so that it aligns with the distribution of the original text embedding.

[0042] In detail, a trainable fully connected layer can be used, whose input dimension is the same as the hidden dimension of the visual text features, and whose output dimension is also the same. The visual text feature matrix with shape [text token count, hidden dimension] is multiplied with the weight matrix of the fully connected layer (with shape [hidden dimension, hidden dimension]), and a bias vector (length of the hidden dimension) is added to obtain a transformed visual feature matrix with the same dimension and numerical distribution as the original text embedding. This provides a matching feature representation for subsequent addition of the residual with the text embedding.

[0043] In this embodiment of the invention, the visually injected text embedding is the feature sequence that is finally input to the drafter. Its format is exactly the same as that of plain text embedding, but the vector at each position has been enriched with semantics from the image.

[0044] In this embodiment of the invention, fusing the linearly transformed visual text features with the text embedding to obtain the visually injected text embedding includes: The linearly transformed visual text features are added element-wise to the text embedding according to a preset position sequence to obtain an intermediate fusion vector sequence. The intermediate fusion vector sequence is standardized by a preset layer normalization layer to obtain a fusion feature sequence; The fused feature sequence is used as a visually injected text embedding.

[0045] In detail, a linear transformation matrix W and a bias term b are applied to the visual text features X to obtain the transformed features Y, i.e., Y = X·W + b. This transformation makes the statistical distribution (mean, variance) of Y similar to the original text embedding. The transformed features Y are then added element-wise to the original text embedding to obtain an intermediate fused vector sequence. The addition of residuals preserves the semantic skeleton of the original text while injecting visual details. For each row of the intermediate fused vector sequence... Perform layer normalization independently and calculate the mean of the vector. and variance Then scale and translate, that is ,in and These are learnable scaling and offset parameters. This is the normalized intermediate fusion vector sequence, i.e., the fusion feature sequence. Layer normalization can stabilize the training process, prevent feature scale drift, and make subsequent drafters easier to process. The fusion feature sequence is determined as a visually injected text embedding, which is completely consistent with the plain text embedding in form. Therefore, it can seamlessly replace the input of a conventional language model drafter without modifying the internal structure of the drafter.

[0046] Furthermore, by using linear transformation and residual addition, visual information is hidden in the standard text embedding format, allowing any lightweight language model to be directly used as a drafter without any visual adapter, which significantly improves the versatility of the method.

[0047] S5. Use the drafter in the visual language model to perform autoregressive analysis on the visually injected text embedding to obtain the candidate tokens corresponding to the visually injected text embedding.

[0048] In this embodiment of the invention, the drafter is a lightweight autoregressive language model, typically consisting of only a text decoder, with a parameter count that is only a fraction of that of the target visual language model; candidate tokens refer to text lexical sequences that are quickly generated by the drafter and have not yet been validated by the target model.

[0049] In this embodiment of the invention, the step of using the drafter in the visual language model to perform autoregressive analysis on the visually injected text embedding to obtain candidate tokens corresponding to the visually injected text embedding includes: The visually injected text embedding is used as the target input sequence of the drafter. The self-attention layer in the drafter performs forward computation on the target input sequence to obtain the hidden state vector. Extract the residual vector corresponding to the hidden state vector from the preset set of step-dependent deviation correction residuals, and add the residual vector to the hidden state vector element-wise to obtain the target state vector; The word probability distribution of the target input sequence is calculated based on the target state vector, and target candidate tokens are selected from the tokens corresponding to the word probability distribution according to a preset sampling strategy. The target candidate token is added to the end of the target input sequence to form an extended input sequence, and the length counter corresponding to the extended input sequence is updated. Determine whether the count value of the length counter has reached the preset draft length threshold. If not, use the extended input sequence as the target input sequence and return to the step of performing forward calculation on the target input sequence through the self-attention layer in the drafter until the count value of the length counter reaches the preset draft length threshold. If the target is achieved, the selected target candidate token will be used as the candidate token corresponding to the visually injected text embedding.

[0050] In detail, the visually injected text embedding is used as the initial target input sequence for the drafter, with the shape [number of text tokens, hidden dimension]. The drafter consists of multiple Transformer decoders (typically 1 / 3 the number of layers in the target model, such as 6 layers). Its self-attention layer first performs forward computation on the input sequence. As the input sequence passes through each layer, a causal mask ensures that each position can only focus on itself and the tokens before it. The last layer outputs a hidden state matrix, and the vector at the last position in the matrix, i.e., the end of the current input sequence, is used as the hidden state vector for the current inference step, denoted as h_t. Since the drafter does not contain a visual encoder and does not receive visual feedback from the target model in real time, its h_t deviates from the ideal hidden state of the target model in the same context. Therefore, during the training phase, by simulating parallel inference between the drafter and the target model, a residual vector r_t is pre-computed for each time step t (e.g., t=1, 2, ..., L, where L is the draft length threshold of 5), forming a set of step-dependent bias correction residuals. During inference, the drafter extracts r_t corresponding to the current step and performs element-wise addition: h'_t=h_t+r_t, to obtain the target state vector h'_t. The additive correction directly compensates for the semantic drift accumulated due to the lack of visual information.

[0051] Specifically, based on the corrected target state vector h'_t, the drafter calculates the word probability distribution through a linear layer (768-dimensional input, output vocabulary size dimension, e.g., 50257) and a Softmax function. A temperature sampling strategy (temperature parameter 0.8) is used to divide the original output score by the temperature before applying Softmax, making the distribution smoother and increasing generation diversity; alternatively, top-k sampling (k=50) is used to retain only the 50 words with the highest probabilities. A token is extracted from the probability distribution as the target candidate token for the current step, and this candidate token is converted into a 768-dimensional vector using the word embedding matrix. This vector is then appended to the end of the current input sequence, forming an extended new input sequence, such as increasing the length from 3 to 4. Simultaneously, the length counter is incremented by 1 (initially 0, now 1).

[0052] Furthermore, the drafter checks whether the length counter value has reached a preset draft length threshold (e.g., 5). If it has not reached the threshold (e.g., current counter = 1 < 5), the extended input sequence is used as the new target input sequence, and the process returns to the step of forward computation through the self-attention layer, i.e., the next time step iteration begins. Based on the sequence containing candidate tokens, a new hidden state is calculated, the corresponding residual r_2 is extracted, and after correction, a second candidate token is generated. This process is repeated until the fifth candidate token is generated, at which point the counter reaches 5, and the iteration stops. Finally, all candidate tokens extracted sequentially during the iteration process are combined into an output sequence, which is the candidate token corresponding to the visually injected text embedding. The candidate tokens are then fed into the target visual language model for parallel verification. Through time-step correction of the step-dependent residual, the drafter can stably generate candidate sequences that are highly consistent with the distribution of the target model, even without real-time visual feedback from the target model. This significantly reduces the number of forward propagations of the target model while ensuring quality, thus accelerating inference.

[0053] For example, in a medical scenario, the initial input to the drafter is a visually injected text embedding (corresponding to description, lung, abnormality). After self-attention computation, the drafter obtains a hidden state vector, adds a pre-trained step-dependent residual r1, corrects it, predicts the next token, samples the ground glass image, appends this token to the end of the sequence, and updates the counter to 1. This process is repeated to generate five candidate tokens in sequence: shadow, size, approximately, 2cm. Each step uses the corresponding residuals r2 to r5 for correction, ensuring that even if the drafter does not directly see the chest X-ray image, it can generate a reasonable candidate sequence that conforms to the imaging description. In a fintech scenario, the drafter generates a visually injected text embedding (corresponding to price and tax, total, for) based on an invoice image. After residual correction, it predicts 1,000, then generates 200, yuan, and whole, for a total of five candidate tokens. The step-dependent residual compensates for the semantic drift caused by the drafter's inability to obtain fine-grained visual features of the invoice's numerical region in real time, making the generated amount highly consistent with the actual number in the image, improving the accuracy and speed of automatic invoice recognition.

[0054] Furthermore, a drafter with far fewer parameters than the target model is used, and the drafter only processes text embeddings, resulting in extremely fast inference speed. It also generates candidate tokens of length L through autoregressive iteration, providing the possibility for subsequent parallel verification. The overall decoding steps are reduced from L steps to 1 step.

[0055] S6. Verify the output probability of the candidate tokens according to the output probability distribution of the visual language model, and use the verified candidate tokens as the decoding output of the visual language model.

[0056] In this embodiment of the invention, the output probability distribution refers to the vocabulary probability calculated by the target visual language model for the complete context (image + original text + candidate tokens); the verification process is used to determine whether the candidate tokens generated by the drafter are consistent with the judgment of the target model, thereby deciding whether to accept or reject.

[0057] In this embodiment of the invention, verifying the output probability of the candidate token based on the output probability distribution of the visual language model includes: The candidate token is concatenated with the target text token to obtain the verification sequence; The visual language model is used to perform autoregressive forward propagation on the verification sequence to obtain the target word probability distribution corresponding to the candidate token; Extract the output probability distribution of the visual language model corresponding to the position order of the candidate tokens, and query the probability value corresponding to the candidate token from the output probability distribution; When the probability value is greater than or equal to the preset acceptance threshold, the candidate token is determined to be a candidate token that has passed verification. When the probability value is less than the preset acceptance threshold, a target token is selected from the tokens corresponding to the output probability distribution to replace the candidate token, and the target token is used as the candidate token that passes the verification.

[0058] In detail, the candidate token sequence generated by the drafter is concatenated sequentially with the original target text tokens to form a complete verification sequence. The length of this verification sequence is the sum of the number of original text tokens and the number of candidate tokens. Simultaneously, the visual embedding of the original image (e.g., 196 image patches, each 768 dimensions) remains unchanged. The entire verification sequence and visual embedding are then input into the target visual language model at once, performing a single autoregressive forward propagation. Due to the causal masking mechanism of the Transformer decoder, the model can only see all tokens up to and including that position when calculating the output at each position. Therefore, this forward propagation can generate the corresponding word probability distribution for each candidate token position in the verification sequence, such as positions 4 to 8, in parallel. Compared to calling the target model word by word, this one-time parallel computation significantly reduces inference time.

[0059] Specifically, for each candidate token position, the probability distribution of that position output by the target model is extracted, and the probability value corresponding to the candidate token itself is retrieved from it. Assuming a preset acceptance threshold of 0.5, the probability value of each candidate token is compared sequentially with the threshold. If the probability value is greater than or equal to 0.5, the token is accepted, and the next candidate token is checked. If the probability value is less than 0.5, the token is rejected, and the verification of all subsequent candidate tokens is immediately terminated. Simultaneously, a new token is selected from the output probability distribution of the target model at the current position as a replacement, typically the token with the highest probability, but a sampling strategy can also be used. Finally, all accepted and replaced tokens are output sequentially as the decoding result of the visual language model. This verification mechanism ensures that the final output is highly consistent with the autoregressive decoding result of the target model in terms of distribution. Furthermore, since the drafter generates most of the acceptable tokens, the actual steps required by the target model are significantly reduced, thus achieving high-quality and high-efficiency speculative decoding.

[0060] Furthermore, for the candidate token sequence generated by the drafter, after the target visual language model performs threshold comparison on the output probability distribution of each position, all candidate tokens that are judged as accepted (i.e., their probability value is greater than or equal to the preset acceptance threshold), as well as the target tokens resampled from the target model distribution at the rejection position, are arranged in their original order in the sequence to form the final output sequence. This sequence is the final response result of the visual language model to the original input image and text prompts. It retains the efficiency of the drafter's rapid generation and ensures that the output quality is highly consistent with the result of directly using the target model's autoregressive decoding through the verification of the target model, thus achieving the dual goals of accelerating inference and maintaining generation fidelity.

[0061] Furthermore, the target model validates the entire candidate sequence at once, ensuring quality without loss through an acceptance threshold mechanism and terminating immediately after a rejection position to avoid invalid computation. This validation process enables speculative decoding to achieve several times the speedup while maintaining the output distribution consistent with the target model.

[0062] As can be seen, in the above scheme, by injecting visual information into the text embedding, the drafter does not need to process the original visual token at all, which significantly reduces the computational burden; by using the cross-modal capability of the target vision-language model itself to generate visual-injected text embedding, the problem of inconsistency between visual and text semantics is solved; through the step-dependent bias correction residual mechanism, the drafter's prediction drift is stabilized in the absence of real-time feedback from the target model. Thus, while ensuring that the output quality is consistent with the target model, efficient speculative decoding acceleration is achieved, which can solve the problem of low efficiency when performing model decoding. It can be widely used in scenarios such as real-time medical image report generation and high-speed automatic recognition of financial bills, significantly improving inference throughput and reducing deployment costs.

[0063] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0064] In one embodiment, a model decoding device based on visual token hiding is provided. This model decoding device 100 based on visual token hiding corresponds one-to-one with the model decoding method based on visual token hiding in the above embodiments. Figure 3 As shown, the visual token-hiding-based model decoding device 100 includes a visual analysis module 101, a text feature extraction module 102, a cross-modal fusion module 103, a feature fusion module 104, an autoregressive analysis module 105, and a probability verification module 106. Detailed descriptions of each functional module are as follows: The visual analysis module 101 is used to acquire the target input image and the target text token, and to perform visual analysis on the target input image using the visual encoder in the preset visual language model to obtain the visual embedding. The text feature extraction module 102 is used to extract text features from the target text token using the text encoder in the visual language model to obtain a text embedding. The cross-modal fusion module 103 is used to perform cross-modal fusion of the visual embedding and the text embedding through the decoder of the visual language model to generate visual text features; The feature fusion module 104 is used to perform a linear transformation on the visual text features, and then fuse the linearly transformed visual text features with the text embedding to obtain a visually injected text embedding. The autoregressive analysis module 105 is used to perform autoregressive analysis on the visually injected text embedding using the drafter in the visual language model to obtain the candidate tokens corresponding to the visually injected text embedding. The probability verification module 106 is used to verify the output probability of the candidate tokens according to the output probability distribution of the visual language model, and to use the candidate tokens that pass the verification as the decoding output result of the visual language model.

[0065] In one embodiment, the visual analysis module 101, when performing visual analysis on the target input image using a visual encoder in a preset visual language model to obtain a visual embedding, is used to: The target input image is divided into multiple non-overlapping image blocks, and the pixel values ​​of each image block are flattened into a one-dimensional pixel sequence; Each one-dimensional pixel sequence is mapped to an initial visual feature vector through the linear projection layer of the visual encoder, and the initial visual feature vector is encoded into an initial visual embedding sequence. Position embedding is performed on each feature vector in the initial visual embedding sequence to obtain a visual embedding sequence. The global attention weights between each image patch in the visual embedding sequence are analyzed using the multi-head self-attention layer of the visual encoder. The global attention weights are weighted and summed with the value vectors of all image blocks to obtain the output feature sequence of the multi-head self-attention layer. The output feature sequence is then linearly transformed to obtain the visual embedding.

[0066] In one embodiment, the text feature extraction module 102, when performing text feature extraction on the target text token using the text encoder in the visual language model to obtain the text embedding, is used to: Each discrete token in the target text tokens is mapped to an initial text feature vector, and the initial text feature vector is encoded into an initial text embedding sequence; Positional encodings are superimposed on each position in the initial text embedding sequence to obtain the text embedding sequence; The text embedding sequence is encoded and transformed using the encoding layer of the text encoder to generate a text embedding.

[0067] In one embodiment, the cross-modal fusion module 103, when performing cross-modal fusion of the visual embedding and the text embedding through the decoder of the visual language model to generate visual text features, is used to: The visual embedding and the text embedding are concatenated according to the sequence dimension to generate a joint embedding sequence. The text embedding in the joint embedding vector is used as the query vector, and the visual embedding in the joint embedding vector is used as the key vector and value vector. The attention weight of each text token for all visual tokens is calculated based on the query vector, the key vector, and the value vector. The attention weight is then used to perform a weighted summation of the value vector to obtain the preliminary visual text intermediate features corresponding to the position of each text token. The intermediate features of the preliminary visual text are processed by the decoding layer of the decoder to obtain a deep visual text feature sequence. The deep visual text feature sequence is segmented to obtain the feature subsequence corresponding to the text embedding, and the feature subsequence is used as the visual text feature.

[0068] In one embodiment, the feature fusion module 104, when performing a linear transformation on the visual text features and fusing the linearly transformed visual text features with the text embedding to obtain a visually injected text embedding, is used to: The linearly transformed visual text features are added element-wise to the text embedding according to a preset position sequence to obtain an intermediate fusion vector sequence. The intermediate fusion vector sequence is standardized by a preset layer normalization layer to obtain a fusion feature sequence; The fused feature sequence is used as a visually injected text embedding.

[0069] In one embodiment, the autoregressive analysis module 105, when performing autoregressive analysis on the visually injected text embedding using the drafter in the visual language model to obtain candidate tokens corresponding to the visually injected text embedding, is used to: The visually injected text embedding is used as the target input sequence of the drafter. The self-attention layer in the drafter performs forward computation on the target input sequence to obtain the hidden state vector. Extract the residual vector corresponding to the hidden state vector from the preset set of step-dependent deviation correction residuals, and add the residual vector to the hidden state vector element-wise to obtain the target state vector; The word probability distribution of the target input sequence is calculated based on the target state vector, and target candidate tokens are selected from the tokens corresponding to the word probability distribution according to a preset sampling strategy. The target candidate token is added to the end of the target input sequence to form an extended input sequence, and the length counter corresponding to the extended input sequence is updated. Determine whether the count value of the length counter has reached the preset draft length threshold. If not, use the extended input sequence as the target input sequence and return to the step of performing forward calculation on the target input sequence through the self-attention layer in the drafter until the count value of the length counter reaches the preset draft length threshold. If the target is achieved, the selected target candidate token will be used as the candidate token corresponding to the visually injected text embedding.

[0070] In one embodiment, the probability verification module 106, when performing the verification of the output probability of the candidate token based on the output probability distribution of the visual language model, is configured to: The candidate token is concatenated with the target text token to obtain the verification sequence; The visual language model is used to perform autoregressive forward propagation on the verification sequence to obtain the target word probability distribution corresponding to the candidate token; Extract the output probability distribution of the visual language model corresponding to the position order of the candidate tokens, and query the probability value corresponding to the candidate token from the output probability distribution; When the probability value is greater than or equal to the preset acceptance threshold, the candidate token is determined to be a candidate token that has passed verification. When the probability value is less than the preset acceptance threshold, a target token is selected from the tokens corresponding to the output probability distribution to replace the candidate token, and the target token is used as the candidate token that passes the verification.

[0071] This invention provides a model decoding device based on visual token hiding. By injecting visual information into text embeddings, the drafter no longer needs to process the original visual tokens, significantly reducing the computational burden. It utilizes the cross-modal capabilities of the target visual-language model to generate visually injected text embeddings, solving the problem of inconsistency between visual and textual semantics. Through a step-dependent bias correction residual mechanism, the drafter's prediction drift is stabilized in the absence of real-time feedback from the target model. Thus, while ensuring that the output quality is consistent with the target model, efficient inference decoding acceleration is achieved. This solves the problem of low efficiency in model decoding and can be widely applied to scenarios such as real-time medical image report generation and high-speed automatic recognition of financial documents, significantly improving inference throughput and reducing deployment costs.

[0072] Specific limitations regarding the model decoding device based on visual token hiding can be found in the limitations of the model decoding method based on visual token hiding mentioned above, and will not be repeated here. Each module in the aforementioned model decoding device based on visual token hiding can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0073] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side method for decoding a model based on visual token hiding.

[0074] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a model decoding method based on visual token hiding.

[0075] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the target input image and the target text token, and use the visual encoder in the preset visual language model to perform visual analysis on the target input image to obtain the visual embedding; The text encoder in the visual language model is used to extract text features from the target text token to obtain a text embedding. The visual language model's decoder performs cross-modal fusion of the visual embedding and the text embedding to generate visual text features; The visual text features are linearly transformed, and the linearly transformed visual text features are fused with the text embedding to obtain the visually injected text embedding. The drafter in the visual language model is used to perform autoregressive analysis on the visually injected text embedding to obtain candidate tokens corresponding to the visually injected text embedding. The output probability of the candidate token is verified based on the output probability distribution of the visual language model, and the candidate token that passes the verification is used as the decoding output of the visual language model.

[0076] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the target input image and the target text token, and use the visual encoder in the preset visual language model to perform visual analysis on the target input image to obtain the visual embedding; The text encoder in the visual language model is used to extract text features from the target text token to obtain a text embedding. The visual language model's decoder performs cross-modal fusion of the visual embedding and the text embedding to generate visual text features; The visual text features are linearly transformed, and the linearly transformed visual text features are fused with the text embedding to obtain the visually injected text embedding. The drafter in the visual language model is used to perform autoregressive analysis on the visually injected text embedding to obtain candidate tokens corresponding to the visually injected text embedding. The output probability of the candidate token is verified based on the output probability distribution of the visual language model, and the candidate token that passes the verification is used as the decoding output of the visual language model.

[0077] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0078] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0080] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0081] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A model decoding method based on visual token hiding, characterized in that, The method includes: Obtain the target input image and the target text token, and use the visual encoder in the preset visual language model to perform visual analysis on the target input image to obtain the visual embedding; The text encoder in the visual language model is used to extract text features from the target text token to obtain a text embedding. The visual language model's decoder performs cross-modal fusion of the visual embedding and the text embedding to generate visual text features; The visual text features are linearly transformed, and the linearly transformed visual text features are fused with the text embedding to obtain the visually injected text embedding. The drafter in the visual language model is used to perform autoregressive analysis on the visually injected text embedding to obtain candidate tokens corresponding to the visually injected text embedding. The output probability of the candidate token is verified based on the output probability distribution of the visual language model, and the candidate token that passes the verification is used as the decoding output of the visual language model.

2. The model decoding method based on visual token hiding as described in claim 1, characterized in that, The step of performing visual analysis on the target input image using a visual encoder in a preset visual language model to obtain a visual embedding includes: The target input image is divided into multiple non-overlapping image blocks, and the pixel values ​​of each image block are flattened into a one-dimensional pixel sequence; Each one-dimensional pixel sequence is mapped to an initial visual feature vector through the linear projection layer of the visual encoder, and the initial visual feature vector is encoded into an initial visual embedding sequence. Position embedding is performed on each feature vector in the initial visual embedding sequence to obtain a visual embedding sequence. The global attention weights between each image patch in the visual embedding sequence are analyzed using the multi-head self-attention layer of the visual encoder. The global attention weights are weighted and summed with the value vectors of all image blocks to obtain the output feature sequence of the multi-head self-attention layer. The output feature sequence is then linearly transformed to obtain the visual embedding.

3. The model decoding method based on visual token hiding as described in claim 1, characterized in that, The step of extracting text features from the target text token using the text encoder in the visual language model to obtain the text embedding includes: Each discrete token in the target text tokens is mapped to an initial text feature vector, and the initial text feature vector is encoded into an initial text embedding sequence; Positional encodings are superimposed on each position in the initial text embedding sequence to obtain the text embedding sequence; The text embedding sequence is encoded and transformed using the encoding layer of the text encoder to generate a text embedding.

4. The model decoding method based on visual token hiding as described in claim 1, characterized in that, The step of performing cross-modal fusion of the visual embedding and the text embedding through the decoder of the visual language model to generate visual text features includes: The visual embedding and the text embedding are concatenated according to the sequence dimension to generate a joint embedding sequence. The text embedding in the joint embedding vector is used as the query vector, and the visual embedding in the joint embedding vector is used as the key vector and value vector. The attention weight of each text token for all visual tokens is calculated based on the query vector, the key vector, and the value vector. The attention weight is then used to perform a weighted summation of the value vector to obtain the preliminary visual text intermediate features corresponding to the position of each text token. The intermediate features of the preliminary visual text are processed by the decoding layer of the decoder to obtain a deep visual text feature sequence. The deep visual text feature sequence is segmented to obtain the feature subsequence corresponding to the text embedding, and the feature subsequence is used as the visual text feature.

5. The model decoding method based on visual token hiding as described in claim 1, characterized in that, The process of fusing the linearly transformed visual text features with the text embedding to obtain a visually injected text embedding includes: The linearly transformed visual text features are added element-wise to the text embedding according to a preset position sequence to obtain an intermediate fusion vector sequence. The intermediate fusion vector sequence is standardized by a preset layer normalization layer to obtain a fusion feature sequence; The fused feature sequence is used as a visually injected text embedding.

6. The model decoding method based on visual token hiding as described in claim 1, characterized in that, The step of using the drafter in the visual language model to perform autoregressive analysis on the visually injected text embedding to obtain candidate tokens corresponding to the visually injected text embedding includes: The visually injected text embedding is used as the target input sequence of the drafter. The self-attention layer in the drafter performs forward computation on the target input sequence to obtain the hidden state vector. Extract the residual vector corresponding to the hidden state vector from the preset set of step-dependent deviation correction residuals, and add the residual vector to the hidden state vector element-wise to obtain the target state vector; The word probability distribution of the target input sequence is calculated based on the target state vector, and target candidate tokens are selected from the tokens corresponding to the word probability distribution according to a preset sampling strategy. The target candidate token is added to the end of the target input sequence to form an extended input sequence, and the length counter corresponding to the extended input sequence is updated. Determine whether the count value of the length counter has reached the preset draft length threshold. If not, use the extended input sequence as the target input sequence and return to the step of performing forward calculation on the target input sequence through the self-attention layer in the drafter until the count value of the length counter reaches the preset draft length threshold. If the target candidate token is selected, it will be used as the candidate token corresponding to the visually injected text embedding.

7. The model decoding method based on visual token hiding as described in claim 1, characterized in that, The step of verifying the output probability of the candidate token based on the output probability distribution of the visual language model includes: The candidate token is concatenated with the target text token to obtain the verification sequence; The visual language model is used to perform autoregressive forward propagation on the verification sequence to obtain the target word probability distribution corresponding to the candidate token; Extract the output probability distribution of the visual language model corresponding to the position order of the candidate tokens, and query the probability value corresponding to the candidate token from the output probability distribution; When the probability value is greater than or equal to the preset acceptance threshold, the candidate token is determined to be a candidate token that has passed verification. When the probability value is less than the preset acceptance threshold, a target token is selected from the tokens corresponding to the output probability distribution to replace the candidate token, and the target token is used as the candidate token that passes the verification.

8. A model decoding device based on visual token hiding, characterized in that, The device includes: The visual analysis module is used to acquire the target input image and the target text token, and to perform visual analysis on the target input image using the visual encoder in the preset visual language model to obtain the visual embedding. The text feature extraction module is used to extract text features from the target text token using the text encoder in the visual language model to obtain the text embedding. The cross-modal fusion module is used to perform cross-modal fusion of the visual embedding and the text embedding through the decoder of the visual language model to generate visual text features; The feature fusion module is used to perform a linear transformation on the visual text features, and then fuse the linearly transformed visual text features with the text embedding to obtain a visually injected text embedding. The autoregressive analysis module is used to perform autoregressive analysis on the visually injected text embedding using the drafter in the visual language model to obtain the candidate tokens corresponding to the visually injected text embedding. The probability verification module is used to verify the output probability of the candidate tokens according to the output probability distribution of the visual language model, and to use the candidate tokens that pass the verification as the decoding output of the visual language model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the model decoding method based on visual token hiding as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the model decoding method based on visual token hiding as described in any one of claims 1 to 7.

Images