An efficient image description method, system, device, program and storage medium based on Transformer-Prefix
By using a Decoder-only Transformer architecture and a prefix injection mechanism, visual features are mapped to text prefixes, solving the problems of computational complexity and inference latency in image description methods and achieving efficient image description generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HARBIN ENG UNIV
- Filing Date
- 2026-03-27
- Publication Date
- 2026-07-03
AI Technical Summary
Existing image description methods suffer from high overhead in terms of computational complexity and inference latency, making them difficult to meet the needs of resource-constrained devices.
By adopting a Decoder-only Transformer architecture and a prefix injection mechanism, visual features are mapped to text prefixes, and image descriptions are generated through a self-attention layer, reducing the computational overhead of the cross-attention module.
It significantly reduces the size of model parameters and computational overhead, improves inference efficiency, is suitable for edge devices and mobile applications, and maintains generation performance.
Smart Images
Figure CN122334191A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of computer vision and natural language processing, and specifically relates to an efficient image description method, system, device, program and storage medium based on Transformer-Prefix. Background Technology
[0002] Image captioning aims to automatically generate natural language descriptions that accurately reflect visual content from input images, and has significant application value in scenarios such as visual impairment assistance, human-computer interaction, and cross-modal information retrieval. Most existing mainstream methods adopt an "encoder-decoder" structure, in which the visual encoder (such as ResNet, ViT) is used to extract spatial features of the image, and the text decoder (usually LSTM or Standard Transformer) is used to generate the descriptive sequence.
[0003] In recent years, Transformer-based methods have achieved significant performance improvements in image captioning tasks due to their powerful modeling capabilities. However, traditional Transformer decoders typically rely on self-attention and cross-attention mechanisms in each layer. The cross-attention mechanism, in particular, requires the decoder to interact with all visual features output by the encoder at every step of generation. This architecture suffers from the following drawbacks:
[0004] High computational complexity: During the inference phase, the cross-attention mechanism introduces a large number of matrix operations, and the computational overhead increases significantly with the sequence length; Large memory usage: A large number of intermediate states need to be cached (Key-Value Cache), which places high demands on hardware resources; Structural redundancy: For simple visual scenarios, frequent visual-text interactions may be redundant, leading to increased inference latency and making it difficult to meet the needs of resource-constrained devices (such as mobile devices and embedded devices) or low-latency application scenarios.
[0005] Therefore, how to reduce model structure complexity and inference cost while maintaining generation quality has become a pressing technical problem in this field. Summary of the Invention
[0006] The purpose of this invention is to address the problems of high computational complexity and large inference latency in existing encoder-decoder architectures by providing an efficient image description method, system, device, program, and storage medium based on Transformer-Prefix.
[0007] An efficient image description method based on Transformer-Prefix is proposed. The global feature vector of the image is extracted and mapped to a continuous vector representation with the same dimension as the text embedding, which serves as a visual prefix. The visual prefix is concatenated at the beginning of the word embedding sequence to form the input sequence of the decoder. The decoder adopts a Decoder-only Transformer architecture and processes the input sequence through stacked self-attention layers to generate descriptive text for the image.
[0008] Furthermore, the global feature vector of the image Mapped to a continuous vector representation consistent with the text embedding dimension. , as a visual prefix:
[0009]
[0010] in, and As weight; and For bias terms; This is the activation function.
[0011] Furthermore, the original natural language sequence is converted into a discrete word tag sequence using a tokenizer. Tag each word Convert to the corresponding continuous vector representation The word embedding sequence is obtained. ;
[0012] visual prefix This sequence is concatenated at the very beginning of the word embedding sequence to form the input sequence for the decoder. ;
[0013]
[0014] An efficient image description system based on Transformer-Prefix, comprising an encoder, a mapping network, and a decoder;
[0015] The encoder is used to extract global feature vectors from the input image;
[0016] The mapping network is used to map the global feature vector into a continuous vector representation consistent with the text embedding dimension, which serves as a visual prefix. The visual prefix is then concatenated at the beginning of the word embedding sequence to form the input sequence of the decoder.
[0017] The decoder adopts a Decoder-only Transformer architecture, which processes the input sequence through stacked self-attention layers to generate descriptive text for the image.
[0018] Furthermore, the system's training process is divided into two stages. In the first stage, the encoder parameters are fixed, and only the mapping network and decoder are trained. In the second stage, the SCST reinforcement learning algorithm is used, with the CIDEr score as the reward function, to optimize the system as a whole and improve the overall semantic consistency of the generated image description text.
[0019] Furthermore, the first stage of training employs the cross-entropy loss function:
[0020]
[0021] in, Indicates the first At each time step, the system generates image description text words. and visual prefix Under the given conditions, descriptive text words for the current real image The predicted probability; This represents the total number of time steps.
[0022] Furthermore, the loss function for the second stage of training is:
[0023]
[0024] in, This refers to the sentence sampled from the image description text generated by the system; This represents the baseline sentence generated by greedy decoding; The reward function is represented by the CIDER score. Sampling sentence The joint generation probability in the current system is calculated by multiplying the conditional probabilities at each time step. , Sampling sentence In the Words generated at each time step; Indicates the first The word sequence generated before time step n, i.e., from the first to the last word... A sequence consisting of all words generated at each time step.
[0025] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described efficient image description method based on Transformer-Prefix.
[0026] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described efficient image description method based on Transformer-Prefix.
[0027] A computer program product includes computer instructions that, when executed by a processor, implement the steps of the above-described efficient image description method based on Transformer-Prefix.
[0028] The beneficial effects of this invention are as follows:
[0029] This invention significantly reduces model parameter size and computational overhead (FLOPs) by employing a Decoder-only architecture and replacing the traditional cross-attention module with a prefix injection mechanism, thereby greatly improving inference efficiency. By using visual features as prefix context for text generation, visual information remains effective throughout the generation process, enhancing the guiding role of image semantics in text generation and avoiding the gradual decay of information. The model structure is lightweight and has low inference latency, making it particularly suitable for deployment in real-world application environments with limited computing power and GPU memory, such as edge devices, mobile apps, or real-time auxiliary systems. Experiments show that this invention achieves description generation performance (BLEU, CIDEr, etc.) comparable to complex Encoder-Decoder architectures while maintaining high efficiency. Attached Figure Description
[0030] Figure 1 This is a diagram of the overall architecture of the present invention.
[0031] Figure 2 This is a schematic diagram of the network architecture based on Transformer-Prefix in this invention. Detailed Implementation
[0032] The present invention will now be further described with reference to the accompanying drawings.
[0033] This invention addresses the inefficiency of traditional architectures that rely on cross-attention. It abandons the traditional cross-attention mechanism and adopts a decoder-only architecture, fusing visual information into the text generation process through prefix injection. First, robust visual features are extracted using the powerful zero-shot transfer capability of a pre-trained model. Second, a lightweight mapping network transforms the visual features into prefix embeddings in the text space. Finally, the self-attention mechanism of the Transformer decoder implicitly guides the entire text generation flow using visual prefixes, eliminating the need for explicit cross-modal interaction modules.
[0034] An efficient image description method based on Transformer-Prefix includes the following steps:
[0035] Step 1: Extract the global feature vector of the image And mapped to a continuous vector representation consistent with the text embedding dimension. , as a visual prefix;
[0036]
[0037] in, and As weight; and For bias terms; For activation functions;
[0038] Step 2: Convert the original natural language sequence into a discrete word tag sequence using a tokenizer. Tag each word Convert to the corresponding continuous vector representation The word embedding sequence is obtained. ; Visual prefix This sequence is concatenated at the very beginning of the word embedding sequence to form the input sequence for the decoder. ;
[0039]
[0040] Step 3: The decoder adopts a Decoder-only Transformer architecture, which processes the input sequence through stacked self-attention layers to generate descriptive text for the image.
[0041] Example 1:
[0042] An efficient image description system based on Transformer-Prefix, comprising an encoder, a mapping network, and a decoder;
[0043] The encoder is used to extract global feature vectors from the input image;
[0044] The mapping network is used to map the global feature vector into a continuous vector representation consistent with the text embedding dimension, which serves as a visual prefix. The visual prefix is then concatenated at the beginning of the word embedding sequence to form the input sequence of the decoder.
[0045] The decoder adopts a Decoder-only Transformer architecture, which processes the input sequence through stacked self-attention layers to generate descriptive text for the image.
[0046] The training process of the efficient image description system based on Transformer-Prefix is divided into two stages. In the first stage, the encoder parameters are fixed, and only the mapping network and decoder are trained. In the second stage, the SCST reinforcement learning algorithm is used to optimize the system as a whole, using the CIDEr score as the reward function, thereby improving the overall semantic consistency of the generated image description text.
[0047] Example 2:
[0048] In this embodiment, a pre-trained vision-language model (such as CLIP) is used as a visual encoder to encode the input image. Global semantic features are extracted from the visual encoder output as a high-level semantic representation of the image. A mapping network is designed to map the visual semantic features into vector representations consistent with the text embedding dimension. These mapped visual features are used as "soft prompts" or prefix information to guide the subsequent text generation process. A decoder-only Transformer architecture is adopted as the text generation model. Visual prefixes and text word embeddings are concatenated in the sequence dimension to form the decoder input sequence. Image description text is generated through an autoregressive approach, where the visual prefixes serve as initial context in sequence modeling. Cross-entropy loss is used for supervised training of the model to learn the correspondence between images and text. In the later stages of training, an optimization strategy based on sequence-level evaluation metrics (such as CIDEr) is introduced to further improve the overall consistency and semantic quality of the generated descriptions.
[0049] 1. Visual Feature Extraction
[0050] Input an RGB image I. This embodiment uses a pre-trained CLIP (ViT-B / 32) model as the visual encoder. Unlike traditional methods that extract grid features, this embodiment only extracts the global feature vector v of the image.
[0051]
[0052] This feature vector contains highly condensed semantic information about the image.
[0053] 2. Prefix Construction
[0054] Due to visual characteristics The embedding dimension (e.g., 512) may differ from the embedding dimension (e.g., 768) of the text decoder and reside in different semantic spaces, thus requiring a mapping network. This embodiment employs a two-layer multilayer perceptron (MLP) as the mapping network:
[0055]
[0056] in This refers to the generated visual prefix. This vector represents the text embedding dimension. It is treated as a virtual "word" embedding in subsequent processing.
[0057] 3. Decoding
[0058] A GPT-2-style Decoder-only Transformer is used as the decoder. The target description text is first processed by a predefined tokenizer, converting the original natural language sequence into a discrete sequence of word tokens. Subsequently, each tag is embedded through a mapping layer. Convert to the corresponding continuous vector representation Thus, the word embedding sequence is obtained. Visual prefixes Concatenate at the very beginning of the text sequence:
[0059]
[0060] The decoder processes the sequence through stacked self-attention layers:
[0061]
[0062] In the When generating words at each time step, the model can only see visual prefixes. and previously generated words Due to visual prefixes Finally, starting from the beginning of the context window, each subsequently generated word will aggregate visual prefixes through a self-attention mechanism. This information allows for control over text generation based on image content.
[0063] 4. Training Strategies
[0064] The training process is divided into two phases:
[0065] Phase 1 (Warm-up Training): With the visual encoder parameters fixed, only the mapping network and decoder are trained. The cross-entropy loss function is used.
[0066]
[0067] in, Indicates the first At each time step, the system generates image description text words. and visual prefix Under the given conditions, descriptive text words for the current real image The predicted probability; This represents the total number of time steps.
[0068] Phase Two (Fine-tuning): Some parameters are opened up for optimization using the SCST (Self-Critical Sequence Training) strategy. The CIDEr score is used as the reward function, and the loss function is defined as:
[0069]
[0070] in, This refers to the sentence sampled from the image description text generated by the system; This represents the baseline sentence generated by greedy decoding; The reward function is represented by a CIDEr score, which indicates the degree of consistency between the generated sentence and the reference sentence. The calculation is performed according to the standard definition of CIDEr, and the generated result is evaluated by n-gram weighted similarity with multiple reference annotations. Sampling sentence The joint generation probability in the current system is calculated by multiplying the conditional probabilities at each time step. , Sampling sentence In the Words generated at each time step; Indicates the first The word sequence generated before time step n, i.e., from the first to the last word... A sequence consisting of all words generated at each time step.
[0071] This invention significantly reduces the inference latency and memory usage of the model while ensuring the accuracy of image description.
[0072] (1) Lightweight alignment: By introducing an MLP mapping network to construct continuous prefixes, the retraining of large visual encoders is avoided, which greatly reduces the number of parameters for model tuning.
[0073] (2) Architectural efficiency: The Decoder-only architecture is used instead of the cross-attention mechanism, which reduces the computational redundancy inside the Transformer layer and improves the inference speed on edge devices.
[0074] (3) Strategy consistency: By combining cross-entropy and CIDEr reward function in two stages, the noise sensitivity of the generated text is improved, making it more in line with human cognitive habits of image semantics.
[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An efficient image description method based on Transformer-Prefix, characterized in that: The global feature vector of the image is extracted and mapped to a continuous vector representation consistent with the text embedding dimension, serving as a visual prefix. The visual prefix is concatenated at the beginning of the word embedding sequence to form the input sequence of the decoder. The decoder adopts a Decoder-only Transformer architecture, which processes the input sequence through stacked self-attention layers to generate descriptive text for the image.
2. The efficient image description method based on Transformer-Prefix according to claim 1, characterized in that: The global feature vector of the image Mapped to a continuous vector representation consistent with the text embedding dimension. , as a visual prefix: in, and As weight; and For bias terms; This is the activation function.
3. The efficient image description method based on Transformer-Prefix according to claim 1, characterized in that: The original natural language sequence is converted into a discrete word tag sequence using a tokenizer. Tag each word Convert to the corresponding continuous vector representation The word embedding sequence is obtained. ; visual prefix This sequence is concatenated at the very beginning of the word embedding sequence to form the input sequence for the decoder. ; 。 4. A high-efficiency image description system based on Transformer-Prefix, characterized in that: Includes encoders, mapping networks, and decoders; The encoder is used to extract global feature vectors from the input image; The mapping network is used to map the global feature vector into a continuous vector representation consistent with the text embedding dimension, which serves as a visual prefix. The visual prefix is then concatenated at the beginning of the word embedding sequence to form the input sequence of the decoder. The decoder adopts a Decoder-only Transformer architecture, which processes the input sequence through stacked self-attention layers to generate descriptive text for the image.
5. The efficient image description system based on Transformer-Prefix according to claim 4, characterized in that: The system's training process is divided into two stages. In the first stage, the encoder parameters are fixed, and only the mapping network and decoder are trained. In the second stage, the SCST reinforcement learning algorithm is used, with the CIDEr score as the reward function, to optimize the system as a whole and improve the overall semantic consistency of the generated image description text.
6. The efficient image description system based on Transformer-Prefix according to claim 5, characterized in that: The first stage of training uses the cross-entropy loss function: in, Indicates the first At each time step, the system generates image description text words. and visual prefix Under the given conditions, descriptive text words for the current real image The predicted probability; This represents the total number of time steps.
7. The efficient image description system based on Transformer-Prefix according to claim 6, characterized in that: The loss function for the second stage of training is: in, This refers to the sentence sampled from the image description text generated by the system; This represents the baseline sentence generated by greedy decoding; The reward function is represented by the CIDER score. Sampling sentence The joint generation probability in the current system is calculated by multiplying the conditional probabilities at each time step. , Sampling sentence In the Words generated at each time step; Indicates the first The word sequence generated before time step n, i.e., from the first to the last word... A sequence consisting of all words generated at each time step.
8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 3.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 3.
10. A computer program product comprising computer instructions, characterized in that: When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 3.