Image description generation method and system based on two-stage progressive fusion coding

Abstract

The application discloses an image description generation method and system based on two-stage progressive fusion coding, and the method comprises the following steps: in the first stage, the features extracted by the image encoder CLIP ViT are gradually interpolated into the features extracted by the corresponding image encoder Swin Transformer to refine the semantic representation; in the second stage, a global perception workspace module is proposed, the workspace integrates the features from the image encoder Swin Transformer and the image encoder CLIP ViT through weighted fusion; and a length-independent expansion module is used to efficiently process the variable-length input; the problems of feature representation fragmentation and unsatisfactory visual language alignment caused by the dependence of the existing method on a single visual encoder are solved, and the method has outstanding performance in the accuracy of image description generation and the richness of semantic expression.

Description

Technical Field

[0001] This invention belongs to the fields of multimodal fusion technology and image description generation technology, and in particular relates to an image description generation method and system based on two-stage progressive fusion coding. Background Technology

[0002] Image captioning generation is a fundamental multimodal task that requires generating natural language descriptions from visual input. This necessitates not only strong image recognition capabilities but also complex semantic understanding to produce coherent text. Traditional image captioning generation methods primarily utilize Convolutional Neural Networks (CNNs) for image feature extraction, followed by Recurrent Neural Networks (RNNs) for text generation. In recent years, the evolution of deep learning algorithms, especially the emergence of the Transformer architecture, has revolutionized the field of image captioning generation. PureT is a fully Transformer-based image captioning generation model that combines a novel word vector and global image feature pre-fusion mechanism to enhance cross-modal interaction. The Locality-Sensitive Transformer Network (LSTNet) introduces a locality-sensitive attention mechanism and cross-layer semantic fusion, significantly improving the model's performance in capturing local visual information and achieving cross-layer semantic complementarity. Furthermore, the Transformer's exceptional capabilities in computer vision have spawned Visual Transformers, primarily used for extracting and encoding high-level features of images in image captioning tasks.

[0003] These models still have limitations in visual feature extraction and visual-semantic feature alignment. The main reason is that the reliance on a single visual encoder limits their ability to simultaneously capture fine-grained details (e.g., object boundaries and motion trajectories) and global semantics (e.g., scene context and spatial relationships between objects). Furthermore, the alignment between cross-modal visual and semantic features remains suboptimal. Current methods typically perform alignment at only one level, lacking multi-stage deep fusion approaches that can effectively bridge the gap between visual features and linguistic semantics. These limitations prevent decoders from fully utilizing visual data, leading to over-reliance on prior text, and thus, while the generated descriptions may appear coherent, they often fail to accurately represent the semantic and visual details of the image. Summary of the Invention

[0004] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and proposes an image description generation method and system based on two-stage progressive fusion coding.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] In a first aspect, the present invention provides an image description generation system based on two-stage progressive fusion coding, comprising an image encoder, a progressive semantic enhancement module, a global awareness workspace module, an extension module, and a multi-head attention module. The image encoder includes an image encoder Swin Transformer and an image encoder CLIP ViT, both of which are connected to the progressive semantic enhancement module and the global awareness workspace module. The progressive semantic enhancement module is located between the image encoder Swin Transformer and the image encoder CLIP ViT. The extension module includes a static extension module and a mask dynamic extension module. The static extension module is connected to the global awareness workspace module on one hand and to the multi-head attention module on the other. The mask dynamic extension module is connected to the multi-head attention module.

[0007] The image encoder is used to extract fine-grained visual features and global visual semantic features of the image;

[0008] The progressive semantic enhancement module is used to perform linear interpolation alignment and element fusion on the features extracted by the image encoder;

[0009] The global perception workspace module is used to perform unified spatial mapping and weighted fusion of high-level features output by the image encoder.

[0010] The extension module is used to process variable-length feature sequences;

[0011] The multi-head attention module is used to input the fused features into the decoder and generate image descriptions through the multi-head attention mechanism.

[0012] Secondly, this invention provides an image description generation method based on two-stage progressive fusion coding, comprising the following steps:

[0013] Step 1: In the image encoding stage of the image description generation system, the image encoder Swin Transformer and the image encoder CLIP ViT synchronously perform hierarchical encoding on the original image: when the image encoder Swin Transformer extracts the i-th layer (i=1,2,3) multi-scale fine-grained visual features V si Simultaneously, the image encoder CLIP ViT extracts the semantically rich global image features V from the i-th layer. ci ; For this layer V ci Perform linear transformation and linear interpolation to obtain the result with V si Alignment features ;Will With V siElement-by-element addition and fusion are performed to generate the initial fusion feature V of this layer. fi and directly replace V si As input to the image encoder Swin Transformer for encoding the features of the next layer; the image encoder Swin Transformer generates the fourth layer feature V based on the fused features of the first three layers. s4 Meanwhile, the image encoder CLIP ViT completes the fourth layer of feature V. c4 Extraction;

[0014] Step 2: Based on Step 1, process the fourth layer feature V output by the image encoder Swing Transformer. s4 The fourth layer feature V output by the image encoder CLIP ViT c4 The dimensions are aligned through linear transformation, and then the final fused features are generated through weighted fusion.

[0015] Step 3: Perform static feature expansion on the fused features output in Step 2 using the static expansion module to output image features;

[0016] Step 4: In the text encoding stage of the image description generation system, the input raw text is segmented, and the length of the text feature sequence is dynamically adjusted through the mask dynamic expansion module to output the text features;

[0017] Step 5: In the description generation stage of the image description generation system, the image features output in Step 3 and the text features output in Step 4 are input into the multi-head attention mechanism to calculate the cross-modal attention weights, realize the deep fusion of visual and semantic features, and the fused features are decoded word by word to generate natural language descriptions, and generate complete descriptions through autoregression.

[0018] Furthermore, in step 1, the linear transformation and linear interpolation processing includes:

[0019] By adjusting the channel dimensions of the features encoded by the CLIP ViT image encoder, linear transform features are obtained: ;

[0020] Where B is the batch size, L ci C is the sequence length of the CLIP ViT encoded features in the image encoder. si The number of channels for encoding features for the image encoder Swin Transformer is processed through linear layers to match the number of channels C. si ;

[0021] After channel adjustment The features are interpolated by sequence length to obtain the interpolated features:

[0022] ;

[0023] Where B is the batch size, L si C is the sequence length of the features encoded by the image encoder Swin Transformer. si The number of channels for encoding features for the image encoder Swin Transformer is processed through interpolation operations to match the feature sequence length L. si .

[0024] Furthermore, step 2 specifically includes:

[0025] The fourth layer feature V encoded by the image encoder CLIP ViT c4 Expand dimensions:

[0026] This makes its dimension the same as the fourth layer feature V encoded by the image encoder Swing Transformer. s4 The dimensions are consistent, among which, For linear transformation, For extended transformation;

[0027] For V s4 The residual join and normalization processes are performed using the following formulas:

[0028] ;

[0029] Among them, V s4 The fourth layer feature encoded by the Swing Transformer image encoder. For layer normalization operation;

[0030] right The residual join and normalization processes are performed using the following formulas:

[0031] ;

[0032] in, The encoding features of the dimension-expanded image encoder CLIP ViT. For layer normalization operation;

[0033] We use weight g to perform weighted summation of features:

[0034] ;

[0035] Among them, F S and F C V after residual connection and layer normalization s4 and Features, Ff denoted as , and g as the weight.

[0036] Furthermore, the implementation process of the static extension module and the mask dynamic extension module is as follows:

[0037] Define a set of learnable parameters E Q E B and the expansion factor N t E Q E is a learnable query vector. B It is a learnable bias vector;

[0038] In the static extension module, the query vector Q is extended. E and bias vector B E They are respectively equal to E Q and E B , used to extend computation;

[0039] In the dynamic expansion module, the query vector Q is expanded. E and bias vector B E The result is obtained through the following formula:

[0040] ;

[0041] H E It is a matrix with a specific structure used to compute Q. E and B E It consists of multiple unit vectors and a zero vector, and the length of these vectors is N. t I E It is by using L units of size The matrix is ​​formed by concatenating the identity matrices column by column, where L is the input length and C represents the linear projection of the input.

[0042] Furthermore, in step 4, dynamically adjusting the length of the text feature sequence includes:

[0043] pass Double-expansion generates variable-length sequences.

[0044] Compared with existing technologies, the image description generation method and system based on two-stage progressive fusion coding described in this invention have the following advantages:

[0045] This invention proposes an image description system based on a two-stage progressive fusion coding approach. This system integrates the image encoder Swin Transformer and the image encoder CLIP ViT to enhance image description generation performance. By combining CLIP-coded features with Swin Transformer visual representations through multiple steps, this system effectively addresses the technical challenges of visual-language modality mismatch and insufficient visual features.

[0046] This invention designs a global perception workspace module that can map the high-level features output by the image encoder Swin Transformer and the image encoder CLIP ViT to a unified feature space, and use a weighted fusion strategy to fuse the features, further fusing visual features.

[0047] To address the limitations of fixed-length feature sequences, which often restrict the performance of image description generation systems, this invention introduces an extension mechanism to achieve efficient processing of longer and more complex input sequences. Attached Figure Description

[0048] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0049] Figure 1 This is a flowchart illustrating the method described in this invention.

[0050] Figure 2 This is a schematic diagram of the system described in this invention.

[0051] Figure 3 This is a schematic diagram of the structure of the extension module described in this invention. Detailed Implementation

[0052] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0053] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0054] Example 1:

[0055] Please see Figure 2A preferred embodiment of the present invention provides an image description generation system based on two-stage progressive fusion coding, including an image encoder, a progressive semantic enhancement module, a global awareness workspace module, an extension module, and a multi-head attention module. The image encoder includes an image encoder Swin Transformer and an image encoder CLIP ViT. Both the image encoder Swin Transformer and the image encoder CLIP ViT are connected to the progressive semantic enhancement module and the global awareness workspace module. The progressive semantic enhancement module is located between the image encoder Swin Transformer and the image encoder CLIP ViT. The extension module includes a static extension module and a mask dynamic extension module. The static extension module is connected to the global awareness workspace module on one side and the multi-head attention module on the other side. The mask dynamic extension module is connected to the multi-head attention module.

[0056] The image encoder is used to extract fine-grained visual features and global visual semantic features of the image;

[0057] The progressive semantic enhancement module is used to perform linear interpolation alignment and element fusion on the features extracted by the image encoder;

[0058] The global perception workspace module is used to perform unified spatial mapping and weighted fusion of high-level features output by the image encoder.

[0059] The extension module is used to process variable-length feature sequences;

[0060] The multi-head attention module is used to input the fused features into the decoder and generate image descriptions through the multi-head attention mechanism.

[0061] like Figure 1 As shown, another preferred embodiment of the present invention provides an image description generation method based on two-stage progressive fusion coding, which overcomes the limitations in visual feature extraction and visual semantic feature alignment, and includes the following steps:

[0062] S1: Original image feature dual encoder encoding and progressive semantic enhancement in the first three stages

[0063] Input: The original image is input into the image encoder Swin Transformer and the image encoder CLIP ViT for image feature extraction.

[0064] deal with:

[0065] In the image encoding stage of the image description generation system, the image encoder Swin Transformer and the image encoder CLIP ViT synchronously perform hierarchical encoding on the original image: when the image encoder Swin Transformer extracts the i-th layer (i=1,2,3) multi-scale fine-grained visual features V si Simultaneously, the image encoder CLIP ViT extracts the semantically rich global image features V from the i-th layer. ci Then, this layer V... ci Perform linear transformation and linear interpolation to obtain the result with V si Alignment features Next, With V si Element-by-element addition and fusion are performed to generate the initial fusion feature V of this layer. fi and directly replace V si This serves as input for the image encoder Swin Transformer to encode the features of the next layer; ultimately, the image encoder Swin Transformer generates the fourth layer feature V based on the fused features of the first three layers. s4 Meanwhile, the image encoder CLIP ViT completes the fourth layer of feature V. c4 Extraction.

[0066] The fused features retain fine-grained visual details while injecting global semantic information, achieving progressive semantic enhancement and improving the semantic understanding ability of low-level features.

[0067] In this embodiment, the linear transformation and linear interpolation processing includes:

[0068] By adjusting the channel dimensions of the features encoded by the CLIP ViT image encoder, linear transform features are obtained: ;

[0069] Where B is the batch size, L ci C is the sequence length of the CLIP ViT encoded features in the image encoder. si The number of channels for encoding features for the image encoder Swin Transformer is processed through linear layers to match the number of channels C. si ;

[0070] After channel adjustment The features are interpolated by sequence length to obtain the interpolated features:

[0071] ;

[0072] Where B is the batch size, L si C is the sequence length of the features encoded by the image encoder Swin Transformer.si The number of channels for encoding features for the image encoder Swin Transformer is processed through interpolation operations to match the feature sequence length L. si .

[0073] S2: Global Aware Workspace (GPW) Final Feature Fusion

[0074] Objective: To deeply integrate complementary features of the image encoder Swin Transformer and the image encoder CLIP ViT in a high-level feature space.

[0075] deal with:

[0076] The fourth layer features (V) encoded by the image encoder Swin Transformer s4 ) and the fourth layer features encoded by the CLIP ViT image encoder (V c4 First, align the dimensions using a linear transformation, then perform a weighted fusion, specifically:

[0077] The fourth layer feature V encoded by the image encoder CLIP ViT c4 Expand dimensions:

[0078] This makes its dimension the same as the fourth layer feature V encoded by the image encoder Swing Transformer. s4 The dimensions are consistent, among which, For linear transformation, For extended transformation;

[0079] For V s4 The residual join and normalization processes are performed using the following formulas:

[0080] ;

[0081] Among them, V s4 The fourth layer feature encoded by the Swing Transformer image encoder. For layer normalization operation;

[0082] right The residual join and normalization processes are performed using the following formulas:

[0083] ;

[0084] in, The encoding features of the dimension-expanded image encoder CLIP ViT. For layer normalization operation;

[0085] We use weight g to perform weighted summation of features:

[0086] ;

[0087] Among them, F S and F C V after residual connection and layer normalization s4 and Features, F f denoted as , and g as the weight.

[0088] This module achieves deep feature fusion alignment by adjusting the feature fusion ratio between fine-grained details (such as object shape) encoded by the image encoder Swin Transformer and semantically rich global image features (such as scene category) encoded by the image encoder CLIP ViT.

[0089] S3: Static extension module processes the sequence length of fused features output from the Global Perception Workspace (GPW). Goal: To overcome the constraint of fixed visual feature length on system performance.

[0090] deal with:

[0091] Static expansion: directly expanding the feature sequence length by learning the query vector (e.g., expansion factor N). t When the input length is 3, the sequence length becomes 3×L (where L is the original input length), which does not depend on the input length and enhances the system's ability to process long sequence visual features.

[0092] S4: Dynamic expansion of raw text input and mask autoregression

[0093] Input: The purpose of the original text input is to provide semantic information. After word segmentation, it is used as an input mask for dynamic expansion modules.

[0094] deal with:

[0095] Text feature length adjustment: Dynamically adjust the length of the text feature sequence (e.g.) This enables cross-modal feature length matching.

[0096] Dynamic expansion: combining input length L and expansion factor N t (e.g. N) t When L=2 and L=4, the sequence length is 8), and the autoregressive properties are maintained through a masking mechanism to ensure temporal consistency after feature expansion.

[0097] Autoregressive Mask Calculation: The triangular mask ensures the autoregressiveness of text generation (i.e., the generation of the t-th output depends only on the previous t-1 outputs), avoiding future information leakage and improving the temporal rationality of description generation.

[0098] S5: Multi-head attention cross-modal fusion and description generation

[0099] Cross-modal feature alignment: The expanded image features (from step S3) and text features (from step S4) are input into the multi-head attention mechanism, and cross-modal attention weights are calculated through key-value query to achieve deep fusion of visual and semantic features.

[0100] Description generation: The fused features are decoded word by word (token) to generate a natural language description. The complete description is generated through an autoregressive method (generating one token each time, updating the input sequence and iterating).

[0101] like Figure 3 As shown, in this embodiment, key components designed to address the fixed-length limitation of the input sequence through static expansion and masked dynamic expansion enhance the system's ability to handle variable-length complex inputs. These components are integrated into the encoder and decoder architectures, respectively. In static expansion, the sequence length is determined by the expansion factor N. t Directly determined (e.g., N) t When the length is 4, the output length is 4; define a learnable query vector E. Q and bias vector E B (All belong to) d m (where Q is the feature dimension) and the extended query vector Q used in subsequent extended computation. E and bias vector B E They are respectively equal to E Q and E B In dynamic mask expansion, the query vector Q is expanded. E and bias vector B E After the BroadSum operation is combined with the input linear projection feature C, the length of the expanded sequence is N. t L (L is the input length, such as N) t When L=3 and L=3, the output length is 9), and a masking mechanism is used to maintain the autoregressive property.

[0102] The implementation process of the extended module of this invention is as follows:

[0103] Feature processing is achieved through a forward expansion process: first, the input is linearly projected to obtain K, V1, and V2, where parameters K and Q are... E Used for calculating the extended matrix M, with parameters V1, V2, and B. E Used for extended sequences The calculation first involves K and Q. E Perform matrix multiplication to calculate the extended matrix M (this matrix is ​​used for...) The calculation, (as intermediate parameters), and obtained through ReLU activation calculation and normalization calculation. (It serves as an intermediate bridge connecting the extended matrix and the final extended sequence, and is further used for...) (calculation), ultimately through V1, V2, B E The extended sequence is calculated. Subsequently, through a backward expansion process, using M T The original sequence length is restored, and the results are combined using a gating mechanism to complete feature expansion.

[0104] In this embodiment, the construction method is verified:

[0105] 1. Dataset and Evaluation Methods

[0106] The system was trained and evaluated using the MSCOCO dataset, which contains 123,287 images, of which 82,783 were used for training and 40,504 for validation. Each image is accompanied by 5 manually annotated text descriptions. For ease of comparison, the experiment followed the Karpathy partitioning rule, dividing the dataset into 113,287 training images, 5,000 validation images, and 5,000 test images. The evaluation metrics employed were five standard measures in the field of image caption generation: BLEU (measuring the n-gram matching degree between generated text and reference text), METEOR (evaluation based on word semantic matching), ROUGE-L (evaluating long-distance semantic coherence), CIDEr-D (consensus-based semantic consistency score), and SPICE (evaluation of semantic propositions analyzing scene graph structure), comprehensively covering dimensions such as grammatical accuracy, semantic richness, and structural rationality.

[0107] 2. Establish benchmarks

[0108] On the MSCOCO dataset, the method of this invention was compared and analyzed with various cutting-edge image captioning generation methods from both domestic and international sources. The comparison methods included traditional CNN-RNN architecture models (such as AoANet, Up-Down), pure Transformer models (such as PureT, X-Transformer), methods based solely on the Swing Transformer (such as CropCap, OSIC), and models incorporating contrastive learning (such as DCCT, ExpansionNet v2). All baseline models underwent multi-metric evaluation (including BLEU, METEOR, ROUGE-L, CIDEr-D, SPICE) under identical experimental conditions to ensure that performance differences accurately reflect the advantages and disadvantages of model architectures in core aspects such as visual-semantic fusion and feature alignment.

[0109] 3. Main Results

[0110] As shown in Table 1, the method of this invention achieves optimal performance on the MSCOCO dataset: in the single-model main experiment comparison, CIDEr-D improves by 2.2% compared to the fusion-based contrastive learning method DCCT; and by 5.3% compared to the SwinTransformer-based method OSIC. As shown in Table 2, in the four-model ensemble experiment comparison, CIDEr-D improves by 1.7% compared to the fusion-based contrastive learning method DCCT.

[0111] Table 1 compares the experimental performance of the methods on the MS-COCO dataset (single-model experimental results). The abbreviations in the table are explained as follows: B=BLEU (Bilingual Evaluation Alternative): assesses accuracy by calculating the n-gram overlap between the generated and reference descriptions. B@1 (BLEU-1) is based on single-word matching, and B@4 (BLEU-4) is based on matching four consecutive words, reflecting word accuracy and sentence fluency / coherence, respectively; M=METEOR (Meteor Evaluation Metric): focuses on semantic matching, combining stemming and synonym matching, and evaluates the semantic consistency between the generated and reference descriptions by harmonizing precision and recall; R=ROUGE-L (ROUGE-L Evaluation Metric): with recall as the core, it measures the semantic and structural similarity between the generated and reference descriptions through the longest common subsequence length, suitable for evaluating the semantic integrity of long texts; C=CIDEr-D (CIDEr-D Evaluation Metric): designed for image descriptions, calculating n-grams based on TF-IDF weights. Similarity, with the addition of diversity penalty, assesses the consistency and diversity of the description and the reference consensus; S=SPICE (SPICE evaluation index): parses the description into semantic propositions, calculates the matching degree between the generated propositions and the reference propositions, and assesses the ability to understand deep semantics such as entity interactions and attributes.

[0112] Table 1

[0113]

[0114] Table 2 shows the experimental performance comparison of the methods on the MS-COCO dataset (experimental results of four-model ensemble).

[0115] Table 2

[0116]

[0117] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. An image description generation system based on two-stage progressive fusion coding, characterized in that: The image encoder includes an image encoder Swin Transformer and an image encoder CLIP ViT, both of which are connected with the progressive semantic enhancement module and the global perception workspace module, the progressive semantic enhancement module is between the image encoder Swin Transformer and the image encoder CLIP ViT, the expansion module includes a static expansion module and a mask dynamic expansion module, the static expansion module is connected with the global perception workspace module on one hand and connected with the multi-head attention module on the other hand, and the mask dynamic expansion module is connected with the multi-head attention module; The image encoder is used for extracting fine-grained visual features and global visual semantic features of an image; The progressive semantic enhancement module is used for linear interpolation alignment and element fusion of the features extracted by the image encoder; The global perception workspace module is used for unified space mapping and weighted fusion of high-level features output by the image encoder; The expansion module is used for processing variable-length feature sequences; The multi-head attention module is used for inputting the fused features into a decoder to generate an image description through a multi-head attention mechanism.

2. An image description generation method based on two-stage progressive fusion coding, characterized in that: The method comprises the following steps: Step 1: In the image encoding stage of the image description generation system, the image encoder Swin Transformer synchronously encodes the original image with the image encoder CLIP ViT: when the image encoder Swin Transformer extracts the i-th layer (i = 1, 2, 3) multi-scale fine-grained visual features V si , the image encoder CLIP ViT synchronously extracts the i-th layer global image features rich in semantic information V ci ; linear transformation and linear interpolation processing are performed on the layer V ci to obtain the features V si aligned with V ; and V and V si are element-wise added and fused to generate the preliminary fusion features V fi of the layer, which directly replaces V si as the input of the image encoder Swin Transformer to encode the next layer features; The image encoder SwinTransformer generates the fourth layer feature V based on the fused features of the first three layers s4 Meanwhile, the image encoder CLIP ViT completes the extraction of the fourth layer feature V c4 ​ Step 2: On the basis of step 1, the fourth layer features V output by the image encoder Swin Transformer are fused with the fourth layer features V output by the image encoder CLIP ViT s4 and the fourth layer features V output by the image encoder CLIP ViT c4 The dimensions are aligned by linear transformation, and the final fused features are generated by weighted fusion. Step 3: performing feature static expansion on the fused features output in step 2 through the static expansion module to output image features; Step 4: performing word segmentation processing on the input original text in the text encoding stage of the image description generation system, dynamically adjusting the length of the text feature sequence through the mask dynamic expansion module, and outputting text features; Step 5: in the description generation stage of the image description generation system, inputting the image features output in step 3 and the text features output in step 4 into the multi-head attention mechanism, calculating cross-modal attention weights, realizing deep fusion of visual-semantic features, and generating natural language descriptions through a decoder, and generating complete descriptions through an autoregressive manner.

3. The image description generation method based on two-stage progressive fusion coding according to claim 2, characterized in that: In step 1, the linear transformation and linear interpolation processing include: The channel dimension of the feature coded by the image encoder CLIP ViT is adjusted to obtain linear transformation features: ; where B is batch size, L ci is the sequence length of the image encoder CLIP ViT encoding features, C si is the number of channels of the image encoder Swin Transformer encoding features, processed through a linear layer to match the number of channels C si ; channel-adjusted characteristics are interpolated with sequence length to obtain interpolated characteristics: ； where B is batch size, L si is the sequence length of the image encoder Swin Transformer encoding features, C si is the number of channels of the image encoder Swin Transformer encoding features, processed by an interpolation operation to match the feature sequence length L si .

4. The image description generation method based on two-stage progressive fusion coding according to claim 2, characterized in that: Step 2 specifically includes: The 4th layer feature V encoded by the image encoder CLIP ViT c4 Dimension expansion is performed: to the dimension of the 4th layer feature V s4 wherein, is a linear transformation, is an expansion transformation; For V s4 Residual connection and normalization are performed, and the specific formula is as follows: ； wherein V s4 is the 4th layer feature of the image encoder Swin Transformer encoding, is the layer normalization operation; right The residual join and normalization processes are performed using the following formulas: ； wherein, is the encoded feature of the dimensionally expanded image encoder CLIP ViT, is a layer normalization operation; The weighted sum of features is realized by using a weight g: ； where F S and F C are the V s4 s after residual connection and layer normalization processing characteristics, F f is the final fusion feature, and g is the weight.

5. The image description generation method based on two-stage progressive fusion coding according to claim 2, characterized in that: The implementation process of the static expansion module and the mask dynamic expansion module is as follows: define a set of learnable parameters E Q , E B and an expansion factor N t , E Q is a learnable query vector, E B is a learnable bias vector; In the static extension module, the extension query vector Q E and the bias vector B E are respectively equal to E Q and E B , used for extension calculation; In the dynamic expansion module, the expansion query vector Q E and the bias vector B E is calculated by the following formula: ； H E is a matrix of a specific structure for calculating Q E and B E is composed of a plurality of unit vectors and zero vectors, the length of which is N t , I E is a matrix obtained by concatenating L unit matrices of size by column, L being the input length, C indicating the linear projection of the input.

6. The image description generation method based on two-stage progressive fusion coding according to claim 5, characterized in that: In step 4, dynamically adjusting the length of the text feature sequence includes: By The length of the sequence is extended by a factor of two.

Images