Image description system based on dual-path dual-layer feature encoder

Abstract

The application relates to an image description system based on a dual-path dual-layer feature encoder, comprising a geometry and vision fusion module for embedding position information of image features; a feature enhancement encoder module based on a self-attention network, which combines regional features and grid features to complementarily enhance feature fusion; a feature fusion encoder module based on a dual-layer Transformer, which is used for modeling and fusing regional features and grid features; a feature fusion module based on a multivariate residual network, which fuses the output of the feature fusion encoder module to obtain encoding information; and a decoder module based on a Transformer, which decodes the encoding information. The application can better generate corresponding description sentences according to image content.

Description

Technical Field

[0001] This invention relates to the field of image description, and more specifically to an image description system based on a dual-channel, dual-layer feature encoder. Background Technology

[0002] In recent years, image description, as an interdisciplinary research area of ​​computer vision and natural language processing, has become a popular cross-modal research problem. When analyzing the visual content of an image, image description needs to accurately capture salient objects, important attributes, and relationships between objects, while also capturing the semantic information of the image to complete the cross-modal conversion from image to text, generating grammatically correct and fluent textual descriptions in human language. Image description can provide assistance to visually impaired individuals and plays a crucial role in applications such as intelligent information dissemination and monitoring, and image retrieval, possessing significant theoretical research value and practical application value.

[0003] Current methods for image description based on deep learning mostly employ an encoder-decoder architecture. The encoder encodes the image into intermediate vector features, and the decoder then decodes these features to generate corresponding descriptive sentences. Early work typically used convolutional neural networks (CNNs) as encoders to extract fixed-size grid features covering the entire image, providing complete object information and detailed background information. More recently, object detection networks (Faster R-CNNs) have been used as encoders, extracting region features to contribute object-level information, significantly reducing the difficulty of visual semantic embedding. After obtaining the encoder results, recurrent neural networks (RNNs) are used as decoders to generate descriptive sentences. However, because the visual encoder is a CNN and the language decoder is an RNN, the encoder-decoder architecture is heterogeneous, and CNNs and RNNs do not share basic network components, resulting in low-quality generated text descriptions. To address this, a Transformer structure is introduced as both the visual encoder and language decoder. This homogeneous architecture facilitates end-to-end training and improves the quality of the model-generated descriptions.

[0004] However, most studies only use one type of image feature. Grid features fail to accurately identify salient objects in the image, while region features lose some details and background information, resulting in a lack of contextual and fine-grained information. This is the limitation of using grid features and region features alone. Moreover, both types of features can obtain information about their own shortcomings from the other feature. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide an image description system based on a dual-channel, dual-layer feature encoder, which aims to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An image description system based on a dual-channel, dual-layer feature encoder, including

[0008] A geometry-vision fusion module is used to embed location information of image features;

[0009] The feature enhancement encoder module based on the self-attention network combines region features and grid features to complementarily enhance feature fusion.

[0010] A feature fusion encoder module based on a two-layer Transformer is used to model and fuse region features and mesh features;

[0011] The feature fusion module based on the multivariate residual network fuses the output of the feature fusion encoder module to obtain encoded information.

[0012] The decoder module based on Transformer decodes the encoded information.

[0013] Furthermore, the geometry-vision fusion module uses Faster R-CNN pre-trained on the VG dataset to extract region features and grid features, embedding absolute and relative geometric information of the input features to simulate the complex visual and positional relationships between the input features.

[0014] Furthermore, the feature enhancement encoder module based on the self-attention network combines region features and grid features to complementaryly enhance feature fusion, specifically as follows:

[0015] The Dot-Self-Gate module is used to enhance the region features, and the DSG module is used to consider the distribution of attention within the object. The formula is as follows:

[0016] H1(V R )=σ(W v1 V R )⊙W v2 V R

[0017] H2(V R ) = softmax(tanh(H1(V R )))

[0018] H3(V R =LayerNorm(H2(V) R ))

[0019] Among them, W v1 and W v2It is a learnable matrix, σ represents the sigmoid activation function, ⊙ is the Hadamard product, and V R H1(V) represents the region features of the input image. R H2(V) R ), H3(V R () represents the hidden state of regional features during the computation process;

[0020] Next, both the region features and the grid features pass through the self-attention module simultaneously:

[0021] Q r K r V r =Linears(V G )

[0022] Q g K g V g =Linears(DSG(V R ))

[0023] Among them, V R This indicates the output after passing through the DSG module, V G Q represents the grid features of the input image, and Q, K, and V represent the vectors of the image features after passing through a linear layer.

[0024] Scoring is calculated using queries and keys that fuse absolute positions with multi-head attention:

[0025]

[0026] Then, the relative position information is used to adjust the attention weights:

[0027] W′ ij =W ij +log(Ω(i,j))

[0028] Finally, a secondary attention module is introduced to filter and enhance the node features encoded by the self-attention module.

[0029] Furthermore, the feature fusion encoder module based on a two-layer Transformer uses two independent attention modules to simultaneously model the intra-layer relationships of region features and grid features, mining their intrinsic attribute information, specifically as follows:

[0030]

[0031]

[0032] in, The hidden state represents the input region features. The hidden state represents the input mesh features, where l represents the layer number; RPE and GPE represent the absolute position matrix of the mesh in the region, respectively, and Ω... rr and Ω gg These represent the relative position matrices of the region and the grid, respectively.

[0033] Then, both the region features and the grid features are processed by a feedforward neural network (FFN).

[0034]

[0035]

[0036] Among them, C l C′ represents the hidden state after passing through the self-attention network. (l) This represents the hidden layer state after passing through the feedforward neural network, where l represents the number of hidden layers.

[0037] Subsequently, local cross-attention is used to simulate the complex interaction between regions and the mesh. Regions are embedded into the mesh to supplement contextual and fine-grained information, while the mesh is embedded into regions to supplement object-level information, thus achieving inter-layer fusion. At the same time, alignment maps are introduced to effectively constrain visual features.

[0038]

[0039]

[0040] Among them, Ω rg Ω represents the relative position matrix between the region and the grid. gr G represents the relative position matrix between the grid and the region; M represents the alignment map constructed from the image geometry; l represents the output after local cross attention; and l represents the number of hidden layers.

[0041] Then, the two types of features are processed by a feedforward neural network (FFN).

[0042]

[0043]

[0044] Furthermore, the feature fusion module based on multivariate residual networks uses multivariate residual networks to fuse the enhanced features with the features fused by the Transformer.

[0045] Furthermore, the multivariate residual network includes a residual projection fusion module and a multilinear relationship fusion module. The residual projection fusion module strengthens the relationship between input and output, and then the multilinear relationship fusion module is used to learn the multilinear relationships between features.

[0046] Furthermore, the Transformer-based decoder module uses Transformer to decode the encoder information and generate a description statement; Ground Truth is first modeled as a vector q through a layer of self-attention, and then the intermediate vector generated by the encoder is used as k and v, which are then passed through Cross Attention to obtain the output vector.

[0047] Furthermore, we first use the cross-entropy loss function to measure the difference between the word sequence generated by the model and the real sequence:

[0048]

[0049] in, This refers to the word generated at time t. p represents the Ground Truth sequence. θ These are the model hyperparameters.

[0050] Then, reinforcement learning is used to further optimize the non-differentiable CIDEr-D score:

[0051]

[0052] Where k is the size of the Beam Search, r is the CIDEr-D scoring function, and b is the baseline. p represents the Ground Truth sequence. θ These are the model hyperparameters.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] This invention can better utilize regional features and grid features, fully realize the complementary advantages of the two features, generate more accurate and richer intermediate vector information, and thus generate more precise representation results. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the system of the present invention;

[0056] Figure 2 This is a feature enhancement encoder module in one embodiment of the present invention;

[0057] Figure 3 This is a feature fusion encoder module in one embodiment of the present invention. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] Please refer to Figure 1This invention provides an image description system based on a dual-channel, dual-layer feature encoder, including...

[0060] A geometry-vision fusion module is used to embed location information of image features;

[0061] The feature enhancement encoder module based on the self-attention network combines region features and grid features to complementarily enhance feature fusion.

[0062] A feature fusion encoder module based on a two-layer Transformer is used to model and fuse region features and mesh features;

[0063] The feature fusion module based on the multivariate residual network fuses the output of the feature fusion encoder module to obtain encoded information.

[0064] The decoder module based on Transformer decodes the encoded information.

[0065] In this embodiment, preferably, each module is as follows:

[0066] 1) A fusion module based on geometry and vision

[0067] First, Faster R-CNN, pre-trained on the VG dataset, is used to extract features. For grid features, they are averaged and pooled to a size of 7×7. For region features, the same model is used to extract features in 2048 dimensions.

[0068] Next, a Transformer is used to model both types of features simultaneously. Since the Transformer is entirely based on Self-Attention, and Self-Attention cannot capture word positional information, but the positional information of words within a sentence is crucial, this embodiment uses DLCT to simultaneously embed corresponding absolute and relative geometric information into both features to simulate the complex visual and positional relationships between input features, thus strengthening the modeling of intra-layer and inter-layer relationships.

[0069] Absolute position indicates the specific location of an object in an image, allowing the model to quickly locate it. If multiple similar visual features are found but their absolute positions differ significantly, this helps the model distinguish whether they are the same object. Sine and cosine embeddings are used to establish absolute position information for grid features, using the coordinates of the top-left and top-right corners of the bounding box to determine the absolute position of the region features.

[0070] Relative positions can better integrate visual features. We add a 4-dimensional vector (x, y, w, h) to the bounding box geometry of both grid features and region features, representing the center coordinates, width, and height of the bounding box, respectively.

[0071] 2) Feature enhancement encoder module based on self-attention network

[0072] Both region features and grid features of the image are extracted by a pre-trained model. For region features, there is also some messy and irrelevant background information. For grid features, the extraction of detailed information is interfered with by significant object information. Therefore, this embodiment proposes a feature enhancement coding module based on these two features to enhance the two input features, remove the less important information of each feature, enhance their respective advantages, and prepare for feature fusion to maximize the advantages.

[0073] First, the Dot-Self-Gate module is used to enhance the region features. Since region features are often located using bounding boxes, if the located bounding boxes contain irrelevant background information or partial information about other objects, it may affect the feature extraction of that object. Therefore, the DSG module is used to consider the distribution of attention within the object, eliminating irrelevant information in the region features and ensuring accurate object-level information is extracted. The formula is as follows:

[0074] H1(V G )=σ(W v1 V G )⊙W v2 V G

[0075] H2(V G )=sofrmax(tanh(H1(V G )))

[0076] H3(V G =LayerNorm(H2(V) G ))

[0077] Among them, W v1 and W v2 It is a learnable matrix, where σ represents the sigmoid activation function, V represents the image features, H represents the hidden state of the region features, and l represents the current layer number;

[0078] Next, both features are simultaneously processed through a self-attention module to further explore their intrinsic properties and enhance the model's ability to capture key information.

[0079] Q r K r V r =Linears(V R)

[0080] Q g K g V g =Linears(DSG(V G ))

[0081] Scoring is calculated using queries and keys that fuse absolute positions with multi-head attention:

[0082]

[0083] Then, the relative position information is used to adjust the attention weights:

[0084] W′ ij =W ij +log(Ω(i,j))

[0085] Finally, a secondary attention module (AoA) is introduced to filter and enhance the node features encoded by the self-attention module. The AoA module can help establish relationships between different attention heads and extract the correct parts from the attention results.

[0086] 3) Feature fusion encoder module based on two-layer Transformer

[0087] By using two independent attention modules, we can simultaneously model the intra-layer relationships of region features and grid features, and mine their intrinsic attribute information.

[0088]

[0089]

[0090] in, RPE and GPE represent the absolute position matrix of the grid in the region, respectively. rr and Ω gg represents the relative position matrix of the region and the grid, respectively, and l represents the layer number.

[0091] Then, both types of features are processed through a feedforward neural network (FFN).

[0092]

[0093]

[0094] Then, local cross-attention is used to simulate the complex interaction between regions and grids. Regions are embedded into grids to supplement contextual and fine-grained information, while grids are embedded into regions to supplement object-level information, so as to achieve inter-layer fusion. At the same time, alignment maps are introduced to effectively constrain visual features and avoid semantic noise caused by feature fusion.

[0095]

[0096]

[0097] Where Ω represents the relative position matrix between the region and the grid, and G represents the alignment map constructed from the image geometry information.

[0098] Then, the two types of features are processed through a feedforward neural network (FFN).

[0099]

[0100]

[0101] 4) Feature fusion module based on multivariate residual network

[0102] This embodiment uses a multivariate residual network (MRM) to fuse the enhanced features with the features fused by the Transformer, resulting in richer encoded information. This network mainly consists of two modules: a residual projection fusion module and a multilinear relationship fusion module. The residual module strengthens the relationship between input and output, and then the multilinear relationship fusion module learns the multilinear relationships between features. This allows for the fusion of region features and grid features without interference, while simultaneously obtaining rich vector information, leveraging the advantages of a two-channel encoder.

[0103] 5) Transformer-based decoder module

[0104] This paper uses Transformer to decode encoder information and generate descriptive statements; Ground Truth is first modeled as vector q through a layer of self-attention, and then the intermediate vector generated by the encoder is used as k and v, which are then passed through CrossAttention to obtain the output vector.

[0105] 6) Loss Function

[0106] In this embodiment, the cross-entropy loss function is first used to measure the difference between the word sequence generated by the model and the real sequence.

[0107]

[0108] Where yt represents the word generated at time t, and θ is the model hyperparameter;

[0109] Then, the non-differentiable CIDEr-D score is further optimized through reinforcement learning;

[0110]

[0111] Where k is the size of the Beam Search, r is the CIDEr-D scoring function, and b is the baseline.

[0112] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. An image description system based on a dual-channel, dual-layer feature encoder, characterized in that, include A geometry-vision fusion module is used to embed location information of image features; The feature enhancement encoder module based on the self-attention network combines region features and grid features to complementarily enhance feature fusion. A feature fusion encoder module based on a two-layer Transformer is used to model and fuse region features and mesh features; The feature fusion module based on the multivariate residual network fuses the output of the feature fusion encoder module to obtain encoded information. The Transformer-based decoder module decodes the encoded information. The feature enhancement encoder module based on a self-attention network combines region features and grid features to complementarily enhance feature fusion, specifically as follows: The Dot-Self-Gate module is used to enhance the region features, and the DSG module is used to consider the distribution of attention within the object. The formula is as follows: H1(V R )=σ(W v1 V R )⊙W v2 V R H2(V R )=softmax(tanh(H1(V R ))) H3(V R )=LayerNorm(H2(V R )) Among them, W v1 and W v2 It is a learnable matrix, σ represents the sigmoid activation function, ⊙ is the Hadamard product, and V R H1(V) represents the region features of the input image. R H2(V) R ), H3(V R () represents the hidden state of regional features during the computation process; Next, both the region features and the grid features pass through the self-attention module simultaneously: Q r ,K r ,V r =Linears(V G ) Q g ,K g ,In g =Linears(DSG(V R )) Among them, DSG(V R ) indicates the output after passing through the DSG module, V G The grid features of the input image are represented by Q, K, and V, which represent the vectors of the image features after passing through a linear layer. Scoring is calculated using queries and keys that fuse absolute positions with multi-head attention: Then, the relative position information is used to adjust the attention weights: W′ ij =W ij +log(Ω(i,j)) Finally, a secondary attention module is introduced to filter and enhance the node features encoded by the self-attention module; The feature fusion module based on multivariate residual networks uses multivariate residual networks to fuse the enhanced features with the features fused by Transformer; The multivariate residual network includes a residual projection fusion module and a multilinear relationship fusion module. The residual projection fusion module strengthens the relationship between input and output, and then the multilinear relationship fusion module is used to learn the multilinear relationships between features.

2. The image description system based on a dual-channel, dual-layer feature encoder according to claim 1, characterized in that, The geometry-vision fusion module uses Faster R-CNN pre-trained on the VG dataset to extract region and grid features, embedding absolute and relative geometric information of the input features to simulate the complex visual and positional relationships between the input features.

3. The image description system based on a dual-channel, dual-layer feature encoder according to claim 1, characterized in that, The feature fusion encoder module based on a two-layer Transformer uses two independent attention modules to simultaneously model the intra-layer relationships of region features and grid features, and mine their intrinsic attribute information, specifically: in, The hidden state represents the input region features. The hidden state represents the input mesh features, where l represents the layer number; RPE and GPE represent the absolute position matrix of the mesh in the region, respectively, and Ω... rr and Ω gg These represent the relative position matrices of the regions and grids, respectively. Then, both the region features and the grid features are processed by a feedforward neural network (FFN). Among them, C l C' represents the hidden state after passing through the self-attention network. (l) This represents the hidden layer state after passing through the feedforward neural network, where l represents the number of hidden layers. Subsequently, local cross-attention is used to simulate the complex interaction between regions and the mesh. Regions are embedded into the mesh to supplement contextual and fine-grained information, while the mesh is embedded into regions to supplement object-level information, thus achieving inter-layer fusion. At the same time, alignment maps are introduced to effectively constrain visual features. Among them, Ω rg Ω represents the relative position matrix between the region and the grid. gr G represents the relative position matrix between the grid and the region; M represents the alignment map constructed from the image geometry; l represents the output after local cross attention; and l represents the number of hidden layers. Then, the two types of features are processed by a feedforward neural network (FFN).

4. The image description system based on a dual-channel, dual-layer feature encoder according to claim 1, characterized in that, The Transformer-based decoder module uses Transformer to decode encoder information and generate descriptive statements; Ground Truth is first modeled as vector q through a layer of self-attention, and then the intermediate vector generated by the encoder is used as k and v, which are then passed through CrossAttention to obtain the output vector.

5. The image description system based on a dual-channel, dual-layer feature encoder according to claim 1, characterized in that, First, use the cross-entropy loss function to measure the difference between the word sequence generated by the model and the real sequence: in, This refers to the word generated at time t. p represents the Ground Truth sequence. θ For model hyperparameters; Then, reinforcement learning is used to further optimize the non-differentiable CIDEr-D score: Where k is the size of the Beam Search, r is the CIDEr-D scoring function, and b is the baseline. p represents the Ground Truth sequence. θ These are the model hyperparameters.

Images