Image description system and method based on image-text feature fusion
By introducing graphic feature fusion technology into the image description model, using attention mechanism and feature fusion module, the problems of insufficient cross-modal reasoning capabilities in the decoding stage and lack of feature interaction during the encoding process in the existing technology are solved, and the more Accurate and relevance image description generation.
Patent Information
- Application Number
- CN202411075186.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
The existing image description model lacks complex cross-modal reasoning capabilities during the decoding stage, cannot effectively deal with the semantic gap problem, and lacks feature interaction during the encoding process, which affects the model performance.
The system based on graphic and text feature fusion is adopted, including text feature extraction module, feature extraction module, image encoder module, feature fusion module and decoder module, through word embedding, attention mechanism, sliding window attention mechanism, self-attention mechanism and feature fusion and other technologies to achieve deep fusion and interaction between image features and text features.
The performance of the image description model is improved, the generated description is more accurate and consistent with the input image, the model's cross-modal understanding ability is enhanced, and the relevance and accuracy of image description is improved.
Smart Images

Figure CN119006964A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of automatic generation of image description text, and in particular relates to an image description system and method based on image and text feature fusion. Background Art
[0002] Image description is a cross-modal task that combines computer vision and natural language processing. The goal of computer vision is to require computers to recognize objects in images and the relationship between objects. It is widely used in target detection, semantic segmentation, image classification and other fields. The goal of natural language processing is to require computers to understand text sentences and realize communication between computers and humans. It is widely used in speech recognition, title generation, text classification and other fields. Compared with single-modal tasks, image description tasks are more challenging. The most important challenge is how to process information between two different modalities and how to interact cross-modally. In the process of describing an image, it is necessary to fully understand the information contained in the original image, and then generate a text sentence describing the image content with the correct grammar. At the same time, image description generation has also promoted the research and development of related applications such as cross-modal retrieval of images and texts, assistance for the blind, and automatic generation of medical reports, so it has attracted much attention from scholars.
[0003] In the early stages of research in the field of image description, traditional machine learning-based methods such as description retrieval or filling in templates were mainly used. With the rapid development of deep learning, especially the widespread application of convolutional neural networks (CNNs) and recurrent neural networks (RNNs), the research focus has gradually shifted to deep learning. These deep learning models can capture the relationship between fine-grained image features and text vocabulary through training. At the same time as the introduction of convolutional neural networks and recurrent neural networks, the attention mechanism has been widely used in natural language processing, prompting in-depth research on the attention mechanism in the field of computer vision. The use of the attention mechanism allows the interactive fusion of image context information and text vocabulary information, thereby improving the relevance of the generated image description. The emergence of multimodal pre-trained large models has had an important impact on image description. These models are committed to unifying tasks in the visual and textual fields, processing visual and textual information at the same time, and significantly improving the performance of image description models. This trend has gradually become a key driving force for the development of the image description field.
[0004] At the same time, there are a series of problems with existing methods and image description models. Existing methods usually integrate the advantages of multiple types of visual features in the encoding stage to form a more powerful visual representation. However, in the decoding stage, only a simple cross-attention mechanism is applied to complete the conversion from vision to language. This method lacks complex cross-modal reasoning capabilities and cannot effectively deal with the semantic gap problem. In addition, although some methods enhance decoding capabilities to achieve more accurate description generation, they do not have any feature interaction in the encoding process and cannot form complementary and robust visual representations, which affects the performance of the model. Summary of the invention
[0005] The present invention aims to solve the deficiencies of the prior art and provides the following solutions:
[0006] An image description system based on image-text feature fusion, comprising: a text feature extraction module, a feature extraction module, an image encoder module, a feature fusion module and a decoder module;
[0007] The text feature extraction module maps the text of the original image into a vector by using a word embedding method to obtain the original text feature;
[0008] The feature extraction module is used to extract features from the original image to obtain grid features and global features of the original image;
[0009] The image encoder module uses an attention mechanism to perform attention interaction on the grid features and the global features;
[0010] The feature fusion module is used to fuse and enhance the grid features and the global features after attention interaction to obtain enhanced image features;
[0011] The decoder module fuses the original text features, the global features and the enhanced image features to obtain an image description sentence.
[0012] Preferably, the feature extraction module comprises: a visual encoder unit, a dimensionality reduction unit and a pooling unit;
[0013] The visual encoder unit is used to extract grid features from the original image to obtain the grid features;
[0014] The dimension reduction unit is used to reduce the dimension of the grid feature to obtain a dimension-reduced feature;
[0015] The pooling unit is used to perform global average pooling on the dimensionally reduced features to obtain the global features.
[0016] Preferably, the image encoder module comprises: a sliding window attention unit and a self-attention unit;
[0017] The sliding window attention unit is used to dynamically adjust the window size, and divide the grid features based on the window size, and perform feature enhancement on the divided grid features to complete the attention interaction of the grid features;
[0018] The self-attention unit is used to perform feature enhancement on the global feature and complete the attention interaction of the global feature.
[0019] Preferably, the feature fusion module includes: a feature fusion unit, a feature segmentation unit and a feature splicing enhancement unit;
[0020] The feature fusion unit is used to fuse and splice the grid features and the global features between adjacent levels to obtain the spliced overall features;
[0021] The feature segmentation unit is used to segment the overall feature along the channel dimension to obtain segmented image features;
[0022] The feature splicing and enhancement unit is used to perform splicing enhancement on the segmented image features in the channel dimension to obtain the enhanced image features.
[0023] Preferably, the decoder module comprises: a picture-text pre-fusion unit and a semantic feature fusion unit;
[0024] The image-text pre-fusion unit is used to preliminarily fuse the original text features and the global features to obtain semantic features;
[0025] The semantic feature fusion unit is used to fuse the enhanced image features and the semantic features to obtain the image description sentence.
[0026] The present invention also provides an image description method based on image-text feature fusion, the method is applied to any of the above-mentioned systems, and comprises the following steps:
[0027] Use word embedding to map the text of the original image into a vector to obtain the original text features;
[0028] Performing feature extraction on the original image to obtain grid features and global features of the original image;
[0029] Using an attention mechanism to perform attention interaction on the grid feature and the global feature;
[0030] The grid feature and the global feature after attention interaction are fused and enhanced to obtain enhanced image features;
[0031] The image description sentence is obtained by fusing the original text features, the global features and the enhanced image features.
[0032] Preferably, the feature extraction method comprises:
[0033] Extracting grid features from the original image to obtain the grid features;
[0034] Performing dimension reduction on the grid features to obtain dimension-reduced features;
[0035] The dimension-reduced features are subjected to global average pooling to obtain the global features.
[0036] Preferably, the attention interaction method includes:
[0037] The window size is dynamically adjusted using a sliding window attention mechanism, and the grid features are divided based on the window size, and the divided grid features are enhanced to complete the attention interaction of the grid features;
[0038] The self-attention mechanism is used to enhance the global features and complete the attention interaction of the global features.
[0039] Preferably, the fusion enhancement method comprises:
[0040] The grid features and the global features between adjacent levels are fused and spliced to obtain the spliced overall features;
[0041] Segmenting the overall feature along the channel dimension to obtain segmented image features;
[0042] The segmented image features are spliced and enhanced in the channel dimension to obtain the enhanced image features.
[0043] Preferably, the method for obtaining the image description sentence includes:
[0044] Preliminarily fusing the original text features and the global features to obtain semantic features;
[0045] The enhanced image features and the semantic features are fused to obtain the image description sentence.
[0046] Compared with the prior art, the present invention has the following beneficial effects:
[0047] The present invention combines image and text information and uses advanced deep learning technology to achieve accurate description of image content. By fusing image features at different layers and pre-fusing image features and text features, it can generate a natural language description corresponding to the input image, further improving the performance of the image description model. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0049] Figure 1 A schematic diagram of the system structure of an embodiment of the present invention;
[0050] Figure 2 A schematic diagram of the overall system architecture of an embodiment of the present invention;
[0051] Figure 3 Schematic diagram of a feature fusion module according to an embodiment of the present invention;
[0052] Figure 4 Schematic diagram of a graphic and text pre-fusion unit according to an embodiment of the present invention. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0055] Embodiment 1
[0056] In this embodiment, if Figure 1 , Figure 2 As shown, an image description system based on image-text feature fusion includes: a text feature extraction module, a feature extraction module, an image encoder module, a feature fusion module and a decoder module.
[0057] The text feature extraction module uses word embedding to map the text of the original image into a vector to obtain the original text features.
[0058] The feature extraction module is used to extract features from the original image to obtain the grid features and global features of the original image.
[0059] The feature extraction module includes: a visual encoder unit, a dimensionality reduction unit and a pooling unit. The visual encoder unit is used to extract grid features from the original image to obtain grid features; the dimensionality reduction unit is used to reduce the dimensions of the grid features to obtain features after dimensionality reduction; the pooling unit is used to perform global average pooling on the features after dimensionality reduction to obtain global features.
[0060] In this embodiment, the constructed feature extraction module mainly uses the visual encoder in the CLIP model, and its main function is to extract grid features from the original image to provide strong support for the subsequent text decoding task. Specifically, the visual encoder unit uses the CLIP-ViT-L visual encoder to extract grid features from the original image, which is formulated as follows:
[0061] V G =CLIP-ViT-L(I)
[0062] Among them, I represents the original image after data enhancement, V G represents the image grid feature obtained after the backbone network, with a dimension of 1024; the number of parameters of the overall network architecture is reduced to improve the training rate, and the dimension reduction unit reduces the dimension of the grid feature from 1024 to 512 through the fully connected layer; in order to obtain the global feature description of the image, the pooling unit converts the grid feature V G Global average pooling obtains the global feature V g .
[0063] The image encoder module uses the attention mechanism to interact the grid features with the global features.
[0064] The image encoder module includes: a sliding window attention unit and a self-attention unit. The sliding window attention unit is used to dynamically adjust the window size, divide the grid features based on the window size, and enhance the grid features after division to complete the attention interaction of the grid features; the self-attention unit is used to enhance the global features to complete the attention interaction of the global features.
[0065] In this embodiment, the attention mechanism of the image encoder module includes self-attention (MSA), window attention (W-MSA) and sliding window attention (SW-MSA).
[0066] The self-attention mechanism is a key technology for image processing. It can learn the relationship between different pixels in an image, thereby capturing long-distance dependencies and expanding the receptive field. The core idea is to map image features to three different spaces: query vector (Q), key vector (K), and value vector (V). Then, by calculating the similarity between the query vector and the key vector, a weight map is obtained. Finally, the weight map is weighted and summed with the corresponding value vector to obtain the attention feature map. The multi-head attention mechanism uses multiple sets of independent query, key, and value vectors, called "heads", at the same time to learn representations of different subspaces, thereby enhancing feature representation.
[0067] The window attention mechanism processes the input of large-size images by dividing the input image features into windows of fixed size, and then performs a self-attention mechanism within each window, that is, calculating the attention weights between the elements within the window. This method reduces the computational complexity and reduces the consumption of computing resources while retaining global information. When processing image grid features, it can effectively capture local and global information and strike a balance between computational efficiency and performance.
[0068] The sliding window attention mechanism is a further improvement of the window attention mechanism, which introduces the concept of sliding window. In the sliding window attention mechanism, the window is no longer a fixed size, and the entire image features can be covered by sliding a certain distance. The model is allowed to dynamically adjust the position and size of the window to adapt to features of different sizes and distributions, so as to better capture the relationship between local features and global features. It can flexibly process image features of different sizes, while reducing the dependence on predefined window sizes and improving the generalization ability of the model. In general, both window attentions are methods of processing image data using attention mechanisms. They differ in processing large-size inputs and capturing local and global information, but both can effectively improve the performance and efficiency of the model.
[0069] This embodiment uses the sliding window attention mechanism and the self-attention mechanism to construct the image encoder module. The sliding window attention unit is used to dynamically adjust the window size, divide the grid features based on the window size, and enhance the features of the divided grid features to complete the attention interaction of the grid features; specifically, given the input image grid feature V G =[v1,v2,...,v n ], where v n It represents the nth patch in the image, divided into i windows, and the grid feature V G Mapped to the space of query vector Q, key K and value V through linear transformation:
[0070]
[0071] in, represents the learnable parameter matrix in the i-th window, Qi represents the query vector of the i-th window, K i represents the key vector of the i-th window, V i Represents the value vector of the i-th window; then the divided grid features are enhanced according to the results of the linear transformation to complete the attention interaction of the grid features:
[0072]
[0073] W-SMA(Q,K,V)=Merge(H1,H2,...,H i )
[0074] Among them, d k represents the scaling factor, H i represents the enhanced image features, represents the transposition of Ki, and Merge represents the inverse operation after window division. The self-attention unit converts the global feature V g Mapped to three different spaces: query vector Q, key vector K, and value vector V. The steps are consistent with the sliding window attention unit. Then, the weight map is obtained by calculating the similarity between the query vector Q and the key vector K. Finally, the weight map is weighted and summed with the corresponding value vector to obtain the global feature V. g Perform feature enhancement and complete attention interaction of global features.
[0075] The feature fusion module is used to fuse and enhance the grid features and global features after attention interaction to obtain enhanced image features.
[0076] The feature fusion module includes: a feature fusion unit, a feature segmentation unit and a feature splicing enhancement unit. The feature fusion unit is used to fuse and splice the grid features and global features between adjacent levels to obtain the spliced overall features; the feature segmentation unit is used to segment the overall features along the channel dimension to obtain the segmented image features; the feature splicing enhancement unit is used to splice and enhance the segmented image features in the channel dimension to obtain the enhanced image features.
[0077] In most image description models, the image features extracted by the encoder are not fully utilized, and there is no other interaction between visual features. However, the features extracted from different layers can be integrated with each other, which helps to reduce visual feature redundancy and enhance feature representation. Figure 3 As shown in the figure, the feature fusion unit uses a multi-level feature fusion method. First, the grid features and global features of different layers are saved in the corresponding queues. The information between different features is used to fuse the grid features and the global features to obtain the overall features after splicing, which can be expressed as follows:
[0078] F All =Concat((F i +G i+1 ), (F i+1 +G i ))
[0079] F All i =Relu(F All W1 i +b1 i )W2 i +b2 i
[0080] F All i+1 =Relu(F All W i+1 +b1 i+1 )W2 i+1 +b2 i+1
[0081] Among them, F i and F i+1 Represents the grid features of two adjacent layers, G i and G i+1 represents the global features of two adjacent layers, F All Represents the overall features after splicing, W1 i 、W2 i 、b1 i 、b2 i 、W1 i+1 、b1 i+1 、b2 i+1 Represents a learnable parameter; afterwards, the feature segmentation unit segments the features along the channel dimension, and the feature splicing enhancement unit is used to perform splicing enhancement on the channel dimension of the segmented image features through a feedforward neural network to obtain the enhanced image features:
[0082]
[0083] F′ i =FFN(Concat(F′ i ×AvgPool(ω i ), F′ i+1 ×AvgPool(ω i+1 )))
[0084] Among them, Split means splitting the features along the channel dimension, F′ i and F′ i+1 represents the enhanced image features, ω i and ω i+1It represents the channel weight of the enhanced image feature, and AvgPool represents weighted average pooling of the corresponding weights.
[0085] The decoder module fuses the original text features, global features and enhanced image features to obtain the image description sentence.
[0086] The decoder module includes: a picture-text pre-fusion unit and a semantic feature fusion unit. The picture-text pre-fusion unit is used to preliminarily fuse the original text features with the global features to obtain semantic features; the semantic feature fusion unit is used to fuse the enhanced image features with the semantic features to obtain the image description sentence.
[0087] In this embodiment, the image and text pre-fusion unit is as follows: Figure 4 As shown in the figure, this unit is designed to simulate the association between vision and language to help the decoder better understand and process information between different modalities. By shortening the connection between image and text features, the unit can obtain information that is closer to the image semantic feature encoding, thereby improving the processing effect of multimodal information. The original text features are increased to 1024 dimensions and fused with the semantic features spliced with image features in high-dimensional space, which can further improve the expressiveness of semantic features and enable them to better capture the association information between text and image. It can be expressed as follows:
[0088]
[0089] The original text features and the global features V g Perform preliminary fusion to obtain semantic features in, represents the text feature encoding input of the l-1 layer image, W1, W2, b1, b2 represent learnable parameters. After that, the enhanced image features and semantic features are input into the Masked MSA module of the semantic feature fusion unit, followed by a residual connection and layer normalization, which can be expressed as:
[0090]
[0091] Subsequent decoding will perform a second fusion between modalities through the cross-attention mechanism, and interact between features through the Cross MSA module to obtain semantic feature encoding containing image context content. It can be expressed as:
[0092]
[0093] Finally, the text feature encoding is output from the decoder to obtain the image description sentence.
[0094] Embodiment 2
[0095] In this embodiment, the training process and experimental results of the image description system will be described.
[0096] This embodiment uses the MS COCO dataset, which contains 82,783 training set images, 40,504 validation set images, and 40,775 test set images, each of which contains 5 different manually annotated reference descriptions (the test set is not provided). The dataset is divided according to the division method disclosed by Karpathy et al., wherein 5,000 images are divided from the validation set for verification during the training process, and another 5,000 images are divided for testing. Finally, the remaining images in the validation set are merged with the training set to obtain a training set containing 113,287 images and their reference descriptions. The descriptions corresponding to all images in the training set are processed, arranged in descending order according to the frequency of occurrence of the words, each word is mapped to an integer id, the original corpus is segmented, and a vocabulary is further constructed. The vocabulary size is 9487, and the original description sentence is replaced with the vocabulary id; then the longest sentence length is set to 17, and the original sentence description is intercepted and filled; finally, all the picture descriptions in the training set are converted into a one-dimensional array to form the original text for input.
[0097] The training process of the model is divided into two stages. The first stage uses supervised learning to perform coarse-grained learning on the model. The second stage uses reinforcement learning strategy for training, mainly targeting the CIDEr indicator in image description.
[0098] First, in the supervised learning stage, we use the cross entropy (XE) loss as the objective function to optimize our model. The loss function is defined as:
[0099]
[0100] in, It indicates the words that have been generated. Represents the word that the model is about to predict, T represents the length of the sentence, and θ represents the parameters that the image description model needs to learn.
[0101] In the reinforcement learning stage, the loss function is defined as:
[0102]
[0103] in, is the sample description statement obtained through sampling, It is the sentence sampled by the current model through the beamsearch algorithm.
[0104] To demonstrate the effectiveness of the present invention, this embodiment shows the quantitative results of the trained image description system and other state-of-the-art methods on the Karpathy test set in Table 1. The CIDEr score of the present invention reached 141.7%, which is 2.9% and 3.8% higher than CropCAP and DSNT. At the same time, in order to further verify the effectiveness and generalization ability of the model, this embodiment achieved CIDEr scores of 137.4% (c5) and 139.4% (c40) on the online MS COCO test server. Table 2 shows the performance in terms of 5 reference titles (c5) and 40 reference titles (c40). It can be seen that the present invention surpasses the current state-of-the-art methods in almost all indicators. It is worth noting that the ROUGE-L (c40) and CIDEr (c40) scores of the present invention reached 76.4% and 138.6%, respectively, which are 1% and 2.3% higher than the best performers.
[0105] Table 1
[0106]
[0107] Table 2 Ranking of the most advanced image description generation models on the MSCOCO online test server, where c5 and c40 represent 5 and 40 reference titles used for evaluation, respectively. All values in the table are expressed as percentage (%), and it can be seen that the performance of the present invention is better than other mainstream methods.
[0108] Table 2
[0109]
[0110] Embodiment 3
[0111] In this embodiment, an image description method based on image-text feature fusion includes the following steps:
[0112] S1. Use word embedding to map the text of the original image into a vector and obtain the original text features.
[0113] S2. Perform feature extraction on the original image to obtain the grid features and global features of the original image.
[0114] The feature extraction method includes: performing grid feature extraction on the original image to obtain grid features; performing dimension reduction on the grid features to obtain dimension reduced features; and performing global average pooling on the dimension reduced features to obtain global features.
[0115] S3. Use the attention mechanism to interact the grid features and global features.
[0116] The attention interaction method includes: using the sliding window attention mechanism to dynamically adjust the window size, dividing the grid features based on the window size, and enhancing the features of the divided grid features to complete the attention interaction of the grid features; using the self-attention mechanism to enhance the features of the global features to complete the attention interaction of the global features.
[0117] S4. The grid features after attention interaction and the global features are fused and enhanced to obtain enhanced image features.
[0118] The fusion enhancement method includes: fusing and splicing the grid features and global features between adjacent levels to obtain the spliced overall features; segmenting the overall features along the channel dimension to obtain the segmented image features; and splicing and enhancing the segmented image features on the channel dimension to obtain the enhanced image features.
[0119] S5. Based on the fusion of original text features, global features and enhanced image features, the image description sentence is obtained.
[0120] The method for obtaining an image description sentence includes: preliminarily fusing original text features and global features to obtain semantic features; and fusing enhanced image features and semantic features to obtain an image description sentence.
[0121] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. An image description system based on image and text feature fusion, characterized in that: include: Text feature extraction module, feature extraction module, image encoder module, feature fusion module and decoder module; The text feature extraction module maps the text of the original image into a vector by using a word embedding method to obtain the original text feature; The feature extraction module is used to extract features from the original image to obtain grid features and global features of the original image; The image encoder module uses an attention mechanism to perform attention interaction on the grid features and the global features; The feature fusion module is used to fuse and enhance the grid features and the global features after attention interaction to obtain enhanced image features; The decoder module fuses the original text features, the global features and the enhanced image features to obtain an image description sentence.
2. According to claim 1, the image description system based on image-text feature fusion is characterized in that: The feature extraction module includes: a visual encoder unit, a dimension reduction unit and a pooling unit; The visual encoder unit is used to extract grid features from the original image to obtain the grid features; The dimension reduction unit is used to reduce the dimension of the grid feature to obtain a dimension-reduced feature; The pooling unit is used to perform global average pooling on the dimensionally reduced features to obtain the global features.
3. The image description system based on image-text feature fusion according to claim 1, characterized in that: The image encoder module includes: a sliding window attention unit and a self-attention unit; The sliding window attention unit is used to dynamically adjust the window size, and divide the grid features based on the window size, and perform feature enhancement on the divided grid features to complete the attention interaction of the grid features; The self-attention unit is used to perform feature enhancement on the global feature and complete the attention interaction of the global feature.
4. The image description system based on image-text feature fusion according to claim 1, characterized in that: The feature fusion module includes: a feature fusion unit, a feature segmentation unit and a feature splicing enhancement unit; The feature fusion unit is used to fuse and splice the grid features and the global features between adjacent levels to obtain the spliced overall features; The feature segmentation unit is used to segment the overall feature along the channel dimension to obtain segmented image features; The feature splicing and enhancement unit is used to perform splicing enhancement on the segmented image features in the channel dimension to obtain the enhanced image features.
5. The image description system based on image-text feature fusion according to claim 1, characterized in that: The decoder module includes: a picture and text pre-fusion unit and a semantic feature fusion unit; The image-text pre-fusion unit is used to preliminarily fuse the original text features and the global features to obtain semantic features; The semantic feature fusion unit is used to fuse the enhanced image features and the semantic features to obtain the image description sentence.
6. An image description method based on image-text feature fusion, the method being applied to the system according to any one of claims 1 to 5, characterized in that: The following steps are involved: Use word embedding to map the text of the original image into a vector to obtain the original text features; Performing feature extraction on the original image to obtain grid features and global features of the original image; Using an attention mechanism to perform attention interaction on the grid feature and the global feature; The grid feature and the global feature after attention interaction are fused and enhanced to obtain enhanced image features; The image description sentence is obtained by fusing the original text features, the global features and the enhanced image features.
7. The image description method based on image-text feature fusion according to claim 6, characterized in that: The feature extraction method comprises: Extracting grid features from the original image to obtain the grid features; Performing dimension reduction on the grid features to obtain dimension-reduced features; The dimension-reduced features are subjected to global average pooling to obtain the global features.
8. The image description method based on image-text feature fusion according to claim 6, characterized in that: The attention interaction method includes: The window size is dynamically adjusted using a sliding window attention mechanism, and the grid features are divided based on the window size, and the divided grid features are enhanced to complete the attention interaction of the grid features; The self-attention mechanism is used to enhance the global features and complete the attention interaction of the global features.
9. The image description method based on image-text feature fusion according to claim 6, characterized in that: The fusion enhancement method comprises: The grid features and the global features between adjacent levels are fused and spliced to obtain the spliced overall features; Segmenting the overall feature along the channel dimension to obtain segmented image features; The segmented image features are spliced and enhanced in the channel dimension to obtain the enhanced image features.
10. The image description method based on image-text feature fusion according to claim 6, characterized in that: The method for obtaining the image description sentence includes: Preliminarily fusing the original text features and the global features to obtain semantic features; The enhanced image features and the semantic features are fused to obtain the image description sentence.
Citation Information
Patent Citations
Adaptive generation system for image semantic description
CN110210499A
Interactive image editing method and device, readable storage medium and electronic equipment
CN113448477A
Transform structure-based image description method and device
CN114399646A
Cited By
Image watermark embedding method and device, electronic equipment and storage medium
CN121074565A
