Text-guided image segmentation method based on attention mechanism for cross-modal text retrieval

Through the cross-modal text retrieval attention mechanism and multi-level features mixing, the problem of inaccurate positioning of text-guided image segmentation methods in the existing technology in complex scenarios is solved, and higher accuracy and robustness are achieved. It is suitable for Human-computer interaction and picture editing fields.

CN113657400BActive Publication Date: 2024-12-03DALIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110952433.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-19
Publication Date
2024-12-03
Estimated Expiration
2041-08-19

AI Technical Summary

Technical Problem

The existing text-guided image segmentation method lacks effective multimodal information fusion when dealing with complex text descriptions or multi-entity scenarios, resulting in inaccurate positioning of target objects and lack of model interpretability.

Method used

Using a method based on the cross-modal text retrieval mechanism, text and image features are extracted through LSTM and ResNet-101 convolutional neural network, Hadamama product fusion is performed, and feature positioning is enhanced using the cross-modal attention mechanism, combined with the memory gate Mix multi-level features with reset door units to generate more accurate segmentation results.

Benefits of technology

It improves the accuracy and robustness of text-guided image segmentation, can correctly locate the target object in complex scenarios, and enhances the detailed accuracy and interpretability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113657400B_ABST
    Figure CN113657400B_ABST
Patent Text Reader

Abstract

A text-guided image segmentation method based on a cross-modal text retrieval attention mechanism. This method uses text as a query vector and multimodal features as a retrieval space, adaptively focusing on regions similar to the text in the multimodal feature space to locate the target object. The method includes: feature extraction, deep fusion of image and text information, deep relationship capture, and multi-level feature mixing. Convolutional networks and long short-term memory networks are used to extract visual and language features; Hadamard products are used to deeply fuse visual and language feature information; an attention mechanism based on text queries is used to obtain regions in the image scene that are similar to the text description, and finally multiple levels of features are mixed to segment the target object. This method can effectively improve the accuracy of locating targets in complex scenes and achieve accurate segmentation of regions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the interdisciplinary technical field of computer vision and natural language processing, and relates to a text-guided image segmentation method based on a cross-modal text retrieval attention mechanism. Background Art

[0002] The goal of the text-guided image segmentation task is to locate and segment the objects or regions described by the text based on the given natural language. It can not only identify fixed categories, but also understand the rich and varied natural language. Compared with ordinary image segmentation tasks, this task can build a bridge for direct and rapid communication between humans and computers. It is more intuitive in practical applications and can be conveniently applied in daily life such as human-computer interaction and image editing. It has broad commercial value and far-reaching research significance. The text-guided image segmentation task requires that the computer's understanding of the objects in the observed scene should not only stay at the "cognitive" level, but also correctly "understand" the attributes of the objects such as type, size, shape, and the correlation with surrounding objects. It is necessary to unify the mapping of the feature information of the text and the image, and to correctly locate the multi-entity relationship in the image based on the object relationship mentioned in the language, and the positioning area needs to be accurately segmented.

[0003] Most of the current text-guided image segmentation methods use a simple method of directly adding or cascading text features and image visual features to integrate multimodal information pixel-by-pixel segmentation prediction results. However, this simple fusion method underestimates the importance of the interaction between text and image information, ignores the guiding decision of text on image information, and lacks model interpretability. This greatly affects the accuracy of the model's target positioning and causes the model to predict the wrong area of ​​the object. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a text-guided image segmentation method based on a cross-modal text retrieval attention mechanism, which explicitly retrieves the multimodal feature space using natural language text and ultimately guides the generation of segmentation results. This method can effectively improve the accuracy of segmentation results in scenes with complex text descriptions or multiple objects of the same category.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A text-guided image segmentation method based on a cross-modal text retrieval attention mechanism comprises the following steps:

[0007] (1) Feature extraction:

[0008] Perform feature extraction on input natural language text and RGB images.

[0009] For natural language text, use on e The -hot method represents each word as a vector, embeds the vector into a low-dimensional vector, and then inputs it into the LSTM long short-term memory network, and uses the final hidden state as the text representation vector F of the entire natural language text. l .

[0010] For RGB images, the Res3, Res4, and Res5 layers of the ResNet-101 convolutional neural network are used to extract image features as visual features. The same processing steps are used in the three-layer feature map, and for convenience, the visual features are collectively referred to as F v The convolutional neural network uses the pre-trained parameters of the DeepLab semantic segmentation model as the initial parameters. Using the DeepLab pre-trained parameters can effectively reduce the network training time and improve the generalization ability of the network. Then, the position coordinates of each visual feature vector are embedded in the visual feature F v The specific process is as follows: for a visual feature map with a length and width of w and h, where w = h = 40. The center position of the visual feature map is taken as the coordinate origin (0, 0), the coordinates of the upper left corner of the feature map are marked as (-1, -1), and the coordinates of the lower right corner are marked as (+1, +1). The spatial coordinates of the pixel in the i-th row and j-th column of the feature map are

[0011] Finally, the visual feature F v With the text representation vector F l After cascading, it is mapped to the 500-dimensional text-image multimodal feature space M.

[0012] (2) Deep integration of graphic and text information:

[0013] The text representation vector F generated in step (1) l By performing Hadamard product with each position in the multimodal feature space M, the multimodal information of the two can be modeled, and the deep fusion of text and image information into a unified multimodal mixed feature space M′ can be achieved, which is conducive to the correct positioning of the target object area by the subsequent modules.

[0014] (3) Deep-level relationship capture:

[0015] The text representation vector F generated in step (1) l As the query vector of the attention mechanism, the multimodal mixed feature space M′ generated in step (2) is used as the retrieval area, and the text-based cross-modal attention mechanism is used to query the area in the multimodal mixed feature space that is similar to the text, and generate a feature map R with the location of the target object.

[0016] The text-based cross-modal attention mechanism is different from the ordinary attention mechanism. By adding a secondary attention mechanism, the correlation between the query vector and the result of the first attention mechanism is enhanced to reduce the difference in different information forms between the query vector and the multimodal mixed features. The information vector I and the attention gating unit G generated by the first attention result are used to promote the correct positioning of the target object. Compared with the ordinary cascade method, this method can effectively resist the interference of redundant words, correctly locate the area where the target object is located in complex scenes, and improve the accuracy of the model.

[0017] (4) Multi-level feature mixing:

[0018] The ResNet101 visual encoder used in step (1) can obtain three visual features F from the Res3, Res4, and Res5 layers. v Therefore, the three levels of features need to be mixed to obtain the final result predicted by the method.

[0019] Using R (i) represents the mixed information feature map i∈{3, 4, 5} of the i-th layer generated in step (3). (i) The feature maps output by the deep relationship capture modules at different layers have different channel dimensions. Therefore, we first use 1×1 convolution to compress the mixed feature map channels of all different layers into the same dimension, and record the output result after mapping as X (i) . Then for the i-th level, use a memory gate unit m (i) and reset gate unit Jointly control the generation of the final result, memory gate unit m (i) and reset gate unit r (i) The function is similar to the gate unit in the long short-term memory network, which controls the contribution of the features in the corresponding layer to the final result. The higher the contribution weight ratio, the more semantic information the corresponding layer provides, and the more important the feature layer is. In addition, it also contains a context controller C (i) , used to represent the information of features from other layers to the i-th layer. Gated multi-level mixing can effectively fuse feature semantic information from different layers, automatically generate the optimal weight ratio through learning and training, without the need for human prior intervention, making the model results more accurate and improving the segmentation detail accuracy of the model.

[0020] Compared with the prior art, the present invention enables the text-based image segmentation method to adapt to complex natural language scenarios with multiple description entities and correctly locate the description area. The method of the present invention can be applied to various fields such as human-computer interaction and interactive image editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1It is the overall architecture diagram of the present invention. DETAILED DESCRIPTION

[0022] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0023] Figure 1 The framework design of the text-based image segmentation method of the present invention is given, and the main process is as follows:

[0024] First, all images are resized to 320 × 320. The Res3, Res4, and Res5 layers of the ResNet-101 convolutional neural network are used to extract image features as visual features. i∈{3, 4, 5}. Use the DeepLab pre-trained feature extraction network to extract visual features F v The convolutional neural network uses the pre-trained parameters of the DeepLab semantic segmentation model as initial parameters; then the position coordinates of each visual feature vector are embedded in the visual feature F v The specific process is as follows:

[0025] For a visual feature map with a length and width of w and h, where w = h = 40; the center position of the visual feature map is taken as the coordinate origin (0, 0), the coordinates of the upper left corner of the feature map are marked as (-1, -1), and the coordinates of the lower right corner are marked as (+1, +1). The spatial position coordinates of the pixel in the i-th row and j-th column of the feature map are

[0026] For natural language, the word vector is initialized randomly. e -hot word vectors are embedded into 1000-dimensional vectors, and the vector representation of the sentence is obtained through the LSTM long short-term memory network. The longest word number of the LSTM text is 20, and the specific calculation process of the long short-term memory network is shown in the formula: t =LSTM(x t ,h t-1 );

[0027] where h t represents the LSTM output vector, x t represents the LSTM input vector, h t-1 Represents the output hidden state of the previous LSTM layer. The hidden state of the last layer is used as the vector representation of the text F l .

[0028] The visual features are then concatenated with the textual features.

[0029] M=Φ(concat(F v , F l , Cord))

[0030] Among them, Cord represents the visual space position coordinate, Φ represents the mapping function, and M represents the generated multimodal mixed features with text and image information.

[0031] Then, the Hadamard product is used to fuse multimodal features.

[0032] F=M⊙F l

[0033] The generated text representation vector F l By performing Hadamard product with each position in the multimodal feature space M, the multimodal information of the two can be modeled, and the deep fusion of text and image information into a unified multimodal mixed feature space M′ can be achieved.

[0034] The generated text representation vector F l The generated multimodal mixed feature space M′ is used as the query vector of the attention mechanism as the retrieval area, and the text-based cross-modal attention mechanism is used to query the same area in the multimodal mixed feature space as the text, generating a feature map R with the location of the target object.

[0035] Based on the multimodal feature F obtained in the previous step, a cross-modal attention mechanism based on text query is used to obtain the area in the feature that is similar to the text. The calculation process is as follows:

[0036]

[0037] The text representation feature F l As the query vector Q, the image-text multimodal hybrid feature M′ is used as the key vector K and the value vector V. The traditional attention mechanism is used for the first time to obtain the decision result with text assistance, which is recorded as Att.

[0038] Fusion = concat(Att, F)

[0039] I=CNN(Fusion)

[0040] G = Sigmoid (CNN (Fusion))

[0041] R=I⊙G

[0042] Then the multimodal feature F is connected to the first attention result Att and the second attention process is performed again. Here, two 1×1 convolution operations are used to generate the information vector I and the attention gating unit G, where the two convolution operation parameters cannot be shared. The information vector I is equivalent to the query vector of the second attention. Compared with the text representation vector Q, it has less difference with the visual features and is more suitable for finding spatial feature vectors similar to the query vector in the multimodal feature space. The attention gating unit G is used to control the contribution between the first attention generation result Att and the multimodal semantic feature F. The model automatically adjusts the weight ratio between the Att and F features through training to generate better model results.

[0043] The adopted ResNet101 visual encoder can obtain three visual features F from Res3, Res4, and Res5 levels. v ; The final result of the prediction method needs to mix the features of the three levels;

[0044] Using R (i) Represents the generated mixed information feature map i∈{3, 4, 5} of the i-th layer; due to R (i) The feature maps output by the deep relationship capture modules at different layers have different channel dimensions. First, 1×1 convolution is used to compress the mixed feature map channels of all different layers into the same dimension, and the output result after mapping is recorded as X( i ); then for the i-th level, use a memory gate unit m( i ) and reset gate unit Jointly control the generation of the final result, the memory gate unit m(i) and the reset gate unit r( i ) has the same function as the gating unit in the long short-term memory network, controlling the contribution of the features in the corresponding layer to the final result. The higher the contribution weight ratio, the more semantic information the corresponding layer provides, and the more important the feature layer is. In addition, it also contains a context controller C( i ) is used to represent the information of features from other layers to the i-th layer; the gated multi-level mixture can effectively fuse the feature semantic information from different levels, and automatically generate the optimal weight ratio through learning and training, making the model results more accurate and improving the segmentation detail accuracy of the model.

[0045] Example

[0046] In this embodiment, the deep learning framework Tensorflow is used on the GTX 1080 graphics card.

[0047] Dataset: The experimental evaluation is conducted on the standard public dataset G-ref. This dataset includes 26,711 images and 104,560 natural language texts, with an average text length of 8.43 words. It is a relatively complex dataset among text-based image segmentation datasets.

[0048] In order to prove the effectiveness of the text-guided image segmentation method based on the cross-modal text retrieval attention mechanism, the IoU indicator is tested on the G-Ref dataset. The results are shown in Table 1. The experiment proves that the method of the present invention can effectively improve the accuracy of the results.

[0049]

[0050] It can be seen from the experimental results that the present invention is more accurate and more robust in locating natural language text descriptions of complex multi-entities compared to the prior art.

Claims

1. A text-guided image segmentation method based on cross-modal text retrieval attention mechanism, It is characterized in that Here are the steps: (1) Feature extraction: Extract features from input natural language text and RGB images; For natural language text, use the one-hot method to represent each word as a vector, embed the vector into a low-dimensional vector and input it into the LSTM long short-term memory network, and use the final hidden state as the text representation vector F of the entire natural language text. l ; For RGB images, the Res3, Res4, and Res5 layers of the ResNet-101 convolutional neural network are used to extract image features as visual features. The same processing steps are used in the three-layer feature map, and the visual features are collectively referred to as F v ; The convolutional neural network uses the pre-trained parameters of the DeepLab semantic segmentation model as initial parameters; then the position coordinates of each visual feature vector are embedded in the visual feature F v The specific process is as follows: For a visual feature map with a length and width of w, h, where w = h = 40; the center position of the visual feature map is taken as the coordinate origin (0,0), the coordinates of the upper left corner of the feature map are marked as (-1, -1), and the coordinates of the lower right corner are marked as (+1, +1). The spatial position coordinates of the pixel in the i-th row and j-th column of the feature map are Finally, the visual feature F v With the text representation vector F l After cascading, it is mapped to a 500-dimensional text-image multimodal feature space M; (2) Deep integration of graphic and text information: The text representation vector F generated in step (1) l The Hadamard product is performed with each position in the multimodal feature space M to model the multimodal information of the two, and to achieve deep fusion of text and image information into a unified multimodal hybrid feature space M. ′ ; (3) Deep-level relationship capture: The text representation vector F generated in step (1) l As the query vector of the attention mechanism, the multimodal hybrid feature space M generated in step (2) ′ As the retrieval area, use the text-based cross-modal attention mechanism to query the same area as the text in the multimodal mixed feature space to generate a feature map R with the location of the target object; (4) Multi-level feature mixing: The ResNet101 visual encoder used in step (1) can obtain three visual features F from the Res3, Res4, and Res5 layers. v ; The final result of the prediction method needs to mix the features of the three levels; Using R (i) represents the mixed information feature map i∈{3,4,5} of the i-th layer generated in step (3); since R (i) The feature maps output by the deep relationship capture modules at different layers have different channel dimensions. First, 1×1 convolution is used to compress the mixed feature map channels of all different layers into the same dimension, and the output result after mapping is recorded as X (i) ; Then for the i-th level, use a memory gate unit m (i) and reset gate unit Joint control of the generation of the final result, memory gate unit m (i) and reset gate unit r (i) The function is the same as the gate unit in the long short-term memory network, which controls the contribution of the features in the corresponding layer to the final result. The higher the contribution weight ratio, the more semantic information the corresponding layer provides, and the more important the feature layer is. In addition, it also contains a context controller C (i) , which is used to represent the information of features from other layers to the i-th layer; the gated multi-level mixture can effectively fuse the feature semantic information from different levels, and automatically generate the optimal weight ratio through learning and training, making the model results more accurate and improving the segmentation detail accuracy of the model.