Text-guided image segmentation method based on structured multi-modal fusion network

Abstract

The application provides a text-guided image segmentation method based on a structured multi-modal fusion network, which models global multi-modal context by constructing multiple multi-modal fusion structures, highlights the spatial region where the object described by the text is located, and further integrates high-level semantic and low-level visual features by constructing a multi-level progressive refinement module to clearly distinguish the object described by the text from similar objects and generate an accurate segmentation mask. The method mainly includes: feature extraction, construction of a progressive multi-modal fusion tree module, construction of a transmembrane fusion Transformer module, and construction of a multi-level progressive fusion refinement module. The semantic features in the picture are extracted using a convolutional neural network; the initial global multi-modal context information is modeled through the progressive multi-modal fusion tree module; the high-quality global multi-modal context is further modeled through the transmembrane fusion Transformer module; finally, the visual details of the spatial region where the object described by the text is located are supplemented by integrating high-level multi-modal semantic and low-level visual detail information to clearly distinguish the boundaries between objects, clearly distinguish the object described by the text from other objects, and finally generate an accurate segmentation mask.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and natural language processing, and relates to a text-guided image segmentation method based on a structured multimodal fusion network, with the starting point of exploring high-quality global multimodal context for collaborative modeling of multimodal fusion structures. Background Technology

[0002] With the advent of the artificial intelligence era, the demand for interaction between humans and computers / intelligent machines is increasing. How to enable machines to understand complex natural language, share the same perspective as humans, observe the world as humans observe it, and perform corresponding actions according to human intentions has become a major hot topic in the industry. Image segmentation, a traditional research area in computer vision, has always attracted much attention and has been widely applied in various fields such as autonomous driving, human-computer interaction, virtual reality, and medical imaging in recent years. Therefore, combining natural language processing with image processing can promote the development of human-computer interaction and achieve barrier-free communication between machines and humans.

[0003] Text-based image segmentation is a research branch of segmentation tasks that closely aligns with practical applications. This task requires a comprehensive understanding of computer vision and natural language processing to correctly segment specific objects within an image described by text. Compared to traditional semantic segmentation tasks that classify each pixel into predefined categories, text-based image segmentation is not limited to predefined categories, making it more challenging. Furthermore, because text may contain diverse semantic content, such as entities (e.g., "person," "house"), attributes (e.g., "large," "white"), and relationships (e.g., "on the right," "at the bottom"), text-based image segmentation is better suited for complex scenarios, such as interactive image editing and text-driven human-computer interaction. In recent years, text-based image segmentation has attracted widespread attention from researchers, and some methods have achieved significant results. However, two major challenges remain unresolved. One major challenge is how the model should understand and align visual and linguistic features to distinguish objects described by text in complex visual and semantic scenes. The other challenge is how the model can capture more information to improve the quality of the segmentation mask for the identified objects. Existing text-based image segmentation methods have modeled the global multimodal context based on a single multimodal fusion structure (such as multimodal trees, multimodal graphs, and cross-membrane Transformers), without exploring the collaborative modeling of high-quality global multimodal contexts by multiple multimodal fusion structures. Our research reveals that fine-tuning the collaborative relationships within and between multimodal fusion structures improves model robustness and generates more accurate segmentation masks. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this invention provides a text-guided image segmentation method based on a structured multimodal fusion network. This method collaboratively models global multimodal contextual information through multiple multimodal fusion structures to distinguish objects described in the text within an image. Furthermore, it aligns high-level multimodal features with low-level visual features to supplement the visual details of the distinguished objects, thereby effectively improving the accuracy of the segmentation results.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A text-guided image segmentation method based on a structured multimodal fusion network includes the following steps:

[0007] (1) Feature extraction:

[0008] Feature extraction is performed on the input RGB images and natural language text. For the RGB images, a convolutional neural network (CNN) is used to extract visual and semantic features. Since this method belongs to the image segmentation branch, the pre-trained parameters of the DeepLab semantic segmentation model are used as the initial parameters of the CNN. Using DeepLab pre-trained parameters effectively reduces network training time and improves the network's generalization ability. For the natural language text, a pre-trained GloVe word embedding model is used to represent each word as a vector. This vector is then input into a bidirectional LSTM (Long Short-Term Memory) network. The hidden state output by the bidirectional LSTM is used as the vector representation of the entire natural language text. This process involves inputting low-rank word vectors into the LSTM and performing multiple iterations to obtain the hidden state, which is then used as the vector representation of the entire sentence.

[0009] (2) Progressive construction of multimodal fusion tree module:

[0010] Since text-based image segmentation tasks require global reasoning based on language and modeling global multimodal context information, a multimodal tree is constructed to perform global reasoning and aggregate global multimodal context information. First, a dependency parser is used to parse the text into a dependency tree, where each tree node represents a word. Then, the high-level visual and linguistic features from step (1) are used as input to the dependency tree. At each tree node, the proposed progressive multimodal fusion module fuses the word features and high-level visual features corresponding to each tree node, forming word-specific multimodal features. To obtain global context information, we further update the information between nodes from bottom to top along the tree structure, that is, using child nodes to update the information of parent nodes until the root node state is updated. Information exchange between nodes is completed by a gating device. At this point, the features of the multimodal tree root node possess global context information.

[0011] (3) Construction of the Transformer module for cross-membrane fusion:

[0012] To further enhance the robustness of the model and obtain more refined multimodal features, we designed a Transformer-based module to further model the deeper interactions between multimodal features and linguistic features based on the multimodal tree modeling in step (2). The cross-membrane fusion Transformer consists of two linear mapping layers, six Transformer encoder layers with positional encoding, and a noise filtering head.

[0013] In the cross-membrane fusion Transformer module, we first process multimodal features and linguistic features, feeding each into a linear mapping layer to obtain new feature maps. Then, the processed features and a special learnable feature are provided to the Transformer encoding layer to model a higher-quality global multimodal context. Furthermore, to obtain more refined multimodal features, a noise filtering head uses a special feature containing rich contextual information to filter noise irrelevant to the object described in the text, highlighting the spatial region containing the object described in the text.

[0014] (4) Multi-level progressive integration and refinement of module construction:

[0015] By constructing the multimodal fusion tree in step (2) and the cross-membrane fusion Transformer in step (3), the visual and textual modal features are precisely aligned, generating a refined multimodal feature that highlights the spatial region where the object described in the text is located. To improve the detail of the segmentation mask and generate an accurate prediction mask, we designed a multi-level progressive fusion refinement module. This module supplements the high-level multimodal features with detailed information from low-level visual features in a progressive fusion manner, enabling the high-level semantic features to have more local details. Finally, the model uses this feature with sufficient local and global information to predict the final segmentation mask.

[0016] Compared with existing technologies, this invention enables text-based image segmentation methods to accurately identify objects described in natural language expressions in complex visual and semantic scenes, distinguishing the text-described objects from other similar objects. The method of this invention can be applied to various fields such as human-computer interaction. Attached Figure Description

[0017] Figure 1 This is a diagram of the overall architecture of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0019] Figure 1 The framework design of the text-based image segmentation method of the present invention is given, and the main process is as follows:

[0020] For visual feature acquisition, all images are first resized to 320×320. A visual feature extraction network is used to extract four-stage visual features V1, V2, V3, and V4. Next, high-level visual features V2 and V3 are downsampled to the size of V4 and then concatenated with V4 along the channel axis to form a multi-level visual feature V. The high-level visual feature V is then fed into a progressive multimodal fusion tree module for multimodal context modeling with language features. The low-level visual feature V1 is fed into a multi-level progressive fusion refinement module for fusion with finer multimodal features to supplement the visual detail information of the highlighted spatial regions. The process of forming the high-level visual feature V is shown in the formula:

[0021] V=ConvBN([Down(V2);Down(V3);V4])

[0022] Where Down(·) represents the downsampling operation, ConvBN(·) represents the 1x1 convolutional layer connected to the BatchNorm layer, and [;] represents the connection operation along the channel axis.

[0023] For text feature acquisition, a pre-trained GloVe word embedding model is used to represent each word as a vector, and then a bidirectional LSTM (Long Short-Term Memory) network is used to obtain the sentence vector representation L. The maximum number of words in the LSTM text is 20. The specific calculation process of the LSTM network is shown in the formula:

[0024] h t =LSTM(x t ,h t-1 )

[0025] Where h t Let x represent the LSTM output vector. t h represents the LSTM input vector. t-1 This represents the hidden output state of the previous LSTM layer.

[0026] To model global contextual information, a progressive multimodal fusion tree is first constructed. Specifically, a dependency parser is used to parse the text into a dependency tree, where each tree node corresponds to a word. The word features output from the bidirectional LSTM are combined with the dependency tree, resulting in each tree node corresponding to a word feature. At each tree node, the proposed progressive multimodal fusion module is used to fuse the high-level visual feature V and the word feature Li corresponding to tree node i. i The multimodal features M corresponding to each tree node i are obtained. i The progressive multimodal fusion module generates the multimodal feature M corresponding to the tree node i. iThe process is shown in the formula:

[0027]

[0028]

[0029]

[0030] M i =Conv([V;M f M vl ])

[0031] in Indicates bilinear fusion. The expression represents element-wise summation, and Conv(·) represents a 1x1 convolution operation.

[0032] To model global contextual information, we use a gating mechanism (GRU) to update the information between nodes from bottom to top along the tree structure until the root node's feature information is updated. At this point, the root node's features have preliminary global contextual information.

[0033] To model higher-quality multimodal contexts, the multimodal features and linguistic features of the multimodal tree root node are fed into the proposed cross-membrane fusion Transformer to model the direct interactions between elements, highlighting spatial regions described by text and suppressing signal responses from spatial regions not described by text. Specifically, the multimodal features and linguistic features are first fed into a linear mapping layer to obtain the multimodal embedding t. m and language embeddings l In addition, a learnable feature t is embedded. s It learns a global context to evaluate whether each pixel contains the target object and fuses it with multimodal embeddings to filter out noise from irrelevant spatial regions (i.e., background) to obtain higher-quality multimodal context. These joint embeddings can be represented as:

[0034]

[0035] Where N m This represents the number of multimodal embeddings, where T represents the number of language embeddings.

[0036] The joint embedding is fed into the Transformer encoder for inference, resulting in the multimodal embedding t′. m Language embedding l ′ and learnable embeddings t s '. The feature t s Expanded into feature t' mThe reshaped shape is then integrated with it to reduce noise interference from spatial regions not described in the text. The noise filtering process can be represented as:

[0037]

[0038] M cmf =ConvBN([M c ;M])

[0039] Where M cmf For cross-modal fusion of Transformer output features, Reshape(·) is the feature shape reshaping operation, and Tile(·) is the feature unfolding operation.

[0040] To generate accurate segmentation masks, fine-grained multimodal features M with global contextual information are obtained. cmft Then, ASPP is first used to extract M. cmf High-level multimodal features M are obtained from multi-scale contextual information. high Then, the proposed multi-level progressive fusion refinement module is used to refine M. high Integration with visual feature V1, which possesses rich visual detail. Multi-layered progressive fusion and refinement module integration M. high The process of generating the final segmentation mask from V1 is shown in the formula:

[0041] M high =Upsample(ConvBN(M high ))

[0042] V1 = ConvBN(V1)

[0043]

[0044]

[0045] M seg =Sig(Conv r ([M high M p M prog ]))

[0046] Where Upsample is the upsampling operation, Conv r (·) represents stacked convolutional layers, and Sig(·) represents the sigmoid activation function.

[0047] This invention uses a multimodal tree and a cross-modal Transformer to jointly model multimodal contextual information. Finally, it integrates refined multimodal features with global contextual information with low-level visual detail features to obtain features with both high-level semantics and low-level visual detail. These features are then fed into stacked convolutional layers and classified pixel-by-pixel using a sigmoid activation function to generate the final segmentation result.

[0048] Example

[0049] In this embodiment, the deep learning framework PyTorch is used on a GTX 2080 12G graphics card.

[0050] Dataset: Experimental evaluation was conducted on the standard public dataset RefCOCO. This dataset includes 19,994 images and 142,209 sentences of natural language text. RefCOCO is one of the most commonly used datasets for text-based image segmentation tasks.

[0051] Ablation experiments: To demonstrate the effectiveness of each step in the text-guided image segmentation method based on a structured multimodal fusion network, the IoU metric was tested on the RefCOCO dataset. The results are shown in Table 1. The ablation experiments demonstrate that the method of this invention can effectively improve the accuracy of the results.

[0052] Table 1. Segmentation results of ablation experiments with different step combinations

[0053]

[0054] Experimental results demonstrate the effectiveness of this invention. Compared to existing technologies, this invention can accurately identify objects described in text from complex visual and semantic contexts, and clearly distinguishes the objects described in text from similar objects by integrating high-level semantic and low-level visual detail information. The method has strong robustness.

Claims

1. A text-guided image segmentation method based on a structured multimodal fusion network, characterized in that, The steps are as follows: (1) Feature extraction Feature extraction is performed on the input RGB images and natural language text. For RGB images, a convolutional neural network is used to extract visual and linguistic features. Since this method belongs to the image segmentation branch, the pre-trained parameters of the DeepLab semantic segmentation model are used as the initial parameters of the convolutional neural network. Using DeepLab pre-trained parameters can effectively reduce the network training time and improve the network's generalization ability. For natural language text, a pre-trained GloVe word embedding model is used to represent each word as a vector. This vector is then input into a bidirectional LSTM long short-term memory network. The hidden state output by the bidirectional LSTM is used as the vector representation of the entire natural language text. This process involves inputting low-rank word vectors into the LSTM and passing them through multiple loops to obtain the hidden state as the vector representation of the entire sentence. (2) Progressive construction of multimodal fusion tree module Since text-based image segmentation tasks require global reasoning of images based on language and modeling of global multimodal context information, a multimodal tree is constructed to perform global reasoning and aggregate global multimodal context information. First, the text is parsed into a dependency tree using a dependency parser, where each tree node represents a word. Then, the high-level visual features and language features from step (1) are used as input to the dependency tree. At each tree node, the word features and high-level visual features corresponding to each tree node are fused using the proposed progressive multimodal fusion module to form word-specific multimodal features. In order to obtain global context information, the information between nodes is further updated from bottom to top along the tree structure, that is, the information of the parent node is updated using the child node until the root node state is updated. The information exchange between nodes is completed by a gating device. At this time, the features of the multimodal tree root node have global context information. (3) Construction of the Transformer module for cross-modal fusion To further improve the robustness of the model and obtain more refined multimodal features, a Transformer-based module was designed to further model the deeper interaction between multimodal features and language features based on the multimodal tree modeling in step (2); the cross-modal fusion Transformer includes two linear mapping layers, six Transformer encoder layers with position encoding, and a noise filtering head; In the cross-modal fusion Transformer module, multimodal features and linguistic features are processed first. The two features are then fed into a linear mapping layer to obtain new feature maps. The processed features and a special learnable feature are then provided to the Transformer encoding layer to model a higher quality global multimodal context. Furthermore, in order to obtain more refined multimodal features, a noise filtering head contains special features with rich contextual information to filter out noise that is irrelevant to the object described in the text and highlights the spatial region where the object described in the text is located. (4) Multi-level progressive integration and refinement of module construction: By constructing the multimodal fusion tree in step (2) and the cross-modal fusion Transformer in step (3), the visual and textual modal features are precisely aligned, generating a refined multimodal feature. This multimodal feature highlights the spatial region where the object described in the text is located. In order to improve the details of the segmentation mask and generate an accurate prediction mask, a multi-level progressive fusion refinement module is designed. This module supplements the high-level multimodal features with the detailed information of the low-level visual features in a progressive fusion manner, so that the high-level semantic features have more local details. Finally, the model uses this feature with sufficient local and global information to predict the final segmentation mask.

2. The text-guided image segmentation method based on a structured multimodal fusion network according to claim 1, characterized in that, The progressive multimodal fusion tree module construction includes the following steps: The progressive multimodal fusion module generates the multimodal features corresponding to tree nodes 𝑖. The process is shown in the following formula: ； ； ； ； in: These are the visual features extracted by the visual encoder; It is the word feature corresponding to the tree node 𝑖; Indicates bilinear fusion; This indicates an element-wise summation operation; This represents a 1x1 convolution operation; [;] indicates a connection along the channel axis.

3. The text-guided image segmentation method based on a structured multimodal fusion network according to claim 1, characterized in that, The specific steps for constructing the cross-modal fusion Transformer module are as follows: First, multimodal features and linguistic features are processed, and the two features are fed into a linear mapping layer to obtain new feature maps. Then, the processed features and a special learnable feature are provided to the Transformer encoding layer to model a higher quality global multimodal context. Furthermore, in order to obtain more refined multimodal features, a special feature containing rich contextual information is used in the noise filtering head to filter out noise that is irrelevant to the object described in the text and to highlight the spatial region where the object described in the text is located. The noise filtering process is represented as follows: ； ； in: For feature shape reshaping operations; Feature expansion operation; The output of the Transformer encoding layer is the inference result corresponding to the input multimodal features; The output of the Transformer encoding layer is the inference result corresponding to the input learnable features; This refers to the output features of the Transformer for cross-modal fusion.

4. The text-guided image segmentation method based on a structured multimodal fusion network according to claim 1, characterized in that, The multi-level progressive fusion and refinement module integrates high-level multimodal features. and low-level visual features The process of generating the final segmentation mask is shown in the following formula: ) ； ； ； ； ； in: This is an upsampling operation; It is a 1x1 convolutional layer connected to the BatchNorm layer; This represents stacked convolutional layers; This represents the sigmoid activation function.

Images