A reference image segmentation method based on cross-modal dual graph alignment

Abstract

This paper discloses a reference image segmentation method based on cross-modal dual graph alignment, belonging to the field of multimodal image segmentation. This method innovatively proposes a "part-unity-whole" paradigm, mapping the extracted visual and textual features to a unified latent representation structure before performing cross-modal fusion. This facilitates the explicit alignment information extracted by the model, effectively enhancing the final segmentation effect.

Description

Technical Field

[0001] This paper belongs to the field of multimodal image segmentation. It uses text as a reference to generate masks that correspond to objects in the image that are most relevant to the text description. This method extracts explicit and detailed cross-modal alignment information through a proposed dual graph module and a hierarchical interaction module, achieving good segmentation results based on the obtained cross-modal information. Background Art

[0002] With the continuous development of computer vision, many single-modal image segmentation networks have been proposed, and they have achieved considerable success in fields such as autonomous driving. However, for other applications, such as text-based image editing and human-computer interaction, single-modal image segmentation cannot meet the needs. Multimodal image segmentation methods have more extensive applications and greater significance in real life.

[0003] In multimodal image segmentation, text-based reference image segmentation is a hot topic. This task is challenging because, on the one hand, the model needs to fully understand the components and context of both modalities; on the other hand, it also needs to accurately align heterogeneous information across modalities. Therefore, in the reference image segmentation task, obtaining initial features of a specific modality and performing cross-modal fusion are crucial.

[0004] Many excellent methods have been proposed for obtaining initial features for specific modalities. For visual information, CNN-based encoders or visual Transformer-based encoders are commonly used to extract visual features corresponding to the input image. For textual information, the mainstream approach currently uses RNN-based encoders or Transformer-based encoders to extract textual features corresponding to the input text. Regarding cross-modal information fusion, current methods mainly adopt concatenation-based methods and attention-based methods. Some work also incorporates information about the syntactic structure of sentences to enhance the representation of textual features. However, these cross-modal fusion methods struggle to capture hierarchical correlations across modal structures. Furthermore, due to the different data styles of the two modalities and the distinct structural characteristics of the encoders for visual and textual modalities, the extracted features of the different modalities have completely different latent representation spaces. Direct cross-modal fusion, however, is difficult to obtain accurate cross-modal alignment information. Therefore, it is particularly meaningful to form a unified representation structure based on dual graphs and to perform hierarchical interactions between cross-modal features.

[0005] In the prior art paper "Linguistic Structure Guided Context Modelling for Refering Image Segmentation," published at ECCV 2020, a leading computer vision conference, the authors constructed a graph neural network for text information to better capture the contextual relationships between words in a sentence. However, this paper only constructed a graph structure for the single modality of text. For the visual modality, the encoder was still used to extract initial visual features, and the unification of the representation space was not achieved by constructing a common graph structure. Therefore, during cross-modal fusion, the features of the two modalities remain in different latent representation spaces, which causes errors in capturing inter-modal alignment information. Summary of the Invention

[0006] The purpose of the present invention is to solve the following technical problems:

[0007] First, existing reference image segmentation methods usually use visual and text encoders with significantly different structures, and the data of different modalities input into each encoder have inherently inconsistent styles. This results in the extracted visual and text features having completely different latent representation spaces, which to a certain extent hinders accurate cross-modal alignment. To address this problem, the present invention designs a dual graph module, which constructs graph structures in different modalities, and maps visual and text features with different latent representation spaces to a unified graph representation space through information propagation and aggregation between nodes in the graph neural network reasoning process. This helps the model capture explicit and accurate cross-modal alignment information, effectively improving the effect of modal fusion.

[0008] Second, existing reference image segmentation methods typically use methods based on splicing, attention mechanisms, and the introduction of syntactic structure information to obtain cross-modal correlation information for global representations or heterogeneous contextual information from different modalities. These fusion methods are not comprehensive in capturing cross-modal correlations because they ignore the capture of cross-modal hierarchical associations. In this invention, by performing entity-level and graph-level interactions on features from different modalities, the cross-modal hierarchical association information is fully captured, effectively improving the segmentation performance of the model.

[0009] The technical solution of the present invention is: a reference image segmentation method based on cross-modal dual graph alignment, the method comprising:

[0010] Step 1: To further enhance text information and facilitate subsequent capture of long-range context of sentences, the input text is first preprocessed. This preprocessing includes word part-of-speech tagging and obtaining syntactic structure based on a dependency parse tree.

[0011] We use the open-source natural language processing library SpaCy to implement part-of-speech tagging. We reduce word part-of-speech categories to seven: noun, adjective, verb, preposition, adverb, other parts of speech, and zero-padding. We use one-hot encoding to represent these seven parts of speech.

[0012] The dependency parse tree method is used to obtain the subordinate relationship between words in the sentence, preparing for the subsequent construction of the graph structure;

[0013] Step 2: Input the image I and text expression S into the encoder of each modality respectively to obtain 4 scale visual features V i and text features L, i = 0, 1, 2, 3; then, the part-of-speech one-hot encoding extracted in step 1 is passed through a multi-layer fully connected layer encoding network to obtain the part-of-speech feature P. Finally, the text feature L and the part-of-speech feature P are element-wise multiplied to fully integrate the features corresponding to each word with its part-of-speech feature. The specific formula is as follows:

[0014] L P =ReLU(Conv(L⊙P)) (0.1)

[0015] Among them, ⊙ represents the matrix element-level multiplication operation, Conv represents the convolution layer, ReLU represents the activation function, and L P Represents text features that incorporate part-of-speech information;

[0016] Step 3: For the lowest level visual feature V0, perform cross-modal fusion;

[0017] First, the initial text feature L obtained from the text encoder is passed through a linear mapping layer and a Softmax calculation function to obtain the weight ω corresponding to each word feature; according to the weight, the feature vector corresponding to each word in the text feature L is weighted and operated to obtain the overall feature vector h of the sentence. c ; Finally, h c Flatten it and concatenate it with V0 and the artificially defined spatial coordinate O0 along the channel dimension, and output the final multimodal feature M0 at this scale through a convolutional layer; the specific formula can be expressed as:

[0018] M0=Conv([V0;Tile(h c );O0]) (0.2)

[0019] Among them, Tile represents the tiling operation, [;] represents the splicing operation, and Conv represents the convolution layer;

[0020] Step 4: For the higher-level visual features V1, V2, and V3, use the dual graph module and the hierarchical interaction module to perform cross-modal fusion. The operations for V1, V2, and V3 are the same, and the corresponding subscripts are omitted below to explain the method.

[0021] The first step is to extract entities and integrate the text features L that incorporate part-of-speech information P Treated as a text entity and marked with L E In terms of vision, the spatial coordinate O is concatenated with the visual feature V and then convolved to obtain the visual entity V E ; Then use the entity-level HIM to interact with entities of different modalities to obtain the entity-level multimodal feature M E ; Then, construct the visual graph and text graph, and E With text entity L E Perform graph reasoning within the modality respectively, and use V G With L G Representation; then use the graph-level HIM to interact with the graph-based features of different modalities to obtain the graph-level multimodal feature M G ; Finally, M E With M G Perform splicing operations to integrate entity-level interaction results with graph-level hierarchical interaction results to obtain the final multimodal feature M at this scale;

[0022] Step 5: After the above steps, multimodal features M of different scales are obtained i , i = 0, 1, 2, 3, a high-to-low path is designed to extract multimodal features at all scales. The specific formula is:

[0023]

[0024] Among them, Upsample represents the upsampling operation, [;] represents the concatenation operation, and Conv represents the convolution layer;

[0025] Step 6: Input Y0 into the segmentation head composed of multiple convolutional layers and upsample the output to obtain the final prediction result Using the binary cross entropy function as the loss function and using Y to represent the true segmentation value, the prediction loss can be expressed as:

[0026]

[0027] Among them, the subscripts i and j represent the pixel positions of the predicted results and the true values; H and W represent the size of the real image mask.

[0028] Furthermore, in order to enhance the multi-layer perception capability of the model, a series of auxiliary losses are set; the specific method is as follows:

[0029] The multimodal features M at each scale i All are input into the same segmentation head as above to obtain multi-scale prediction values Then downsample the true value Y by different multiples to obtain the true value at different scales Multi-scale auxiliary loss Expressed as:

[0030]

[0031] The final loss L T Expressed as:

[0032]

[0033] Among them, λ, λ0, λ1, λ2, and λ3 are hyperparameters to balance different losses. λ is set to 1.0, and λ0, λ1, λ2, and λ3 are set to 0.5.

[0034] Furthermore, the dual graph module in step 4 is divided into two branches: a text graph branch and a visual graph branch;

[0035] In the text graph branch, the part-of-speech-aware text feature L P Considered as a text entity, denoted as L E , then, the text entity L E Mapped to the initial node Δ of the text graph L , and construct the directed edges of the graph based on the result of the dependency parse tree in the preprocessing step, and express it in the form of an adjacency matrix, denoted as M L Finally, the graph convolutional network is used to perform graph reasoning on the initial nodes and edges to update the information between nodes; update the nodes The process formula is expressed as:

[0036]

[0037] in, and is a learnable parameter, d(·) represents the degree of the node, η(i) represents the set of child nodes of node i, and finally, the graph-based text feature output by the text graph branch is denoted as L G ;

[0038] In the visual graph branch, the visual features after incorporating the absolute position information are regarded as verse nodes, denoted as V E , V E Mapped to the initial node Δ of the visual graph V, and use two learnable matrices W1 and W2 and the visual initial node Δ V The edges of the graph are constructed and represented in the form of an adjacency matrix, denoted as M V Finally, the graph convolutional network is also used to perform graph reasoning on the initial nodes and edges. This process is expressed as follows:

[0039]

[0040] in, is a learnable parameter, and the graph-based visual features output by the visual graph branch are denoted as V G .

[0041] Furthermore, the hierarchical interaction module in step 4 is implemented based on the attention mechanism, using two identical modules to implement entity-level and graph-level interactions respectively; the input visual features and text features are denoted as V and L respectively, and they are respectively subjected to 1×1 convolution to obtain V p With L p , which is used to calculate the cross-modal affine matrix, which is expressed as:

[0042] M Affine =Softmax(L p reshape(V p )) (0.9)

[0043] Among them, M Affine Represents the cross-modal affine matrix, reshape represents the matrix shape transformation operation, and Softmax represents the Softmax function. After obtaining the cross-modal affine matrix, the multimodal features can be calculated by the following formula:

[0044] L a =Conv(L q T M Affine ) (0.10)

[0045] M=ReLU(Conv(L a ⊙V a )) (0.11)

[0046] Among them, T represents the transpose operation, Conv represents the convolution layer, ⊙ represents the matrix element-level multiplication operation, ReLU represents the activation function, V a With L q They are obtained by 1×1 convolution mapping from V and L respectively, and M represents the entity-level / graph-level multimodal features obtained after interaction.

[0047] In the present invention, a dual graph module and a hierarchical interaction module are innovatively proposed. The dual graph module constructs graph structures for the visual modality and the text modality respectively, and uses a graph convolutional network to perform graph reasoning. While capturing long-distance contextual relationships within the modalities, it maps the features of the visual and text modalities in different latent representation structures to a unified latent representation structure based on a graph. This is conducive to the subsequent capture of explicit alignment information between modalities and reduces the error of cross-modal fusion. In terms of the hierarchical interaction module, it is used for entity-level interaction and graph-level interaction, which is conducive to capturing sufficient cross-modal correlation. Thanks to the proposed dual graph module and hierarchical interaction module, on the one hand, the method of the present invention has a high gain in objective indicators compared to other current high-performance methods; on the other hand, the method of the present invention can obtain high-quality segmentation masks based on the input image and text. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a schematic diagram of the "part-unity-whole" paradigm of the present invention;

[0049] Figure 2 The present invention provides a reference image segmentation method based on cross-modal dual graph alignment;

[0050] Figure 3 Schematic diagram of the dual graph module of the present invention;

[0051] Figure 4 is a schematic diagram of the hierarchical interaction module of the present invention;

[0052] Figure 5 Schematic diagram of the visualization result of reference image segmentation of the present invention. DETAILED DESCRIPTION

[0053] The present invention proposes a reference image segmentation method based on cross-modal dual graph alignment. Most existing methods adopt a "part-whole" paradigm, that is, first use the encoders of each modality to extract visual and text features respectively, and then adopt a multimodal fusion strategy to obtain the overall multimodal features. Different from the "part-whole" paradigm, the present invention innovatively proposes a "part-unified-whole" paradigm, that is, first map the extracted visual and text features to a unified potential representation structure, and then perform cross-modal fusion. This is conducive to the explicit alignment information extracted by the model, which can effectively enhance the final segmentation effect. The "part-unified-whole" paradigm is as follows Figure 1 As shown;

[0054] To achieve the above-mentioned "part-unification-whole" paradigm, the present invention first uses two deep encoders to extract initial features from the input image and text respectively; then, a dual graph module is used to construct graph structures for the visual and text modalities respectively, and a graph convolutional network (GCN) is used to perform graph reasoning respectively to fully propagate and aggregate node information. In this process, a unified representation structure is formed for heterogeneous data of different modalities; then, in order to fully extract cross-modal mutual correlation information, a hierarchical interaction module based on the attention mechanism is adopted, and it is applied to entity-level and graph-level hierarchical interactions; finally, multi-scale features are integrated through a high-to-low path, and the final prediction mask is output through the convolutional layer. Figure 2 The overall network architecture of the method proposed in this invention is presented;

[0055] Figure 2 In, V i (i=0,1,2,3) represents the multi-scale visual features extracted by the visual encoder, HIM represents the hierarchical interaction module, Softmax represents the Softmax function operation, M i (i=0,1,2,3) represents the multimodal features at different scales obtained by model inference, Represents the prediction results generated based on multimodal features of different scales, Represents the true value of the mask at different scales, Represents the auxiliary losses corresponding to different scales, represents the final prediction generated by the model after top-down multi-scale integration, Y represents the true value of the mask corresponding to the input image, L seg is the loss function between the final prediction and the true value.

[0056] This method was experimentally evaluated on the public reference image segmentation dataset RefCOCO and compared with other mainstream reference image segmentation methods. As shown in Table 1, compared with the current high-performance ReSTR method, using the segmentation intersection over union (IoU) as the evaluation metric, the proposed method improved by 3.11% on the validation set of the RefCOCO dataset, 2.99% on test set A, and 2.44% on test set B. In addition, the proposed method also visualized the masks predicted by the model and compared them with the true masks. The comparison results are shown in Table 1. Figure 5 The results show that the proposed model can generate high-quality image segmentation masks.

[0057] Table 1 Objective performance evaluation of this research method on the RefCOCO dataset

[0058] method Image resolution Validation set Test set A Test set B RMI 320×320 45.18 45.69 45.57 CMPC 320×320 61.36 64.54 59.64 LSCM 320×320 61.47 64.99 59.55 EFN 320×320 62.76 65.69 59.67 CGAN 416×416 64.86 68.04 62.07 VLT 416×416 65.65 68.29 62.73 ReSTR 480×480 67.22 69.30 64.45 Methods of this study 416×416 70.33 72.29 66.89

Claims

1. A reference image segmentation method based on cross-modal dual graph alignment, the method comprising: Step 1: To further enhance text information and facilitate subsequent capture of long-range context of sentences, the input text is first preprocessed. This preprocessing includes word part-of-speech tagging and obtaining syntactic structure based on a dependency parse tree. We use the open-source natural language processing library SpaCy to implement part-of-speech tagging. We reduce word part-of-speech categories to seven: noun, adjective, verb, preposition, adverb, other parts of speech, and zero-padding. We use one-hot encoding to represent these seven parts of speech. The dependency parse tree method is used to obtain the subordinate relationship between words in the sentence, preparing for the subsequent construction of the graph structure; Step 2: Input the image I and text expression S into the encoder of each modality respectively to obtain 4 scale visual features V i and text features L, i = 0, 1, 2, 3; then, the part-of-speech one-hot encoding extracted in step 1 is passed through a multi-layer fully connected layer encoding network to obtain the part-of-speech feature P. Finally, the text feature L and the part-of-speech feature P are element-wise multiplied to fully integrate the features corresponding to each word with its part-of-speech feature. The specific formula is as follows: IT P =ReLU(Conv(L⊙P)) Among them, ⊙ represents the matrix element-level multiplication operation, Conv represents the convolution layer, ReLU represents the activation function, and L P Represents text features that incorporate part-of-speech information; Step 3: For the lowest level visual feature V0, perform cross-modal fusion; First, the initial text feature L obtained from the text encoder is passed through a linear mapping layer and a Softmax calculation function to obtain the weight ω corresponding to each word feature; according to the weight, the feature vector corresponding to each word in the text feature L is weighted and operated to obtain the overall feature vector h of the sentence. c ; Finally, h c Flatten it and concatenate it with V0 and the manually defined spatial coordinate O0 along the channel dimension, and output the final multimodal feature M0 at this scale through a convolutional layer; the specific formula is expressed as: M0=Conv([V0;Tile(h c );O0]) Among them, Tile represents the tiling operation, [;] represents the splicing operation, and Conv represents the convolution layer; Step 4: For the higher-level visual features V1, V2, and V3, use the dual graph module and the hierarchical interaction module to perform cross-modal fusion. The operations for V1, V2, and V3 are the same, and the corresponding subscripts are omitted below to explain the method. The first step is to extract entities and integrate the text features L that incorporate part-of-speech information P Treated as a text entity and marked with L E In terms of vision, the spatial coordinate O is concatenated with the visual feature V and then convolved to obtain the visual entity V E ; Then use the entity-level HIM to interact with entities of different modalities to obtain the entity-level multimodal feature M E ; Then, construct the visual graph and text graph, and E With text entity L E Perform graph reasoning within the modality respectively, and use V G With L G Representation; then use the graph-level HIM to interact with the graph-based features of different modalities to obtain the graph-level multimodal feature M G ; Finally, M E With M G Perform splicing operations to integrate entity-level interaction results with graph-level hierarchical interaction results to obtain the final multimodal feature M at this scale; Step 5: After the above steps, multimodal features M of different scales are obtained i , i = 0, 1, 2, 3, a high-to-low path is designed to integrate multimodal features at each scale. The specific formula is: Among them, Upsample represents the upsampling operation, [;] represents the concatenation operation, and Conv represents the convolution layer; Step 6: Input Y0 into the segmentation head composed of multiple convolutional layers and upsample the output to obtain the final prediction result Using the binary cross entropy function as the loss function and using Y to represent the true value of the segmentation, the prediction loss is expressed as: Among them, the subscripts i and j represent the pixel positions of the predicted results and the true values; H and W represent the size of the real image mask.

2. The reference image segmentation method based on cross-modal dual graph alignment according to claim 1, characterized in that: In order to enhance the multi-layer perception ability of the model, a series of auxiliary losses are set; the specific method is: The multimodal features M at each scale i All are input into the same segmentation head as above to obtain multi-scale prediction values , and then downsample the true value Y by different multiples to obtain the true value at different scales Multi-scale auxiliary loss Expressed as: The final loss L T Expressed as: Among them, λ, λ0, λ1, λ2, and λ3 are hyperparameters to balance different losses. λ is set to 1.0, and λ0, λ1, λ2, and λ3 are set to 0.

5.

3. The reference image segmentation method based on cross-modal dual graph alignment according to claim 1, characterized in that: The dual graph module in step 4 is divided into two branches: a text graph branch and a visual graph branch; In the text graph branch, the part-of-speech-aware text feature L P Considered as a text entity, denoted as L E , then, the text entity L E Mapped to the initial node Δ of the text graph L , and construct the directed edges of the graph based on the result of the dependency parse tree in the preprocessing step, and express it in the form of an adjacency matrix, denoted as M L Finally, the graph convolutional network is used to perform graph reasoning on the initial nodes and edges to update the information between nodes; update the node δ i L The process formula is expressed as: in, and is a learnable parameter, d(·) represents the degree of the node, η(i) represents the set of child nodes of node i, and finally, the graph-based text feature output by the text graph branch is denoted as L G ; In the visual graph branch, the visual features after incorporating the absolute position information are regarded as visual nodes, denoted as V E , V E Mapped to the initial node Δ of the visual graph V , and use two learnable matrices W1 and W2 and the visual initial node Δ V The edges of the graph are constructed and represented in the form of an adjacency matrix, denoted as M V Finally, the graph convolutional network is also used to perform graph reasoning on the initial nodes and edges. This process is expressed as follows: in, is a learnable parameter, and the graph-based visual features output by the visual graph branch are denoted as V G .

4. The reference image segmentation method based on cross-modal dual graph alignment according to claim 1, wherein: The hierarchical interaction module in step 4 is implemented based on the attention mechanism. Two identical modules are used to implement entity-level and graph-level interactions respectively. The input visual features and text features are denoted as V and L respectively, and V is obtained by performing 1×1 convolution on them. p With L p , which is used to calculate the cross-modal affine matrix, which is expressed as: M Affine =Softmax(L p reshape(V p )) Among them, M Affine Represents the cross-modal affine matrix, reshape represents the matrix shape transformation operation, and Softmax represents the Softmax function. After obtaining the cross-modal affine matrix, the multimodal features are calculated by the following formula: L a =Conv(L q T M Affine ) M=ReLU(Conv(L a ⊙V a )) Among them, T represents the transpose operation, Conv represents the convolution layer, ⊙ represents the matrix element-level multiplication operation, ReLU represents the activation function, V a With L q They are obtained by 1×1 convolution mapping from V and L respectively, and M represents the entity-level / graph-level multimodal features obtained after interaction.

Images