A method for image segmentation based on semantic relation mining and reasoning

Abstract

The application belongs to the technical field of computer vision, and is a referring image segmentation method based on semantic relation mining and reasoning. First, a text parser is used to parse the input text to obtain entities in the text and the relations between the entities, and a corresponding scene graph is generated; then the text and the image are respectively input into a text and visual encoder to obtain text and visual features, and the features are fused to obtain visual features embedded with the text; then the visual features embedded with the text and the scene graph are jointly input into a graph convolution module for feature updating; finally, a reasoning fusion module is used to finally fuse the graph convolution output and the visual features embedded with the text, and a contrast learning loss function is used for training, and finally more accurate segmentation masks are obtained. The application uses a scene graph, graph convolution and an attention mechanism to further and deeply fuse and update the context information and image features, and further improves the accuracy of the referring image segmentation method on multiple data sets.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a referring image segmentation method based on semantic relation mining and reasoning. BACKGROUND

[0002] In our daily life, we are exposed to a wealth of information, among which text and image information occupy an important position. Referring image segmentation is an intersection of computer vision and natural language processing, aiming to combine natural language description with image information to accurately locate and segment the target in the image. This task requires the model not only to understand the relationship between entities expressed in the text, but also to be able to effectively apply these information to image segmentation.

[0003] Current referring image segmentation algorithms can be roughly divided into two categories: single-stage scheme and multi-stage scheme.

[0004] The general process of single-stage scheme is to first extract the features of image and text using convolutional neural network (CNN) and text encoder (such as LSTM) respectively. Then, different methods are used to fuse these features to obtain the target segmentation result. The main difference between these methods is how to effectively integrate the information of the two modalities. Early methods use cascading way to fuse, but this method is difficult to effectively align the long distance information between the two modalities. Some researchers, such as Chen Yiwu of University of California, published Referring Expression Object Segmentation with Caption-Aware Consistency, use dynamic filters to realize fusion. These filters can be regarded as the convolution kernel of convolutional neural network, abstracting the text information into convolution kernel, and then fusing the information of the two modalities through corresponding convolution operation. However, since convolution operation mainly focuses on local information, this way may cause the model to pay too much attention to details.

[0005] Multi-stage approaches, on the other hand, require first extracting candidate object regions from the image using traditional object detection methods (such as Fast R-CNN), and then selecting the object that best matches the input text from these regions as the final result. In the paper MAttNet: Modular Attention Network for Referring Expression Comprehension by Yuxing Yu et al. from Shanghai Jiao Tong University, a modular attention model is proposed, which parses the input text into subject, location, and relationship parts, and then uses different attention modules to match candidate regions for each part. The final target candidate box is obtained by weighted summation, and then input into the segmentation branch to get the segmentation result. On the other hand, in the paper Learning to Assemble Neural Module Tree Networks for Visual Grounding by Daqing Liu et al. from University of Science and Technology of China, a tree-shaped neural network module is proposed, which uses a semantic parsing tree to parse the input text. The nodes in the tree represent the words in the text, and the edges represent the relationship between the words. Each word calculates the similarity with the candidate region, and selects the highest score region as the output, and then inputs the segmentation branch to get the final segmentation result. However, one obvious disadvantage of multi-stage methods is that the selection of candidate regions is very time-consuming, which affects the test speed of the model. In contrast, single-stage algorithms do not require candidate box extraction and are generally more efficient.

[0006] Through the above methods, we can see that there are many methods in the field of referring image segmentation, each method has its unique advantages and challenges. Continuous research and exploration will help better combine natural language and image information and promote the development of this field.

[0007] The above methods have the following two problems:

[0008] (1) The complex and difficult-to-understand deep context semantic information in the input text is not fully utilized;

[0009] (2) Existing models do not align different types of semantic features and visual features step by step. SUMMARY

[0010] In order to solve the above problems, the present application proposes a referential image segmentation method based on semantic relation mining and reasoning. The algorithm first uses a text parser to parse the text in the data set to obtain entity words and relation words in the text, and generates a corresponding text scene graph; meanwhile, the text and image are input into the text encoder and visual encoder respectively to extract features, and the two features are input into the attention fusion module in the encoding stage to obtain the fused multi-modal feature; then the fused multi-modal feature is added as a global node to the scene graph information, and the scene graph is input into the graph convolution module for feature updating and fusion; finally, the features obtained by the graph convolution are used as the enhanced scene graph features, which are input into the proposed reasoning module together with the enhanced multi-modal features obtained before for step-by-step reasoning and fusion, and a contrast learning loss function is used for training, and finally the segmentation mask of the object expressed in the expression is obtained.

[0011] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0012] A referential image segmentation method based on semantic relation mining and reasoning, the specific implementation steps are as follows:

[0013] Step S1: feature preprocessing: according to the text and its corresponding image obtained from the data set, the text and visual encoder are input to obtain the text feature and image feature respectively;

[0014] Further, the step S1 is specifically:

[0015] Step S11: input the text in the data set into the text encoder, which uses the BERT model, thereby extracting the word-level text feature , wherein is the number of channels of the text feature, and T is the number of words in the text;

[0016] Step S12: use the four-stage visual transformer layer to extract rich image features from the image and the output of the previous stage, that is, the visual feature, represented as , respectively represent the number of channels in the stage, the height and width of the feature map.

[0017] Step S2: input the text feature and the visual feature into the visual fusion module to obtain the multi-modal feature, that is, the text embedded visual feature;

[0018] Further, the step S2 is specifically:

[0019] Step S21: the visual feature extracted by step S12 is input into the visual fusion module together with the text feature combining to generate a set of multi-modal features, i.e., visual features of text embeddings;

[0020] Step S22: each element in each visual feature of text embedding is weighted by a learnable gating unit, and then added element by element to the visual feature to generate a set of enhanced multi-modal features, denoted as , where , respectively, represent the number of channels, the height and the width of the feature map in the level.

[0021] Step S3: send the text in the data set into the text parser to generate a scene graph in the form of a dictionary, and send it into the text encoder to obtain a node feature matrix in the scene graph and an entity relationship adjacency matrix;

[0022] Further, the step S3 is specifically:

[0023] Step S31: send the text in the data set into the text parser to obtain a scene graph in the form of a dictionary, which contains all entity words and the relationships between the entity words;

[0024] Step S32: parse the scene graph in the form of a dictionary, and create an entity relationship adjacency matrix for each text according to the parsing result;

[0025] Step S33: extract the entity words in the scene graph in the form of a dictionary as node information, and send it into the text encoder to obtain node features containing entity word information.

[0026] Step S4: add the enhanced multi-modal features to the scene graph node feature matrix to form a scene graph node feature containing image information and an adjacency matrix, and input them into the graph convolutional neural network for updating;

[0027] Further, the step S4 is specifically:

[0028] Step S41: according to the enhanced multi-modal features of the fourth stage obtained in step S22 , as a global node feature, add it to the node feature matrix obtained in step S33, and add a global entity in the adjacency matrix, and set the adjacency relationship between the global entity and each entity word in the dictionary scene graph to 1, indicating that the global node feature has a relationship with all entity nodes, to obtain an enhanced scene graph;

[0029] Step S42: input the enhanced node feature matrix in the enhanced scene graph and the relationship adjacency matrix obtained in step S41 into the graph convolutional neural network, update the node features by the enhanced multi-modal features, enhance the part corresponding to the vision in the text information, mine the relationship between the nodes, highlight the features related to the referents, and output the features of the GCN network as the updated scene graph features.

[0030] Step S5: gradually fuse the multi-level enhanced multi-modal features and the updated scene graph features in the attention reasoning module, cascade the fusion results, and obtain the features of the object to be segmented finally.

[0031] Further, the step S5 is specifically:

[0032] Step S51: use three attention reasoning modules to fuse the enhanced multi-modal features obtained in step S22 respectively as one of the inputs of the attention reasoning module, and the other input uses the updated scene graph features obtained in step S42.

[0033] Step S52: fuse the two inputs in the attention reasoning module, calculate two attention weight distributions by using the softmax and sigmoid activation functions respectively, and obtain two attention maps, namely the aggregation attention map and the propagation attention map .

[0034] Step S53: use the aggregation attention map as the weight to perform weighted summation operation on the enhanced multi-modal features , and obtain the aggregation attention feature .

[0035] Step S54: based on the propagation attention map , convert the aggregation attention feature into a feature matrix with the same dimension as the enhanced multi-modal features , denoted as ; combine and by using element addition, and obtain a new feature mapping by another convolutional layer.

[0036] Step S55: obtain three feature mappings , and finally obtain the segmentation features of the segmented object by the step-by-step cascading operation.

[0037] Step S6: the attention reasoning module is trained by a contrastive learning loss function, the overall model is trained by a cross-entropy loss function, and the weight parameters of the two losses are adaptively updated, so that the algorithm prediction result fits the correct result; the obtained overall algorithm model is a tool for the referential image segmentation method.

[0038] The beneficial effects of the present application are:

[0039] (1) We innovatively introduce scene graph knowledge and graph convolutional neural network to combine to mine complex context information in the text and the relationship between entity words, compared to ordinary text encoding, such a method has stronger representation ability, and can process deeper semantics, and deepen the understanding of the text.

[0040] (3) We use the attention reasoning structure to gather the features related to the segmentation object at the whole image level, diffuse and propagate the effective features, gradually reason to align the text features and image features, highlight the correct features on the whole image features, and suppress the invalid features, and more efficiently realize the interaction of cross-modal information. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is the overall structure diagram of the present application;

[0042] Figure 2 is the specific process of obtaining the visual feature of the text embedding by the visual fusion module in the present application;

[0043] Figure 3 is the specific process of extracting entity words and relationship words and constructing a scene graph by the text mining module in the present application;

[0044] Figure 4 is the specific process of aligning and enhancing the multi-modal features and updating the scene graph features by the attention reasoning module in the present application. Specific implementation method

[0045] The technical solutions of the present application will be further described below in combination with specific embodiments and drawings.

[0046] A referential image segmentation method based on text relationship mining and reasoning, the steps are as follows:

[0047] Step S1: feature preprocessing: according to the text and its corresponding image obtained from the data set, the text and visual encoder are sent to obtain the text feature and image feature, i.e. visual feature;

[0048] The step S1 is specifically:

[0049] Step S11: input the text in the data set into the text encoder, using the BERT model, thereby extracting the word-level text features wherein is the number of channels of the text features, and T is the number of words in the text.

[0050] Step S12: use the four-stage visual transformer layer to extract rich image features, i.e., visual features, from the image and the output of the previous stage, represented as , respectively represent the number of channels, the height and the width of the feature map in the first

[0051] Step S2: input the text features and the visual features into the visual fusion module to obtain multi-modal features, i.e., text-embedded visual features.

[0052] The step S2 is specifically:

[0053] Step S21: the visual features extracted from step S12 are combined with the text features via the visual fusion module to generate a set of multi-modal features, i.e., text-embedded visual features.

[0054] Step S22: each element in each text-embedded visual feature is weighted by a learnable gating unit, and then added element-wise to the visual features to generate a set of enhanced multi-modal features, represented as wherein respectively represent the number of channels, the height and the width of the feature map in the first

[0055] Step S3: input the text in the data set into the text parser to generate a scene graph in the form of a dictionary, and input it into the text encoder to obtain a node feature matrix in the scene graph and an entity relationship adjacency matrix.

[0056] The step S3 is specifically:

[0057] Step S31: input the text in the data set into the text parser to obtain a scene graph file in the form of a dictionary, which contains all entity words and the relationships between the entity words.

[0058] Step S32: parse the scene graph file in the form of a dictionary, and create an entity relationship adjacency matrix for each text according to the parsing result.

[0059] ​​Step S33: Extract the entity words in the scene graph in the form of a dictionary as node information, input into the text encoder, and obtain a node feature matrix containing entity word information.

[0060] Step S4: Add the enhanced multi-modal feature to the scene graph node feature matrix to form a scene graph node feature containing image information and an adjacency matrix, and input them into the graph convolutional neural network for updating.

[0061] The step S4 is specifically:

[0062] Step S41: According to the enhanced multi-modal feature of the fourth stage obtained in step S22 , as a global node feature, is added to the node feature matrix obtained in step S33, and a global entity is added to the adjacency matrix. The adjacency relationship between the global entity and each entity word in the dictionary scene graph is set to 1, indicating that the global node feature has a relationship with all entity nodes, and an enhanced scene graph is obtained.

[0063] Step S42: The node feature matrix and the relationship adjacency matrix of the enhanced scene graph obtained in step S41 are input into the graph convolutional neural network as inputs. The node features are updated by the enhanced multi-modal feature, the part corresponding to the visual information in the text information is enhanced, the relationship between the nodes is mined, and the features related to the object are highlighted. The features output by the GCN network are used as the updated scene graph features.

[0064] Step S5: Gradually fuse the multi-level enhanced multi-modal features and the updated scene graph features in the attention reasoning module, concatenate the fusion results, and obtain the features of the object to be segmented finally.

[0065] The step S5 is specifically:

[0066] Step S51: Use three attention reasoning modules to input the enhanced multi-modal feature obtained in step S22 as one of the inputs of the attention reasoning module, and the updated scene graph features obtained in step S42 are used as the other input.

[0067] Step S52: The two inputs are fused inside the attention reasoning module, and two attention weight distributions are calculated through softmax and sigmoid activation functions respectively. Two attention maps are obtained, which are the aggregation attention map and the propagation attention map . The calculation formulas of each value and in the aggregation attention map and the propagation attention map are as follows:

[0068] ,

[0069] where

[0070] ,

[0071] where

[0072] where h, w are the height and width of the feature, is the single feature vector of the enhanced multi-modal feature, is the single feature vector of the updated scene graph feature. , are the projection weight matrices respectively. and is the scalar product of the two projection vectors, indicating the correlation between the features of the two modalities;

[0073] Step S53: The aggregated attention map is taken as the weight to perform a weighted sum operation on the enhanced multi-modal feature to obtain an aggregated attention feature vector , and the formula is as follows:

[0074]

[0075] where h, w are the height and width of the feature, is the single feature vector of the enhanced multi-modal feature, is each value in the aggregated attention map .

[0076] Step S54: Based on the propagation attention map , the aggregated attention feature is converted into a feature matrix with the same dimension as the enhanced multi-modal feature , denoted by , where each feature vector in is calculated according to the following formula:

[0077]

[0078] combines and using element addition to obtain a new feature map through another convolution layer;

[0079] Step S55: The three feature maps obtained are concatenated step by step:

[0080]

[0081]

[0082]

[0083] where Gate denotes a gating operation, and concat[;] denotes concatenation. Finally, the segmentation feature of the desired referent object is obtained .

[0084] Step S6: training the attention reasoning module using the contrastive learning loss function, and the positive sample feature representation is calculated according to the following formula:

[0085]

[0086] where F denotes the feature representation of the anchor sample, f(.) denotes a feature mapping function, and ||.|| denotes the Euclidean distance, is a hyperparameter called margin, which is used to control the distance between the positive sample and the negative sample.

[0087] The contrastive loss is calculated according to the following formula:

[0088]

[0089]

[0090] where F denotes the feature representation of the anchor sample, f(.) denotes a feature mapping function, and ||.|| denotes the Euclidean distance, is a hyperparameter called margin, which is used to control the distance between the positive sample and the negative sample.

[0091] The cross-entropy loss function is used to train the overall model, and the exponential moving average is used to adaptively calculate the weight parameters for updating the two losses:

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] where is the exponential moving average value for each loss, is the exponential moving average value of the last time, ​is the current loss list, is a smoothing factor, respectively are the weights of two losses. Minimizing the final overall loss function makes the algorithm prediction results fit the correct results; the resulting overall algorithm model is the tool for the referential image segmentation method.

[0099] The above algorithm is the step of the referential image segmentation method based on text relationship mining and reasoning. Multiple sets of control tests are carried out for accuracy comparison under the same data set, which proves that the performance of the referential image segmentation method based on semantic relationship mining and reasoning proposed by the algorithm exceeds that of other benchmark algorithms, and the effectiveness of the algorithm is proved.

Claims

1. A method for referential image segmentation based on semantic relation mining and reasoning, characterized in that, The steps are as follows: Step S1: feature preprocessing: the text and its corresponding image obtained from the data set are sent into the text and visual encoder to obtain the text feature and image feature respectively; Step S11: input the text in the data set into the text encoder, using the BERT model, thereby extracting the word-level text features wherein is the number of channels of the text features, and T is the number of words in the text; Step S12: using a four-stage visual transformer layer to extract rich image features, i.e., visual features, from the image and the output of the previous stage, denoted as , respectively represent the number of channels, the height and the width of the feature map in the i-th stage. ​ Step S2: input the text feature and visual feature into the visual fusion module to obtain the multi-modal feature, i.e. the visual feature of the text embedding; Step S21: visual features extracted from step S12 combined via the vision fusion module with the text features to produce a set of multi-modal features, i.e. visual features of text embeddings; Step S22: Each element in each text embedding's visual feature is weighted by a learnable gating unit and then added element-wise to the visual feature to produce a set of enhanced multimodal features, denoted as where denote the number of channels, height and width of the feature map in the th stage, respectively Step S3: send the text in the data set into the text parser to generate a scene graph in the form of a dictionary, and send it into the text encoder to obtain the node feature matrix in the scene graph and the entity relationship adjacency matrix; Step S31: send the text in the data set into the text parser to obtain a scene graph file in the form of a dictionary, which contains all entity words and the relationships between the entity words; Step S32: parse the scene graph file in the form of a dictionary, and create an entity relationship adjacency matrix for each text according to the parsing result; Step S33: extract the entity words in the scene graph in the form of a dictionary as node information, and send them into the text encoder to obtain a node feature matrix containing entity word information; Step S4: add the enhanced multi-modal feature to the scene graph node feature matrix to form a scene graph node feature containing image information and an adjacency matrix, and input them into the graph convolutional neural network for updating; Step S41: obtaining the enhanced multi-modal features of the fourth stage according to the features obtained in step S22 As a global node feature, it is added to the node feature matrix obtained in step S33, and a global entity is added to the adjacency matrix. The adjacency relationship between the global entity and each entity word in the dictionary scene graph is set to 1, indicating that the global node feature has a relationship with all entity nodes, and an enhanced scene graph is obtained. Step S42: input the node feature matrix and relationship adjacency matrix in the enhanced scene graph obtained in step S41 into the graph convolutional neural network, update the node features by the enhanced multi-modal feature, mine the relationships between the nodes, highlight the features related to the referents, and output the features of the GCN network as the updated scene graph features; Step S5: gradually fuse the multi-level enhanced multi-modal feature and the updated scene graph feature in the attention reasoning module, cascade the fusion results, and obtain the feature of the object to be segmented finally; Step S51: using three attention reasoning modules, the enhanced multi-modal features obtained in step S22 are used to obtain three updated scene graph features, respectively. respectively as one of the inputs of the attention reasoning module, and the other input uses the updated scene graph features obtained by step S42; Step S52: The attention reasoning module internally uses two inputs to be fused, and two attention weight distributions are calculated through softmax and sigmoid activation functions respectively, and two attention maps are obtained, which are the aggregation attention map and the propagation attention map , respectively. Each value in the aggregation attention map and the propagation attention map is calculated according to the following formula:​ ； wherein ; ； wherein ; where h, w are the height and width of the feature, is the single feature vector of enhanced multi-modal features, is the single feature vector of updated scene graph features, , are projection weight matrices, respectively, and is the scalar product of two projection vectors, indicating the correlation between the features of the two modalities; Step S53: taking the attention aggregation map as a weight to enhance the multi-modal feature performing weighted summation operation to obtain the attention aggregation feature ; Step S54: propagating attention map , the aggregated attention feature is converted into an enhanced multi-modal feature with the same dimension, represented by ; using element addition to combine and , a new feature map is obtained through another convolution layer Step S55: Obtaining 3 feature maps By the step-by-step cascading operation, the segmentation features of the segmented object required are finally obtained ; Step S6: train the attention reasoning module by the contrastive learning loss function, and the positive sample feature representation is calculated as follows: ； wherein represents the multi-modal feature; the negative sample feature is calculated according to the following formula: ； Wherein, gt represents the true value, i.e. the correct segmentation feature of the segmented object given in the data set; The contrastive loss is calculated as follows: ； ； where F denotes the feature representation of anchor samples, f(.) denotes the feature mapping function, and ||.|| denotes the Euclidean distance, is a hyper-parameter called margin, which is used to control the distance between positive and negative samples, and the cross-entropy loss function The overall model is trained, and the weight parameters of the two losses are adaptively calculated using the exponential moving average to make the algorithm prediction results fit the correct results; the resulting overall algorithm model is the tool referred to as the image segmentation method.

Images