A language-guided cross-modal instance segmentation approach

Through the language-guided cross-modal instance segmentation method, the multi-scale feature fusion and cross-modal attention module are used, combined with the visual reasoning module, the problem of insufficient interaction between visual and language features in the existing technology is solved, and a more accurate entity is achieved Identification and distinction improve the effect of instance segmentation.

CN114119975BActive Publication Date: 2024-11-29CHINESE PEOPLE'S PUBLIC SECURITY UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111408303.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-25
Publication Date
2024-11-29
Estimated Expiration
2041-11-25

AI Technical Summary

Technical Problem

The existing instance segmentation method based on natural language description has shortcomings in the interaction between visual and language modal features, resulting in invalid cross-modal matching, difficulty in distinguishing multiple entities in the same category, and lack effective visual reasoning capabilities.

Method used

Using the language-guided cross-modal instance segmentation method, visual and language features are extracted through CNN and language encoder, multi-scale feature fusion module (MSFR) is introduced to enhance visual features, construct a cross-modal attention module (CAM) for interaction, and Multimodal feature representation is generated using the visual inference module (VR) to achieve cross-modal feature matching and entity distinction.

Benefits of technology

It improves the accuracy and distinction ability of instance segmentation, captures local and global details, adapts to match visual and language features, effectively implements instance segmentation, and enhances the discrimination of multimodal features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114119975B_ABST
    Figure CN114119975B_ABST
Patent Text Reader

Abstract

The present invention discloses a language-guided cross-modal instance segmentation method, which comprises the following steps: extracting image visual features and expression language features; using the MSFR module to capture global and local detail information of visual features and fuse them to enhance the representation ability of visual features; using the CAM module to align entity words of reference expressions with important image regions to highlight all entities in the image; and using the VR module to generate a multimodal feature representation for mask prediction. The present invention constructs a CMVR model to obtain a more discriminative feature representation for instance segmentation; it can capture local and global detailed information and pay more attention to details; it adaptively aligns information keywords in expressions with important regions in input images to promote matching between different modal features. At the same time, graph-based reasoning can be performed with the help of relational words, which can more effectively achieve instance segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an instance segmentation method, and in particular to a language-guided cross-modal instance segmentation method. Background Art

[0002] Instance segmentation based on natural language description is a challenging problem, which is academically called referring image segmentation. Instance segmentation based on natural language description is different from traditional computer vision semantic segmentation. Traditional semantic segmentation has no restrictions on target semantics and categories, while the objects of instance segmentation are specified by natural language. The challenge lies in the difference between visual and language modalities. The key to RIS is to learn effective multimodal features between visual and language modes to accurately identify the referred target. This task has a wide range of applications in human-computer interaction, scene perception, image retrieval, and localization of specific targets in video surveillance, and is one of the key focuses in the field of computer vision and pattern recognition.

[0003] Earlier instance segmentation methods used convolutional neural networks (CNN) and long short-term memory (LSTM) to extract visual and language unimodal features. Then, the concatenated convolution method was used to fuse visual and language features to generate segmentation masks. However, since these methods model language and visual information separately and lack sufficient interaction between the two methods, cross-modal matching between visual and language features is invalid. Later, an attention mechanism was gradually introduced to adaptively focus on informative keywords in language expressions and important areas in images. However, these methods do not fully utilize the context of words in expressions, so it is difficult to form different information when there are multiple entities in the same category. Recent methods mainly learn multimodal context representations by using different types of informative keywords in expressions. Although these methods have made great progress, none of them have effective visual reasoning capabilities to obtain recognizable multimodal features. Summary of the invention

[0004] In order to address the deficiencies of the above technologies, the present invention provides a language-guided cross-modal instance segmentation method.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a language-guided cross-modal instance segmentation method, comprising the following four steps:

[0006] Step 1: Extract visual features from the input image through the CNN encoder, and extract language features from the expression through the language encoder;

[0007] Step 2: Introduce the MSFR module to obtain multi-scale refined information from the visual features extracted by the CNN encoder, including global and local multi-scale detail information, fuse the information, and enhance the representation ability of visual features;

[0008] Step 3: Construct a CAM module to perform cross-modal interaction between the visual features obtained by the CNN encoder and the language features obtained by the language encoder, calculate cross-modal attention, and align the information keywords in the expression with the corresponding areas in the input image;

[0009] Step 4: Fuse the visual features enhanced by the MSFR module with the language features obtained by the language encoder, and input the fused features into the VR module to generate a multimodal feature representation for mask prediction.

[0010] Furthermore, in step 1, the specific process of visual feature extraction is:

[0011] 1.1 Input 3-channel image Where W and H are the width and height of the image respectively, and R represents the W×H×3 dimensional real space; visual features are extracted through the CNN backbone network Among them w, h, D v are the width, height and channel dimensions of the visual feature respectively; in order to save the spatial coordinate information, the visual feature x I Connected with the 8-dimensional space coordinate feature, that is The visual features are transformed through 1*1 convolution The size is converted to

[0012] Furthermore, in step 1, the specific process of language feature extraction is:

[0013] 1.2 Input Expression to the language encoder, where S l , L are the word and sentence lengths respectively, and l represents the index of the lth word; first, the word embedding of each word is initialized using the pre-trained word embedding model, as shown in Formula 1:

[0014] e l =embedding(s l ) Formula 1

[0015] Among them, embedding() represents the pre-trained word embedding model; then the word embedding is passed into the bidirectional LSTM network to encode the context of each word; the context of the word represents h l By connecting its forward and backward hidden state vectors, it is expressed as shown in Equation 2:

[0016]

[0017] el represents the word embedding representation of the lth word, Represents the forward and backward hidden states in the bidirectional LSTM network;

[0018] The language features of the entire expression are expressed as {h l} represents the dimension of word context; h l represents the hidden state of the lth word.

[0019] Furthermore, in step 2, a MSFR module is constructed to capture the detailed information of visual features x 1 ; This module consists of a Local PyConv branch and a Global PyConv branch; the Local PyConv branch is responsible for capturing local detailed information; the Global PyConv branch is used to capture global detail information; the output features of the Local PyConv branch and the Global PyConv branch are fused through 1*1 convolution to obtain a 1024-dimensional feature map containing local and global detail information.

[0020] Furthermore, the specific processing process of step 3 is:

[0021] 3.1 All words in the sentence are divided into four types, including entities, attributes, positional relationships, and unnecessary words, and a four-dimensional vector is calculated to represent the probability that it is one of the four types. The definition of the probability vector is shown in Formula 3:

[0022] q l =softmax(W 1 σ(W 0 h l +b 0 )+b 1 ) Formula 3

[0023] in, and represents a trainable parameter, σ represents a learnable parameter; Softmax( ) represents a normalized exponential function; D l , D h Respectively represent the word classification and the dimension of this feature; Represents word S l The probabilities of entities, attributes, relations, and non-essential words;

[0024] 3.2 Based on word features and entity and attribute words The probability of , calculates the cross-modal attention between each word and different image regions; the weighted normalized attention of the word is defined as shown in Equation 4 and Equation 5:

[0025]

[0026]

[0027] in, and is the transfer matrix, Dn represents the hyperparameter, D x , D h Represent the dimensions of visual features and word features respectively, α l represents cross-modal attention; β l is the weighted normalized attention, representing word s l Refers to the weighted probability of the relevant image area; aggregate the attention weighted features of all words and calculate the language feature C of the corresponding entity. The calculation process is as follows:

[0028]

[0029] h l represents the hidden state of the lth word.

[0030] Furthermore, the specific processing process of step 4 is as follows:

[0031] 4.1 The global language feature C and visual feature X of the entity are fused to obtain the multimodal feature Y; as shown in Formula 7:

[0032] Y=L2Norm((tanh(W p C)⊙tanh(W x X))) Formula 7

[0033] in, and represents a trainable parameter, ⊙ represents the multiplication of corresponding elements of the matrix, L2Norm(·) represents the L2 norm; Dn, Dm, Dv represent the dimensions of global language features, multimodal features, and visual features, respectively, and R represents the real number space of Dn×Dm or Dv×Dm dimensions;

[0034] 4.2 The size of the multimodal feature Y is transformed into C×W×H, where H, W, and C represent the height, width, and number of channels of Y, respectively, and the total number of nodes in the multimodal graph G is K=W×H; each position in the multimodal feature is represented as an image region, each image region is represented as a node of the multimodal graph G, and the multimodal feature Y is assigned as a feature of the node F; the weighted normalized context definition of the relational terms on the edge of the multimodal graph is shown in Formula 8:

[0035]

[0036] in, represents the trainable parameters, D e represents the hyperparameter, D l , D h Respectively represent the word classification and the dimension of this feature, δ represents the activation function;

[0037] The affinity matrix between the multimodal features of a node and the weighted relational context of a word is defined as shown in Equation 9:

[0038]

[0039] in, and represents trainable parameters; Feature vector representing positional relation words;

[0040] Then the normalized affinity matrix is ​​shown in Equation 10 and Equation 11:

[0041] A 1 =softmax(A F ) Formula 10

[0042] A 2 =softmax(A F T ) Formula 11

[0043] in, and Through the affinity matrix A F The row and column normalization of A F T .; The adjacency matrix is ​​expressed as shown in formula 12:

[0044] A=A 1 A 2 Formula 12

[0045] Among them, the element a of A i,j Represents the normalized size of the information flow between node i and node j in the multimodal graph G;

[0046] 4.3 Generate a gate for each node and edge of the multimodal graph to limit the information propagation between nodes; the gate g of the edge v It is defined as the sum of the weighted normalized attention values ​​of individual words in the expression, as shown in Equation 13:

[0047]

[0048] Among them, β l represents weighted normalized attention;

[0049] The gate g of the node e It is defined as the sum of all weighted probabilities of individual words in the expression, as shown in Equation 14:

[0050]

[0051] Among them, r l Represents the weighted normalized contextual definition of relational terms on the edge of a multimodal graph;

[0052] 4.4 Compare the adjacency matrix A with the edge gate g v Multiply the feature of node F and the gate g of the node e Multiply; then, the node features are updated through graph convolution operation, as shown in formula 15-17:

[0053]

[0054]

[0055]

[0056] in, b 1 、b 2 and are all trainable parameters, and represents the feature of bidirectional relationship, σ(·) represents the sigmoid activation function, represents the t-th inference multimodal feature, A T Represents the transposed matrix of matrix A; after multiple map convolution operations, it generates a more discriminative multimodal feature representation;

[0057] 4.5 Multimodal Features The multimodal feature map is upsampled by the deconvolution method through the ASPP decoder to predict the segmentation mask; then, the pixel-level cross entropy loss function is adopted, and the predicted segmentation mask and the Ground Truth segmentation mask are used to train the entire model end-to-end.

[0058] Furthermore, the end-to-end training is divided into two stages;

[0059] 1) Train at a low resolution scale without upsampling, and then train at a high resolution scale; the basic parameters are set as follows: word embedding size and hidden state size are both set to 1024, and the number of dynamic filters is set to 10;

[0060] 2) The Adam optimizer is used for training, with an initial learning rate of 1×10 and a batch size of 1. -5, the graph convolution is set to 3 layers, and the loss function adopts the cross entropy loss function.

[0061] The present invention discloses a language-guided cross-modal instance segmentation method, which constructs a cross-modal attention guided visual reasoning (CMVR) model to obtain more discriminative feature representation for instance segmentation. First, the model can capture local and global detailed information and pay more attention to details; second, the cross-modal attention module adaptively aligns the information keywords in the expression with the important areas in the input image to promote the matching between different modal features. At the same time, the constructed visual reasoning module can perform graph-based reasoning with the help of relational words to distinguish the referent from other objects of the same category, which can achieve instance segmentation more effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the process of the present invention.

[0063] Figure 2 This is the overall framework diagram of the cross-modal attention guided visual reasoning model of the present invention.

[0064] Figure 3 This is a schematic diagram of multi-scale feature optimization of the present invention.

[0065] Figure 4 Schematic diagram of visual reasoning segmentation of the present invention.

[0066] Figure 5 This is a schematic diagram of attention visualization of the present invention.

[0067] Figure 6 It is a schematic diagram of output prediction of the present invention under fixed image and changed language expression.

[0068] Figure 7 This is a schematic diagram of the language-guided cross-modal instance segmentation results of the present invention. DETAILED DESCRIPTION

[0069] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0070] The present invention proposes a language-guided cross-modal instance segmentation method, which constructs a cross-modal attention guided visual reasoning model (CMVR), which consists of three modules: first, a multi-scale feature refinement (MSFR) module is constructed to process small and large targets, effectively capture multi-scale local and global optimized visual characteristics, and fuse them to generate multi-scale features to enhance visual representation. Secondly, a cross-modal attention (CMA) module is introduced to adaptively align the information keywords in the expression with the important information in the image; the CMA module highlights all entities described by the language by calculating the weighted normalized attention value of the word; at the same time, the relationship context of the words in the expression is calculated as a relationship clue to associate with the relevant entities in the image. Finally, a visual reasoning (VR) module is proposed, which constructs a fully connected multimodal graph using multimodal features and expressed relational clues, and gradually performs visual reasoning on the graph to highlight the correct entities while suppressing other irrelevant entities; the VR module helps to generate more distinctive multimodal feature representations for reference segmentation. Therefore, the language-guided cross-modal instance segmentation method proposed in the present invention, based on the pre-trained Chinese word embedding matrix, makes full use of the semantic information of the target, combines the target visual features to form cross-modal attention, and realizes cross-modal instance segmentation based on the Chinese context.

[0071] like Figure 1 As shown in FIG. 1 , the overall process framework of the language-guided cross-modal instance segmentation method of the present invention includes the following four parts:

[0072] First, the CNN encoder extracts visual features from the input image, and the language encoder extracts language features from the expression;

[0073] Second, the MSFR module is introduced to obtain multi-scale refined information from the visual features extracted by the CNN encoder, including global and local multi-scale detail information, and fuse the information to enhance the representation ability of visual features;

[0074] Third, construct a CAM module to perform cross-modal interaction between the visual features obtained by the CNN encoder and the language features obtained by the language encoder, calculate cross-modal attention, and align the information keywords in the expression with the corresponding areas in the input image;

[0075] Fourth, the visual features enhanced by the MSFR module are fused with the language features obtained by the language encoder, and the fused features are input into the VR module to generate a multimodal feature representation for mask prediction, that is, the final output segmentation target mask map.

[0076] like Figure 2 As shown in the figure, it is the overall framework diagram of the cross-modal attention guided visual reasoning model proposed in the present invention. The model takes the image and the expression as input. First, the CNN encoder (specifically, the ResNet101 structure can be used) is used to extract the visual features (Feature Map) in the image, and the spatial location (Spatial Location) features are spliced ​​to obtain a new visual feature vector. Then, the pre-trained language encoder (word embedding model) is used to process the citation expression to obtain the word embedding vector (Word Embedding), and then the word vector is input into the LSTM network to obtain the language feature vector. The MSFR module takes the visual features (Feature Map) as input, and obtains multi-scale detail information in the extracted visual feature map, namely, global multi-scale features (Global Feature Maps) and local multi-scale features (Local Feature Maps), and splices and merges the global and local multi-scale features (Merge Local-Global) to achieve enhanced feature representation capabilities. The language features and the visual features extracted by the CNN encoder are used as the input of the CMA module to capture the inter-modal dependencies between keywords in the citation expression and different image regions. Moreover, the CMA module focuses on all entities mentioned in the expression, so it can help align entity words in the expression with important areas in the image. Subsequently, a fully connected multimodal graph (Multimodal Feature) is constructed on the spatial area of ​​the image; in the VR module, the multimodal information between nodes is propagated through the edge connections between nodes; using graph convolution, visual reasoning (VisualReasoning) is performed on the multimodal graph according to the relationship in the expression, effectively highlighting the relevant entities that match the language description and suppressing other irrelevant entities to generate a more discriminative multimodal feature representation, and the multimodal features are passed through the ASPP (Atrous Spatial Pyramid Pooling) decoder, and the multimodal feature graph is upsampled using the deconvolution method to predict the segmentation mask.

[0077] For CNN encoder and language encoder, the specific process of feature extraction is:

[0078] 1) Input 3-channel image Where W and H are the width and height of the image respectively, and R represents the W×H×3 dimensional real space; through the visual features in the CNN backbone network Among them w, h, D v are the width, height and channel dimensions of the visual feature respectively; in order to save the spatial coordinate information, the visual feature x I Connected with the 8-dimensional space coordinate feature, that is The visual features are transformed through 1*1 convolution The size is converted to

[0079] 2) Enter the expression to the language encoder, where s l , L are the word and sentence lengths respectively, and l represents the index of the lth word; first, the word embedding of each word is initialized using the pre-trained word embedding model, as shown in Formula 1:

[0080] e l =embedding(s l ) Formula 1

[0081] Where embedding( ) represents the pre-trained word embedding model; the word embedding is then passed into the bidirectional LSTM network to encode the context of each word; the context of the word is represented by h l By connecting its forward and backward hidden state vectors, it is expressed as shown in Equation 2:

[0082]

[0083] el represents the word embedding representation of the lth word, Represents the forward and backward hidden states in the bidirectional LSTM network;

[0084] The language features of the entire expression are expressed as {h l} represents the dimension of word context, h l represents the hidden state of the lth word.

[0085] like Figure 3 As shown in the figure, it is a schematic diagram of the multi-scale feature refinement (MSFR) module. The MSFR module captures the detailed information of the visual features x 1 ; This module consists of (Local PyConv, Local Pyramid Convolution branch) and (GlobalPyConv, Global Pyramid Convolution branch), and each layer of branches involves different types of kernel sizes. The Local PyConv branch is mainly responsible for capturing local detailed information; the Global PyConv branch is used to capture global detailed information; the specific steps are as follows:

[0086] 1) If Figure 3 As shown in the dotted part above, the Local PyConv branch is mainly responsible for processing entities of smaller size and capturing local detail information. It contains four convolutional layers, and the sizes of each convolution kernel are 9*9, 7*7, 5*5 and 3*3 respectively; the size of the convolution kernel increases from left to right, and the depth of the kernel decreases. Before the feature map is input to the Local PyConv branch, a 1*1 convolution is first applied to reduce the number of feature maps to 512; then pyramid convolutions with different kernels are performed from left to right to obtain local detailed information at four scales; finally, a 1*1 convolution is used to combine the information obtained at different scales. This branch also includes a normalization layer and a ReLU activation function after the four convolutional layers.

[0087] 2) If Figure 3 As shown in the dotted part below, the Global PyConv branch processes entities of larger size and captures global details; at the same time, it ensures that the kernel can cover the entire feature map. First, adaptive average pooling is used to reduce the spatial size of the input feature map to 9*9, and 1*1 convolution is used to reduce the number of feature maps to 512; then, cone convolutions with different kernels are performed in sequence; finally, 1*1 convolution is used to fuse information of different scales. In addition, before adaptive average pooling, bilinear interpolation is used to reconstruct the feature map to its original size by upsampling. The output feature maps from the Local PyConv branch and the Global PyConv branch are concatenated to obtain 1024 feature maps containing local and global details.

[0088] In the CAM module, the specific processing process is:

[0089] 3.1 All words in the sentence are divided into four types, including entity (Entry), attribute (Property), location relationship (Location) and unnecessary words (Unnecessary words), and a four-dimensional vector is calculated to represent the probability of it being one of the four types. The definition of the probability vector is shown in Formula 3:

[0090] q l =sof tmax(W 1 σ(W 0 h l +b 0 )+b 1 ) Formula 3

[0091] in, and represents a trainable parameter, σ represents a learnable parameter; Softmax( ) represents a normalized exponential function; D l, D h Respectively represent the word classification and the dimension of this feature; Represents word S l The probabilities of entities, attributes, relations, and non-essential words;

[0092] 3.2 Based on word features and entity and attribute words The probability of , calculates the cross-modal attention between each word and different image regions; the weighted normalized attention of the word is defined as shown in Equation 4 and Equation 5:

[0093]

[0094]

[0095] in, and is the transfer matrix, D n represents the hyperparameter, D x , D h Represent the dimensions of visual features and word features respectively, α l represents cross-modal attention; β l is the weighted normalized attention, representing word s l Refers to the weighted probability of the relevant image area; aggregate the attention weighted features of all words and calculate the language feature C of the corresponding entity. The calculation process is as follows:

[0096]

[0097] h l represents the hidden state of the lth word.

[0098] like Figure 4 As shown in Figure 1, it is a schematic diagram of the visual reasoning (VR) module, in which a fully connected multimodal graph is constructed on the spatial region of the image in order to effectively highlight the relevant entities that match the language description and suppress other irrelevant entities. The multimodal graph is represented by G = (V, E, F, A), where is a set of vertices containing multimodal information of an entity, v k represents the kth node, K represents the number of nodes; is a set of edges, e i,j It corresponds to the edge between neighbor nodes, i, j represents the adjacent nodes in the multimodal graph; is the collective property of vertices, Fi represents the multimodal feature in the i-th node; the multimodal information between nodes is propagated through the edge connections between nodes; using graph convolution, visual reasoning is performed on the multimodal graph based on the relationship in the expression. The specific steps are as follows:

[0099] 4.1 The global language feature C and visual feature X of the entity are fused to obtain the multimodal feature Y; as shown in Formula 7:

[0100] Y=L2Norm((tanh(W p C)⊙tanh(W X X))) Formula 7

[0101] in, and represents a trainable parameter, ⊙ represents the multiplication of corresponding elements of the matrix, L2Norm(·) represents the L2 norm; Dn, Dm, Dv represent the dimensions of global language features, multimodal features, and visual features, respectively, and R represents the real number space of Dn×Dm or Dv×Dm dimensions;

[0102] 4.2 The size of the multimodal feature Y is transformed into C×W×H, where H, W, and C represent the height, width, and number of channels of Y, respectively, and the total number of nodes in the multimodal graph G is K=W×H; each position in the multimodal feature is represented as an image region, each image region is represented as a node of the multimodal graph G, and the multimodal feature Y is assigned as a feature of the node F; the weighted normalized context definition of the relational terms on the edge of the multimodal graph is shown in Formula 8:

[0103]

[0104] in, represents the trainable parameters, D e represents the hyperparameter, D l , D h Respectively represent the word classification and the dimension of this feature, δ represents the activation function;

[0105] The affinity matrix between the multimodal features of a node and the weighted relational context of a word is defined as shown in Equation 9:

[0106]

[0107] in, and represents trainable parameters; Feature vector representing positional relation words;

[0108] Then the normalized affinity matrix is ​​shown in Equation 10 and Equation 11:

[0109] A 1 =softmax(A F ) Formula 10

[0110] A 2 =softmax(A F T ) Formula 11

[0111] in, and Through the affinity matrix A F The row and column normalization of A F T .; The adjacency matrix is ​​expressed as shown in formula 12:

[0112] A=A 1 A 2 Formula 12

[0113] Among them, the element a of A i,j Represents the normalized size of the information flow between node i and node j in the multimodal graph G;

[0114] 4.3 In a fully connected multimodal graph, the multimodal information of nodes is propagated through the adjacency matrix. However, the information in some nodes is independent of each other, and the unconstrained information propagation between these nodes may introduce a lot of noise and redundant information, resulting in confusing results; therefore, in the present invention, a gate is generated for each node and edge of the multimodal graph to restrict the information propagation between nodes; the gate g of the edge is v It is defined as the sum of the weighted normalized attention values ​​of individual words in the expression, as shown in Equation 13:

[0115]

[0116] Among them, β l represents weighted normalized attention;

[0117] The gate g of the node e It is defined as the sum of all weighted probabilities of individual words in the expression, as shown in Equation 14:

[0118]

[0119] Among them, r l Represents the weighted normalized contextual definition of relational terms on the edge of a multimodal graph;

[0120] 4.4 Compare the adjacency matrix A with the edge gate g v Multiply the feature of node F and the gate g of the node e Multiply; then, the node features are updated through graph convolution operation, as shown in formula 15-17:

[0121]

[0122]

[0123]

[0124] in, b 1 、b 2 and are all trainable parameters, and represents the feature of bidirectional relationship, σ(·) represents the sigmoid activation function, represents the t-th inference multimodal feature, A T Represents the transposed matrix of matrix A; after multiple map convolution operations, it generates a more discriminative multimodal feature representation;

[0125] 4.5 Multimodal features The multimodal feature map is upsampled by the deconvolution method through the ASPP (Atrous Spatial Pyramid Pooling) decoder to predict the segmentation mask; then, the pixel-level cross entropy loss function is used, and the predicted segmentation mask and the ground truth segmentation mask are used to train the entire model end-to-end;

[0126] The model is trained end-to-end in two stages. First, it is trained at a low resolution without upsampling, and then it is trained at a high resolution. The basic parameters are set as follows: the word embedding size and hidden state size are both set to 1024, and the number of dynamic filters is set to 10. The Adam optimizer is used for training, with an initial learning rate of 1×10 and a batch size of 1. The initial learning rate of the Adam optimizer is 1×10 -5 , the graph convolution is set to 3 layers, and the loss function adopts the cross entropy loss function.

[0127] like Figure 5 As shown, to better illustrate the advantages of cross-modal attention in the model, the attention heat map between the image and the expression is visualized. In the figure, (a) is the original image, (b) and (c) are the attention maps of only a single word and the complete expression in the test sample; (d) is the prediction result of the present invention; (e) is the attention map of the expression manually modified.

[0128] like Figure 6 As shown in the figure, in the case of a fixed image, by changing the expression to predict the segmentation result, such as "bottom center dark brown donut" refers to the third donut in the third row, rewriting the expression to "top row middle sprinkled donut" refers to the second donut in the first row, it can still accurately segment. As can be seen from the figure, the model can correctly segment all instances, which significantly highlights the flexibility and adaptability of the proposed model.

[0129] like Figure 7As shown, the language-guided cross-modal image segmentation results of the present invention are shown. In the figure, each column represents the original image, the ground-truth, the prediction result produced by the RNN-CNN as the baseline method, the prediction result produced by the baseline+CMA module, the prediction result produced by the baseline+MSRF module, and the prediction result produced by the present invention. Figure 7 The first row (f) shows that CMVR can segment objects that are severely occluded and have no localization words; Figure 7 The second row (f) shows that CMVR can distinguish objects in similar backgrounds. In contrast, the baseline model can correctly segment the instance referred to by the language, but confuses it with similar backgrounds. Figure 7 As can be seen from the third row of , the baseline model does not locate the orange in the picture (e.g., “little orange to the right of the big orange in the back”), while the present invention successfully distinguishes the referent in the complex scene under the guidance of attention. Figure 7 The first row (e) shows that CMVR can capture both global and local detailed information and accurately segment reference points, highlighting the potential of CMVR to perform relative position reasoning. Figure 7 The fourth row (f) in FIG. 4 shows the ability of CMVR to accurately identify the reference object, indicating that the present invention can effectively reduce the edge adhesion effect and highlight the details of the reference object.

[0130] The above implementation modes are not limitations of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the scope of the technical solution of the present invention also belong to the protection scope of the present invention.

Claims

1. A language-guided cross-modal instance segmentation method, Features: It includes the following four steps: Step 1: Extract visual features from the input image through the CNN encoder, and extract language features from the expression through the language encoder; Step 2: Introduce the multi-scale feature refinement MSFR module to obtain multi-scale refined information from the visual features extracted by the CNN encoder, including global and local multi-scale detail information, fuse the information, and enhance the representation ability of visual features; In step 2, a multi-scale feature refinement MSFR module is constructed to capture the detailed information of visual features x 1 ; This module consists of a Local PyConv branch and a Global PyConv branch; the Local PyConv branch is responsible for capturing local detailed information; the Global PyConv branch is used to capture global detailed information; the output features of the Local PyConv branch and the Global PyConv branch are fused through 1*1 convolution to obtain a 1024-dimensional feature map containing local and global detail information; Step 3: Construct a cross-modal attention CMA module to perform cross-modal interaction between the visual features obtained by the CNN encoder and the language features obtained by the language encoder, calculate cross-modal attention, and align the information keywords in the expression with the corresponding areas in the input image; include: 3.1 Classify all words in the sentence into four types, including entities, attributes, positional relationships, and unnecessary words, and calculate a four-dimensional vector to represent the probability that it is one of the four types, 3.2 Based on word features and the probabilities of entities and attribute words, calculate the cross-modal attention between each word and different image regions; aggregate the attention weighted features of all words to calculate the global language feature C of the corresponding entity; Step 4: Fuse the visual features enhanced by the multi-scale feature refinement (MSFR) module with the global language features (C) of the entity, and input the fused features into the visual reasoning (VR) module to generate a multimodal feature representation for mask prediction.

2. The language-guided cross-modal instance segmentation method according to claim 1, Features: In step 1, the specific process of visual feature extraction is: 1.1 Input 3-channel image Where W and H are the width and height of the image respectively, and R represents the W×H×3 dimensional real space; visual features are extracted through the CNN backbone network Among them w, h, D v are the width, height and channel dimensions of the visual feature respectively; in order to save the spatial coordinate information, the visual feature x I Connected with the 8-dimensional space coordinate feature, that is The visual features are transformed through 1*1 convolution The size is converted to 3. The language-guided cross-modal instance segmentation method according to claim 2, Features: In step 1, the specific process of language feature extraction is: 1.2 Input Expression to the language encoder, where S l , L are the word and sentence lengths respectively, and l represents the index of the lth word; first, the word embedding of each word is initialized using the pre-trained word embedding model, as shown in Formula 1: e l = embedding(s l ) Equation 1 Among them, embedding() represents the pre-trained word embedding model; then the word embedding is passed into the bidirectional LSTM network to encode the context of each word; the context of the word represents h l By connecting its forward and backward hidden state vectors, it is expressed as shown in Equation 2: el represents the word embedding representation of the lth word, Represents the forward and backward hidden states in the bidirectional LSTM network; The language features of the entire expression are expressed as {h l } represents the dimension of word context; h l represents the hidden state of the lth word.

4. The language-guided cross-modal instance segmentation method according to claim 3, Features: The specific processing process of step 3 is: In 3.1, the definition of the probability vector is shown in Equation 3: q l = softmax(W 1 σ(W 0 h l + b 0 ) + b 1 ) Equation 3 in, and represents a trainable parameter, σ represents a learnable parameter; Softmax() represents a normalized exponential function; D l , D h Respectively represent the word classification and the dimension of this feature; Represents word S l The probabilities of entities, attributes, relations, and non-essential words; In 3.2, word features are recorded as Entity and attribute words are denoted by The weighted normalized attention of words is defined as shown in Equation 4 and Equation 5: in, and is the transfer matrix, D n represents the hyperparameter, D x , D h Represent the dimensions of visual features and word features, α l represents cross-modal attention; β l is the weighted normalized attention, representing word s l refers to the weighted probability of the relevant image region; The calculation process of the entity's global language feature C is as follows: h l represents the hidden state of the lth word.

5. The language-guided cross-modal instance segmentation method according to claim 4, Features: The specific processing process of step 4 is as follows: 4.1 The global language feature C and visual feature X of the entity are fused to obtain the multimodal feature Y; as shown in Formula 7: Y = L2Norm((tanh(W p C) ⊙ tanh(W X X))) Equation 7 in, and represents a trainable parameter, ⊙ represents the multiplication of corresponding elements of the matrix, L2Norm(·) represents the L2 norm; Dn, Dm, Dv represent the dimensions of global language features, multimodal features, and visual features, respectively, and R represents the real number space of Dn×Dm or Dv×Dm dimensions; 4.2 The size of the multimodal feature Y is transformed into C×W×H, where H, W, and C represent the height, width, and number of channels of Y, respectively, and the total number of nodes in the multimodal graph G is K=W×H; each position in the multimodal feature is represented as an image region, each image region is represented as a node of the multimodal graph G, and the multimodal feature Y is assigned as a feature of the node F; the weighted normalized context definition of the relational terms on the edge of the multimodal graph is shown in Formula 8: in, represents the trainable parameters, D e represents the hyperparameter, D l , D h Respectively represent the word classification and the dimension of this feature, δ represents the activation function; The affinity matrix between the multimodal features of a node and the weighted relational context of a word is defined as shown in Equation 9: in, and represents trainable parameters; Feature vector representing positional relation words; Then the normalized affinity matrix is ​​shown in Equation 10 and Equation 11: A 1 = softmax(A F ) Equation 10 A 2 =softmax(AF T ) formula 11 in, and Through the affinity matrix A F The row and column normalization of A F T ; The adjacency matrix is ​​expressed as shown in formula 12: A=A 1 A 2 Formula 12 Among them, the element a of A i,j Represents the normalized size of the information flow between node i and node j in the multimodal graph G; 4.3 Generate a gate for each node and edge of the multimodal graph to limit the information propagation between nodes; the gate g of the edge v It is defined as the sum of the weighted normalized attention values ​​of individual words in the expression, as shown in Equation 13: Among them, β l represents weighted normalized attention; The gate g of the node e It is defined as the sum of all weighted probabilities of individual words in the expression, as shown in Equation 14: Among them, r l Represents the weighted normalized contextual definition of relational terms on the edge of a multimodal graph; 4.4 Compare the adjacency matrix A with the edge gate g v Multiply the feature of node F and the gate g of the node e Multiply; then, the node features are updated through graph convolution operation, as shown in formula 15-17: in, b 1 、b 2 and are all trainable parameters, and represents the feature of bidirectional relationship, σ(·) represents the sigmoid activation function, represents the t-th inference multimodal feature, A T Represents the transposed matrix of matrix A; after multiple map convolution operations, it generates a more discriminative multimodal feature representation; 4.5 Multimodal Features The multimodal feature map is upsampled by the deconvolution method through the ASPP decoder to predict the segmentation mask; then, the pixel-level cross entropy loss function is adopted, and the predicted segmentation mask and the Ground Truth segmentation mask are used to train the entire model end-to-end.

6. The language-guided cross-modal instance segmentation method according to claim 5, Features: End-to-end training is divided into two stages; 1) Train at a low resolution scale without upsampling, and then train at a high resolution scale; the basic parameters are set as follows: word embedding size and hidden state size are both set to 1024, and the number of dynamic filters is set to 10; 2) The Adam optimizer is used for training, with an initial learning rate of 1×10 and a batch size of 1. -5 , the graph convolution is set to 3 layers, and the loss function adopts the cross entropy loss function.

Citation Information

Patent Citations

  • Stereoscopic image uniflow visual saliency detection method based on joint information coding

    CN112257509A

  • RGBD saliency object detection method based on 3D convolutional neural network

    CN112749712A