An indicative segmentation method using interleaved coding

Abstract

This invention belongs to the field of artificial intelligence technology and proposes an interleaved coding method for indicative segmentation. First, an image feature extraction network and a language feature extraction network are constructed separately. Then, a cross-modal feature fusion module is used to establish the connection between the two feature extraction networks, while a sampling method is introduced to suppress background noise. Next, image and language gating modules control the proportion of the fused features flowing into the original coding branches. Finally, the fused features obtained from interleaved coding are upsampled by a decoder to output the image segmentation result. This invention achieves a high-precision, lightweight image segmentation model for natural language indicators. The model fully utilizes multi-level language and image features, effectively locating the image target indicated by the language, and the segmentation process is fast and accurate.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an indicative segmentation method using interleaved coding. Background Technology

[0002] Natural language-indicated image segmentation is a task that, given an image and a linguistic description of the target object, segments the relevant pixel regions in the image based on the input natural language description. This task differs from semantic image segmentation, combining image segmentation with natural language, and is a relatively recent problem. It has brought significant value to various applications, such as language-based human-computer interaction and image editing. Compared to traditional unimodal visual segmentation tasks based on fixed category conditions, natural language-indicated image segmentation is more challenging because it needs to handle the richer vocabulary and sentence grammatical diversity of human natural language, and understand the relationship between linguistic expressions and their corresponding instances. The target object for segmentation is inferred from a free-form expression, which includes words and phrases representing concepts such as entities, actions, attributes, and locations, and these words and phrases are organized according to grammatical rules. The difficulty of this task lies in locating and segmenting the relevant visual target using the given linguistic conditions.

[0003] In convolutional neural networks, features from different layers can reflect details or global semantic information of an image. Therefore, deep and shallow features are complementary. However, existing natural language-indicated image segmentation methods, such as "GuangFeng, Zhiwei Hu, Lihe Zhang, and Huchuan Lu. Encoder fusion network with co-attention embedding for referring image segmentation[C]. In Computer Vision and Pattern Recognition, IEEE, 2021," typically use the same language features or interact with language features in the visual encoder, or as in "Ye L, Rochan M, Liu Z, et al. Cross-Modal Self-Attention Network for Referring Image Segmentation[C]. In Computer Vision and Pattern Recognition, Long Beach: IEEE, 2019.", interact with language features in the decoder. Interaction in the decoder does not fully utilize the pre-trained parameters of the encoder; interaction in the encoder ignores the fact that the language branch is as important as the visual branch, and it is not possible to update only the visual features without updating the language features. In addition, natural language instructions also contain rich global and local semantics. Both of these methods ignore the multi-layered information inherent in language itself, which may weaken the consistency of multi-granularity cross-modal alignment. Interleaved encoding, on the other hand, considers multi-granularity language context and, with the help of a visual-language interaction module, can effectively solve the above problems and achieve deep interleaving between modalities. Furthermore, previous methods simply insert fused features directly into the encoding network, ignoring the issue of the unequal information content between the visual and linguistic streams. By adding a gating structure to control the proportion of fused features flowing into the original encoding branch, the localization of feature fusion becomes more efficient. Summary of the Invention

[0004] The technical problem to be solved by this invention is to make up for the shortcomings of current image segmentation methods based on natural language indications, and to propose an indicative segmentation method based on interleaved coding. This method involves deep learning and computer vision, and is based on gated sampling of natural language indications through interleaved coding to improve the accuracy of image segmentation.

[0005] The technical solution of the present invention is an indicative segmentation method for interleaved coding, which specifically includes the following steps;

[0006] Step 1: Construct an image feature extraction network;

[0007] An image feature extraction network is established using ResNeSt-50 or ResNeSt-101 network structures as the backbone network to extract image features. The proportional features corresponding to the input image size are extracted from three layers of either the ResNeSt-50 or ResNeSt-101 network as the output image features.

[0008] Step 2: Construct a language feature extraction network;

[0009] A language feature extraction network is constructed using the 12-layer network structure of the BERT model as the backbone network to extract natural language features; multi-layer natural language features are extracted from each layer of the BERT network and output.

[0010] Step 3: Construct an image-to-language cross-modal feature fusion module;

[0011] The cross-modal feature fusion module is used to locate the image region indicated by natural language; firstly, the image features V... i and natural language features L j Channel dimensionality is reduced to a single dimension using 2D convolution, and this reduced dimension is used as input to the image-to-language cross-modal feature fusion module. Image features V i and natural language features L j The dimensions are C×H×W and C×T, respectively; where H, W, and C represent image features V. i The height, width, and number of channels, where T represents the natural language feature L. j The length of the sentence in the image; the image features V i Reorganize into a 2D matrix of size C×(HW), and calculate image features V. i and natural language features L j The image-to-language affinity matrix; where and These are learnable convolution parameters;

[0012] The process of obtaining the affinity matrix is ​​defined as follows:

[0013]

[0014] A′ was calculated vl ∈R (HW)×T Describes image features V i Each pixel and natural language feature L j The similarity between each word;

[0015] For A′ vlA sampling operation is performed along the H×W dimension, selecting the first K maximum values ​​along H×W, where K is the number of sampling points, and setting the remaining values ​​to zero. This is used to suppress interference from background pixels and reduce noise in the affinity matrix.

[0016] A vl "=Sampling)A vl ′)

[0017] The affinity matrix after sampling is normalized along the first dimension using the Softmax function; finally, A is used. vl Image features V i Projecting onto the language space, we obtain the fused language features L′ j The dimensions are C×T; the calculation process is defined as follows:

[0018] A vl =Softmax(A vl ")

[0019] L′ j =V i A vl

[0020] Step 4: Construct a language-to-image cross-modal feature fusion module; first, integrate natural language features L... j and image features V i The features are reduced to the same number of channels through convolution, and then input into the language-to-image cross-modal feature fusion module. The features from the two inputs are multiplied to obtain the language-to-image affinity matrix A. lv ';in, and These are learnable parameters; for the affinity matrix A lv A sampling operation is performed along dimension T, selecting the top K largest words along T, where K is the number of sampling points. The remaining values ​​are set to zero to highlight keywords. The operation process is defined as follows:

[0021]

[0022] A lv " = Sample(A lv ′)

[0023] A lv "∈R (HW)×T Finally, a softmax function is used to process the natural language features L... j Mapping to image space to obtain fused image features V i The operation process is defined as follows:

[0024] A lv =Softmax(A lv")

[0025] V i ′=L j+1 A lv

[0026] Step 5: Construct the image gating module

[0027] The image gating module controls the proportion of image features flowing into the image feature extraction network for fusion; it also controls the fusion of image features V. i The image features are added to the original image features by the image gating module, and then further increased in dimensionality through convolution before being fed into the next layer of the image feature extraction network. The specific operation process of the image gating module is designed as follows:

[0028] G1=P(S(Conv(Cat(V i V i ′))))

[0029] V i "=Norm(V i )+Norm(G1·V i ′)

[0030] Where Cat(·) represents concatenation along the channel dimension, Conv(·) represents a convolutional layer, S(·) represents an element-level sigmoid operation, P(·) represents a global average pooling operation, and Norm is used to represent L2 norm-based normalization.

[0031] Step 6: Construct the language gating module

[0032] The language gating module controls the proportion of language features flowing into the language feature extraction network for fusion; it also controls the fused language features L′. j After being added to the original language features by the language feature gating module, the feature is then increased in dimensionality through convolution before being fed into the next layer of the language feature extraction network. The specific computational process is designed as follows:

[0033] G2=P(S(Conv(Cat(L j ,L′ j ))))

[0034] L″ j =L j +LN(G2·L′ j )

[0035] Where LN(·) represents the LayerNorm operation;

[0036] Step 7: Construct the overall network structure

[0037] The input consists of an image and a short natural language sentence. The image is fed into an image feature extraction network to extract multi-scale image features. The short natural language sentence is fed into a language feature extraction network to extract multi-scale natural language features. The extracted image features and natural language features are then processed by a language-to-image cross-modal feature fusion module and an image-to-language cross-modal feature fusion module, respectively, to generate a fused image and fused language features. The fused image features are then fed back into the original image feature extraction network through an image feature gating module. The fused language features are then fed back into the original natural language feature extraction network through a language gating module. Finally, the fused image features from the deepest layer of the image feature extraction network are upsampled to output the segmentation result.

[0038] The three network layers of the ResNeSt-50 or ResNeSt-101 are selected as layer 2, layer 3, and layer 4, respectively; the corresponding proportions of the image size are 1 / 4, 1 / 8, and 1 / 16, respectively, and the number of channels corresponding to the output image features are 512, 1024, and 2048, respectively; the multiple network layers in the BERT network are layer 6, layer 9, and layer 12, and the number of channels corresponding to the natural language features is 768.

[0039] Training phase: ResNeSt-101 was used as the image feature extraction network, pre-trained on the ImageNet image classification dataset, and the pre-trained network weight parameters were saved; bert-base-uncased was used as the pre-trained model for the language feature extraction network; and training was performed on the UNC database, and finally validated on the UNC val, UNC testA, and UNC testB test sets respectively; the overall loss function of the network used the BCE algorithm; the network optimizer adopted the SGD algorithm, the batch size was set to 24, and the cosine annealing function was used to adjust the learning rate.

[0040] The beneficial effects of this invention are as follows: This invention fully utilizes the interleaved coding method, embedding an image-to-language cross-modal feature fusion module and a language-to-image cross-modal feature fusion module between the image feature extraction network and the language feature extraction network. Image-language interaction guides the progressive fusion of features at different levels between the two feature extraction networks. Through a shallow-to-deep approach, the mutual embedding of multimodal features is gradually completed, realizing the process of mutual instruction encoding between image and language. Simultaneously, a sampling mechanism is added to the fusion module to filter out background noise interference. Furthermore, the flow ratio of the fused feature stream is dynamically controlled by introducing image and language gating modules. This invention captures the multi-granular semantic context of natural language, achieving progressive fusion of multimodal features across multiple levels from multiple perspectives. Attached Figure Description

[0041] Figure 1This is a schematic diagram of an interleaved coding network.

[0042] Figure 2 This is a diagram showing the internal structure of the cross-modal feature fusion module.

[0043] Figure 3 This refers to the internal structure of the image gating module, where conv represents a single-layer convolutional structure.

[0044] Figure 4 This describes the internal structure of the language gating module. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0046] Figure 1 This is a schematic diagram of an interleaved coding network, which includes an image feature extraction network, a language feature extraction network, an image-to-language cross-modal feature fusion module, and a language-to-image cross-modal feature fusion module. Figure 1 The top row, Layers 1, 2, 3, and 4, represents the image feature extraction network; their different sizes indicate that the extracted feature map size gradually decreases. The bottom row, Bert Layer, represents the language feature extraction network. L2V represents the language-to-image cross-modal feature fusion module, and V2L represents the image-to-language cross-modal feature fusion module. The fused feature maps are then upsampled by the decoder to output the segmentation result.

[0047] Figure 2 This describes the specific structures of the image-to-language cross-modal feature fusion module and the language-to-image cross-modal feature fusion module, where V i L represents image features j Represents natural language features. G1 represents the image gating module, G2 represents the language gating module, and + represents pixel addition.

[0048] An indicative segmentation method using interleaved coding, comprising the following steps:

[0049] (1) Constructing an image feature extraction network

[0050] The image feature extraction network uses ResNeSt-50 or ResNeSt-101 network structures as the backbone to extract image features. It extracts features representing 1 / 4, 1 / 8, and 1 / 16 of the input image size from ResNeSt layers 2, 3, and 4, respectively, as the output image features. like Figure 1 As shown in the top row, its input is a three-channel RGB image V0, and the number of channels corresponding to the output image features are 512, 1024, and 2048 respectively;

[0051] (2) Constructing a language feature extraction network

[0052] The language feature extraction network uses a 12-layer BERT-based network structure as the backbone to extract natural language features, extracting multi-layer natural language features from layers 6, 9, and 12 of the BERT network, respectively. like Figure 1 The Bert Layer 6, Bert Layer 9, and Bert Layer 12 in the bottom row show that the language input is a one-hot encoded feature L0 of length 20 after padding, and the natural language feature output has 768 corresponding channels.

[0053] (3) Constructing an image-to-language cross-modal feature fusion module

[0054] Figure 2 The line below illustrates the process of image-to-language cross-modal feature fusion, first by fusing image features V... i and natural language features L j As input to the fusion module, its feature dimensions are C×H×W and C×T, respectively; where H, W, and C represent the height, width, and number of channels of the feature, respectively, and T represents the length of the sentence; then, the image features V i Reorganize into a 2D matrix of size C×(HW) and calculate image features V. i and natural language features L j The image-to-language affinity matrix; where and These are learnable convolution parameters; the computation process is defined as follows:

[0055]

[0056] A′ was calculated vl ∈R (HW)×T V was described i Each pixel and L j The similarity between each word; for A′ vl A sampling operation was performed along the H×W dimension, selecting the first K maximum values ​​along H×W (K is the number of sampling points), and setting the remaining values ​​to zero. Experiments showed that the segmentation accuracy was highest when the number of sampling points K was 7.

[0057] A vl " = Sample(A vl ′)

[0058] Next, the Softmax function is used to normalize the affinity matrix along the first dimension. Finally, A... vl Image features Vi Projected onto the language space, where the fused linguistic features L′ j The dimensions are C×T; the calculation process is defined as follows:

[0059] A vl =Softmax(A vl ")

[0060] L′ j =V i A vl

[0061] (4) Construct a language-to-image cross-modal feature fusion module

[0062] Similar to image-to-language, input L j and V i Features, such as Figure 2 As shown in the top row, multiplying the two input features yields the language-to-image affinity matrix A. lv ';in and These are learnable parameters; next, for A lv A sampling operation is performed along the T dimension, selecting the top K largest words along T (K is the number of sampling points), and setting the rest to zero. Experiments show that the segmentation accuracy is highest when the number of sampling points K is 7.

[0063]

[0064] A lv "=Sampling)A lv ′)

[0065] A lv "∈R (HW)×T Finally, a Softmax function is used to process the language features L. j Mapping to image space to obtain fused image features V′ i The operation process is defined as follows:

[0066] A lv =Softmax(A lv ")

[0067] V′ i =L j+1 A lv

[0068] (5) Constructing an image gating module

[0069] The image gating module is used to control the proportion of image features flowing into the image feature extraction network for fusion, such as... Figure 3 As shown; the fused image features V′ iAfter being added to the original image features by the image gating module, the image is then enlarged through convolution before being fed into the next layer of the image feature extraction network. The specific operation process of the image gating module is designed as follows:

[0070] G1=P(S(Conv(Cat(V i V i ′))))

[0071] V i "=Norm(V i )+Norm(G1·V i ′)

[0072] Where Cat(·) represents concatenation along the channel dimension, Conv(·) represents a convolutional layer, S(·) represents an element-level sigmoid operation, P(·) represents a global average pooling operation, and Norm is used to represent L2 norm-based normalization.

[0073] (6) Constructing a language gating module

[0074] The language gating module is used to control the proportion of language features flowing into the language feature extraction network for fusion, such as... Figure 4 As shown; the fused linguistic features L′ j After being added to the original language features by the language feature gating module, the feature is then increased in dimensionality through convolution before being fed into the next layer of the language feature extraction network. The specific computational process is designed as follows:

[0075] G2=P(S(Conv(Cat(L j ,L′ j ))))

[0076] L″ j =L j +LN(G2·L′ j )

[0077] Where LN(·) represents the LayerNorm operation;

[0078] Using ResNeSt-101 as the backbone network, it was first pre-trained on the ImageNet image classification dataset, and the pre-trained network weight parameters were saved. Then, experiments were conducted on two natural language instruction image segmentation databases, UNC and UNC+. The BCE algorithm was used as the loss function. The network optimizer adopted the SGD algorithm, with a batch size of 24. Cosine annealing was used to adjust the learning rate, with an initial learning rate of 0.0005, and 125,000 iterations were performed. The model accuracy improved by 6.1% compared to existing techniques.

Claims

1. An indicative partitioning method for interleaved coding, characterized in that, Specifically comprising the following steps: Step 1, constructing an image feature extraction network; The ResNeSt-50 or ResNeSt-101 network structure is used as the backbone network to establish the image feature extraction network for extracting image features; extracting corresponding scale features of an input image size from each of three layers of a ResNeSt-50 or ResNeSt-101 network as output image features Step 2, constructing a language feature extraction network; The 12-layer network structure of the Bert model is used as the backbone network to construct the language feature extraction network for extracting natural language features; extracting multi-layer natural language feature outputs from multi-layer network layers in a bert network, respectively Step 3, constructing an image-to-language cross-modal feature fusion module; The cross-modal feature fusion module is used for locating to the image region indicated by the natural language; first, the image feature V i and the natural language feature L j are reduced in dimension by 2D convolution to the same dimension, and are taken as the input of the image-to-language cross-modal feature fusion module, the image feature V i and the natural language feature L j have sizes of C×H×W and C×T respectively; wherein H, W and C respectively represent the height, width and channel number of the image feature V i , and T represents the length of the sentence in the natural language feature L j ; the image feature V i is reorganized into the form of a 2D matrix with the size of C×(HW), and the image-to-language affinity matrix of the image feature V i and the natural language feature L j is calculated; wherein and are learnable convolution parameters; The affinity matrix operation process is defined as follows: A is calculated ′ vl ∈R (HW)×T The similarity between each pixel of the image feature V i and each word of the natural language feature L j is described; A ′ vl Sampling operation is performed along the HxW dimension, the first K maximum values along the HxW are selected, K is the number of sampling points, and the remaining values are set to zero, which is used to suppress the interference of background pixels and reduce the noise of the affinity matrix. A vl " = Sampling(A vl ′ ) The Softmax function is used to normalize the affinity matrix along the first dimension after sampling; finally, A vl The image feature V i is projected into the language space to obtain the fused language feature L ′ j with the size of CxT; the operation process is defined as follows: A vl = Softmax(A vl ") L ′ j =V i A vl Step 4, constructing a language-to-image cross-modal feature fusion module; First, the natural language feature L j and the image feature V i are reduced in dimension through convolution to the same number of channels, input into the language-to-image cross-modal feature fusion module, and multiplied to obtain the language-to-image affinity matrix A lv ′ ; wherein, and are learnable parameters; the affinity matrix A lv ′ is sampled, sampling is performed along the T dimension, the first K largest words along T are selected, K is the number of sampling points, and the remaining values are set to zero, for highlighting key words; the operation process is defined as follows: A lv " = Sampling(A lv ′ ) A lv "∈R (HW)×T Finally, the natural language feature L j is mapped to the image space by a Softmax function to obtain the fused image feature V i ′ The operation process is defined as follows: A lv = Softmax(A lv ") V i ′ = LA lv Step 5, constructing an image gating module The image gating module is used to control the proportion of the fused image features flowing into the image feature extraction network; the fused image features V i ′ After the image gating module is added to the original image features, the dimension is finally increased through convolution, and the image features are sent into the next layer of the image feature extraction network; the specific operation process of the image gating module is designed as follows: G1 = P(S(Conv(Cat(V i , V i ′ )))) V i " = Norm(V i ) + Norm(G1 · V i ′ ) Wherein, Cat(·) represents concatenation along the channel dimension, Conv(·) represents a convolutional layer, S(·) represents an element-level Sigmoid operation, P(·) represents a global average pooling operation; Norm is used to represent L2 norm-based normalization; Step 6, constructing a language gating module The language gating module is used to control the proportion of the language features fused into the language feature extraction network; the fused language features L ′ j The language feature gating module is added to the original language features, and finally the convolution is performed to increase the dimension, and the language features are sent into the next layer of the language feature extraction network. The specific operation process is designed as follows: G2 = P(S(Conv(Cat(L j , L j ′ )))) L j "= L j + LN(G2· L ′ j ) Wherein, LN(·) represents a LayerNorm operation; Step 7, constructing an overall network structure An image and a natural language short sentence are input; the image is input into the image feature extraction network to extract multi-scale image features; the natural language short sentence is input into the language feature extraction network to extract multi-scale natural language features; the extracted image features and natural language features are respectively input into the language-to-image cross-modal feature fusion module and the image-to-language cross-modal feature fusion module to generate fused image features and fused language features; the fused image features are input into the original image feature extraction network through the image feature gating module; the fused language features are input into the original natural language feature extraction network through the language gating module; finally, the fused image features of the deepest layer of the image feature extraction network are output by upsampling to obtain a segmentation result.

2. The interleaved coded indicative partitioning method of claim 1, wherein, Three network layers of the ResNeSt-50 or ResNeSt-101 are respectively selected as layer2, layer3, and layer4; the corresponding proportions of the corresponding image sizes are 1 / 4, 1 / 8, and 1 / 16, respectively; and the multi-layer network layers in the Bert network are respectively layer6, layer9, and layer12.

Images