A method and device for image object segmentation

Abstract

The present invention relates to the field of computer vision, and in particular to a method and device for image target segmentation. It mainly includes: inputting an input image to be segmented into an encoder, using the encoder to extract input image features to obtain a first-order feature map; wherein each layer of the encoder network contains a hybrid Transformer module composed of a convolution structure and a Transformer structure; inputting the first-order feature map into a multi-layer attention module to obtain a second-order feature map of multi-scale features; inputting the second-order feature map into a decoder of a pure convolution structure, restoring the second-order feature map to a third-order feature map consistent with the size of the input image, and using the third-order feature map as a prediction result of image target segmentation. The present invention can accurately segment image targets of different scales from the image to be segmented when the training sample set is small.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and device for image target segmentation. Background Art

[0002] The rise of artificial intelligence has brought new breakthroughs in the field of computer vision. With the development of deep learning and neural networks, the attention mechanism has also become a technology widely used in various fields. As an important part of the human visual system, the basic idea of ​​the attention mechanism is to re-weight the features by calculating the attention weights, thereby strengthening the effective features and suppressing the invalid features. In addition to the common spatial attention and channel attention mechanisms, there is also a special self-attention mechanism, which calculates the response of each position as the weighted sum of all position features. Since the weight calculation can be calculated in parallel in matrix form, it can efficiently capture long-range contextual interactions.

[0003] Thanks to this, the Transformer model based on the self-attention mechanism has a powerful ability to describe long-range dependencies and capture global features, and is widely used in natural language processing, object detection, object segmentation and other fields. However, to achieve this global information interaction capability, a corresponding price must be paid. The accuracy of existing Transformer-based object segmentation models depends on a large number of parameters and large-scale data sets, with high computational costs and lack of good robustness and generalization, and poor performance on small sample data sets. In addition, a design structure that focuses too much on global features will also lead to the neglect of local features and multi-scale features, making it difficult to accurately segment smaller targets.

[0004] In view of this, how to overcome the defects of the existing technology and solve the problem that the existing Transformer model is not easy to accurately segment smaller targets is a problem to be solved in this technical field. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention solves the problem that the existing Transformer model is not easy to accurately segment smaller objects.

[0006] The embodiment of the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for image target segmentation, specifically: inputting an input image to be segmented into an encoder, and using the encoder to extract input image features to obtain a first-order feature map; wherein each layer of the encoder network contains a hybrid Transformer module composed of a convolutional structure and a Transformer structure; inputting the first-order feature map into a multi-layer attention module to obtain a second-order feature map of multi-scale features; inputting the second-order feature map into a decoder of a pure convolutional structure, restoring the second-order feature map to a third-order feature map consistent with the size of the input image, and using the third-order feature map as a prediction result of image target segmentation.

[0008] Preferably, in each hybrid Transformer module, the use of an encoder to extract input image features to obtain a first-order feature map specifically includes: using multiple convolution kernels of different scales to extract local spatial features in the input image, performing cross-scale embedding and image sequence conversion on the local spatial features to obtain an embedding vector with a specified number of channels; using two consecutive Transformer layers to extract global spatial features of the embedding vector, and converting the extracted global spatial features into a corresponding first-order feature map.

[0009] Preferably, the method of using two consecutive Transformer layers to extract the global spatial features of the embedded vector specifically includes: according to the number of layers of the current hybrid Transformer module, setting a self-attention window with a specified number of attention heads in the Transformer layer, inputting the embedded vector into the self-attention window to obtain a global attention vector containing multi-head features; expanding the global attention vector and obtaining a linear projection of the global attention vector, and learning the nonlinear features in the linear projection through an activation function to obtain the global spatial features.

[0010] Preferably, the first-order feature map is input into the multi-layer attention module to obtain the second-order feature map of the multi-scale feature, specifically including: obtaining the first-order feature map output by the hybrid Transformer module of the current layer as the feature map of the current layer, obtaining the first-order feature map output by the hybrid Transformer module of the previous layer as the feature map of the previous layer, and obtaining the first-order feature map output by the hybrid Transformer module of the next layer as the feature map of the next layer; converting the feature map of the current layer into a first vector of a first length through a global pooling layer, the first vector generates a second vector of a second length through a first fully connected layer, and the second vector generates a third vector and a fourth vector of a first length through a second fully connected layer; normalizing the third vector and the fourth vector to generate a first weight and a second weight of the corresponding channel, and the sum of the first weight and the second weight is 1; calculating the first product of the first weight and the feature map of the previous layer, and calculating the second product of the second weight and the feature map of the next layer, adding the first product, the second product and the feature map of the current layer element by element, and using the addition result as the second-order feature map.

[0011] Preferably, the calculation of the first product of the first weight and the previous layer feature map, and the calculation of the second product of the second weight and the next layer feature map, specifically include: using a downsampling function to downsample the previous layer feature map, multiplying the first weight by the downsampling result, to obtain the first product; using an upsampling function to upsample the next layer feature map, multiplying the second weight by the upsampling result, to obtain the second product; wherein the upsampling function and the downsampling function use convolution and transposed convolution with a step size of 2, and a linear rectification layer, so as to obtain the nonlinear characteristics of the previous layer feature map and the next layer feature map.

[0012] Preferably, the element-by-element addition of the first product, the second product and the current layer feature map specifically includes: using the first parameter as a weighting parameter of the sum of the first product and the second product, using the second parameter as a weighting parameter of the current layer feature map, and calculating the corresponding weighted addition result; wherein the first parameter and the second parameter are adaptive trainable parameters with a length of 2.

[0013] Preferably, the method of restoring the second-order feature map to a third-order feature map having the same size as the input image specifically includes: the decoder receives the second-order feature map, splices the second-order feature map with the upsampling result of the deep feature map of the input image in the channel dimension, and restores the splicing result to the same dimension through convolution to obtain the third-order feature map.

[0014] Preferably, the concatenation result is restored to the same dimension by convolution, specifically including: upsampling the second-order feature map to obtain a sampling result consistent with the input image size, and then using 1×1 convolution and normalization function to map the probability value of each pixel of the sampling result belonging to different categories to different channels to obtain a third-order feature map of the same dimension as the input image.

[0015] Preferably, the method also includes: using a cross entropy loss function to calculate the loss of the neural network model, updating the parameters of the neural network model according to the calculated loss, using the updated parameters to train and optimize the neural network model, and using the optimized neural network model to perform image target segmentation on the input image.

[0016] On the other hand, the present invention provides a device for image target segmentation, specifically: including at least one processor and a memory, the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by at least one processor, and after the instructions are executed by the processor, they are used to complete the method for image target segmentation in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are: effectively combining the convolution kernel and the Transformer model in the network model, and embedding a multi-scale attention module, comprehensively capturing global information, local information and multi-scale information, improving the model's feature extraction capability and robustness, reducing the model's complexity, and being able to accurately segment image targets of different scales from the image to be segmented when the training sample set is small. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of a neural network model used in an embodiment of the present invention;

[0020] Figure 2 A flow chart of a method for image object segmentation provided by an embodiment of the present invention;

[0021] Figure 3 A flow chart of another method for image object segmentation provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of the structure of a hybrid Transformer module used in an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of the image size that needs to be processed by the convolution kernel used in the embodiment of the present invention;

[0024] Figure 6 A schematic diagram of the structure of the Transformer layer used in an embodiment of the present invention;

[0025] Figure 7 A flow chart of another method for image object segmentation provided by an embodiment of the present invention;

[0026] Figure 8 A schematic diagram of the structure of a multi-layer attention module used in an embodiment of the present invention;

[0027] Fig. 9 A flow chart of another method for image object segmentation provided by an embodiment of the present invention;

[0028] Fig.10 A schematic diagram of the structure of an image object segmentation device provided by an embodiment of the present invention;

[0029] The reference numerals are as follows:

[0030] 11: processor; 12: memory. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] The present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0033] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0034] Embodiment 1:

[0035] Compared with the Convolutional Neural Network (CNN) structure, the Transformer model has a stronger ability to describe long-range dependencies and can better extract global features. However, it requires a large amount of data sets and computation, and performs poorly on small data sets. In contrast, the CNN structure is relatively simple and has good local feature extraction capabilities. Therefore, the computational cost of the model can be reduced and the generalization of the model can be improved by effectively combining the CNN structure, and the segmentation accuracy and segmentation performance on small sample data sets can be improved by combining multi-scale features.

[0036] In specific implementation, the following can be used: Figure 1 The neural network model shown includes an encoder, a multi-layer attention module and a decoder. In the method provided in this embodiment, a multi-layer hybrid Transformer module is first used as an encoder, and each layer of the hybrid Transformer module includes a convolution structure and a Transformer structure to effectively extract global and local features of the image; a multi-layer attention module is used to connect the encoder and the decoder, which enhances the multi-scale feature extraction capability of the neural network model and further improves the processing capability for small targets and small data sets; finally, a decoder with a pure CNN structure is used to generate the segmentation result.

[0037] like Figure 2 As shown, the specific steps of the method for image object segmentation provided by the embodiment of the present invention are as follows:

[0038] Step 101: Input an input image to be segmented into an encoder, and use the encoder to extract features in the input image to obtain a first-order feature map.

[0039] In the method provided in this embodiment, a hybrid Transformer module combining a convolutional structure and a Transformer structure is used as an encoder, and each layer of the encoder network contains a hybrid Transformer module composed of a convolutional structure and a Transformer structure, wherein a plurality of convolutional layers constitute a CNN structure, a plurality of Transformer layers constitute a Transformer structure, and convolutional layers and Transformer layers are used alternately. The hybrid Transformer module of this embodiment is equivalent to an effective combination of a CNN encoder that only uses convolutional kernels and a Transformer encoder that only uses multi-head attention, which can improve the global feature and local feature extraction capabilities, the robustness of the model, generalization, and adaptability to small data sets. On the other hand, the input and output of the Transformer structure are one-dimensional sequences, while the input and output of the CNN structure are two-dimensional feature maps. In order to combine the two, it is necessary to complete the conversion between the feature map and the one-dimensional sequence through a conversion layer.

[0040] Step 102: Input the first-order feature map into a multi-layer attention module to obtain a second-order feature map of multi-scale features.

[0041] In this embodiment, the encoder can extract feature maps of multiple scales. In order to achieve information interaction between neural network layers of different depths, it is also necessary to use a multi-layer attention module to connect the encoder and decoder to extract the second-order feature map of multi-scale features. By embedding the cross-scale multi-layer attention module in the hybrid Transformer module, the perception ability of multi-scale targets and small-scale targets can be improved, and the segmentation ability of small sample data sets can be further improved.

[0042] Step 103: Input the second-order feature map into a decoder of a pure convolutional structure, restore the second-order feature map to a third-order feature map having the same size as the input image, and use the third-order feature map as a prediction result of image target segmentation.

[0043] After obtaining the second-order feature map of multi-scale features, the second-order feature map can be decoded by the decoder to generate a prediction result. In this embodiment, a decoder with a pure CNN structure containing only convolution kernels is used to receive the second-order feature map, and up-sampled through the convolution layer to restore the second-order feature map to the input image size, thereby obtaining a third-order feature map consistent with the input image size. The third-order feature map can be used as a prediction result for image target segmentation. The encoder composed of the Transformer structure and the CNN structure and the decoder of the pure CNN structure avoid information redundancy and reduce model parameters and computational costs.

[0044] After steps 101 to 103 provided in this embodiment, the input image can be segmented into multi-scale image objects, and accurate prediction results can be obtained when the object scale is uncertain.

[0045] In the encoder used in this embodiment, in the hybrid Transformer module, each convolutional layer in the CNN structure and each Transformer layer in the Transformer structure are connected together in a specified order.

[0046] like Figure 3 As shown, in each hybrid Transformer module, the encoder can be used to extract the first-order feature map in the image to be segmented in the following way.

[0047] Step 201: Use multiple convolution kernels of different scales to extract local spatial features in the input image, perform cross-scale embedding and image sequence conversion on the local spatial features to obtain an embedding vector with a specified number of channels.

[0048] When performing feature extraction, we first use CNN's good local feature extraction capabilities to extract local spatial features in the input image. In actual implementation, multiple convolution kernels can be used to extract local spatial features of different scales, and the extracted local spatial features are input to subsequent layers in the form of sequence tokens for processing.

[0049] Step 202: Use two consecutive Transformer layers to extract global spatial features of the embedded vector, and convert the extracted global spatial features into corresponding first-order feature maps.

[0050] In order to better extract global spatial features, it is also necessary to use the Transformer layer to extract global spatial features from the embedded vector.

[0051] After step 201 and step 202 provided in this embodiment, the extraction of spatial features of various scales can be completed.

[0052] In the specific implementation, the structure of the hybrid Transformer module in each layer of the network is as follows: Figure 4 As shown, it includes the following layers, and each layer is connected in sequence.

[0053] (1) Convolutional layer (abbreviated as Conv), used to execute step 201 and extract local spatial features.

[0054] (2) Cross-Scale Embedding (CEL) is a layer that embeds features at multiple scales.

[0055] (4) Image to Sequence (Img2Seq) is a layer that uses a matrix transformation (reshape) operation to convert the two-dimensional first-order feature map into an embedding vector in the form of a one-dimensional sequence.

[0056] (3) Transformer layer: In this embodiment, two consecutive Transformer layers are used to execute step 202 to extract global spatial features.

[0057] (4) Sequence to Image (Seq2Img) is the inverse operation of the image sequence conversion layer, which converts the embedding vector in the form of a one-dimensional sequence into a second-order feature map.

[0058] In this embodiment, a cross-scale embedding layer (CEL) is used to perform patch embedding and patch merging on the image to enhance context information extraction, enhance the encoder's inter-layer multi-scale feature perception ability, and compensate for information loss in the downsampling process; the global and local features of the image are effectively extracted through alternating convolutional layers and Transformer layers, which improves the robustness of the model and its adaptability to small data sets. In a specific implementation, in each convolutional layer, convolution kernels of different sizes can be used to convolve the input image, and the convolution results are obtained and directly spliced ​​together as output. Through the convolution of different convolution kernels, information on features of different scales can be obtained, which is relatively effective for objects of unknown scales.

[0059] In a specific scenario, the CEL in the first layer of the encoder hybrid Transformer module is called block embedding. First, initialize multiple convolution kernels of different sizes and the same step length. In this scenario, Figure 5 As shown in the figure, each dotted box indicates the image size that needs to be processed by a convolution kernel. Among them, the convolution kernel step size is 4, and the convolution kernel sizes from inside to outside are (4×4, 8×8, 16×16, 32×32), and their number corresponds to the number of output channels, which are C / 2, C / 4, C / 8, and C / 8 from inside to outside, where C is the number of channels. Then the input image is convolved to obtain the result after convolution, and the convolution results are spliced ​​together. In this scenario, the size of the input image is 224x224. Through block embedding, a first-order feature map of size 56x56 can be obtained, which is then expanded into an embedding vector in the form of a one-dimensional sequence.

[0060] In this scenario, the CEL in the hybrid Transformer module of the remaining layers of the encoder is called block merging. It uses two convolution kernels with a stride of 2, with sizes of (4×4, 8×8), which is basically the same as the convolution operation above, except that the number of channels corresponding to the final output is C / 2 and C / 2, respectively. In this scenario, taking the 56x56 first-order feature map obtained after block embedding as an example, a 28×28 feature map can be obtained after block merging. After the block embedding process, the input image is transformed into a high-level, low-resolution second-order feature map. Only a smaller convolution kernel is needed to obtain rich context information, while also reducing the amount of calculation.

[0061] Through the above-mentioned cross-scale embedding mechanism of the two different convolution kernels, the network can learn rich feature information and reduce the information loss caused by the network during the downsampling process.

[0062] Further, such as Figure 6 As shown in the figure, the Transformer layer contains the following layers in sequence: Multi-Head Attention (MHA), the first addition and normalization (Layer Normalization, Layer Norm) layer, the feed-forward network (FFN) layer and the second addition and normalization layer.

[0063] like Figure 7 As shown, two consecutive Transformer layers can be used to extract the global spatial features of the embedding vector. The execution process of each Transformer layer is as follows:

[0064] Step 301: According to the number of layers of the current hybrid Transformer module model, a self-attention window with a specified number of attention heads is set in the Transformer layer, and the embedding vector is input into the self-attention window to obtain a global attention vector containing multi-head features.

[0065] In a specific scenario, the size of the self-attention window in the multi-head attention layer can be set to 16×16. Take the example that the number of Transformer layers in the hybrid Transformer module of each layer of the network is 2, and the encoder includes a 4-layer hybrid Transformer module. In the 4-layer hybrid Transformer module, the number of attention heads needs to be set to 2, 4, 8 and 16 respectively. The multi-head attention layer assigns self-attention weights to the input embedding vector through the global attention core mechanism, and linear projections V, K and Q are generated in the process. The multi-head attention layer can repeat the process of generating linear projections V, K and Q several times in different linear spaces and take the average to obtain a global attention vector with multi-head features, thereby enhancing the feature extraction capability. Its calculation function is as shown in formula (1):

[0066]

[0067] In formula (1), Softmax is a normalization function; Q(x), K(x), and V(x) represent the query matrix, key matrix, and value matrix, respectively, which are obtained by inputting local spatial features (sequence tokens) into three linear projection layers; d is the dimension of the sequence token. As can be seen from the formula, the multi-head attention layer can transform the embedding vector into a global attention vector with multi-head features.

[0068] Step 302: Expand the global attention vector and obtain a linear projection of the global attention vector, and learn the nonlinear features in the expanded linear projection through an activation function to obtain the global spatial features of the embedding vector.

[0069] After obtaining the linear projection and completing the normalization, the global attention vector needs to be expanded using the feedforward network layer. The feedforward network layer includes a linear projection layer Linear and a nonlinear activation layer (Rectified Linear Unit, abbreviated as ReLU). The linear projection of the global attention vector is obtained by the linear projection layer Linear, and the linear projection is rectified to obtain the global spatial features. The calculation function of the feedforward network layer is shown in formula (2).

[0070] FFN(x)=Linear(ReLU(x)) (2)

[0071] In formula (2), Linear is the linear function of the linear projection layer, ReLU(x) is the linear rectification function of the nonlinear activation layer, and the feedforward network layer learns nonlinear features through the ReLU activation function. In a real scenario, the expansion ratio of the first linear projection layer in FFN is set to 4. When projecting, it is first expanded to 1:4 and then restored to 4:1, which can enhance the semantic expression ability and obtain more complex feature information.

[0072] After step 301 and step 302 provided in this embodiment, the extraction of global spatial features can be completed through two Transformer layers.

[0073] After the above processing, the output of the hybrid Transformer module is expressed as follows:

[0074] F′=CEL(Conv(F)) (3)

[0075] X=Img2Seq(F′) (4)

[0076] X′=MHA(LayerNorm(X))+X (5)

[0077] Y=Seq2Img(FFN(LayerNorm(X′)))+X′ (6)

[0078] Among them, F is the input of the hybrid Transformer module, that is, the input image; F′ is the first-order feature map obtained using the cross-scale embedding mechanism; X is the one-dimensional vector sequence obtained by flattening the first-order feature map; X′ is the output result of the multi-layer attention module; Y is the output result of the hybrid Transformer module.

[0079] The jump connection between the encoder and the decoder can improve the network segmentation performance, and it can also be improved using the attention mechanism or multi-scale method. The method provided in this embodiment is different from the common multi-scale fusion method in that feature maps of different dimensions are taken into account at the time of input. Figure 8 As shown, the structure of the multi-layer attention module used in this embodiment.

[0080] In this embodiment, the encoder includes multiple layers of hybrid Transformer modules, and each layer of hybrid Transformer modules outputs first-order feature maps of different scales. In order to combine the first-order feature maps of different scales, first, the first-order feature map output by the hybrid Transformer module of the current layer is obtained as the feature map of the current layer, the first-order feature map output by the hybrid Transformer module of the previous layer is obtained as the feature map of the previous layer, and the first-order feature map output by the hybrid Transformer module of the next layer is obtained as the feature map of the next layer. Then, as Fig. 9 As shown in Figure 1, the first-order feature map is input into a multi-layer attention module to obtain a second-order feature map of multi-scale features.

[0081] Step 401: The feature map of the current layer is converted into a first vector of a first length through a global pooling layer, the first vector generates a second vector of a second length through a first fully connected layer, and the second vector generates a third vector and a fourth vector of the first length through a second fully connected layer.

[0082] The feature map of the current layer in the encoder is passed through the global pooling layer to obtain a vector of length c, and then passed through the first fully connected layer (Fully Connected Layers, abbreviated as FC), that is, Figure 8 FC1 in generates a vector d of length n. The FC1 layer can reduce the dimension, reduce the complexity and computational effort to achieve better generalization, and enhance the ability to learn nonlinear transformations through the ReLU layer.

[0083] Step 402: Normalize the third vector and the fourth vector to generate a first weight and a second weight of the corresponding channel, where the sum of the first weight and the second weight is 1.

[0084] The vector d is respectively Figure 8 The second fully connected layer (FC2) in generates two vectors of length c: a and b, and then uses the softmax function to normalize and generate the weight value W1 (i.e., the first weight) and weight value W2 (i.e., the second weight) of the corresponding channel, W1+W2=1.

[0085] Step 403: Calculate the first product of the first weight and the feature map of the previous layer, and calculate the second product of the second weight and the feature map of the next layer, add the first product, the second product and the feature map of the current layer element by element, and use the addition result as the second-order feature map.

[0086] The weight value W1 is multiplied by the feature map xi-1 of the previous layer in the encoder, and the weight value W2 is multiplied by the feature map xi+1 of the next layer. The multiplied result is added element by element to the feature map xi of the current layer to obtain the second-order feature map of multi-scale features.

[0087] After steps 401 to 403 provided in this embodiment, the feature maps output by each layer of the encoder can be superimposed to form a second-order feature map of multi-scale features.

[0088] In actual implementation, the specific calculation method of step 403 is: use the downsampling function to downsample the feature map of the previous layer, multiply the first weight by the downsampling result to obtain the first product; use the upsampling function to upsample the feature map of the next layer, multiply the second weight by the upsampling result to obtain the second product; wherein the upsampling function and the downsampling function use convolution and transposed convolution with a step size of 2, and a linear rectification layer, so as to obtain the nonlinear features of the feature map of the previous layer and the feature map of the next layer. The calculation process can be expressed by the following formulas (7) to (11):

[0089] s=F gp (x i ) (7)

[0090] d = ReLU(FC 1(s)) (8)

[0091] (a, b) = FC 2(d) (9)

[0092] (W1, W2) = Softmax (a, b) (10)

[0093] u i =α(W1·Down(x i-1 )+W2·Up(x i+1 ))+β·x i (11)

[0094] Among them, F gp (xi) represents global pooling, x i-1 、x i 、x i+1 、u i They are the second-order feature maps output by the (i-1), i, and (i+1)th layer encoders, respectively. Down(x) is the downsampling function, and Up(x) is the upsampling function. In the specific implementation, the upsampling function and the downsampling function can use convolution and transposed convolution with a step size of 2, and are followed by a ReLU layer to obtain nonlinear features.

[0095] In actual implementation, formula (11) also includes a first parameter α and a second parameter β, both of which are adaptive trainable parameters to adjust the possible differences in the weight requirements of different layers. Specifically: the first parameter is used as a weighting parameter of the sum of the first product and the second product, and the second parameter is used as a weighting parameter of the feature map of the current layer, and the corresponding weighted addition result is calculated; wherein the first parameter and the second parameter are adaptive trainable parameters of length 2. The calculation method of the first parameter α and the second parameter β is as follows formula (12):

[0096]

[0097] It can be seen that the multi-layer attention module can effectively combine multi-scale feature information, and utilize the commonalities between adjacent layers to globally pool the current feature map as a reference weight, so as to absorb the complementary information between adjacent feature layers in a targeted manner, and avoid the homogeneity between layers by adding a custom trainable weight to each layer, which helps each convolutional layer to extract more accurate feature information, enhance the segmentation ability of small targets and the adaptability to small data sets.

[0098] After obtaining the second-order feature map of multi-scale features, the decoder of pure CNN structure can be used to generate prediction results. Specifically: the decoder receives the second-order feature map, concatenates the second-order feature map with the up-sampled result of the deep feature map of the input image in the channel dimension, and restores the concatenation result to the same dimension through convolution to obtain the third-order feature map.

[0099] In the decoding stage, the decoder receives the second-order feature maps obtained by the multi-layer attention module. These second-order feature maps are concatenated with the upsampling results of the deep feature maps in the channel dimension and restored to the same dimension through convolution to obtain the third-order feature maps output by the encoder, as shown in formulas (13) and (14):

[0100]

[0101] Among them, Concat represents channel dimension concatenation, Up(x) is the upsampling operation, and Conv i The convolution operation is used to change the number of channels of the second-order feature map, and the final D i With F i Same dimensions.

[0102] In the specific implementation, for the input image H, W, C are the length, width, and number of channels of the image, respectively. First, a block embedding (a series of convolutions with a stride of 4) is performed to obtain Then, in the next few hybrid Transformer modules, the first-order feature maps of different stages are obtained by block merging (a series of convolution operations with a step size of 2). These first-order feature maps are passed through a multi-layer attention module to obtain a second-order feature map M that integrates multi-scale feature information. i , and then concatenated with the upsampling result of the second-order feature map in the channel dimension, and restored to the same dimension through convolution to obtain the third-order feature map output by the encoder.

[0103] In the above process, the concatenation result is restored to the same dimension through convolution, which specifically includes: upsampling the second-order feature map to obtain a sampling result consistent with the input image size, and then using 1×1 convolution and normalization function to map the probability value of each pixel of the sampling result belonging to different categories to different channels to obtain a third-order feature map of the same dimension as the input image. In a specific implementation, the second-order feature map D1 can be upsampled to the original image size, and then 1×1 convolution and softmax function are used to map the probability value of each pixel belonging to different categories to different channels to obtain the final prediction result.

[0104] In actual implementation, in order to improve the accuracy of image target segmentation, the network parameters can be continuously updated through neural network loss calculation, the network can be trained, and the final segmentation result can be generated. Specifically: the cross entropy loss function is used to calculate the loss of the neural network model, the parameters of the neural network model are updated according to the calculated loss, the updated parameters are used to train and optimize the neural network model, and the optimized neural network model is used to segment the input image.

[0105] In a certain actual scenario, the loss function used is expressed as follows (15):

[0106]

[0107] In formula (15), K represents the number of categories, N represents the number of samples, and p ik It indicates the probability of predicting that the i-th sample is the k-th class, y ik Indicates whether the i-th sample belongs to the k-th category (ie, the true value).

[0108] In the specific implementation, the neural network model proposed in this embodiment can use the PyTorch framework and use a single NVIDIA GeForce RTX 3090 (24GB) graphics card for GPU acceleration. All experiments are run in the same environment. The Adam optimizer is used in training, the learning rate is 1e-4, the weight decay is set to 1e-4, the epoch is set to 200, and the batch size is 8. In the network weight initialization, the CNN part uses the Resnet-34 pre-trained on ImageNet as the backbone network, and the Transformer part uses random initialization.

[0109] Compared with other target segmentation methods based on the Transformer model, the image target segmentation method provided in this embodiment can utilize global features and local features at the same time, and combines multi-scale features to extract features from three different dimensions, thereby enhancing the model feature expression ability and improving the adaptability to small sample data sets; and the pure CNN structure adopted by the decoder can avoid generating too much redundant information and significantly reduce the network computing power cost, achieving a good balance between network performance and computing overhead. The model structure designed by this method enhances the robustness and generalization of the network, improves the network segmentation accuracy at unit cost, and also provides a new idea for the target segmentation task on small data sets.

[0110] Embodiment 2:

[0111] Based on the image target segmentation method provided in the above embodiment 1, the present invention also provides an image target segmentation device that can be used to implement the above method, such as Fig.10, which is a schematic diagram of the device architecture of an embodiment of the present invention. The device for image object segmentation of this embodiment includes one or more processors 11 and a memory 12. Fig.10 A processor 11 is taken as an example.

[0112] The processor 11 and the memory 12 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0113] The memory 12 is a non-volatile computer-readable storage medium for the method of image object segmentation, and can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the method of image object segmentation in Example 1. The processor 11 executes various functional applications and data processing of the device for image object segmentation by running the non-volatile software programs, instructions and modules stored in the memory 12, that is, the method of image object segmentation in Example 1 is implemented.

[0114] The memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 12 may optionally include a memory remotely arranged relative to the processor 11, and these remote memories may be connected to the processor 11 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The program instructions / modules are stored in the memory 12, and when executed by one or more processors 11, the method for image object segmentation in the above embodiment 1 is executed, for example, the above described Figure 2 , Figure 3 Figure 7 and Fig. 9 The steps shown.

[0116] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for image object segmentation, characterized in that: include: The input image to be segmented is input into the encoder, and the encoder is used to extract the features of the input image to obtain a first-order feature map; wherein each layer of the encoder network contains a hybrid Transformer module composed of a convolutional structure and a Transformer structure; Obtain the first-order feature map output by the hybrid Transformer module of the current layer as the feature map of the current layer, obtain the first-order feature map output by the hybrid Transformer module of the previous layer as the feature map of the previous layer, and obtain the first-order feature map output by the hybrid Transformer module of the next layer as the feature map of the next layer; convert the feature map of the current layer into a first vector of a first length through a global pooling layer, the first vector generates a second vector of a second length through a first fully connected layer, and the second vector generates a third vector and a fourth vector of a first length through a second fully connected layer; normalize the third vector and the fourth vector to generate the first weight and the second weight of the corresponding channel, and the sum of the first weight and the second weight is 1; calculate the first product of the first weight and the feature map of the previous layer, and calculate the second product of the second weight and the feature map of the next layer, add the first product, the second product and the feature map of the current layer element by element, and use the addition result as the second-order feature map; The second-order feature map is input into the decoder of the pure convolution structure, the second-order feature map is restored to a third-order feature map with the same size as the input image, and the third-order feature map is used as the prediction result of image target segmentation.

2. The method for image object segmentation according to claim 1, characterized in that: In each hybrid Transformer module, the encoder is used to extract input image features to obtain a first-order feature map, specifically including: Use multiple convolution kernels of different scales to extract local spatial features in the input image, perform cross-scale embedding and image sequence conversion on the local spatial features to obtain an embedding vector with a specified number of channels; Two consecutive Transformer layers are used to extract the global spatial features of the embedded vector, and the extracted global spatial features are converted into corresponding first-order feature maps.

3. The method for image object segmentation according to claim 2, characterized in that: The method of extracting the global spatial features of the embedded vector using two consecutive Transformer layers specifically includes: According to the number of layers of the current hybrid Transformer module, a self-attention window with a specified number of attention heads is set in the Transformer layer, and the embedding vector is input into the self-attention window to obtain a global attention vector containing multi-head features; The global attention vector is expanded and its linear projection is obtained. The nonlinear features in the linear projection are learned through the activation function to obtain the global spatial features.

4. The method for image object segmentation according to claim 1, characterized in that: The calculating of the first product of the first weight and the previous layer feature map, and the calculating of the second product of the second weight and the next layer feature map, specifically includes: Use the downsampling function to downsample the feature map of the previous layer, and multiply the first weight by the downsampling result to obtain the first product; Use the upsampling function to upsample the feature map of the next layer, and multiply the second weight by the upsampling result to obtain the second product; Among them, the upsampling function and the downsampling function use convolution and transposed convolution with a step size of 2, as well as a linear rectification layer, to obtain the nonlinear characteristics of the feature map of the previous layer and the feature map of the next layer.

5. The method for image object segmentation according to claim 1, characterized in that: The adding the first product, the second product and the feature map of the current layer element by element specifically includes: Using the first parameter as a weighting parameter of the sum of the first product and the second product, using the second parameter as a weighting parameter of the feature map of the current layer, and calculating the corresponding weighted addition result; The first parameter and the second parameter are adaptive trainable parameters with a length of 2.

6. The method for image object segmentation according to claim 1, characterized in that: Restoring the second-order feature map to a third-order feature map having the same size as the input image specifically includes: The decoder receives the second-order feature map, concatenates the second-order feature map with the upsampling result of the deep feature map of the input image in the channel dimension, and restores the concatenated result to the same dimension through convolution to obtain the third-order feature map.

7. The method for image object segmentation according to claim 6, characterized in that: The concatenated result is restored to the same dimension by convolution, specifically including: The second-order feature map is upsampled to obtain a sampling result that is consistent with the input image size, and then a 1×1 convolution and normalization function is used to map the probability value of each pixel of the sampling result belonging to different categories to different channels to obtain a third-order feature map of the same dimension as the input image.

8. The method for image object segmentation according to claim 1, characterized in that: The method further comprises: The cross entropy loss function is used to calculate the loss of the neural network model, the parameters of the neural network model are updated according to the calculated loss, the updated parameters are used to train and optimize the neural network model, and the optimized neural network model is used to segment the input image.

9. A device for image object segmentation, characterized in that: The method comprises at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions executable by the at least one processor, and after the instructions are executed by the processor, they are used to complete the method for image target segmentation according to any one of claims 1 to 8.

Images