A segmentation method and device based on a hybrid structure and a storage medium

Abstract

The application discloses a segmentation method and device based on a mixed structure and a storage medium, and the method comprises the following steps: inputting an image to be segmented into a backbone network model based on CNN or a Transformer, and obtaining multi-scale features of the image to be segmented; further aligning the multi-scale features by using a variability encoder; dividing the aligned multi-scale features into two parts, inputting one part into a pixel decoder to obtain a feature map, and inputting the other part into a mixed decoder to generate spatial mixed features; obtaining corresponding feature representations by using a feedforward neural network on the generated spatial mixed features; and regressing the feature representations to obtain the category and the corresponding bounding box of an object, performing convolution operation on the feature map generated by the pixel decoder, and obtaining a corresponding segmentation mask. The method has the characteristics of good segmentation effect and fast convergence.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a segmentation method and device based on a hybrid structure and a storage medium. BACKGROUND

[0002] Existing segmentation methods are generally based on convolutional neural networks or Transformer models. Segmentation models based on convolutional neural networks have developed a lot, but convolutional neural networks have the characteristic of being difficult to capture global correlations between picture features. Therefore, segmentation models based on convolutional neural networks currently encounter some bottlenecks. Although some solutions that use data augmentation and neural architecture search can continue to improve the segmentation effect of segmentation models based on convolutional neural networks, the cost is also very high.

[0003] In recent years, especially since the DETR model was proposed in 2020, visual detection and segmentation models based on Transformers have received more attention and development. Models based on Transformers can establish global connections between features, which brings a greater performance improvement compared to convolutional neural networks. However, Transformers have the characteristic of slow convergence. For example, DETR requires 300 iterations (epochs) to converge, and the computational requirements are high, which is unacceptable for large-scale applications of the model. SUMMARY

[0004] The present application provides a segmentation method and device based on a hybrid structure and a storage medium, which has the characteristics of good segmentation effect and fast convergence.

[0005] The first aspect of the present application provides a segmentation method based on a hybrid structure, the method comprising the following steps:

[0006] inputting a to-be-segmented image into a CNN or Transformer-based backbone network model to obtain multi-scale features of the to-be-segmented image;

[0007] further aligning the multi-scale features using a variability encoder;

[0008] dividing the aligned multi-scale features into two parts, one part being input into a pixel decoder to obtain a feature map, and the other part being input into a hybrid decoder to generate spatial mixed features;

[0009] The generated spatial mixed features are input into a feedforward neural network to obtain corresponding feature representations, and the feature representations are regressed to obtain the category and corresponding bounding box of the object, and then are convolved with the feature map generated by the pixel decoder to obtain a corresponding segmentation mask.

[0010] A further technical solution of the present application is that the specific method for further aligning the multi-scale features by the variable encoder comprises: extracting the upper and lower features adjacent to the features to perform 3D convolution, performing up-sampling on the lower features, performing down-sampling on the upper features, then splicing the upper features and the lower features, and if there is no upper feature or lower feature, the corresponding feature is not used, and only two layers are used for alignment.

[0011] A further technical solution of the present application is that the pixel decoder is a learnable adaptive pixel decoder, and specifically comprises: adaptively learning a feature mixture suitable for the current scale according to the position vector.

[0012] A further technical solution of the present application is that the specific method for generating the spatial mixed features in the mixed decoder comprises:

[0013] A position-aware sampling coordinate is generated according to the Query and the position vector, and sampling is performed on the last layer of the deformable encoder according to the sampling coordinate to obtain a sampled feature set;

[0014] The Query interacts with the feature set, and specifically comprises: after the feature set is subjected to Layer norm, the channel mixed features are obtained by mixing the weight generated by the Content Query;

[0015] The channel mixed features are transposed and then mixed with the weight generated by the Content Query to obtain spatial mixed features.

[0016] A further technical solution of the present application is that the channel of the mixed decoder is divided into multiple groups of mixed structures to generate multiple groups of spatial mixed features, the multiple groups of spatial mixed features are projected to the same dimension as the Content Query through a linear layer, and then residual addition is performed to obtain the final spatial mixed features.

[0017] The second aspect of the present application provides a segmentation device based on a mixed structure, and the device comprises:

[0018] The multi-scale feature acquisition module is configured to input the image to be segmented into a CNN or Transformer based backbone network model to obtain multi-scale features of the image to be segmented.

[0019] A multi-scale feature preprocessing module is configured to further align the multi-scale features by using a variability encoder;

[0020] A multi-scale feature processing module is configured to divide the aligned multi-scale features into two parts, one of which is input into a pixel decoder to obtain a feature map, and the other of which is input into a mixed decoder to generate spatial mixed features.

[0021] A segmentation mask obtaining module is configured to obtain a corresponding feature representation by inputting the generated spatial mixed features into a feedforward neural network, regress a class and a corresponding bounding box of an object from the feature representation, and perform convolution operation on the feature map generated by the pixel decoder to obtain a corresponding segmentation mask.

[0022] In a third aspect, the application provides a segmentation device based on a mixed structure, comprising a processor and a memory, wherein the memory stores computer executable programs, and when the computer executable programs are executed by the processor, the segmentation method based on the mixed structure is executed.

[0023] In a fourth aspect, the application provides a computer readable storage medium, which stores instructions, and when the instructions are executed by a processor, the processor executes the segmentation method based on the mixed structure.

[0024] The segmentation method, device and storage medium based on the mixed structure provided by the application effectively accelerate the model convergence by using the soft restriction of target detection as mixed sampling, and can be used for universal segmentation as the current Query-based method. The scale mixing method is proposed, the previous Query-base segmentation model usually only focuses on how to learn a good Query, but ignores learning a good feature representation, the scale mixing method is proposed to automatically adjust the mixing strategy before different scale features by using a group of learnable adaptive pixel decoders, which effectively improves the segmentation effect, and the additional computational cost can be ignored. The multi-head mixing structure is proposed, which refers to the multi-head attention mechanism in the Transformer model, can effectively improve the segmentation effect, and can reduce certain computational cost and parameter amount.

[0025] 1. The application proposes a universal segmentation method based on a mixed structure, which has the characteristics of good segmentation effect and adaptability to various segmentation tasks (semantic segmentation, instance segmentation and panoramic segmentation).

[0026] 2. The application introduces a deformable encoder in the mixed architecture, which can effectively align the features and greatly help the mixed sampling in the later stage, and further improve the final segmentation effect.

[0027] 3. The application proposes a scale mixing method, which enables the pixel decoder to adaptively learn the mixing strategy of different scale features, and effectively improves the segmentation effect.

[0028] 4. The application proposes a multi-head mixing structure, which uses a principle similar to the multi-head attention mechanism in the mixed structure, and uses multiple sets of different mixing parameters to learn features, which can effectively improve the final segmentation effect. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flow diagram of the segmentation method based on the mixed structure in the embodiment of the application;

[0030] Figure 2 is a flow diagram of the adaptive mixing update in the embodiment of the application;

[0031] Figure 3 is a structure diagram of the segmentation device based on the mixed structure in the embodiment of the application;

[0032] Figure 4 is the architecture of the computer device in the embodiment of the application. DETAILED DESCRIPTION

[0033] To further describe the technical solutions of the application, the embodiment is implemented on the premise of the technical solutions of the application, and gives detailed implementation modes and specific steps.

[0034] The features, structures or characteristics described in the disclosure can be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to give a sufficient understanding of the embodiments of the disclosure. However, those skilled in the art will realize that one or more of the specific details can be omitted, or other methods, components, devices, steps, etc. can be used to practice the technical solutions of the disclosure. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring the aspects of the disclosure.

[0035] The accompanying drawings are only intended to schematically illustrate the present disclosure, and the same reference signs in the drawings represent the same or similar parts, and thus repeated description thereof will be omitted. Some block diagrams shown in the drawings do not necessarily have to correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or hardware acquisition modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] The abbreviations and key terms involved in the embodiments are defined as follows:

[0037] Instance Segmentation: further separates the foreground and background of an object pixel by pixel on the basis of object detection and accurately classifies them;

[0038] Transformer model: a modeling algorithm and a corresponding model with self-attention as the core.

[0039] MLP (Multi-layer Perceptrons): a kind of neural network with feed-forward networks as the core, which usually has a differentiable nonlinear activation function and can be effectively learned using the backpropagation algorithm.

[0040] MLP-Mixer model: a computer vision backbone network with MLP as the core, which is different from general convolutional neural network-based vision networks and Transformer-based vision networks, and has MLP as the main part in the network.

[0041] Query-based method: generally uses a set of fixed-dimension queries to represent an object, and the query is usually a 256-dimensional vector.

[0042] Mixing structure: a network structure using a fully connected layer, which is a short name for MLP-Mixer.

[0043] The embodiments of the present application provide the following embodiments for the segmentation method, device and storage medium based on the mixing structure:

[0044] Based on the embodiment 1 of the present application

[0045] This embodiment is used to illustrate the segmentation method based on the mixing structure, and the flowchart is as shown in Figure 1 The method comprises the following steps:

[0046] The image to be segmented is input into a CNN or Transformer-based backbone network model to obtain multi-scale features of the image to be segmented;

[0047] The multi-scale features are further aligned using a deformable encoder;

[0048] The aligned multi-scale features are divided into two parts, one part is input into a pixel decoder to obtain a feature map, and the other part is input into a mixing decoder to generate spatial mixing features;

[0049] The generated spatial mixing features are input into a feedforward neural network to obtain corresponding feature representations, and the feature representations are regressed to obtain the category and corresponding bounding box of the object. The feature map generated by the pixel decoder is convolved to obtain the corresponding segmentation mask.

[0050] In the specific implementation process, as shown in Figure 1 , first, the embodiment fully utilizes the structure of decoupling Content query and position information commonly used in current target detection in structural design, which has the characteristics of local attention and can significantly accelerate model convergence. First, the image to be segmented is input into a CNN or Transformer-based backbone network model to extract multi-scale features of the image to be segmented, such as the image feature pyramid (ImagePyramidFeatures) shown in Figure 1 ; then, a deformable encoder is used to further align the multi-scale features. It should be noted that the process of further aligning the multi-scale features by the deformable encoder in the embodiment can be repeated multiple times, and in the preferred embodiment of the present application, it is set to be repeated M=4 times; further, the aligned multi-scale features are divided into two parts, one part is input into a pixel decoder to obtain a feature map, and the other part is input into a mixing decoder (Mixing Decoder) to generate spatial mixing features, wherein the pixel decoder (Pixel decoder) is composed of a semantic feature pyramid network (Semantic FPN) and a scale mixing (Scale Mixing) method.

[0051] In the preferred embodiment, the specific method of further aligning the multi-scale features by the deformable encoder includes: extracting the upper and lower features adjacent to the features to do 3D convolution, upsampling the lower features, downsampling the upper features, then splicing the upper and lower features, if there is no upper or lower feature, then do not use the corresponding feature, only use two layers to align.

[0052] In the implementation process, in the target detection and segmentation task, using different levels of features for prediction is a common operation. However, how to fuse different feature layers is a problem worth considering. The current method is basically to scale first, scale to the same resolution, and then add or splice. However, different levels of feature layers have semantic differences, which is particularly evident in the mixing of cross-scale sampling, resulting in poor results. Considering that the correlation between different layers is different, and the closer the feature layers are, the greater the correlation is, the present application proposes to use dynamic convolution to align the features.

[0053] Specifically, the features adjacent to the features are extracted to do 3D convolution, and then the features of the lower layer are up-sampled and the features of the upper layer are down-sampled, and then the features are spliced. If there is no upper feature or lower feature, the corresponding feature is not used, and only two layers are used for alignment. This process is repeated M times. In a preferred embodiment, M = 4.

[0054] In a preferred embodiment, the pixel decoder (Pixel Decoder) is a kind of learnable adaptive pixel decoder, which specifically includes: learning a feature mixing suitable for the current scale according to the positional vector.

[0055] In the implementation process, the current Query-based general segmentation method generally only emphasizes how to make the learning better Query, and ignores the design of the pixel decoder (Pixel Decoder). The present application proposes a kind of learnable adaptive Pixel Decoder, which learns a feature mixing suitable for the current scale according to the positional vector (Positional vector). The Pixel Decoder can adaptively sample features according to the position of the current positional vector, and these samples will be used as the input of the Mixer to further update the learning Query.

[0056] In a preferred embodiment, the specific method of generating spatial mixed features in the mixing decoder (Mixing Decoder) includes:

[0057] Generating a sampling coordinate with position awareness according to the Query and the positional vector, sampling in the last layer of the deformable encoder according to the sampling coordinate to obtain a set of sampled features;

[0058] The Query interacts with the set of features, specifically including: the set of features will be mixed with the weight generated by the content Query after Layer norm to obtain channel mixed features;

[0059] The channel mixing features are transposed and mixed with the weights generated by the Content Query again to obtain spatial mixed features.

[0060] In the implementation process, the mixing decoder is a key part of the embodiment. First, a content-related sampling coordinate with positional awareness is generated according to the Query and the positional vector. Then, the sampling coordinate is used to sample the last layer of the deformable encoder to obtain a set of sampled features.

[0061] It should be noted that the Queries will first undergo a Query Interaction process. This process is similar to the self-attention mechanism in the Transformer-based method, but different from it. In order to fully utilize the decoupling advantage of the Content Query and the Positional vector, the embodiment takes the positional vector as the positional encoding information of the content query.

[0062] The Query interacts with the feature set. In the embodiment, this process is called Adaptive Mixing Update, as shown in Figure 2 After the Layer norm, the features are mixed with the weights generated by the Content Query to obtain channel mixed features, i.e., Channel Mixing in Figure 2 Then, the obtained features are transposed and mixed with the weights generated by the Content Query again to obtain spatial mixed features, i.e., Spatial mixing in Figure 2

[0063] In the preferred embodiment, the mixing decoder channels are divided into multiple groups of mixing structures to generate multiple groups of spatial mixed features. The multiple groups of spatial mixed features are projected to the same dimension as the Content Query through a linear layer, and then a residual addition operation is performed to obtain the final spatial mixed features.

[0064] ​In the implementation process, this part adopts the multi-head design in the Transformer, called Multi-head mixing. That is, the channel is divided into multiple parts, and then a multi-group mixing structure is used to generate spatial mixing features respectively, and then the obtained multiple groups of spatial mixing features are projected to the same dimension as the Content Query through a linear layer, and then residual addition operation is performed.

[0065] This process is repeated N times, and in the preferred embodiment, N = 6.

[0066] The generated spatial mixing features are input into the feedforward neural network to obtain the corresponding feature representation, and the feature representation is used to regress the category and corresponding bounding box of the object. The feature map generated by the pixel decoder is convolved to obtain the corresponding segmentation mask.

[0067] In the implementation process, as shown in Figure 1 , the generated spatial mixing features are input into different feedforward neural networks (FFN) to obtain the corresponding representation, and then the category and corresponding bounding box of the object are regressed. After convolution (FFN generates corresponding convolution kernel weights) operation with the feature map generated by the pixel decoder, the corresponding segmentation mask can be obtained.

[0068] It should be noted that the CNN or Transformer-based backbone network model can be any visual backbone network model, including but not limited to ResNet, Swin-Transformer, PVT, etc. neural network model.

[0069] Based on the embodiment 2 of the present application

[0070] In the following, with reference to Figure 3To describe the device corresponding to the method according to the embodiment 1 of the present disclosure, a segmentation device based on a hybrid structure, the device 300 comprises: a multi-scale feature acquisition module 301, configured to input an image to be segmented into a CNN or a Transformer based backbone network model to obtain multi-scale features of the image to be segmented; a multi-scale feature preprocessing module 302, configured to further align the multi-scale features by using a variability encoder; a multi-scale feature processing module 303, configured to divide the aligned multi-scale features into two parts, one part is input into a pixel decoder to obtain a feature map, and the other part is input into a hybrid decoder to generate spatial mixed features; a segmentation mask acquisition module 304, configured to obtain a corresponding feature representation by feeding the generated spatial mixed features through a feedforward neural network, regress the class and the corresponding bounding box of the object, and perform convolution operation on the feature map generated by the pixel decoder to obtain a corresponding segmentation mask. In addition to the above four modules, the device 300 can also include other components, however, since these components are irrelevant to the content of the present disclosure, their illustrations and descriptions are omitted here.

[0071] The specific working process of the segmentation device 300 based on the hybrid structure is described with reference to the description of the segmentation method based on the hybrid structure in the above embodiment 1, and will not be repeated here.

[0072] Based on the embodiment 3 of the present disclosure

[0073] The device according to the embodiment of the present disclosure can also be implemented by means of Figure 4 The architecture of the computing device is shown. Figure 4 The architecture of the computing device is shown. As Figure 4 shown, the computer system 401, the system bus 403, one or more CPUs 404, input / output 402, memory 405, etc. The memory 405 can store various data or files used by the computer processing and / or communication and the program instructions executed by the CPU including the method of embodiment 1. Figure 4 The architecture shown is only exemplary, and when implementing different devices, one or more components in Figure 4 are adjusted according to actual needs.

[0074] Based on the embodiment 4 of the present disclosure

[0075] The embodiment of the present disclosure can also be implemented as a computer readable storage medium. The computer readable storage medium according to the embodiment 4 stores computer readable instructions. When the computer readable instructions are run by a processor, the segmentation method based on the hybrid structure according to the embodiment 1 of the present disclosure described with reference to the above figures can be executed.

[0076] The embodiment of the application is directed to the above-mentioned segmentation method, device and storage medium based on the mixed structure, which uses the target detection as a soft limit of mixed sampling, effectively accelerates the model convergence, and can be used for universal segmentation as the current Query-based method; a variable encoder is used to align features of different scales, effectively suppressing the phenomenon of inconsistent convolution in different scales, effectively accelerating the convergence speed and segmentation effect of the model; a scale mixing method is proposed, the previous Query-base segmentation model usually only focuses on how to learn a good query, but ignores learning a good feature representation, the application proposes a scale mixing method, which uses a set of learnable adaptive pixel decoders to automatically adjust the mixing strategy before different scale features, effectively improving the segmentation effect, and the additional computational cost can be ignored; a multi-head mixing structure is proposed, which refers to the multi-head attention mechanism in the Transformer model, can effectively improve the segmentation effect, and can reduce certain computational amount and parameter amount. In summary, the beneficial effects achieved by the embodiment are:

[0077] 1. The application proposes a universal segmentation method based on a mixed structure, which has the characteristics of good segmentation effect and can adapt to various segmentation tasks (semantic segmentation, instance segmentation, panoramic segmentation).

[0078] 2. The application introduces a deformable encoder in the mixed architecture, which can effectively align the features and greatly help the mixed sampling in the next stage, and further improve the final segmentation effect.

[0079] 3. The application proposes a scale mixing method, which enables the pixel decoder to adaptively learn the mixing strategy of different scale features, which can effectively improve the segmentation effect.

[0080] 4. The application proposes a multi-head mixing structure, which uses a principle similar to the multi-head attention mechanism in the mixed structure to learn features using multiple sets of different mixing parameters, which can effectively improve the final segmentation effect.

[0081] In this document, the terms "comprise", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a step, method, process, machine, article of manufacture, or composition of matter that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such step, method, process, machine, article of manufacture, or composition of matter.

[0082] The above description is further to specific preferred embodiments of the present application and cannot be deemed to limit the specific implementation of the present application to these descriptions. For those of ordinary skill in the art to which the present application belongs, a number of simple deductions or replacements can be made without departing from the concept of the present application, and all of these should be deemed to fall within the protection scope of the present application.

Claims

1. A method for segmenting based on a hybrid structure, characterized in that, The method comprises the following steps: inputting an image to be segmented into a backbone network model based on CNN or Transformer to obtain multi-scale features of the image to be segmented; further aligning the multi-scale features by using a deformable encoder; dividing the aligned multi-scale features into two parts, one part being input into a pixel decoder to obtain a feature map, and the other part being input into a hybrid decoder to generate spatial mixed features; obtaining corresponding feature representations by using a feedforward neural network on the generated spatial mixed features, and regressing the feature representations to obtain the category and corresponding bounding box of an object, and performing convolution operation on the feature map generated by the pixel decoder to obtain a corresponding segmentation mask; the specific method for generating spatial mixed features in the hybrid decoder comprises: generating a sampling coordinate with position perception according to a query and a position vector, sampling features in the last layer of the deformable encoder according to the sampling coordinate to obtain a sampled feature set; interacting the query with the feature set, specifically including: mixing the channel mixed features with the weight generated by the content query after the layer norm to obtain the channel mixed features; transposing the channel mixed features and mixing them with the weight generated by the content query again to obtain the spatial mixed features.

2. The hybrid structure-based segmentation method of claim 1, wherein, The specific method for further aligning the multi-scale features by using the deformable encoder comprises: extracting the upper and lower features adjacent to the features to perform 3D convolution, up-sampling the lower features, down-sampling the upper features, and then splicing the upper features and the lower features, if there is no upper feature or lower feature, then the corresponding feature is not used, and only two layers are used for alignment.

3. The hybrid structure based segmentation method of claim 1, wherein, The pixel decoder is a learnable adaptive pixel decoder, specifically including: adaptively learning a feature mixing suitable for the current scale according to the position vector.

4. The hybrid structure based segmentation method of claim 1, wherein, The channel of the hybrid decoder is divided into multiple groups of mixed structures to generate multiple groups of spatial mixed features, the multiple groups of spatial mixed features are projected to the same dimension as the content query by using a linear layer, and then residual addition operation is performed to obtain the final spatial mixed features.

5. A segmentation apparatus based on a hybrid structure, characterized by The device comprises: a multi-scale feature acquisition module configured to input an image to be segmented into a backbone network model based on CNN or Transformer to obtain multi-scale features of the image to be segmented; a multi-scale feature preprocessing module configured to further align the multi-scale features by using a deformable encoder; a multi-scale feature division processing module configured to divide the aligned multi-scale features into two parts, one part being input into a pixel decoder to obtain a feature map, and the other part being input into a hybrid decoder to generate spatial mixed features; a segmentation mask acquisition module configured to obtain corresponding feature representations by using a feedforward neural network on the generated spatial mixed features, and regress the feature representations to obtain the category and corresponding bounding box of an object, and perform convolution operation on the feature map generated by the pixel decoder to obtain a corresponding segmentation mask; the specific method for generating spatial mixed features in the hybrid decoder comprises: A sampling coordinate with position awareness is generated for the multi-scale features according to the Query and the position vector, and sampling is performed on the last layer of the deformable encoder according to the sampling coordinate to obtain a sampled feature set; The Query interacts with the feature set, specifically including that the feature set is mixed with the weight generated by the Content Query after Layer norm to obtain a channel mixed feature; The channel mixed feature is transposed and mixed with the weight generated by the Content Query again to obtain a spatial mixed feature.

6. A hybrid structure-based segmentation apparatus, characterized by, Comprise: a processor; and a memory, wherein the memory has stored therein computer executable programs which, when executed by the processor, perform the segmentation method based on the hybrid structure according to any one of claims 1-4.

7. A computer readable medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the segmentation method based on the hybrid structure according to any one of claims 1-4. The computer program is executed by the processor to implement the segmentation method based on the hybrid structure according to any one of claims 1-4.

Images