Superpixel segmentation method and system based on attention mechanism and convolutional neural network

Abstract

The present disclosure provides a superpixel segmentation method and system based on an attention mechanism and a convolutional neural network. The scheme first embeds an attention module into a convolutional neural network. Secondly, the resulting squeeze and excitation network is trained end-to-end. Then, a flexible loss function is used to learn superpixels for specific tasks. Finally, the trained network can be used to perform superpixel segmentation of the image, greatly reducing the dimension and eliminating some abnormal pixels. The algorithm based on the attention mechanism and convolutional neural network superpixel segmentation can obtain better segmentation results, providing a method with obvious advantages in the field of image superpixel segmentation.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and in particular, relates to a superpixel segmentation method and system based on an attention mechanism and a convolutional neural network. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Superpixel segmentation is an important preprocessing step in computer image processing. In computer vision, a superpixel is a grouping of pixels with similar features to form a more representative large element. This new element will become the basic unit of other image processing algorithms. It can not only greatly reduce the size, but also eliminate some abnormal pixels. A large number of experimental analyses have shown that the superpixel segmentation method based on deep learning not only outperforms the existing superpixel algorithms on traditional segmentation benchmarks, but also can learn superpixels for other tasks. In addition, deep learning networks can be easily integrated into downstream deep networks to improve performance. At present, due to their representativeness and computational efficiency, superpixels have been widely used in computer vision algorithms such as object detection, semantic segmentation, saliency estimation, optical flow estimation, depth estimation, tracking, etc.

[0004] The inventors found that the existing super pixel segmentation method has the following disadvantages:

[0005] (1) Most gradient-based superpixels start with an initial clustering of pixels and iteratively update the clustering by gradient changes until certain criteria are met to form a superpixel.

[0006] (2) SLIC (Simple Linear Iterative Clustering) is a super pixel segmentation algorithm based on K-means clustering. Although it can control the number and compactness of super pixel blocks, this method only considers the color and coordinate relationship between the pixel point and the seed point, but does not consider the relationship between the pixel point and the boundary, so the degree of fitting to the image boundary is not very good.

[0007] (3) DB-SCAN (Density-based Spatial Clustering of Applications with Noise) is a superpixel segmentation algorithm based on density clustering. Since the DB-SCAN clustering algorithm can find clusters of arbitrary shapes, it has good segmentation potential for objects with complex and irregular shapes, but it does not consider the spatial relationship between pixels and seed points, resulting in irregular shapes of superpixels. Summary of the invention

[0008] In order to solve the above problems, the present disclosure provides a superpixel segmentation method and system based on attention mechanism and convolutional neural network. The image superpixel segmentation result obtained by the scheme can not only greatly reduce the image dimension, but also effectively eliminate some abnormal pixels, laying a foundation for subsequent applications and improving computational efficiency.

[0009] According to a first aspect of an embodiment of the present disclosure, a superpixel segmentation method based on an attention mechanism and a convolutional neural network is provided, comprising:

[0010] Obtain an image to be segmented by superpixels;

[0011] Inputting the image into a pre-trained superpixel segmentation model to obtain a predicted superpixel association map, and determining a superpixel segmentation result of the image based on the superpixel association map;

[0012] Among them, the superpixel segmentation model adopts an encoder-decoder design, the encoder includes several convolutional layers, and the image generates feature maps of different scales through convolutional layers of different levels in the encoder; the decoder includes several deconvolutional layers, and the feature maps generated by convolutional layers of different levels in the encoder are transmitted to the deconvolution layers of the corresponding level in the decoder through jump connections; at the same time, an attention module is set before the input of each deconvolution layer in the decoder.

[0013] Furthermore, the attention module adopts the SE-Net attention module, which includes a squeezing operation and an excitation operation. The squeezing operation generates a channel descriptor by aggregating feature maps in the spatial dimension; the excitation operation adopts a gating mechanism and takes the embedding of the global distribution generated by the channel descriptor as input to obtain a set of modulation weights for each channel.

[0014] Furthermore, based on the superpixel association map, the image superpixel segmentation result is determined, specifically: a predicted superpixel association map is obtained through the superpixel segmentation model, wherein the superpixel association map determines the probability of each pixel being assigned to a different grid unit based on a soft association map instead of the actual hard assignment of pixels; and a superpixel segmentation result is obtained by assigning each pixel to the grid unit with the highest probability.

[0015] Furthermore, the loss function used in the training process of the superpixel segmentation model includes two parts. The first part is used to combine pixels with similar attributes; the second part is used to force the superpixels to remain compact in space.

[0016] According to a second aspect of an embodiment of the present disclosure, a superpixel segmentation system based on an attention mechanism and a convolutional neural network is provided, comprising:

[0017] A data acquisition unit, which is used to acquire an image to be subjected to superpixel segmentation;

[0018] A superpixel segmentation unit, which is used to input the image into a pre-trained superpixel segmentation model to obtain a predicted superpixel association map, and determine the image superpixel segmentation result based on the superpixel association map;

[0019] Among them, the superpixel segmentation model adopts an encoder-decoder design, the encoder includes several convolutional layers, and the image generates feature maps of different scales through convolutional layers of different levels in the encoder; the decoder includes several deconvolutional layers, and the feature maps generated by convolutional layers of different levels in the encoder are transmitted to the deconvolution layers of the corresponding level in the decoder through jump connections; at the same time, an attention module is set before the input of each deconvolution layer in the decoder.

[0020] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device comprising a memory, a processor and a computer program stored and running on the memory, wherein when the processor executes the program, the superpixel segmentation method based on an attention mechanism and a convolutional neural network is implemented.

[0021] According to the fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the superpixel segmentation method based on the attention mechanism and convolutional neural network is implemented.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] (1) The scheme disclosed in the present invention provides a superpixel segmentation method based on an attention mechanism and a convolutional neural network. The scheme introduces the SE-Net attention module into the convolutional neural network to construct a superpixel segmentation model. The scheme uses the squeeze and excitation network generated by SE-Net for end-to-end training, and uses a flexible loss function to learn superpixels for specific tasks. Finally, the trained network can be used to perform superpixel segmentation of the image, greatly reducing the dimensionality and eliminating some abnormal pixels.

[0024] (2) Compared with the traditional FCN convolution, the scheme disclosed in the present invention adds an attention module to the convolutional neural network, which can better simulate the dependency between channels and adaptively adjust the characteristic response value of each channel. Adding an attention module to the network will only increase a small amount of computational overhead, but can greatly improve network performance.

[0025] (3) The scheme described in the present disclosure finds the correlation scores between image pixels and regular grid cells and uses a squeeze and excitation network to directly predict these scores. The final superpixel segmentation result is obtained by assigning each pixel to the regular grid cell with the highest probability to obtain a superpixel, which can effectively improve the efficiency and accuracy of superpixel segmentation.

[0026] Advantages of additional aspects of the present disclosure will be given in part in the following description and in part will become apparent from the following description or will be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0028] Figure 1 This is a schematic diagram of the overall network structure of the superpixel segmentation method based on the attention mechanism and convolutional neural network described in the first embodiment of the present disclosure;

[0029] Figure 2 This is a schematic diagram of the superpixel extrusion and excitation network structure described in the first embodiment of the present disclosure;

[0030] Figure 3 The first embodiment of the present disclosure is a schematic diagram of the structure of the extrusion and excitation module;

[0031] Figure 4 It is a schematic diagram of the superpixel segmentation result on the BSDS500s dataset described in the first embodiment of the present disclosure;

[0032] Figure 5 This is a schematic diagram of the superpixel segmentation results on the NYUv2 dataset described in the first embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0035] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0036] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0037] Embodiment 1:

[0038] The purpose of this embodiment is to provide a superpixel segmentation method based on attention mechanism and convolutional neural network.

[0039] A superpixel segmentation method based on attention mechanism and convolutional neural network, comprising:

[0040] Obtain an image to be segmented by superpixels;

[0041] Inputting the image into a pre-trained superpixel segmentation model to obtain a predicted superpixel association map, and determining a superpixel segmentation result of the image based on the superpixel association map;

[0042] Among them, the superpixel segmentation model adopts an encoder-decoder design, the encoder includes several convolutional layers, and the image generates feature maps of different scales through convolutional layers of different levels in the encoder; the decoder includes several deconvolutional layers, and the feature maps generated by convolutional layers of different levels in the encoder are transmitted to the deconvolution layers of the corresponding level in the decoder through jump connections; at the same time, an attention module is set before the input of each deconvolution layer in the decoder.

[0043] Furthermore, the attention module adopts the SE-Net (Squeeze-and-Excitation Networks) attention module, which includes a squeezing operation and an excitation operation. The squeezing operation generates a channel descriptor by aggregating feature maps in the spatial dimension; the excitation operation adopts a gating mechanism and takes the embedding of the global distribution generated by the channel descriptor as input to obtain a set of modulation weights for each channel.

[0044] Furthermore, based on the superpixel association map, the image superpixel segmentation result is determined, specifically: a predicted superpixel association map is obtained through the superpixel segmentation model, wherein the superpixel association map determines the probability of each pixel being assigned to a different grid unit based on a soft association map instead of the actual hard assignment of pixels; and a superpixel segmentation result is obtained by assigning each pixel to the grid unit with the highest probability.

[0045] Furthermore, the loss function used in the training process of the superpixel segmentation model includes two parts. The first part is used to combine pixels with similar attributes; the second part is used to force the superpixels to remain compact in space.

[0046] Furthermore, the predetermined loss function is specifically expressed as follows:

[0047]

[0048] Where p is a pixel in the image, p' is a pixel in the reconstructed image, f(p) is the feature of the pixel, f'(p) is the feature of the pixel in the reconstructed image, dist() represents the difference between the reconstructed feature and the original feature and the difference between the reconstructed position and the original position, m is the weight to balance these two items, and s is the sampling interval of the superpixel.

[0049] Specifically, for ease of understanding, the solution disclosed in the present disclosure is described in detail below with reference to the accompanying drawings:

[0050] like Figure 1 As shown, the scheme disclosed in the present invention adopts an encoder-decoder design to predict the super pixel association map Q through a super pixel segmentation model with a skip connection and an attention module. The encoder takes the color image as input and generates a high-level feature map through a convolutional network. Then, the decoder gradually upsamples the feature map through a deconvolution layer. An attention module is added before each deconvolution to pay more attention to the feature weight of each super pixel, thereby improving the accuracy of segmentation and making the final prediction.

[0051] For any given transformation that maps input to a feature map, such as a convolution operation, the scheme disclosed in the present invention sets a corresponding attention module before each convolution operation to perform feature recalibration. In this embodiment, the SE-Net attention module is used, and the SE-Net attention module includes a squeeze operation and an excitation operation. The input features of the convolution operation are first subjected to a squeeze operation to aggregate the feature map in the spatial dimension to produce a channel descriptor. The function of the descriptor is to produce an embedding of the global distribution of channel feature responses so that the information of the global receptive field of the network can be used by all its layers. The squeeze operation is followed by an excitation operation in the form of a simple gating mechanism that takes the embedding produced by the squeeze operation as input and produces a set of modulation weights for each channel. These weights are applied to the feature map to produce the output of the attention module, which can be directly input into the subsequent layers of the network. Then, we can write the output as U=[u1,u2,...,u C ],in:

[0052]

[0053] Among them, * represents convolution, X=[X 1 ,X 2 ,...,X C' ],u c ∈R H×W ; is a two-dimensional space kernel, representing v c A single channel acts on the corresponding channel of X; X is the input image, v c is the cth convolution kernel parameter, V c 1 is the first channel parameter of the cth convolution kernel, V c 2 The second channel parameter of the cth convolution kernel, R is the image feature map, V c C’ is the parameter of the C'th channel of the cth convolution kernel, c is the number of convolution kernels, C' is the number of feature channels after convolution transformation, u1 is the output of convolution kernel 1 operation, u2 is the output of convolution kernel 2 operation, u c is the output of the convolution kernel c operation, H is the height of the pixel, and W is the width of the pixel.

[0054] Furthermore, the solution disclosed in the present invention uses a soft association graph Q∈R H×W×|NP| (i.e., a superpixel association map, which indicates the probability of each pixel belonging to each superpixel) instead of the hard assignment G of the pixels. For example: Among them, s is the superpixel center, N p is the set of superpixels around a pixel, q s (p) indicates that a pixel p is assigned to each s∈N p Finally, a superpixel is obtained by assigning each pixel to the grid cell with the highest probability: * = arg max s q s (p). Different from the method based on fully convolutional neural network, the present invention adds an attention mechanism to the original network structure, which can effectively improve the accuracy of segmentation.

[0055] Furthermore, the scheme disclosed in the present disclosure has good flexibility in terms of loss functions. Generally speaking, f(p) represents the pixel attributes that we want the superpixel to retain. In this embodiment, f(p) includes a 3-dimensional CIELab color vector and an N-dimensional semantic label single encoding vector, where N represents the number of classes. We further use image coordinates p = [x, y] T represents the location of a pixel. Given the predicted superpixel association map Q, we can compute the center of any superpixel, where u s is the attribute vector, ls is the position vector, which is expressed as follows:

[0056]

[0057]

[0058] Where Np is the set of super pixels around p, q s (p) is the probability predicted by the network that p is associated with a superpixel. In equations (2) and (3), each sum is performed over all pixels that could be assigned to s. The reconstructed properties and location of any pixel p are then given by:

[0059]

[0060] Finally, the general formulation of our loss function has two terms. The first term encourages the trained model to group pixels with similar properties, and the second term enforces that superpixels be spatially compact.

[0061]

[0062] The superpixel segmentation model is trained based on the determined loss function, and the trained model is used to implement superpixel segmentation of the image.

[0063] Furthermore, in order to verify the advantages of the present disclosure for image superpixel segmentation, a large number of superpixel segmentation experiments were carried out on the BSDS500s dataset and the NYUv2 dataset in this embodiment. The model was first trained on the training set in the dataset, then verified on the validation set, and tested on the test set. The experimental results are shown in Figure 2. Figure 4 , Figure 5 As shown. Figure 4 , Figure 5 It can be seen that the convolutional neural network superpixel segmentation method based on the attention mechanism established in the present disclosure can achieve good results for the superpixel segmentation of images. The superpixel segmentation of images can not only greatly reduce the image dimension, but also eliminate some abnormal pixels. This shows that the convolutional neural network superpixel segmentation method based on the attention mechanism established in the present disclosure is effective, and improves the computational efficiency in subsequent object detection, semantic segmentation, saliency estimation, optical flow estimation, depth estimation, tracking and other fields, and has certain practicality.

[0064] Embodiment 2:

[0065] The purpose of this embodiment is to provide a superpixel segmentation system based on attention mechanism and convolutional neural network.

[0066] A superpixel segmentation system based on attention mechanism and convolutional neural network, comprising:

[0067] A data acquisition unit, which is used to acquire an image to be subjected to superpixel segmentation;

[0068] A superpixel segmentation unit, which is used to input the image into a pre-trained superpixel segmentation model to obtain a predicted superpixel association map, and determine the image superpixel segmentation result based on the superpixel association map;

[0069] Among them, the superpixel segmentation model adopts an encoder-decoder design, the encoder includes several convolutional layers, and the image generates feature maps of different scales through convolutional layers of different levels in the encoder; the decoder includes several deconvolutional layers, and the feature maps generated by convolutional layers of different levels in the encoder are transmitted to the deconvolution layers of the corresponding level in the decoder through jump connections; at the same time, an attention module is set before the input of each deconvolution layer in the decoder.

[0070] Furthermore, the attention module adopts the SE-Net attention module, which includes a squeezing operation and an excitation operation. The squeezing operation generates a channel descriptor by aggregating feature maps in the spatial dimension; the excitation operation adopts a gating mechanism and takes the embedding of the global distribution generated by the channel descriptor as input to obtain a set of modulation weights for each channel.

[0071] Furthermore, the image superpixel segmentation result is determined based on the superpixel association map, specifically: a predicted superpixel association map is obtained through the superpixel segmentation model, wherein the superpixel association map is based on a soft association map instead of the actual pixel hard assignment, and the probability of each pixel being assigned to a different grid unit is determined; and the superpixel segmentation result is obtained by assigning each pixel to the grid unit with the highest probability. In more embodiments, it is also provided:

[0072] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, no further description is given here.

[0073] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0074] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0075] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0076] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0077] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in the present embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0078] The superpixel segmentation method and system based on attention mechanism and convolutional neural network provided in the above embodiment can be implemented and have broad application prospects.

[0079] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A superpixel segmentation method based on attention mechanism and convolutional neural network, characterized in that: include: Obtain an image to be segmented by superpixel; Inputting the image into a pre-trained superpixel segmentation model to obtain a predicted superpixel association map, and determining a superpixel segmentation result of the image based on the superpixel association map; The superpixel segmentation model adopts an encoder-decoder design, wherein the encoder includes several convolutional layers, and the image generates feature maps of different scales through convolutional layers of different levels in the encoder; the decoder includes several deconvolutional layers, and the feature maps generated by convolutional layers of different levels in the encoder are transmitted to the deconvolutional layers of the corresponding levels in the decoder through jump connections; at the same time, an attention module is set before the input of each deconvolutional layer in the decoder; The attention module adopts the SE-Net attention module, which includes a squeeze operation and an excitation operation. The squeeze operation generates a channel descriptor by aggregating feature maps in the spatial dimension; the excitation operation adopts a gating mechanism, takes the embedding of the global distribution generated by the channel descriptor as input, and obtains a set of modulation weights for each channel; The method of determining the image superpixel segmentation result based on the superpixel association map is specifically as follows: a predicted superpixel association map is obtained through the superpixel segmentation model, wherein the superpixel association map determines the probability of each pixel being assigned to a different grid unit based on a soft association map instead of the actual hard assignment of pixels; and a superpixel segmentation result is obtained by assigning each pixel to the grid unit with the highest probability.

2. A superpixel segmentation method based on attention mechanism and convolutional neural network as claimed in claim 1, characterized in that: The loss function used in the training process of the superpixel segmentation model includes two parts. The first part is used to combine pixels with similar attributes; the second part is used to enforce the superpixels to remain compact in space.

3. A superpixel segmentation method based on attention mechanism and convolutional neural network as claimed in claim 2, characterized in that: The loss function is specifically expressed as follows: Where p is a pixel in the image, p' is a pixel in the reconstructed image, f(p) is the feature of the pixel, f'(p) is the feature of the pixel in the reconstructed image, dist() represents the difference between the reconstructed feature and the original feature and the difference between the reconstructed position and the original position, m is the weight to balance these two items, and s is the sampling interval of the superpixel.

4. A superpixel segmentation system based on attention mechanism and convolutional neural network, characterized in that: include: A data acquisition unit, which is used to acquire an image to be subjected to superpixel segmentation; A superpixel segmentation unit, which is used to input the image into a pre-trained superpixel segmentation model to obtain a predicted superpixel association map, and determine the image superpixel segmentation result based on the superpixel association map; The superpixel segmentation model adopts an encoder-decoder design, wherein the encoder includes several convolutional layers, and the image generates feature maps of different scales through convolutional layers of different levels in the encoder; the decoder includes several deconvolutional layers, and the feature maps generated by convolutional layers of different levels in the encoder are transmitted to the deconvolutional layers of the corresponding levels in the decoder through jump connections; at the same time, an attention module is set before the input of each deconvolutional layer in the decoder; The attention module adopts the SE-Net attention module, which includes a squeeze operation and an excitation operation. The squeeze operation generates a channel descriptor by aggregating feature maps in the spatial dimension; the excitation operation adopts a gating mechanism, takes the embedding of the global distribution generated by the channel descriptor as input, and obtains a set of modulation weights for each channel; The method of determining the image superpixel segmentation result based on the superpixel association map is specifically as follows: a predicted superpixel association map is obtained through the superpixel segmentation model, wherein the superpixel association map determines the probability of each pixel being assigned to a different grid unit based on a soft association map instead of the actual hard assignment of pixels; and a superpixel segmentation result is obtained by assigning each pixel to the grid unit with the highest probability.

5. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, characterized in that: When the processor executes the program, it implements a superpixel segmentation method based on an attention mechanism and a convolutional neural network as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a superpixel segmentation method based on an attention mechanism and a convolutional neural network is implemented as described in any one of claims 1 to 3.

Images