A novel feature up-sampling method for semantic segmentation

Abstract

The application provides a novel feature up-sampling method for semantic segmentation, which improves the resolution of an image by combining pixel reorganization Pixel Shuffle with an attention mechanism, prevents a chessboard effect, and avoids a 0 filling operation on unknown pixel points compared with a previous traditional up-sampling module; secondly, the attention mechanism is introduced to combine global information with local information, obtain a feature map with a larger receptive field and more detailed information, increase effective feature information, and realize the function of effectively improving the segmentation performance of the network. The application has universality, eliminates the limitations of previous methods by simply replacing all up-sampling modules, obtains more representative feature mappings, obtains an enlarged image with good features, makes the restored image obtain more feature information, and is beneficial to improving the segmentation result of the network model.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to a novel feature upsampling method for semantic segmentation. Background Technology

[0002] Image semantic segmentation is gradually becoming a research hotspot for computer vision and machine learning researchers. Numerous applications require accurate and efficient segmentation mechanisms, such as autonomous driving, indoor navigation, and virtual / augmented reality systems. This need aligns with the goals of deep learning in machine vision, including semantic segmentation and scene understanding. With the advent of deep learning, many computer vision problems, including semantic segmentation, have begun to be solved using deep architectures, achieving significant results; among them, the application of Convolutional Neural Networks (CNNs) has greatly improved segmentation accuracy. With the progress of deep learning research, commonly used segmentation models can be broadly categorized into two architectures: multi-path parallel architecture and encoding / decoding architecture.

[0003] In multi-parallel architectures, semantic information from deep, coarse network layers is combined with surface information from shallow, fine network layers to generate accurate segmentation. Common multi-parallel architectures include FCN, Fast-SCNN, and PSPNet. In encoder-decoder architectures, the encoder uses convolutional and pooling layers to shrink the feature map size, making it a lower-dimensional representation. The decoder receives this representation and performs upsampling through transposed convolutions to restore the spatial dimension. Semantic segmentation not only needs discriminative power at the pixel level but also a mechanism to project the discriminative features learned by the encoder at different stages back to the pixel space (i.e., map back to the original image size). Different architectures employ different mechanisms (skip connections, pyramid pooling, etc.) as part of the decoding mechanism. Common encoder-decoder architectures include U-Net, LinkNet, and Deepabv3+.

[0004] In both architectures described above, downsampling is performed first to shrink the feature map and reduce computational cost, followed by upsampling for image restoration; upsampling plays a crucial role in this process. Currently, three common upsampling methods are interpolation, transposed convolution, and unpooling. Interpolation is the simplest method for image restoration, including nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. Interpolation-based upsampling methods predict and estimate unknown points based on known pixels, but this method does not allow the enlarged image to contain more information; instead, it amplifies noise, increases computational cost, and blurs the results. Convolution only reduces or keeps the input size unchanged, while transposed convolution is used to increase the input size, for example, to refine coarse feature maps, and has specific applications in FCN models. However, transposed convolution also has a significant drawback: the checkerboard effect can occur, hindering segmentation performance. Unpooling is based on the traditional downsampling pooling method. During pooling, the input values ​​are placed back into their original positions according to their indices, resulting in a sparse matrix. Then, convolutional layers (DeconvNet uses transposed convolution, SegNet uses convolution) are used to refine the coarse feature map. However, it cannot reconstruct all the original data from the pooling result, leading to information loss. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a novel feature upsampling method for semantic segmentation, which can improve the segmentation performance of the network.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a novel feature upsampling method for semantic segmentation, comprising the following steps:

[0007] S1: Upsample the features of the original image using the Pixel Shuffle method, and restore the size of the original image by image magnification;

[0008] S2: Enhance the features obtained by image magnification using the channel attention mechanism to increase the connection between different channels of the image;

[0009] S3: Perform pixel addition on the results obtained from image magnification and feature enhancement to obtain the final output result.

[0010] According to the above scheme, the specific steps in step S1 are as follows:

[0011] S11: The original image with C input channels and C×H×W feature size;

[0012] S12: Let the magnification factor be r when upsampling the original image. Using Pixel Shuffle, the low-resolution C×H×W feature map with C channels is convolved to obtain a feature map with 2r channels. 2 The feature map of C has the same size as the input feature map.

[0013] S13: Select r 2 Pixels at the same position in the channel feature map are filled into the corresponding r×r pixels in the high-resolution image in a fixed manner; the same rule is followed to fill the small grids divided by each low-resolution pixel, resulting in a channel number of 2r. 2 The feature map of C is recombined among multiple channels to obtain a high-resolution 2C×rH×rW upsampled feature map with 2C channels, which preserves the relative spatial location information in the original image.

[0014] According to the above scheme, the specific steps in step S2 are as follows:

[0015] The ECA module is used to enhance the features of the image. One-dimensional convolution is used to achieve local cross-channel interaction without dimensionality reduction. The channel attention mechanism is used to change the weights between different channels to correct the features, retain valuable features and remove worthless features, thereby enhancing the feature representation ability.

[0016] Global average pooling (GAP) is used to compress global spatial information into channel descriptors, reducing network parameters and preventing overfitting; let X a (i, j) represents the a-th channel X a The value at position (i, j), where G is the global average pooling function, and F is the value at position (i, j). a The input channel attention module's original feature map is used; a global average pooling (GAP) operation is performed on the input 2C×H×W feature map to obtain a 2C×1×1 matrix; channel attention operations are then performed on the 2C×1×1 matrix to compress the C×H×W feature map into a C×1×1 channel descriptor G. a :

[0017]

[0018] Obtain matrix G a The weights are then passed through two convolutional layers, a ReLU layer, and a sigmoid function to obtain the attention weight matrix F for different channels. a :

[0019] P a =Sigymoid(Conv(ReLU(Comv(G a ))));

[0020] P aThe calibrated attention feature map is obtained by multiplying the original input with the original input.

[0021] According to the above scheme, the specific steps in step S3 are as follows:

[0022] Since the feature maps output after image magnification and feature enhancement each have 2C channels, a dual-branch structure is used to process the features to reduce the amount of computation.

[0023] The outputs of steps S2 and S3 are reshaped to reconstruct two feature maps of the same channel size: the feature map with 2C channels obtained from image magnification is reshaped into two feature maps F with the same number of channels. ai (i = 1, 2); The feature map with 2C channels obtained by feature enhancement is reshaped into two matrices P of the same size. ai (i = 1, 2):

[0024] F ai (i=12) and P ai (i = 1, 2) are multiplied by their corresponding output weights to obtain features F of the same size for both branches. bi (i = 1, 2):

[0025]

[0026] The feature F of the two branches bi Adding them together, we get the final feature output Fb with C channels:

[0027] F b =F b1 +F b2 .

[0028] A computer storage medium storing a computer program executable by a computer processor, the computer program performing a novel feature upsampling method for semantic segmentation.

[0029] The beneficial effects of this invention are as follows:

[0030] 1. The present invention provides a novel feature upsampling method for semantic segmentation, which improves image resolution by combining pixel reorganization (PixelShuffle) with an attention mechanism. Compared with the original traditional upsampling module, it prevents the checkerboard effect and avoids padding zeros on unknown pixels. Secondly, by introducing an attention mechanism, global and local information are combined to obtain feature maps with a larger receptive field and more detailed information, thereby increasing effective feature information and effectively improving the segmentation performance of the network.

[0031] 2. The novel general feature upsampling architecture proposed in this invention effectively alleviates common difficulties in existing upsampling operations, such as information loss caused by interpolation and checkerboard effect caused by transposed convolution, resulting in a general upsampling module with superior performance.

[0032] 3. This invention is universal. By simply replacing all the upsampling modules, it eliminates the limitations of previous methods and obtains more representative feature maps. It also obtains magnified images with good features, so that the restored images have more feature information, which is beneficial to improving the segmentation results of the network model. Attached Figure Description

[0033] Figure 1 This is a functional block diagram of an embodiment of the present invention.

[0034] Figure 2 This is a flowchart of an embodiment of the present invention.

[0035] Figure 3 This is a diagram illustrating the pixel recombination process of the image magnification section in an embodiment of the present invention. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0037] See Figure 1 This invention discloses a novel feature upsampling method for semantic segmentation, comprising two steps: image magnification and feature enhancement. The method utilizes the different characteristics of these two parts to obtain an image with good features. Then, the features obtained from the two steps are multiplied and added to obtain the final output result. The method includes the following steps:

[0038] (1) Upsample the input features using Pixel Shuffle; input the image to the image magnification section to restore the image size;

[0039] See Figure 2 The two parts (within the dashed boxes) consist of a pixel shuffle operation and an attention mechanism, respectively. The original C×H×W input feature map needs to be processed by an image magnification part to obtain a 2C×rH×rW output feature map (where r is the magnification factor of the image during upsampling). The main implementation part is the pixel shuffle operation. The specific structure diagram of the pixel shuffle operation is shown below. Figure 3 As shown.

[0040] After the image is input, an image enlargement section is first performed to resize the image. To avoid the checkerboard effect and information loss in transposed convolution, this section uses a pixel shuffle operation to obtain a high-resolution image through convolution. The main function of pixel shuffle is to obtain a high-resolution feature map from a low-resolution feature map through convolution and multi-channel recombination.

[0041] The specific process of the Pixel Shuffle operation is as follows: Figure 3 As shown. First, the input image or feature map of H×W×C with C channels is convolved to obtain a 2r channel image. 2 The feature map of C is such that the size of the obtained feature map is the same as the size of the input feature map, and r is the magnification factor of the image during upsampling. Secondly, the main function of Pixel Shuffle is to transform the obtained 2r... 2 The feature maps of channel C are recombined into a new upsampled feature map of size rH×rW×2C. The implementation process involves selecting r... 2 Pixels at the same position in the channel feature map are filled into the corresponding r×r pixels in the high-resolution image in a fixed manner. This process is repeated until each small grid divided by the low-resolution pixel is filled, completing the reconstruction. During this process, the model can adjust the r×r channel weights to continuously optimize the generated result. This magnification method allows the feature map to contain spatial information, thus preserving the relative spatial location information in the original image.

[0042] (2) The features obtained in the previous part are enhanced using the channel attention mechanism; after restoration, the feature map is enhanced to increase the connection between different channels of the image.

[0043] After the Pixel Shuffle section, the original low-resolution image is enlarged to a high-resolution image of rH×rW, and the resulting feature map has 2C channels. Since directly performing convolution to reduce the number of channels is detrimental to the interaction of channel information, the image is further processed in the feature enhancement section. This section enhances feature representation by introducing channel attention. The attention mechanism can correct features, retaining valuable features and discarding worthless ones. A commonly used channel attention module is the SE module, which adaptively recalibrates the channel-related feature responses by explicitly modeling the interdependencies between channels. However, this invention uses the ECA module for image feature enhancement, which is a non-dimensionality-reducing local cross-channel interaction strategy; compared to fully connected layers, it can be effectively implemented through one-dimensional convolution. The specific implementation process is as follows.

[0044] In the feature enhancement section, channel attention is used to change the weights between different channels, that is, the importance (weights) of each channel of the input image is calculated by the network to enhance the feature representation capability. This part first compresses the spatial dimension of the input feature map. This step can be achieved through global average pooling (GAP) and global max pooling (GMP). This invention uses global average pooling to compress global spatial information into the channel descriptors, which reduces network parameters and prevents overfitting. The input 2C×H×W feature map is first subjected to global average pooling to obtain a 2C×1×1 matrix, and then channel attention is applied; that is, the C×H×W feature map is compressed into a C×1×1 channel descriptor, as shown in the following formula:

[0045]

[0046] Among them, X a (i, j) represents the a-th channel X a The value at position (i, j), where G is the global average pooling function, and F is the value at position (i, j). a This is the original feature mapping for the input channel attention module.

[0047] For the pooled matrix G a To obtain the weights, the fully connected layers in the SE module are replaced with convolutional layers. The attention weight matrix P for different channels is obtained through two convolutional layers, a ReLU function, and a sigmoid function. a :

[0048] P a =Sigmoid(Conv(ReLU(Conv(G a )))),

[0049] Then, multiply the input with the original input to obtain the calibrated attention feature map.

[0050] (3) Perform pixel addition on the results obtained from these two parts to obtain the final output;

[0051] Since both image upscaling and feature enhancement outputs have 2C channels, a dual-branch structure is used to process the features, reducing computational cost. As shown in the figure, both outputs are reshaped to reconstruct two feature maps of the same size and channel count. These are then multiplied by their respective output weights to obtain two features of the same size. Finally, the feature maps from both branches are added to obtain the module's final output. The dual-branch structure reduces computation, and fusing them improves the correlation between features and enhances their semantic representation.

[0052] Since the feature map obtained from the magnified portion of the image has 2C channels, it is reshaped into two feature maps F with the same number of channels. ai (i = 1, 2); and the weights obtained from the feature enhancement part are also divided into two matrices of the same size P in the same way. ai (i = 1, 2); Perform a weighted operation (channel multiplication) on the obtained weights and features to obtain the output F of each branch after multiplying by the weights. bi (i = 1, 2):

[0053]

[0054] Finally, the features F of the two branches obtained are analyzed. bi Summing yields the final feature output F of the module with C channels. b :

[0055] F b =F b1 +F b2 .

[0056] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A novel feature upsampling method for semantic segmentation, characterized in that: Includes the following steps: S1: Upsample the features of the original image using the Pixel Shuffle method, and restore the size of the original image by image magnification; S2: Enhance the features obtained through image magnification using a channel attention mechanism to increase the connection between different channels of the image; the specific steps are as follows: The ECA module is used to enhance the features of the image. One-dimensional convolution is used to achieve local cross-channel interaction without dimensionality reduction. The channel attention mechanism is used to change the weights between different channels to correct the features, retain valuable features and remove worthless features, thereby enhancing the feature representation ability. Global average pooling (GAP) is used to compress global spatial information into channel descriptors, reducing network parameters and preventing overfitting; Let... Indicates the first One channel exist The value of the position, This is the global average pooling function. The original feature mapping for the input channel attention module; for the input The feature map is obtained by performing a global average pooling (GAP) operation. The matrix; for Perform channel attention operations on the matrix, Feature map compression Channel descriptor G a : ； Get matrix G a The weights are then processed through two convolutional layers, a ReLU layer, and a sigmoid function to obtain the attention weight matrices for different channels. : ； Will The calibrated attention feature map is obtained by multiplying the original input with the calibrated attention feature map. S3: Perform pixel addition on the results obtained from image magnification and feature enhancement to obtain the final output result.

2. The novel feature upsampling method for semantic segmentation according to claim 1, characterized in that: The specific steps in step S1 are as follows: S11: Number of input channels is C, feature size is... The original image; S12: Let the magnification factor be the magnification factor when upsampling the original image. r Pixel Shuffle is used to reconstruct low-resolution images with C channels. The feature map is convolved to obtain the number of channels. The feature map, whose size is the same as the size of the input features; S13: Select Pixels at the same position in the channel feature map are filled into the corresponding pixels in the high-resolution image in a fixed manner. Within each pixel; following the same rules, fill in the small squares divided by each low-resolution pixel, resulting in a channel number of... The feature map is obtained by recombination among multiple channels, resulting in a channel number of... high resolution Upsample the feature map to preserve the relative spatial location information in the original image.

3. The novel feature upsampling method for semantic segmentation according to claim 1, characterized in that: The specific steps in step S3 are as follows: Since the feature maps output after image magnification and feature enhancement each have the same number of channels... A dual-branch structure is used to process features to reduce computational load; The outputs of steps S2 and S1 are reshaped to reconstruct two feature maps of the same channel size: the number of channels obtained by image magnification is... Feature map reshape is used to reshape two feature maps with the same number of channels. The number of channels obtained by feature enhancement is The feature map is reshaped into two matrices of the same size. ; Will and Multiply each feature by its corresponding output weight to obtain features of the same size for both branches. : ； Features of the two branches Adding them together yields the final feature output result with channel number C. : 。 4. A computer storage medium, characterized in that: It contains a computer program that can be executed by a computer processor, which performs a novel feature upsampling method for semantic segmentation as described in any one of claims 1 to 3.

Images