Unstructured environment significance semantic segmentation method and system

Abstract

The invention provides an unstructured environment significance semantic segmentation method and system, and belongs to the technical field of image semantic segmentation. According to the method, in the original feature extraction stage, more ghost feature maps are generated through low-cost linear transformation by utilizing the existing feature maps, so that the calculation efficiency of the network is improved; affinity channel attention and affinity space attention are introduced, so that feature expressions of different channels are enhanced, key information of different positions of the space is extracted, and performance is improved under the condition that network complexity is not increased. According to the method, features output by the network are subjected to difference expansion, pixels with low probability are inhibited, pixels with high probability are enhanced, a probability value is used for representing a gray level or haze concentration value, similar dust or haze probability values are mapped to a more reasonable gray level interval, fine differences of dust or haze concentration are enhanced, and segmentation precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image semantic segmentation, and in particular relates to a method and system for saliency semantic segmentation in an unstructured environment. Background Art

[0002] Image semantic segmentation is an important research direction in the field of computer vision. Traditional semantic segmentation methods based on deep learning mainly rely on stacked convolutional layers for feature extraction. Although this feature extraction method is simple to establish a model, the convolutional layer usually only adopts a simple superposition or splicing method, and does not consider the weight of the features, making it difficult for the model to learn accurate image detail features. When dealing with shape changes or small targets, especially dust and haze in unstructured environments, important features will be lost, resulting in reduced segmentation accuracy.

[0003] Therefore, it is necessary to provide a saliency semantic segmentation method and system for unstructured environments to solve the above problems. Summary of the Invention

[0004] The present invention provides a method and system for saliency semantic segmentation in unstructured environments. By introducing an affinity attention mechanism into the semantic segmentation network, the generated segmentation map can effectively reflect the concentration differences of dust and haze in the unstructured environment, thereby improving the segmentation accuracy and the generalization ability of the model, thereby solving at least one technical problem involved in the background technology.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows: A saliency semantic segmentation method for an unstructured environment comprises the following steps: Step S1, obtaining an unstructured environment image to be segmented, and generating an original feature map of the unstructured environment image using ghost convolution; Step S2: Affinity spatial attention and affinity channel attention are used to learn the importance of different positions and channels in the original feature map, respectively, to generate a feature map that takes affinity attention into account. Step S3: Map the feature map considering affinity attention to the category space, generate the category probability distribution of each pixel, and obtain the segmentation result; In step S4, the segmentation result is subjected to Fourier transform, bandpass filtering and inverse Fourier transform in sequence for denoising, and a difference magnification mapping function is introduced to magnify the image grayscale difference in the segmentation result.

[0006] As a preferred improvement, the process of generating the original feature map specifically includes the following steps: Step S11: Perform convolution operation on the unstructured environment image to extract features and generate a size of Feature map , where 、 、 Represents feature maps Number of channels, height, and width; Step S12: feature map Perform ghost convolution to generate feature maps ; Step S13: feature map Perform a convolution operation to generate a feature map ; Step S14: The feature map With feature map The size of the splicing along the channel dimension is Original feature map ; Step S15: convert the original feature map Perform batch normalization and use hardswish function activation.

[0007] As a preferred improvement, the affinity spatial attention processing process specifically includes the following steps: Step S211: the original feature map Perform global maximum pooling and global average pooling on the channel dimension respectively, and get two sizes is the channel attention vector 、 ; Step S212: the original feature map Division is a feature unit and is flattened and reshaped into a feature map ;in, , represents the set of real numbers; Step S213: The feature map After two different conversion functions and Perform conversion to generate feature maps and , where the conversion function and They all consist of a convolution layer with a convolution kernel of 1, batch normalization, and ReLU activation function; Step S214: The feature map and feature maps Perform matrix multiplication to obtain the affinity matrix ; Step S215: Affinity matrix Reshape into rows and columns Two relationship feature graphs and ; Step S216: channel attention vector 、 And the relationship feature graph and Splicing in the channel direction to obtain the feature map ; Step S217: The feature map Through two conversion functions and Extract global information and generate size The affinity relationship space attention weight matrix ,in, It consists of a convolution layer with a convolution kernel of 1, a batch normalization layer, and a ReLU activation function. It consists of a convolution layer with a convolution kernel of 1, a batch normalization layer, and a Sigmoid activation function.

[0008] As a preferred improvement, the affinity channel attention processing process specifically includes the following steps: Step S221: the original feature map Perform global maximum pooling and global average pooling on the channel dimension respectively, and you can get two sizes of Channel attention vector 、 ; Step S222: Divide the feature map into feature units and flattened to reshape into a feature map ; Step S223: The feature map After two different conversion functions and Convert to feature map and , conversion function and They all consist of a convolution layer with a convolution kernel of 1, batch normalization, and ReLU activation function; Step S224: The feature map and Perform matrix multiplication to obtain the affinity matrix ; Step S225: Affinity matrix Reshape into rows and columns Two relationship feature graphs and ; Step S226: channel attention vector 、 and relationship feature graph and Splicing in the channel direction to obtain a new feature map ; Step S227: The feature map Through the conversion function and Extract global information and generate size The affinity relationship space attention weight matrix .

[0009] As a preferred improvement, the feature map generation process considering affinity attention specifically includes the following steps: Step S231: The original feature map Attention weights in affinity relationship space Multiply element by element along the channel direction to get the feature map ; Step S232: then convert the feature map Attention weights of affinity channels Multiply elements along the spatial position to obtain the feature map ; Step S233, finally the original feature map With feature map Add element by element to get the feature map considering affinity attention .

[0010] As a preferred improvement, step S3 specifically includes the following process: using 1×1 convolution to adjust the feature map The number of channels is set to be equal to the number of categories, and then the Sigmoid function is used to compress the output value of each pixel to [0,1], indicating the probability of belonging to dust or haze.

[0011] As a preferred improvement, the Fourier transform process is as follows: Where, Indicates at frequency point The complex-valued Fourier coefficients at , Represents the vertical spatial frequency component index in the frequency domain, Represents the horizontal spatial frequency component index in the frequency domain; Represents spatial domain coordinates The intensity of the pixel at represents the row index of the pixel in the spatial domain, Represents the column index of the pixel in the spatial domain; represents an imaginary unit; The bandpass filter is expressed as: Where, Indicates the bandpass filter at the frequency point The response value at Indicates frequency point Euclidean distance to the frequency center point; represents the center frequency; Indicates the frequency bandwidth; The inverse Fourier transform is expressed as: As a preferred improvement, the difference magnification mapping function is expressed as: , Where, Indicates intermediate parameters; Represents the difference magnification mapping relationship function; represents the amplification factor; Indicates the maximum value of the grayscale pixel value of the image, which is 255; represents the inverse Fourier transform operator; represents the Fourier transform operator; Represents the ceiling function.

[0012] A system for performing the above-mentioned method for saliency semantic segmentation in an unstructured environment comprises: A semantic segmentation network is obtained by replacing the feature extraction module in the traditional Unet network with a ghost affinity attention module. The ghost affinity attention module includes a ghost convolution layer, an affinity attention layer, and an output layer connected in sequence, wherein: The ghost convolution layer is used to obtain the unstructured environment image to be segmented, and generate the original feature map of the unstructured environment image using ghost convolution; The affinity attention layer is used to use affinity spatial attention and affinity channel attention to respectively learn the importance of different positions and different channels in the original feature map, and generate a feature map that takes affinity attention into account; The output layer is used to map the feature map considering affinity attention to the category space, generate the category probability distribution of each pixel, and obtain the segmentation result; The difference magnification module is used to perform denoising on the segmentation results by sequentially subjecting them to Fourier transform, bandpass filtering and inverse Fourier transform, and introduce a difference magnification mapping function to magnify the image grayscale difference in the segmentation results.

[0013] As a preferred improvement, the semantic segmentation network includes 6 layers of encoders and 5 layers of decoders. Each layer of encoder and decoder uses the ghost affinity attention module as the basic module for feature extraction. The encoder layer extracts the abstract features of the input image by downsampling, and the decoder reconstructs the details of the image by upsampling, and outputs a saliency map. The saliency map uses the predicted probability value as the dust concentration value to reflect the concentration difference of the dust; the encoder and the decoder are skipped to pass the local features of the encoder directly to the decoder to compensate for the loss of spatial information in downsampling; each decoder of the semantic segmentation network outputs a segmentation result, and the output results of all decoders are weighted and summed to obtain the final segmentation result map.

[0014] The beneficial effects of the present invention are: (1) In the original feature extraction stage, the existing feature maps are used to generate more ghost feature maps through low-cost linear transformation, thereby improving the computational efficiency of the network; (2) Introducing affinity channel attention and affinity spatial attention to enhance the feature expression of different channels and extract key information at different locations in space, thereby improving performance without increasing network complexity; (3) The features output by the network are expanded by using the difference, which suppresses the pixels with lower probability and enhances the pixels with higher probability. The grayscale or haze concentration value is represented by the probability value, and the similar dust or haze probability values are mapped to a more reasonable grayscale range, which enhances the subtle differences in dust or haze concentration and improves the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which: Figure 1 This is a framework diagram of the semantic segmentation network in Example 2; Figure 2 This is a framework diagram of the ghost affinity attention module in Example 2; Figure 3 Flowchart of the affinity spatial attention processing process in the present invention; Figure 4 Flowchart of the affinity channel attention processing process in the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0017] Example 1 This embodiment provides a method for saliency semantic segmentation in an unstructured environment, comprising the following steps: Step S1: obtain an unstructured environment image to be segmented, and use ghost convolution to generate an original feature map of the unstructured environment image.

[0018] The unstructured environment image is acquired using existing technologies in the art, such as collecting it from existing image data, or photographing it through a camera or other camera equipment.

[0019] The generation process of the original feature map specifically includes the following steps: Step S11: Perform convolution operation on the unstructured environment image to extract features and generate a size of Feature map , where 、 、 Represents feature maps Number of channels, height, and width; Step S12: feature map Perform ghost convolution to generate feature maps ; Step S13: feature map Perform a convolution operation to generate a feature map ; Step S14: The feature map With feature map The size of the splicing along the channel dimension is Original feature map ; Step S15: convert the original feature map Perform batch normalization and use hardswish function activation.

[0020] In the original feature extraction stage, the existing feature maps are used to generate more ghost feature maps through low-cost linear transformation, thereby improving the computational efficiency of the network.

[0021] In step S2, affinity spatial attention and affinity channel attention are used to respectively learn the importance of different positions and different channels in the original feature map, and generate a feature map considering affinity attention.

[0022] The processing process of affinity spatial attention includes the following steps: Step S211: the original feature map Perform global maximum pooling and global average pooling on the channel dimension respectively, and get two sizes is the channel attention vector 、 ; Step S212: the original feature map Division is a feature unit and is flattened and reshaped into a feature map ;in, , represents the set of real numbers; Step S213: The feature map After two different conversion functions and Perform conversion to generate feature maps and , where the conversion function and They all consist of a convolution layer with a convolution kernel of 1, batch normalization, and ReLU activation function; Step S214: The feature map and feature maps Perform matrix multiplication to obtain the affinity matrix ; Step S215: Affinity matrix Reshape into rows and columns Two relationship feature graphs and ; Step S216: channel attention vector 、 And the relationship feature graph and Splicing in the channel direction to obtain the feature map ; Step S217: The feature map Through two conversion functions and Extract global information and generate size The affinity relationship space attention weight matrix ,in, It consists of a convolution layer with a convolution kernel of 1, a batch normalization layer, and a ReLU activation function. It consists of a convolution layer with a convolution kernel of 1, a batch normalization layer, and a Sigmoid activation function.

[0023] Common methods for learning feature vector attention weights are divided into local attention and global attention. For local attention, a feature unit is calculated by other local feature units through a common conversion function to form a weight. This strategy ignores global information. For global attention, a feature unit is learned by all feature units together to form an attention weight. However, this strategy is inefficient and consumes a lot of computing resources when there are many feature units. This application considers the affinity between feature units, explores global feature information and performs knowledge mining, and integrates the features themselves and the relationship between features to jointly learn attention, which can better extract the features of dust and haze areas.

[0024] The process of affinity channel attention includes the following steps: Step S221: the original feature map Perform global maximum pooling and global average pooling on the channel dimension respectively, and you can get two sizes of Channel attention vector 、 ; Step S222: Divide the feature map into feature units and flattened to reshape into a feature map ; Step S223: The feature map After two different conversion functions and Convert to feature map and , conversion function and They all consist of a convolution layer with a convolution kernel of 1, batch normalization, and ReLU activation function; Step S224: The feature map and Perform matrix multiplication to obtain the affinity matrix ; Step S225: Affinity matrix Reshape into rows and columns Two relationship feature graphs and ; Step S226: channel attention vector 、 and relationship feature graph and Splicing in the channel direction to obtain a new feature map ; Step S227: The feature map Through the conversion function and Extract global information and generate size The affinity relationship space attention weight matrix .

[0025] The feature map generation process considering affinity attention specifically includes the following steps: Step S231: The original feature map Attention weights in affinity relationship space Multiply element by element along the channel direction to get the feature map ; Step S232: then convert the feature map Attention weights of affinity channels Multiply elements along the spatial position to obtain the feature map ; Step S233, finally the original feature map With feature map Add element by element to get the feature map considering affinity attention .

[0026] In step S3, the feature map considering affinity attention is mapped to the category space, and the category probability distribution of each pixel is generated to obtain the segmentation result.

[0027] Step S3 specifically includes the following process: Use 1×1 convolution to adjust the feature map The number of channels is set to be equal to the number of categories, and then the Sigmoid function is used to compress the output value of each pixel to [0,1], indicating the probability of belonging to dust or haze.

[0028] In step S4, the segmentation result is subjected to Fourier transform, bandpass filtering and inverse Fourier transform in sequence for denoising, and a difference magnification mapping function is introduced to magnify the image grayscale difference in the segmentation result.

[0029] The Fourier transform process is as follows: Where, Indicates at frequency point The complex-valued Fourier coefficients at , Represents the vertical spatial frequency component index in the frequency domain, Represents the horizontal spatial frequency component index in the frequency domain; Represents spatial domain coordinates The intensity of the pixel at represents the row index of the pixel in the spatial domain, Represents the column index of the pixel in the spatial domain; represents an imaginary unit; The bandpass filter is expressed as: Where, Indicates the bandpass filter at the frequency point The response value at Indicates frequency point Euclidean distance to the frequency center point; represents the center frequency; Indicates the frequency bandwidth; The inverse Fourier transform is expressed as: In order to further expand the image grayscale difference in the segmentation result, this application defines a difference expansion mapping function, which is expressed as: , Where, Indicates intermediate parameters; Represents the difference magnification mapping relationship function; represents the amplification factor; Indicates the maximum value of the grayscale pixel value of the image, which is 255; represents the inverse Fourier transform operator; represents the Fourier transform operator; Represents the ceiling function.

[0030] Difference magnification mapping function middle, The larger the value, the more significant the difference in areas with high dust density, but at the same time, the grayscale value in areas with low dust density will be lower, achieving differentiated processing.

[0031] The final segmentation result map is obtained by weighting the segmentation maps after the above image processing.

[0032] Example 2 like Figure 1-Figure 4 As shown, this embodiment provides a system for executing the method for saliency semantic segmentation in an unstructured environment of embodiment 1, including: A semantic segmentation network is obtained by replacing the feature extraction module in the traditional Unet network with a ghost affinity attention module. The ghost affinity attention module includes a ghost convolution layer, an affinity attention layer, and an output layer connected in sequence, wherein: The ghost convolution layer is used to obtain the unstructured environment image to be segmented, and generate the original feature map of the unstructured environment image using ghost convolution; The affinity attention layer is used to use affinity spatial attention and affinity channel attention to respectively learn the importance of different positions and different channels in the original feature map, and generate a feature map that takes affinity attention into account; The output layer is used to map the feature map considering affinity attention to the category space, generate the category probability distribution of each pixel, and obtain the segmentation result; The difference magnification module is used to perform denoising on the segmentation results by sequentially performing Fourier transform, bandpass filtering and inverse Fourier transform, thereby magnifying the image grayscale difference in the segmentation results.

[0033] The semantic segmentation network retains the nested U-shaped structure of the Unet network, consisting of a six-layer encoder and a five-layer decoder. Each encoder and decoder layer uses the ghost affinity attention module as the basic module for feature extraction. The encoder layer extracts abstract features of the input image through downsampling, while the decoder reconstructs image details through upsampling, outputting a saliency map that uses predicted probability values as dust concentration values to reflect dust concentration differences. Skip connections between the encoder and decoder directly transmit local features from the encoder to the decoder, compensating for the loss of spatial information during downsampling.

[0034] Each decoder of the semantic segmentation network outputs a segmentation result, and the output results of all decoders are weighted and summed to obtain the final segmentation result map.

[0035] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. A method for salient semantic segmentation in unstructured environments, characterized by: The steps include: Step S1, obtaining an unstructured environment image to be segmented, and generating an original feature map of the unstructured environment image using ghost convolution; Step S2: Affinity spatial attention and affinity channel attention are used to learn the importance of different positions and channels in the original feature map, respectively, to generate a feature map that takes affinity attention into account. Step S3: Map the feature map considering affinity attention to the category space, generate the category probability distribution of each pixel, and obtain the segmentation result; In step S4, the segmentation result is subjected to Fourier transform, bandpass filtering and inverse Fourier transform in sequence for denoising, and a difference magnification mapping function is introduced to magnify the image grayscale difference in the segmentation result.

2. The method for saliency semantic segmentation in an unstructured environment according to claim 1, wherein: The generation process of the original feature map specifically includes the following steps: Step S11: Perform convolution operation on the unstructured environment image to extract features and generate a size of Feature map , where 、 、 Represents feature maps Number of channels, height, and width; Step S12: feature map Perform ghost convolution to generate feature maps ; Step S13: feature map Perform a convolution operation to generate a feature map ; Step S14: The feature map With feature map The size of the splicing along the channel dimension is Original feature map ; Step S15: convert the original feature map Perform batch normalization and use hardswish function activation.

3. The method for saliency semantic segmentation in an unstructured environment according to claim 2, wherein: The processing process of affinity spatial attention includes the following steps: Step S211: the original feature map Perform global maximum pooling and global average pooling on the channel dimension respectively, and get two sizes is the channel attention vector 、 ; Step S212: the original feature map Division is a feature unit and is flattened and reshaped into a feature map ;in, , represents the set of real numbers; Step S213: The feature map After two different conversion functions and Perform conversion to generate feature maps and , where the conversion function and They all consist of a convolution layer with a convolution kernel of 1, batch normalization, and ReLU activation function; Step S214: The feature map and feature maps Perform matrix multiplication to obtain the affinity matrix ; Step S215: Affinity matrix Reshape into rows and columns Two relationship feature graphs and ; Step S216: channel attention vector 、 And the relationship feature graph and Splicing in the channel direction to obtain the feature map ; Step S217: The feature map Through two conversion functions and Extract global information and generate size The affinity relationship space attention weight matrix ,in, It consists of a convolution layer with a convolution kernel of 1, a batch normalization layer, and a ReLU activation function. It consists of a convolution layer with a convolution kernel of 1, a batch normalization layer, and a Sigmoid activation function.

4. The method for saliency semantic segmentation in an unstructured environment according to claim 3, wherein: The process of affinity channel attention includes the following steps: Step S221: the original feature map Perform global maximum pooling and global average pooling on the channel dimension respectively, and you can get two sizes of Channel attention vector 、 ; Step S222: Divide the feature map into feature units and flattened to reshape into a feature map ; Step S223: The feature map After two different conversion functions and Convert to feature map and , conversion function and They all consist of a convolution layer with a convolution kernel of 1, batch normalization, and ReLU activation function; Step S224: The feature map and Perform matrix multiplication to obtain the affinity matrix ; Step S225: Affinity matrix Reshape into rows and columns Two relationship feature graphs and ; Step S226: channel attention vector 、 and relationship feature graph and Splicing in the channel direction to obtain a new feature map ; Step S227: The feature map Through the conversion function and Extract global information and generate size The affinity relationship space attention weight matrix .

5. The method for saliency semantic segmentation in an unstructured environment according to claim 4, wherein: The feature map generation process considering affinity attention specifically includes the following steps: Step S231: The original feature map Attention weights in affinity relationship space Multiply element by element along the channel direction to get the feature map ; Step S232: then convert the feature map Attention weights of affinity channels Multiply elements along the spatial position to obtain the feature map ; Step S233, finally the original feature map With feature map Add element by element to get the feature map considering affinity attention .

6. The method for saliency semantic segmentation in an unstructured environment according to claim 5, wherein: Step S3 specifically includes the following process: Use 1×1 convolution to adjust the feature map The number of channels is set to be equal to the number of categories, and then the Sigmoid function is used to compress the output value of each pixel to [0,1], indicating the probability of belonging to dust or haze.

7. The method for saliency semantic segmentation in an unstructured environment according to claim 6, wherein: The Fourier transform process is as follows: Where, Indicates at frequency point The complex-valued Fourier coefficients at , Represents the vertical spatial frequency component index in the frequency domain, Represents the horizontal spatial frequency component index in the frequency domain; Represents spatial domain coordinates The intensity of the pixel at represents the row index of the pixel in the spatial domain, Represents the column index of the pixel in the spatial domain; represents an imaginary unit; The bandpass filter is expressed as: Where, Indicates the bandpass filter at the frequency point The response value at Indicates frequency point Euclidean distance to the frequency center point; represents the center frequency; Indicates the frequency bandwidth; The inverse Fourier transform is expressed as: 。 8. The method for saliency semantic segmentation in an unstructured environment according to claim 7, wherein: The difference magnification mapping function is expressed as: ， Where, Indicates intermediate parameters; Represents the difference magnification mapping relationship function; represents the amplification factor; Indicates the maximum value of the grayscale pixel value of the image, which is 255; represents the inverse Fourier transform operator; represents the Fourier transform operator; Represents the ceiling function.

9. A system for executing the method for salient semantic segmentation of an unstructured environment according to any one of claims 1 to 8, characterized in that: include: A semantic segmentation network is obtained by replacing the feature extraction module in the traditional Unet network with a ghost affinity attention module. The ghost affinity attention module includes a ghost convolution layer, an affinity attention layer, and an output layer connected in sequence, wherein: The ghost convolution layer is used to obtain the unstructured environment image to be segmented, and generate the original feature map of the unstructured environment image using ghost convolution; The affinity attention layer is used to use affinity spatial attention and affinity channel attention to respectively learn the importance of different positions and different channels in the original feature map, and generate a feature map that takes affinity attention into account; The output layer is used to map the feature map considering affinity attention to the category space, generate the category probability distribution of each pixel, and obtain the segmentation result; The difference magnification module is used to perform denoising on the segmentation results by sequentially subjecting them to Fourier transform, bandpass filtering and inverse Fourier transform, and introduce a difference magnification mapping function to magnify the image grayscale difference in the segmentation results.

10. The system according to claim 9, characterized in that The semantic segmentation network includes a 6-layer encoder and a 5-layer decoder. Each layer of encoder and decoder uses the ghost affinity attention module as the basic module for feature extraction. The encoder layer extracts abstract features of the input image through downsampling, and the decoder reconstructs the details of the image through upsampling, outputting a saliency map. The saliency map uses the predicted probability value as the dust concentration value to reflect the concentration difference of dust. The encoder and decoder are skipped to directly transfer the local features of the encoder to the decoder to compensate for the loss of spatial information in downsampling. Each decoder in the semantic segmentation network outputs a segmentation result, and the output results of all decoders are weighted and summed to obtain the final segmentation result map.

Images