Image segmentation method and device

Abstract

The present invention provides an image segmentation method and device, which includes: obtaining a first feature map of a target image; applying a pyramid pooling module to extract second feature maps of different sizes of receptive fields of the first feature map at multiple resolutions, adjusting the feature map channels and resolutions after performing maximum pooling and average pooling on each second feature map, and obtaining each global context information feature map; fusing each global context information feature map with the first feature map to obtain a fused feature map, applying a channel attention fusion module to adjust the features of different sizes of receptive fields corresponding to the fused feature map, and then performing upsampling to obtain a final feature map; determining the image information category of each pixel position in the final feature map, and generating an image segmentation result map based on the image information category of each pixel position in the final feature map. By applying this method, more accurate global context information can be extracted by performing average pooling and maximum pooling on the feature map, thereby improving the accuracy of image segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image segmentation method and device. Background Art

[0002] Image segmentation is the process of dividing an image into several specific sub-regions with unique properties. This simplifies or even changes the image representation, adds labels to image pixels, and pixels with the same label have certain common visual characteristics, which is helpful for locating objects and boundaries in the image. At present, the image segmentation method is mainly based on deep learning. Compared with traditional image segmentation methods, deep learning image segmentation methods have significantly improved accuracy and speed.

[0003] Although the existing image segmentation methods can perform image segmentation learning through a large amount of data, they are unable to effectively extract the context information of the image during the specific segmentation process, resulting in low image segmentation accuracy. Summary of the invention

[0004] In view of this, the present invention provides an image segmentation method, through which more accurate global context information can be extracted by performing average pooling and maximum pooling on the feature map, thereby improving the accuracy of image segmentation.

[0005] The present invention also provides an image segmentation device to ensure the implementation and application of the above method in practice.

[0006] An image segmentation method, comprising:

[0007] Obtaining a first feature map of a target image to be segmented;

[0008] Applying a preset pyramid pooling module, extracting second feature maps of different sizes of receptive fields of the first feature map at multiple resolutions, and adjusting the feature map channels and resolutions after performing maximum pooling and average pooling on each of the second feature maps to obtain each global context information feature map;

[0009] Fusing each of the global context information feature maps with the first feature map to obtain a fused feature map, and applying a pre-trained channel attention fusion module to adjust features of receptive fields of different sizes corresponding to the fused feature map, and recalibrating the fused feature map;

[0010] Applying a preset upsampling prediction module to perform an upsampling operation on the adjusted fusion feature map to obtain a final feature map;

[0011] The image information category of each pixel position in the final feature map is determined, and an image segmentation result map is generated based on the image information category of each pixel position in the final feature map.

[0012] In the above method, optionally, the step of obtaining a first feature map of the target image to be segmented includes:

[0013] The target image is input into a feature extraction network ResNet-101, and the ResNet-101 is controlled to perform multi-layer convolution operations and pooling operations on the target image to obtain a first feature map of the target image output by the ResNet-101.

[0014] In the above method, optionally, extracting a second feature map of a first feature map with a receptive field of different sizes at multiple resolutions includes:

[0015] The pyramid pooling module is controlled to apply pooling kernels of 1×1, 2×2, 3×3 and 6×6 to downsample the first feature map, so as to generate second feature maps with resolutions of 1×1, 2×2, 3×3 and 6×6 respectively.

[0016] In the above method, optionally, the step of adjusting the feature map channel and resolution after performing maximum pooling and average pooling on each of the second feature maps to obtain each global context information feature map includes:

[0017] Performing maximum pooling and average pooling on each of the second feature maps to obtain a first branch map and a second branch map corresponding to each of the second feature maps;

[0018] Splicing the first branch graph and the second branch graph corresponding to each of the second feature graphs to obtain a spliced ​​graph corresponding to each of the second feature graphs, wherein the feature graph channel of the spliced ​​graph is twice the feature graph channel of the corresponding second feature graph;

[0019] Performing a convolution operation on each of the spliced ​​graphs so that the number of feature graph channels of each of the spliced ​​graphs is consistent with the number of feature graph channels of the first feature graph;

[0020] Apply a preset bilinear interpolation method to adjust the resolution of the mosaic of the reduced feature map channels according to the resolution of the first feature map to obtain a resolution equal to the first feature map. Figure 1 A consistent global context information feature map.

[0021] In the above method, optionally, the application of a pre-trained channel attention fusion module to adjust the features of receptive fields of different sizes corresponding to the fused feature map and recalibrate the fused feature map includes:

[0022] Inputting the fused feature map into the channel attention fusion module, controlling the channel attention fusion module to compress the fused feature map, and obtaining a first processed feature map;

[0023] Controlling the channel attention fusion module to call the fully connected layer according to a preset multiple, reducing the number of feature map channels of the first processing feature map, and obtaining a second processing feature map;

[0024] Control the channel attention fusion module to pass the second processed feature map through a ReLu activation layer, and then through another fully connected layer to restore the number of channels of the second processed feature map to be consistent with the number of channels of the fused feature map, so as to obtain a third processed feature map;

[0025] The channel attention fusion module is controlled to normalize the channel attention weights in the third processing feature map based on the resolution of the fused feature map through a preset Sigmoid function and weight them to the features of each feature map channel, thereby realizing the recalibration of the fused feature map.

[0026] An image segmentation device, comprising:

[0027] An acquisition unit, used for acquiring a first feature map of a target image to be segmented;

[0028] A pooling unit is used to apply a preset pyramid pooling module to extract second feature maps of different sizes of receptive fields of the first feature map at multiple resolutions, and adjust the feature map channel and resolution after performing maximum pooling and average pooling on each of the second feature maps to obtain each global context information feature map;

[0029] An adjustment unit, used to fuse each of the global context information feature maps with the first feature map to obtain a fused feature map, and apply a pre-trained channel attention fusion module to adjust the features of receptive fields of different sizes corresponding to the fused feature map, and identify the feature map channel to which each pixel position of the fused feature map belongs;

[0030] An upsampling unit, used to apply a preset upsampling prediction module to perform an upsampling operation on the adjusted fusion feature map to obtain a final feature map;

[0031] The segmentation unit is used to determine the image information category of each pixel position in the final feature map, and generate an image segmentation result map based on the image information category of each pixel position in the final feature map.

[0032] In the above device, optionally, the acquisition unit includes:

[0033] The first operation subunit is used to input the target image into the feature extraction network ResNet-101, control the ResNet-101 to perform multi-layer convolution operations and pooling operations on the target image, and obtain a first feature map of the target image output by the ResNet-101.

[0034] In the above device, optionally, the pooling unit comprises:

[0035] A generating subunit is used to control the pyramid pooling module to apply pooling kernels of 1×1, 2×2, 3×3 and 6×6 to downsample the first feature map respectively, so as to generate second feature maps with resolutions of 1×1, 2×2, 3×3 and 6×6 respectively.

[0036] In the above device, optionally, the pooling unit comprises:

[0037] A pooling subunit, used to perform maximum pooling and average pooling on each of the second feature maps to obtain a first branch map and a second branch map corresponding to each of the second feature maps;

[0038] A splicing subunit, used for splicing the first branch graph and the second branch graph corresponding to each of the second feature graphs to obtain a spliced ​​graph corresponding to each of the second feature graphs, wherein the feature graph channel of the spliced ​​graph is twice the feature graph channel of the corresponding second feature graph;

[0039] A second operation subunit is used to perform a convolution operation on each of the spliced ​​graphs so that the number of feature graph channels of each of the spliced ​​graphs is consistent with the number of feature graph channels of the first feature graph;

[0040] The adjustment subunit is used to adjust the resolution of the mosaic of the reduced feature map channels according to the resolution of the first feature map by applying a preset bilinear interpolation method to obtain a resolution equal to that of the first feature map. Figure 1 A consistent global context information feature map.

[0041] In the above device, optionally, the adjustment unit comprises:

[0042] A first control subunit is used to input the fused feature map into the channel attention fusion module, control the channel attention fusion module to compress the fused feature map, and obtain a first processed feature map;

[0043] A second control subunit is used to control the channel attention fusion module to call the fully connected layer to reduce the number of feature map channels of the first processing feature map according to a preset multiple to obtain a second processing feature map;

[0044] A third control subunit is used to control the channel attention fusion module to restore the number of channels of the second processed feature map to be consistent with the number of channels of the fused feature map after passing the second processed feature map through a ReLu activation layer and then through another fully connected layer, so as to obtain a third processed feature map;

[0045] The fourth control subunit is used to control the channel attention fusion module to normalize the channel attention weights in the third processing feature map based on the resolution of the fused feature map through a preset Sigmoid function and weight them to the features of each feature map channel, thereby realizing the recalibration of the fused feature map.

[0046] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned image segmentation method.

[0047] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be used by one or more processors to execute the above-mentioned image segmentation method.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] The present invention provides an image segmentation method, comprising: obtaining a first feature map of a target image to be segmented; applying a preset pyramid pooling module to extract second feature maps of different sizes of receptive fields of the first feature map at multiple resolutions, and adjusting the feature map channels and resolutions after performing maximum pooling and average pooling on each of the second feature maps to obtain each global context information feature map; fusing each of the global context information feature maps with the first feature map to obtain a fused feature map, and applying a pre-trained channel attention fusion module to adjust the features of the receptive fields of different sizes corresponding to the fused feature map, and recalibrating the fused feature map; applying a pre-set upsampling prediction module to perform an upsampling operation on the adjusted fused feature map to obtain a final feature map; determining the image information category of each pixel position in the final feature map, and generating an image segmentation result map based on the image information category of each pixel position in the final feature map. By applying the method provided by the present invention, more accurate global context information can be extracted by performing average pooling and maximum pooling on the feature map, thereby improving the accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0051] Figure 1 A method flow chart of an image segmentation method provided by an embodiment of the present invention;

[0052] Figure 2An execution structure diagram of an image segmentation method provided by an embodiment of the present invention;

[0053] Figure 3 A flowchart of another method of an image segmentation method provided by an embodiment of the present invention;

[0054] Figure 4 A device structure diagram of an image segmentation device provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0058] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.

[0059] The embodiment of the present invention provides an image segmentation method, which can be applied to various system platforms, and its execution subject can be a processor of a computer terminal or various mobile devices. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:

[0060] S101: Obtain a first feature map of a target image to be segmented.

[0061] In the present invention, a first feature map of the target image is generated by extracting image features of the target image.

[0062] S102: Apply a preset pyramid pooling module to extract second feature maps of different sizes of receptive fields of the first feature map at multiple resolutions, and adjust the feature map channels and resolutions after performing maximum pooling and average pooling on each of the second feature maps to obtain each global context information feature map.

[0063] It should be noted that the Pyramid Pooling Module (PPM) is composed of a group of pooling modules of different scales. By pooling from more to less, the receptive field is effectively increased and the utilization efficiency of global information is increased. The receptive field refers to the size of the area where a pixel position on the second feature map is mapped back to the first feature map.

[0064] The resolutions are 1×1, 2×2, 3×3, and 6×6. After performing maximum pooling and average pooling on the feature map, two feature maps with the same number of channels and resolution as the feature map are obtained. After the two feature maps are fused, the feature map channels and resolution are adjusted. The obtained global context information feature map is consistent with the feature map channels and resolution of the first feature map.

[0065] Specifically, the pyramid pooling module is controlled to apply pooling kernels of 1×1, 2×2, 3×3 and 6×6 to downsample the first feature map, and generate second feature maps with resolutions of 1×1, 2×2, 3×3 and 6×6, respectively.

[0066] Therefore, in the pyramid pooling module, the depth convolution feature map is downsampled using 1×1, 2×2, 3×3 and 6×6 pooling kernels to generate feature maps of 1×1, 2×2, 3×3 and 6×6 resolution sizes respectively.

[0067] Furthermore, after performing maximum pooling and average pooling on each second feature map, adjusting the feature map channel and resolution, and obtaining each global context information feature map may include:

[0068] Performing maximum pooling and average pooling on each second feature map to obtain a first branch map and a second branch map corresponding to each second feature map;

[0069] Splicing the first branch graph and the second branch graph corresponding to each second feature graph to obtain a spliced ​​graph corresponding to each second feature graph;

[0070] Performing a convolution operation on each of the stitched images so that the number of feature map channels of each stitched image is consistent with the number of feature map channels of the first feature map;

[0071] Apply the preset bilinear interpolation method to adjust the resolution of the mosaic of the reduced feature map channels according to the resolution of the first feature map to obtain a resolution equal to that of the first feature map. Figure 1 A consistent global context information feature map.

[0072] Among them, 1×1, 2×2, 3×3 and 6×6 pooling kernels are used for downsampling to generate feature maps of 1×1, 2×2, 3×3 and 6×6 resolution sizes. Each pooled feature map is downsampled using the maximum pooling and average pooling methods to obtain the first branch map after maximum pooling and the second branch map after average pooling. The first branch map and the second branch map obtained by the two pooling methods are spliced. The number of feature map channels of the original feature map is N, and the number of feature map channels of each spliced ​​map after pooling is 2N. Then, a 1x1 convolution operation is used for each spliced ​​map to reduce the number of spliced ​​map channels to make it consistent with the number of feature map channels of the first feature map. Subsequently, a bilinear interpolation method is used for each branch feature map to restore the resolution of the feature map to the size before pooling.

[0073] It should be noted that after the first feature map is pooled to generate the second feature map, only the resolution of the image is changed, but the number of feature map channels does not change. When the second feature map is subjected to maximum pooling and average pooling to generate two branch maps and then spliced, the number of feature map channels is twice the original.

[0074] It should also be noted that the global context information can be used to represent the relationship between each pixel position in the image and its surrounding pixel positions after the target image has been pooled and concatenated multiple times.

[0075] S103: Fusing each of the global context information feature maps with the first feature map to obtain a fused feature map, and applying a pre-trained channel attention fusion module to adjust features of receptive fields of different sizes corresponding to the fused feature map, and recalibrating the fused feature map.

[0076] Specifically, each global context information feature map is fused with the first feature map to obtain a fused feature map, and the channel attention fusion module is applied to adaptively adjust the features of receptive fields of different sizes to the fused feature map, so that the weights of receptive field features of different sizes in the fused feature map are different, thereby improving the accuracy of the global context information in the fused feature map. Among them, the channel attention fusion module compresses the feature map channels of the fused feature map and restores them to the original number of channels, thereby enhancing the fused feature map.

[0077] Furthermore, the fused feature map is first compressed from the spatial dimension using global average pooling, and then the correlation between channels is modeled using a fully connected layer. Specifically, a fully connected layer is first used to reduce the number of channels, and then after the ReLu activation layer, the number of channels is restored to the original dimension using a fully connected layer. Finally, the generated channel attention weights are normalized by the Sigmoid function, and then weighted to the features of each feature map channel to recalibrate the fused feature map. For example, the fused feature map is a 50×50×16 feature map. After global average pooling, its fragmentation rate is reduced to obtain a 1×1×16 feature map. The number of channels is then reduced by 1 / 16 to obtain a 1×1×1 feature map. After the ReLu activation layer, the number of channels is restored to obtain a 1×1×16 feature map using a fully connected layer. Finally, after normalization, the weight of each feature map channel is between (0 and 1).

[0078] S104: Apply a preset upsampling prediction module to perform an upsampling operation on the adjusted fusion feature map to obtain a final feature map.

[0079] It should be noted that the upsampling prediction module performs an upsampling operation on the adjusted fused feature map according to the resolution of the original target image, so that the resolution of the final feature map is consistent with the resolution of the target image.

[0080] S105: Determine the image information category of each pixel position in the final feature map, and generate an image segmentation result map based on the image information category of each pixel position in the final feature map.

[0081] Among them, the fused feature map after adjusting the receptive fields of different sizes is upsampled to the input image resolution size, and the image information category of each pixel position is predicted. The number of channels of the feature map for prediction is the number of categories + 1. If the pixel value of the channel corresponding to the image information category of a certain pixel position is close to 1, and the pixel values ​​of the channels corresponding to other categories are close to 0, then the category of the pixel position is the image information category corresponding to the channel with a pixel value close to 1.

[0082] It should be noted that an image may contain a variety of image information, such as people, trees, houses, roads, etc. The image information categories at different pixel positions determine the image information corresponding to different areas of the target image, and perform image segmentation on the fused feature map.

[0083] In the method provided by the embodiment of the present invention, when it is necessary to perform image segmentation on the target image, the image features of the target image are extracted, and a first feature map is generated based on the image features. The first feature map is pooled using 1x1, 2x2, 3x3 and 6x6 pooling kernels through the pyramid pooling module to obtain second feature maps of different sizes of receptive fields, and then each second feature map is subjected to maximum pooling and average pooling to obtain a first branch map and a second branch map corresponding to each second feature map, and the first branch map and the second score map corresponding to each second feature map are fused, and the number and resolution of the feature map channels are adjusted so that the number and resolution of the feature map channels are consistent with the first feature map. Figure 1 A global context information feature map is obtained, the first feature map is fused with the global context information feature map, and the features of different sizes of receptive fields corresponding to the fused feature map are adjusted through the channel attention fusion module, and then up-sampled to obtain the final feature map; the image information category of each pixel position in the final feature map is determined, and an image segmentation result map is generated based on the image information category of each pixel position in the final feature map.

[0084] By applying the method provided in the embodiment of the present invention, more accurate global context information can be extracted by performing average pooling and maximum pooling on the feature map, thereby improving the accuracy of image segmentation.

[0085] In the present invention, when it is necessary to segment the target image, it is necessary to extract the features of the target image to generate a first feature map of the target image. Therefore, obtaining the first feature map of the target image to be segmented may specifically include:

[0086] The target image is input into a feature extraction network ResNet-101, and the ResNet-101 is controlled to perform multi-layer convolution operations and pooling operations on the target image to obtain a first feature map of the target image output by the ResNet-101.

[0087] It should be noted that the deep convolutional features, i.e., the first feature map, are generated by ResNet-101. In ResNet-101, convolution operations, pooling operations, and residual modules are mainly used multiple times to reduce the gradient vanishing phenomenon in the deep convolutional feature network, thereby extracting more effective deep convolutional features. After the operation of ResNet-101, the resolution of the first feature map obtained is different from the resolution of the target image. For example, the resolution of the target image is 100×100, and the resolution of the first feature map obtained after feature extraction by ResNet-101 is 50×50.

[0088] refer to Figure 2Based on the method provided by the above embodiment, the present invention provides a deep neural network, which includes ResNet-101, pyramid pooling module, channel attention fusion module and upsampling prediction module. Wherein ResNet-101 is used to characterize the image, obtain the first feature map, and perform maximum pooling and average pooling through the pyramid pooling module, obtain the global context information feature map, fuse the global context information feature map with the first feature map, adjust the fusion feature map through the channel attention fusion module, further enhance the accuracy of the fusion feature map, and finally perform upsampling prediction for image segmentation to obtain the image segmentation result. Wherein, before the actual application of the deep neural network, it is necessary to perform deep training to improve the accuracy of the entire network. In the process of deep training of the deep neural network, each module in the entire deep neural network (for example: ResNet-101, pyramid pooling module, channel attention fusion module and upsampling prediction module) can be uniformly trained at the same time, and each module cooperates with each other in the same training process, and the gradient descent method is used to perform multiple trainings and adjust the module parameters of each module to improve the accuracy of the entire deep neural network. In addition, the training process of the entire deep neural network can also be to train a single module. When the accuracy of a single module reaches the set conditions, it will be applied to the entire deep neural network.

[0089] Therefore, in a deep neural network, no matter whether all modules are trained uniformly or a single module is trained separately, each module needs to perform the training process according to its corresponding training process. For ResNet-101, it is trained to improve the accuracy of image feature extraction; for the pyramid pooling module, it is trained to extract the global context information of the feature map through operations such as maximum pooling and average pooling; for the channel attention fusion module, it is trained to adaptively adjust the features of receptive fields of different sizes, so that the weights of receptive field features of different sizes in the fused feature map are different; for the upsampling prediction module, it is trained to perform image segmentation on the feature map according to the image information category.

[0090] In the embodiment of the present invention, the process of applying the pre-trained channel attention fusion module to adjust the features of the receptive fields of different sizes corresponding to the fused feature map and recalibrating the fused feature map is as follows: Figure 3 As shown, it may specifically include:

[0091] S301: Input the fused feature map into the channel attention fusion module, control the channel attention fusion module to compress the fused feature map, and obtain a first processed feature map.

[0092] In the present invention, compressing the fused feature map may specifically be compressing it into a feature map with a resolution of 1×1, and the number of channels of the feature map remains unchanged during the compression process.

[0093] S302: Control the channel attention fusion module to call the fully connected layer according to a preset multiple, reduce the number of feature map channels of the first processed feature map, and obtain a second processed feature map.

[0094] In the present invention, the resolution of the feature map remains unchanged during the process of reducing the number of channels.

[0095] S303: Control the channel attention fusion module to pass the second processed feature map through a ReLu activation layer, and then through another fully connected layer to restore the number of channels of the second processed feature map to be consistent with the number of channels of the fused feature map, so as to obtain a third processed feature map.

[0096] S304: Control the channel attention fusion module to normalize the channel attention weights in the third processed feature map based on the resolution of the fused feature map through a preset Sigmoid function and weight them to the features of each feature map channel, so as to achieve recalibration of the fused feature map.

[0097] In the present invention, after the generated global context information feature map is fused with the first feature map, the channel attention fusion module is used to adaptively adjust the features of receptive fields of different sizes, so that the weights of receptive field features of different sizes in the fused feature map are different, and the attention weights are learned using the gradient descent method, so that they are adjusted with the accuracy of image segmentation, until the attention weights are adjusted to obtain higher image segmentation accuracy. In the training process of the channel attention fusion module, the fused feature map is first feature compressed from the spatial dimension using global average pooling, and then the correlation between feature map channels is modeled using a fully connected layer. Specifically, a fully connected layer is first used to reduce the number of feature map channels, and then after the ReLu activation layer, the fully connected layer is used to restore the number of feature map channels to the original dimension, and finally the generated feature map channel attention weights are normalized by the Sigmoid function, weighted to the features of each feature map channel, and the fused feature map is recalibrated.

[0098] It should be noted that the channel attention fusion module is trained by the gradient descent method. The gradient descent method is a way to adjust the module parameters in combination with the loss function during the training process of the channel attention fusion module, so that after the channel attention fusion module inputs the feature map, it can adjust the attention weight of the feature map as accurately as possible, thereby improving the accuracy of image segmentation.

[0099] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.

[0100] and Figure 1 Corresponding to the method described above, the embodiment of the present invention further provides an image segmentation device for Figure 1 In the specific implementation of the method, the image segmentation device provided in the embodiment of the present invention can be applied to a computer terminal or various mobile devices, and its structural diagram is as follows Figure 4 As shown, specifically including:

[0101] An acquisition unit 401 is used to acquire a first feature map of a target image to be segmented;

[0102] A pooling unit 402 is used to apply a preset pyramid pooling module to extract second feature maps of different sizes of receptive fields of the first feature map at multiple resolutions, and adjust the feature map channel and resolution after performing maximum pooling and average pooling on each of the second feature maps to obtain each global context information feature map;

[0103] An adjustment unit 403 is used to fuse each of the global context information feature maps with the first feature map to obtain a fused feature map, and use a pre-trained channel attention fusion module to adjust the features of receptive fields of different sizes corresponding to the fused feature map, and identify the feature map channel to which each pixel position of the fused feature map belongs;

[0104] An upsampling unit 404 is used to apply a preset upsampling prediction module to perform an upsampling operation on the adjusted fusion feature map to obtain a final feature map;

[0105] The segmentation unit 405 is used to determine the image information category of each pixel position in the final feature map, and generate an image segmentation result map based on the image information category of each pixel position in the final feature map.

[0106] In the device provided by the embodiment of the present invention, when it is necessary to perform image segmentation on the target image, the image features of the target image are extracted, and a first feature map is generated based on the image features. The first feature map is pooled by using a pyramid pooling module using 1x1, 2x2, 3x3 and 6x6 pooling kernels to obtain second feature maps of different sizes of receptive fields, and then each second feature map is subjected to maximum pooling and average pooling to obtain a first branch map and a second branch map corresponding to each second feature map, and the first branch map and the second score map corresponding to each second feature map are fused, and the number and resolution of the feature map channels are adjusted so that the number and resolution of the feature map channels are consistent with the first feature map. Figure 1 A global context information feature map is obtained, the first feature map is fused with the global context information feature map, and the features of different sizes of receptive fields corresponding to the fused feature map are adjusted through the channel attention fusion module, and then up-sampled to obtain the final feature map; the image information category of each pixel position in the final feature map is determined, and an image segmentation result map is generated based on the image information category of each pixel position in the final feature map.

[0107] By applying the device provided by the embodiment of the present invention, more accurate global context information can be extracted by performing average pooling and maximum pooling on the feature map, thereby improving the accuracy of image segmentation.

[0108] In the device provided by the embodiment of the present invention, the acquisition unit 401 includes:

[0109] The first operation subunit is used to input the target image into the feature extraction network ResNet-101, control the ResNet-101 to perform multi-layer convolution operations and pooling operations on the target image, and obtain a first feature map of the target image output by the ResNet-101.

[0110] In the device provided in the embodiment of the present invention, the pooling unit 402 includes:

[0111] A generating subunit is used to control the pyramid pooling module to apply pooling kernels of 1×1, 2×2, 3×3 and 6×6 to downsample the first feature map respectively, so as to generate second feature maps with resolutions of 1×1, 2×2, 3×3 and 6×6 respectively.

[0112] In the device provided in the embodiment of the present invention, the pooling unit 404 includes:

[0113] A pooling subunit, used to perform maximum pooling and average pooling on each of the second feature maps to obtain a first branch map and a second branch map corresponding to each of the second feature maps;

[0114] A splicing subunit, used for splicing the first branch graph and the second branch graph corresponding to each of the second feature graphs to obtain a spliced ​​graph corresponding to each of the second feature graphs, wherein the feature graph channel of the spliced ​​graph is twice the feature graph channel of the corresponding second feature graph;

[0115] A second operation subunit is used to perform a convolution operation on each of the spliced ​​graphs so that the number of feature graph channels of each of the spliced ​​graphs is consistent with the number of feature graph channels of the first feature graph;

[0116] The adjustment subunit is used to adjust the resolution of the mosaic of the reduced feature map channels according to the resolution of the first feature map by applying a preset bilinear interpolation method to obtain a resolution equal to that of the first feature map. Figure 1 A consistent global context information feature map.

[0117] In the device provided by the embodiment of the present invention, the adjustment unit includes:

[0118] A first control subunit is used to input the fused feature map into the channel attention fusion module, control the channel attention fusion module to compress the fused feature map, and obtain a first processed feature map;

[0119] A second control subunit is used to control the channel attention fusion module to call the fully connected layer to reduce the number of feature map channels of the first processing feature map according to a preset multiple to obtain a second processing feature map;

[0120] A third control subunit is used to control the channel attention fusion module to restore the number of channels of the second processed feature map to be consistent with the number of channels of the fused feature map after passing the second processed feature map through a ReLu activation layer and then through another fully connected layer, so as to obtain a third processed feature map;

[0121] The fourth control subunit is used to control the channel attention fusion module to normalize the channel attention weights in the third processing feature map based on the resolution of the fused feature map through a preset Sigmoid function and weight them to the features of each feature map channel, thereby realizing the recalibration of the fused feature map.

[0122] The specific working process of each unit and sub-unit in the image segmentation device disclosed in the above embodiment of the present invention can refer to the corresponding content in the image segmentation method disclosed in the above embodiment of the present invention, and will not be repeated here.

[0123] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned image segmentation method.

[0124] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Figure 5 As shown, it specifically includes a memory 501 and one or more instructions 502, wherein the one or more instructions 502 are stored in the memory 501 and are configured to be executed by one or more processors 503 to perform the following operations:

[0125] Obtaining a first feature map of a target image to be segmented;

[0126] Applying a preset pyramid pooling module, extracting second feature maps of different sizes of receptive fields of the first feature map at multiple resolutions, and adjusting the feature map channels and resolutions after performing maximum pooling and average pooling on each of the second feature maps to obtain each global context information feature map;

[0127] Fusing each of the global context information feature maps with the first feature map to obtain a fused feature map, and applying a pre-trained channel attention fusion module to adjust features of receptive fields of different sizes corresponding to the fused feature map, and recalibrating the fused feature map;

[0128] Applying a preset upsampling prediction module to perform an upsampling operation on the adjusted fusion feature map to obtain a final feature map;

[0129] The image information category of each pixel position in the final feature map is determined, and an image segmentation result map is generated based on the image information category of each pixel position in the final feature map.

[0130] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0131] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of both.

[0132] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0133] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image segmentation method, characterized in that: include: Obtaining a first feature map of a target image to be segmented; Applying a preset pyramid pooling module, extracting second feature maps of different sizes of receptive fields of the first feature map at multiple resolutions, and adjusting the feature map channels and resolutions after performing maximum pooling and average pooling on each of the second feature maps to obtain each global context information feature map; Fusing each of the global context information feature maps with the first feature map to obtain a fused feature map, and applying a pre-trained channel attention fusion module to adjust features of receptive fields of different sizes corresponding to the fused feature map, and recalibrating the fused feature map; Applying a preset upsampling prediction module to perform an upsampling operation on the adjusted fusion feature map to obtain a final feature map; Determine the image information category of each pixel position in the final feature map, and generate an image segmentation result map based on the image information category of each pixel position in the final feature map; The step of performing maximum pooling and average pooling on each of the second feature maps and adjusting the feature map channel and resolution to obtain each global context information feature map includes: Performing maximum pooling and average pooling on each of the second feature maps to obtain a first branch map and a second branch map corresponding to each of the second feature maps; Splicing the first branch graph and the second branch graph corresponding to each of the second feature graphs to obtain a spliced ​​graph corresponding to each of the second feature graphs, wherein the feature graph channel of the spliced ​​graph is twice the feature graph channel of the corresponding second feature graph; Performing a convolution operation on each of the spliced ​​graphs so that the number of feature graph channels of each of the spliced ​​graphs is consistent with the number of feature graph channels of the first feature graph; Applying a preset bilinear interpolation method to adjust the resolution of the mosaic of the reduced feature map channels according to the resolution of the first feature map, to obtain a global context information feature map having a resolution consistent with that of the first feature map; The applying a pre-trained channel attention fusion module to adjust the features of receptive fields of different sizes corresponding to the fused feature map and recalibrate the fused feature map includes: Inputting the fused feature map into the channel attention fusion module, controlling the channel attention fusion module to compress the fused feature map, and obtaining a first processed feature map; Controlling the channel attention fusion module to call the fully connected layer according to a preset multiple, reducing the number of feature map channels of the first processing feature map, and obtaining a second processing feature map; Control the channel attention fusion module to pass the second processed feature map through a ReLu activation layer, and then through another fully connected layer to restore the number of channels of the second processed feature map to be consistent with the number of channels of the fused feature map, so as to obtain a third processed feature map; The channel attention fusion module is controlled to normalize the channel attention weights in the third processing feature map based on the resolution of the fused feature map through a preset Sigmoid function and weight them to the features of each feature map channel, thereby realizing the recalibration of the fused feature map.

2. The method according to claim 1, characterized in that: The step of obtaining a first feature map of a target image to be segmented comprises: The target image is input into a feature extraction network ResNet-101, and the ResNet-101 is controlled to perform multi-layer convolution operations and pooling operations on the target image to obtain a first feature map of the target image output by the ResNet-101.

3. The method according to claim 1, characterized in that The step of extracting a second feature map of receptive fields of different sizes at multiple resolutions of the first feature map includes: The pyramid pooling module is controlled to apply pooling kernels of 1×1, 2×2, 3×3 and 6×6 to downsample the first feature map, so as to generate second feature maps with resolutions of 1×1, 2×2, 3×3 and 6×6 respectively.

4. An image segmentation device, characterized in that: include: An acquisition unit, used for acquiring a first feature map of a target image to be segmented; A pooling unit is used to apply a preset pyramid pooling module to extract second feature maps of different sizes of receptive fields of the first feature map at multiple resolutions, and adjust the feature map channel and resolution after performing maximum pooling and average pooling on each of the second feature maps to obtain each global context information feature map; An adjustment unit, used to fuse each of the global context information feature maps with the first feature map to obtain a fused feature map, and apply a pre-trained channel attention fusion module to adjust the features of receptive fields of different sizes corresponding to the fused feature map, and identify the feature map channel to which each pixel position of the fused feature map belongs; An upsampling unit, used to apply a preset upsampling prediction module to perform an upsampling operation on the adjusted fusion feature map to obtain a final feature map; A segmentation unit, used to determine the image information category of each pixel position in the final feature map, and generate an image segmentation result map based on the image information category of each pixel position in the final feature map; The pooling unit comprises: A pooling subunit, used to perform maximum pooling and average pooling on each of the second feature maps to obtain a first branch map and a second branch map corresponding to each of the second feature maps; A splicing subunit, used for splicing the first branch graph and the second branch graph corresponding to each of the second feature graphs to obtain a spliced ​​graph corresponding to each of the second feature graphs, wherein the feature graph channel of the spliced ​​graph is twice the feature graph channel of the corresponding second feature graph; A second operation subunit is used to perform a convolution operation on each of the spliced ​​graphs so that the number of feature graph channels of each of the spliced ​​graphs is consistent with the number of feature graph channels of the first feature graph; an adjusting subunit, configured to adjust the resolution of the mosaic of the reduced feature map channels according to the resolution of the first feature map by applying a preset bilinear interpolation method, so as to obtain a global context information feature map having a resolution consistent with that of the first feature map; The adjustment unit comprises: A first control subunit is used to input the fused feature map into the channel attention fusion module, control the channel attention fusion module to compress the fused feature map, and obtain a first processed feature map; A second control subunit is used to control the channel attention fusion module to call the fully connected layer to reduce the number of feature map channels of the first processing feature map according to a preset multiple to obtain a second processing feature map; A third control subunit is used to control the channel attention fusion module to restore the number of channels of the second processed feature map to be consistent with the number of channels of the fused feature map after passing the second processed feature map through a ReLu activation layer and then through another fully connected layer, so as to obtain a third processed feature map; The fourth control subunit is used to control the channel attention fusion module to normalize the channel attention weights in the third processing feature map based on the resolution of the fused feature map through a preset Sigmoid function and weight them to the features of each feature map channel, thereby realizing the recalibration of the fused feature map.

5. The device according to claim 4, characterized in that The acquisition unit comprises: The first operation subunit is used to input the target image into the feature extraction network ResNet-101, control the ResNet-101 to perform multi-layer convolution operations and pooling operations on the target image, and obtain a first feature map of the target image output by the ResNet-101.

6. The device according to claim 5, characterized in that The pooling unit comprises: A generating subunit is used to control the pyramid pooling module to apply pooling kernels of 1×1, 2×2, 3×3 and 6×6 to downsample the first feature map respectively, so as to generate second feature maps with resolutions of 1×1, 2×2, 3×3 and 6×6 respectively.

Images