An image segmentation method based on multi-scale space adaptive hole convolution

Abstract

The application discloses an image segmentation method based on a multi-scale space adaptive hole convolution, and comprises the following steps: a multi-scale space adaptive hole convolution model is established, which comprises a hole convolution module, a space scale attention module and a feature recombination module, wherein the hole convolution module comprises n parallel hole convolution layers of different scales, is used for performing convolution on an input feature map to generate n groups of first feature maps with the same size, the space scale attention module is used for synchronously generating m*n channel attention maps according to the input feature map, and the feature recombination module is used for recombining the corresponding first feature maps and the attention maps to obtain a final output result; and the image segmentation network model is used for performing image segmentation on the final output result to obtain a segmentation result. According to the method, the feature semantics extracted by the convolution is more explicit, unnecessary interference information can be removed, and ambiguity is reduced, so that a more accurate segmentation result can be obtained.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an image segmentation method based on multi-scale spatial adaptive dilated convolution. Background Technology

[0002] In everyday life, images often exhibit a wide variety of object sizes due to varying distances and shapes. This multi-scale irregularity presents a significant challenge for convolution. Dilated convolution, when extracting information from different spatial locations, can only use a single-size kernel, employing the same sampling method for both large and small objects. While a large dilation rate allows for accurate feature capture when sampling large objects like trees, it becomes problematic when sampling small objects like cows. The sampling region of dilated convolution is prone to incorporating background information and creating ambiguity, significantly interfering with semantic understanding and object classification. This scale uniformity in convolution is a major cause of inaccurate segmentation.

[0003] The usual approach to solving this type of problem is to use convolutional kernels of multiple scales to extract information, such as using dilated convolutions with multiple parallel branches or using different dilation rates at different layers of the network to achieve the network's ability to extract multi-scale information. However, these methods all have a problem: this multi-scale extraction capability can only be applied holistically and cannot be adjusted according to different targets. The aforementioned conventional methods simply use convolutional kernels of different sizes for the entire feature map in a general way, without considering that objects at different spatial locations on the feature map have different requirements for the sampling area. Summary of the Invention

[0004] The purpose of this invention is to address the above-mentioned problems by proposing an image segmentation method based on multi-scale spatial adaptive dilated convolution, which makes the semantic features extracted by convolution clearer, eliminates unnecessary interference information and reduces ambiguity, and helps to obtain more accurate segmentation results.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] This invention proposes an image segmentation method based on multi-scale spatial adaptive dilated convolution, comprising the following steps:

[0007] S1. Establish a multi-scale spatial adaptive dilated convolution model, which includes a dilated convolution module, a spatial scale attention module, and a feature reconstruction module, wherein:

[0008] The dilated convolution module includes n parallel dilated convolution layers of different scales, used to convolve the input feature map to generate n sets of first feature maps of the same size;

[0009] The spatial scale attention module consists of a convolutional layer, a BN layer, and a sigmoid activation function connected in sequence, used to synchronously generate an attention map with m*n channels based on the input feature map;

[0010] The feature reorganization module is used to reorganize the corresponding first feature map and attention map to obtain the final output result. The process is as follows:

[0011] The attention maps of m*n channels are divided into m groups. The n attention maps in each group correspond one-to-one with the n dilated convolutional layers in the dilated convolutional module, representing a combination of dilation rates.

[0012] Each attention map is multiplied by the first feature map of the corresponding channel in the dilated convolution module in sequence, and the results of the multiplication are added and fused to obtain the corresponding output feature map for each group.

[0013] All output feature maps are concatenated to obtain the final output result;

[0014] S2. Use an image segmentation network model to segment the final output result obtained by stitching to obtain the segmentation result.

[0015] Preferably, the porosity of the parallel dilated convolutional layers of n scales is determined by a set of hyperparameters r = [r1, r2, r3, ..., r n [Control, where n represents the number of dilated convolutional layers.]

[0016] Preferably, the number of dilated convolutional layers n = 4, and the hyperparameter r = [1, 3, 5, 7].

[0017] Preferably, the spatial size of the attention map is the same as the spatial size of the first feature map.

[0018] Preferably, the image segmentation network model is one of the DeepLabv3 network model, PSPNet network model, or UltraNet network model.

[0019] Preferably, the output feature maps are stitched together by using the Concat operation along the channels.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] This method employs a multi-scale spatial adaptive dilated convolution model for feature extraction. Compared to traditional dilated convolution, it uses a reweighted approach after stacking multiple dilated convolutions. By applying multiple dilated convolutions of different scales to the same spatial location, and using a spatial scale attention module to weight the feature maps generated by each dilated convolution, the weighted feature maps are finally recombined and fused. This weighted fusion method allows the network to adaptively select a reasonable combination of dilation rates based on the size and shape of the object at the current location, thereby selecting a sampling region of appropriate size. This can be adjusted according to different targets, overcoming the limitations of fixed receptive fields in traditional dilated convolution when extracting features and the single scale defect of dilated convolution used in traditional image segmentation. Furthermore, the attention mechanism of the spatial scale attention module removes unnecessary interference information and reduces ambiguity, making the semantics of the features extracted by convolution clearer, which is conducive to obtaining more accurate segmentation results and has strong generalization ability. Attached Figure Description

[0022] Figure 1 This is a flowchart of the image segmentation method based on multi-scale spatial adaptive dilated convolution of the present invention;

[0023] Figure 2 This is a schematic diagram of the multi-scale spatial adaptive dilated convolution model structure of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application.

[0026] like Figure 1-2 As shown, an image segmentation method based on multi-scale spatial adaptive dilated convolution includes the following steps:

[0027] S1. Establish a multi-scale spatial adaptive dilated convolution model, which includes a dilated convolution module, a spatial scale attention module, and a feature reconstruction module, wherein:

[0028] The dilated convolution module includes n parallel dilated convolution layers of different scales, used to convolve the input feature map to generate n sets of first feature maps of the same size;

[0029] The spatial scale attention module consists of a convolutional layer, a BN layer, and a sigmoid activation function connected in sequence, used to synchronously generate an attention map with m*n channels based on the input feature map;

[0030] The feature reorganization module is used to reorganize the corresponding first feature map and attention map to obtain the final output result. The process is as follows:

[0031] The attention maps of m*n channels are divided into m groups. The n attention maps in each group correspond one-to-one with the n dilated convolutional layers in the dilated convolutional module, representing a combination of dilation rates.

[0032] Each attention map is multiplied by the first feature map of the corresponding channel in the dilated convolution module in sequence, and the results of the multiplication are added and fused to obtain the corresponding output feature map for each group.

[0033] All output feature maps are concatenated to obtain the final output result.

[0034] In one embodiment, the void ratio of the n-scale parallel dilated convolutional layers is determined by a set of hyperparameters r = [r1, r2, r3, ..., r n [Control, where n represents the number of dilated convolutional layers.]

[0035] In one embodiment, the number of dilated convolutional layers n = 4, and the hyperparameter r = [1, 3, 5, 7]. It should be noted that the specific values ​​can be adjusted according to actual needs, and the number of channels in the spatial scale attention module should be adjusted accordingly.

[0036] In one embodiment, the spatial size of the attention map is the same as the spatial size of the first feature map.

[0037] In one embodiment, the output feature maps are stitched together by using a Concat operation along the channels.

[0038] S2. Use an image segmentation network model to segment the final output result obtained by stitching to obtain the segmentation result.

[0039] In one embodiment, the image segmentation network model is one of the DeepLabv3 network model, the PSPNet network model, and the UPERNet network model.

[0040] The following detailed description is provided through specific embodiments.

[0041] In this embodiment, as Figure 2As shown, the multi-scale spatial adaptive dilated convolution model includes a dilated convolution module, a spatial scale attention block (SSA), and a feature recombination block.

[0042] The dilated convolution module consists of multiple parallel dilated convolutional layers, whose dilation rate is determined by a set of hyperparameters r = [r1, r2, r3, ..., r n The number of dilated convolutional layers is controlled by ], where n represents the number of dilated convolutional layers. Taking n=4 and r=[1,3,5,7] as an example, the dilated convolution module uses four parallel dilated convolutional layers with dilation rates of 1, 3, 5 and 7 to convolve the input feature map, generating four sets of first feature maps with the same size.

[0043] Furthermore, the input feature map also passes through a spatial scale attention module, which sequentially passes through a convolutional layer, a batch normalization (BN) layer, and a sigmoid activation function to obtain attention maps with the same spatial size as the first feature map of the dilated convolutional module, containing 4*n = 16 channels. These 16 channels are divided into 4 groups, each containing 4 channels, corresponding to the 4 dilated convolutional layers in the dilated convolutional module. The size of each attention map is the same as the size of the first feature map of the dilated convolutional module. Each pixel on the attention map represents the importance of its corresponding scale branch at the current spatial location. Each group of n channels represents a combination of dilation rates. The dilated convolutional module provides 4 combinations to reduce information loss due to attention map misjudgments and increase the diversity of output features, which is beneficial for guiding subsequent recombination and fusion through the attention map.

[0044] The feature reconstruction module receives the first feature maps output from the four dilated convolutional layers of the dilated convolution module and the attention maps generated by the spatial scale attention module, and reconstructs the feature maps according to certain rules. Specifically, the 16-channel attention maps generated by the spatial scale attention module are divided into four groups, and the attention maps within each group are sequentially associated with the first feature maps output from the corresponding dilated convolutional layers. Figure 1 Each pair of dot products is then multiplied, and the results are added together to obtain a set of scale-filtered output feature maps. A total of four output maps are generated in the diagram. Finally, these four sets of output feature maps are concatenated to obtain the final output result.

[0045] The image segmentation result can be obtained by using the deeplabv3 network model to segment the final output obtained from the concatenation. In practical applications, the image to be segmented is used as the input feature map to a multi-scale spatial adaptive dilated convolutional model for feature extraction to obtain the final output result. The final output result is then used for image segmentation by the deeplabv3 network model to obtain the segmentation result.

[0046] This method uses a multi-scale spatial adaptive dilated convolution model for feature extraction. It optimizes the structure of traditional dilated convolution by adopting a multi-diffusion rate design to enrich the sampling information and make the semantics of the features extracted by convolution clearer. It also uses an attention mechanism to remove unnecessary interference information and reduce ambiguity. Compared to traditional dilated convolution, this method employs a reweighted approach by stacking multiple dilated convolutions. Multiple dilated convolutions of different scales are applied to the same spatial location, and a spatial scale attention module weights the feature maps generated by each dilated convolution. Finally, the weighted feature maps are recombined and fused. This weighted fusion method allows the network to adaptively select a reasonable combination of dilation rates based on the size and shape of the object at the current location, thereby choosing an appropriate sampling region. This can be adjusted according to different targets, overcoming the limitations of fixed receptive fields in traditional dilated convolutions and the single-scale dilated convolution used in traditional image segmentation. Furthermore, the attention mechanism of the spatial scale attention module removes unnecessary interference information and reduces ambiguity, making the semantics of the features extracted by the convolution more explicit, which is conducive to obtaining more accurate segmentation results and has strong generalization ability.

[0047] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0048] The embodiments described above are merely specific and detailed examples of the embodiments described in this application, and should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the appended claims.

Claims

1. An image segmentation method based on multi-scale spatial adaptive dilated convolution, characterized in that: The image segmentation method based on multi-scale spatial adaptive dilated convolution includes the following steps: S1. Establish a multi-scale spatial adaptive dilated convolution model, which includes a dilated convolution module, a spatial scale attention module, and a feature reconstruction module, wherein: The dilated convolution module includes n parallel dilated convolutional layers of different scales, used to convolve the input feature map to generate n sets of first feature maps of the same size; the dilation rate of the n parallel dilated convolutional layers of different scales is determined by a set of hyperparameters. Control involves applying multiple dilated convolutional layers of different scales to the same spatial location in the input feature map, where n represents the number of dilated convolutional layers; The spatial scale attention module includes a convolutional layer, a BN layer, and a sigmoid activation function connected in sequence, used to synchronously generate an attention map with m*n channels based on the input feature map; the spatial size of the attention map is the same as the spatial size of the first feature map; the spatial scale attention module is used to weight the first feature map generated by each dilated convolutional layer; The feature recombination module is used to recombine and fuse multiple weighted first feature maps and attention maps to obtain the final output result, as follows: m The attention maps of the n channels are divided into m groups. The n attention maps in each group correspond one-to-one with the n dilated convolutional layers in the dilated convolutional module, representing a combination of dilation rates. Each attention map is multiplied by the first feature map of the corresponding channel in the dilated convolution module in sequence, and the results of the multiplication are added and fused to obtain the corresponding output feature map for each group. All output feature maps are concatenated to obtain the final output result. This weighted fusion method enables the multi-scale spatial adaptive dilated convolution model to learn to adaptively select a reasonable combination of dilation rates based on the size and shape of the object at the current spatial location. S2. Use an image segmentation network model to segment the final output result obtained by stitching to obtain the segmentation result.

2. The image segmentation method based on multi-scale spatial adaptive dilated convolution as described in claim 1, characterized in that: The number of dilated convolutional layers n=4, the hyperparameters .

3. The image segmentation method based on multi-scale spatial adaptive dilated convolution as described in claim 1, characterized in that: The image segmentation network model is one of the following: DeepLabv3 network model, PSPNet network model, or UltraNet network model.

4. The image segmentation method based on multi-scale spatial adaptive dilated convolution as described in claim 1, characterized in that: The output feature maps are stitched together by using the Concat operation along the channels.

Images