An image semantic segmentation method and system based on adaptive sampling

Abstract

This invention relates to an image semantic segmentation method and system based on adaptive sampling, belonging to the field of computer vision technology. First, a semantic segmentation network using adaptive sampling is constructed. Then, a semantic high-frequency loss function and a semantic segmentation task loss function are used to train the semantic segmentation deep convolutional network. Finally, the trained adaptive sampling semantic segmentation network is used to perform semantic segmentation on the image, obtaining segmentation results with rich details and accurate edges. This invention utilizes an adaptive sampling strategy to improve the semantic segmentation network's ability to retain key region details during downsampling and optimizes the upsampling results using multi-scale local information during upsampling. This enables the semantic segmentation network to significantly improve its ability to recognize key region details in images with extremely low additional computational cost, resulting in refined, accurate, and highly consistent semantic segmentation results.

Description

Technical Field

[0001] This invention relates to a method and system for semantic segmentation of images, specifically to an image semantic segmentation method and system based on adaptive sampling, belonging to the field of computer vision technology. Background Technology

[0002] Semantic segmentation is a computer vision technique that associates predefined categories of interest with each pixel in an image. It requires the computer to understand the semantic information of the image content and densely determine whether each pixel belongs to a specific category of interest. It can be used to accurately identify the set of pixels that constitute a target of interest. Examples include identifying vehicles, pedestrians, and traffic signals in autonomous driving scenarios, and forests, rivers, and mountains in remote sensing images. Because it requires dense prediction of images, extracting and preserving details and high-frequency information is crucial for final accuracy.

[0003] Sampling refers to the sampling process in deep neural networks used for semantic segmentation, including downsampling and upsampling. Downsampling reduces the spatial size of the feature map in the deep neural network and can be achieved using spatial pooling layers or convolutional layers with strides. Upsampling increases the spatial size of the feature map and can be achieved using bilinear interpolation or transposed convolutional layers.

[0004] In semantic segmentation deep neural networks, a typical approach involves alternating convolutional blocks and downsampling modules to gradually reduce the feature map size, decrease computational complexity, and increase the receptive field of subsequent convolutional blocks. Traditional upsampling and downsampling processes assume that every pixel is equally important and use a spatially uniform sampling strategy. However, in real-world natural images, not every pixel is equally important. For example, pixels in the region containing the target of interest are more important than background pixels and have a greater impact on the final recognition accuracy. A uniform downsampling process can lead to the loss of high-frequency detail information in these important regions. Summary of the Invention

[0005] The purpose of this invention is to address the technical problem that uniform downsampling strategies used in semantic segmentation methods can easily lead to the loss of detailed information in important regions, in order to meet the needs of intelligent image recognition. This invention creatively proposes an image semantic segmentation method and system based on adaptive sampling.

[0006] This invention can adaptively adjust the spatial sampling rate during neural network downsampling, performing dense sampling on important regions and sparse sampling on unimportant regions. This significantly improves the ability of the semantic segmentation model to retain detailed information of important regions during downsampling with minimal additional computation, and fully utilizes multi-scale contextual information during upsampling to achieve high-precision, low-complexity semantic segmentation.

[0007] The present invention is achieved using the following technical solution.

[0008] On the one hand, this invention proposes an image semantic segmentation method based on adaptive sampling, comprising the following steps:

[0009] Step 1: Construct an adaptive sampling semantic segmentation network;

[0010] Step 2: Train the semantic segmentation deep convolutional network using the semantic high-frequency loss function and the semantic segmentation task loss function;

[0011] Step 3: Use the trained adaptive sampling semantic segmentation network to perform semantic segmentation on the image to obtain segmentation results with rich details and accurate edges.

[0012] On the other hand, this invention proposes an image semantic segmentation system based on adaptive sampling, including an adaptive sampling semantic segmentation network module, a semantic high-frequency loss function training module, and a semantic segmentation inference module.

[0013] Beneficial effects

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] 1. This method significantly improves the ability of semantic segmentation deep neural networks to retain important regional details during downsampling.

[0016] 2. This method makes full use of local contextual information to optimize the upsampling process, improve the upsampling precision, and thus improve the accuracy of semantic segmentation.

[0017] 3. This method has extremely low additional computational overhead, and is simple to implement, high-performing, and robust. It is beneficial for achieving low-latency, high-speed semantic segmentation. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention.

[0019] Figure 2 This is a schematic diagram of the adaptive sampling semantic segmentation network structure of the method of the present invention.

[0020] Figure 3 This is a schematic diagram of the adaptive downsampling method described in this invention.

[0021] Figure 4 This is a schematic diagram of the adaptive multi-scale upsampling method described in this invention.

[0022] Figure 5 This is a flowchart of the system of the present invention. Detailed Implementation

[0023] To better illustrate the purpose and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0024] like Figure 1 As shown, an image semantic segmentation method based on adaptive sampling includes the following steps:

[0025] Step 1: Construct a semantic segmentation network using adaptive sampling.

[0026] Semantic segmentation, as a dense prediction task, relies heavily on the high-frequency details preserved in the feature maps of deep neural networks for high prediction accuracy. However, traditional uniform sampling strategies lead to the loss of details in important regions. This invention replaces the traditional uniform sampling strategy with adaptive sampling, thereby significantly preserving key details in important regions.

[0027] Specifically, adaptive sampling comprises two parts: adaptive downsampling and adaptive multi-scale upsampling. By replacing the ordinary uniform downsampling and upsampling in a typical semantic segmentation deep neural network with adaptive downsampling and adaptive multi-scale upsampling, an adaptive sampling semantic segmentation network is constructed. The overall network structure is as follows: Figure 2 As shown. It should be noted that adaptive sampling is applicable to semantic segmentation models (FCN, PSPNet, CCNet, OCNet, PCAA, etc.).

[0028] Specifically, adaptive downsampling, such as Figure 3 As shown. First, generate the attention map S:

[0029] S = Softmax(f att (X)) (1)

[0030] Where X is the input feature map; f att The attention generator can be implemented by a lightweight convolutional subnetwork, such as a single 1×1 convolutional layer, a pyramid spatial pooling layer, etc., for optimizing the pyramid spatial pooling layer.

[0031] Then, adaptive sampling coordinates are generated based on the attention map:

[0032] u = f u (i,j,S),v=f v (i,j,S) (2)

[0033] Where u and v are the coordinates of the adaptive sampling. And constraints:

[0034] f u (0,j,S)=0,f u (1,j,S)=1 (3)

[0035] f v(i,0,S)=0,f v (i,1,S)=1 (4)

[0036] Here, (i,j) are uniform two-dimensional coordinates, which are mapped by the coordinate mapping function f. u f v Mapped to (u,v) adaptive sampling coordinates:

[0037]

[0038]

[0039] Among them, (i ′ ,j ′ Let G be the position of the surrounding pixels and G be a two-dimensional Gaussian function. This coordinate mapping function causes pixels with high attention values ​​to pull the sampling coordinates of their surrounding pixels, resulting in denser sampling coordinates in the high attention value region and achieving non-uniform sampling.

[0040] X N (i,j)=X(u,v) (7)

[0041] Among them, X N (i,j) represents the feature located at coordinates (i,j) after non-uniform sampling. X N The dimension size is the same as the input X.

[0042] Finally, the adaptive downsampling process is completed by performing downsampling:

[0043] X ′ =Downsample(X N (8)

[0044] Among them, X ′ The obtained adaptive downsampled features; Downsample is the downsampling operation, which can be accomplished using pooling layers, convolutional layers with strides, etc.

[0045] The adaptive multi-scale upsampling process is as follows: Figure 4 As shown, firstly, non-uniform upsampling is used to upsample the segmentation result predicted from the adaptive downsampling features back to its original size. Then, Local Pixel Relation Modules (LPRM) are used to mine multi-scale local information and optimize the upsampling result. Specifically, non-uniform upsampling uses Delaunay triangulation and barycentric coordinate interpolation to reconstruct uniform sampling positions from known non-uniform adaptive sampling points. After non-uniform upsampling, the LPRM constructs local pixel relations R to optimize the upsampling result.

[0046]

[0047]

[0048] Among them, X comp For the compressed features, This represents a convolutional layer with a kernel size of k×k and a dilation rate of d, where Ω represents... The constructed local pixel region. Y and Y' represent the original semantic segmentation result and the semantic segmentation result optimized using local pixel relationships, respectively. Y represents the local pixel relationship at pixel (i,j), where p and q represent the offset coordinates of a k×k neighborhood near (i,j). i+p,j+q This represents the segmentation result at (i+p,j+q).

[0049] like Figure 4 As shown, multiple local pixel relation modules can be cascaded together, and subsequent modules can use a larger hole ratio to obtain a larger receptive field, thereby constructing a larger-scale local pixel relation R. ′ :

[0050]

[0051] Where d' = 2d, that is, the receptive field of the subsequent local pixel relation module is twice that of the former, and the optimization process is consistent with Equations 10 and 11, and is carried out iteratively.

[0052] Step 2: Train the semantic segmentation deep convolutional network using the semantic high-frequency loss function and the semantic segmentation task loss function.

[0053] Specifically, the total loss function L total Represented as:

[0054] L total =L seg +λL SHF (12)

[0055] Among them, L seg L represents the loss function for semantic segmentation tasks. SHF Let L represent the semantic high-frequency loss function, and λ represent the loss function balance coefficient. Further, L... seg Let L be the pixel-wise cross-entropy loss function. SHF Specifically, it is expressed as follows:

[0056]

[0057] in, To segment high-frequency regions using real semantic tags As an attention map, it is mapped via the coordinate mapping function f uf v The adaptive sampling coordinates obtained by mapping are:

[0058]

[0059]

[0060] Where Δ represents the Laplacian operator, used to extract the ground truth labels for semantic segmentation. The high-frequency region in the semantic high-frequency region is usually the edge of the semantic object.

[0061] Sampling points When the semantic edges of objects are densely distributed, adaptive sampling can be guided to sample the semantic edges of objects densely, thereby preserving as much edge detail information as possible during the downsampling process, which is beneficial for the model to accurately segment object edges.

[0062] The adaptive sampling semantic segmentation network can be trained and optimized end-to-end using the loss function described above.

[0063] Step 3: Use the trained adaptive sampling semantic segmentation network to perform semantic segmentation on the image, thereby obtaining segmentation results with rich details and accurate edges.

[0064] To achieve the objectives described in this invention, this invention further proposes an image semantic segmentation system based on adaptive sampling, such as... Figure 5 As shown, it includes an adaptive sampling semantic segmentation network construction module 10, a semantic high-frequency region dense sampling learning module 20, and an image semantic segmentation inference module 30.

[0065] The adaptive sampling semantic segmentation network model building module 10 is used to build a semantic segmentation network model using adaptive sampling. This module transforms a regular semantic segmentation network into a semantic segmentation network model using adaptive sampling, improving the model's ability to preserve details during downsampling and its ability to utilize multi-scale local pixel relationships during upsampling.

[0066] The semantic high-frequency region dense sampling learning module 20 is used to guide the adaptive sampling semantic segmentation model to identify key semantic high-frequency regions (i.e. object edge regions of the target of interest) in the image, and to perform dense sampling on such regions, thereby greatly preserving semantic edge details, improving the accuracy of the model in object edge segmentation, and improving the stability and consistency of the segmentation results.

[0067] The image semantic segmentation inference module 30 is responsible for using the trained model to extract image features through adaptive sampling, and obtaining the final image semantic segmentation result.

[0068] The connection relationships between the above modules are as follows:

[0069] The output of the adaptive sampling semantic segmentation network model building module 10 is connected to the input of the semantic high-frequency region dense sampling learning module 20.

[0070] The output of the semantic high-frequency region dense sampling learning module 20 is connected to the input of the image semantic segmentation inference module 30.

Claims

1. An image semantic segmentation method based on adaptive sampling, characterized in that, Includes the following steps: Step 1: Construct an adaptive sampling semantic segmentation network; Adaptive sampling includes adaptive downsampling and adaptive multi-scale upsampling. By replacing ordinary uniform downsampling and upsampling in ordinary semantic segmentation deep neural networks with adaptive downsampling and adaptive multi-scale upsampling, an adaptive sampling semantic segmentation network is constructed. Step 2: Train the semantic segmentation deep convolutional network using the semantic high-frequency loss function and the semantic segmentation task loss function; Wherein, the total loss function L total Represented as: THE total =L seg +λL SHF Among them, L seg L represents the loss function for semantic segmentation tasks. SHF λ represents the semantic high-frequency loss function, and λ represents the loss function balance coefficient. L seg Let L be the pixel-wise cross-entropy loss function. SHF Specifically, it is expressed as follows: Where (u,v) are adaptive sampling coordinates, u=f u (i,j,S),v=f v (i,j,S), where S is the attention map, S = Softmax(f att (X)), where X is the input feature map, f att For attention generator; f u f v Let (i,j) be the coordinate mapping function, where (i,j) are uniform two-dimensional coordinates. To segment high-frequency regions of labels using real semantic segmentation As an attention map, it is mapped via the coordinate mapping function f u f v The adaptive sampling coordinates obtained by mapping are: Where Δ represents the Laplacian operator, used to extract the ground truth labels for semantic segmentation. The high-frequency region in the text, namely the semantic high-frequency region, is the edge of the semantic object; Sampling points When the semantic edges of objects are densely distributed, it can guide adaptive sampling to densely sample the semantic edges of objects. The adaptive sampling semantic segmentation network is trained and optimized end-to-end using the loss function described above. Step 3: Use the trained adaptive sampling semantic segmentation network to perform semantic segmentation on the image to obtain segmentation results with rich details and accurate edges.

Images