A semantic segmentation method based on background blur

Abstract

The present invention relates to a semantic segmentation method based on background blur, which is implemented by embedding a BOKEH module in an existing FCN semantic segmentation network. The algorithm of the BOKEH module is described as #imgabs0##imgabs1#, where img(:,:,k) represents the pixel value size on the k channel of the image; σ represents the blur factor; R B* Represents the overall background rate of the training set; R B The background domain ratio (BGL) represents the background position information of the image. This semantic segmentation method based on background blurring (BOKEH) undergoes in-depth analysis and optimization of the dataset. Without changing the existing network architecture or increasing the data size, it effectively improves the network segmentation performance by embedding the background blurring module (BOKEH) into the existing FCN semantic segmentation network.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a semantic segmentation method based on background blurring. Background Art

[0002] In recent years, deep convolutional neural networks have been increasingly used to achieve pixel-level, end-to-end image segmentation—semantic segmentation. As an essential component of computer vision tasks, semantic segmentation is widely used in autonomous driving, robotic perception, augmented reality, video surveillance, and other fields.

[0003] The emergence and application of FCNs (fully convolutional neural networks) have greatly simplified traditional approaches to solving the semantic segmentation problem. Based on existing datasets, segmentation results are relatively high, even reaching saturation. However, from semantic segmentation to real-time semantic segmentation, and from redundant to streamlined network architectures, existing improvements generally achieve better segmentation by designing and improving the network structure itself and using extensive data augmentation methods, while ignoring the impact of dataset characteristics on segmentation results. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a semantic segmentation method based on background blur, which aims to perform in-depth analysis and optimization of the data set, and improve the network segmentation performance by embedding the background blur module BOKEH in the existing FCN semantic segmentation network without changing the existing network structure or increasing the data.

[0005] This semantic segmentation method based on background blur is implemented by embedding the BOKEH module in the existing FCN semantic segmentation network. The algorithm of the BOKEH module is described as

[0006] Among them, img(:,:,k) represents the pixel value size on the k channel of the image; σ represents the blur factor; R B* Represents the overall background rate of the training set; R B Indicates the background area ratio; BGL indicates the background position information of the image.

[0007] Furthermore, the

[0008] in, Indicates the domain of interest; Represents the background blur area.

[0009] Furthermore, the

[0010] Among them, 0≤i≤H-1,0≤j≤W-1,k=0,1,2,σ∈(0,1]; img(i,j,k) represents the pixel value at position (i,j) on the k channel of the image; * represents the matrix Hadamard product; img(i,j,k)*BGL(i,j) represents the Hadamard product of img(i,j,k) with BGL(i,j) on the k channel domains respectively.

[0011] Furthermore, the σ selection strategy is σ=-R B ×R B* +1;

[0012] Among them, R B* Represents the overall background rate of the training set; R B Represents the background domain ratio.

[0013] Furthermore, the

[0014] in, is the sum of the background pixel values; H represents the image height; W represents the image width.

[0015] Specifically, the FCN network model includes an input end, a convolution unit, a pooling unit, an encoder unit, a decoder unit, and an output end, wherein the BOKEH module is embedded between the input end and the first convolution unit.

[0016] The present invention proposes a semantic segmentation method based on background blurring. After in-depth analysis and optimization of the data set, the network segmentation performance is effectively improved by embedding the background blurring module BOKEH into the existing FCN semantic segmentation network without changing the existing network architecture or increasing the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The following is a further description of a semantic segmentation method based on background blurring of the present invention with reference to the accompanying drawings:

[0018] Figure 1 This is a comparison table of the segmentation performance of the three FCNs before and after the BOKEH module on the PASCAL VOC 2012 validation set;

[0019] Figure 2 This is a detailed accuracy comparison table of FCN-8s on the PASCAL VOC 2012 Val dataset before and after this BOKEH module;

[0020] Figure 3 This is a comparison table of the segmentation performance of BiSeNet on the CamVid validation set before and after this BOKEH module;

[0021] Figure 4This is a detailed accuracy comparison table of BiSeNet on the CamVid dataset before and after this BOKEH module;

[0022] Figure 5 This is the FCN network architecture diagram in which this BOKEH module is embedded. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described below with specific examples, but the protection scope of the present invention is not limited to the following examples.

[0024] Implementation 1: This semantic segmentation method based on background blur is implemented by embedding the BOKEH module in the existing FCN segmentation network. The algorithm of the BOKEH module is described as follows: Among them, img(:,:,k) represents the pixel value size on the k channel of the image; σ represents the blur factor; R B* Represents the overall background rate of the training set; R B Indicates the background area ratio; BGL indicates the background position information of the image.

[0025] described in, Indicates the domain of interest; Represents the background blur area. Where, 0≤i≤H-1,0≤j≤W-1,k=0,1,2,σ∈(0,1]); img(i,j,k) represents the pixel value at position (i,j) on channel k of the image; * represents the matrix Hadamard product; img(i,j,k)*BGL(i,j) represents the Hadamard product of the k channel domains of img(i,j,k) with BGL(i,j). The σ selection strategy is σ=-R B ×R B* +1; where R B* Represents the overall background rate of the training set; R B Represents the background domain ratio. in, is the sum of the background pixel values; H represents the image height; W represents the image width.

[0026] The specific algorithm principle is:

[0027] Assume that the entire image is divided into the domain of interest and the background domain. The set of all pixels of interest is denoted as φ I ; The set of all pixels of no interest is denoted as φ B .

[0028] For any input image and corresponding label during training Where H represents the image height, W represents the image width, and C represents the number of image channels (for RGB images, C = 3). The background blur can be described as:

[0029]

[0030]

[0031]

[0032] Among them, 0≤i≤h-1,0≤j≤w-1,k=0,1,2,σ∈(0,1]; represents the domain of interest, represents the background blur area, Represents the background location information at the coordinate position (i, j) of the image plane. When BGL(i, j) = 1, it means that the point belongs to the background, and 0 means it is the region of interest. img(i, j, k) represents the pixel value at position (i, j) on the k channel of the image. * represents the matrix Hadamard product. img(i, j, k) * BGL(i, j) represents the Hadamard product of the k channel domains of img(i, j, k) with BGL(i, j). σ represents the blur factor. Then:

[0033] Among them, R B Represents the background domain ratio, R I Indicates the proportion of interest domains, obviously there is R B +R I =1, R B 、R I Defined as:

[0034] in, is the sum of the number of pixels in the background domain, is the sum of the number of pixels in the interest area. For PASCALVOC 2012 Trainset, we know that

[0035] For the selection of the blur factor σ, let R B* Represents the overall background rate of the dataset training set (such as PASCAL VOCTrainsetR B* =0.7481), let σ∈[1-R B* ,1],For each image in the data set, if the background ratio is larger, the background blur degree should be larger, that is, the corresponding σ should be smaller; if the background ratio is smaller, the background blur degree should be smaller, that is, σ should be larger, especially when RB = 0, that is, only the region of interest is included, and the corresponding background blur factor σ should take the maximum value; when R B = 1, that is, only the background domain is included. At this time, the corresponding background blur factor σ should take the minimum value. Let σ, R B Satisfies the following relationship: σ=-R B ×α+β;

[0036] Among them, α, From the above assumptions, we can see Substituting into That is: σ=-R B ×R B* +1;

[0037] Therefore, the BOKEH model algorithm can be described as:

[0038] Implementation method 2: Figure 5 As shown, specifically with respect to the FCN network model, the FCN network of the semantic segmentation method based on background blurring includes an input end, a convolution unit, a pooling unit, an encoder unit, a decoder unit, and an output end, wherein the BOKEH module is embedded between the input end and the first convolution unit. The BOKEH module is as described in Implementation 1 and will not be described again.

[0039] Comparative analysis of technical effects:

[0040] 1. Use the classic FCN network and the real-time bilateral network BiSeNet.

[0041] In order to more accurately capture the impact of background information on segmentation accuracy, all data augmentation methods except cropping are removed. They are denoted as (Re)FCN-8s, (Re)FCN-16s, (Re)FCN-32s, and (Re)BiSeNet.

[0042] 2. Apply this method to multiple segmentation networks on two datasets and report the results on both datasets. Specifically:

[0043] (1) Experiments were conducted on three FCN network architectures on the PASCAl VOC dataset, such as Figure 1 As shown in the table, the background blur module BOKEH improves the (Re)FCN-32s, (Re)FCN-16s and (Re)FCN-8s by 4.7, 4.6 and 4.8 points respectively. The specific accuracy of each category under the (Re)FCN-8s and (Re)FCN-8s+BOKEH results is as follows Figure 2 As shown. Figure 2As shown in the table, sheep (0.86%), boats (0.58%), bicycles (0.29%), and potted plants (0.64%) account for relatively small proportions and have the largest performance improvements. Therefore, embedding the BOKEH module achieves significant improvements in relatively small proportions of categories.

[0044] (2) The segmentation results of the BOKEH module on the CamVid dataset are as follows Figure 3 As shown in the table, the results show that the mean IOU is improved by 3.7, proving that the BOKEH module is suitable for real-time network architecture. From the results of PASCAL VOC 2012, BOKEH plays a good role in objects with a small proportion of categories in the dataset. Figure 4 As shown in the table, it can be seen from the table data that the background accounts for a relatively small proportion, but has the largest improvement, which is in line with the above rules.

[0045] A semantic segmentation method based on background blurring in the present invention conducts in-depth analysis and optimization of the data set. Without changing the existing network structure or increasing the data, the network segmentation performance is effectively improved by embedding the background blurring module BOKEH into the existing FCN semantic segmentation network.

[0046] The above description shows the main features, basic principles, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments or examples described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The above embodiments or examples should therefore be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be encompassed within the present invention. Any reference numerals in the claims should not be construed as limiting the claims to which they relate.

[0047] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A semantic segmentation method based on background blur, characterized by: This method is implemented by embedding the BOKEH module in the existing FCN semantic segmentation network. The algorithm of the BOKEH module is described as follows: Among them, img(:,:,k) represents the pixel value size on the k channel of the image; σ represents the blur factor; R B* Represents the overall background rate of the training set; R B Indicates the background area ratio; BGL indicates the background position information of the image.

2. The semantic segmentation method based on background blurring according to claim 1, wherein: described in, Indicates the domain of interest; Represents the background blur area.

3. The semantic segmentation method based on background blurring according to claim 2, wherein: described Among them, 0≤i≤H-1,0≤j≤W-1,k=0,1,2,σ∈(0,1]; img(i,j,k) represents the pixel value at position (i,j) on the k channel of the image; * represents the matrix Hadamard product; img(i,j,k)*BGL(i,j) represents the Hadamard product of img(i,j,k) with BGL(i,j) on the k channel domains respectively.

4. The semantic segmentation method based on background blurring according to claim 3, wherein: The σ selection strategy is σ=-R B ×R B* +1; Among them, R B* Represents the overall background rate of the training set; R B Represents the background domain ratio.

5. The semantic segmentation method based on background blurring according to claim 4, wherein: described in, is the sum of the background pixel values; H represents the image height; W represents the image width.

6. The semantic segmentation method based on background blurring according to claim 5, wherein: The FCN semantic segmentation network includes an input end, a convolution unit, a pooling unit, an encoder unit, a decoder unit, and an output end, wherein the BOKEH module is embedded between the input end and the first convolution unit.

Images