A high-resolution salient object detection method based on graffiti supervision

Abstract

The invention discloses a high-resolution salient target detection method based on graffiti supervision, image processing, sending the original resolution image to the edge structure preservation branch to extract rich edge detail features at a higher resolution; feature extraction, the scaled image is passed through a lightweight MobileNet backbone network to extract a feature map with a larger receptive field, and then the different scale features are sent to the multi-scale context fusion module through a short connection to obtain multi-scale features; edge extraction, the original resolution image is sent to the edge structure preservation branch to extract the edge features of the salient target at a higher resolution; feature fusion, the feature maps of the two branches are expanded to the same size and dimension through 1×1 convolution and bilinear interpolation respectively to perform feature fusion. The problem of high-resolution image processing is solved, the quality of the segmentation result is improved, and the computer memory consumption caused by the high-resolution image is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a high-resolution salient object detection method based on graffiti supervision. Background Art

[0002] Salient object detection (SOD) aims to accurately detect and segment the target region to attract people's attention in the image, which leads to an impressive performance in various visual recognition tasks. It can be used as a preprocessing step in robot intelligent control, visual tracking, image cropping, etc.

[0003] Methods based on deep convolutional neural networks (DCNNs) are more effective in obtaining salient foreground areas in complex backgrounds, thereby obtaining accurate segmentation results. However, these fully supervised methods rely heavily on large-scale pixel-level annotated datasets to train and learn the models. Even for experienced dataset annotators, pixel-level ground-truth annotation is very time-consuming and laborious. Therefore, this inefficient annotation work will inevitably limit the total number of pixel-level training samples, and thus become a bottleneck for the further development of fully supervised learning. For this reason, weakly supervised learning methods that do not rely on fine annotations have attracted the attention of researchers.

[0004] Compared with the fully supervised method, the weakly supervised method can only use one or more of the following weakly labeled datasets, which can save a lot of manpower and material resources. Weak labels: image-level category labels, bounding box labels, graffiti labels, and point labels. Category labels are easy to obtain, but lack annotation of semantic spatial information and boundary annotation of salient foreground areas. Bounding box labels provide location and category information for most target objects, but ignore semantic pixel annotation. Salient object detection is a class-agnostic task that focuses on segmenting salient foregrounds into white and binarizing backgrounds into black, and is insensitive to categories. Graffiti only marks the internal recognition target of class agnosticism and indicates the spatial location information of the target. Experiments show that the results obtained by graffiti are better than those obtained with other weak labels and are even close to those obtained through supervised learning.

[0005] In addition, almost all of these previous works that have achieved high-quality salient object segmentation results have focused on low-resolution images. In real-world scenarios, images taken by electronic devices are all high-resolution (such as 1920×1080 pixels or higher), and the processing of high-resolution images is a problem that needs to be solved at present. The problems faced by high-resolution input image processing are the quality of segmentation results and the computer memory consumption caused by high-resolution images. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a high-resolution salient object detection method based on graffiti supervision to solve the problems in the background technology.

[0007] The present invention provides the following technical solutions:

[0008] A high-resolution salient object detection method based on graffiti supervision includes the following steps:

[0009] Step 1: Image processing. Image processing uses the high reasoning quality of high-resolution images and the high efficiency of low-resolution images. In the feature extraction stage, the high-resolution input image is first scaled to 1 / 4 of the original size and sent to the feature extraction branch to efficiently extract multi-scale feature maps. The original resolution image is sent to the edge structure preservation branch to extract rich edge detail features at a higher resolution.

[0010] Step 2: Feature extraction: Send the scaled image to the lightweight context semantic branch, and extract the feature map with a large receptive field through the lightweight MobileNet backbone network. Then, the different scale features of the second, third, fourth, and fifth stages are sent to the multi-scale context fusion module through short connections to obtain multi-scale features.

[0011] Step 3: Edge extraction: the original resolution image is sent to the edge structure preservation branch and processed by the convolution layer to extract the edge features of the salient target at a higher resolution.

[0012] Step 4: Feature fusion: The feature maps of the two branches are expanded to the same size and dimension through 1×1 convolution and bilinear interpolation respectively to perform feature fusion, and the final prediction result is obtained by concatenate fusion.

[0013] Preferably, the two branches of the pre-image processing model simultaneously process two images of different resolutions, one is the original high-resolution image, and the other is a lower-resolution image that has been resized to 1 / 4 of the original resolution.

[0014] Preferably, the feature extraction backbone network adopts the MobileNet V2 network, which uses deep separable convolution and inverted residual structure to reduce the number of model parameters and reduce the loss of features caused by low-dimensional processing.

[0015] Preferably, the multi-scale context fusion module first uses bilinear interpolation to upsample the features of the second, third, fourth and fifth stages by 4, 8, 16 and 32 times respectively, uses concatenate to concatenate the features of different scales, and then obtains a multi-scale feature map after global average pooling, batch normalization and convolution operations.

[0016] Preferably, in the context semantic feature extraction stage, a partial cross entropy loss function is selected.

[0017] Preferably, three convolutional layers and a spatial position attention module are used in the edge structure preserving branch to suppress useless detail features and retain effective edge features of salient objects.

[0018] Preferably, the features extracted by the three-layer convolution are subjected to global maximum pooling and global average pooling operations respectively using the spatial position attention module, and then the two feature maps are fused, and then weighted by convolution operation and sigmoid operation to give useful edge structures, and then element-wise multiplication operation is performed with the features of the previous stage.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] (1) The present invention provides a high-resolution salient object detection method based on graffiti supervision, which greatly saves the time of large-scale pixel-level annotation of the data set and improves the working efficiency of the pixel-level annotation of the data set by preparing and marking the first graffiti-annotated high-resolution salient object detection data set.

[0021] (2) The present invention provides a high-resolution salient target detection method based on graffiti supervision. The lightweight context semantic branch adopts a lightweight MobileNet V2 feature extraction backbone network, and utilizes deep separable convolution and inverted residual structure to greatly reduce the number of model parameters and reduce the loss of features caused by low-dimensional processing.

[0022] (3) The present invention provides a high-resolution salient target detection method based on graffiti supervision, which adopts three convolutional layers and a spatial position attention module to suppress useless detail features and retain the effective edge features of salient targets, and can extract rich edge structures of salient targets at higher resolution.

[0023] (4) The present invention provides a high-resolution salient target detection method based on graffiti supervision. The feature maps of the two branches are expanded to the same size and dimension through 1×1 convolution and bilinear interpolation respectively to perform feature fusion. A simple concatenate fusion method is used to obtain the final prediction result, thereby reducing the number of parameters and ensuring work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 This is the overall network framework diagram of the present invention.

[0026] Figure 2 This is the percentage of annotated pixels in the first graffiti annotated dataset proposed by the present invention.

[0027] Figure 3 This is an example diagram of the annotation of the first graffiti annotation dataset proposed by the present invention.

[0028] Figure 4 This is a diagram of the multi-scale context feature fusion module of the present invention.

[0029] Figure 5 This is a diagram of the spatial position attention module of the present invention.

[0030] Figure 6 This is a table of evaluation results of the present invention on different data sets.

[0031] Figure 7 It is a comparison diagram of the PR curve and the F value curve of the present invention.

[0032] Figure 8 It is a MAE comparison chart of the present invention.

[0033] Fig. 9 It is a schematic diagram of the visualization process of the present invention.

[0034] Fig.10 This is a diagram of the submodule ablation experiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Embodiment 1:

[0038] like Figure 1As shown, a high-resolution salient object detection method based on graffiti supervision includes the following steps:

[0039] Step 1: Image processing. Image processing uses the high reasoning quality of high-resolution images and the high efficiency of low-resolution images. In the feature extraction stage, the high-resolution input image is first scaled to 1 / 4 of the original size and sent to the feature extraction branch to efficiently extract multi-scale feature maps. The original resolution image is sent to the edge structure preservation branch to extract rich edge detail features at a higher resolution.

[0040] Step 2: Feature extraction: Send the scaled image to the lightweight context semantic branch, and extract the feature map with a large receptive field through the lightweight MobileNet backbone network. Then, the different scale features of the second, third, fourth, and fifth stages are sent to the multi-scale context fusion module through short connections to obtain multi-scale features.

[0041] Step 3: Edge extraction: the original resolution image is sent to the edge structure preservation branch and processed by the convolution layer to extract the edge features of the salient target at a higher resolution.

[0042] Step 4: Feature fusion: The feature maps of the two branches are expanded to the same size and dimension through 1×1 convolution and bilinear interpolation respectively to perform feature fusion, and the final prediction result is obtained by concatenate fusion.

[0043] The two branches of the pre-image processing model simultaneously process two images of different resolutions, one is the original high-resolution image, and the other is a lower-resolution image that is resized to 1 / 4 of the original resolution.

[0044] The feature extraction backbone network adopts the MobileNet V2 network, and uses deep separable convolution and inverted residual structure to reduce the number of model parameters and reduce the loss of features caused by low-dimensional processing.

[0045] The multi-scale context fusion module first uses bilinear interpolation to upsample the features of the second, third, fourth and fifth stages by 4, 8, 16 and 32 times respectively, uses concatenate to fuse the features of different scales, and then obtains a multi-scale feature map after global average pooling, batch normalization and convolution operations.

[0046] In the context semantic feature extraction stage, a partial cross entropy loss function is selected.

[0047] In the edge structure preserving branch, three convolutional layers and a spatial position attention module are used to suppress useless detail features and retain the effective edge features of salient objects.

[0048] The spatial position attention module is used to perform global maximum pooling and global average pooling operations on the features extracted by the three-layer convolution. Then the two feature maps are fused and weighted by convolution and sigmoid operations to give useful edge structures, and then element-wise multiplication is performed with the features of the previous stage.

[0049] Embodiment 2

[0050] like Figure 1-10 As shown, a high-resolution salient object detection method based on graffiti supervision includes the following steps:

[0051] Step 1: Image processing. Image processing uses the high reasoning quality of high-resolution images and the high efficiency of low-resolution images. In the feature extraction stage, the high-resolution input image is first scaled to 1 / 4 of the original size and sent to the feature extraction branch to efficiently extract multi-scale feature maps. The original resolution image is sent to the edge structure preservation branch to extract rich edge detail features at a higher resolution.

[0052] Step 2: Feature extraction: Send the scaled image to the lightweight context semantic branch, and extract the feature map with a large receptive field through the lightweight MobileNet backbone network. Then, the different scale features of the second, third, fourth, and fifth stages are sent to the multi-scale context fusion module through short connections to obtain multi-scale features.

[0053] Step 3: Edge extraction: the original resolution image is sent to the edge structure preservation branch and processed by the convolution layer to extract the edge features of the salient target at a higher resolution.

[0054] Step 4: Feature fusion: The feature maps of the two branches are expanded to the same size and dimension through 1×1 convolution and bilinear interpolation respectively to perform feature fusion, and the final prediction result is obtained by concatenate fusion.

[0055] When inputting high-resolution images, in order to balance the inference speed and the quality of the prediction results, Mobilenet V2 is used as the backbone network to process lower resolution images (1 / 4 of the original image). MobileNet V2 uses depthwise separable convolution instead of standard convolution. It also constructs inverted residuals and linear bottlenecks, which greatly reduces the parameters in the network and avoids the loss of high-dimensional information. Especially for the input of high-resolution images, the reduction in the number of parameters means that the network's inference speed will be faster.

[0056] High-level features obtained through convolution or pooling operations have large receptive fields and good semantic encoding capabilities, but they will result in low resolution feature maps and weak representation of geometric information. The characteristics of low-level features are opposite to high-level features and can complement each other. Therefore, multi-scale feature representation and multi-scale receptive fields can improve the detection effect of salient objects of different scales. A multi-scale context aggregation module (MCAM) is constructed to connect the side outputs of different stages, and then the multi-scale global context semantic information is encoded through global average pooling operations and convolution λ to obtain feature maps with multi-scale receptive fields. This operation process does not greatly increase the computational complexity of the network.

[0057] Under graffiti supervision, the network only uses incomplete foreground labeled pixels. In the absence of boundary position information, it is necessary to accurately identify and segment salient objects. Graffiti tags cannot clearly identify the location and boundary details of the most discriminative objects. In order to avoid poor prediction and segmentation results caused by directly using graffiti supervision, a boundary structure maintenance branch is introduced, which only contains three convolutional layers and a spatial location attention module (SLAM) to maintain the full-resolution input image as a supplement, providing as much obvious boundary information as possible to the other branch.

[0058] In order to achieve a compromise between running efficiency and performance accuracy, the simplest concatenate method is used to fuse the features of the two branches without using a post-processing algorithm to refine the saliency map again. In the post-concatenation operation, a bilinear interpolation upsampling operation is used to restore the final fine-grained image, mapping it to the original high-resolution input size.

[0059] In order to verify the effectiveness of this method, comparative experiments were conducted on a high-resolution dataset (HRSOD,) and four public low-resolution datasets (DUT-TE, ECSSD, DUTS-OMORN, HKU-IS) with 10 of the latest salient object detection methods. The experiment used the deep learning framework Pytorch to build the network. The experimental environment was a Linux system. NVIDIA TeslaT4 GPU16GB was used to train and test the neural network model. The model was trained on Scr-HRSOD, the first graffiti-annotated high-resolution salient object detection dataset. In addition, the proposed method was trained using the Scr-HRSOD and S-DUTS datasets to verify the generalization ability of low-resolution inputs. The experimental performance was compared with the latest methods on five datasets, such as Figure 6 , Figure 7 As shown in Figure 2, the PR curve and F value curve of this method perform well. Figure 7 As shown in the figure, from the comparison of saliency map results, this method has strong capabilities of high-resolution salient target edge segmentation, position detection and detail segmentation.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A high-resolution salient object detection method based on graffiti supervision, characterized in that: The following steps are involved: Step 1: Image processing. Image processing uses the high reasoning quality of high-resolution images and the high efficiency of low-resolution images. In the feature extraction stage, the high-resolution input image is first scaled to 1 / 4 of the original size and sent to the feature extraction branch to efficiently extract multi-scale feature maps. The original resolution image is sent to the edge structure preservation branch to extract rich edge detail features at a higher resolution. Step 2: Feature extraction: Send the scaled image to the lightweight context semantic branch, and extract the feature map with a large receptive field through the lightweight MobileNet backbone network. Then, the different scale features of the second, third, fourth, and fifth stages are sent to the multi-scale context fusion module through short connections to obtain multi-scale features. Step 3: Edge extraction: the original resolution image is sent to the edge structure preservation branch and processed by the convolution layer to extract the edge features of the salient target at a higher resolution. Step 4: Feature fusion: The feature maps of the two branches are expanded to the same size and dimension through 1×1 convolution and bilinear interpolation to perform feature fusion, and the final prediction result is obtained by concatenate fusion. When inputting high-resolution images, in order to balance the inference speed and the quality of the prediction results, MobilenetV2 is used as the backbone network to process lower-resolution images (1 / 4 of the original image). MobileNet V2 uses depthwise separable convolution instead of standard convolution, and also constructs inverted residuals and linear bottlenecks, which greatly reduces the parameters in the network and avoids the loss of high-dimensional information. Especially for the input of high-resolution images, the reduction in the number of parameters means that the network's inference speed will be faster; MobileNet V2 uses depthwise separable convolution to replace standard convolution, and also constructs inverted residual and linear bottleneck. It encodes multi-scale global contextual semantic information through global average pooling operation and convolution λ to obtain feature maps with multi-scale receptive fields. Under graffiti supervision, the network only uses incomplete foreground labeled pixels. In the absence of boundary position information, it is necessary to accurately identify and segment salient targets. Graffiti tags cannot clearly identify the location and boundary details of the most discriminative objects. In order to avoid poor prediction segmentation results caused by directly using graffiti supervision, a boundary structure maintenance branch is introduced, which only contains three convolutional layers and a spatial location attention module (SLAM) to maintain the full-resolution input image as a supplement, providing as much obvious boundary information as possible to the other branch.

2. According to claim 1, a high-resolution salient object detection method based on graffiti supervision is characterized in that: The two branches of the pre-image processing model simultaneously process two images of different resolutions, one is the original high-resolution image, and the other is a lower-resolution image that is resized to 1 / 4 of the original resolution.

3. According to claim 1, a high-resolution salient object detection method based on graffiti supervision is characterized in that: The feature extraction backbone network adopts the MobileNet V2 network, and uses deep separable convolution and inverted residual structure to reduce the number of model parameters and reduce the loss of features caused by low-dimensional processing.

4. According to claim 1, a high-resolution salient object detection method based on graffiti supervision is characterized in that: The multi-scale context fusion module first uses bilinear interpolation to upsample the features of the second, third, fourth and fifth stages by 4, 8, 16 and 32 times respectively, uses concatenate to fuse the features of different scales, and then obtains a multi-scale feature map after global average pooling, batch normalization and convolution operations.

5. According to claim 1, a high-resolution salient object detection method based on graffiti supervision is characterized in that: In the context semantic feature extraction stage, a partial cross entropy loss function is selected.

6. The high-resolution salient object detection method based on graffiti supervision according to claim 1 is characterized in that: In the edge structure preserving branch, three convolutional layers and a spatial position attention module are used to suppress useless detail features and retain the effective edge features of salient objects.

7. A high-resolution salient object detection method based on graffiti supervision according to any one of claims 1 to 6, characterized in that: The spatial position attention module is used to perform global maximum pooling and global average pooling operations on the features extracted by the three-layer convolution. Then the two feature maps are fused and weighted by convolution and sigmoid operations to give useful edge structures, and then element-wise multiplication is performed with the features of the previous stage.

Images