A human-like perception mechanism driven remote sensing image salient object detection method

Abstract

The present application belongs to the field of image analysis processing, and relates to a kind of human-like perceptual mechanism driven remote sensing image saliency target detection method, comprising: obtaining the remote sensing image to be measured and inputting the trained target detection model, to obtain detection result;The training process of target detection model includes: obtaining data set X, pre-process X, to obtain the pre-processed remote sensing image and its label data;The pre-processed remote sensing image is input into MiT network, to obtain feature and high-frequency information;Feature and high-frequency information are input into high-frequency feature supplement module, to obtain enhanced feature;Enhanced feature is input into decoder, to obtain detection result;According to detection result and label data, the parameters of target detection model are updated until the trained target detection model is obtained;The present application extracts the high-frequency information around the pixels of remote sensing image using wavelet transform, and uses guided filtering to supplement the high-frequency information of the features of remote sensing image, so that the features can better supplement saliency target.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image analysis and processing, and relates to a remote sensing image saliency target detection method driven by a human-like perception mechanism. BACKGROUND

[0002] Nowadays, with the development of the times, the application of computer vision field has become a hot spot. Among them, saliency target detection is a widely used technology in the field of computer vision, which aims to accurately identify and highlight the most salient and eye-catching target area in the image or video, so as to help users quickly locate the key information and improve efficiency. Saliency target detection can also assist in target tracking, image segmentation, scene understanding and other visual tasks, and provide valuable prior information.

[0003] Saliency target detection of remote sensing images is mainly used in the fields of agricultural detection, urban planning, environmental detection, geological disaster detection, etc. It has important application value. Due to the obvious differences in shooting equipment, scene and angle between remote sensing images and natural images, there are differences in resolution, data type and scale between the two. At the same time, the saliency target detection of remote sensing images generally detects the ground cover, topography and other large-scale continuous and spatial structure features. In some scenes, there is no prominent area, so the saliency target detection of remote sensing images and natural images are obviously different, which makes the saliency target detection methods of the two data forms different in many ways, especially the complex topological structure in remote sensing images has a great influence on the saliency target detection of remote sensing images.

[0004] At present, the saliency target detection method of remote sensing images is mostly along the development route of natural images. However, due to the differences with natural images, the method of saliency target detection of remote sensing images is subject to many conditions. The main solution ideas of many methods are as follows. First, use auxiliary information to enhance the final result. That is, use edge information and skeleton information to enhance the result and features, which is a direct way to enhance the final result. Second, expand the feature receptive field and enhance the local information. That is, constantly expand the generated feature receptive field, and then enhance the local information on this basis. Such a method is in line with the idea of segmentation tasks such as semantic segmentation, that is, to get a good feature to help achieve a good result. In recent years, GeleNet proposed is a framework of the second solution idea, which to some extent alleviates the detection of difficult samples.

[0005] However, these frameworks all use semantic segmentation to solve the problem of salient object detection, without considering the characteristics of the salient object detection task and the characteristics of remote sensing images, so there are often problems of missed detection and interference by shadows, which affects the quality of the detection results to some extent. SUMMARY

[0006] To solve the above prior art problems, the present application adopts a remote sensing image salient object detection method driven by a human-like perception mechanism, comprising: obtaining a remote sensing image to be detected, inputting the remote sensing image to be detected into a trained salient object detection model to obtain a detection result; the salient object detection model comprises: a MiT network, a high-frequency feature supplement module HFSM and a decoder; the training process of the salient object detection model comprises:

[0007] S1, obtain a remote sensing image salient object detection image data set X, and pre-process the data set X to obtain a pre-processed data set X'; the pre-processed data set X' comprises: pre-processed remote sensing images and their label data;

[0008] S2, input the pre-processed remote sensing image into the MiT network to obtain features and high-frequency information; wherein MiT is a hybrid visual transformer;

[0009] S3, input the features and high-frequency information into the high-frequency feature supplement module HFSM to obtain enhanced features;

[0010] S4, input the enhanced features into the decoder to obtain the detection result;

[0011] S5, calculate the loss function value according to the detection result and the label data, update the parameters of the salient object detection model according to the loss function value, and obtain the trained salient object detection model when the loss function value is the smallest.

[0012] The data set X comprises: remote sensing images and their GT images; the pre-processing of the data set X comprises:

[0013] S11, normalize, randomly rotate and randomly crop the remote sensing images to obtain pre-processed remote sensing images;

[0014] S12, Gaussian blur the GT images to obtain Gaussian blurred GT images; calculate the inverse distance of the GT images to obtain an inverse distance map; use the Gaussian blurred GT images and the inverse distance map as the label data of the pre-processed remote sensing images; wherein the GT image is a real situation image.

[0015] The MiT network comprises: N MiT modules M i, i ∈ [1, N], N MiT modules are connected in series; the MiT network processing data in the pre-processed data set X' includes:

[0016] S21, inputting the pre-processed remote sensing image into the MiT module M1 to obtain the feature f 1 and high frequency information

[0017] S22, inputting the feature f 1 into the MiT module M2 to obtain the feature f 2 and high frequency information

[0018] S23, inputting the feature f i-1 into the MiT module M i to obtain the feature f i and high frequency information

[0019] S24, repeating step S23 until all MiT modules are passed through.

[0020] The MiT module includes a wavelet transform module and a feature extraction module; the MiT module M i processing the remote sensing image includes: inputting the feature f i-1 into the wavelet transform module to obtain the intermediate feature f i,* and high frequency information inputting the intermediate feature f i,* into the feature extraction module to obtain the feature f i .

[0021] The high-frequency feature supplement module HFSM includes: N sub-modules HFSM-i, i ∈ [1, N]; the high-frequency feature supplement module HFSM processing the feature and the high frequency information includes: inputting the feature f i and high frequency information into the corresponding HFSM-i to obtain the enhanced feature

[0022] HFSM-i includes a first guided filter module, a second guided filter module, a third guided filter module and a convolution block, and the high frequency information includes: HFSM-i processing the feature f i and high frequency information includes: inputting the feature f i and high frequency information into the first guided filter module, inputting the feature f i and high frequency information into the second guided filter module, inputting the feature f i and high frequency information The outputs of the first guiding filtering module, the second guiding filtering module and the third guiding filtering module are combined, and the combined result is input into a convolution block to obtain an enhanced feature. respectively represent high-frequency information in horizontal, vertical and diagonal directions.

[0023] The decoder comprises a plurality of decoding layers and a plurality of dimension adjustment modules, the decoding layers are connected with the dimension adjustment modules one by one, the decoding layers comprise convolution blocks and skip connections, and the number of the convolution blocks and the number of times of the skip connections in the decoding layers increase layer by layer.

[0024] The loss function L total is:

[0025]

[0026] wherein, and are respectively an L1 loss and an IoU loss of a detection result, L distance is a reciprocal distance loss of the detection result, K is a number of levels of the decoder, and j is an index of the decoder level.

[0027] wherein, GT' j is a GT image blurred by a Gaussian function, || l1 is an L1 loss, S j is a detection result output by a dimension adjustment module corresponding to the decoder level j.

[0028] The reciprocal distance loss of the detection result is:

[0029] L distance = |pred, GT_distance| l1 + |pred, GT_distance| iou

[0030] wherein, pred is a detection result output by a dimension adjustment module corresponding to a last level of the decoder, GT_distance is a reciprocal distance map of a GT image, || l1 , and iou are respectively an L1 loss and an IoU loss.

[0031] Beneficial effects:

[0032] 1、The present application starts from the process of human perception of objects, first from bottom to top using the wavelet transform module of MiT network to extract the high frequency information around the pixels of remote sensing image, that is, the shape features of objects, and then from top to bottom using guided filtering to supplement the high frequency information of the features output by the MiT network, so that the shape features can be used to guide the features to better supplement the salient objects, quickly locate the interested area and refine it, thereby reducing the missed detection of the target and reducing the interference of the shadow of the remote sensing image; 2, The present application adopts a new hybrid loss, which contains inverse distance loss, IoU loss and L1 loss; wherein the inverse distance loss needs to calculate the inverse distance of the GT map, so that the edge and salient object positioning of the final result can be better enhanced, and the improved IoU loss and L1 loss also simulate the process of human perception from coarse to fine, and accelerate the network convergence. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The flow chart of the remote sensing image salient object detection method driven by the human-like perception mechanism provided by the embodiment of the present application is shown in the figure.

[0034] Figure 2 The schematic diagram of the target detection model provided by the embodiment of the present application is shown in the figure.

[0035] Figure 3 The schematic diagram of Wavlet Block provided by the embodiment of the present application is shown in the figure.

[0036] Figure 4 The schematic diagram of the sub-module HFSM-i provided by the embodiment of the present application is shown in the figure.

[0037] Figure 5 The schematic diagram of the decoder provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0039] As Figure 1 , Figure 2As shown, the present application adopts a remote sensing image salient target detection method driven by a human-like perception mechanism, which comprises: acquiring an optical remote sensing image (ORSI) to be detected, inputting the optical remote sensing image to be detected into a trained salient target detection model to obtain a detection result; the salient target detection model comprises: a MiT network, a high-frequency feature supplement module HFSM and a decoder; the training process of the salient target detection model comprises:

[0040] S1, acquiring a remote sensing image salient target detection image dataset X, and pre-processing the dataset X to obtain a pre-processed dataset X'; the pre-processed dataset X' comprises: pre-processed remote sensing images and their label data;

[0041] The dataset X comprises: remote sensing images and their GT images; the pre-processing of the dataset X comprises:

[0042] S11, normalizing, randomly rotating and randomly cropping the remote sensing images to obtain pre-processed remote sensing images with unified size; the size is HxWx3, H=W=512;

[0043] S12, Gaussian blurring the GT images to different degrees to obtain the Gaussian blurred GT images GT' j , and performing inverse distance calculation on the GT images to obtain an inverse distance map, taking the Gaussian blurred GT images and the inverse distance map as the label data of the pre-processed remote sensing images; wherein when j is 1, GT' j represents the GT image without Gaussian blurring processing, and when j is not 1, GT' j represents the result obtained by Gaussian blurring with a kernel of 5 and a variance of 2 based on the previous GT' j-1 ; the GT image is a Ground Truth image, i.e. a real situation image or a reference image.

[0044] S2, inputting the pre-processed remote sensing images into the MiT network to obtain features and high-frequency information;

[0045] The MiT network is a backbone network of Transformer, and MiT (MixVisionTransformer) is a mixed vision transformer; the MiT network comprises: N MiT modules M i ,i∈[1,N], and the N MiT modules are connected in series; the MiT network processes the pre-processed remote sensing images, which comprises:

[0046] S21, inputting the pre-processed remote sensing images into the MiT module M1 to obtain features f 1 and high-frequency information

[0047] S22, the feature f 1 is input into the MiT module M2, to obtain the feature f 2 and high-frequency information

[0048] S23, the feature f i-1 is input into the MiT module M i , to obtain the feature f i and high-frequency information

[0049] S24, repeat step S23 until all MiT modules are passed through.

[0050] The MiT module comprises a wavelet transform module and a feature extraction module, and the feature extraction module comprises a combination of L forward feedback modules (Mix-FFN) and an efficient self-attention module (Efficient Self-Attn); the wavelet transform module takes the result of the previous MiT module as input, and the first wavelet transform takes the remote sensing image as input.

[0051] As shown in Figure 3 , the MiT module M i processes the remote sensing image, which includes wavelet transform (Wavelet Transform) on the feature f i-1 to separate the high-frequency information, to obtain four components Among them, is low-frequency information, respectively represent the high-frequency information in the horizontal, vertical and diagonal directions, and the high-frequency information is reserved separately in order to simulate the first step of human perception of objects, that is, to abstract the shape features of objects from the high-frequency information around the image pixels according to the image pixels from bottom to top.

[0052] The feature f i-1 is processed by a convolution combination to obtain an intermediate feature f i ; the convolution combination includes a convolution layer, a batch normalization layer and an activation layer, and its purpose is to reduce the size of the feature while expanding the channel dimension C i of the feature so that the feature can obtain more rich semantic information;

[0053] The intermediate feature f i,* is input into the feature extraction module, so that the feature can pay more attention to global information, expand the receptive field, and make the final obtained feature f i represent more rich information.

[0054] S3, inputting the feature and the high-frequency information into a high-frequency feature supplement module HFSM to supplement the feature with the high-frequency information, to obtain an enhanced feature;

[0055] As shown in Figure 4 , the high-frequency feature supplement module HFSM includes N sub-modules HFSM-i, i∈[1,N]; processing the feature and the high-frequency information by the high-frequency feature supplement module HFSM includes: inputting the feature f i and the high-frequency information into the corresponding HFSM-i, to obtain the enhanced feature

[0056] The HFSM-i includes a first guided filter module, a second guided filter module, a third guided filter module and a convolution block (Convs); the convolution block includes a convolution layer, a batch normalization layer and an activation layer.

[0057] Processing the feature f i and the high-frequency information by the HFSM-i includes: inputting the feature f i and the high-frequency information into the first guided filter module, inputting the feature f i and the high-frequency information into the second guided filter module, inputting the feature f i and the high-frequency information into the third guided filter module, combining (Concatenation) the outputs of the first guided filter module, the second guided filter module and the third guided filter module, and inputting the combined result into the convolution block, to obtain the enhanced feature. The main role of this module is to imitate the second step of human perception of objects, to guide the feature to focus on the salient target area using the above high-frequency information, to scan the entire field of view from top to bottom to identify the size, direction and other features of the object, and to quickly locate and refine the area of interest.

[0058] Different high-frequency information obtains different guided results, which are expressed as follows:

[0059]

[0060] Among them, Guided() is a guided filter, and the three high-frequency information are used as a guide map to guide the feature with different high-frequency features.

[0061] Combining the outputs of the first guided filter module, the second guided filter module and the third guided filter module includes:

[0062]

[0063] Among them, represents the enhanced feature, convs are convolution blocks, and cat represents concatenation of the guided results Concatenate along the channel dimension.

[0064] In one embodiment, the number N of MiT modules and sub-modules HFSM-i is 4.

[0065] S4, input the enhanced feature into the decoder to obtain a detection result;

[0066] The enhanced feature obtained by the sub-module HFSM-i is directly used in the decoder, and the decoder includes K decoding layers, which are designed according to the idea of Unet++. The traditional pyramid structure is adopted, and the number of convolution blocks and the number of skip connections are gradually increased from shallow to deep, so as to fully interact between features and better utilize the context information between adjacent features, thereby better positioning and refining the salient target. The decoder also includes a plurality of dimension adjustment modules, which are connected one by one with the decoding layers. The dimension adjustment module includes a convolution layer (Conv) and a convolution block (Convs), so that the final output result of the decoder corresponds to the dimension of the label data.

[0067] In one embodiment, as shown in Figure 5 , the number of layers K of the decoder is 3, and the number of convolution blocks (convs) contained in each layer is 1, 2 and 3 respectively.

[0068] S5, calculate the loss function value according to the detection result and the label data, and update the parameters of the salient target detection model according to the loss function value, and when the loss function value is the smallest, the trained salient target detection model is obtained.

[0069] The loss function L total is:

[0070]

[0071] L distance = |pred, GT_distance| l1 + |pred, GT_distance| iou

[0072] wherein, and are the L1 loss and the IoU loss of the detection result respectively, the L1 loss is the Mean Absolute Error (MAE), the IoU loss is the Intersection over Union Loss, and L distanceFor the inverse distance loss of the detection result, prde is the detection result output by the dimension adjustment module corresponding to the last layer of the decoder, S j For the detection result output by the dimension adjustment module corresponding to the decoder layer j, GT′ j GT_distance is the inverse distance map of the GT image, and l1 iou L1 loss and IoU loss, respectively.

[0073] The above examples further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made to the present application within the spirit and principles of the present application should be included in the protection scope of the present application.​

Claims

1. A human-like perceptual mechanism driven salient object detection method for remote sensing images, characterized in that, The application relates to the technical field of remote sensing image processing, and discloses a remote sensing image salient target detection method and device. The method comprises the following steps: The salient target detection model comprises an MiT network, a high-frequency feature supplement module HFSM and a decoder, and the training process of the salient target detection model comprises the following steps: S1, obtain a remote sensing image salient target detection image dataset X, and pre-process the dataset X to obtain a pre-processed dataset X ′ ; the pre-processed dataset X ′ comprise: a pre-processed remote sensing image and label data thereof S2, inputting the preprocessed remote sensing image into the MiT network to obtain features and high-frequency information; wherein, the MiT is a hybrid visual transformer; The MiT network comprises N MiT modules M i , i ∈ [1, N], and the N MiT modules are connected in series in turn; and the MiT network processes data in the preprocessed data set X ′ includes the following steps. S21, input the pretreated remote sensing image into the MiT module M1 to obtain the feature f 1 and high frequency information S22, the feature f 1 inputting the MiT module M2, to obtain the feature f 2 and high frequency information S23, the feature f i-1 inputting the MiT module M i , obtaining the feature f i and high-frequency information S24, repeating step S23 until all MiT modules are passed through; The MiT module comprises a wavelet transform module and a feature extraction module; the MiT module M i Processing the remote sensing image comprises inputting the feature f i-1 into the wavelet transform module to obtain intermediate features f i,* and high-frequency information The intermediate features f i,* are input into the feature extraction module to obtain the feature f i ; S3, inputting the features and the high-frequency information into the high-frequency feature supplement module HFSM to obtain enhanced features; The high-frequency feature supplement module HFSM comprises: N sub-modules HFSM-i, i∈[1, N]; the high-frequency feature supplement module HFSM processes the feature and the high-frequency information, comprising: inputting the feature f i and the high-frequency information into the corresponding HFSM-i, to obtain the enhanced feature The HFSM-i includes a first guided filter module, a second guided filter module, a third guided filter module, and a convolution block, and the high-frequency information comprises: The HFSM-i processes the feature f i and the high-frequency information comprises: i The HFSM-i processes the feature f inputs the first guided filter module, processes the feature f i and the high-frequency information inputs the second guided filter module, processes the feature f i and the high-frequency information inputs the third guided filter module, combines outputs of the first guided filter module, the second guided filter module, and the third guided filter module, and inputs the combined result into the convolution block to obtain an enhanced feature; wherein, respectively represent high-frequency information in horizontal, vertical, and diagonal directions. S4, inputting the enhanced features into the decoder to obtain a detection result; S5, calculating a loss function value according to the detection result and label data, updating the parameters of the salient target detection model according to the loss function value, and obtaining a trained salient target detection model when the loss function value is the minimum; Loss function L total is: wherein, and L1 loss and IoU loss of the detection result, respectively, L distance is the inverse distance loss of the detection result, K is the number of levels of the decoder, and j is the index of the decoder level.

2. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 1, characterized in that: The data set X comprises remote sensing images and GT images thereof, and the preprocessing of the data set X comprises the following steps: S11, normalizing, randomly rotating and randomly cropping the remote sensing images to obtain preprocessed remote sensing images; S12, performing Gaussian blurring on the GT images to obtain Gaussian-blurred GT images, and performing inverse distance calculation on the GT images to obtain an inverse distance map; the Gaussian-blurred GT images and the inverse distance map are taken as label data of the preprocessed remote sensing images; wherein, the GT image is a real situation image.

3. The human-like perceptual mechanism driven remote sensing image saliency object detection method according to claim 1, characterized in that, The decoder comprises multiple decoding layers and multiple dimension adjustment modules, and the decoding layers and the dimension adjustment modules are connected in a one-to-one correspondence; the decoding layer comprises a convolution block and a skip connection, and the number of the convolution blocks and the number of times of the skip connection in the decoding layer increase layer by layer.

4. The human-like perceptual mechanism driven remote sensing image saliency object detection method according to claim 1, characterized in that, where GT j ′ is the GT image after Gaussian blur in the label data, l1 is the L1 loss, S j is the detection result output by the dimension adjustment module corresponding to the decoder layer j.

5. The human-like perceptual mechanism driven remote sensing image saliency object detection method according to claim 1, characterized in that, The inverse distance loss of the detection result is L distance = |pred, GT_distance| l1 + |pred, GT_distance| iou wherein pred is the detection result output by the dimension adjustment module corresponding to the last layer of the decoder, GT_distance is the inverse distance map of the GT image in the label data, and l1 , iou are the L1 loss and the IoU loss, respectively.

Images