Remote sensing image saliency target detection method driven by human-like perception mechanism

Abstract

The invention belongs to the field of image analysis and processing, and relates to a human-like perception mechanism-driven remote sensing image saliency target detection method, which comprises the following steps of: obtaining a to-be-detected remote sensing image and inputting the to-be-detected remote sensing image into a trained target detection model to obtain a detection result; the training process of the target detection model comprises the following steps: acquiring a data set X, and preprocessing the X to obtain a preprocessed remote sensing image and label data thereof; inputting the preprocessed remote sensing image into a MiT network to obtain characteristics and high-frequency information; inputting the features and the high-frequency information into a high-frequency feature supplementing module to obtain enhanced features; inputting the enhanced features into a decoder to obtain a detection result; updating parameters of the target detection model according to the detection result and the label data until a trained target detection model is obtained; according to the method, high-frequency information around pixels of the remote sensing image is extracted through wavelet transform, high-frequency information supplementation is carried out on the features of the remote sensing image through guide filtering, and therefore the features can better supplement significant targets.

Description

Technical Field

[0001] The invention belongs to the field of image analysis and processing, and relates to a remote sensing image salient target detection method driven by a human-like perception mechanism. Background Art

[0002] Nowadays, with the development of the times, the application of computer vision has become a hot topic. Among them, salient object detection is a widely used technology in the field of computer vision. Its purpose is to accurately identify and highlight the most prominent and eye-catching target areas in images or videos, so as to help users quickly locate key information and improve efficiency. Salient object detection can also assist in visual tasks such as target tracking, image segmentation, and scene understanding, providing valuable prior information for them.

[0003] The salient target detection of remote sensing images is mainly used in the fields of agricultural detection, urban planning, environmental detection, geological disaster detection, etc. It has important application value. Since remote sensing images are significantly different from natural images in terms of shooting equipment, scenes and angles, the two have different resolutions, data types and scales. At the same time, the salient target detection of remote sensing images generally detects surface cover, topography, etc., which present a large range of continuity and spatial structure characteristics. In some scenes, there are even no prominent areas. Therefore, the salient target detection of remote sensing images is significantly different from that of natural images, which also makes the salient target detection methods of the two data forms have many differences. In particular, the complex topological structure in remote sensing images has a great impact on the salient target detection of remote sensing images.

[0004] At present, most of the salient target detection methods of remote sensing images follow the development path of natural images. However, due to the differences from natural images, the salient target detection methods of remote sensing images are subject to many conditions. There are two main solutions for many methods. First, use auxiliary information to gain the final result. That is, using edge information and skeleton information to enhance the results and features is a way to directly enhance the final result. Second, expand the receptive field of the feature and enhance the local information. That is, continuously expand the receptive field of the generated features, and at the same time enhance the local information on this basis. This method is in line with the idea of ​​current segmentation tasks such as semantic segmentation, that is, to obtain a good feature to help achieve a good result. In recent years, the proposed GeleNet is a framework for the second solution, which has alleviated the detection of difficult samples to a certain extent.

[0005] However, these frameworks all use the idea of ​​semantic segmentation to solve the problem of salient object detection. They do not take into account the characteristics of the salient object detection task, nor do they take into account the characteristics of remote sensing images. Therefore, problems such as missed detection and interference from shadows often occur, which affects the quality of the detection results to a certain extent. Summary of the invention

[0006] In order to solve the above problems in the prior art, the present invention adopts a remote sensing image salient target detection method driven by a human-like perception mechanism, comprising: obtaining a remote sensing image to be detected, inputting the remote sensing image to be detected into a trained salient target detection model, and obtaining a detection result; the salient target detection model comprises: a MiT network, a high-frequency feature supplementation module HFSM and a decoder; the training process of the salient target detection model comprises:

[0007] S1. Obtain a remote sensing image salient target detection image dataset X, and preprocess the dataset X to obtain a preprocessed dataset X′; the preprocessed dataset X′ includes: a preprocessed remote sensing image and its label data;

[0008] S2, input the preprocessed remote sensing image into the MiT network to obtain features and high-frequency information; wherein MiT is a hybrid visual transformer;

[0009] S3, inputting the features and high-frequency information into the high-frequency feature supplementation module HFSM to obtain enhanced features;

[0010] S4, input the enhanced features into the decoder to obtain the detection result;

[0011] S5. Calculate the loss function value according to the detection results and label data, and update the parameters of the salient object detection model according to the loss function value. When the loss function value is minimum, the trained salient object detection model is obtained.

[0012] The data set X includes: remote sensing images and their GT images; the preprocessing of the data set X includes:

[0013] S11, normalizing, randomly rotating, and randomly cropping the remote sensing image to obtain a preprocessed remote sensing image;

[0014] S12, performing Gaussian blur on the GT image to obtain a Gaussian blurred GT image; performing inverse distance calculation on the GT image to obtain an inverse distance map; using the Gaussian blurred GT image and the inverse distance map as label data of the preprocessed remote sensing image; wherein the GT image is a real situation image.

[0015] The MiT network includes: N MiT modules M i, i∈[1,N], N MiT modules are connected in series in sequence; the MiT network processes the data in the preprocessed data set X′ including:

[0016] S21, input the preprocessed remote sensing image into the MiT module M1, and obtain the feature f 1 and high frequency information

[0017] S22, feature f 1 Input MiT module M2 and get feature f 2 and high frequency information

[0018] S23, feature f i-1 Enter MiT module M i , and get the feature f i and high frequency information

[0019] S24. Repeat step S23 until all MiT modules are passed.

[0020] The MiT module includes: wavelet transform module and feature extraction module; MiT module M i The remote sensing image processing includes: i-1 Input the wavelet transform module to obtain the intermediate feature f i,* and high frequency information The intermediate feature f i,* Input the feature extraction module to obtain the feature f i .

[0021] The high-frequency feature supplementation module HFSM includes: N submodules HFSM-i, i∈[1,N]; the high-frequency feature supplementation module HFSM processes the features and high-frequency information by: i and high frequency information Input the corresponding HFSM-i to get the enhanced features

[0022] HFSM-i includes a first guided filtering module, a second guided filtering module, a third guided filtering module and a convolution block. include: HFSM-i for feature f i and high frequency information The processing includes: transforming the feature f i and high frequency information Input the first guided filtering module and transform the feature f i and high frequency information Input the second guided filtering module and transform the feature f i and high frequency information The third guided filtering module is input, the outputs of the first guided filtering module, the second guided filtering module, and the third guided filtering module are combined, and the combined result is input into the convolution block to obtain enhanced features; wherein, Respectively represents the high-frequency information in the horizontal, vertical and diagonal directions.

[0023] The decoder includes multiple decoding layers and multiple dimension adjustment modules, and the decoding layers are connected to the dimension adjustment modules in a one-to-one correspondence; the decoding layers include convolution blocks and jump connections, and the number of convolution blocks and the number of jump connections in the decoding layers increase layer by layer.

[0024] Loss function L total for:

[0025]

[0026] in, and They are the L1 loss and IoU loss of the detection results, L distance is the inverse distance loss of the detection result, K is the number of decoder levels, and j is the index of the decoder level.

[0027] Among them, GT′ j is the GT image after Gaussian blur, || l1 is the L1 loss, S j It is the detection result output by the dimension adjustment module corresponding to the decoder level j.

[0028] The inverse distance loss of the detection result is:

[0029] L distance =|pred,GT_distance| l1 +|pred,GT_distance| iou

[0030] Among them, pred is the detection result output by the dimension adjustment module corresponding to the last level of the decoder, GT_distance is the inverse distance map of the GT image, || l1 、|| iou They are L1 loss and IoU loss respectively.

[0031] Beneficial effects:

[0032] 1. Starting from the process of human perception of objects, the present invention first uses the wavelet transform module of the MiT network from bottom to top to extract the high-frequency information around the pixels of the remote sensing image, that is, the shape features of the object, and then uses the guided filter from top to bottom to supplement the features output by the MiT network with the high-frequency information, so that the features can be guided according to the shape features to better supplement the significant targets, quickly locate the area of ​​interest and refine it, thereby reducing the missed detection of the target and reducing the interference of the shadow of the remote sensing image; 2. The present invention adopts a new hybrid loss, which includes inverse distance loss, IoU loss and L1 loss; wherein the inverse distance loss requires the inverse distance calculation of the GT map, so that the edge and significant target positioning of the final result can be better enhanced, and the improved IoU loss and L1 loss also simulate the process from coarse to fine of human perception, and accelerate the convergence of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a method for detecting salient objects in remote sensing images driven by a human-like perception mechanism provided in an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of a target detection model provided by an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of a Wavelet Block provided by an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of a submodule HFSM-i provided in an embodiment of the present invention;

[0037] Figure 5 A schematic diagram of a decoder provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] like Figure 1 , Figure 2As shown, the present invention adopts a remote sensing image salient target detection method driven by a human-like perception mechanism, including: obtaining an optical remote sensing image (ORSI) to be detected, inputting the optical remote sensing image to be detected into a trained salient target detection model, and obtaining a detection result; the salient target detection model includes: a MiT network, a high-frequency feature supplementation module HFSM and a decoder; the training process of the salient target detection model includes:

[0040] S1. Obtain a remote sensing image salient target detection image dataset X, and preprocess the dataset X to obtain a preprocessed dataset X′; the preprocessed dataset X′ includes: a preprocessed remote sensing image and its label data;

[0041] The data set X includes: remote sensing images and their GT images; the preprocessing of the data set X includes:

[0042] S11, normalizing, randomly rotating, and randomly cropping the remote sensing image to obtain a remote sensing image with a uniform size after preprocessing; the size is H×W×3, H=W=512;

[0043] S12, perform Gaussian blurring on the GT image to different degrees to obtain the Gaussian blurred GT image GT′ j , calculate the inverse distance of the GT image to obtain the inverse distance map, and use the Gaussian blurred GT image and the inverse distance map as the label data of the preprocessed remote sensing image; among them, when j is 1, GT′ j represents the GT image without Gaussian blur processing. When j is not 1, GT′ j Indicates that in the previous GT′ j-1 The result is obtained by applying Gaussian blur with a kernel of 5 and a variance of 2 on the basis of . GT image is the Ground Truth image, that is, the real situation image or reference image.

[0044] S2, input the preprocessed remote sensing image into the MiT network to obtain features and high-frequency information;

[0045] MiT network is a backbone network of Transformer. MiT (MixVisionTransformer) is a mixed vision transformer. MiT network includes: N MiT blocks M i , i∈[1,N], N MiT modules are connected in series in sequence; the MiT network processes the preprocessed remote sensing image including:

[0046] S21, input the preprocessed remote sensing image into the MiT module M1, and obtain the feature f 1 and high frequency information

[0047] S22, feature f 1 Input MiT module M2 and get feature f 2 and high frequency information

[0048] S23, feature f i-1 Enter MiT module M i , and get the feature f i and high frequency information

[0049] S24. Repeat step S23 until all MiT modules are passed.

[0050] The MiT module includes: a wavelet transform module (Wavelet Block) and a feature extraction module. The feature extraction module includes: a combination of L forward feedback modules (Mix-FFN) and an efficient self-attention module (Efficient Self-Attn); the wavelet transform module takes the result of the previous MiT module as input, and the first wavelet transform takes the remote sensing image as input.

[0051] like Figure 3 As shown, MiT module M i The remote sensing image processing includes: i-1 Perform wavelet transform to separate high-frequency information and obtain four components in, is low frequency information, They represent high-frequency information in the horizontal, vertical and diagonal directions respectively. The purpose of retaining the high-frequency information separately is to simulate the first step of human perception of objects, that is, abstracting the shape characteristics of the object from the bottom up based on the high-frequency information around the image pixels.

[0052] Use convolution combination to combine the features f i-1 Processing is performed to obtain the intermediate feature f i The convolutional layer consists of a convolutional layer, a batch normalization layer, and an activation layer. Its purpose is to reduce the size of the feature while expanding the channel dimension C of the feature. i This enables features to obtain richer semantic information;

[0053] The intermediate feature f i,* Input feature extraction module, so that the features can pay more attention to the global information and expand the receptive field, so that the final feature f i Indicates richer information.

[0054] S3, inputting the feature and high-frequency information into the high-frequency feature supplementation module HFSM to supplement the feature with high-frequency information to obtain enhanced features;

[0055] like Figure 4 As shown, the high-frequency feature supplementation module HFSM includes: N submodules HFSM-i, i∈[1,N]; the high-frequency feature supplementation module HFSM processes the feature and high-frequency information by: i and high frequency information Input the corresponding HFSM-i to get the enhanced features

[0056] HFSM-i includes a first guided filter module (Guided Filter), a second guided filter module, a third guided filter module and a convolution block (Convs); the convolution block includes: a convolution layer, a batch normalization layer and an activation layer.

[0057] HFSM-i for feature f i and high frequency information The processing includes: transforming the feature f i and high frequency information Input the first guided filtering module and transform the feature f i and high frequency information Input the second guided filtering module and transform the feature f i and high frequency information The third guided filter module is input, and the outputs of the first guided filter module, the second guided filter module, and the third guided filter module are combined (concatenation), and the combined result is input into the convolution block to obtain enhanced features. The main function of this module is to imitate the second step of human perception of objects, using the high-frequency information above to guide the features to focus on the significant target area, scanning the entire field of view scene from top to bottom to identify the size, direction and other features of the object, quickly locate the area of ​​interest and refine it.

[0058] Different high-frequency information leads to different guidance results, which are shown as follows:

[0059]

[0060] Among them, Guided() is a guided filter, which uses three high-frequency information as a guide map to guide the features with different high-frequency features.

[0061] Combining the outputs of the first guided filtering module, the second guided filtering module, and the third guided filtering module includes:

[0062]

[0063] in, Indicates enhanced features, convs is a convolution block, and cat indicates that the result will be guided Stitching is done along the channel dimension.

[0064] In one embodiment, the number N of MiT modules and sub-modules HFSM-i is 4.

[0065] S4, input the enhanced features into the decoder to obtain the detection result;

[0066] The enhanced features obtained by the submodule HFSM-i are directly used in the decoder. The decoder includes K decoding layers. The K decoding layers are designed according to the idea of ​​Unet++ and adopt the traditional pyramid structure. The number of convolution blocks and the number of jump connections are gradually increased from shallow to deep. The purpose is to fully interact between features and better utilize the context information between adjacent features, so as to better locate and refine significant targets. The decoder also includes multiple dimension adjustment modules, which are connected to the decoding layers one by one. The dimension adjustment module includes a convolution layer (Conv) and a convolution block (Convs), so that the final output result of the decoder corresponds to the dimension of the label data.

[0067] In one embodiment, Figure 5 As shown, the number of layers K of the decoder is 3, and the number of convolution blocks (convs) contained in each layer is 1, 2, and 3 respectively.

[0068] S5. Calculate the loss function value according to the detection results and label data, and update the parameters of the salient object detection model according to the loss function value. When the loss function value is minimum, the trained salient object detection model is obtained.

[0069] Loss function L total for:

[0070]

[0071] L distance =|pred,GT_distance| l1 +|pred,GT_distance| iou

[0072] in, and They are the L1 loss and IoU loss of the detection results, L1 loss is the mean absolute error (MAE), IoU loss is the intersection over union loss (Intersection over Union Loss), L distanceis the inverse distance loss of the detection result, prde is the detection result output by the dimension adjustment module corresponding to the last level of the decoder, and S j is the detection result output by the dimension adjustment module corresponding to decoder level j, GT′ j is the GT image after Gaussian blur of different degrees, GT_distance is the inverse distance map of the GT image, || l1 、|| iou They are L1 loss and IoU loss respectively.

[0073] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting salient objects in remote sensing images driven by a human-like perception mechanism, characterized in that: include: Obtain a remote sensing image to be detected, input the remote sensing image to be detected into a trained salient target detection model, and obtain a detection result; The salient object detection model includes: MiT network, high-frequency feature supplementation module HFSM and decoder; the training process of the salient object detection model includes: S1. Obtain a remote sensing image salient target detection image dataset X, and preprocess the dataset X to obtain a preprocessed dataset X ′ ; Preprocessed dataset X ′ Includes: preprocessed remote sensing images and their label data; S2, input the preprocessed remote sensing image into the MiT network to obtain features and high-frequency information; wherein MiT is a hybrid visual transformer; S3, inputting the features and high-frequency information into the high-frequency feature supplementation module HFSM to obtain enhanced features; S4, input the enhanced features into the decoder to obtain the detection result; S5. Calculate the loss function value according to the detection results and label data, and update the parameters of the salient object detection model according to the loss function value. When the loss function value is minimum, the trained salient object detection model is obtained.

2. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 1, characterized in that: The data set X includes: remote sensing images and their GT images; the preprocessing of the data set X includes: S11, normalizing, randomly rotating, and randomly cropping the remote sensing image to obtain a preprocessed remote sensing image; S12, performing Gaussian blur on the GT image to obtain a Gaussian blurred GT image; performing inverse distance calculation on the GT image to obtain an inverse distance map; using the Gaussian blurred GT image and the inverse distance map as label data of the preprocessed remote sensing image; wherein the GT image is a real situation image.

3. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 1, characterized in that: The MiT network includes: N MiT modules M i , i∈[1,N], N MiT modules are connected in series; the MiT network processes the preprocessed data set X ′ The data processing includes: S21, input the preprocessed remote sensing image into the MiT module M1, and obtain the feature f 1 and high frequency information S22, feature f 1 Input MiT module M2 and get feature f 2 and high frequency information S23, feature f i-1 Enter MiT module M i , and get the feature f i and high frequency information S24. Repeat step S23 until all MiT modules are passed.

4. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 3, characterized in that: The MiT module includes: wavelet transform module and feature extraction module; MiT module M i The remote sensing image processing includes: i-1 Input the wavelet transform module to obtain the intermediate feature f i,* and high frequency information The intermediate feature f i,* Input the feature extraction module to obtain the feature f i .

5. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 3, characterized in that: The high-frequency feature supplementation module HFSM includes: N submodules HFSM-i, i∈[1,N]; the high-frequency feature supplementation module HFSM processes the features and high-frequency information by: i and high frequency information Input the corresponding HFSM-i to get the enhanced features 6. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 5, characterized in that: HFSM-i includes a first guided filtering module, a second guided filtering module, a third guided filtering module and a convolution block. include: HFSM-i for feature f i and high frequency information The processing includes: transforming the feature f i and high frequency information Input the first guided filtering module and transform the feature f i and high frequency information Input the second guided filtering module and transform the feature f i and high frequency information The third guided filtering module is input, the outputs of the first guided filtering module, the second guided filtering module, and the third guided filtering module are combined, and the combined result is input into the convolution block to obtain enhanced features; wherein, Respectively represents the high-frequency information in the horizontal, vertical and diagonal directions.

7. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 1, characterized in that: The decoder includes multiple decoding layers and multiple dimension adjustment modules, and the decoding layers are connected to the dimension adjustment modules in a one-to-one correspondence; the decoding layers include convolution blocks and jump connections, and the number of convolution blocks and the number of jump connections in the decoding layers increase layer by layer.

8. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 1, characterized in that: Loss function L total for: in, and They are the L1 loss and IoU loss of the detection results, L distance is the inverse distance loss of the detection result, K is the number of decoder levels, and j is the index of the decoder level.

9. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 8, characterized in that: Among them, GT j ′ is the GT image after Gaussian blur in the label data, || l1 is the L1 loss, S j It is the detection result output by the dimension adjustment module corresponding to the decoder level j.

10. The method for detecting salient objects in remote sensing images driven by a human-like perception mechanism according to claim 8, characterized in that: The inverse distance loss of the detection result is: L distance =|pred,GT_distance| l1 +|pred,GT_distance| iou Among them, pred is the detection result output by the dimension adjustment module corresponding to the last level of the decoder, GT_distance is the inverse distance map of the GT image in the label data, || l1 、|| iou They are L1 loss and IoU loss respectively.

Images