Optical remote sensing image detection method, device and storage medium

Abstract

The present invention provides an optical remote sensing image detection method, device and storage medium, belonging to the field of image detection, the method comprises: dividing a plurality of optical remote sensing images to obtain a plurality of optical remote sensing training images and a plurality of optical remote sensing test images; respectively labeling each optical remote sensing training image to obtain an optical remote sensing labeled image; constructing a training model, training the training model according to a plurality of optical remote sensing labeled images to obtain a detection model for an image to be tested; testing the detection model for an image to be tested according to a plurality of optical remote sensing test images to obtain an image detection model; detecting the optical remote sensing image to be tested by the image detection model to obtain a detection result. The present invention realizes the distinction between the target area and the background area, can more accurately predict the saliency map, improves the accuracy of optical remote sensing image detection, and reduces a lot of labor and time.

Description

Technical Field

[0001] The present invention mainly relates to the field of image detection technology, and in particular to an optical remote sensing image detection method, device and storage medium. Background Art

[0002] Traditional salient object detection models usually rely on some low-level features, such as color contrast and background priors. With the great success of deep learning technology in the field of computer vision, models based on fully convolutional networks have become the mainstream of salient object detection architectures. In recent years, research on salient object detection in natural scenes has made great progress, but there are few studies on optical remote sensing images. Usually, the foreground area of ​​optical remote sensing images is similar to the surrounding environment and has objects of different scales.

[0003] At present, some salient object detection methods usually use attention mechanisms to capture semantic contextual relationships, or use pyramid structures to solve the scale variation problem of salient objects. However, these methods fail to introduce the boundary of the object to define the foreground area while targeting objects of different scales, resulting in inaccurate object detection in the foreground area, which makes it difficult to distinguish salient objects in complex scenes. In addition, existing deep learning-based salient object detection algorithms for optical remote sensing images focus on using pixel-by-pixel precise annotation, which requires a lot of labor and time. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an optical remote sensing image detection method, device and storage medium in view of the deficiencies in the prior art.

[0005] The technical solution of the present invention to solve the above technical problem is as follows: an optical remote sensing image detection method, comprising the following steps:

[0006] Importing a plurality of optical remote sensing images, dividing the plurality of optical remote sensing images to obtain an optical remote sensing image training set and an optical remote sensing image test set, wherein the optical remote sensing image training set includes a plurality of optical remote sensing training images, and the optical remote sensing image test set includes a plurality of optical remote sensing test images;

[0007] Annotating each of the optical remote sensing training images respectively to obtain an optical remote sensing annotated image corresponding to each of the optical remote sensing training images;

[0008] Constructing a training model, and performing model training on the training model according to the plurality of optical remote sensing annotated images to obtain an image detection model to be tested;

[0009] Testing the image detection model to be tested according to the plurality of optical remote sensing test images to obtain an image detection model;

[0010] The optical remote sensing image to be tested is imported, and the optical remote sensing image to be tested is tested by using the image detection model to obtain a detection result.

[0011] Another technical solution of the present invention to solve the above technical problem is as follows: an optical remote sensing image detection device, comprising:

[0012] An image segmentation module is used to import a plurality of optical remote sensing images, segment the plurality of optical remote sensing images, and obtain an optical remote sensing image training set and an optical remote sensing image test set, wherein the optical remote sensing image training set includes a plurality of optical remote sensing training images, and the optical remote sensing image test set includes a plurality of optical remote sensing test images;

[0013] An image annotation module, used to annotate each of the optical remote sensing training images to obtain an optical remote sensing annotated image corresponding to each of the optical remote sensing training images;

[0014] A model training module is used to construct a training model, and to perform model training on the training model according to the plurality of optical remote sensing annotated images to obtain a detection model for the image to be tested;

[0015] A model testing module, used for testing the image detection model to be tested according to the plurality of optical remote sensing test images to obtain an image detection model;

[0016] The detection result acquisition module is used to import the optical remote sensing image to be tested, detect the optical remote sensing image to be tested through the image detection model, and obtain the detection result.

[0017] Another technical solution of the present invention to solve the above technical problem is as follows: an optical remote sensing image detection device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the optical remote sensing image detection method as described above is implemented.

[0018] Another technical solution of the present invention to solve the above technical problem is as follows: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the optical remote sensing image detection method as described above is implemented.

[0019] The beneficial effects of the present invention are as follows: a plurality of optical remote sensing training images and a plurality of optical remote sensing test images are obtained by dividing a plurality of optical remote sensing images, optical remote sensing annotated images are obtained by annotating each optical remote sensing training image, a detection model for an image to be tested is obtained by training a training model with a plurality of optical remote sensing annotated images, an image detection model is obtained by testing the detection model for an image to be tested with a plurality of optical remote sensing test images, and a detection result is obtained by detecting the optical remote sensing image to be tested with the image detection model, thereby achieving the distinction between the target area and the background area, being able to more accurately predict the saliency map, improving the accuracy of optical remote sensing image detection, and reducing a lot of labor and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a flow chart of an optical remote sensing image detection method provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a flow chart of a model analysis of an optical remote sensing image detection method provided by an embodiment of the present invention;

[0022] Figure 3 A module block diagram of an optical remote sensing image detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0024] Figure 1 A schematic flow chart of an optical remote sensing image detection method provided by an embodiment of the present invention.

[0025] like Figure 1 As shown, an optical remote sensing image detection method comprises the following steps:

[0026] Importing a plurality of optical remote sensing images, dividing the plurality of optical remote sensing images to obtain an optical remote sensing image training set and an optical remote sensing image test set, wherein the optical remote sensing image training set includes a plurality of optical remote sensing training images, and the optical remote sensing image test set includes a plurality of optical remote sensing test images;

[0027] Annotating each of the optical remote sensing training images respectively to obtain an optical remote sensing annotated image corresponding to each of the optical remote sensing training images;

[0028] Constructing a training model, and performing model training on the training model according to the plurality of optical remote sensing annotated images to obtain an image detection model to be tested;

[0029] Testing the image detection model to be tested according to the plurality of optical remote sensing test images to obtain an image detection model;

[0030] The optical remote sensing image to be tested is imported, and the optical remote sensing image to be tested is tested by using the image detection model to obtain a detection result.

[0031] It should be understood that the plurality of optical remote sensing images may be data in an EORSSD data set, and the plurality of optical remote sensing training images may be data in an EORSSD training set.

[0032] It should be understood that the image data set (i.e., the multiple optical remote sensing images) is divided into a training set (i.e., the optical remote sensing image training set) and a test set (i.e., the optical remote sensing image test set), and the images in the training set (i.e., the optical remote sensing image training set) (i.e., the optical remote sensing training images) are annotated using graffiti annotations, and the annotated images (i.e., the optical remote sensing annotated images) are used as graffiti labels.

[0033] It should be understood that the original image I is annotated with scribbles. rgb The foreground area and background area in the (i.e., the optical remote sensing training image) are arbitrarily labeled, and the foreground, background and unknown pixels in the graffiti annotation are correspondingly recorded as: {1, 2, 0}; and the training data set (i.e., multiple optical remote sensing labeled images) is defined as: where x i is the input image, y i is the corresponding annotation, and N is the size of the training dataset.

[0034] It should be understood that the trained network (ie, the image detection model to be tested) is tested using the test sample set (ie, the plurality of optical remote sensing test images) to evaluate the network (ie, the image detection model to be tested).

[0035] It should be understood that the parameters of the trained optimal model (i.e., the image detection model to be tested) are saved for subsequent testing, and the testing is performed and the test results are evaluated.

[0036] It should be understood that the present invention is an algorithm proposed on two public datasets, ORSSD and EORSSD. The existing ORSSD dataset includes 600 training samples and 200 test samples. It is the first publicly available dataset for the task of salient object detection in optical remote sensing images, but the data scale is relatively small. The EORSSD dataset is an extended dataset of the ORSSD dataset, which contains 2000 images and corresponding pixel-level annotated real images, of which the training set contains 1400 images and the test set contains 600 images.

[0037] In the above embodiment, multiple optical remote sensing training images and multiple optical remote sensing test images are obtained by dividing multiple optical remote sensing images, and optical remote sensing annotated images are obtained by annotating each optical remote sensing training image. The model of the training model is trained by the multiple optical remote sensing annotated images to obtain a detection model for the image to be tested. The image detection model is tested by the multiple optical remote sensing test images to obtain an image detection model. The detection result is obtained by detecting the optical remote sensing image to be tested by the image detection model, thereby achieving the distinction between the target area and the background area, being able to more accurately predict the saliency map, improving the accuracy of optical remote sensing image detection, and reducing a lot of labor and time.

[0038] Optionally, as an embodiment of the present invention, the process of respectively annotating each of the optical remote sensing training images to obtain an optical remote sensing annotated image corresponding to each of the optical remote sensing training images includes:

[0039] The optical remote sensing training images are annotated respectively by using the Quick Selection tool to obtain optical remote sensing annotated images corresponding to the optical remote sensing training images.

[0040] It should be understood that the Quick Selection tool is the Quick Selection tool in Adobe Photoshop, and the marking is implemented by the Quick Selection tool in Adobe Photoshop.

[0041] In the above embodiment, the Quick Selection tool is used to annotate each optical remote sensing training image to obtain an optical remote sensing annotated image, which provides accurate data for subsequent processing, realizes the distinction between the target area and the background area, and can more accurately predict the saliency map.

[0042] Optionally, as an embodiment of the present invention, the process of constructing a training model, training the training model according to the plurality of optical remote sensing annotated images, and obtaining a detection model for the image to be tested includes:

[0043] S1: construct a training model, and extract features from each of the optical remote sensing annotated images according to the training model to obtain a first feature map to be processed corresponding to each of the optical remote sensing annotated images, a second feature map to be processed corresponding to each of the optical remote sensing annotated images, and a third feature map to be processed corresponding to each of the optical remote sensing annotated images;

[0044] S2: performing target positioning feature map analysis on each of the optical remote sensing annotated images and the third feature map to be processed corresponding to each of the optical remote sensing annotated images, to obtain a target positioning feature map corresponding to each of the optical remote sensing annotated images;

[0045] S3: performing edge feature map analysis on each of the first feature maps to be processed, the second feature maps to be processed corresponding to each of the optical remote sensing annotated images, and the third feature maps to be processed corresponding to each of the optical remote sensing annotated images, respectively, to obtain an edge feature map corresponding to each of the optical remote sensing annotated images;

[0046] S4: performing prediction analysis on each of the target positioning feature maps and the edge feature maps corresponding to each of the optical remote sensing annotated images, respectively, to obtain a saliency map corresponding to each of the optical remote sensing annotated images;

[0047] S5: Analyze the loss values ​​of the optical remote sensing annotated images and the saliency maps corresponding to the optical remote sensing annotated images respectively to obtain a detection model for the image to be tested.

[0048] It should be understood that the foreground features of the top-level feature map (i.e., the first feature map to be processed, the second feature map to be processed, and the third feature map to be processed) are extracted through graffiti annotation (i.e., the optical remote sensing annotated image), and the target positioning feature map is obtained using relevant filtering operations; an edge detection network is constructed using the FPN feature pyramid network to extract the edge feature map; the target positioning feature map and the edge feature map are spliced, and the significance prediction of the spliced ​​feature map is performed.

[0049] In the above embodiment, an image detection model is obtained by analyzing the model of multiple optical remote sensing annotated images and an optical remote sensing image test set through a training model, thereby achieving the distinction between the target area and the background area, integrating the multi-scale information and edge information of the image into the learning process of the salient map, and being able to predict the salient map more accurately.

[0050] Optionally, as an embodiment of the present invention, the training model includes a first feature block, a second feature block, a third feature block, a fourth feature block and a fifth feature block, and the process of step S1 includes:

[0051] Performing a first feature extraction on each of the optical remote sensing annotated images using the first feature block to obtain a first initial feature map corresponding to each of the optical remote sensing annotated images;

[0052] Performing a second feature extraction on each of the first initial feature maps using the second feature block to obtain a second initial feature map corresponding to each of the optical remote sensing annotated images;

[0053] Performing a third feature extraction on each of the second initial feature maps through the third feature block to obtain a third initial feature map corresponding to each of the optical remote sensing annotated images, and using the third initial feature map as the first feature map to be processed;

[0054] Performing a fourth feature extraction on each of the first feature maps to be processed by using the fourth feature block, to obtain a second feature map to be processed corresponding to each of the optical remote sensing annotated images;

[0055] The fifth feature block is used to perform a fifth feature extraction on each of the second feature maps to be processed, so as to obtain a third feature map to be processed corresponding to each of the optical remote sensing annotated images.

[0056] It should be understood that the top-level feature map in the VGG16 network (ie, the third feature map to be processed) is extracted.

[0057] It should be understood that the VGG16 fully connected layer is removed and used as the backbone network (i.e., the training model). The backbone network (i.e., the training model) includes five convolution blocks, each of which is composed of several convolutions, and the channels output by each convolution block are 64, 128, 256, 512, and 512 respectively.

[0058] Specifically, the structure of VGG16 without the fully connected layer (i.e., the training model) is as follows:

[0059] The input image (i.e., the optical remote sensing annotated image) has a size of 512x512 and a channel number of 3 (three RGB channels); the first layer of the first convolution block (convolution layer): 512x512x64; the second layer of the first convolution block (convolution layer): 512x512x64; the maximum pooling layer: 256x256x128 (the pooling layer upsamples the image); the first layer of the second convolution block (convolution layer): 256x256x128; the second layer of the second convolution block (convolution layer): 256x256x128; the maximum pooling layer: 128x128x256; the first layer of the third convolution block (convolution layer): 128x128x256; the second layer of the third convolution block (convolution layer) Layer): 128x128x256; the third convolution block third layer (convolution layer): 128x128x256; maximum pooling layer: 64x64x512; the fourth convolution block first layer (convolution layer): 64x64x512; the fourth convolution block second layer (convolution layer): 64x64x512; the fourth convolution block third layer (convolution layer): 64x64x512; maximum pooling layer: 32x32x512; the fifth convolution block first layer (convolution layer): 32x32x512; the fifth convolution block second layer (convolution layer): 32x32x512; the fifth convolution block third layer (convolution layer): 32x32x512; maximum pooling layer: 16x16x512.

[0060] It should be understood that the pre-trained VGG16 network is used as the backbone network (i.e., the training model), and the backbone network (i.e., the training model) includes five convolution blocks, each of which is composed of several convolutions, and the channels output by each convolution block are 64, 128, 256, 512, and 512. The parameters of the network model are initialized, the batch size is set to 6, the initial learning rate is set to 0.0001, and the number of training epochs is set to 60.

[0061] It should be understood that the VGG16 fully connected layer is removed and used as the backbone network (i.e., the training model); and the color image I rgb (i.e., the optical remote sensing annotated image) is input into the pre-trained VGG16 convolutional neural network (i.e., the training model), and the feature map of the last layer of the VGG16 convolutional layer (i.e., the third feature map to be processed) is obtained, which is defined as Where H and W represent the height and width of the feature map, and d represents the number of channels of the feature map.

[0062] In the above embodiment, the features of each optical remote sensing annotated image are extracted by training the model to obtain the first feature map to be processed, the second feature map to be processed, and the third feature map to be processed, thereby achieving the distinction between the target area and the background area, and integrating the multi-scale information and edge information of the image into the learning process of the saliency map.

[0063] Optionally, as an embodiment of the present invention, the process of step S2 includes:

[0064] Multiplying each of the optical remote sensing annotated images and the third feature map to be processed corresponding to each of the optical remote sensing annotated images pixel by pixel to obtain a foreground feature map corresponding to each of the optical remote sensing annotated images;

[0065] Performing average pooling processing on each of the foreground feature maps respectively to obtain a filter corresponding to each of the optical remote sensing annotated images;

[0066] Deep convolution processing is performed on each of the third feature maps to be processed and the filters corresponding to each of the optical remote sensing annotated images to obtain a target positioning feature map corresponding to each of the optical remote sensing annotated images.

[0067] It should be understood that the filter can be a 512×1×1 filter, that is, a "kernel" is more like a 2D weight matrix. A "filter" refers to a 3D structure of multiple Kernels stacked. Here, a 1*1 kernel has 512 channels and is considered 3D.

[0068] It should be understood that the last layer feature map of the VGG16 network (i.e. the third feature map to be processed) and its corresponding graffiti annotation y i (i.e., the optical remote sensing annotated image) is multiplied to obtain a feature map Y containing only foreground features. f (i.e., the foreground feature map); f (i.e., the foreground feature map) is average pooled to obtain a filter kernel of 512×1×1 (i.e., the filter); the filter kernel of 5121×1×1 (i.e., the filter) is combined with the image feature map (i.e., the third feature map to be processed) is subjected to depth-by-depth related convolution to obtain the target positioning feature map Y o .

[0069] It should be understood that the output of the last layer of the VGG16 network (i.e., the third feature map to be processed) is combined with its corresponding graffiti foreground annotation y i (i.e., the optical remote sensing annotated image) is multiplied to obtain a feature map Y containing only foreground features. f (i.e., the foreground feature map); f(i.e., the foreground feature map) is subjected to an average pooling operation to obtain a 512×1×1 filter; the filter is subjected to a depth-wise correlation convolution with the output of the last layer of the VGG16 network (i.e., the third feature map to be processed) to obtain a target positioning feature map Y o .

[0070] Specifically, the image I rgb Input to the network, use VGG16 to extract features, compare the top feature map (i.e. the third feature map to be processed) output by the fifth layer C5 of VGG16 with the marked graffiti annotation y i The foreground area (i.e., the optical remote sensing annotated image) is multiplied pixel by pixel to obtain the foreground feature map Y f , for the foreground feature map Y f After the average pooling operation, a single filter of 512×1×1 is obtained, and deep convolution is performed with C5 (i.e., the third feature map to be processed) through the filter to achieve correlation filtering of image features, that is, to use the correlation between multi-scale visual features of the image and high-level semantic information to mine discriminative features.

[0071] It should be understood that the extracted foreground features (ie, the foreground feature map) are integrated into a 512×1×1 filter, and the filter is used to perform correlation filtering on the image features (ie, the third feature map to be processed) to obtain the target positioning feature map Y o .

[0072] In the above embodiment, the target positioning feature map of each optical remote sensing annotated image and the third feature map to be processed is analyzed to obtain a target positioning feature map, thereby realizing correlation filtering of image features and mining discriminative features by utilizing the correlation between multi-scale visual features of the image and high-level semantic information.

[0073] Optionally, as an embodiment of the present invention, the process of step S3 includes:

[0074] Based on the 1×1 convolutional layer, each of the third feature maps to be processed is horizontally connected to obtain a first pyramid feature map corresponding to each of the optical remote sensing annotated images;

[0075] Performing a first element-by-element addition on each of the first pyramid feature maps and the second feature map to be processed corresponding to each of the optical remote sensing annotated images, respectively, to obtain a second pyramid feature map corresponding to each of the optical remote sensing annotated images;

[0076] performing a second element-by-element addition on each of the second pyramid feature maps and the first feature map to be processed corresponding to each of the optical remote sensing annotated images, respectively, to obtain a third pyramid feature map corresponding to each of the optical remote sensing annotated images;

[0077] The first pyramid feature maps, the second pyramid feature maps corresponding to the optical remote sensing annotated images, and the third pyramid feature maps corresponding to the optical remote sensing annotated images are respectively spliced ​​and calculated by the first formula to obtain a fusion feature map corresponding to each optical remote sensing annotated image. The first formula is:

[0078] P fusion =Cat{P3,P4,P5},

[0079] Among them, P5 is the first pyramid feature map, P4 is the second pyramid feature map, P3 is the third pyramid feature map, {Cat} is the concatenation operation along the channel dimension, P fusion is the fusion feature map;

[0080] The edge map of each fused feature map is calculated by the second formula to obtain the edge feature map corresponding to each optical remote sensing annotated image. The second formula is:

[0081] S e =δ(C RCAB (P fusion )),

[0082] Among them, S e is the edge feature map, P fusion is the fusion feature map, δ is the sigmoid activation function, C RCAB It is the residual channel attention mechanism.

[0083] It should be understood that the feature maps (P3, P4, P5) (i.e., the third pyramid feature map, the second pyramid feature map, and the first pyramid feature map) are output through the last three layers of the FPN feature pyramid network respectively; the feature maps (P3, P4, P5) (i.e., the third pyramid feature map, the second pyramid feature map, and the first pyramid feature map) output by the last three layers of the FPN feature pyramid network are connected in series to obtain a multi-scale feature map (i.e., the fused feature map) output in the series connection stage; the local receptive field of the multi-scale feature map is expanded by using the residual channel attention block and the sigmoid function is used for weighting to obtain the edge feature map S e .

[0084] It should be understood that the FPN feature pyramid network is used to extract the multi-scale features of the image (i.e., the fused feature map), and the residual channel attention block is used to expand the local receptive field of the multi-scale feature map (i.e., the fused feature map), and the sigmoid function is used to weight the feature map with a larger local receptive field to obtain the edge feature map S e .

[0085] Specifically, C1, C2, C3, C4, and C5 (i.e., the first initial feature map, the second initial feature map, the first feature map to be processed, the second feature map to be processed, and the third feature map to be processed) in the VGG16 convolutional neural network are horizontally connected using 1×1 convolution, and element-by-element addition is used to merge the feature maps of the same spatial size of the bottom-up path and the top-down path, and the final output features (P3, P4, and P5) (i.e., the third pyramid feature map, the second pyramid feature map, and the first pyramid feature map). In order to adapt to targets of different scales, a pyramid feature map is generated at each layer through a top-down connection method, and each layer of the pyramid feature map is represented as follows:

[0086]

[0087] Where σ represents the lateral connection implemented by the learnable 1×1 convolutional layer, and τ represents the bilinear interpolation upsampling with a scaling factor of 2. This indicates that the resolution of adjacent layers of the features {P3, P4, P5} (i.e., the third pyramid feature map, the second pyramid feature map, and the first pyramid feature map) differ by a factor of 2. Finally, to maintain the consistency of the feature output size and facilitate the final fusion operation, the fused feature map is represented as follows:

[0088] P fusion =Cat{P3,P4,P5},

[0089] Among them, {P3, P4, P5} represent the output features of the pyramid feature map of 256, 512, and 512 channels. {Cat} represents the concatenation operation along the channel dimension, P fusion is the final splicing result (i.e., the fusion feature map).

[0090] On this basis, the residual channel attention mechanism is introduced to generate different attention to each channel feature by utilizing the interdependence between feature channels. Then the edge map (i.e. the edge feature map) is generated by the sigmoid activation function. The specific formula is as follows:

[0091] S e =δ(C RCAB (P fusion )),

[0092] Where S e is the edge map (i.e., the edge feature map), δ represents the sigmoid activation function, C RCAB It is the residual channel attention mechanism.

[0093] It should be understood that the output of the last layer of each level (when the image is input to the network, that is, the feature map) is selected as the corresponding layer number of the top-down path, and the pixels are added after 1x1 convolution. In general, FPN can also be understood as a model.

[0094] It should be understood that first, C5 (i.e., the third feature map to be processed) is horizontally connected to obtain P5 (i.e., the first pyramid feature map), then P5 (i.e., the first pyramid feature map) and C4 (i.e., the second feature map to be processed) are merged to obtain P4 (i.e., the second pyramid feature map), then P4 (i.e., the second pyramid feature map) and C3 (i.e., the first feature map to be processed) are merged to obtain P3 (i.e., the third pyramid feature map), and P3 (i.e., the third pyramid feature map) and C2 (i.e., the second initial feature map) are merged to obtain P2.

[0095] In the above embodiment, the edge feature maps of the first feature map to be processed, the second feature map to be processed, and the third feature map to be processed are analyzed to obtain edge feature maps, thereby achieving the distinction between the target area and the background area, integrating the multi-scale information and edge information of the image into the learning process of the saliency map, and being able to predict the saliency map more accurately.

[0096] Optionally, as an embodiment of the present invention, the process of step S4 includes:

[0097] Separately splicing the target positioning feature maps and the edge feature maps corresponding to the optical remote sensing annotated images to obtain spliced ​​feature maps corresponding to the optical remote sensing annotated images;

[0098] Based on the 1×1 convolutional layer, each of the spliced ​​feature maps is predicted respectively to obtain a saliency map corresponding to each of the optical remote sensing annotated images.

[0099] It should be understood that the target positioning feature map Y o and the edge feature map S e Merge and perform image saliency prediction through a 1×1 convolutional layer to obtain a saliency map S y .

[0100] It should be understood that the target positioning feature map Y o and the edge feature map S e The concatenation is performed and input into a 1×1 convolutional layer to generate a saliency map.

[0101] In the above embodiment, the feature maps of each target positioning feature map and the edge feature map are spliced ​​to obtain a spliced ​​feature map, and each spliced ​​feature map is predicted based on a 1×1 convolutional layer to obtain a saliency map. The multi-scale information and edge information of the image are simultaneously integrated into the learning process of the saliency map, and the saliency map can be predicted more accurately.

[0102] Optionally, as an embodiment of the present invention, the process of step S5 includes:

[0103] The loss values ​​of the optical remote sensing annotated images and the saliency maps corresponding to the optical remote sensing annotated images are calculated respectively by the third formula to obtain the loss values ​​corresponding to the optical remote sensing annotated images. The third formula is:

[0104] L(θ)=SlogS y +(1-S)log(1-S y ),

[0105] Among them, L(θ) is the loss value, S is the optical remote sensing annotated image, S y is a salient map;

[0106] The parameters of the training model are updated according to all the loss values, and the process returns to step S1 until a preset number of iterations is reached. The training model after the parameter update is used as the image detection model to be tested.

[0107] It should be understood that the loss function is constructed by using the training sample set, automatic differentiation technology, and using the stochastic gradient descent and back propagation algorithm as follows:

[0108] L(θ)=SlogS y +(1-S)log(1-S y ),

[0109] Where, S represents the graffiti annotation (i.e., the optical remote sensing annotated image), S y The final saliency map (i.e., the saliency map) is represented, and the loss function L(θ) (i.e., the loss value) is used to optimize the entire network, update the network parameter θ, and obtain the weight of the training sample set.

[0110] In the above embodiment, the loss values ​​of the optical remote sensing annotated image and the salient map are analyzed respectively to obtain the image detection model to be tested, which increases the accuracy of the training model, realizes the distinction between the target area and the background area, and can more accurately predict the salient map.

[0111] Optionally, as another embodiment of the present invention, Figure 2As shown, a pre-trained VGG16 network is used as a feature extractor to extract features of the image (i.e., the optical remote sensing annotated image) to obtain a convolutional network of C1, C2, C3, C4, and C5 (i.e., the first initial feature map, the second initial feature map, the first feature map to be processed, the second feature map to be processed, and the third feature map to be processed). The top feature map (i.e., the third feature map to be processed) output by the fifth layer C5 of the VGG16 network is combined with the marked graffiti annotation y i The foreground area (i.e., the optical remote sensing annotated image) is multiplied pixel by pixel to obtain the foreground feature map Y f , for the foreground feature map Y f After the average pooling operation, a single filter of 512×1×1 is obtained. Through the filter and C5 (i.e., the third feature map to be processed), deep convolution is performed to realize the correlation filtering of image features, that is, the correlation between the multi-scale visual features of the image and the high-level semantic information is used to mine discriminative features. Multi-scale image features can be extracted through the FPN feature pyramid network, that is, C1, C2, C3, C4, C5 in the VGG16 convolutional neural network (i.e., the first initial feature map, the second initial feature map, the first feature map to be processed, the second feature map to be processed, and the third feature map to be processed) are horizontally connected using 1×1 convolution, and the feature maps of the same spatial size of the bottom-up path and the top-down path are merged by element-by-element addition, and the features (P3, P4, P5) (i.e., the third pyramid feature map, the second pyramid feature map, and the first pyramid feature map) are finally output. In order to adapt to targets of different scales, a pyramid feature map is generated at each layer through a top-down connection method, and each layer of the pyramid feature map is represented as follows:

[0112]

[0113] Where σ represents the lateral connection implemented by the learnable 1×1 convolutional layer, and τ represents the bilinear interpolation upsampling with a scaling factor of 2. This means that the resolution of adjacent layers of features {P3, P4, P5} differs by a factor of 2. Finally, to maintain the consistency of the feature output size, the fused feature map is represented as follows for the final fusion operation:

[0114] P fusion =Cat{P3,P4,P5},

[0115] Among them, {P3, P4, P5} represent the output features of the pyramid feature map of 256, 512, and 512 channels. {Cat} represents the concatenation operation along the channel dimension, P fusion The final stitching result.

[0116] On this basis, the residual channel attention mechanism is introduced to use the interdependence between feature channels to generate different attention for each channel feature. Then the edge map is generated by the sigmoid activation function:

[0117] S e =δ(C RCAB (P fusion ))

[0118] Where S e is the edge map, δ represents the sigmoid activation function, C RCAB It is the residual channel attention mechanism.

[0119] The target is located in the feature map Y o and edge feature map S e Merge and perform image saliency prediction through a 1×1 convolutional layer to obtain a saliency map S y .

[0120] Optionally, as another embodiment of the present invention, after the process of testing the image detection model to be tested based on multiple optical remote sensing test images, four commonly used evaluation indicators can be used to perform performance testing on the test results: Mean Absolute Error (MAE), F-Measure, E-Measure, and S-Measure.

[0121] Among them, (1) Mean Absolute Error (MAE) is used to calculate the difference between the predicted saliency map G(i, j) and the true value S(i, j), as follows:

[0122]

[0123] Where W and H represent the width and height of the image, and i and j represent the horizontal and vertical coordinates of the pixel points in the image.

[0124] (2) F-Measure is the weighted average of precision and recall. β Indicates that the larger its value, the more effective the method is. The formula is as follows:

[0125]

[0126] where β 2 = 0.3, Precision represents precision, Recall represents recall. The precision and recall are calculated by comparing the predicted saliency map with the true value, and the F is obtained by the weighted average of the precision and recall. β .

[0127] (3) E-Measure reflects the correlation between the predicted saliency map and the true value minus the global average, and the formula is as follows:

[0128]

[0129] Where W and H represent the width and height of the image, and Φs(i,j) is the enhanced alignment matrix, which reflects the correlation between the predicted saliency map and the true value minus the overall average of the error between them. The higher the correlation, the better the effect.

[0130] (4) S-Measure is used to measure the structural similarity between the predicted saliency map and the true value. The formula is as follows:

[0131] S m =α×S0+(1-α)×S r

[0132] where α is usually set to 0.5, S0 and S r They represent the target perceived structural similarity and regional structural similarity between the predicted saliency map and the ground truth, respectively. m The larger the value, the more consistent the experimental results are with the true value.

[0133] For these four commonly used evaluation indicators, the smaller the better, and the smaller the error between the true value and the true value, there is no threshold value to determine whether it is good or bad. It can only be said that compared with other existing methods, the numerical value of the present invention is smaller.

[0134] Optionally, as another embodiment of the present invention, the present invention is compared with several salient target detection methods, including DSG, SMD, SSOD, DSS and PoolNet. It is worth noting that DSG, SMD and SSOD are unsupervised / weakly supervised salient target detection methods, while DSS and PoolNet are fully supervised salient target detection methods. For a fair comparison, the present invention is implemented using an NVIDIA RTX A6000 GPU and an open source machine learning library Pytorch, and the present invention shows better performance gain compared with other methods. The experimental results show that the present invention can more effectively capture multi-scale information and distinguish the difference between foreground and background. The test results show that the present invention is superior to other methods.

[0135] Optionally, as another embodiment of the present invention, the present invention can be written in Python language on a Linux operating system platform.

[0136] Optionally, as another embodiment of the present invention, the present invention realizes the distinction between the target area and the background area through correlation filtering, and constructs a multi-scale edge detection network through FPN, and integrates the multi-scale information and edge information of the image into the learning process of the salient map at the same time. The present invention can predict the salient map more accurately.

[0137] Figure 3 A module block diagram of an optical remote sensing image detection device provided by an embodiment of the present invention.

[0138] Optionally, as another embodiment of the present invention, Figure 3 As shown, an optical remote sensing image detection device comprises:

[0139] An image segmentation module is used to import a plurality of optical remote sensing images, segment the plurality of optical remote sensing images, and obtain an optical remote sensing image training set and an optical remote sensing image test set, wherein the optical remote sensing image training set includes a plurality of optical remote sensing training images, and the optical remote sensing image test set includes a plurality of optical remote sensing test images;

[0140] An image annotation module, used to annotate each of the optical remote sensing training images to obtain an optical remote sensing annotated image corresponding to each of the optical remote sensing training images;

[0141] A model training module is used to construct a training model, and to perform model training on the training model according to the plurality of optical remote sensing annotated images to obtain a detection model for the image to be tested;

[0142] A model testing module, used for testing the image detection model to be tested according to the plurality of optical remote sensing test images to obtain an image detection model;

[0143] The detection result acquisition module is used to import the optical remote sensing image to be tested, detect the optical remote sensing image to be tested through the image detection model, and obtain the detection result.

[0144] Optionally, another embodiment of the present invention provides an optical remote sensing image detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the optical remote sensing image detection method described above is implemented. The device may be a computer or other device.

[0145] Optionally, another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the optical remote sensing image detection method as described above is implemented.

[0146] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An optical remote sensing image detection method, characterized in that: The steps include: Importing a plurality of optical remote sensing images, dividing the plurality of optical remote sensing images to obtain an optical remote sensing image training set and an optical remote sensing image test set, wherein the optical remote sensing image training set includes a plurality of optical remote sensing training images, and the optical remote sensing image test set includes a plurality of optical remote sensing test images; Annotating each of the optical remote sensing training images respectively to obtain an optical remote sensing annotated image corresponding to each of the optical remote sensing training images; Constructing a training model, and performing model training on the training model according to the plurality of optical remote sensing annotated images to obtain an image detection model to be tested; Testing the image detection model to be tested according to the plurality of optical remote sensing test images to obtain an image detection model; Importing the optical remote sensing image to be tested, and detecting the optical remote sensing image to be tested by using the image detection model to obtain a detection result; The process of constructing a training model and training the training model according to the plurality of optical remote sensing annotated images to obtain a detection model for the image to be tested includes: S1: construct a training model, and extract features from each of the optical remote sensing annotated images according to the training model to obtain a first feature map to be processed corresponding to each of the optical remote sensing annotated images, a second feature map to be processed corresponding to each of the optical remote sensing annotated images, and a third feature map to be processed corresponding to each of the optical remote sensing annotated images; S2: performing target positioning feature map analysis on each of the optical remote sensing annotated images and the third feature map to be processed corresponding to each of the optical remote sensing annotated images, to obtain a target positioning feature map corresponding to each of the optical remote sensing annotated images; S3: performing edge feature map analysis on each of the first feature maps to be processed, the second feature maps to be processed corresponding to each of the optical remote sensing annotated images, and the third feature maps to be processed corresponding to each of the optical remote sensing annotated images, respectively, to obtain an edge feature map corresponding to each of the optical remote sensing annotated images; S4: performing prediction analysis on each of the target positioning feature maps and the edge feature maps corresponding to each of the optical remote sensing annotated images, respectively, to obtain a saliency map corresponding to each of the optical remote sensing annotated images; S5: analyzing the loss values ​​of the optical remote sensing annotated images and the saliency maps corresponding to the optical remote sensing annotated images respectively to obtain a detection model for the image to be tested; The training model includes a first feature block, a second feature block, a third feature block, a fourth feature block and a fifth feature block, and the process of step S1 includes: Performing a first feature extraction on each of the optical remote sensing annotated images using the first feature block to obtain a first initial feature map corresponding to each of the optical remote sensing annotated images; Performing a second feature extraction on each of the first initial feature maps using the second feature block to obtain a second initial feature map corresponding to each of the optical remote sensing annotated images; Performing a third feature extraction on each of the second initial feature maps through the third feature block to obtain a third initial feature map corresponding to each of the optical remote sensing annotated images, and using the third initial feature map as the first feature map to be processed; Performing a fourth feature extraction on each of the first feature maps to be processed by using the fourth feature block, to obtain a second feature map to be processed corresponding to each of the optical remote sensing annotated images; Performing a fifth feature extraction on each of the second feature maps to be processed by using the fifth feature block, to obtain a third feature map to be processed corresponding to each of the optical remote sensing annotated images; The process of step S2 includes: Multiplying each of the optical remote sensing annotated images and the third feature map to be processed corresponding to each of the optical remote sensing annotated images pixel by pixel to obtain a foreground feature map corresponding to each of the optical remote sensing annotated images; Performing average pooling processing on each of the foreground feature maps respectively to obtain a filter corresponding to each of the optical remote sensing annotated images; Performing deep convolution processing on each of the third feature maps to be processed and the filters corresponding to each of the optical remote sensing annotated images, respectively, to obtain a target positioning feature map corresponding to each of the optical remote sensing annotated images; The process of step S3 includes: based on The convolution layer horizontally connects each of the third feature maps to be processed to obtain a first pyramid feature map corresponding to each of the optical remote sensing annotated images; Performing a first element-by-element addition on each of the first pyramid feature maps and the second feature map to be processed corresponding to each of the optical remote sensing annotated images, respectively, to obtain a second pyramid feature map corresponding to each of the optical remote sensing annotated images; performing a second element-by-element addition on each of the second pyramid feature maps and the first feature map to be processed corresponding to each of the optical remote sensing annotated images, respectively, to obtain a third pyramid feature map corresponding to each of the optical remote sensing annotated images; The first pyramid feature maps, the second pyramid feature maps corresponding to the optical remote sensing annotated images, and the third pyramid feature maps corresponding to the optical remote sensing annotated images are respectively spliced ​​and calculated by the first formula to obtain a fusion feature map corresponding to each optical remote sensing annotated image. The first formula is: ， in, is the first pyramid feature map, is the second pyramid feature map, is the third pyramid feature map, is the concatenation operation along the channel dimension, is the fusion feature map; The edge map of each fused feature map is calculated by the second formula to obtain the edge feature map corresponding to each optical remote sensing annotated image. The second formula is: ， in, is the edge feature map, is the fusion feature map, for Activation function, It is the residual channel attention mechanism.

2. The optical remote sensing image detection method according to claim 1, characterized in that: The process of respectively labeling each of the optical remote sensing training images to obtain an optical remote sensing labeled image corresponding to each of the optical remote sensing training images comprises: The optical remote sensing training images are annotated respectively by using the Quick Selection tool to obtain optical remote sensing annotated images corresponding to the optical remote sensing training images.

3. The optical remote sensing image detection method according to claim 1, characterized in that: The process of step S4 includes: Separately splicing the target positioning feature maps and the edge feature maps corresponding to the optical remote sensing annotated images to obtain spliced ​​feature maps corresponding to the optical remote sensing annotated images; based on The convolution layer predicts each of the spliced ​​feature maps respectively to obtain a saliency map corresponding to each of the optical remote sensing annotated images.

4. The optical remote sensing image detection method according to claim 1, characterized in that: The process of step S5 includes: The loss values ​​of the optical remote sensing annotated images and the saliency maps corresponding to the optical remote sensing annotated images are calculated respectively by the third formula to obtain the loss values ​​corresponding to the optical remote sensing annotated images. The third formula is: ， in, is the loss value, Annotating images for optical remote sensing, is a salient map; The parameters of the training model are updated according to all the loss values, and the process returns to step S1 until a preset number of iterations is reached. The training model after the parameter update is used as the image detection model to be tested.

5. An optical remote sensing image detection device, characterized in that: include: An image segmentation module, used for importing a plurality of optical remote sensing images, segmenting the plurality of optical remote sensing images, and obtaining an optical remote sensing image training set and an optical remote sensing image test set, wherein the optical remote sensing image training set includes a plurality of optical remote sensing training images, and the optical remote sensing image test set includes a plurality of optical remote sensing test images; An image annotation module, used to annotate each of the optical remote sensing training images to obtain an optical remote sensing annotated image corresponding to each of the optical remote sensing training images; A model training module is used to construct a training model, and to perform model training on the training model according to the plurality of optical remote sensing annotated images to obtain a detection model for the image to be tested; A model testing module, used for testing the image detection model to be tested according to the plurality of optical remote sensing test images to obtain an image detection model; A detection result acquisition module is used to import the optical remote sensing image to be tested, detect the optical remote sensing image to be tested through the image detection model, and obtain the detection result; The model training module is specifically used for: S1: construct a training model, and extract features from each of the optical remote sensing annotated images according to the training model to obtain a first feature map to be processed corresponding to each of the optical remote sensing annotated images, a second feature map to be processed corresponding to each of the optical remote sensing annotated images, and a third feature map to be processed corresponding to each of the optical remote sensing annotated images; S2: performing target positioning feature map analysis on each of the optical remote sensing annotated images and the third feature map to be processed corresponding to each of the optical remote sensing annotated images, to obtain a target positioning feature map corresponding to each of the optical remote sensing annotated images; S3: performing edge feature map analysis on each of the first feature maps to be processed, the second feature maps to be processed corresponding to each of the optical remote sensing annotated images, and the third feature maps to be processed corresponding to each of the optical remote sensing annotated images, respectively, to obtain an edge feature map corresponding to each of the optical remote sensing annotated images; S4: performing prediction analysis on each of the target positioning feature maps and the edge feature maps corresponding to each of the optical remote sensing annotated images, respectively, to obtain a saliency map corresponding to each of the optical remote sensing annotated images; S5: analyzing the loss values ​​of the optical remote sensing annotated images and the saliency maps corresponding to the optical remote sensing annotated images respectively to obtain a detection model for the image to be tested; The training model includes a first feature block, a second feature block, a third feature block, a fourth feature block and a fifth feature block, and the process of step S1 includes: Performing a first feature extraction on each of the optical remote sensing annotated images using the first feature block to obtain a first initial feature map corresponding to each of the optical remote sensing annotated images; Performing a second feature extraction on each of the first initial feature maps using the second feature block to obtain a second initial feature map corresponding to each of the optical remote sensing annotated images; Performing a third feature extraction on each of the second initial feature maps through the third feature block to obtain a third initial feature map corresponding to each of the optical remote sensing annotated images, and using the third initial feature map as the first feature map to be processed; Performing a fourth feature extraction on each of the first feature maps to be processed by using the fourth feature block, to obtain a second feature map to be processed corresponding to each of the optical remote sensing annotated images; Performing a fifth feature extraction on each of the second feature maps to be processed by using the fifth feature block, to obtain a third feature map to be processed corresponding to each of the optical remote sensing annotated images; The process of step S2 includes: Multiplying each of the optical remote sensing annotated images and the third feature map to be processed corresponding to each of the optical remote sensing annotated images pixel by pixel to obtain a foreground feature map corresponding to each of the optical remote sensing annotated images; Performing average pooling processing on each of the foreground feature maps respectively to obtain a filter corresponding to each of the optical remote sensing annotated images; Performing deep convolution processing on each of the third feature maps to be processed and the filters corresponding to each of the optical remote sensing annotated images, respectively, to obtain a target positioning feature map corresponding to each of the optical remote sensing annotated images; The process of step S3 includes: based on The convolution layer horizontally connects each of the third feature maps to be processed to obtain a first pyramid feature map corresponding to each of the optical remote sensing annotated images; Performing a first element-by-element addition on each of the first pyramid feature maps and the second feature map to be processed corresponding to each of the optical remote sensing annotated images, respectively, to obtain a second pyramid feature map corresponding to each of the optical remote sensing annotated images; performing a second element-by-element addition on each of the second pyramid feature maps and the first feature map to be processed corresponding to each of the optical remote sensing annotated images, respectively, to obtain a third pyramid feature map corresponding to each of the optical remote sensing annotated images; The first pyramid feature maps, the second pyramid feature maps corresponding to the optical remote sensing annotated images, and the third pyramid feature maps corresponding to the optical remote sensing annotated images are respectively spliced ​​and calculated by the first formula to obtain a fusion feature map corresponding to each optical remote sensing annotated image. The first formula is: ， in, is the first pyramid feature map, is the second pyramid feature map, is the third pyramid feature map, is the concatenation operation along the channel dimension, is the fusion feature map; The edge map of each fused feature map is calculated by the second formula to obtain the edge feature map corresponding to each optical remote sensing annotated image. The second formula is: ， in, is the edge feature map, is the fusion feature map, for Activation function, It is the residual channel attention mechanism.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the optical remote sensing image detection method according to any one of claims 1 to 4 is implemented.

Images