Optical remote sensing image salient target detection method and device, equipment and storage medium

Abstract

The invention relates to an optical remote sensing image salient target detection method, device and equipment and a storage medium, and the method comprises the steps: employing a lightweight encoder to extract feature maps of multiple layers of an optical remote sensing image layer by layer from a shallow layer to a deep layer; a side network of a specific task including a multi-scale semantic difference attention unit and a compact spatial feature fusion unit is used to carry out cascade feature fusion and enhancement on feature images of adjacent levels, so that a shallow-level feature map is improved by a deep-level feature map, and a multi-level spatial information fusion enhanced feature map is obtained; and finally, carrying out multi-stage step-by-step decoding on the multi-level spatial information fusion enhancement feature map from a deep layer to a shallow layer by adopting a lightweight decoder to obtain decoding output, and generating a prediction saliency map according to the decoding output by adopting a detection head so as to realize detection of a salient target. By adopting the method, the detection accuracy can be ensured while the salient target detection efficiency can be effectively improved for the optical remote sensing image.

Description

Technical Field

[0001] The present application relates to the technical field of optical remote sensing image target detection, and in particular to an optical remote sensing image salient target detection method, device, equipment and storage medium. Background Art

[0002] Salient object detection (SOD) strives to replicate the ability of the human visual system to quickly identify objects / regions that attract human attention as a key preprocessing step in various computer vision fields, such as object detection [, object tracking, image quality assessment, and image description. In addition to natural scene images (NSIs), SOD has been widely explored in multiple domains, including videos, RGB-D data, and optical remote sensing images (ORSIs).

[0003] We apply salient object detection in optical remote sensing images, called ORSI-SOD. Compared with NSIs, ORSIs exhibit diverse object scales and types, as well as complex backgrounds caused by high shooting angles and large-scale equipment.

[0004] However, the superior performance of existing state-of-the-art ORSI-SOD models often relies on complex architectures or heavy encoders, which introduce a large number of parameters and computational requirements, limiting their deployment on resource-constrained devices such as unmanned aerial vehicles (UAVs) and satellites. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, equipment and storage medium for detecting significant targets in optical remote sensing images that can improve efficiency while ensuring accuracy in response to the above technical problems.

[0006] A method for detecting salient objects in an optical remote sensing image, the method comprising:

[0007] Acquiring an optical remote sensing image for salient target detection;

[0008] A lightweight encoder is used to extract feature maps of multiple levels of the optical remote sensing image layer by layer from a shallow layer to a deep layer;

[0009] Using a task-specific side network to perform cascade feature fusion and enhancement on feature images of adjacent layers, so that a deep-layer feature map improves a shallow-layer feature map, and obtains multiple layers of spatial information fusion enhanced feature maps, wherein the task-specific side network includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit;

[0010] A lightweight decoder is used to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then a detection head is used to generate a predicted saliency map according to the decoded output;

[0011] According to the predicted saliency map, detection of salient objects in the optical remote sensing image is achieved.

[0012] In one embodiment, the lightweight encoder adopts a MobileNet-V2 network.

[0013] In one embodiment, the side network of the specific task includes a plurality of the multi-scale semantic difference attention units and a plurality of the compact spatial feature fusion units;

[0014] The spatial information fusion enhanced feature map at the i-th level is expressed as:

[0015]

[0016] In the above formula, F i and F i+1 They represent the feature graph of the i-th level and the feature graph of the i+1-th level respectively. and They represent the multi-scale semantic difference attention unit and the compact spatial feature fusion unit corresponding to the i-th level feature map, respectively. Represents the multi-scale semantic difference attention unit corresponding to the i+1th level feature map;

[0017] When the i+1th level is the deepest level, the spatial information fusion enhanced feature map of the i+1th level is expressed as:

[0018]

[0019] In one embodiment, in the multi-scale semantic difference attention unit:

[0020] The feature map of the current level and the output feature map of the multi-scale semantic difference attention unit at a deeper level are used as the first input feature and the second input feature of the multi-scale semantic difference attention unit at the current level;

[0021] After upsampling, convolution and batch normalization processing are performed on the second input feature in sequence, the second input feature and the first input feature are respectively subjected to a multi-scale gradient-aware convolution layer to extract local contrast information across multiple scales, thereby obtaining a corresponding first local contrast feature and a second local contrast feature;

[0022] The first local contrast feature and the second local contrast feature are processed by a cross-gate attention mechanism, so as to remove redundant features in the first local contrast feature with the help of the second local contrast feature, and obtain an enhanced feature;

[0023] The enhanced features are processed using an edge-aware channel attention mechanism to obtain an output feature map of a multi-scale semantic difference attention unit at the current level.

[0024] In one embodiment, each of the compact spatial feature fusion units includes a compact dilated convolutional base part and a compact spatial attention part.

[0025] In one embodiment, the lightweight decoder includes a plurality of decoder units corresponding to processing each level of spatial information fusion enhanced feature map, and the following decoding process is implemented in each of the decoder units:

[0026]

[0027] In the above formula, represents the output of a deeper decoder unit, represents the output of the decoder unit at the current level, represents the corresponding spatial information fusion enhanced feature map, Cat[·] represents the splicing operation in the channel dimension, and Up[·] represents and Conv, BN, and ReLU represent convolution, batch normalization, and activation function processing, respectively.

[0028] In one embodiment, a salient object detection network is constructed according to the lightweight encoder, the task-specific side network, and the lightweight decoder;

[0029] The optical remote sensing image is input into the salient object detection network to obtain a predicted saliency map, so as to detect the salient objects in the optical remote sensing image.

[0030] The present application also provides a device for detecting salient objects in optical remote sensing images, the device comprising:

[0031] The module for acquiring images to be detected is used to acquire optical remote sensing images for significant target detection;

[0032] A feature image acquisition module is used to extract feature images of multiple levels of the optical remote sensing image layer by layer from shallow to deep layers using a lightweight encoder;

[0033] A spatial information fusion enhanced feature map acquisition module is used to use a side network of a specific task to perform cascade feature fusion and enhancement on feature images of adjacent levels, so that a deep-level feature map improves a shallow-level feature map, and obtains multiple levels of spatial information fusion enhanced feature maps, wherein the side network of the specific task includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit;

[0034] The predicted saliency map generation module uses a lightweight decoder to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then uses a detection head to generate a predicted saliency map based on the decoded output;

[0035] The salient target detection module detects the salient targets in the optical remote sensing image according to the predicted saliency map.

[0036] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] Acquiring an optical remote sensing image for salient target detection;

[0038] A lightweight encoder is used to extract feature maps of multiple levels of the optical remote sensing image layer by layer from a shallow layer to a deep layer;

[0039] Using a task-specific side network to perform cascade feature fusion and enhancement on feature images of adjacent layers, so that a deep-layer feature map improves a shallow-layer feature map, and obtains multiple layers of spatial information fusion enhanced feature maps, wherein the task-specific side network includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit;

[0040] A lightweight decoder is used to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then a detection head is used to generate a predicted saliency map according to the decoded output;

[0041] According to the predicted saliency map, detection of salient objects in the optical remote sensing image is achieved.

[0042] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0043] Acquiring an optical remote sensing image for salient target detection;

[0044] A lightweight encoder is used to extract feature maps of multiple levels of the optical remote sensing image layer by layer from a shallow layer to a deep layer;

[0045] Using a task-specific side network to perform cascade feature fusion and enhancement on feature images of adjacent layers, so that a deep-layer feature map improves a shallow-layer feature map, and obtains multiple layers of spatial information fusion enhanced feature maps, wherein the task-specific side network includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit;

[0046] A lightweight decoder is used to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then a detection head is used to generate a predicted saliency map according to the decoded output;

[0047] According to the predicted saliency map, detection of salient objects in the optical remote sensing image is achieved.

[0048] The above-mentioned optical remote sensing image salient target detection method, device, equipment and storage medium first use a lightweight encoder to extract the feature maps of multiple levels of the optical remote sensing image layer by layer from the shallow layer to the deep layer, and then use a side network of a specific task including a multi-scale semantic difference attention unit and a compact spatial feature fusion unit to perform cascade feature fusion and enhancement on the feature images of adjacent levels, so that the deep-level feature map improves the shallow-level feature map, and obtains multiple levels of spatial information fusion enhanced feature maps, and finally uses a lightweight decoder from the deep layer to the shallow layer to perform multi-stage step-by-step decoding on the multiple levels of spatial information fusion enhanced feature maps to obtain a decoded output, and then uses a detection head to generate a predicted saliency map according to the decoded output to achieve the detection of salient targets. The use of this method can effectively improve the efficiency of salient target detection for optical remote sensing images while ensuring the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 1 is a flow chart of a method for detecting salient objects in an optical remote sensing image in one embodiment;

[0050] Figure 2 is a schematic diagram of the structure of a salient object detection network in one embodiment;

[0051] Figure 3 A schematic diagram showing a comparison between the existing lightweight ORSI-SOD methods CorrNet ​​and SeaNe and the general architecture proposed by the present method in one embodiment;

[0052] Figure 4 is a schematic diagram of the structure of a multi-scale semantic difference attention unit in one embodiment;

[0053] Figure 5 is a schematic structural diagram of a compact spatial feature fusion unit in one embodiment;

[0054] Figure 6 is a structural block diagram of a device for detecting salient objects in optical remote sensing images in one embodiment;

[0055] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] In the existing technology, the superior performance of the latest ORSI-SOD models often relies on complex architectures or heavy encoders, which introduces a large number of parameters and computational requirements, limiting the deployment on resource-constrained devices, such as Figure 1 As shown, in the present application, a method for detecting salient targets in optical remote sensing images is provided, comprising the following steps:

[0058] Step S100: acquiring an optical remote sensing image for salient object detection.

[0059] Step S110, using a lightweight encoder to extract feature maps of multiple levels of the optical remote sensing image layer by layer from shallow layers to deep layers.

[0060] Step S120, using a side network for a specific task to perform cascade feature fusion and enhancement on feature images of adjacent levels, so that the deep-level feature map improves the shallow-level feature map, and obtains multiple levels of spatial information fusion enhanced feature maps, wherein the side network for the specific task includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit.

[0061] Step S130, a lightweight decoder is used to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then a detection head is used to generate a predicted saliency map according to the decoded output.

[0062] Step S140: detecting salient objects in the optical remote sensing image according to the predicted saliency map.

[0063] In this application, a general bottom-up pathway architecture for lightweight ORSI-SOD is proposed, which gradually integrates semantic features from deep to shallow layers. This intuitive design enriches the semantic information of each layer and alleviates the representation degradation caused by limited model capacity. In addition, it enhances the adaptability of the output layer to diverse objects, thereby generating accurate saliency maps. Subsequently, local contrast cues are captured by explicitly modeling the relationship between adjacent pixels to generate unique patterns. To this end, a multi-scale semantic difference attention unit (SDAM) is designed to capture multi-scale local semantic differences, and a difference convolution operator is used to dynamically aggregate high-order feature information to highlight the regions of salient objects.

[0064] In this method, based on the general architecture and SDAM, a new lightweight ORSI-SOD model named Semantic Differential Attention Network (SDANet), also known as the Salient Object Detection Network mentioned later, is designed. SDANet aims to improve feature representation within a limited parameterization range by effectively fusing features and integrating local contrast information.

[0065] Specifically, in the salient object detection network, there are three parts, namely, a lightweight encoder, a task-specific side network, and a lightweight decoder. These three parts can be regarded as separate components or as an overall network, such as Figure 2 shown.

[0066] In this article, the three parts of the lightweight encoder, the side network for a specific task, and the lightweight decoder are first introduced, that is, the process from step S110 to step S130 described above. In the process of introduction, the optical remote sensing image obtained in step S100 is used as the processing object for explanation. Among them, the salient targets in the optical remote sensing image can be moving targets such as vehicles and airplanes.

[0067] In step S110, the lightweight encoder adopts the MobileNet-V2 network.

[0068] In this embodiment, the first, third, sixth, thirteenth and last bottleneck layers in the MobileNet-V2 network are divided into five different levels from shallow to deep, and are represented as E i , (i=1,2,3,4,5). The output features of each level, i.e., the feature map, are represented as where c i ∈{16,24,32,96,320}. At the i-th level, the encoder feature F i Before being input to the decoder, it is processed by a task-specific side network.

[0069] In step S120, for the feature map F 1-4 , in the task-specific side network, feature extraction is performed by the multi-scale semantic difference attention unit (SDAM) and the compact spatial feature fusion unit (CSFM). For the deep feature map F 5 , only CSFM is used to extract global information from it. In the feature interaction of SDAM, from the deeper feature map F i+1 Extract local contrast cues as guiding information to improve the shallower feature map F iFeature representation. SDAM focuses on modeling local features, which may lead to the neglect of long-range features of large-scale objects (such as bridges). To address this limitation, a lightweight module CSFM is introduced after SDAM to achieve efficient multi-scale spatial feature enhancement.

[0070] Considering that in the prior art, lightweight methods are to independently fuse multi-level features at a deep or shallow level. For example, some methods focus on the interaction at high-level features, while some methods use different interaction methods for low-level and high-level features. These methods pass deep semantic features to the output layer in the decoding stage, and fail to make full use of the information of high-level salient objects to filter a large number of features in the low-level features and guide the object positioning in the output layer. Therefore, in this embodiment, a general bottom-up architecture is proposed to enhance the representation capability through cascaded feature fusion. Unlike previous methods, this method focuses on transferring deep semantic information within the side network of a specific task. Figure 3 As shown, Figure 3 (a) and Figure 3 (b) shows two existing neural network structures, Figure 3 (c) Shown is the network structure of this method.

[0071] like Figure 3 (c) and They represent the feature fusion module and feature enhancement module of the i-th level respectively. The process can be formulated as:

[0072]

[0073] The advantages of this general architecture include: deep semantic features can be effectively passed to the feature enhancement module of the previous layer, significantly improving the representation ability of this module. This effective information flow allows the model to better utilize the rich semantic information captured by the deep network when processing salient objects. Cascaded feature fusion gradually filters out redundant information at each stage, highlights salient object areas, and narrows the gap between model capacity and feature representation. Through this step-by-step refinement method, the model is able to focus more on the key parts of the image, thereby improving the accuracy of saliency detection.

[0074] Based on the general architecture proposed above, in designing the interaction module Crucial to improving the feature representation of salient objects, the goal is to explicitly model the visual cues of salient objects in deep semantic features and pass this prior information to shallower layers. To achieve this, a novel feature interaction module, Multi-scale Semantic Difference Attention Unit (SDAM), is introduced, which uses multi-scale local contrast cues as guidance to enhance the feature representation of salient regions in shallow features. At the same time, to enhance the spatial representation, a lightweight module is used To enhance the local contrast information extraction of salient object features. However, in order to effectively handle large salient objects (such as bridges), the model also needs to fully interact with pixels within a large receptive field. To address this issue, a Compact Spatial Feature Fusion Module (CSFM) is integrated after SDAM to refine the feature representation through multi-receptive field learning.

[0075] In summary, in this embodiment, the side network of a specific task includes multiple multi-scale semantic difference attention units and multiple compact spatial feature fusion units. Each multi-scale semantic difference attention unit and compact spatial feature fusion unit processes feature maps of different levels respectively, and the spatial information fusion enhanced feature of the i-th level is expressed as:

[0076]

[0077] In the above formula, F i and F i+1 They represent the feature graph of the i-th level and the feature graph of the i+1-th level respectively. and They represent the multi-scale semantic difference attention unit and the compact spatial feature fusion unit corresponding to the i-th level feature map, respectively. Represents the multi-scale semantic difference attention unit corresponding to the i+1th level feature map.

[0078] When the i+1th level is the deepest level, the spatial information fusion enhanced feature map of the i+1th level is expressed as:

[0079]

[0080] like Figure 4 As shown in Figure 2, SDAM combines multi-scale gradient-aware convolution (MSGAC) to capture the i-th and i+1-th level feature maps F i and F i+1 Then, a cross-gating mechanism is applied based on the multi-scale local contrast cues from the feature map F. i+1 Extracted guiding features to emphasize feature map F iFinally, edge channel attention (ECA) is applied to the cross-gating mechanism output features to further enhance the feature responses of channels containing salient object information, based on the edge information existing in each channel.

[0081] Specifically, in the multi-scale semantic difference attention unit: the feature map F of the current level i , and the output feature map of the deeper multi-scale semantic difference attention unit As the first input feature X of the multi-scale semantic difference attention unit at the current level i And the second input feature X i+1 . For the second input feature X i+1 After upsampling, convolution and batch normalization, the first input feature X i After passing through the multi-scale gradient-aware convolutional layer, local contrast information is extracted across multiple scales to obtain the corresponding first local contrast feature Y i And the second local contrast feature Y i+1 , for the first local contrast feature Y i And the second local contrast feature Y i+1 The cross-gate attention mechanism is used to process the second local contrast feature Y i+1 Remove the first local contrast feature Y i The redundant features in the enhanced features are obtained The edge-aware channel attention mechanism is used to enhance the features Processing is performed to obtain the output feature map of the current level multi-scale semantic difference attention unit

[0082] Specifically, multi-scale gradient-aware convolution (MSGAC) is used to extract local contrast information across multiple scales. Figure 4 As shown in Figure 1, the center value of the kernel is fixed to -1 to capture the difference properties of the object edge while retaining the high-frequency information in the feature. In order to adapt to the multi-scale characteristics of ORSI objects, the convolution kernel is expanded to multiple gradient-aware convolution (GAC) groups with different dilation rates to enhance the perception of pixel differences in local areas. Given an input feature X∈R c×h×w , first divide X into N groups [X1,...,X K ,...,X N ], where each group Include Each set of features is then associated with a designed set of convolutions with dilation rates ranging from 1 to N.

[0083] Furthermore, the process of MSGAC(·) can be formulated as follows:

[0084] {X k}=Split[X],k=1:N

[0085]

[0086] X out =Cat[X1,...,X k ,...,X N ]

[0087] In the above formula, X out is the output of MSGAC, is the weight of the k-th group of GAC, d is the dilation rate of the k-th group of GAC, and Cat[·] represents the concatenation in the channel dimension. It is worth noting that before passing through MSGAC, the deep feature X i+1 It undergoes an upsampling layer followed by a 3×3 convolution to make its spatial and channel dimensions consistent with X i Alignment.

[0088] Then the local contrast cue extraction process in SDAM can be formulated as:

[0089] Y i =MSGAC(F i )

[0090] Y i+1 =MSGAC(BN(Conv(Up(F i+1 ))))

[0091] In this embodiment, the method of processing using MSGAC is regarded as using each set of features to learn edge information at a specific scale, which has two advantages: Efficiently capture multi-scale features: multi-scale features can be captured without additional computational overhead. Learn semantic feature representations at a specific scale: adapt to objects of different sizes in ORSIs.

[0092] Furthermore, in the multi-scale semantic difference attention unit, the deep feature F i+1 Contains rich semantic information about salient objects and can be used to guide shallow object features F i Enhancements. From F i and F i+1 The obtained feature Y i and Y i+1 Contains rich boundary information, which makes Y i+1 Can help Y iRemove redundant background information and highlight the boundary structure of salient objects. In order to further reduce the interference of redundant features in the guide item, use Y i+1 The square term of Y i Multiply to get the enhanced output This process can be formulated as:

[0093]

[0094] Furthermore, the features obtained are enhanced by guiding information Contains rich boundary structure information of salient objects. These features are combined with the original features F i By adding, the prominent features of the salient object area can be obtained. In order to further amplify the response of the channel containing more target information, the boundary-rich To adaptively generate a set of coefficients that amplify the activation of feature channels that contain more object boundary information. The edge-aware channel attention mechanism is used to enhance the feature The processing process is expressed as:

[0095]

[0096] In the above formula, represents the final output of SDAM, Fi is the residual term, GAP(·) is the global average pooling layer, Conv1D k represents a one-dimensional convolution with kernel size k, where k is set to 3.

[0097] like Figure 5 As shown in Figure 1, the compact spatial feature fusion unit (CSFM) consists of a compact dilated convolutional base part (CDCM) and a compact spatial attention part (CSAM). To reduce the computational cost, CDCM first projects the input features to a low channel dimension (e.g., 24 channels) through a 1×1 convolution. Then, a set of convolutional layers with different dilation rates are used to capture and aggregate multi-scale spatial features.

[0098] Specifically, the configuration in the compact spatial feature fusion unit is shown in Table 1.

[0099] Table 1

[0100]

[0101] In Table 1, (d1, d2) in the row of Conv1×1 represents input channel d1 and output channel d2. (d1, d2, k) in the row of Conv3×3 represents input channel d1, output channel d2, and expansion channel k.

[0102] In this method, based on the salient object regions provided by the first two units, a lightweight decoder is designed to generate saliency maps step by step.

[0103] In this embodiment, the lightweight decoder is composed of four lightweight decoder units, denoted as D i Each D i A 1×1 convolution is used to aggregate information from the adjacent task-specific side network. Finally, a detection head is used at the end of the decoder unit to generate the predicted saliency map.

[0104] Specifically, the four decoder units are D1, D2, D3, and D4, each of which contains a 1×1 convolution block and an upsampling operation. Unlike the 3×3 convolution or 3×3 depth convolution used in previous methods, the convolution in the decoding layer is replaced by a 1×1 convolution in this method. This replacement effectively reduces the computational cost and parameter size of the decoding part, thereby enhancing the feature representation capability by expanding the capacity of the task-specific side network.

[0105] Further, use and They represent the spatial information fusion enhanced feature maps output by the task-specific side networks at the i-th level and i+1-th level respectively. The processing process in the decoder unit corresponding to the i-th level can be expressed as:

[0106]

[0107] In the above formula, represents the output of a deeper decoder unit, represents the output of the decoder unit at the current level, represents the corresponding spatial information fusion enhanced feature map, Cat[·] represents the splicing operation in the channel dimension, and Up[·] represents and Conv, BN, and ReLU represent convolution, batch normalization, and activation function processing, respectively.

[0108] In this implementation, a detection head is used to generate the final predicted saliency map. The detection head consists of a dropout layer and a 1×1 convolutional layer, which can be expressed as:

[0109]

[0110] In this embodiment, a salient target detection network is constructed according to the above-mentioned lightweight encoder, task-specific side network and lightweight decoder, and the optical remote sensing image to be subjected to salient target detection is input into the network to obtain a predicted saliency map, so as to realize the detection of salient targets in the optical remote sensing image.

[0111] In this embodiment, when training the salient object detection network, binary cross-entropy (BCE) loss and intersection over union (IoU) loss are used for joint training. The total loss can be expressed as:

[0112] L total =L bce (Y pred ,Y true )+L iou (Y pred ,Y true )

[0113] In the above formula, L bce and L iou They are the binary cross entropy loss function and the intersection-over-union loss, Y pred is the final model output, Y true It’s a true label.

[0114] In this paper, the effectiveness of this method is also demonstrated through experimental results. From Table 2, it can be seen that our method achieves a better balance between accuracy and computational efficiency, narrows the performance gap between small models and large models, and is conducive to low-power deployment in actual applications, such as satellites and other devices.

[0115] Table 2

[0116]

[0117] In the above-mentioned salient object detection method for optical remote sensing images, a lightweight salient object detection network consisting of three parts is proposed to detect salient objects in optical remote sensing images. The three parts are: a lightweight encoder using MobileNet-V2 as a lightweight backbone network to reduce the number of parameters and computational cost of the framework, integrating multi-scale semantic difference attention units (SDAM) and compact spatial feature fusion units (CSFM), a task-specific side network that enhances feature representation through local contrast information modeling and global feature refinement, and a lightweight decoder that gradually processes the semantic features of each level to generate the final prediction. In this method, a general bottom-up path architecture suitable for lightweight ORSI-SOD tasks is explored to enhance the feature representation of diverse objects. The design adopts a cascade feature fusion method to gradually integrate features from deep to shallow layers, effectively transferring high-level semantic information to improve the output layer's perception of salient objects of different types and scales. At the same time, a multi-scale semantic difference attention unit (SDAM), a novel feature fusion unit, is proposed, which explicitly models the multi-scale boundary information and category semantics in deep features as semantic difference signals to highlight the regions of salient objects, thereby helping lightweight models to effectively distinguish foreground and background pixels. Based on the above two designs, a new lightweight ORSI-SOD model named Semantic Difference Attention Neural Network (SDANet) is proposed, which utilizes MobileNet-V2. SDANet surpasses the performance of current lightweight ORSI-SOD methods and achieves comparable results to the state-of-the-art ORSI-SOD models, while processing images with a resolution of 288×288 using only 2.73M parameters and 1.2G FLOPs.

[0118] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0119] In one embodiment, Figure 6As shown, an optical remote sensing image salient target detection device is provided, comprising: a to-be-detected image acquisition module 200, a feature image acquisition module 210, a spatial information fusion enhanced feature map acquisition module 220, a predicted saliency map generation module 230 and a salient target detection module 240, wherein:

[0120] The to-be-detected image acquisition module 200 is used to acquire an optical remote sensing image for significant target detection;

[0121] A feature image obtaining module 210 is used to extract feature images of multiple levels of the optical remote sensing image layer by layer from shallow to deep layers using a lightweight encoder;

[0122] A spatial information fusion enhanced feature map obtaining module 220 is used to perform cascade feature fusion and enhancement on feature images of adjacent levels using a side network of a specific task, so that a deep-level feature map improves a shallow-level feature map, and obtains multiple levels of spatial information fusion enhanced feature maps, wherein the side network of the specific task includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit;

[0123] A predicted saliency map generation module 230 is used to use a lightweight decoder to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then use a detection head to generate a predicted saliency map according to the decoded output;

[0124] The salient object detection module 240 is used to detect salient objects in the optical remote sensing image according to the predicted saliency map.

[0125] For the specific definition of the optical remote sensing image salient target detection device, please refer to the definition of the optical remote sensing image salient target detection method above, which will not be repeated here. Each module in the above optical remote sensing image salient target detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0126] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting significant targets in an optical remote sensing image is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0127] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0129] Acquiring an optical remote sensing image for salient target detection;

[0130] A lightweight encoder is used to extract feature maps of multiple levels of the optical remote sensing image layer by layer from a shallow layer to a deep layer;

[0131] Using a task-specific side network to perform cascade feature fusion and enhancement on feature images of adjacent layers, so that a deep-layer feature map improves a shallow-layer feature map, and obtains multiple layers of spatial information fusion enhanced feature maps, wherein the task-specific side network includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit;

[0132] A lightweight decoder is used to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then a detection head is used to generate a predicted saliency map according to the decoded output;

[0133] According to the predicted saliency map, detection of salient objects in the optical remote sensing image is achieved.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0135] Acquiring an optical remote sensing image for salient target detection;

[0136] A lightweight encoder is used to extract feature maps of multiple levels of the optical remote sensing image layer by layer from a shallow layer to a deep layer;

[0137] Using a task-specific side network to perform cascade feature fusion and enhancement on feature images of adjacent layers, so that a deep-layer feature map improves a shallow-layer feature map, and obtains multiple layers of spatial information fusion enhanced feature maps, wherein the task-specific side network includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit;

[0138] A lightweight decoder is used to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then a detection head is used to generate a predicted saliency map according to the decoded output;

[0139] According to the predicted saliency map, detection of salient objects in the optical remote sensing image is achieved.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for detecting salient objects in optical remote sensing images, characterized in that: The method comprises: Acquiring an optical remote sensing image for salient target detection; A lightweight encoder is used to extract feature maps of multiple levels of the optical remote sensing image layer by layer from a shallow layer to a deep layer; Using a task-specific side network to perform cascade feature fusion and enhancement on feature images of adjacent layers, so that a deep-layer feature map improves a shallow-layer feature map, and obtains multiple layers of spatial information fusion enhanced feature maps, wherein the task-specific side network includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit; A lightweight decoder is used to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then a detection head is used to generate a predicted saliency map according to the decoded output; According to the predicted saliency map, detection of salient objects in the optical remote sensing image is achieved.

2. The optical remote sensing image salient object detection method according to claim 1, characterized in that: The lightweight encoder adopts the MobileNet-V2 network.

3. The optical remote sensing image salient object detection method according to claim 2, characterized in that: In the side network of the specific task, a plurality of the multi-scale semantic difference attention units and a plurality of the compact spatial feature fusion units are included; The spatial information fusion enhanced feature map at the i-th level is expressed as: In the above formula, F i and F i+1 They represent the feature graph of the i-th level and the feature graph of the i+1-th level respectively. and They represent the multi-scale semantic difference attention unit and the compact spatial feature fusion unit corresponding to the i-th level feature map, respectively. Represents the multi-scale semantic difference attention unit corresponding to the i+1th level feature map; When the i+1th level is the deepest level, the spatial information fusion enhanced feature map of the i+1th level is expressed as:

4. The optical remote sensing image salient object detection method according to claim 3, characterized in that: In the multi-scale semantic difference attention unit: The feature map of the current level and the output feature map of the multi-scale semantic difference attention unit at a deeper level are used as the first input feature and the second input feature of the multi-scale semantic difference attention unit at the current level; After upsampling, convolution and batch normalization processing are performed on the second input feature in sequence, the second input feature and the first input feature are respectively subjected to a multi-scale gradient-aware convolution layer to extract local contrast information across multiple scales, thereby obtaining a corresponding first local contrast feature and a second local contrast feature; The first local contrast feature and the second local contrast feature are processed by a cross-gate attention mechanism, so as to remove redundant features in the first local contrast feature with the help of the second local contrast feature, and obtain an enhanced feature; The enhanced features are processed using an edge-aware channel attention mechanism to obtain an output feature map of a multi-scale semantic difference attention unit at the current level.

5. The optical remote sensing image salient object detection method according to claim 3, characterized in that: Each of the compact spatial feature fusion units includes a compact dilated convolution base part and a compact spatial attention part.

6. The optical remote sensing image salient object detection method according to any one of claims 4 or 5, characterized in that: The lightweight decoder includes a plurality of decoder units corresponding to processing each level of spatial information fusion enhanced feature map, and the following decoding process is implemented in each of the decoder units: In the above formula, represents the output of a deeper decoder unit, represents the output of the decoder unit at the current level, Y t i represents the corresponding spatial information fusion enhanced feature map, Cat[·] represents the splicing operation in the channel dimension, and Up[·] represents and Y t i Conv, BN, and ReLU represent convolution, batch normalization, and activation function processing, respectively.

7. The optical remote sensing image salient object detection method according to claim 6, characterized in that: Building a salient object detection network according to the lightweight encoder, the task-specific side network, and the lightweight decoder; The optical remote sensing image is input into the salient object detection network to obtain a predicted saliency map, so as to detect the salient objects in the optical remote sensing image.

8. An optical remote sensing image salient object detection device, characterized in that: The device comprises: The module for acquiring images to be detected is used to acquire optical remote sensing images for significant target detection; A feature image acquisition module is used to extract feature images of multiple levels of the optical remote sensing image layer by layer from shallow to deep layers using a lightweight encoder; A spatial information fusion enhanced feature map acquisition module is used to use a side network of a specific task to perform cascade feature fusion and enhancement on feature images of adjacent levels, so that a deep-level feature map improves a shallow-level feature map, and obtains multiple levels of spatial information fusion enhanced feature maps, wherein the side network of the specific task includes a multi-scale semantic difference attention unit and a compact spatial feature fusion unit; The predicted saliency map generation module uses a lightweight decoder to perform multi-stage progressive decoding of multiple levels of spatial information fusion enhanced feature maps from deep layers to shallow layers to obtain a decoded output, and then uses a detection head to generate a predicted saliency map based on the decoded output; The salient target detection module detects the salient targets in the optical remote sensing image according to the predicted saliency map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images