Method and device for detecting aircraft targets in remote sensing images based on visual saliency detection

Abstract

The present invention discloses a method and device for detecting aircraft targets in remote sensing images based on visual saliency detection. The method comprises obtaining input features and determining corresponding masks; dividing foreground and background regions based on the input features according to the masks; concatenating the input features with foreground and background task labels to generate queries, concatenating the foreground region and foreground task labels to generate keys, and concatenating the background region and background task labels to generate values; performing cross-attention based on the generated queries, keys, and values ​​to generate a visual saliency map; and detecting aircraft based on the visual saliency map. The present invention can effectively improve the accuracy and efficiency of detecting aircraft targets in remote sensing images.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and more particularly to a method and device for detecting aircraft targets in remote sensing images based on visual saliency detection. Background Art

[0002] With the continuous advancement of remote sensing technology, remote sensing images have become an important data source for key fields such as military reconnaissance, environmental monitoring, urban planning, and disaster management. In these applications, the detection and recognition of aircraft targets plays a vital role, which is of great significance for ensuring aviation safety and optimizing traffic management.

[0003] Traditional aircraft target detection methods primarily rely on image processing techniques such as edge detection, template matching, and feature extraction. While these methods can achieve target detection to a certain extent, their performance in complex backgrounds is often unsatisfactory. Specifically, they are susceptible to factors such as lighting variations, shadow interference, target occlusion, and image resolution, resulting in poor detection results.

[0004] In recent years, the rapid development of deep learning technology has brought new opportunities to the field of object detection. Object detection algorithms based on convolutional neural networks (CNNs) have achieved remarkable results on multiple standard datasets, demonstrating high detection accuracy and fast detection speed. However, applying these algorithms to remote sensing image processing still faces several challenges:

[0005] Remote sensing images usually have high resolution and contain a large amount of background information, which requires the target detection algorithm to be able to effectively process massive data;

[0006] The annotation of remote sensing images is expensive and time-consuming, which limits the generalization ability of deep learning models;

[0007] The features of aircraft targets in remote sensing images may be subtle and diverse in different images, which increases the difficulty of detection;

[0008] Existing target detection methods often lack effective means to process background information and tend to regard background information as noise, resulting in a decrease in recognition ability at the edges and contours of the target.

[0009] Therefore, how to effectively capture the subtle features of aircraft targets in complex backgrounds and use background information to improve detection performance has become an urgent problem to be solved in the field of remote sensing image aircraft target detection. Summary of the Invention

[0010] Background information not only provides contextual information in target detection, but also helps reduce the false detection rate. However, existing target detection methods usually lack effective processing of background information. To this end, the present invention provides a method and device for detecting aircraft targets in remote sensing images based on visual saliency detection.

[0011] In order to achieve the above object, the present invention adopts the following technical solutions:

[0012] In a first aspect, the present invention provides a method for detecting aircraft targets in remote sensing images based on visual saliency detection, comprising:

[0013] Get input features and determine the corresponding mask;

[0014] dividing a foreground region and a background region based on the input features according to the mask;

[0015] The input features are concatenated with the foreground task marker and the background task marker to generate a query. The foreground area and the foreground task marker are concatenated to generate a key, and the background area and the background task marker are concatenated to generate a value.

[0016] Perform cross-attention based on the generated query, key, and value to obtain a visual saliency map;

[0017] Aircraft are detected based on the visual saliency map.

[0018] Furthermore, determining the corresponding mask includes sequentially upsampling and binarizing the input features.

[0019] Furthermore, dividing the foreground area and the background area based on the input features according to the mask includes:

[0020] For the foreground area, patches are selected from the input features according to the mask for segmentation;

[0021] For the background region, it is obtained by averaging the pooled integration of patches with a median value of 0 in the mask.

[0022] Furthermore, the process of generating the visual saliency map is executed multiple times, and the feature saliency map output by the previous execution process is used as the input of the next execution process, wherein the input features are obtained by converting the visual saliency map output by the previous execution process;

[0023] For the first execution process, the original self-attention is performed on the remote sensing image to be detected to obtain the initial visual saliency map.

[0024] Furthermore, after obtaining the input features, the depth position is embedded first, where the depth position encoding includes:

[0025] The input features are resized to match the spatial dimension h of the sinusoidal positional encoding. i ×wi ,

[0026] Normalize the adjusted input features and make the depth value consistent with h i Multiply and round to get a discrete depth map;

[0027] Encoding is performed according to the sinusoidal position encoding method.

[0028] Furthermore, a learnable scaling factor is set, multiplied by the depth position code, and concatenated with the two-dimensional spatial position code to be embedded in the query and key as a three-dimensional position code to provide three-dimensional structure prior information.

[0029] Furthermore, an inner product operation is performed on the foreground task label and the input feature, and the two-dimensional saliency map is reshaped by the Sigmoid activation function; an inner product operation is performed on the background task label and the input feature, and the boundary map is reshaped by the Sigmoid activation function;

[0030] And the process of generating visual saliency maps is trained by combining two-dimensional saliency maps and boundary maps.

[0031] Furthermore, the training loss function is:

[0032]

[0033] Where, represents the total loss, represents the significant prediction loss, represents the boundary prediction loss, represents the dense prediction loss, represents the supervised prediction loss based on the task label, n represents the number of times the visual saliency map generation process is performed, represents the true value of the saliency map, represents the dense prediction value corresponding to the saliency map, represents the predicted value of the saliency map based on the task label, represents the boundary truth value, represents the dense prediction value corresponding to the boundary, Represents the boundary prediction value based on the task label.

[0034] In a second aspect, the present invention provides a device for detecting aircraft targets in remote sensing images based on visual saliency detection. The device generates a visual saliency map using a decoder, wherein the decoder comprises:

[0035] A feature extraction unit is used to extract features from the input remote sensing image or feature saliency map and determine the corresponding mask;

[0036] a selective-integrated attention unit for dividing a foreground region and a background region based on the extracted features according to the mask; concatenating the input features with the foreground task label and the background task label to generate a query, concatenating the foreground region and the foreground task label to generate a key, and concatenating the background region and the background task label to generate a value; and performing cross-attention based on the generated query, key, and value to obtain a visual saliency map;

[0037] An aircraft detection unit is configured to detect an aircraft based on the visual saliency map.

[0038] Furthermore, there are multiple decoders connected in series.

[0039] It can be seen from the above technical solution that the present invention discloses a method and device for detecting aircraft targets in remote sensing images based on visual saliency detection, which aims to improve the detection accuracy and efficiency of aircraft targets in remote sensing images. Specifically, a novel detection framework is proposed. By introducing visual saliency detection technology, important foreground information can be effectively extracted from complex remote sensing images and effectively distinguished from background information.

[0040] Compared with the existing technology, the beneficial effects include:

[0041] 1) Significantly improved detection accuracy and speed; by introducing a selective-aggregate attention mechanism, the processing of foreground and background information is more efficient, reducing the computational burden of the model at the decoder layer and lowering resource consumption. Furthermore, the fusion of fine-grained and coarse-grained information enables the model to maintain efficient target recognition capabilities in complex scenes, especially in complex backgrounds or with severe occlusion, and the model can still effectively distinguish aircraft targets.

[0042] 2) Through deep position encoding, depth information is effectively integrated, which improves the model's ability to understand three-dimensional spatial structure and makes target detection more accurate in spatial positioning.

[0043] 3) Combining token-based supervised prediction loss enables more effective capture of task-related information during training, thereby improving learning efficiency.

[0044] Ultimately, this invention not only surpasses existing technologies in accuracy, but also reaches new heights in inference speed, demonstrating strong potential in practical applications and being applicable to various complex remote sensing image processing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 This is a flowchart of the method for detecting aircraft targets in remote sensing images based on visual saliency detection;

[0047] Figure 2 Schematic diagram of the structure of the remote sensing image aircraft target detection device based on visual saliency detection. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] The present invention discloses a method for detecting aircraft targets in remote sensing images based on visual saliency detection. Currently, existing saliency detection methods are mostly based on the fusion of global and local features, utilizing multiple information to improve the accuracy of saliency maps. However, these methods may be limited when processing complex scenes because they cannot effectively distinguish long-range dependencies between background and foreground. Therefore, the present invention improves the accuracy and efficiency of aircraft target detection in remote sensing images by introducing a selective-integration attention module and deep position encoding technology.

[0050] Example 1

[0051] In object detection tasks, foreground information is crucial, while background information provides essential contextual clues. To achieve a balance between foreground and background, this application aggregates background information into a single token to represent a coarse-grained region, while dividing the foreground region into fine-grained parts.

[0052] In one embodiment, a spatial attention mechanism is used to replace the original self-attention mechanism; specifically, a remote sensing image aircraft target detection method based on visual saliency detection is proposed, such as Figure 1 As shown, the steps include:

[0053] 1) Obtain input features and determine the corresponding mask; in this embodiment, determining the corresponding mask includes upsampling and binarizing the input features in sequence. In an practicable embodiment, the input features are upsampled twice and binarized using a threshold of 0.5, thereby processing them into a mask.

[0054] 2) Dividing the foreground region and the background region based on the input features according to the mask; so as to simultaneously consider fine-grained and coarse-grained context propagation.

[0055] For the foreground area, fine-grained segmentation is required, that is, according to the mask M i From the input features Select patches for division; the formula is expressed as:

[0056]

[0057] For the background area, the present invention performs coarse-grained segmentation, that is, the background patch corresponds to the mask M i The median index is 0, these patches are integrated with the background area by average pooling The formula is:

[0058]

[0059] 3) Input features The foreground task mark and background task mark are spliced ​​together to generate a query. The foreground area and the foreground task mark are spliced ​​together to generate a key, and the background area and the background task mark are spliced ​​together to generate a value.

[0060] 4) Perform cross attention based on the generated query, key, and value to obtain a visual saliency map; i.e.

[0061]

[0062] Where MCA means using standard multi-head attention in criss-cross attention, and MLP means multi-layer perceptron.

[0063] 5) Detecting the aircraft based on the visual saliency map.

[0064] In a preferred embodiment, the process of generating a visual saliency map is performed multiple times, and the feature saliency map output by the previous execution process is used as the input of the next execution process, wherein the input features are obtained by converting the visual saliency map output by the previous execution process;

[0065] For the first execution process, since there is no feature saliency map from the previous execution process, the original self-attention is used to obtain the initial visual saliency map according to the remote sensing image to be detected during the first execution of this embodiment.

[0066] This application designs a selective-aggregate attention module that can intelligently distinguish foreground and background regions in remote sensing images. In this module, the foreground region is subdivided into fine-grained segments to preserve key aircraft target details, while the background region is aggregated into a coarse-grained global label to reduce unnecessary computational burden.

[0067] By combining multi-scale feature fusion technology and self-attention mechanism, the selective-integrated attention module can capture the subtle features of the aircraft target and learn the long-range dependencies between the foreground and background, thereby significantly improving the detection accuracy.

[0068] Example 2

[0069] In order to better utilize the three-dimensional depth structure, the present invention designs a depth position encoding method based on two-dimensional sinusoidal position encoding; it mainly solves the essential difference between depth as a continuous variable and two-dimensional coordinates as discrete variables.

[0070] In this embodiment, the method includes:

[0071] The input features are resized to match the spatial dimension h of the sinusoidal positional encoding. i ×w i ,

[0072] Normalize the adjusted input features to adjust their value range to [0,1], and make the normalized depth value consistent with h i Multiply and round to get a discrete depth map;

[0073] This process makes the range of depth values ​​the same as the two-dimensional coordinates, so that they can be encoded according to the sinusoidal position encoding method. The mathematical expression is:

[0074]

[0075] Among them, dep represents the depth value and m represents the dimension.

[0076] The above method of the present invention can efficiently integrate depth information into the generation process of the visual saliency map, while maintaining the stability of the overall architecture and avoiding the additional computational costs brought by previous structures.

[0077] In one embodiment, a learnable scaling factor is proposed, denoted as Z i , used to establish the relative importance between depth position encoding and two-dimensional spatial position encoding.

[0078] Specifically, the scaling factor is multiplied with the depth position code and concatenated with the two-dimensional spatial position code to be embedded in the query and key as a three-dimensional position code to provide three-dimensional structure prior information.

[0079] This application is based on the application of depth position coding, which effectively encodes depth information by converting continuous depth values ​​into a discrete form compatible with two-dimensional image coordinates. The depth position coding technology converts depth values ​​into three-dimensional position codes using sine and cosine functions. Combined with two-dimensional spatial position coding, it provides the model with rich spatial position information, thereby enhancing the model's ability to understand three-dimensional spatial structures and improving the accuracy of aircraft target detection.

[0080] Example 3

[0081] The model training and validation process includes data preprocessing, model training, performance evaluation, and hyperparameter tuning.

[0082] First, the remote sensing image data is preprocessed to enhance the model's generalization and robustness. Then, a model integrating a selective-ensemble attention module and deep positional encoding techniques is trained using a well-labeled dataset. The model's performance is improved by optimizing loss functions such as cross-entropy loss and IoU loss.

[0083] In one embodiment, since some predictions are made by enhancing patch tokens and The dense predictions are obtained directly after two linear transformations, and therefore, they are not directly related to the two task tokens, which may hinder their ability to effectively learn task-relevant information.

[0084] Therefore, the present invention applies a token-based supervised prediction loss to directly provide supervision for saliency and boundary tokens. Specifically, in the process of generating the visual saliency map, the inner product operation is directly performed on the foreground task tokens and the input features, and the two-dimensional saliency map is reshaped by the Sigmoid activation function. And perform inner product operation on background task mark and input features, and reshape the boundary map through Sigmoid activation function In order to apply binary cross entropy loss; the expression is as follows:

[0085]

[0086] The process of generating visual saliency maps is then trained by combining the 2D saliency map and the boundary map. In this way, the gradient of the loss can be directly transferred to the task-related tokens. and Thereby enhancing their learning efficiency.

[0087] To further optimize the above technical solution, the loss function expression in this embodiment is:

[0088]

[0089] Where, represents the total loss, represents the significant prediction loss, represents the boundary prediction loss, represents the dense prediction loss, represents the supervised prediction loss based on the task label, n represents the number of times the visual saliency map generation process is performed, represents the true value of the saliency map, represents the dense prediction value corresponding to the saliency map, represents the predicted value of the saliency map based on the task label, represents the boundary truth value, represents the dense prediction value corresponding to the boundary, Represents the boundary prediction value based on the task label.

[0090] In an exemplary embodiment, dense prediction loss is used at the 1st, 1 / 4th, 1 / 8th and 1 / 16th layers. and token-based supervised prediction loss To optimize the generation process of each visual feature map, the details are as follows:

[0091]

[0092] It should be noted that the 1st layer, 1 / 4 layer, 1 / 8 layer, and 1 / 16 layer in this application can be understood as feature layers of different resolutions in the feature extraction network. These layers are usually obtained by layer-by-layer downsampling (for example, convolution operations combined with strides or pooling operations) during the encoding phase of a deep neural network. The specific meaning is:

[0093] Layer 1, representing the full-resolution feature map, usually corresponding to the original resolution of the input image;

[0094] 1 / 4 layer, indicating that the resolution is reduced to 1 / 4 of the input image (the width and height are reduced to 1 / 2 of the original);

[0095] 1 / 8 layer, indicating that the resolution is reduced to 1 / 8 of the input image;

[0096] The 1 / 16 layer indicates that the resolution is reduced to 1 / 16 of the input image, which is usually a feature map in a deeper layer of the network.

[0097] These layers are used for multi-scale supervision, optimizing the network's generation process at different scales to capture feature information at different scales. This embodiment applies a loss function to the feature maps of each of these four layers for optimization. If the number of decoders matches the scale of the feature maps, there will typically be four decoders, each processing a corresponding feature scale.

[0098] During the training phase, a model integrating the Select-Ensemble Attention module and deep position embedding technology was trained using a well-labeled remote sensing image dataset. Model performance was improved by optimizing the aforementioned loss function. Model performance was evaluated on an independent validation set, and hyperparameters were adjusted based on the evaluation results to further improve the model's detection accuracy. Finally, the model's generalization and applicability were verified by testing on multiple remote sensing image datasets.

[0099] It is important to emphasize that the training and testing methods of the select-ensemble attention mechanism of this invention are different. Due to the uncertainty of the number of selected foreground patches, which affects parallel computation, all patch labels are still used during training, and mask attention is used to filter background patch labels.

[0100] The remote sensing image target detection method provided by the present invention not only improves the accuracy of segmentation, but also reduces the resource requirements for model training and deployment through efficient parameter fine-tuning and the use of optimizers, making it suitable for remote sensing image processing tasks in various computing environments.

[0101] Example 4

[0102] This embodiment further provides a device for detecting aircraft targets in remote sensing images based on visual saliency detection. The device generates a visual saliency map using a decoder. The decoder includes:

[0103] The feature extraction unit is used to extract features from the input remote sensing image or feature saliency map and determine the corresponding mask; wherein, referring to Figure 2 , set the feature extraction unit as the converter;

[0104] a selective-integrated attention unit, designed to reduce the computational burden of the decoder layer; used to divide the foreground region and the background region based on the extracted features according to the mask; and concatenate the input features with the foreground task label and the background task label to generate a query, concatenate the foreground region and the foreground task label to generate a key, and concatenate the background region and the background task label to generate a value; and perform cross-attention based on the generated query, key, and value to obtain a visual saliency map;

[0105] An aircraft detection unit is configured to detect an aircraft based on the visual saliency map.

[0106] In one embodiment, the decoders are connected in series, wherein the first decoder layer uses the original self-attention to directly process the remote sensing image because there is no mask from the previous decoder, and the selection-integration attention mechanism is applied in the subsequent two decoder layers to generate a new visual saliency map based on the results output by the previous decoder.

[0107] In one embodiment, a depth position embedding unit is further included for embedding the depth position of the input features, keys, and values. The specific process is described above and will not be repeated here.

[0108] The selective-integrated attention unit in this embodiment intelligently processes foreground and background information, ensuring the model's focus on the detailed features of the aircraft target while effectively reducing unnecessary computational overhead. Furthermore, the application of deep position encoding technology enables the model to more accurately process three-dimensional spatial structures, further enhancing the target's spatial perception capabilities.

[0109] In general, the present invention addresses the challenges faced by aircraft target detection in remote sensing images and can not only improve the accuracy and efficiency of detection, but also promote the widespread application of remote sensing technology in practical applications and meet the needs of multiple fields for efficient and accurate target detection.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0111] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting aircraft targets in remote sensing images based on visual saliency detection, characterized in that: Get input features and determine the corresponding mask; dividing a foreground region and a background region based on the input features according to the mask; The input features are concatenated with the foreground task marker and the background task marker to generate a query. The foreground area and the foreground task marker are concatenated to generate a key, and the background area and the background task marker are concatenated to generate a value. After obtaining the input features, the depth position is embedded first. The depth position encoding includes: The input features are resized to match the spatial dimension h of the sinusoidal positional encoding. i ×w i ; Normalize the adjusted input features and make the depth value consistent with h i Multiply and round to get a discrete depth map; Encode according to the sinusoidal position encoding method; Set a learnable scaling factor, multiply the scaling factor by the depth position code, and concatenate it with the two-dimensional spatial position code to embed it in the query and key as a three-dimensional position code; Perform cross-attention based on the generated query, key, and value to obtain a visual saliency map; Aircraft are detected based on the visual saliency map.

2. The aircraft target detection method according to claim 1, characterized in that: Determine the corresponding mask, including upsampling and binarization of the input features in sequence.

3. The aircraft target detection method according to claim 1, characterized in that: Dividing a foreground area and a background area based on input features according to the mask, including: For the foreground area, patches are selected from the input features according to the mask for segmentation; For the background region, it is obtained by averaging the pooled integration of patches with a median value of 0 in the mask.

4. The aircraft target detection method according to claim 1, characterized in that: The process of generating a visual saliency map is executed multiple times, and the feature saliency map output by the previous execution process is used as the input of the next execution process, wherein the input features are obtained by converting the visual saliency map output by the previous execution process; For the first execution process, the original self-attention is performed on the remote sensing image to be detected to obtain the initial visual saliency map.

5. The aircraft target detection method according to claim 1, characterized in that: Perform an inner product operation on the foreground task mark and the input features, and reshape them using the Sigmoid activation function to obtain a two-dimensional saliency map; perform an inner product operation on the background task mark and the input features, and reshape them using the Sigmoid activation function to obtain a boundary map; And the process of generating visual saliency maps is trained by combining two-dimensional saliency maps and boundary maps.

6. The aircraft target detection method according to claim 5, characterized in that: The training loss function is: Where, represents the total loss, represents the significant prediction loss, represents the boundary prediction loss, represents the dense prediction loss, represents the supervised prediction loss based on the task label, n represents the number of times the visual saliency map generation process is performed, represents the true value of the saliency map, represents the dense prediction value corresponding to the saliency map, represents the predicted value of the saliency map based on the task label, represents the boundary truth value, represents the dense prediction value corresponding to the boundary, Represents the boundary prediction value based on the task label.

7. A remote sensing image aircraft target detection device based on visual saliency detection, characterized in that: A visual saliency map is generated using a decoder, the decoder comprising: A feature extraction unit is used to extract features from the input remote sensing image or feature saliency map and determine the corresponding mask; A selective-integrated attention unit is configured to divide the foreground area and the background area based on the extracted features according to the mask; and concatenate the input features with the foreground task label and the background task label to generate a query, concatenate the foreground area and the foreground task label to generate a key, and concatenate the background area and the background task label to generate a value. After obtaining the input features, the depth position is preferentially embedded, and the depth position encoding includes: The input features are resized to match the spatial dimension h of the sinusoidal positional encoding. i ×w i , Normalize the adjusted input features and make the depth value consistent with h i Multiply and round to get a discrete depth map; Encode according to the sinusoidal position encoding method; Set a learnable scaling factor, multiply the scaling factor by the depth position code, and concatenate it with the two-dimensional spatial position code to embed it in the query and key as a three-dimensional position code; and performing cross attention based on the generated query, key, and value to obtain a visual saliency map; An aircraft detection unit is configured to detect an aircraft based on the visual saliency map.

8. The aircraft target detection device according to claim 7, characterized in that: There are multiple decoders connected in series.

Images