Remote sensing image aircraft target detection method and device based on visual saliency detection

Abstract

The invention discloses a remote sensing image aircraft target detection method and device based on visual saliency detection. The method comprises the following steps: acquiring an input feature, and determining a corresponding mask; dividing a foreground region and a background region based on input features according to the mask; splicing the input features with the foreground task marks and the background task marks to generate query, foreground area and foreground task mark splicing generation keys and background area and background task mark splicing generation values, and executing cross attention according to the generated query, keys and values to obtain a visual saliency map; and detecting the airplane according to the visual saliency map. The method can effectively improve the detection accuracy and efficiency of the aircraft target in the remote sensing image.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and more particularly to a remote sensing image aircraft target detection method and device based on visual saliency detection. Background Art

[0002] With the continuous advancement of remote sensing technology, remote sensing images have become an important data source for key fields such as military reconnaissance, environmental monitoring, urban planning, and disaster management. In these applications, the detection and recognition of aircraft targets plays a vital role, which is of great significance for ensuring aviation safety and optimizing traffic management.

[0003] Traditional aircraft target detection methods mainly rely on image processing techniques, such as edge detection, template matching, feature extraction, etc. These methods can achieve target detection to a certain extent, but their performance in complex backgrounds is often unsatisfactory. Specifically, they are easily affected by factors such as lighting changes, shadow interference, target occlusion, and image resolution, resulting in poor detection results.

[0004] In recent years, the rapid development of deep learning technology has brought new opportunities to the field of target detection. Target detection algorithms based on convolutional neural networks (CNNs) have achieved remarkable results on multiple standard datasets, demonstrating high detection accuracy and fast detection speed. However, there are still some challenges when applying these algorithms to remote sensing image processing:

[0005] Remote sensing images usually have high resolution and contain a lot of background information, which requires the target detection algorithm to be able to effectively process massive data;

[0006] The annotation of remote sensing images is expensive and time-consuming, which limits the generalization ability of deep learning models;

[0007] The features of aircraft targets in remote sensing images may be relatively subtle and diverse in different images, which increases the difficulty of detection;

[0008] Existing target detection methods often lack effective means to process background information and tend to regard background information as noise, resulting in reduced recognition capabilities at target edges and contours.

[0009] Therefore, how to effectively capture the subtle features of aircraft targets in complex backgrounds and use background information to improve detection performance has become an urgent problem to be solved in the field of remote sensing image aircraft target detection. Summary of the invention

[0010] Background information not only provides contextual information in target detection, but also helps to reduce the false detection rate; however, existing target detection methods usually lack effective processing of background information. In this regard, the present invention provides a remote sensing image aircraft target detection method and device based on visual saliency detection.

[0011] In order to achieve the above object, the present invention adopts the following technical solution:

[0012] In a first aspect, the present invention provides a method for detecting aircraft targets in remote sensing images based on visual saliency detection, comprising:

[0013] Get input features and determine the corresponding mask;

[0014] dividing a foreground region and a background region based on the input feature according to the mask;

[0015] The input features are concatenated with the foreground task mark and the background task mark to generate a query. The foreground area and the foreground task mark are concatenated to generate a key, and the background area and the background task mark are concatenated to generate a value.

[0016] Perform cross-attention based on the generated query, key, and value to obtain a visual saliency map;

[0017] Aircraft are detected based on the visual saliency map.

[0018] Further, determining the corresponding mask includes sequentially upsampling and binarizing the input features.

[0019] Further, dividing the foreground area and the background area based on the input features according to the mask includes:

[0020] For the foreground area, patches are selected from the input features for segmentation according to the mask;

[0021] For the background area, it is obtained by averaging the pooled integration of patches with value 0 in the mask.

[0022] Further, the process of generating the visual saliency map is executed multiple times, and the feature saliency map output by the previous execution process is used as the input of the next execution process, wherein the input feature is obtained by converting the visual saliency map output by the previous execution process;

[0023] For the first execution process, the original self-attention is performed according to the remote sensing image to be detected to obtain the initial visual saliency map.

[0024] Furthermore, after obtaining the input features, the depth position is preferentially embedded, wherein the depth position encoding includes:

[0025] The input features are resized to match the spatial dimension h of the sinusoidal positional encoding. i ×i ,

[0026] Normalize the adjusted input features and make the depth value consistent with h i Multiply and round to get a discrete depth map;

[0027] The encoding is performed according to the sinusoidal position encoding method.

[0028] Furthermore, a learnable scaling factor is set, multiplied with the depth position code, and concatenated with the two-dimensional space position code to be embedded in the query and key as the three-dimensional position code to provide three-dimensional structure prior information.

[0029] Furthermore, an inner product operation is performed on the foreground task mark and the input feature, and the two-dimensional saliency map is reshaped by the Sigmoid activation function; an inner product operation is performed on the background task mark and the input feature, and the boundary map is reshaped by the Sigmoid activation function;

[0030] And the process of generating visual saliency maps is trained by combining the two-dimensional saliency map and the boundary map.

[0031] Furthermore, the training loss function is:

[0032]

[0033] In the formula, represents the total loss, represents the significant prediction loss, represents the boundary prediction loss, represents the dense prediction loss, represents the supervised prediction loss based on the task label, n represents the number of times the visual saliency map generation process is performed, represents the true value of the saliency map, represents the dense prediction value corresponding to the saliency map, represents the predicted value of the saliency map based on the task label, represents the boundary truth value, represents the dense prediction value corresponding to the boundary, Represents the boundary prediction value based on the task label.

[0034] In a second aspect, the present invention provides a remote sensing image aircraft target detection device based on visual saliency detection, wherein the device generates a visual saliency map using a decoder, wherein the decoder comprises:

[0035] A feature extraction unit, used to extract features from an input remote sensing image or a feature saliency map and determine a corresponding mask;

[0036] A selection-integration attention unit is used to divide the foreground area and the background area based on the extracted features according to the mask; and the input features are respectively concatenated with the foreground task label and the background task label to generate a query, the foreground area and the foreground task label are concatenated to generate a key, and the background area and the background task label are concatenated to generate a value, and cross attention is performed according to the generated query, key and value to obtain a visual saliency map;

[0037] An aircraft detection unit is used to detect an aircraft according to the visual saliency map.

[0038] Furthermore, the decoders are multiple connected in series.

[0039] It can be seen from the above technical solutions that the present invention discloses a method and device for detecting aircraft targets in remote sensing images based on visual saliency detection, aiming to improve the detection accuracy and efficiency of aircraft targets in remote sensing images. A novel detection framework is specifically proposed. By introducing visual saliency detection technology, important foreground information can be effectively extracted from complex remote sensing images and effectively distinguished from background information.

[0040] Compared with the prior art, the beneficial effects include:

[0041] 1) Significantly improved detection accuracy and speed; by introducing the selection-integration attention mechanism, the processing of foreground and background information is more efficient, and the computational burden of the model at the decoder layer is reduced, reducing resource consumption. At the same time, the fusion of fine-grained and coarse-grained information enables the model to maintain efficient target recognition capabilities in complex scenes, especially in the case of complex backgrounds or severe occlusion, the model can still effectively distinguish aircraft targets;

[0042] 2) Through deep position encoding, depth information is effectively integrated, which improves the model's ability to understand the three-dimensional spatial structure and makes target detection more accurate in spatial positioning.

[0043] 3) Combined with the token-based supervised prediction loss, the task-related information can be captured more effectively during the training process, thereby improving the learning efficiency.

[0044] Finally, the invention not only surpasses the existing technology in accuracy, but also reaches a new height in inference speed, showing great potential in practical applications and is suitable for various complex remote sensing image processing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 The flowchart of the remote sensing image aircraft target detection method based on visual saliency detection is shown in FIG.

[0047] Figure 2 This is a schematic diagram of the structure of the remote sensing image aircraft target detection device based on visual saliency detection. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The embodiment of the present invention discloses a method for detecting aircraft targets in remote sensing images based on visual saliency detection. At present, existing saliency detection methods are mostly based on the fusion of global features and local features, and use multiple information to improve the accuracy of saliency maps; however, these methods may be limited when processing complex scenes because they cannot effectively distinguish the long-distance dependency between the background and the foreground; therefore, the present invention improves the detection accuracy and efficiency of aircraft targets in remote sensing images by introducing a selection-integration attention module and deep position encoding technology.

[0050] Embodiment 1

[0051] In the object detection task, foreground information is crucial, while background information provides necessary contextual clues. To achieve a balance between foreground and background, this application aggregates the background information into a single token to represent a coarse-grained region, while dividing the foreground region into fine-grained parts.

[0052] In one embodiment, a spatial attention mechanism is used to replace the original self-attention; specifically, a remote sensing image aircraft target detection method based on visual saliency detection, such as Figure 1 As shown, the steps include:

[0053] 1) Obtain input features and determine corresponding masks; in this embodiment, determining the corresponding masks includes sequentially upsampling and binarizing the input features. In an practicable embodiment, the input features are upsampled twice and binarized using a threshold of 0.5, thereby processing them into masks.

[0054] 2) Dividing the foreground region and the background region based on the input features according to the mask; so as to simultaneously consider fine-grained and coarse-grained context propagation.

[0055] For the foreground area, fine-grained segmentation is required, that is, according to the mask M i From the input features Select patches for division; the formula is expressed as:

[0056]

[0057] For the background area, the present invention performs coarse-grained segmentation, that is, the background patch corresponds to the mask M i The median index is 0, these patches are integrated with the background area by average pooling The formula is:

[0058]

[0059] 3) Input features The foreground task mark and background task mark are respectively concatenated to generate a query. The foreground area and the foreground task mark are concatenated to generate a key, and the background area and the background task mark are concatenated to generate a value.

[0060] 4) Perform cross-attention based on the generated query, key, and value to obtain a visual saliency map; i.e.

[0061]

[0062] Where MCA means using standard multi-head attention in criss-cross attention and MLP means multi-layer perceptron.

[0063] 5) Detecting the aircraft based on the visual saliency map.

[0064] In a preferred embodiment, the process of generating a visual saliency map is performed multiple times, and the feature saliency map output by the previous execution process is used as the input of the next execution process, wherein the input feature is obtained by converting the visual saliency map output by the previous execution process;

[0065] For the first execution process, since there is no feature saliency map from the previous execution process, the original self-attention is used to obtain the initial visual saliency map according to the remote sensing image to be detected during the first execution of this embodiment.

[0066] This application designs a selective-integrated attention module that can intelligently distinguish foreground and background regions in remote sensing images. In this module, the foreground region is subdivided into fine-grained segments to retain key aircraft target details, while the background region is aggregated into a coarse-grained global label to reduce unnecessary computational burden.

[0067] By combining multi-scale feature fusion technology and self-attention mechanism, the selective-integrated attention module can capture the subtle features of the aircraft target and learn the long-distance dependencies between the foreground and background, thereby significantly improving the detection accuracy.

[0068] Embodiment 2

[0069] In order to better utilize the three-dimensional depth structure, the present invention designs a depth position encoding method based on two-dimensional sinusoidal position encoding; it mainly solves the essential difference between depth as a continuous variable and two-dimensional coordinates as discrete variables.

[0070] In this embodiment, the method includes:

[0071] The input features are resized to match the spatial dimension h of the sinusoidal positional encoding. i × i ,

[0072] The adjusted input features are normalized to adjust their value range to [0,1], and the normalized depth values ​​are made consistent with h i Multiply and round to get a discrete depth map;

[0073] This process makes the range of depth values ​​the same as the two-dimensional coordinates, so that they can be encoded in the subsequent sinusoidal position encoding method. The mathematical expression is:

[0074]

[0075] Among them, dep represents the depth value and m represents the dimension.

[0076] The above method of the present invention can efficiently integrate depth information into the generation process of the visual saliency map, and at the same time, can maintain the stability of the overall architecture and avoid the additional computational costs brought by the previous structure.

[0077] In one embodiment, a learnable scaling factor is proposed, denoted as Z i , used to establish the relative importance between depth position encoding and two-dimensional spatial position encoding.

[0078] Specifically, the scaling factor is multiplied with the depth position code and concatenated with the two-dimensional spatial position code to be embedded in the query and key as the three-dimensional position code to provide three-dimensional structure prior information.

[0079] This application is based on the application of depth position coding, which realizes the effective coding of depth information by converting continuous depth values ​​into discrete forms compatible with two-dimensional image coordinates. The depth position coding technology converts the depth value into three-dimensional position coding through sine and cosine functions, and combines it with two-dimensional spatial position coding to provide the model with rich spatial position information, thereby enhancing the model's ability to understand three-dimensional spatial structure and improving the accuracy of aircraft target detection.

[0080] Embodiment 3

[0081] The model training and validation process includes data preprocessing, model training, performance evaluation, and hyperparameter tuning.

[0082] First, the remote sensing image data is preprocessed to enhance the generalization and robustness of the model. Then, the model integrating the selection-integration attention module and deep position encoding technology is trained using the labeled dataset, and the model performance is improved by optimizing the loss functions such as cross entropy loss and IoU loss.

[0083] In one embodiment, since some predictions are made through enhanced patch tokens and The dense predictions are directly obtained after two linear transformations, and therefore, are not directly related to the two task tokens, which may hinder their ability to effectively learn task-relevant information.

[0084] Therefore, the present invention applies a token-based supervised prediction loss to directly provide supervision for saliency and boundary tokens. Specifically, in the process of generating the visual saliency map, the inner product operation is directly performed on the foreground task tokens and the input features, and the two-dimensional saliency map is reshaped by the Sigmoid activation function. And perform inner product operation on the background task label and input features, and reshape the boundary map through the Sigmoid activation function In order to apply binary cross entropy loss; the expression is as follows:

[0085]

[0086] The process of generating visual saliency maps is then trained by combining the 2D saliency map and the boundary map. In this way, the gradient of the loss can be directly transferred to the task-related tokens. and Thereby enhancing their learning efficiency.

[0087] To further optimize the above technical solution, the loss function expression in this embodiment is:

[0088]

[0089] In the formula, represents the total loss, represents the significant prediction loss, represents the boundary prediction loss, represents the dense prediction loss, represents the supervised prediction loss based on the task label, n represents the number of times the visual saliency map generation process is performed, represents the true value of the saliency map, represents the dense prediction value corresponding to the saliency map, represents the predicted value of the saliency map based on the task label, represents the boundary truth value, represents the dense prediction value corresponding to the boundary, Represents the boundary prediction value based on the task label.

[0090] In an exemplary embodiment, dense prediction loss is used at the 1st layer, 1 / 4 layer, 1 / 8 layer and 1 / 16 layer. and token-based supervised prediction loss To optimize the generation process of each visual feature map, as follows:

[0091]

[0092] It should be noted that the 1st layer, 1 / 4 layer, 1 / 8 layer and 1 / 16 layer of the present application can be understood as feature layers of different resolutions in the feature extraction network. These layers are usually obtained by downsampling layer by layer (such as convolution operation combined with stride or pooling operation) in the encoding stage of the deep neural network. The specific meaning is:

[0093] Layer 1, representing the full-resolution feature map, usually corresponding to the original resolution of the input image;

[0094] 1 / 4 layer, indicating that the resolution is reduced to 1 / 4 of the input image (the width and height are reduced to 1 / 2 of the original);

[0095] 1 / 8 layer, indicating that the resolution is reduced to 1 / 8 of the input image;

[0096] The 1 / 16 layer means that the resolution is reduced to 1 / 16 of the input image, which is usually a feature map in a deeper layer of the network.

[0097] These layers are used for multi-scale supervision, that is, optimizing the network generation process at different scales to capture feature information at different scales; this embodiment applies loss functions to optimize the feature maps of these four layers. If the number of decoders matches the scale of the feature map, there will usually be four decoders, each processing a corresponding feature scale.

[0098] In the training phase, the model integrating the selection-integration attention module and deep position embedding technology is trained using annotated remote sensing image datasets, and the model performance is improved by optimizing the above loss function. The model performance is evaluated on an independent validation set, and the hyperparameters are adjusted based on the evaluation results to further improve the detection accuracy of the model. Finally, the generalization ability and applicability of the model are verified by testing on multiple different remote sensing image datasets.

[0099] It should be emphasized that the training method and the testing method of the selection-integration attention mechanism of the present invention are different. Due to the uncertainty of the number of selected foreground patches, which affects the parallel calculation, all patch labels are still used in the training process, and mask attention is used to filter the background patch labels.

[0100] The remote sensing image target detection method provided by the present invention not only improves the accuracy of segmentation, but also reduces the resource requirements for model training and deployment through efficient parameter fine-tuning and the use of optimizers, making it suitable for remote sensing image processing tasks in various computing environments.

[0101] Embodiment 4

[0102] This embodiment further provides a remote sensing image aircraft target detection device based on visual saliency detection, wherein the device generates a visual saliency map using a decoder, wherein the decoder includes:

[0103] The feature extraction unit is used to extract features from the input remote sensing image or feature saliency map and determine the corresponding mask; wherein, reference Figure 2 , set the feature extraction unit as the converter;

[0104] A selective-integrated attention unit is used to reduce the computational burden of the decoder layer; it is used to divide the foreground area and the background area based on the extracted features according to the mask; and the input features are respectively concatenated with the foreground task label and the background task label to generate a query, the foreground area and the foreground task label are concatenated to generate a key, and the background area and the background task label are concatenated to generate a value, and cross attention is performed according to the generated query, key and value to obtain a visual saliency map;

[0105] An aircraft detection unit is used to detect an aircraft according to the visual saliency map.

[0106] In one embodiment, the decoders are connected in series, wherein the first decoder layer uses the original self-attention to directly process the remote sensing image because there is no mask from the previous decoder, and the selection-integration attention mechanism is applied in the subsequent two decoder layers to generate a new visual saliency map based on the result output by the previous decoder.

[0107] In one embodiment, a depth position embedding unit is further included, which is used to embed the depth position of the input features and keys and values. The specific process is described above and will not be repeated here.

[0108] The selective-integrated attention unit in this embodiment can ensure that the model focuses on the detailed features of the aircraft target and effectively reduce unnecessary computational burden by intelligently processing foreground and background information. At the same time, the application of deep position encoding technology makes the model more accurate in processing three-dimensional spatial structures, further enhancing the spatial perception ability of the target.

[0109] In general, the present invention addresses the challenges faced by aircraft target detection in remote sensing images and can not only improve the accuracy and efficiency of detection, but also promote the widespread application of remote sensing technology in practical applications and meet the needs of multiple fields for efficient and accurate target detection.

[0110] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0111] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting aircraft targets in remote sensing images based on visual saliency detection, characterized in that: Get input features and determine the corresponding mask; dividing a foreground region and a background region based on the input feature according to the mask; The input features are concatenated with the foreground task mark and the background task mark to generate a query. The foreground area and the foreground task mark are concatenated to generate a key, and the background area and the background task mark are concatenated to generate a value. Perform cross-attention based on the generated query, key, and value to obtain a visual saliency map; Aircraft are detected based on the visual saliency map.

2. The aircraft target detection method according to claim 1, characterized in that: Determine the corresponding mask, including upsampling and binarizing the input features in sequence.

3. The aircraft target detection method according to claim 1, characterized in that: Dividing a foreground area and a background area based on the input feature according to the mask includes: For the foreground area, patches are selected from the input features for segmentation according to the mask; For the background area, it is obtained by averaging the pooled integration of patches with value 0 in the mask.

4. The aircraft target detection method according to claim 1, characterized in that: The process of generating a visual saliency map is executed multiple times, and the feature saliency map output by the previous execution process is used as the input of the next execution process, wherein the input feature is obtained by converting the visual saliency map output by the previous execution process; For the first execution process, the original self-attention is performed according to the remote sensing image to be detected to obtain the initial visual saliency map.

5. The aircraft target detection method according to claim 1, characterized in that: After obtaining the input features, the depth position is embedded first, where the depth position encoding includes: The input features are resized to match the spatial dimension h of the sinusoidal positional encoding. i × i , Normalize the adjusted input features and make the depth value consistent with h i Multiply and round to get a discrete depth map; The encoding is performed according to the sinusoidal position encoding method.

6. The aircraft target detection method according to claim 5, characterized in that: Set a learnable scaling factor, multiply the scaling factor with the depth position code, concatenate it with the two-dimensional spatial position code, and embed it in the query and key as the three-dimensional position code.

7. The aircraft target detection method according to claim 1, characterized in that: An inner product operation is performed on the foreground task mark and the input feature, and the two-dimensional saliency map is reshaped by the Sigmoid activation function. An inner product operation is performed on the background task mark and the input feature, and the boundary map is reshaped by the Sigmoid activation function. And the process of generating visual saliency maps is trained by combining the two-dimensional saliency map and the boundary map.

8. The method for detecting an aircraft target according to claim 7, characterized in that: The training loss function is: In the formula, represents the total loss, represents the significant prediction loss, represents the boundary prediction loss, represents the dense prediction loss, represents the supervised prediction loss based on the task label, n represents the number of times the visual saliency map generation process is performed, represents the true value of the saliency map, represents the dense prediction value corresponding to the saliency map, represents the predicted value of the saliency map based on the task label, represents the boundary truth value, represents the dense prediction value corresponding to the boundary, Represents the boundary prediction value based on the task label.

9. A remote sensing image aircraft target detection device based on visual saliency detection, characterized in that: A visual saliency map is generated using a decoder, the decoder comprising: A feature extraction unit, used to extract features from an input remote sensing image or a feature saliency map and determine a corresponding mask; A selection-integration attention unit is used to divide the foreground area and the background area based on the extracted features according to the mask; and the input features are respectively concatenated with the foreground task label and the background task label to generate a query, the foreground area and the foreground task label are concatenated to generate a key, and the background area and the background task label are concatenated to generate a value, and cross attention is performed according to the generated query, key and value to obtain a visual saliency map; An aircraft detection unit is used to detect an aircraft according to the visual saliency map.

10. The aircraft target detection device according to claim 9, characterized in that: The decoders are connected in series in plurality.

Images