Remote sensing target detection method and system based on attention pyramid and transfer guidance

Abstract

The embodiment of the application provides a remote sensing target detection method and system based on attention pyramid and migration guidance, relates to the technical field of remote sensing image processing, and can extract multiscale discriminative spectral attention pyramid features from multispectral indexes of hyperspectral resolution images after acquiring multi-source remote sensing image data, and converts up-sampling of the spectral attention pyramid features into a spectral guide weight map through super-resolution feature migration, then injects the spectral guide weight map into different levels of a high-resolution backbone network through a multi-level spectral modulation in a complementary mechanism of spatial adaptive gating and channel-level spectral excitation, so as to output a detection result through the target detection model, realize full-process fine spectral guidance from local detail enhancement to global semantic calibration of the target detection task, and solve the problems of spectral spatial information fragmentation and invalid shallow fusion.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing technology, and in particular to a remote sensing target detection method and system based on attention pyramid and migration guidance. Background Technology

[0002] Remote sensing target detection is an image detection technology that identifies targets based on remote sensing imagery. After acquiring remote sensing images from satellites, aircraft, or other remote sensing vehicles, target detection models are used to identify geometric and structural information such as target contours, edges, textures, and shapes, thereby achieving target localization and classification. High-resolution remote sensing imagery provided by high spatial resolution commercial satellites or wide-swath multispectral satellites can be used as the basis for target detection during the process.

[0003] Due to the inherent incompleteness of information from a single remote sensing data source, its detection performance is low for targets with small pixel areas, weak spectral features, or those exhibiting spectral similarity. Therefore, a complementary detection approach using multi-source remote sensing data can be employed to overcome the limitations of a single data source. For example, a target detector based on Convolutional Neural Networks (CNN) can be run on high spatial resolution images, while a spectral classifier or anomaly detection algorithm can be run on low spatial resolution and high spectral resolution images. Then, spatial overlay or logical AND operations can be performed on the results to achieve multi-source remote sensing image detection fusion.

[0004] However, this process of independently processing and deciding on multi-source remote sensing images before fusing them is completely fragmented. This means that the discriminative information contained in the hyperspectral data is only used for post-hoc verification or coarse screening of high-resolution detection results. The discriminative power of hyperspectral data fails to provide active and continuous guidance in any form during the detector's feature learning process, leading to fragmented spectral spatial information and ineffective shallow fusion, thus reducing the accuracy of remote sensing target detection results. Summary of the Invention

[0005] In view of this, embodiments of this application provide a remote sensing target detection method and system based on attention pyramid and migration guidance to solve the problems of fragmented spectral spatial information and ineffective shallow fusion.

[0006] According to a first aspect of this application, a remote sensing target detection method based on attention pyramid and migration guidance is provided, the method comprising: Acquire multi-source remote sensing image data, including hyperspectral resolution images and high spatial resolution images; Spectral attention pyramid features are extracted from the hyperspectral resolution image, and two-stream features are extracted from the high spatial resolution image; the spectral attention pyramid features include a multi-scale spectral attention feature map obtained by spectral attention pyramid encoding; the two-stream features include detail features and context features; The spectral attention pyramid feature map is transformed into a spectral guidance signal through super-resolution spectral attention transfer; the spectral guidance signal includes a spectral guidance weight map aligned with the high-resolution feature space; A multi-level spectral modulation detection backbone network is used to generate spectral excitation features based on the spectral guidance weight map and the dual-stream features. The backbone network includes a shallow spatial gated modulation module and a deep channel spectral excitation module. The shallow spatial gated modulation module is used to perform spectral modulation on the detail features at the pixel level according to the spectral guidance signal. The deep channel spectral excitation module is used to reweight each channel of the deep features according to the context features. The spectral excitation features are input into the target detection model to output detection results, which include the bounding box coordinates, class label, and confidence score of the target predicted by the target detection model.

[0007] According to a second aspect of this application, a remote sensing target detection system based on attention pyramid and migration guidance is provided, the system comprising: The data preparation module is used to acquire multi-source remote sensing image data, which includes high-spectral resolution images and high-spatial resolution images. The feature extraction module is used to extract spectral attention pyramid features from the hyperspectral resolution image and dual-stream features from the high spatial resolution image; the spectral attention pyramid features include a multi-scale spectral attention feature map obtained through spectral attention pyramid encoding; the dual-stream features include detail features and context features; An attention transfer module is used to convert the spectral attention pyramid feature map into a spectral guidance signal through super-resolution spectral attention transfer; the spectral guidance signal includes a spectral guidance weight map aligned with the high-resolution feature space; A spectral modulation detection module is used to detect a backbone network using multi-level spectral modulation, generating spectral excitation features based on the spectral guidance weight map and the dual-stream features. The backbone network includes a shallow spatial gated modulation module and a deep channel spectral excitation module. The shallow spatial gated modulation module is used to perform spectral modulation on the detail features at the pixel level according to the spectral guidance signal. The deep channel spectral excitation module is used to reweight each channel of the deep features according to the context features. The result output module is used to input the spectral excitation features into the target detection model so as to output the detection result through the target detection model. The detection result includes the bounding box coordinates, class label and confidence score of the target to be detected predicted by the target detection model.

[0008] Based on the above technical solutions, this application provides a remote sensing target detection method and system based on attention pyramids and transfer guidance. After acquiring multi-source remote sensing image data, the method extracts spectral attention pyramid features from high-spectral-resolution images and detail and contextual features from high-spatial-resolution images. Then, through super-resolution spectral attention transfer, the spectral attention pyramid features are mapped and transformed into spectral guidance signals. Next, a multi-level spectral modulation detection backbone network is used to generate spectral excitation features based on the spectral guidance weight map and dual-stream features. The spectral excitation features are then input into a target detection model to output detection results. This method can extract multi-scale discriminative attention features from multispectral indices, and transform pyramid features into spectral guidance weight maps through super-resolution feature transfer. Then, through multi-level spectral modulation, the spectral guidance weight maps are injected into different levels of the high-resolution backbone network using a complementary mechanism of spatial adaptive gating and channel-level spectral excitation. This achieves refined spectral guidance for the entire target detection task, from local detail enhancement to global semantic calibration, solving the problems of fragmented spectral spatial information and ineffective shallow fusion.

[0009] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic diagram of the remote sensing target detection method based on attention pyramid and migration guidance provided in the embodiments of this application; Figure 2 This is a schematic diagram of the entire remote sensing target detection process provided in the embodiments of this application; Figure 3 This is a schematic diagram of the spectral attention pyramid encoding process provided in an embodiment of this application; Figure 4 This is a schematic diagram of the super-resolution migration process provided in an embodiment of this application; Figure 5This is a schematic diagram of the spectral attention weight generation process provided in an embodiment of this application; Figure 6 This is a schematic diagram of the channel spectrum excitation process provided in the embodiments of this application; Figure 7 This is a schematic diagram of the target detection model training process provided in the embodiments of this application; Figure 8 This is a schematic diagram of the multi-task collaborative training loss calculation process provided in an embodiment of this application; Figure 9 A schematic diagram of the structure of a remote sensing target detection system based on attention pyramid and migration guidance provided in an embodiment of this application. Detailed Implementation

[0011] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0012] In this embodiment, remote sensing target detection refers to an image detection process based on remote sensing imagery for target identification. Remote sensing target detection can be performed after remote sensing images are acquired by satellites, aircraft, or other remote sensing vehicles. A target detection model is then used to identify geometric and structural information such as target contours, edges, textures, and shapes within the remote sensing images, thereby achieving target localization and classification.

[0013] In some embodiments, high-resolution remote sensing imagery provided by high spatial resolution satellites or wide-swath multispectral satellites can be used as the basis for target detection during remote sensing target detection. High spatial resolution satellites, with their onboard sensors, can identify extremely small targets on the ground and can be used to capture detailed ground information. Wide-swath multispectral satellites cover a very wide area of ​​the ground in a single image, allowing for large-scale coverage and spectral analysis.

[0014] High-resolution remote sensing imagery can include hyperspectral resolution imagery and high spatial resolution imagery. Hyperspectral resolution imagery, also known as multispectral resolution imagery, refers to remote sensing imagery where the sensor records a large number of light wavebands, and each band has a narrow wavelength range. High spatial resolution imagery refers to remote sensing imagery where the actual size of the ground represented by a single pixel is relatively small.

[0015] Hyperspectral resolution imagery and high spatial resolution imagery can be acquired using different remote sensing image sensors or remote sensing satellites. For example, hyperspectral resolution imagery can be acquired using wide-swath multispectral satellites such as Sentinel-2 and Landsat-8 / 9.

[0016] High-spectral-resolution imagery acquired by wide-swath multispectral satellites provides rich spectral information, covering the visible, red-edge, near-infrared, short-wave infrared, mid-infrared, and thermal infrared ranges, with spatial resolutions ranging from 10 meters to 60 meters. Utilizing these abundant bands, a series of spectral indices with clear physical meaning and strong discriminative power can be calculated. Examples include the Fmask classic cloud index for cloud detection, the Normalized Dirt Standardized Index (NDBSI) for identifying bare land, the NDISI for identifying built-up areas, and the Floating Debris Index (FDI) for identifying marine debris. These indices effectively amplify the spectral differences between the target and background water bodies, vegetation, or other land features, providing strong discriminative evidence at the material property level.

[0017] High spatial resolution (HSR) imagery can be acquired using commercial HSR satellites such as WorldView-3 / 4, Gaofen-2, and Gaofen-7. These satellites provide panchromatic (PAN) imagery with sub-meter or even higher resolution (0.3-1 meter) and multispectral (MS) imagery with meter-level resolution (1-4 meters). HSR imagery data possesses exceptional spatial detail rendering capabilities. The geometric and structural information of targets, such as their outlines, edges, textures, and shapes, is clearly displayed in HSR imagery, providing a foundation for precise target location and preliminary classification. For example, the widely used Gaofen-2 satellite's multispectral sensor only covers four bands: blue, green, red, and near-infrared.

[0018] Due to the inherent incompleteness of information from a single remote sensing data source, its detection performance is low for targets with small pixel areas, weak spectral characteristics, or those exhibiting the phenomenon of foreign objects sharing the same spectrum. Therefore, a detection method that utilizes complementary multi-source remote sensing data can be used to improve the limitations of a single data source.

[0019] In some embodiments, a target detector based on Convolutional Neural Networks (CNN) can be run on high spatial resolution images and a spectral classifier or anomaly detection algorithm can be run on low spatial resolution and high spectral resolution images through independent processing and post-decision fusion paths. Then, spatial overlay or logical AND operation is performed at the result level to detect and fuse multi-source remote sensing images.

[0020] However, this process of independently processing and deciding on multi-source remote sensing images before fusing them is completely fragmented. This means that the discriminative information contained in the hyperspectral data is only used for post-hoc verification or coarse screening of high-resolution detection results. The discriminative power of hyperspectral data fails to provide active and continuous guidance in any form during the detector's feature learning process. In some embodiments, based on the resolution-aligned channel stacking fusion path, low-resolution multispectral data can be upsampled by interpolation and then directly concatenated with high-resolution data in the channel dimension before being input into the detection network.

[0021] The simple mathematical upsampling method used in the resolution-aligned channel stacking fusion path processing approach merely smooths the original low-resolution information and cannot obtain true high-resolution spectral information. Instead, it introduces severe blurring and spurious spectral patterns, contaminating the input data. Furthermore, the processing network lacks an explicit mechanism to process and distinguish information streams from different sources with different physical meanings and confidence levels. Spectral and spatial information compete equally at the channel level, making it difficult for the network to learn to effectively utilize spectral information. This results in weak spectral guidance and limited fusion effects.

[0022] In some embodiments, high-resolution images can be segmented based on shallow fusion paths of objects or features, and then low-resolution spectral values ​​can be assigned to the segmented images to achieve joint classification based on the spatial and spectral features of the segmented objects.

[0023] The detection accuracy of object- or feature-based shallow fusion path detection methods depends on the accuracy of the initial image segmentation. For small targets at the sub-pixel level with blurred edges and low contrast to the background, obtaining accurate and complete segmentation results is inherently challenging. Errors in the initial segmentation can directly lead to the complete failure of subsequent spectral feature assignment, resulting in poor fault tolerance and robustness of the entire process, making it unsuitable for small target detection scenarios.

[0024] As can be seen, in the above embodiments, it is difficult to refine and transfer the highly discriminative high-dimensional spectral features contained in low spatial resolution data in a "learnable, controllable, and faithful" manner at the deep feature level, and use them as a continuous guiding signal to dynamically modulate and enhance the learning of high-resolution spatial features. As a result, the advantages of the two data sources cannot be deeply, adaptively, and strongly correlated with the detection task within the neural network. Therefore, there are problems of fragmented spectral spatial information and ineffective shallow fusion, which reduces the accuracy of remote sensing target detection results.

[0025] To address the problems of fragmented spectral spatial information and ineffective shallow fusion, this application provides a remote sensing target detection method based on attention pyramids and transfer guidance in some embodiments. This method addresses the issues of fragmented spectral and spatial information and ineffective shallow fusion through end-to-end trainable multi-source remote sensing detection for small target scenarios. The method introduces a spectral attention pyramid into a guided network, extracting multi-scale discriminative attention features from multispectral indices. Then, through differentiable super-resolution feature transfer, the pyramid features are upsampled and transformed into pixel-level spectral attention maps aligned with high-resolution spatial features. Finally, through a multi-layered spectral modulation architecture, the spectral attention maps are adaptively and selectively injected into different layers of the high-resolution backbone network using two complementary mechanisms: spatial adaptive gating and channel-level spectral excitation. This achieves refined spectral guidance for the entire target detection task, from local detail enhancement to global semantic calibration.

[0026] The method can be applied to electronic devices with data processing capabilities. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control machines. For ease of description, this application embodiment uses an electronic device as the execution subject of the method. It should be understood that the method can also be applied to other types of execution subjects, which are not illustrated in this application embodiment. Figure 1 As shown, the method includes: S101. Acquire multi-source remote sensing image data.

[0027] When performing remote sensing target detection, multi-source remote sensing image data can be acquired first. This multi-source remote sensing image data includes hyperspectral resolution imagery and high spatial resolution imagery. The hyperspectral resolution imagery and high spatial resolution imagery are two remote sensing images acquired via a satellite platform that have undergone preprocessing such as radiometric calibration and atmospheric correction, and require spatial registration. The hyperspectral resolution imagery can be represented as:

[0028] In the formula, A data cube representing a hyperspectral resolution image; R Represents the set of real numbers; Indicates the image height of a hyperspectral resolution image; This represents the image width of a hyperspectral resolution image; This indicates the number of bands in a hyperspectral resolution image.

[0029] Similarly, high spatial resolution images can be represented as:

[0030] In the formula, Data cubes representing high spatial resolution images; R Represents the set of real numbers; This represents the image height of a high spatial resolution image; This represents the image width of a high spatial resolution image; This indicates the number of bands in a high spatial resolution image.

[0031] Hyperspectral resolution and high spatial resolution imagery can be acquired through different satellite platforms. For example, L2A-level surface reflectance products can be downloaded from the ESA Copernicus Open Access Centre to obtain hyperspectral resolution imagery acquired by the Sentinel-2 satellite. Similarly, high spatial resolution imagery, i.e., orthorectified surface reflectance data, can be obtained from the Gaofen-2 (GF-2) data download platform.

[0032] High spatial resolution imagery can include panchromatic (PAN) imagery and multispectral (MS) imagery. For example, panchromatic and multispectral imagery acquired using the GF-2 satellite constitute high spatial resolution imagery. The center wavelength of the panchromatic imagery is approximately 610 nm, with a spatial resolution of 0.8 meters; the multispectral imagery includes blue, green, red, and near-infrared bands, with a spatial resolution of 3.2 meters.

[0033] The image closest to the time of the Gaofen-2 (GF-2) satellite's transit within 2 hours can be selected as the hyperspectral resolution image. For example, a hyperspectral resolution image can include 7 bands: B2 (blue, 490nm, 10m), B3 (green, 560nm, 10m), B4 (red, 665nm, 10m), B8 (near-infrared, 842nm, 10m), B8A (red edge, 865nm, 20m), B11 (shortwave infrared 1, 1610nm, 20m), and B12 (shortwave infrared 2, 2190nm, 20m). The 20-meter resolution bands are upsampled to 10 meters using bicubic interpolation to align with other 10-meter bands.

[0034] In the process of acquiring multi-source remote sensing image data, cross-sensor registration can also be performed. That is, in some embodiments, when acquiring multi-source remote sensing image data, raw image data can be acquired first, wherein the raw image data includes first image data acquired from a first satellite platform and second image data acquired from a second satellite platform; the resolution of the second image data is higher than that of the first image data.

[0035] Next, based on the resolution scale of the first image data, the second image data is first upsampled and then downsampled to obtain a high spatial resolution image. Then, the target band image is extracted from the first image data and the high spatial resolution image respectively. Then, based on the scale-invariant feature transform algorithm, key points and key point descriptors are extracted from the target band image. The key point pairs are then filtered according to the random sampling consistency principle, and an affine transformation matrix is ​​fitted based on the key points.

[0036] Then, based on the affine transformation matrix, all band images in the first image data are resampled to obtain a hyperspectral resolution image. The hyperspectral resolution image is aligned on the spatial grid corresponding to the panchromatic band image of the high spatial resolution image.

[0037] For example, since Sentinel-2 and GF-2 come from different satellite platforms and have significantly different resolutions (Sentinel-2 has a resolution of 10 meters, while GF-2 has a resolution of 0.8 meters), high-precision registration can be used to facilitate subsequent fusion. During high-precision registration, resolution unification and feature extraction can be performed first. This involves upsampling the multispectral bands of GF-2 to 0.8 meters and aligning it with the PAN image, then downsampling it to 10 meters to match the scale of Sentinel-2. Because the near-infrared band and Sentinel-2's B8 band spectral range are similar, the near-infrared band of the downsampled GF-2 image and the B8 band of Sentinel-2 can be extracted separately at a 10-meter resolution.

[0038] Next, key points and their descriptors are extracted from two 10-meter resolution near-infrared images using feature matching and the Scale Invariant Feature Transform (SIFT) algorithm. Then, the Random Sample Consensus (RANSAC) algorithm is used to filter the matched key point pairs, eliminating mismatches, and an affine transformation matrix is ​​fitted. The fitting process requires the root mean square error (RMSE) of the matched point pairs to be less than 5 meters, or 0.5 Sentinel-2 pixels.

[0039] Then, through resampling and alignment, that is, using the affine transformation matrix obtained above, bicubic interpolation is used to resample all bands (B2, B3, B4, B8, B8A, B11, B12) of Sentinel-2, so that they are precisely aligned with the panchromatic band image of GF-2 on a 0.8-meter spatial grid.

[0040] Let the spatial resolution scaling factor between the two images be . And after registration, it satisfies , For a 640×640 pixel GF-2 PAN image, since 640 / 8=80, the resolution scaling factor s=10 / 0.8=12.5, it can be approximated by downsampling by 8 times to simplify the network design. The corresponding Sentinel-2 image block size is 80×80 pixels.

[0041] S102. Extract spectral attention pyramid features from hyperspectral resolution images and extract dual-stream features from high spatial resolution images.

[0042] After acquiring multi-source remote sensing image data, feature extraction can be performed on the multi-source remote sensing images. Specifically, spectral attention pyramid features can be extracted from hyperspectral resolution images, and dual-stream features can be extracted from high spatial resolution images. Among them, spectral attention pyramid features include multi-scale spectral attention feature maps obtained through spectral attention pyramid encoding. The multi-scale spectral attention feature maps are multi-scale discriminative attention features extracted from multispectral indices.

[0043] To extract spectral attention pyramid features, such as Figure 2 As shown, in some embodiments, when extracting spectral attention pyramid features from hyperspectral resolution imagery, a spectral index map can be computed in parallel from the hyperspectral resolution imagery. This spectral index map includes at least one of the following: floating debris index, normalized difference vegetation index, normalized difference water index, and cloud index. Then, an initial spectral attention map is generated by normalizing the spectral index map.

[0044] When extracting pyramid features, one can start with high-spectral resolution images. In this process, a set of M physically meaningful spectral index maps are computed in parallel, i.e. , where each spectral index plot Spectral index maps can highlight the spectral differences between the target and the background from different perspectives.

[0045] After calculating the spectral index plots, then analyze each spectral index plot... Normalization is performed using the Sigmoid function, mapping its range to the [0, 1] interval, to generate the initial spectral attention map, i.e. The closer the value of the initial spectral attention map is to 1, the higher the probability that the pixel location has the target spectral characteristics.

[0046] For example, on a registered 80×80 pixel Sentinel-2 image patch, spectral indices can be calculated, which may include floating debris index, normalized difference vegetation index, normalized difference water index, and cloud index.

[0047] The floating debris index can be calculated using the following formula:

[0048] in, ρ B8 , ρ B8A , ρ B11 These are the surface reflectance values ​​for the B8, B8A, and B11 bands, respectively; the wavelength unit for 842 is nanometers.

[0049] The normalized difference vegetation index can be calculated using the following formula:

[0050] Where, ρ B4 This represents the surface reflectance in the B4 band.

[0051] The normalized difference water index can be calculated using the following formula:

[0052] in, This represents the surface reflectance in the B3 band.

[0053] The cloud index can be calculated using the following formula:

[0054] in, ρ B2 This represents the surface reflectance in the B3 band.

[0055] Using the above formula for calculating spectral indices, four index graphs can be obtained, namely... I FDI , I NDVI , I NDWI , I CI ∈ R 80×80×1 .

[0056] Then, spectral attention maps are generated based on spectral indices. Specifically, the Sigmoid function is applied to each index map to normalize its values, mapping them to the [0, 1] interval, resulting in the initial spectral attention maps: , , , .

[0057] After generating the initial spectral attention map, the original hyperspectral band image from the hyperspectral resolution image and the initial spectral attention map can be stitched together along the channel dimension to form enhanced input features. These enhanced input features are then input into a spectral attention pyramid coding network, which encodes them to obtain a multi-scale spectral attention feature map output by the network. The spectral attention pyramid coding network comprises a network structure with multiple downsampling stages, and each downsampling stage includes convolutional layers, batch normalization layers, activation functions, and convolutional pooling layers.

[0058] When performing spectral attention pyramid coding, the original high-spectral resolution image can be used to... The hyperspectral band images are concatenated with the aforementioned M spectral attention maps along the channel dimension to form enhanced input features. The enhanced input features can be represented as:

[0059] In the formula, This indicates enhanced input features; A data cube representing a hyperspectral resolution image; , ..., These represent M spectral attention maps respectively; R Represents the set of real numbers; Indicates the image height of a hyperspectral resolution image; This represents the image width of a hyperspectral resolution image; This indicates the number of bands in a hyperspectral resolution image.

[0060] The enhanced input features are then fed into a lightweight spectral attention pyramid coding network. Spectral Attention Pyramid Coding Network It adopts an encoder-like structure, containing L downsampling stages. Each stage consists of a convolutional layer (Conv), a batch normalization layer (BN), an activation function (such as ReLU), and a convolutional or pooling layer with a stride of 2, achieving a gradual reduction in spatial size and an increase in the number of channels.

[0061] Spectral Attention Pyramid Coding Network It can output a set of multi-scale spectral attention feature maps, namely:

[0062] In the formula, Indicates the first l Spectral attention feature maps generated in each downsampling stage; L This indicates the number of downsampling stages contained in the spectral attention pyramid coding network; R Represents the set of real numbers; Indicates the image height of a hyperspectral resolution image; This represents the image width of a hyperspectral resolution image; This indicates the number of bands extracted in the l-th downsampling stage.

[0063] In the spectral attention feature map, the bottom feature map Preserving more spatial details and local spectral patterns, top-level feature map This includes more global and semantically stronger spectral contextual information. Multiple feature maps can be used to construct a multi-scale spectral attention pyramid, providing rich and robust spectral discrimination cues for subsequent feature transfer.

[0064] For example, such as Figure 3 As shown, the spectral attention pyramid coding network The original band images (B2, B3, B4, B8, B8A, B11, B12) acquired by the seven Sentinel-2 satellite platforms can be stitched together with four spectral attention maps along the channel dimension to obtain enhanced input features. F in ∈ R 80×80×11 Spectral Attention Pyramid Coding Network The network structure employs three downsampling stages: The network structure for stage 1 is Conv2d(11, 64, k=3, s=1, p=1)-BN-ReLU-Conv2d(64, 64, k=3, s=2, p=1)-BN-ReLU, and the output is... In this context, the first convolutional kernel Conv2d has 11 input channels and 64 output channels, k represents the kernel size, s represents the convolution stride, and p represents the padding width.

[0065] The stage 2 network structure is Conv2d(64, 128, k3, s=2, p=1)-BN-ReLU, and the output is... .

[0066] The stage 3 network structure is Conv2d(128, 256, k=3, s=2, p=1)-BN-ReLU, and the output is... Therefore, through three downsampling stages, spectral attention pyramid features can be constructed, i.e. .

[0067] The dual-stream features include detail features and contextual features. Detail features are derived from high spatial resolution imagery. The input consists of panchromatic imagery, or a high-resolution image resulting from the fusion of panchromatic and multispectral images, processed by a shallow convolutional neural network. Local information obtained through feature extraction.

[0068] A convolutional neural network (CNN) for extracting detailed features can consist of 2-3 convolutional blocks, each containing convolution, batch normalization, and ReLU activation units, and using a convolutional kernel with a stride of 2 for downsampling. The output of the CNN is then a high-resolution spatial detail feature, i.e.:

[0069] in, Indicate detailed features; R Represents the set of real numbers; This represents the image height of a high spatial resolution image; This represents the image width of a high spatial resolution image; This indicates the number of bands used in the detail feature extraction process. Detail feature extraction allows the extraction of detailed features containing rich local information such as edges and textures from high spatial resolution images.

[0070] Contextual features are high spatial resolution images The multispectral bands are used as input, and then processed by another convolutional neural network. The features obtained through feature extraction. Convolutional neural networks used for extracting contextual features can extract medium-resolution spectral contextual information, i.e.:

[0071] In the formula, Represents contextual features; R Represents the set of real numbers; This represents the image height of a high spatial resolution image; This represents the image width of a high spatial resolution image; This indicates the number of bands used in the context feature extraction process. Context features can provide region-level semantic information for subsequent detection.

[0072] To extract dual-stream features, in some embodiments, when extracting dual-stream features from high spatial resolution images, the panchromatic band image of the high spatial resolution image can first be normalized, and then the normalized panchromatic band image can be used as input to extract detail features through a detail feature network.

[0073] Furthermore, bicubic interpolation is used to upsample the multispectral band data of the high spatial resolution image to the target pixel size, followed by normalization. Then, the upsampled and normalized multispectral band data is used as input to extract contextual features through a context feature network.

[0074] For example, spectral attention-guided enhanced detection networks can be constructed to achieve spectral attention guidance. These networks can be based on the network structures of target detection models such as YOLO v11, and can be deeply modified to embed core modules such as high-resolution two-stream feature extraction, super-resolution spectral attention transfer, and multi-level spectral modulation detection backbone networks. Spectral attention-guided enhanced detection networks can perform two-stream feature extraction through high-resolution two-stream feature extraction.

[0075] Before performing dual-stream feature extraction, high spatial resolution imagery can be preprocessed. This can be done by using panchromatic (PAN) imagery from the GF-2 satellite platform. Normalization was performed to ensure pixel values ​​ranged between [0, 1]. Furthermore, the GF-2 satellite platform's multispectral (MS) imagery was upsampled to 640×640 pixels using bicubic interpolation and then normalized again to obtain the upsampled and normalized multispectral data. .

[0076] Then, a high-resolution dual-stream feature extraction network is invoked, which may include a detail feature network. and context feature network Among them, detail feature network The input is a normalized PAN image. The network structure is Conv2d(1, 16, k=3, s=2, p=1)-BN-ReLU-Conv2d(16, 32, k=3, s=2, p=1)-BN-ReLU, and the final output detailed features are... .

[0077] Context Feature Network The input is an upsampled and normalized MS image. The network structure is Conv2d(4, 64, k=3, s=2, p=1)-BN-ReLU-Conv2d(64, 64, k=3, s=2, p=1)-BN-ReLU, and the final output context features are... .

[0078] S103. Through super-resolution spectral attention transfer, the spectral attention pyramid feature map is transformed into a spectral guidance signal.

[0079] After feature extraction from multi-source remote sensing images, the spectral attention pyramid feature map can be transformed into a spectral guidance signal through super-resolution spectral attention transfer. This spectral guidance signal includes a spectral guidance weight map aligned with the high-resolution feature space. Therefore, electronic devices can use a super-resolution spectral attention transfer module to transform low-resolution spectral attention pyramid feature maps into a spectral guidance signal strictly aligned with the high-resolution feature space.

[0080] In order to perform super-resolution spectral attention transfer, in some embodiments, when mapping the spectral attention pyramid features into spectral guidance signals, the spectral attention pyramid features can be first upsampled from top to bottom and laterally connected to obtain preliminary fused features.

[0081] When performing super-resolution spectral attention transfer, the super-resolution spectral attention transfer module can perform learnable upsampling and feature pyramid fusion. Based on the idea of ​​Feature Pyramid Network (FPN), it integrates the spectral attention pyramid... Perform top-down upsampling and lateral connections.

[0082] like Figure 4 As shown, the specific process of upsampling and lateral connection is as follows: starting from the top-level features... Initially, its spatial size is magnified by a factor of 2 using transposed convolution or nearest neighbor upsampling, and then compared with features from the encoder at the same scale. The features are obtained by adjusting the number of channels using a 1×1 convolution and then adding them element-wise, followed by a 3×3 convolution for fusion. This process is repeated iteratively until the data is restored to the level of the underlying features. Similar scale. Then, through an additional upsampling operation, such as 4x upsampling, the fused features are adjusted to match the high-resolution detail features. F d Same space size (H) hr / 4)×(W hr / 4), thus obtaining preliminary fusion characteristics, namely:

[0083] In the formula, Indicates preliminary fusion characteristics; This represents the image height of a high spatial resolution image; This represents the image width of a high spatial resolution image; This indicates the number of bands used in the feature fusion process. The process of generating preliminary fused features can effectively aggregate multi-scale spectral information, thereby improving spatial resolution.

[0084] In some embodiments, to obtain preliminary fused features, a 2x nearest neighbor upsampling can be performed on the top-level features of the spectral attention pyramid to obtain top-level upsampled features. Then, a first convolution operation is applied to the second-to-top-level features of the spectral attention pyramid to generate first-scale features. The top-level upsampled features and the first-scale features are then added element-wise and fused using a second convolution operation to obtain the first feature to be fused. The first feature to be fused is then subjected to a 2x nearest neighbor upsampling to obtain the second-scale features, and a first convolution operation is applied to the bottom-level features of the spectral attention pyramid to generate third-scale features. The second-scale features and the third-scale features are then added and fused using a second convolution operation to obtain the second feature to be fused. Finally, a 4x nearest neighbor upsampling is performed on the second feature to obtain the preliminary fused features. It can be seen that the spatial size of the preliminary fused features is consistent with the detail features.

[0085] For example, such as Figure 5 As shown, the FPN-style upsampling fusion path can be achieved through the super-resolution spectral attention transfer module, that is, by upsampling the top-level features... Perform double nearest neighbor upsampling to obtain 20×20×256 features, which will be used as the top-level upsampled features. Then, perform upsampling on the next lower-level features... Adjust the number of channels using 1×1 convolution: Conv2d(128, 256, k=1), outputting first-scale features with a size of 20×20×256.

[0086] Next, add the two features from the previous step element-wise, and then fuse them using a 3×3 convolution: Conv2d(256, 128, k=3, p=1)-BN-ReLU, to obtain the first feature to be fused. Then, for the first feature to be fused... Performing double nearest neighbor upsampling yields a second-scale feature of 40×40×128.

[0087] By analyzing the underlying features A 1×1 convolution operation, Conv2d(64, 128, k=1), is applied to output a 40×40×128 third-scale feature. Then, the second-scale and third-scale features are added together and fused using a 3×3 convolution: Conv2d(128, 64, k=3, p=1)-BN-ReLU, to obtain the second feature to be fused. For the second feature to be fused Perform 4x nearest neighbor upsampling to improve its spatial size and detail features. Consistency was obtained, and preliminary fusion characteristics were obtained. .

[0088] After obtaining the initial fused features, a spatial attention mechanism is employed to perform convolutional calculations on the initial fused features and generate a spatial attention weight map through an activation function. A channel attention mechanism is then used to perform global average pooling on the initial fused features to obtain channel description vectors, and a channel weight vector is generated based on the channel description vectors using a multilayer perceptron and an activation function. Finally, feature fusion is performed based on the spatial attention weight map and the channel weight vectors to generate a spectral guided weight map.

[0089] In order to To extract more accurate and robust spectral guidance weights, a lightweight, refined unit can be used as the spectral attention modulation head. The spectral attention modulation head can employ spatial attention and channel attention mechanisms in parallel, forming spatial attention branches and channel attention branches respectively.

[0090] For spatial attention branches, preliminary feature fusion can be performed. A 3×3 convolutional layer is applied to compress the number of channels to 1, and then a sigmoid activation function is used to generate a spatial attention weight map. , used to emphasize spatial locations with strong spectral responses.

[0091] For channel attention branches, preliminary feature fusion can be performed. Global average pooling (GAP) is performed to obtain a channel description vector, and then a channel weight vector is generated through a small multilayer perceptron (MLP) and a sigmoid function. This is used to highlight the characteristic channels most relevant to the target spectral properties.

[0092] Then, through feature fusion, the features after channel weighting are... Features weighted by spatial attention Element-wise addition is performed, followed by fusion and dimensionality reduction through a 1×1 convolutional layer, ultimately outputting a high-resolution spectral guided weight map, i.e.:

[0093] In the formula, This represents a spectral guided weighting diagram; This represents the image height of a high spatial resolution image; This represents the image width of a high spatial resolution image. The value of each pixel in the spectral-guided weighted map is between 0 and 1, which visually reflects the likelihood of that location being identified as a target based on multispectral information.

[0094] For example, the input to the spectral attention weight generation module is the preliminary fusion features. Spatial attention and channel attention mechanisms are employed in parallel to form spatial attention branches and channel attention branches. The network structure of the spatial attention branch is Conv2d(64, 32, k=3, p=1)-BN-ReLU-Conv2d(32, 1, k=3, p=1)-Sigmoid, and the output of the spatial attention branch is the spatial attention weights, i.e. .

[0095] The network structure of the channel attention branch is GlobalAvgPool2d-Linear(64, 16)-ReLU (bottleneck structure, reduction ratio r=4)-Linear(16, 64)-Sigmoid, and the output of the channel attention branch is the channel weight vector. This can be reshaped into a 1×1×64 vector and combined with the initial fusion features. Perform channel multiplication.

[0096] Then, through feature fusion, the channel-weighted features are combined with the spatial attention-weighted features (which will...) Element-wise addition is performed (replicated 64 times in the channel dimension). The network structure Conv2d(64, 1, k=1)-Sigmoid is then used for fusion and dimensionality reduction to ensure the output is within [0, 1]. The final output is a high-resolution spectral guided weight map, i.e.: .

[0097] S104. Use multi-level spectral modulation to detect the backbone network, and generate spectral excitation features based on the spectral guided weight map and dual-stream features.

[0098] By employing super-resolution spectral attention transfer, the spectral attention pyramid feature map is transformed into a spectral guidance weight map. A multi-level spectral modulation detection backbone network can then be used to generate spectral excitation features based on the spectral guidance weight map and dual-stream features. G spec It is injected into the backbone network of target detection, and by modifying the backbone network of target detection models such as YOLO v11, a spectral modulation mechanism is introduced at multiple levels.

[0099] Therefore, the backbone network includes a shallow spatial gated modulation module and a deep channel spectral excitation module. The shallow spatial gated modulation module is used to perform spectral modulation on detailed features at the pixel level according to the spectral guidance signal; the deep channel spectral excitation module is used to reweight each channel of the deep features according to the context features.

[0100] In order to generate spectral excitation features, in some embodiments, the detail features can be multiplied element-wise with the spectral guided weight map through a shallow spatial gating modulation module to obtain modulation features, and then the modulation features and context features can be spliced ​​together in the channel dimension to form fused input features.

[0101] Shallow spatial gating modulation modules can be located in the input layer or shallow layers of the backbone network, and can guide the weight map of the spectrum. G spec Upsampling to detail features via bilinear interpolation F d Exactly the same size. Then upsampled... G spec With detailed features F d Element-wise multiplication (Hadamard Product) yields the modulation characteristics, namely:

[0102] in, Indicates modulation characteristics; Indicate detailed features; This represents a spectral guided weighting diagram; This indicates an upsampling operation. By generating modulation features, high spatial resolution spatial details can be spectrally modulated at the pixel level, enhancing the intensity of detail features in regions with strong responses in the spectral attention map while suppressing detail features in regions with weak responses. The modulation features generated through modulation... Then, with contextual features The data is stitched together along the channel dimension to form a fused feature. , as input to the improved backbone network.

[0103] Next, the deep channel spectral excitation module performs global average pooling on the spectral guided weight map or fused input features to obtain a global spectral description vector. This global spectral description vector is then input into a multilayer perceptron to learn channel weight vectors. Finally, the channel weight vector is multiplied with the feature map of the current layer along the channel direction to obtain spectral excitation features.

[0104] Deep channel spectral activation modules can be located deep within the backbone network, such as after the 3rd or 5th cross-stage partial network (CSP) layers. The input to a deep channel spectral activation module includes the feature map of the current layer. And the global spectral context information obtained from the super-resolution transfer module. Therefore, when performing channel spectral excitation, a spectral context vector can be generated first, i.e., a weighted graph is guided by the high-resolution spectrum. or the high-dimensional fusion features behind it Perform global average pooling to obtain a global spectral description vector. .

[0105] Next, channel weight prediction is performed by inputting the global spectral description vector z into a small MLP to learn the channel weight vector. ,in W 1, W 2 is the learnable weight matrix. δ It is the ReLU activation function. σ This is the Sigmoid function. This channel weight vector encodes which feature channels are more relevant to the spectral properties of the target for the current task.

[0106] Then feature recalibration is performed, applying the learned channel weights. With the feature map of the current layer Perform multiplication along the channel direction, that is: Therefore, based on the global spectral context information, the channels of deep features are reweighted, enhancing the channel responses related to the target spectrum and suppressing the responses of irrelevant channels.

[0107] For example, the multi-level spectral modulation detection backbone network is modified based on the YOLOv11 backbone network (CSPDarknet). Then, the input feature map size for the backbone network is... We can first use shallow spatial gating modulation to guide the weighting of the spectrum. Upsample to 160×160×1 using bilinear interpolation; no further action is needed if the dimensions already match. Then refine the details. Compared with upsampling Perform element-wise multiplication Modulated detailed features With context features By concatenating along the channel dimension, the fused features are obtained. ,Right now Then, the fusion features will be... As input to the improved backbone network.

[0108] The multi-level spectral modulation detection backbone network can also integrate a deep channel spectral excitation module, such as... Figure 6 As shown, in the YOLO v11 backbone network, Channel Spectrum Excitation Modules (CSE Modules) are inserted after the 3rd and 5th CSPrayers. Taking the insertion after the 3rd CSPrayer as an example, the output features of the 3rd CSPrayer are... Then the CSE module can first generate the spectral context vector, and then use the high-dimensional fusion features generated in the modulation unit. Perform global average pooling to obtain a global spectral description vector. The global spectral description vector encodes the global spectral attribute context of the entire image.

[0109] Then, channel weight prediction is performed. The global spectral description vector z is input into an MLP to learn the channel weight vector. This MLP structure employs a bottleneck design to reduce the number of parameters and introduce nonlinearity, such as Linear (in_features=64, out_features=16) - ReLU (bottleneck layer, reduction rate r=4) - Linear (in_features=16, out_features=256) - Sigmoid, outputting the channel weight vector. ,in and For learnable weight matrix, δ Activated for ReLU σ It's a Sigmoid vector. Each element corresponds to A feature channel, with a value between 0 and 1, represents the importance of the channel feature in relation to the current global spectral context.

[0110] Then feature recalibration is performed, and the learned channel weight vectors are... It is reshaped to a 1×1×256 dimension, and then broadcast to the feature map. Perform channel-by-channel multiplication, that is This adaptively enhances the response of feature channels highly correlated with the target spectral characteristics based on global spectral context information, while suppressing the responses of irrelevant or interfering channels. (Calibrated features) Replace the original features The input is fed into subsequent layers of the backbone network. The CSE module inserted after the 5th CSPlayer has a similar structure, but the output dimension of the second fully connected layer of the MLP needs to be adjusted according to the number of channels in the output feature map of that layer.

[0111] S105. Input the spectral excitation features into the target detection model so that the target detection model can output the detection results.

[0112] After generating spectral excitation features based on the spectral guided weight map and two-stream features, these features can be input into the target detection model to output detection results. The features extracted by the backbone network, after shallow spatial modulation and deep channel modulation, simultaneously contain fine spatial details enhanced by spectral information and high-level semantic information calibrated by spectral context. These features can be fed into the detection head of feature extraction models such as YOLO v11 to predict the target's bounding box, category, and confidence score. Therefore, the detection results include the bounding box coordinates, category label, and confidence score of the target predicted by the target detection model.

[0113] For example, the spectral excitation features extracted by the backbone network through the aforementioned multi-level spectral modulation can be fed into the original detection head of YOLO v11. The detection head includes a path aggregation network (PANet) and detection layers at multiple scales to predict the bounding box, class, and objectness score of the target. By maintaining consistency with the YOLOv11 configuration in the detection head structure, it can output feature maps at three scales: 20×20, 40×40, and 80×80, to accommodate targets of different sizes.

[0114] In order to obtain detection results, in some embodiments, when inputting spectral excitation features into the target detection model to output detection results through the target detection model, the spectral excitation features can be input into the target detection model to obtain the prediction tensor output by the target detection model, wherein the prediction tensor includes bounding box offset, object confidence and class probability.

[0115] The bounding box offsets are then decoded into absolute bounding boxes in the image coordinate system. Preset object confidence thresholds and class probability thresholds are applied to filter these absolute bounding boxes to determine reliable prediction boxes. Then, a non-maximum suppression algorithm is applied to remove overlapping redundant boxes from the reliable prediction boxes. Finally, the object confidence score of each reliable prediction box after removing overlapping redundant boxes is multiplied by its class probability to obtain the confidence score of the reliable prediction box.

[0116] For example, after a target detection model is trained, it can be deployed for inference. Therefore, for a new pair of remote sensing images acquired by the Sentinel-2 and Gaofen-2 satellite platforms, data preprocessing can be performed first to ensure that the two input images have undergone radiometric calibration, atmospheric correction, and orthorectification. Then, a registration process is executed to resample the Sentinel-2 images to ensure strict alignment with the Gaofen-2 PAN images. Finally, spectral index calculation and spectral attention pyramid encoding are performed on the registered Sentinel-2 image blocks to obtain... Normalize the GF-2 PAN image and upsample and normalize the GF-2 MS image to obtain... and .

[0117] Then, through the forward propagation of the object detection model network, , , The input is fed into a pre-trained spectral attention-guided augmentation detection network. This network can then automatically perform high-resolution feature extraction, spectral attention super-resolution transfer and refinement, multi-level spectral modulation, and finally, object detection prediction. The network outputs a prediction tensor at three scales, including bounding box offsets, object confidence, and class probability.

[0118] Then, through post-processing, the predicted offsets output by the network are decoded into absolute coordinate bounding boxes in the image coordinate system, i.e., [ x min , y min , x max , y max The predicted bounding boxes are initially filtered using object confidence thresholds and class probability thresholds to remove obviously unreliable boxes. Non-maximum suppression (NMS) is then applied to the filtered boxes to remove highly overlapping redundant boxes. The mean intersection-overlap (IoU) threshold for the NMS algorithm is set to 0.5. For the retained predicted boxes, their object confidence score is multiplied by their class probability to obtain the final confidence score.

[0119] Then, the results are output, that is, the object detection model outputs the final list of detection results. Each detection result in the list can include bounding box coordinates (which can be converted to geographic coordinates), class label, and confidence score.

[0120] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a remote sensing target detection method based on attention pyramid and transfer guidance. The difference between this method and the above embodiments is that the target detection model can be trained based on the feature extraction process in the above embodiments, such as... Figure 7 As shown, the method includes: S201. Initialize the network parameters of the target detection model; S202. Input the training data into the object detection model to obtain the training detection results output by the object detection model; S203. Calculate the training loss based on the training detection results according to the total loss function; S204. Iteratively optimize the network parameters of the target detection model based on training loss.

[0121] To train the object detection model, the electronic device can first initialize the network parameters of the object detection model. Specifically, the convolutional layers of the spectral attention pyramid coding network, detail feature network, and context feature network in the object detection model are initialized with a normal distribution; the transposed convolutional layers in the backbone network are initialized with bilinear interpolation kernels; and the fully connected layers in the backbone network are initialized with a uniform distribution.

[0122] The training data is then input into the object detection model to obtain the training detection results output by the model. The training loss is then calculated based on the training detection results using the total loss function. This total loss function is a weighted sum of the original detection loss, the spectral attention alignment loss, and the spectral guidance effectiveness loss. The network parameters of the object detection model are then iteratively optimized based on the training loss.

[0123] To ensure the accuracy of spectral attention transfer and the effectiveness of the spectral guidance mechanism, electronic devices can perform multi-task co-optimization training with spectral consistency constraints, enabling end-to-end joint training through a multi-task loss function. The total loss function can be expressed as:

[0124] in, The standard loss for object detection includes bounding box regression loss such as CIoU Loss, classification loss such as FocalLoss, and object-specific loss. This represents the spectral attention alignment loss, used to provide a supervisory signal for the super-resolution migration module, which constructs pseudo-true values ​​using high-resolution multispectral data.

[0125] When calculating the spectral attention alignment loss, one can start with high-resolution images. The corresponding spectral index maps are calculated in the multispectral bands and normalized using Sigmoid to obtain high-resolution spectral attention pseudo-true value maps. Then, the spectral guided weight map predicted by the network is calculated. With spectral attention pseudo-true value map The loss between these parameters is used to constrain the spectral attention map generated by the network to be as spatially consistent as possible with the true high-resolution spectral response.

[0126] This represents the loss of spectral guidance effectiveness. The loss of spectral guidance effectiveness can enhance the guiding role of the spectral attention map in the detection process. Therefore, an adaptive loss weighting strategy can be designed to calculate the weight of the corresponding region in the spectral guidance map for each anchor or predicted box labeled as a positive sample (containing the target). Average response value Then, the classification loss for the positive sample is weighted, i.e.:

[0127] In the formula, y Indicates the true label, that is y =1 indicates that the target is... y =0 indicates background; p α and γ are the class probabilities predicted by the network, and α and γ are hyperparameters in Focal Loss. Therefore, for positive sample regions with strong spectral responses, the network is encouraged to make higher-confidence correct predictions; for regions with weak spectral responses but still representing the target, the loss weights are increased, prompting the network to strengthen its learning. Thus, spectral-guided effectiveness loss... It is the sum of the weighted classification losses of all positive samples, which can establish a positive feedback loop between the spectral response and the detection confidence, forcing the network to effectively utilize spectral guidance information.

[0128] For example, such as Figure 8 As shown, when training an object detection model, network parameter initialization can be performed first, i.e., for the spectral attention pyramid coding network... High-resolution dual-stream feature extraction network and The convolutional layers are initialized using a Kaiming normal distribution. The transposed convolutional layers in the super-resolution transfer module are initialized using a bilinear interpolation kernel. The fully connected layers in the Channel Spectral Excitation (CSE) module are initialized using a Xavier uniform distribution.

[0129] Then, the loss function components are calculated, that is, for L total =L det +λ 1 L align +λ 2 L guide The total loss function can be configured with balancing hyperparameters. λ 1 = 0.5 λ 2 = 0.2. Wherein, the original detection loss function... L det You can use the default YOLOv11 settings, which include: bounding box loss. CIoU loss; classification loss Binary cross-entropy loss (single-class case); target-oriented loss Binary cross-entropy loss, etc.

[0130] Spectral attention alignment lossL align The aim is to supervise the super-resolution migration module to generate accurate Since pixel-level true values ​​of spectral attention are unavailable, approximate supervisory signals can be constructed using high-resolution GF-2 multispectral data. Spectral indices are approximated using the four MS bands of GF-2. Although the lack of key bands such as SWIR in GF-2 prevents the calculation of indices like FDI, this process provides a reasonably reasonable spatial distribution of the spectral response. Sigmoid normalization is then applied to each approximate index map, and the pixel-level maximum value is taken from the four normalized index maps to obtain the final pseudo-true value map. The maximum value operation here is to fuse the responses of multiple exponentials to form a more complete indication of the potential target region. The loss is calculated using binary cross-entropy loss (BCE Loss). and The difference between them, therefore: ; in, N =160×160 is the total number of pixels. i For pixel index; This represents a spectral guided weighting diagram; This represents the pseudo-true value map of spectral attention.

[0131] Loss of spectral guidance effectiveness The aim is to establish a strong correlation between the spectral response and the detection results, forcing the network to utilize... To make a decision, the following step is needed: calculate the loss of spectral guidance effectiveness. First, the response value can be calculated. For each predicted bounding box that is matched as a positive sample in the training batch, the corresponding region in the spectral guide map can be calculated. The average pixel value is used as the spectral response intensity of the sample. The weighted classification loss is then applied to the original classification loss for the positive sample. (i.e., Focal Loss) is dynamically weighted. The weighting strategy is as follows: the higher the r, the stronger the spectral response, indicating that the location is more likely to be the target, and the network should be more confident. Therefore, the weighting coefficients should make the loss relative to the weight not less than 1 to reinforce learning.

[0132] At the same time, attention should also be paid to difficult samples or positive samples with low r due to annotation noise. That is, a smooth weighting function should be used: Where β=0.5 is a scaling factor, the weighted classification loss is: Total guiding loss Defined as the ratio of the sum of the weighted classification losses of all positive samples in the batch to the number of positive samples, i.e.:

[0133] in, The number of positive samples. k For sample index.

[0134] Based on the aforementioned total loss function, the optimizer and hyperparameters of the object detection model can be configured. The optimizer can be a stochastic gradient descent (SGD) optimizer, with momentum and weight decay. The base learning rate is set to 0.01; the momentum is set to 0.937; and the weight decay is set to 5 × 10⁻⁶. -4 The learning rate scheduling strategy is to use a warm-up cosine annealing scheduler.

[0135] During the warmup epochs, 3 epochs can be used to linearly increase the learning rate from an initial value of 0.00 to a set base learning rate of 0.01. The total number of training epochs can be set to 300 epochs. After the warmup period, the learning rate decays from lr=0.01 to a minimum learning rate of 1×10⁻⁶ according to a cosine function. -5 The batch size can be set to 8 based on GPU memory. Gradient clipping thresholds can also be set to 10.0 to prevent gradient explosion during the early stages of training.

[0136] Data augmentation strategies can also be implemented during training. Specifically, during the training phase, identical random spatial and color transformations are applied synchronously to the input GF-2 PAN image, GF-2 MS image, and corresponding bounding boxes to ensure data consistency.

[0137] During data augmentation, Mosaic augmentation can be used. Four training images are randomly selected with a probability of 0.5 and stitched together into one image for training. This increases the contextual diversity of the single batch of input data and provides more small target samples, which helps improve the model's ability to detect small targets.

[0138] Image enhancement can also be achieved through basic geometric transformations, namely, random horizontal flipping with a probability of 0.5, random vertical flipping with a probability of 0.3, random rotation angle range of ±10 degrees, and random scaling ratio range of 0.5 to 1.5 times.

[0139] Image enhancement can also be achieved through color dithering, which involves making slight adjustments to the brightness, contrast, and saturation of GF-2 MS images by an increment of 0.1 to simulate different lighting and atmospheric conditions. During all these enhancement operations, the Sentinel-2 data and its derived spectral attention pyramid features do not participate in spatial geometric transformations because they are precisely registered with the GF-2 data. Any spatial transformation of the GF-2 data should result in an identical transformation of the corresponding Sentinel-2 data block. In practice, the paired GF-2 and Sentinel-2 data can be enhanced synchronously first, and then the spectral attention pyramid can be calculated from the enhanced Sentinel-2 data.

[0140] By applying the technical solutions of the above embodiments, the remote sensing target detection method based on attention pyramid and transfer guidance described in the above embodiments can achieve explicit modeling and utilization of multi-scale spectral discrimination features based on spectral attention pyramid encoding and guidance mechanisms. By transforming multiple physically meaningful spectral indices into attention maps and constructing multi-scale pyramid features, it can not only more robustly capture the spectral patterns of targets in different spatial contexts, but also express spectral information in a form compatible with deep learning, such as attention, providing a rich, structured, and easily fused information source for subsequent guidance.

[0141] The method can also achieve semantic fidelity and spatial projection of spectral guidance signals from low resolution to high resolution through differentiable super-resolution spectral attention transfer. By employing a learnable network incorporating FPN-style fusion and attention modulation, it adaptively learns how to map low-resolution spectral discrimination information to high-resolution space according to task requirements, generating a spatial attention map. G spec It retains the discriminative nature of spectral semantics and possesses spatial details aligned with high-resolution features, thus overcoming the drawbacks of interpolation methods that introduce false information.

[0142] The proposed method employs a multi-layered spectral modulation architecture to achieve deep coupling of spectral information with the detection network throughout the entire process, from local to global and from details to semantics. Through shallow spatial gating modulation, spectral attention is used to enhance relevant spatial details at the pixel level, while a deep-channel spectral excitation module recalibrates high-level semantic features at the feature channel level using global spectral context. This modulation strategy allows spectral guidance information to permeate all stages of feature extraction, achieving the fusion of two information sources within the neural network and enhancing the model's ability to reason based on spectral attributes.

[0143] The method also constructs a spectrally consistent multi-task optimization framework, ensuring the accuracy and effectiveness of the guidance mechanism through joint learning. It also introduces... L alignTo mitigate losses, the quality of the generated spectral attention map is constrained by the supervisory signal provided by high-resolution data, ensuring the physical reliability of the guiding signal. Simultaneously, this is combined with... L guide The auxiliary loss creates an adaptive weighted feedback between the spectral response and detection performance, forcing the network to proactively and effectively utilize spectral information for decision-making. By co-optimizing the auxiliary loss and the main detection loss, it is possible to ensure from different perspectives that the entire system learns correct and useful spectral spatial correlations.

[0144] Furthermore, the proposed method supports end-to-end training, features a flexible model architecture, and possesses good versatility and scalability. By integrating spectral attention generation, super-resolution transfer, multi-level modulation, and detection into a unified end-to-end deep learning framework, all parameters can be jointly optimized using gradient descent. Moreover, this framework is not dependent on specific satellite sensor combinations and is applicable to any multi-source remote sensing data with complementary characteristics of hyperspectral low-spatial and high-spatial low-spectral data, providing a feasible technical solution for solving a wider range of multi-source remote sensing small target detection problems.

[0145] In some embodiments, as a specific implementation of the remote sensing target detection method based on attention pyramid and migration guidance described in the above embodiments, some embodiments of this application also provide a remote sensing target detection system based on attention pyramid and migration guidance, such as... Figure 9 As shown, the system includes: The data preparation module is used to acquire multi-source remote sensing image data, which includes hyperspectral resolution images and high spatial resolution images. The feature extraction module is used to extract spectral attention pyramid features from the hyperspectral resolution image and dual-stream features from the high spatial resolution image; the spectral attention pyramid features include a multi-scale spectral attention feature map obtained through spectral attention pyramid encoding; the dual-stream features include detail features and context features; An attention transfer module is used to convert the spectral attention pyramid feature map into a spectral guidance signal through super-resolution spectral attention transfer; the spectral guidance signal includes a spectral guidance weight map aligned with the high-resolution feature space; A spectral modulation detection module is used to detect a backbone network using multi-level spectral modulation, generating spectral excitation features based on the spectral guidance weight map and the dual-stream features. The backbone network includes a shallow spatial gated modulation module and a deep channel spectral excitation module. The shallow spatial gated modulation module is used to perform spectral modulation on the detail features at the pixel level according to the spectral guidance signal. The deep channel spectral excitation module is used to reweight each channel of the deep features according to the context features. The result output module is used to input the spectral excitation features into the target detection model so as to output the detection result through the target detection model. The detection result includes the bounding box coordinates, class label and confidence score of the target to be detected predicted by the target detection model.

[0146] By applying the technical solutions of the above embodiments, the remote sensing target detection system based on attention pyramid and transfer guidance described in the above embodiments can extract spectral attention pyramid features from high-spectral-resolution images and detail features and contextual features from high-spatial-resolution images after acquiring multi-source remote sensing image data. Then, through super-resolution spectral attention transfer, the spectral attention pyramid features are mapped and transformed into spectral guidance signals. Next, a multi-level spectral modulation detection backbone network is used to generate spectral excitation features based on the spectral guidance weight map and dual-stream features. The spectral excitation features are then input into the target detection model to output detection results. The system can extract multi-scale discriminative attention features from multispectral indices, and transform pyramid features into spectral guidance weight maps through super-resolution feature transfer. Then, through multi-level spectral modulation, the spectral guidance weight maps are injected into different levels of the high-resolution backbone network using a complementary mechanism of spatial adaptive gating and channel-level spectral excitation. This achieves refined spectral guidance for the entire target detection task, from local detail enhancement to global semantic calibration, solving the problems of fragmented spectral spatial information and ineffective shallow fusion.

[0147] It should be noted that other corresponding descriptions of the functional units involved in the remote sensing target detection system based on attention pyramid and migration guidance provided in the embodiments of this application can be found in the corresponding descriptions in the remote sensing target detection method based on attention pyramid and migration guidance provided in the above embodiments, and will not be repeated here.

[0148] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A remote sensing target detection method based on attention pyramid and transfer guidance, characterized in that, The method includes: Acquire multi-source remote sensing image data, including hyperspectral resolution images and high spatial resolution images; Spectral attention pyramid features are extracted from the hyperspectral resolution image, and two-stream features are extracted from the high spatial resolution image; the spectral attention pyramid features include a multi-scale spectral attention feature map obtained by spectral attention pyramid encoding; the two-stream features include detail features and context features; Super-resolution spectral attention transfer transforms the spectral attention pyramid feature map into a spectral guidance signal. The spectral guidance signal includes a spectral guidance weight map aligned with the high-resolution feature space. The transformation of the spectral attention pyramid feature map into the spectral guidance signal involves: top-down upsampling and lateral concatenation of the spectral attention pyramid features to obtain preliminary fused features; performing convolutional computation on the preliminary fused features using a spatial attention mechanism, and generating a spatial-dimensional attention weight map using an activation function; performing global average pooling on the preliminary fused features using a channel attention mechanism to obtain channel description vectors, and generating channel weight vectors based on the channel description vectors using a multilayer perceptron and an activation function; and performing feature fusion based on the spatial-dimensional attention weight map and the channel weight vectors to generate the spectral guidance weight map. A multi-level spectral modulation detection backbone network is used to generate spectral excitation features based on the spectral guided weight map and the two-stream features. The backbone network includes a shallow spatial gated modulation module and a deep channel spectral excitation module. The shallow spatial gated modulation module is used to perform spectral modulation on the detail features at the pixel level according to the spectral guided signal. The deep channel spectral excitation module is used to reweight each channel of the deep features according to the context features. Generating spectral excitation features based on the spectral guided weight map and the two-stream features includes: multiplying the detail features element-wise with the spectral guided weight map through the shallow spatial gated modulation module to obtain modulation features; concatenating the modulation features with the context features along the channel dimension to form fused input features; performing global average pooling on the spectral guided weight map or the fused input features through the deep channel spectral excitation module to obtain a global spectral description vector; inputting the global spectral description vector into a multilayer perceptron to learn channel weight vectors; and multiplying the channel weight vector with the feature map of the current layer along the channel direction to obtain the spectral excitation features. The spectral excitation features are input into the target detection model to output detection results, which include the bounding box coordinates, class label, and confidence score of the target predicted by the target detection model.

2. The method according to claim 1, characterized in that, Acquire multi-source remote sensing image data, including: Acquire raw image data, which includes first image data acquired from a first satellite platform and second image data acquired from a second satellite platform; the resolution of the second image data is higher than that of the first image data. Based on the resolution scale of the first image data, the second image data is first upsampled and then downsampled to obtain the high spatial resolution image; Extract the target band image from the first image data and the high spatial resolution image respectively; Based on the scale-invariant feature transform algorithm, key points and descriptors of the key points are extracted from the target band image. According to the principle of consistency of random sampling, the key point pairs are screened, and an affine transformation matrix is ​​fitted based on the key points; According to the affine transformation matrix, all band images in the first image data are resampled to obtain the hyperspectral resolution image; the hyperspectral resolution image is aligned with the spatial grid of the panchromatic band image of the high spatial resolution image at the corresponding resolution.

3. The method according to claim 1, characterized in that, Extracting spectral attention pyramid features from the hyperspectral resolution image includes: Spectral index maps are calculated in parallel from the hyperspectral resolution imagery, and the spectral index maps include at least one of the following: floating debris index, normalized difference vegetation index, normalized difference water index, and cloud index. An initial spectral attention map is generated by normalizing the spectral index map. The original hyperspectral band image in the hyperspectral resolution image is stitched together with the initial spectral attention map in the channel dimension to form enhanced input features; The enhanced input features are input into the spectral attention pyramid coding network to obtain the multi-scale spectral attention feature map output by the spectral attention pyramid coding network; the spectral attention pyramid coding network contains a network structure with multiple downsampling stages; the network structure of each downsampling stage includes a convolutional layer, a batch normalization layer, an activation function, and a convolutional pooling layer.

4. The method according to claim 1, characterized in that, Extracting dual-stream features from the high spatial resolution image includes: Normalization processing is performed on the panchromatic band image of the high spatial resolution image; Using the normalized panchromatic band image as input, the detail features are extracted through a detail feature network; the detail feature network is a shallow convolutional neural network. The multispectral band data of the high spatial resolution image is upsampled to the target pixel size by bicubic interpolation and then normalized. The multispectral band data after the above sampling and normalization processing is used as input, and the context features are extracted through a context feature network; the context feature network is a convolutional neural network with a different kernel specification than the detail feature network.

5. The method according to claim 1, characterized in that, The spectral attention pyramid features are upsampled from top to bottom and laterally connected to obtain preliminary fused features, including: Perform double nearest neighbor upsampling on the top-level features in the spectral attention pyramid features to obtain the top-level upsampled features; A first convolution operation is applied to the second-to-top-level features in the spectral attention pyramid features to generate first-scale features; The top-level upsampled features and the first-scale features are added element by element and then fused through a second convolution operation to obtain the first feature to be fused. The first feature to be fused is upsampled by two nearest neighbors to obtain the second-scale feature. The first convolution operation is applied to the bottom-level features in the spectral attention pyramid features to generate the third-scale features; The second-scale features and the third-scale features are added together and then fused through a second convolution operation to obtain the second feature to be fused. The second feature to be fused is subjected to 4x nearest neighbor upsampling to obtain the preliminary fused feature; the spatial size of the preliminary fused feature is consistent with the detail feature.

6. The method according to claim 1, characterized in that, The method further includes: The target detection model is initialized with network parameters; wherein the convolutional layers of the spectral attention pyramid coding network, detail feature network, and context feature network in the target detection model are initialized with a normal distribution; the transposed convolutional layers in the backbone network are initialized with bilinear interpolation kernels; and the fully connected layers in the backbone network are initialized with a uniform distribution. The training data is input into the object detection model to obtain the training detection results output by the object detection model; The training loss is calculated based on the training detection results according to the total loss function, which is a weighted sum of the original detection loss, the spectral attention alignment loss, and the spectral guidance effectiveness loss. The network parameters of the object detection model are iteratively optimized based on the training loss.

7. The method according to claim 1, characterized in that, The spectral excitation features are input into the target detection model to output detection results, including: The spectral excitation features are input into the target detection model to obtain the prediction tensor output by the target detection model. The prediction tensor includes bounding box offset, object confidence, and class probability. The bounding box offset is decoded into an absolute coordinate bounding box in the image coordinate system; The absolute coordinate bounding boxes are filtered by applying preset object confidence thresholds and category probability thresholds to determine reliable prediction boxes; The non-maximum suppression algorithm is applied to remove overlapping redundant boxes in the reliable prediction boxes; The confidence score of the reliable prediction box is obtained by multiplying the object confidence score corresponding to the reliable prediction box after removing overlapping redundant boxes by the class probability.

8. A remote sensing target detection system based on attention pyramid and migration guidance, characterized in that, The system includes: The data preparation module is used to acquire multi-source remote sensing image data, which includes hyperspectral resolution images and high spatial resolution images. The feature extraction module is used to extract spectral attention pyramid features from the hyperspectral resolution image and dual-stream features from the high spatial resolution image; the spectral attention pyramid features include a multi-scale spectral attention feature map obtained through spectral attention pyramid encoding; the dual-stream features include detail features and context features; An attention transfer module is used to transform the spectral attention pyramid feature map into a spectral guidance signal through super-resolution spectral attention transfer. The spectral guidance signal includes a spectral guidance weight map aligned with the high-resolution feature space. The transformation of the spectral attention pyramid feature map into a spectral guidance signal includes: performing top-down upsampling and lateral concatenation on the spectral attention pyramid features to obtain preliminary fused features; employing a spatial attention mechanism to perform convolution calculation on the preliminary fused features and generate a spatial dimension attention weight map through an activation function; employing a channel attention mechanism to perform global average pooling on the preliminary fused features to obtain channel description vectors, and generating channel weight vectors based on the channel description vectors using a multilayer perceptron and an activation function; and performing feature fusion based on the spatial dimension attention weight map and the channel weight vectors to generate the spectral guidance weight map. A spectral modulation detection module is used to detect a backbone network using multi-level spectral modulation, generating spectral excitation features based on the spectral guidance weight map and the two-stream features. The backbone network includes a shallow spatial gated modulation module and a deep channel spectral excitation module. The shallow spatial gated modulation module is used to perform spectral modulation on the detail features at the pixel level according to the spectral guidance signal. The deep channel spectral excitation module is used to reweight each channel of the deep features according to the context features. Generating spectral excitation features based on the spectral guidance weight map and the two-stream features includes: multiplying the detail features element-wise with the spectral guidance weight map through the shallow spatial gated modulation module to obtain modulation features; concatenating the modulation features with the context features along the channel dimension to form fused input features; performing global average pooling on the spectral guidance weight map or the fused input features through the deep channel spectral excitation module to obtain a global spectral description vector; inputting the global spectral description vector into a multilayer perceptron to learn channel weight vectors; and multiplying the channel weight vector with the feature map of the current layer along the channel direction to obtain the spectral excitation features. The result output module is used to input the spectral excitation features into the target detection model so as to output the detection result through the target detection model. The detection result includes the bounding box coordinates, class label and confidence score of the target to be detected predicted by the target detection model.

Images