An end-to-end weakly aligned multispectral remote sensing image target detection method and device

Abstract

The present invention belongs to the field of computer vision and pattern recognition technology, and specifically relates to an end-to-end weakly aligned multispectral remote sensing image target detection method and device. The specific process of the method is as follows: training data construction: pre-set weakly aligned visible light and infrared remote sensing images are annotated to obtain a weakly aligned multispectral remote sensing image target detection dataset; target detection network construction: an end-to-end weakly aligned multispectral remote sensing image target detection network is constructed, including a feature encoding network, a query selection network, and a decoding network; target detection network training: the target detection network is trained using the target detection dataset; remote sensing image target detection: the trained target detection network is used to perform remote sensing image target detection, and the detection results include: target category, visible light target position, and infrared target position.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and pattern recognition, and in particular relates to an end-to-end weakly aligned multispectral remote sensing image target detection method and device. Background Art

[0002] Remote sensing image target detection aims to locate and identify targets from a given remote sensing image and has been widely used in fields such as intelligent surveillance, urban planning, port management, and military reconnaissance. With the rapid development of artificial intelligence and multi-sensor technology, multispectral remote sensing image target detection has garnered widespread attention by leveraging the complementarity of visible and infrared imagery to achieve more robust and accurate target recognition. However, in practical applications, due to factors such as imaging conditions and equipment accuracy, multispectral data often suffers from positional offsets. This means that visible and infrared image pairs are not strictly aligned, resulting in an object having different positions in different modalities. This weak alignment of multispectral remote sensing image target detection poses challenges to deep learning methods, as feature fusion and neural network training may be affected.

[0003] Existing weakly aligned remote sensing image target detection technologies mainly adopt the strategy of alignment first and then fusion. The paper WeaklyAligned Cross-Modal Learning for Multispectral Pedestrian Detection designs a regional feature alignment (RFA) module to capture position offsets and adaptively align regional features of the two modalities, and then proposes a new multimodal fusion method to select more reliable features and suppress useless features through feature reweighting. In addition, this paper proposes a novel RoI jittering strategy to improve the robustness to unexpected offset patterns under different devices and system settings. The paper Cross-Modality Proposal-guided Feature Mining for UnregisteredRGB-Thermal Pedestrian Detection proposes a cross-modal proposal-guided feature mining (CPFM) mechanism to predict target proposal candidate regions in the two modalities respectively, and studies two non-maximum suppression (NMS) methods for post-processing to simultaneously improve the detection accuracy of visible light and infrared images.

[0004] Although existing methods can improve detection accuracy, they still have the following shortcomings:

[0005] 1. Extracting local features of visible light and infrared images through a region candidate network to achieve feature alignment. This model is highly complex and cannot meet real-time detection requirements.

[0006] 2. The feature fusion method of first aligning and then fusing is used, which fails to fully utilize cross-modal complementarity and results in poor model detection capabilities.

[0007] 3. More complex post-processing operations are required to accurately locate targets in both visible light and infrared images, and the model has low timeliness.

[0008] Therefore, to achieve high-precision and high-efficiency weakly aligned remote sensing image target detection, it is necessary to propose a parallel alignment-fusion method that does not require display extraction of local features of visible light and infrared images, so as to make more full use of multimodal complementarity. In addition, this method also needs to meet the end-to-end paradigm without complex post-processing operations, so as to simultaneously detect targets in visible light and infrared images while ensuring model speed. This is also the technical problem to be solved by the present invention. Summary of the Invention

[0009] In view of this, the present invention provides an end-to-end weakly aligned multispectral remote sensing image target detection method and device, which can fully explore and utilize the complementarity of multimodal targets, and accurately locate targets in visible light and infrared images without the need for complex post-processing operations.

[0010] The technical solutions for implementing the present invention are as follows:

[0011] In the first aspect, the embodiment of the present application provides an end-to-end weakly aligned multispectral remote sensing image target detection method, the specific process of which is as follows:

[0012] Construction of training data: The preset weakly aligned visible light and infrared remote sensing images are annotated to obtain a weakly aligned multispectral remote sensing image target detection dataset;

[0013] Target detection network construction: Construct an end-to-end weakly aligned multispectral remote sensing image target detection network, including feature encoding network, query selection network and decoding network, where

[0014] The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features;

[0015] a query selection network, configured to process the visible light coded features and the infrared coded features, and output an initial shared content query, an initial visible light reference point, and an initial infrared reference point;

[0016] The decoding network is used to interact with the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and infrared encoding features to achieve multimodal feature alignment and fusion, and ultimately output the target classification result and visible light infrared target position;

[0017] Target detection network training: Use the target detection dataset to train the target detection network;

[0018] Remote sensing image target detection: Use the trained target detection network to perform remote sensing image target detection. The detection results include: target category, visible light target position, and infrared target position.

[0019] Furthermore, the query selection network of the present invention includes: a classification network, a visible light regression network and an infrared regression network;

[0020] The classification network is used to obtain the classification score of each pixel feature of the input fusion feature, and then select the K pixel features with the highest scores as the initial shared content query through the Top-K operation;

[0021] The visible light regression network is used to select the regression output corresponding to K pixel features from the visible light coding features and add them to the initial target frame (the initial target frame is preset and the same for each image) to form an initial visible light reference point;

[0022] The infrared regression network is used to select regression outputs corresponding to K pixel features from the infrared coding features and add them to the initial target frame to form an initial infrared reference point.

[0023] Furthermore, the specific processing process of the decoding network of the present invention is:

[0024] (1) The initial visible light reference point and the initial infrared reference point are passed through a shared position coding network to obtain a location query, and the location query is added to the initial shared content query to form a shared query;

[0025] (2) After the shared query is enhanced through the multi-head self-attention network to distinguish between different queries, the shared query and multi-modal multi-scale features are input into the dual multi-modal multi-scale multi-head sparse attention module to obtain the refined shared content query;

[0026] (3) The refined shared content query is passed through the classification network to output the target classification result, and the refined shared content query is passed through the visible light infrared dual regression network. The output is added to the initial visible light reference point and the initial infrared reference point to obtain the visible light target position and the infrared target position, where the visible light target position and the infrared target position are regarded as refined reference points and input into the next decoder to achieve the purpose of gradually refining the target frame.

[0027] Furthermore, the algorithm of the dual multimodal multi-scale multi-head sparse attention module described in the present invention is:

[0028] Then the process of the dual multimodal multi-scale multi-head sparse attention module can be expressed as follows:

[0029]

[0030] Δp qhlk =f offse t(F q )

[0031] Among them, H represents the number of attention heads, L represents the number of multi-scale layers in each modality, K represents the number of feature sampling points in each layer, and W h and W′ h is the linear mapping network weight of the attention mechanism, is a hierarchical mapping function, and Through the sampling point weight network f weight Generate, represents the kth attention weight in the hth attention head on the lth layer feature map, Δp qhlk Offset the network by sampling point f offset Generate, represents the k-th sampling point offset in the h-th attention head on the l-th layer feature map, F q To share query features, F′ q To refine shared content queries, is a multi-modal and multi-scale feature, and represent infrared characteristics and visible light characteristics respectively, is the visible light reference point, Infrared reference point.

[0032] Furthermore, the present invention constructs dual regression loss and dual IoU-aware classification loss to guide the training of the target detection network;

[0033] Let the true value and network output be and z={c,b vis ,b ir}, the total loss L of the network can be expressed as follows:

[0034]

[0035] Among them, L box is the L1 loss function, L GIoU is the GIoU loss function, L1 is the mean absolute error loss function, IoU vis and IoU ir is the IoU between the predicted value and the true box, L cls is the dual IoU-aware classification loss;

[0036]

[0037] Where d = (IoU vis IoU ir ), a, y, e are all hyper parameters.

[0038] Furthermore, during training, the present invention introduces category noise by randomly flipping the true value labels to other labels. The intensity of the noise is controlled by a hyperparameter, and the noise of negative queries is greater than that of positive queries.

[0039] During training, the content noise query and the position noise query are input into the decoding network simultaneously with the initial content query and position query output by the query selection network. The content noise query and the position noise query are trained through denoising in the decoding network.

[0040] Furthermore, the present invention introduces an attention mask to ensure that the denoised query is invisible to the original query.

[0041] In a second aspect, an embodiment of the present application provides an end-to-end weakly aligned multispectral remote sensing image target detection device, which is a target detection network including a feature encoding network, a query selection network, and a decoding network, wherein:

[0042] The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features;

[0043] a query selection network, configured to process the visible light coded features and the infrared coded features, and output an initial shared content query, an initial visible light reference point, and an initial infrared reference point;

[0044] The decoding network is used to interact the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and infrared encoding features to achieve multimodal feature alignment and fusion, and finally output the target classification result and the visible light infrared target position.

[0045] Beneficial effects:

[0046] First, the present invention designs a dual multimodal multi-scale multi-head sparse attention module. Based on the query feature approach, through sparse sampling of multimodal features guided by dual reference positions, it realizes the direct alignment and fusion of multimodal multi-scale target features under weak alignment conditions, fully explores and utilizes the complementarity of multimodal targets, and accurately locates targets in visible light and infrared images without the need for complex post-processing operations.

[0047] Secondly, the present invention designs a dual regression network and dual IoU-aware classification loss, adopts an end-to-end target detection method based on query features, and simultaneously outputs visible light and infrared target positions, making full use of the advantage of no post-processing operation. Furthermore, the dual-path target IoU score is used as a soft label for the classification loss, ensuring the common accuracy of object classification, visible light target position and infrared target position of the same target in weakly aligned bimodal images, which is conducive to the network to fully explore the alignment features and their complementarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is a flow chart of the method of the present invention;

[0050] Figure 2 Schematic diagram of an end-to-end weakly aligned multispectral remote sensing image target detection basic network according to one embodiment of the present invention;

[0051] Figure 3 Schematic diagram of a dual multimodal multi-scale multi-head sparse attention module according to one embodiment of the present invention;

[0052] Figure 4 Schematic diagram of a dual contrast denoising training strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0054] It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments may be combined with each other; and, based on the embodiments in this disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of this disclosure.

[0055] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0056] The embodiment of the present application provides an end-to-end weakly aligned multispectral remote sensing image target detection method, the specific process is as follows:

[0057] Construction of training data: The preset weakly aligned visible light and infrared remote sensing images are annotated to obtain a weakly aligned multispectral remote sensing image target detection dataset;

[0058] Target detection network construction: Construct an end-to-end weakly aligned multispectral remote sensing image target detection network, including feature encoding network, query selection network and decoding network, where

[0059] The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features;

[0060] a query selection network, configured to process the visible light coded features and the infrared coded features, and output an initial shared content query, an initial visible light reference point, and an initial infrared reference point;

[0061] The decoding network is used to interact with the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and infrared encoding features to achieve multimodal feature alignment and fusion, and ultimately output the target classification result and visible light infrared target position;

[0062] Target detection network training: Use the target detection dataset to train the target detection network;

[0063] Remote sensing image target detection: Use the trained target detection network to perform remote sensing image target detection. The detection results include: target category, visible light target position, and infrared target position.

[0064] like Figures 1-4 As shown, this embodiment provides an end-to-end weakly aligned multispectral remote sensing image target detection method, including the following steps, where steps S1-S6 are only one of the multiple execution orders of this application:

[0065] Step S1: labeling the preset weakly aligned visible light and infrared remote sensing images to obtain a weakly aligned multispectral remote sensing image target detection dataset;

[0066] Step S2: construct an end-to-end weakly aligned multispectral remote sensing image target detection basic network;

[0067] Step S3: construct a dual multimodal multi-scale multi-head sparse attention module based on S2;

[0068] Step S4: construct dual regression loss and dual IoU-aware classification loss to guide model training;

[0069] Step S5: construct a dual contrast denoising training strategy to accelerate model training;

[0070] Step S6: Perform model training, verification, and testing based on the data set in S1.

[0071] The following is a detailed description of each step:

[0072] Step S1: Annotate preset weakly aligned visible light and infrared remote sensing images to obtain a weakly aligned multispectral remote sensing image target detection dataset.

[0073] A target classification system and sample annotation specifications for weakly aligned multispectral (including visible light and infrared) remote sensing images were constructed. Based on this classification system and sample annotation specifications, the target positions and categories in the weakly aligned visible light and infrared remote sensing images were annotated with horizontal boxes. In addition, the targets were annotated with ID numbers to ensure that the same target appeared in pairs in images of different modalities, but with offset positions. The annotated images were sliced ​​with a slice size of 640*640 and an overlap coefficient of 20% to form a weakly aligned multispectral remote sensing image target detection dataset. This dataset was then divided into training, validation, and test sets in a ratio of 7:1:2.

[0074] Step S2: Construct an end-to-end weakly aligned multispectral remote sensing image target detection basic network.

[0075] like Figure 1 As shown in the figure, an end-to-end weakly aligned multispectral remote sensing image target detection basic network is constructed to achieve feature encoding, query selection and decoding. The specific process is as follows:

[0076] 1) Feature encoding: First, a pair of weakly aligned infrared and visible light images are used as input. The features of each image are first encoded by a modality-specific backbone network (e.g., ResNet50) and a feature pyramid network (e.g., FPN), outputting visible light encoded features and infrared encoded features.

[0077] 2) Query selection: The visible light coding features and infrared coding features are added and fused to obtain the fused features, which are input into the query selection network to output the initial shared content query, initial visible light reference point, and infrared reference point.

[0078] The query selection network consists of a classification network, a visible light regression network, and an infrared regression network. The classification network outputs a classification score for each pixel feature. A Top-K operation then selects the K pixel features with the highest scores as the initial shared content query. Similarly, the visible light regression network and the infrared regression network each output regression outputs corresponding to the K pixel features. These are added to the initial target bounding box to form the initial visible light reference point and initial infrared reference point.

[0079] 3) Decoding: The decoding process interacts the initial shared content query with the multimodal and multi-scale features, aligns and fuses the multimodal features, and ultimately outputs the target classification result and the visible light infrared target location. Specifically:

[0080] (1) The initial visible light reference point and the initial infrared reference point are passed through a shared position coding network to obtain a position query, and the position query is added to the initial shared content query to form a shared query.

[0081] (2) Shared queries enhance the distinction between different queries through a multi-head self-attention network, preventing different shared queries from matching the same feature point at the same time; then the shared query and multimodal multi-scale features are input into the dual multimodal multi-scale multi-head sparse attention module to obtain a refined shared content query, while achieving alignment and fusion of multimodal features;

[0082] (3) The refined shared content query is passed through the classification network to output the target classification result, and the refined shared content query is passed through the visible light infrared dual regression network to output the visible light target frame position offset, which is added to the initial visible light reference point to obtain the visible light target position, and the infrared target frame position offset is output, which is added to the initial infrared reference point to obtain the infrared target position. The visible light target position and the infrared target position are regarded as refined reference points and input into the next decoder to achieve the purpose of gradually refining the target frame.

[0083] Step S3: construct a dual multimodal multi-scale multi-head sparse attention module based on S2;

[0084] like Figure 2 As shown in the figure, a dual multimodal, multi-scale, multi-head sparse attention module is constructed to achieve alignment and fusion of multimodal features. In fact, the refined shared content query represents the fused features, but it is still necessary to dynamically interact the query features with the multimodal, multi-scale feature fusion to enable it to find the target area in the multimodal, multi-scale features and accurately express the target characteristics.

[0085] Specifically, given the input multi-semantic infrared feature map and visible light feature map Use the qth normalized center point of the visible light and infrared reference points as the visible light reference point and infrared reference points Assumption F q For query features ( Figure 2 Q) after the multi-head self-attention module, F′ q is the updated query feature ( Figure 2 The process of the dual multimodal multi-scale multi-head sparse attention module can be expressed as follows:

[0086]

[0087] Δp qhlk =f offset (F q )

[0088] Where H represents the number of attention heads, L represents the number of multi-scale layers in each modality, and K represents the number of feature sampling points in each layer. h and W′ h is the linear mapping network weight of the attention mechanism, It is a hierarchical mapping function used to map the normalized reference point to the l-th level feature map. and pass Figure 2 The sampling point weight network f in weight Generate, represents the kth attention weight in the hth attention head on the lth layer feature map, Δp qhlk pass Figure 2 The sampling point offset network f in offset Generate, which represents the k-th sampling point offset in the h-th attention head on the l-th layer feature map. Note that the visible light and infrared modalities share a Δp qhlk ,This approach helps the network learn the offset difference of features and achieve ,feature alignment.

[0089] Step S4: Construct dual regression loss and dual IoU-aware classification loss to guide model training.

[0090] The end-to-end weakly aligned multispectral remote sensing image target detection basic network outputs the target classification score, visible light target offset and infrared target offset in the query selection and decoding stages, so a dual regression loss and dual IoU-aware classification loss are constructed to guide the training of the model. Assume that the true value and network output are and z={c,b vis ,b ir}, the total loss L of the network can be expressed as follows:

[0091]

[0092] Among them, L box is the L1 loss function, L GIoU is the GIoU loss function, L1 is the commonly used mean absolute error loss function, IoU vis and IoU ir is the IoU between the predicted value and the true box, L cls The dual IoU-aware classification loss can be expressed as follows:

[0093]

[0094] Where d = (IoU vis IoU ir ), a, y, e are all hyperparameters. L clsIt can simultaneously reflect object classification, visible light modality position, and infrared modality position, rather than just representing the possibility of foreground.

[0095] Step S5: Construct a dual contrast denoising training strategy to accelerate model training.

[0096] End-to-end object detection networks typically use the Hungarian matching algorithm to achieve a one-to-one match between query features and targets, eliminating the NMS post-processing algorithm. However, this also leads to problems such as unstable network training and slow convergence. Therefore, a dual contrast denoising training strategy is constructed to accelerate the training of end-to-end weakly aligned multispectral remote sensing image object detection networks. Specifically:

[0097] Category noise is introduced by randomly flipping the ground truth labels to other labels. For paired visible and infrared bounding box noise, such as Figure 3 As shown in Figure 2, random displacement, scale, and angle noise are applied to the visible and infrared bounding boxes of an object to generate pairs of denoised queries for both the positive and negative modalities. The intensity of the noise is controlled by a hyperparameter, with negative queries being noisier than positive queries. Applying random noise to the bounding boxes of both modalities also increases the diversity of object misalignment.

[0098] During training, content-noise queries and position-noise queries are fed into the decoder network simultaneously with the initial content and position queries output by the query selection network. These queries undergo denoising training in the decoder, accelerating network convergence and stability. Furthermore, to prevent information leakage during self-attention, we introduce an attention mask to ensure that the denoised query is invisible to the initial query.

[0099] Step S6: Perform model training, verification, and testing based on the data set in S1.

[0100] The training set in step S1 is used to train the end-to-end weakly aligned multispectral remote sensing image target detection network. The backbone network is initialized using COCO dataset pre-training. The stochastic gradient descent method is used for 72 rounds of iterative training with a batch size of 16 and an initial learning rate of 0.0001. The parameters of the end-to-end weakly aligned multispectral remote sensing image target detection network are updated. The detection performance of the end-to-end weakly aligned multispectral remote sensing image target detection network is observed using the validation set in step S1 until the network performance converges, thereby obtaining an end-to-end weakly aligned multispectral remote sensing image target detection model. The test set in S1 is input into the model to obtain detection results.

[0101] The present invention uses query features to align visible light and infrared modalities. When query features are used to find sparse sampling points related to modality-specific targets, the target features are automatically aligned without the need for explicit target candidate areas.

[0102] While querying features to find sparse sampling points related to modality-specific targets, the present invention also outputs sampling point weights, thereby simultaneously realizing the fusion of alignment features and making full use of multimodal complementarity.

[0103] The present invention adopts an end-to-end target detection method based on query features, and simultaneously outputs visible light and infrared target positions. With the help of a one-to-one matching method, it completely abandons the two-way NMS operation of traditional multimodal target detection.

[0104] In a second aspect, an embodiment of the present application provides an end-to-end weakly aligned multispectral remote sensing image target detection device, which is a target detection network including a feature encoding network, a query selection network, and a decoding network, wherein:

[0105] The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features;

[0106] a query selection network, configured to process the visible light coded features and the infrared coded features, and output an initial shared content query, an initial visible light reference point, and an initial infrared reference point;

[0107] The decoding network is used to interact the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and infrared encoding features to achieve multimodal feature alignment and fusion, and finally output the target classification result and the visible light infrared target position.

[0108] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An end-to-end weakly aligned multispectral remote sensing image target detection method, characterized in that: The specific process is: Construction of training data: The preset weakly aligned visible light and infrared remote sensing images are annotated to obtain a weakly aligned multispectral remote sensing image target detection dataset; Target detection network construction: Build an end-to-end weakly aligned multispectral remote sensing image target detection network, It includes feature encoding network, query selection network and decoding network, among which, The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features; a query selection network, configured to process the visible light coded features and the infrared coded features, and output an initial shared content query, an initial visible light reference point, and an initial infrared reference point; The decoding network is used to interact with the initial shared content query, initial visible light reference point, initial infrared reference point, visible light coded features, and infrared coded features to achieve multimodal feature alignment and fusion, and ultimately output the target classification result and visible light infrared target position; Target detection network training: Use the target detection dataset to train the target detection network; Remote sensing image target detection: Detect remote sensing image targets using the trained target detection network. Detection results include target category, visible light target location, and infrared target location. The specific processing process of the decoding network is: (1) The initial visible light reference point and the initial infrared reference point are passed through the shared position encoding network to obtain a location query, and the location query is added to the initial shared content query to form a shared query; (2) After the shared query is enhanced to distinguish between different queries through the multi-head self-attention network, the shared query and multi-modal multi-scale features are input into the dual multi-modal multi-scale multi-head sparse attention module to obtain a refined shared content query; (3) The refined shared content query is passed through the classification network to output the target classification result, and the refined shared content query is passed through the visible light infrared dual regression network. The output is added to the initial visible light reference point and the initial infrared reference point to obtain the visible light target position and the infrared target position. The visible light target position and the infrared target position are regarded as refined reference points and input into the next decoder to achieve the purpose of gradually refining the target frame; The algorithm of the dual multimodal multi-scale multi-head sparse attention module: Then the process of the dual multimodal multi-scale multi-head sparse attention module is expressed as follows: in, represents the number of attention heads, represents the number of multi-scale levels in each modality, Represents the number of feature sampling points in each layer, and is the linear mapping network weight of the attention mechanism, is a hierarchical mapping function, and Through the sampling point weight network Generate, which represents the kth attention weight in the hth attention head on the lth layer feature map, Offset network by sampling points Generate, which represents the k-th sampling point offset in the h-th attention head on the l-th layer feature map, To share query features, To refine shared content queries, is a multi-modal and multi-scale feature, represent infrared characteristics and visible light characteristics respectively, is the visible light reference point, Infrared reference point.

2. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 1, characterized in that The query selection network includes: a classification network, a visible light regression network and an infrared regression network; The classification network is used to obtain the classification score of each pixel feature of the input fusion feature, and then select the K pixel features with the highest scores as the initial shared content query through the Top-K operation; The visible light regression network is used to select regression outputs corresponding to K pixel features from the visible light coding features, and add them to the initial target frame to form an initial visible light reference point; The infrared regression network is used to select regression outputs corresponding to K pixel features from the infrared coding features and add them to the initial target frame to form an initial infrared reference point.

3. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 2, characterized in that: Construct dual regression loss and dual IoU-aware classification loss to guide the training of target detection network; Let the true value and network output be and If , the total loss L of the network is expressed as follows: in, is the L1 loss function, is the GIoU loss function, is the mean absolute error loss function, and is the IoU between the predicted value and the true box, is the dual IoU-aware classification loss.

4. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 3, characterized in that: described in , These are all hyperparameters.

5. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 1, characterized in that During training, category noise is introduced by randomly flipping the true value labels to other labels. The intensity of the noise is controlled by a hyperparameter, and the noise of negative queries is greater than that of positive queries.

6. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 5, characterized in that: During training, the content noise query and the position noise query are input into the decoding network simultaneously with the initial content query and position query output by the query selection network. The content noise query and the position noise query are trained through denoising in the decoding network.

7. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 1, characterized in that: An attention mask is introduced to ensure that the denoised query is not seen by the initial query.

8. An end-to-end weakly aligned multispectral remote sensing image target detection device, used to execute the method according to any one of claims 1 to 7, wherein the device is a target detection network. Including feature encoding network, query selection network and decoding network, in, The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features; a query selection network, configured to process the visible light coded features and the infrared coded features, and output an initial shared content query, an initial visible light reference point, and an initial infrared reference point; The decoding network is used to interact the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and infrared encoding features to achieve multimodal feature alignment and fusion, and finally output the target classification result and the visible light infrared target position.

Images