End-to-end weak alignment multispectral remote sensing image target detection method and device

Abstract

The invention belongs to the technical field of computer vision and mode recognition, and particularly relates to an end-to-end weak alignment multispectral remote sensing image target detection method and device. The method specifically comprises the following steps of: constructing training data: labeling preset weak alignment visible light and infrared remote sensing images to obtain a weak alignment multispectral remote sensing image target detection data set; target detection network construction: constructing an end-to-end weak alignment multispectral remote sensing image target detection network, wherein the end-to-end weak alignment multispectral remote sensing image target detection network comprises a feature coding network, a query selection network and a decoding network; training a target detection network: training the target detection network by using the target detection data set; and remote sensing image target detection: performing remote sensing image target detection by using the trained target detection network, wherein a detection result comprises a target category, a visible light target position and an infrared target position.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and pattern recognition, and in particular relates to an end-to-end weakly aligned multispectral remote sensing image target detection method and device. Background Art

[0002] Remote sensing image target detection aims to locate and identify targets from a given remote sensing image, and has been widely used in intelligent monitoring, urban planning, port management, and military reconnaissance. With the rapid development of artificial intelligence technology and multi-sensor technology, multispectral remote sensing image target detection has gained widespread attention by making full use of the complementarity of visible light and infrared images to achieve more robust and accurate target recognition. However, in practical applications, due to the influence of imaging conditions, equipment accuracy and other factors, multispectral data often have position offset problems, that is, the visible light and infrared image pairs are not strictly aligned, resulting in an object having different positions in different modalities. This weakly aligned multispectral remote sensing image target detection poses a challenge to deep learning methods, as feature fusion and neural network training may be affected.

[0003] Existing weakly aligned remote sensing image target detection technologies mainly adopt the strategy of alignment first and then fusion. The paper WeaklyAligned Cross-Modal Learning for Multispectral Pedestrian Detection designs a regional feature alignment (RFA) module to capture position offsets and adaptively align regional features of the two modalities, and then proposes a new multimodal fusion method to select more reliable features and suppress useless features through feature reweighting. In addition, this paper proposes a novel RoI jittering strategy to improve the robustness to unexpected offset patterns under different devices and system settings. The paper Cross-Modality Proposal-guided Feature Mining for UnregisteredRGB-Thermal Pedestrian Detection proposes a cross-modal proposal-guided feature mining (CPFM) mechanism to predict target proposal candidate regions in the two modalities respectively, and studies two non-maximum suppression (NMS) methods for post-processing to simultaneously improve the detection accuracy of visible light and infrared images.

[0004] Although existing methods can improve the accuracy of detection, they still face the following disadvantages:

[0005] 1. Extract local features of visible light and infrared images through the region candidate network to achieve the purpose of aligning features. The model is highly complex and difficult to meet the needs of real-time detection;

[0006] 2. The feature fusion method of aligning first and then fusing is adopted, and the cross-modal complementarity cannot be fully utilized, and the model detection ability is poor;

[0007] 3. More complex post-processing operations are required to accurately locate targets in both visible light and infrared images, and the model has low timeliness.

[0008] Therefore, in order to achieve high-precision and high-efficiency weakly aligned remote sensing image target detection, it is necessary to propose an alignment-fusion parallel method that does not require explicit extraction of local features of visible light and infrared images, so as to make more full use of multimodal complementarity. In addition, this method also needs to meet the end-to-end paradigm that does not require complex post-processing operations, so as to simultaneously detect targets in visible light and infrared images while ensuring the model speed. This is also the technical problem to be solved by the present invention. Summary of the invention

[0009] In view of this, the present invention provides an end-to-end weakly aligned multispectral remote sensing image target detection method and device, which can fully explore and utilize the complementarity of multimodal targets, and accurately locate targets in visible light and infrared images without the need for complex post-processing operations.

[0010] The technical solution for implementing the present invention is as follows:

[0011] In the first aspect, the embodiment of the present application provides an end-to-end weakly aligned multispectral remote sensing image target detection method, the specific process is:

[0012] Construction of training data: Annotate the preset weakly aligned visible light and infrared remote sensing images to obtain a weakly aligned multispectral remote sensing image target detection dataset;

[0013] Target detection network construction: Construct an end-to-end weakly aligned multispectral remote sensing image target detection network, including feature encoding network, query selection network and decoding network, where:

[0014] The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features;

[0015] A query selection network, used for processing the visible light coded features and the infrared coded features, and outputting an initial shared content query, an initial visible light reference point, and an initial infrared reference point;

[0016] The decoding network is used to interact the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and the infrared encoding features to achieve the alignment and fusion of multimodal features, and finally output the target classification result and the visible light infrared target position;

[0017] Target detection network training: Use the target detection dataset to train the target detection network;

[0018] Remote sensing image target detection: Use the trained target detection network to perform remote sensing image target detection. The detection results include: target category, visible light target position, and infrared target position.

[0019] Furthermore, the query selection network of the present invention includes: a classification network, a visible light regression network and an infrared regression network;

[0020] The classification network is used to obtain the classification score of each pixel feature of the input fusion feature, and then select the K pixel features with the highest scores as the initial shared content query through the Top-K operation;

[0021] The visible light regression network is used to select regression outputs corresponding to K pixel features from the visible light coding features, and add them to the initial target frame (the initial target frame is preset and the same for each image) to form an initial visible light reference point;

[0022] The infrared regression network is used to select regression outputs corresponding to K pixel features from the infrared coding features, and add them to the initial target frame to form an initial infrared reference point.

[0023] Furthermore, the specific processing process of the decoding network of the present invention is:

[0024] (1) The initial visible light reference point and the initial infrared reference point are used to obtain a location query through a shared location coding network, and the location query is added to the initial shared content query to form a shared query;

[0025] (2) After the shared query is enhanced through the multi-head self-attention network to distinguish between different queries, the shared query and multi-modal multi-scale features are input into the dual multi-modal multi-scale multi-head sparse attention module to obtain a refined shared content query;

[0026] (3) The refined shared content query is passed through the classification network to output the target classification result, and the refined shared content query is passed through the visible light infrared dual regression network. The output is added to the initial visible light reference point and the initial infrared reference point to obtain the visible light target position and the infrared target position, where the visible light target position and the infrared target position are regarded as refined reference points and input into the next decoder to achieve the purpose of gradually refining the target frame.

[0027] Furthermore, the algorithm of the dual multi-modal multi-scale multi-head sparse attention module described in the present invention is:

[0028] Then the process of the dual multi-modal multi-scale multi-head sparse attention module can be expressed as follows:

[0029]

[0030] Δp qhlk =f offse t(F q )

[0031] Among them, H represents the number of attention heads, L represents the number of multi-scale levels in each modality, K represents the number of feature sampling points in each layer, and W h and W′ h is the linear mapping network weight of the attention mechanism, is the hierarchical mapping function, and Through the sampling point weight network f weight Generate, represents the kth attention weight in the hth attention head on the lth layer feature map, Δp qhlk By offsetting the network f offset Generate, represents the kth sampling point offset in the hth attention head on the lth layer feature map, F q is the shared query feature, F′ q To refine shared content queries, is a multi-modal and multi-scale feature. and Represent infrared characteristics and visible light characteristics respectively, is the visible light reference point, Infrared reference point.

[0032] Furthermore, the present invention constructs dual regression loss and dual IoU-aware classification loss to guide the training of the target detection network;

[0033] Let the true value and network output be and z={c,b vis ,b ir}, the total loss L of the network can be expressed as follows:

[0034]

[0035] Among them, L box is the L1 loss function, L GIoU is the GIoU loss function, L1 is the mean absolute error loss function, IoU vis and IoU ir is the IoU between the predicted value and the true box, L cls is the dual IoU-aware classification loss;

[0036]

[0037] Where d = (IoU vis IoU ir ), a, y, and e are all hyperparameters.

[0038] Furthermore, during training, the present invention introduces category noise by randomly flipping the true value label to other labels. The intensity of the noise is controlled by a hyperparameter, and the noise of negative queries is greater than that of positive queries.

[0039] During training, the content noise query and the position noise query are input into the decoding network simultaneously with the initial content query and the position query output by the query selection network, and the content noise query and the position noise query are trained by denoising in the decoding network.

[0040] Furthermore, the present invention introduces an attention mask to ensure that the denoised query is invisible to the initial query.

[0041] In a second aspect, an embodiment of the present application provides an end-to-end weakly aligned multispectral remote sensing image target detection device, which is a target detection network, including a feature encoding network, a query selection network and a decoding network, wherein:

[0042] The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features;

[0043] A query selection network, used for processing the visible light coded features and the infrared coded features, and outputting an initial shared content query, an initial visible light reference point, and an initial infrared reference point;

[0044] The decoding network is used to interact the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and the infrared encoding features to achieve the alignment and fusion of multimodal features, and finally output the target classification results and the visible light infrared target position.

[0045] Beneficial effects:

[0046] First, the present invention designs a dual multimodal multi-scale multi-head sparse attention module. Based on the query feature approach, through the sparse sampling of multimodal features guided by dual reference positions, it realizes the direct alignment and fusion of multimodal multi-scale target features under weak alignment, fully explores and utilizes the complementarity of multimodal targets, and accurately locates targets in visible light and infrared images without the need for complex post-processing operations.

[0047] Second, the present invention designs a dual regression network and a dual IoU-aware classification loss, adopts an end-to-end target detection method based on query features, and simultaneously outputs visible light and infrared target positions, making full use of the advantage of no post-processing operations, and then borrows the dual-path target IoU score as a soft label for the classification loss at the same time, ensuring the common accuracy of object classification, visible light target position and infrared target position of the same target in weakly aligned bimodal images, which is conducive to the network to fully explore the alignment features and their complementarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a flow chart of the method of the present invention;

[0050] Figure 2 A schematic diagram of an end-to-end weakly aligned multispectral remote sensing image target detection basic network according to one embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a dual multi-modal multi-scale multi-head sparse attention module according to an embodiment of the present invention;

[0052] Figure 4 Schematic diagram of a dual contrast denoising training strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0054] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0055] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0056] The embodiment of the present application provides an end-to-end weakly aligned multispectral remote sensing image target detection method, the specific process is as follows:

[0057] Construction of training data: Annotate the preset weakly aligned visible light and infrared remote sensing images to obtain a weakly aligned multispectral remote sensing image target detection dataset;

[0058] Target detection network construction: Construct an end-to-end weakly aligned multispectral remote sensing image target detection network, including feature encoding network, query selection network and decoding network, where:

[0059] The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features;

[0060] A query selection network, used for processing the visible light coded features and the infrared coded features, and outputting an initial shared content query, an initial visible light reference point, and an initial infrared reference point;

[0061] The decoding network is used to interact the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and the infrared encoding features to achieve the alignment and fusion of multimodal features, and finally output the target classification result and the visible light infrared target position;

[0062] Target detection network training: Use the target detection dataset to train the target detection network;

[0063] Remote sensing image target detection: Use the trained target detection network to perform remote sensing image target detection. The detection results include: target category, visible light target position, and infrared target position.

[0064] like Figure 1-Figure 4 As shown, the present embodiment provides an end-to-end weakly aligned multispectral remote sensing image target detection method, comprising the following steps, wherein the following steps S1-S6 are only one of the multiple execution orders of the present application:

[0065] Step S1, annotating preset weakly aligned visible light and infrared remote sensing images to obtain a weakly aligned multispectral remote sensing image target detection data set;

[0066] Step S2, constructing an end-to-end weakly aligned multispectral remote sensing image target detection basic network;

[0067] Step S3: construct a dual multi-modal multi-scale multi-head sparse attention module based on S2;

[0068] Step S4: construct dual regression loss and dual IoU-aware classification loss to guide model training;

[0069] Step S5, constructing a dual contrast denoising training strategy to accelerate model training;

[0070] Step S6: Perform model training, verification and testing based on the data set in S1.

[0071] Each step is described in detail below:

[0072] Step S1: annotate preset weakly aligned visible light and infrared remote sensing images to obtain a weakly aligned multispectral remote sensing image target detection dataset.

[0073] Construct a target classification system and sample annotation specification for weakly aligned multispectral (including visible light and infrared) remote sensing images. Based on the classification system and sample annotation specification, the target positions and categories in weakly aligned visible light and infrared remote sensing images are annotated in horizontal box format. In addition, the targets are annotated with ID numbers to ensure that the same target appears in pairs in different modal images, but the target position is offset. The annotated images are sliced, the slice size is set to 640*640, and the overlap coefficient is set to 20%, forming a weakly aligned multispectral remote sensing image target detection dataset, which is divided into training set, validation set and test set in a ratio of 7:1:2.

[0074] Step S2: construct an end-to-end weakly aligned multispectral remote sensing image target detection basic network.

[0075] like Figure 1 As shown in the figure, an end-to-end weakly aligned multispectral remote sensing image target detection basic network is constructed to realize feature encoding, query selection and decoding. The specific process is as follows:

[0076] 1) Feature encoding: First, a pair of weakly aligned infrared and visible light images are used as input. The features of each image are first encoded by a modality-specific backbone network (e.g., ResNet50) and a feature pyramid network (e.g., FPN), outputting visible light encoded features and infrared encoded features.

[0077] 2) Query selection: The visible light coding features and infrared coding features are added and fused to obtain the fused features, which are input into the query selection network to output the initial shared content query, initial visible light reference point and infrared reference point.

[0078] The query selection network consists of a classification network, a visible light regression network, and an infrared regression network. The classification network outputs the classification score of each pixel feature, and then selects the K pixel features with the highest scores through the Top-K operation and uses them as the initial shared content query. Similarly, the visible light regression network and the infrared regression network output the regression outputs corresponding to the K pixel features respectively, and add them to the initial target box to form the initial visible light reference point and the initial infrared reference point.

[0079] 3) Decoding: The decoding process is to interact the initial shared content query with the multi-modal and multi-scale features, and realize the alignment and fusion of multi-modal features, and finally output the target classification results and visible light infrared target position. Specifically:

[0080] (1) The initial visible light reference point and the initial infrared reference point are passed through a shared position coding network to obtain a position query, and the position query is added to the initial shared content query to form a shared query.

[0081] (2) Shared queries enhance the distinction between different queries through a multi-head self-attention network to prevent different shared queries from matching the same feature point at the same time; then the shared query and multi-modal multi-scale features are input into the dual multi-modal multi-scale multi-head sparse attention module to obtain a refined shared content query, while achieving alignment and fusion of multi-modal features;

[0082] (3) The refined shared content query is passed through the classification network to output the target classification result, and the refined shared content query is passed through the visible light infrared dual regression network to output the visible light target frame position offset, which is added to the initial visible light reference point to obtain the visible light target position, and the infrared target frame position offset is output, which is added to the initial infrared reference point to obtain the infrared target position, where the visible light target position and the infrared target position are regarded as refined reference points and input into the next decoder to achieve the purpose of gradually refining the target frame.

[0083] Step S3, constructing a dual multi-modal multi-scale multi-head sparse attention module based on S2;

[0084] like Figure 2 As shown in the figure, a dual multimodal multiscale multi-head sparse attention module is constructed to achieve the alignment and fusion of multimodal features. In fact, the refined shared content query represents the fused features, but it is still necessary to dynamically interact the query features with the multimodal multiscale feature fusion so that it can find the target area in the multimodal multiscale features and accurately express the characteristics of the target.

[0085] Specifically, given the input multi-semantic infrared feature map and visible light feature map Use the qth normalized center point of the visible light and infrared reference points as the visible light reference point and infrared reference points Assumption F q For query features ( Figure 2 Q) after the multi-head self-attention module, F′ q is the updated query feature ( Figure 2 The process of the dual multi-modal multi-scale multi-head sparse attention module can be expressed as follows:

[0086]

[0087] Δp qhlk =f offset (F q )

[0088] Among them, H represents the number of attention heads, L represents the number of multi-scale levels in each modality, and K represents the number of feature sampling points in each layer. h and W′ h is the linear mapping network weight of the attention mechanism, It is a hierarchical mapping function used to map the normalized reference point to the l-th level feature map. and pass Figure 2 The sampling point weight network f in weight Generate, represents the kth attention weight in the hth attention head on the lth layer feature map, Δp qhlk pass Figure 2 The sampling point offset network f offset Generate, which represents the k-th sampling point offset in the h-th attention head on the l-th layer feature map. Note that the visible light and infrared modalities share a Δp qhlk ,This approach helps the network learn the offset difference of features and achieve ,feature alignment.

[0089] Step S4: construct dual regression loss and dual IoU-aware classification loss to guide model training.

[0090] The end-to-end weakly aligned multispectral remote sensing image target detection basic network outputs the target classification score, visible light target offset and infrared target offset in the query selection and decoding stages. Therefore, the dual regression loss and dual IoU-aware classification loss are constructed to guide the training of the model. Assume that the true value and network output are and z={c,b vis ,b ir}, the total loss L of the network can be expressed as follows:

[0091]

[0092] Among them, L box is the L1 loss function, L GIoU is the GIoU loss function, L1 is the commonly used mean absolute error loss function, IoU vis and IoU ir is the IoU between the predicted value and the true box, L cls The dual IoU-aware classification loss can be expressed as follows:

[0093]

[0094] Where d = (IoU vis IoU ir ), a, y, e are all hyper parameters. clsRather than just representing the possibility of foreground, it can simultaneously reflect object classification, visible light modality position, and infrared modality position.

[0095] Step S5: construct a dual contrast denoising training strategy to accelerate model training.

[0096] End-to-end object detection networks usually use the Hungarian matching algorithm to achieve one-to-one matching between query features and targets, eliminating the NMS post-processing algorithm, but it also leads to problems such as unstable network training and slow convergence. Therefore, a dual contrast denoising training strategy is constructed to accelerate the training of end-to-end weakly aligned multispectral remote sensing image object detection networks. Specifically:

[0097] Class noise is introduced by randomly flipping the ground truth labels to other labels. For pairs of visible and infrared bounding box noise, such as Figure 3 As shown in Figure 1, random displacement, scaling, and angle noise are applied to the visible and infrared bounding boxes of an object to generate pairs of denoised queries in both positive and negative modalities. The strength of the noise is controlled by a hyperparameter, with negative queries having more noise than positive queries. Applying random noise to the bounding boxes of both modalities also enhances the diversity of object misalignment situations.

[0098] During training, the content noise query and position noise query are input into the decoder network together with the initial content query and position query output by the query selection network. The content noise query and position noise query are denoised in the decoder, which accelerates the convergence and stability of the network. In addition, in order to prevent information leakage in the self-attention process, we introduce an attention mask to ensure that the denoised query cannot be seen by the initial query.

[0099] Step S6: Perform model training, verification and testing based on the data set in S1.

[0100] The training set in step S1 is used to train the end-to-end weakly aligned multispectral remote sensing image target detection network, the COCO dataset is used for pre-training to initialize the backbone network, the random gradient descent method is used for iterative training for 72 rounds, the batch size is 16, the initial learning rate is set to 0.0001, the parameters of the end-to-end weakly aligned multispectral remote sensing image target detection network are updated, and the validation set in step S1 is used to observe the detection performance of the end-to-end weakly aligned multispectral remote sensing image target detection network until the network performance converges, and an end-to-end weakly aligned multispectral remote sensing image target detection model is obtained. The test set in S1 is input into the model to obtain the detection result.

[0101] The present invention uses query features to align visible light and infrared modalities. When query features are used to find modality-specific target-related sparse sampling points, target feature alignment is automatically achieved without explicit target candidate areas.

[0102] While querying features to find modality-specific target-related sparse sampling points, the present invention also outputs sampling point weights, synchronously realizing the fusion of alignment features, and fully utilizing multi-modal complementarity.

[0103] The present invention adopts an end-to-end target detection method based on query features, outputs visible light and infrared target positions simultaneously, and uses a one-to-one matching method to completely abandon the operation of two-way NMS in traditional multimodal target detection.

[0104] In a second aspect, an embodiment of the present application provides an end-to-end weakly aligned multispectral remote sensing image target detection device, which is a target detection network, including a feature encoding network, a query selection network and a decoding network, wherein:

[0105] The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features;

[0106] A query selection network, used for processing the visible light coded features and the infrared coded features, and outputting an initial shared content query, an initial visible light reference point, and an initial infrared reference point;

[0107] The decoding network is used to interact the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and the infrared encoding features to achieve the alignment and fusion of multimodal features, and finally output the target classification results and the visible light infrared target position.

[0108] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An end-to-end weakly aligned multispectral remote sensing image target detection method, characterized in that: The specific process is: Construction of training data: Annotate the preset weakly aligned visible light and infrared remote sensing images to obtain a weakly aligned multispectral remote sensing image target detection dataset; Target detection network construction: Build an end-to-end weakly aligned multispectral remote sensing image target detection network. It includes feature encoding network, query selection network and decoding network, among which, The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features; A query selection network, used for processing the visible light coded features and the infrared coded features, and outputting an initial shared content query, an initial visible light reference point, and an initial infrared reference point; The decoding network is used to interact the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and the infrared encoding features to achieve the alignment and fusion of multimodal features, and finally output the target classification result and the visible light infrared target position; Target detection network training: Use the target detection dataset to train the target detection network; Remote sensing image target detection: Use the trained target detection network to perform remote sensing image target detection. The detection results include: target category, visible light target position, and infrared target position.

2. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 1, characterized in that: The query selection network includes: a classification network, a visible light regression network and an infrared regression network; The classification network is used to obtain the classification score of each pixel feature of the input fusion feature, and then select the K pixel features with the highest scores as the initial shared content query through the Top-K operation; The visible light regression network is used to select regression outputs corresponding to K pixel features from the visible light coding features, and add them to the initial target frame to form an initial visible light reference point; The infrared regression network is used to select regression outputs corresponding to K pixel features from the infrared coding features, and add them to the initial target frame to form an initial infrared reference point.

3. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 2, characterized in that: The specific processing process of the decoding network is: (1) The initial visible light reference point and the initial infrared reference point are used to obtain a location query through a shared location coding network, and the location query is added to the initial shared content query to form a shared query; (2) After the shared query is enhanced through the multi-head self-attention network to distinguish between different queries, the shared query and multi-modal multi-scale features are input into the dual multi-modal multi-scale multi-head sparse attention module to obtain a refined shared content query; (3) The refined shared content query is passed through the classification network to output the target classification result, and the refined shared content query is passed through the visible light infrared dual regression network. The output is added to the initial visible light reference point and the initial infrared reference point to obtain the visible light target position and the infrared target position, where the visible light target position and the infrared target position are regarded as refined reference points and input into the next decoder to achieve the purpose of gradually refining the target frame.

4. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 3, characterized in that: The algorithm of the dual multimodal multi-scale multi-head sparse attention module is: Then the process of the dual multi-modal multi-scale multi-head sparse attention module can be expressed as follows: Δp qhlk =f offset (F q ) Among them, H represents the number of attention heads, L represents the number of multi-scale levels in each modality, K represents the number of feature sampling points in each layer, and W h and W h ′ is the linear mapping network weight of the attention mechanism, is a hierarchical mapping function, and Through the sampling point weight network f weight Generate, represents the kth attention weight in the hth attention head on the lth layer feature map, Δp qhlk By offsetting the network f offset Generate, represents the kth sampling point offset in the hth attention head on the lth layer feature map, F q is the shared query feature, F′ q To refine shared content queries, is a multi-modal and multi-scale feature. and Represent infrared characteristics and visible light characteristics respectively, is the visible light reference point, Infrared reference point.

5. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 2, characterized in that: Construct dual regression loss and dual IoU-aware classification loss to guide the training of target detection network; Let the true value and network output be and z={c,b vis ,b ir }, the total loss L of the network can be expressed as follows: Among them, L box is the L1 loss function, L GIoU is the GIoU loss function, L1 is the mean absolute error loss function, IoU vis and IoU ir is the IoU between the predicted value and the true box, L cls It is the dual IoU-aware classification loss.

6. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 5, characterized in that: The L cls for: Where d = (IoU vis IoU ir ), a, y, and e are all hyperparameters.

7. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 1, characterized in that: During training, category noise is introduced by randomly flipping the true value labels to other labels. The intensity of the noise is controlled by a hyperparameter, and the noise of negative queries is larger than that of positive queries.

8. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 7, characterized in that: During training, the content noise query and the position noise query are input into the decoding network simultaneously with the initial content query and the position query output by the query selection network, and the content noise query and the position noise query are trained by denoising in the decoding network.

9. The end-to-end weakly aligned multispectral remote sensing image target detection method according to claim 1, characterized in that: An attention mask is introduced to ensure that the denoised query is not visible to the initial query.

10. An end-to-end weakly aligned multispectral remote sensing image target detection device, the device is a target detection network, including a feature encoding network, a query selection network and a decoding network, in, The feature encoding network is used to encode the input infrared and visible light images and output visible light encoding features and infrared encoding features; A query selection network, used for processing the visible light coded features and the infrared coded features, and outputting an initial shared content query, an initial visible light reference point, and an initial infrared reference point; The decoding network is used to interact the initial shared content query, the initial visible light reference point, the initial infrared reference point visible light encoding features and the infrared encoding features to achieve the alignment and fusion of multimodal features, and finally output the target classification results and the visible light infrared target position.

Images