Methods, apparatus, equipment and media for dual-modal image registration of unmanned aerial vehicles

Abstract

This application relates to a method, apparatus, device, and medium for dual-modal image registration of unmanned aerial vehicles (UAVs). The method involves using a feature extraction network to extract multi-scale features from the visible light and infrared images in a visible-infrared dual-modal image pair. Then, a feature aggregation network is used to fuse common features between the two features at the same scale. An encoder is then used to fuse the common fused features of visible light and infrared at multiple scales. A decoder constructed from a deformable convolutional network and sparse sampling is used to decode the encoder output to obtain an offset vector based on sampling points. Finally, a corner regression network and a regression network are used to obtain transformation parameters for registering the corresponding dual-modal image pair based on the offset vector. This method enables fast and effective online dual-modal registration.

Description

Technical Field

[0001] This application relates to the field of image registration technology, and in particular to a method, apparatus, device and medium for dual-modal image registration of unmanned aerial vehicles (UAVs). Background Technology

[0002] With the widespread use of various types of sensors, multimodal images are attracting increasing attention in fields such as autonomous driving and drone image processing due to their stronger anti-interference capabilities and larger information content compared to traditional visible light images. Drones equipped with multiple sensors offer a new way to obtain comprehensive information about the scene. By processing the multi-sensor image information acquired by drones, richer perceptual information can be obtained, effectively expanding the application scenarios of drones. Currently, visible light-infrared images have been widely used in many fields such as drone target detection and target tracking.

[0003] However, current UAV dual-modal image processing efforts primarily focus on the fusion of dual-modal images, assuming that the different modalities have already been registered. The visible light / infrared images acquired by the UAV are limited by the image acquisition equipment, resulting in unregistered dual-modal images that require offline pre-registration. This significantly restricts the application of dual-modal algorithms. Furthermore, for UAVs, the visible light payload includes wide-angle and zoom cameras. Wide-angle cameras have a fixed focal length and a large imaging range, but the target is small, making long-distance target observation impossible. Zoom cameras, by changing the focal length, can magnify the target, suitable for close-range target observation. The infrared payload is a fixed-focus camera with a fixed imaging area. In UAV dual-modal ground target detection and tracking tasks, both wide-angle and zoom cameras are needed simultaneously. Therefore, an algorithm directly applicable to UAV dual-modal image registration is crucial for UAV dual-modal visual perception tasks. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, device, and medium for dual-modal image registration of UAVs that can be directly used to address the above-mentioned technical problems.

[0005] A dual-modal image registration method for unmanned aerial vehicles (UAVs), the method comprising:

[0006] Acquire a real-time bimodal image pair dataset, which consists of multiple one-to-one corresponding visible light-infrared bimodal image pairs ordered by time from drone images;

[0007] A feature extraction network was used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain the visible light multi-scale features and the infrared multi-scale features;

[0008] For visible light features and infrared features at the same scale, a feature aggregation network is used to fuse the common features between the two features to obtain the common fused features of visible light and infrared features at the corresponding scale.

[0009] An encoder is used to fuse the common fusion features of visible light and infrared light at multiple scales to obtain the multi-scale fusion features of visible light and infrared light.

[0010] A decoder constructed from a deformable convolutional network and sparse sampling is used to decode based on the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling points.

[0011] A corner regression network is used to obtain the corner coordinates based on the offset vector, and a regression network is used to obtain the transformation parameters based on the corner coordinates.

[0012] The infrared image in the visible-infrared dual-modal image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible image.

[0013] In one embodiment, the encoder includes two encoder branches that respectively process visible light common fusion features and infrared common fusion features;

[0014] In each of the encoder branches, after splicing the common fusion features of visible light or infrared light at multiple scales, an attention mechanism is then used to enhance the spliced ​​features to obtain the multi-scale fusion features of visible light or infrared light.

[0015] The visible light multi-scale fusion feature and the infrared multi-scale fusion feature are stitched together and used as the output of the encoder.

[0016] In one embodiment, in the decoder:

[0017] The query vector is autocorrelated using a multi-head self-attention network to obtain the autocorrelated query vector.

[0018] The offset vector is obtained by sequentially processing the autocorrelation-derived query vector, the encoder's output features, and sparse sampling points using a multi-scale deformable attention network and a feedforward network.

[0019] In one embodiment, the processing flow in the decoder is represented as follows:

[0020] F MSDA =f q ' uery+MSDA(LN(f q ' uery ),R' point ,f encoder ))

[0021] f decoder =FFN(LN(f MSDA ))+f MSDA

[0022] In the above formula, f q ' uery R' represents the autocorrelation-adjusted query vector. point f represents the sparse sampling points. encoder F represents the output characteristics of the encoder. MSDA f MSDA This represents the output data of a multi-scale deformable attention network, f. decoder The output data of the decoder is the offset vector, LN(·) represents a linear layer, MSDA(·) represents a multi-scale deformable attention network, and FFN(·) represents a feedforward network.

[0023] In one embodiment, the corner regression network obtains the corner coordinates based on the offset vector, and the processing flow is as follows:

[0024] f output =MLP(f decoder )+Sig inv (R' point )

[0025] In the above formula, f decoder 、R' point These represent the output data of the decoder, i.e., the offset vector, and the sparse sampling points, respectively. Sig inv (·) indicates the inverse sigmoid transformation, f output This represents the coordinates of the corner point.

[0026] In one embodiment, the method includes:

[0027] The network is obtained by constructing parameter transformation based on the feature extraction network, feature aggregation network, encoder, decoder, corner regression network, and regression network.

[0028] The visible light-infrared dual-modal image pair is input into the trained parameter transformation network to obtain the corresponding transformation parameters.

[0029] In one embodiment, training the parameter transformation to obtain the network includes:

[0030] Obtain a bimodal image pair training dataset, which includes multiple training image pairs, each of which includes a visible light image and a corresponding infrared image;

[0031] The bimodal images are preprocessed for each training image in the training dataset, including size unification, grayscale variation, and pixel normalization.

[0032] A transformation matrix is ​​constructed based on preset transformation parameters, and the preprocessed infrared image is spatially transformed using a two-dimensional network and the transformation matrix.

[0033] The parameters are transformed by comparing the spatially transformed infrared image with the corresponding visible light image to train the network, and the transformation matrix is ​​used as the ground truth label.

[0034] This application provides a dual-modal image registration device for unmanned aerial vehicles (UAVs), the device comprising:

[0035] The dual-modal image pair dataset acquisition module is used to acquire a real-time dual-modal image pair dataset, which consists of multiple one-to-one corresponding visible light-infrared dual-modal image pairs obtained by UAV and sorted by time.

[0036] The feature extraction module is used to perform multi-scale feature extraction on the visible light image and the infrared image in the visible light-infrared dual-modal image pair using a feature extraction network, so as to obtain visible light multi-scale features and infrared multi-scale features.

[0037] The common feature fusion module is used to perform common feature fusion between visible light features and infrared features at the same scale using a feature aggregation network to obtain visible light common fusion features and infrared common fusion features at the corresponding scale.

[0038] The encoding fusion module is used to fuse the common fusion features of visible light and infrared light at multiple scales using an encoder, respectively, to obtain the multi-scale fusion features of visible light and the multi-scale fusion features of infrared light.

[0039] The decoding module is used to employ a decoder constructed from a deformable convolutional network and sparse sampling to decode based on the visible light multi-scale fusion features and infrared multi-scale fusion features to obtain an offset vector based on the sampling points.

[0040] The transformation parameter acquisition module is used to obtain corner coordinates based on the offset vector using a corner regression network, and to obtain transformation parameters based on the corner coordinates using a regression network.

[0041] The dual-modal image registration module is used to transform the infrared image in the visible-infrared dual-modal image pair according to the transformation parameters, so as to achieve registration with the corresponding visible light image.

[0042] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0043] Acquire a real-time bimodal image pair dataset, which consists of multiple one-to-one corresponding visible light-infrared bimodal image pairs ordered by time from drone images;

[0044] A feature extraction network was used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain the visible light multi-scale features and the infrared multi-scale features;

[0045] For visible light features and infrared features at the same scale, a feature aggregation network is used to fuse the common features between the two features to obtain the common fused features of visible light and infrared features at the corresponding scale.

[0046] An encoder is used to fuse the common fusion features of visible light and infrared light at multiple scales to obtain the multi-scale fusion features of visible light and infrared light.

[0047] A decoder constructed from a deformable convolutional network and sparse sampling is used to decode based on the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling points.

[0048] A corner regression network is used to obtain the corner coordinates based on the offset vector, and a regression network is used to obtain the transformation parameters based on the corner coordinates.

[0049] The infrared image in the visible-infrared dual-modal image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible image.

[0050] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0051] Acquire a real-time bimodal image pair dataset, which consists of multiple one-to-one corresponding visible light-infrared bimodal image pairs ordered by time from drone images;

[0052] A feature extraction network was used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain the visible light multi-scale features and the infrared multi-scale features;

[0053] For visible light features and infrared features at the same scale, a feature aggregation network is used to fuse the common features between the two features to obtain the common fused features of visible light and infrared features at the corresponding scale.

[0054] An encoder is used to fuse the common fusion features of visible light and infrared light at multiple scales to obtain the multi-scale fusion features of visible light and infrared light.

[0055] A decoder constructed from a deformable convolutional network and sparse sampling is used to decode based on the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling points.

[0056] A corner regression network is used to obtain the corner coordinates based on the offset vector, and a regression network is used to obtain the transformation parameters based on the corner coordinates.

[0057] The infrared image in the visible-infrared dual-modal image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible image.

[0058] The aforementioned method, apparatus, device, and medium for dual-modal image registration of UAVs employ a feature extraction network to extract multi-scale features from the visible light and infrared images in a visible-infrared dual-modal image pair. Then, a feature aggregation network is used to fuse common features between the two features at the same scale. An encoder is then used to fuse the common fused features of visible light and infrared at multiple scales. A decoder constructed from a deformable convolutional network and sparse sampling decodes the encoder output to obtain an offset vector based on sampling points. Finally, a corner regression network and a regression network are used to obtain transformation parameters for registering the corresponding dual-modal image pair based on the offset vector. This method enables fast and effective online dual-modal registration. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating a dual-modal image registration method for a drone in one embodiment.

[0060] Figure 2 This is a schematic diagram of the feature extraction network structure in one embodiment;

[0061] Figure 3This is a schematic diagram of the feature aggregation network in one embodiment;

[0062] Figure 4 This is a schematic diagram of the encoder, decoder, and corner regression network in one embodiment;

[0063] Figure 5 This is a schematic diagram of the training process of the network obtained by parameter transformation in one embodiment;

[0064] Figure 6 This is a schematic diagram illustrating the effects of image registration using various traditional methods and the proposed method in an experiment.

[0065] Figure 7 This is a schematic diagram of the thermal block diagram visualization of each module in the network for image pairs obtained by parameter transformation in an experiment.

[0066] Figure 8 This is a structural block diagram of a dual-modal image registration device for a drone in one embodiment;

[0067] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] Image registration is a fundamental task in image processing. Its core is to match images from different viewpoints, sensors, and times based on their correspondences, i.e., solving for the spatial relationships between images to ensure a one-to-one correspondence between points at the same spatial location in different images. There are two main methods for aligning visible-infrared images: traditional methods and depth-based methods.

[0070] Traditional methods extract feature point pairs from images and calculate image variation parameters based on the feature relationships between images, such as the SIFT algorithm. Meanwhile, to reduce errors in the extracted feature point pairs, the RANSAC algorithm is used to remove outlier feature point pairs. Depth-based image registration methods have now become mainstream.

[0071] Depth-based image registration methods, using CNN feature extraction networks as their backbone, calculate the correlation between extracted features to obtain the homography matrix between images. This method is a dense matching approach, considering only the relationships between feature pairs without taking into account global feature relationships. LOFTR uses a Transformer to model the feature relationships between features, avoiding the influence of anomalous feature pairs by considering global feature relationships. However, LOFTR is a two-stage matching process, computationally intensive, and inefficient.

[0072] Compared to visible image registration, bimodal image registration currently faces three main challenges. First, similar to UAV target detection tasks, targets in images captured from a UAV perspective are at different scales. Therefore, fully extracting multi-scale features for image representation is crucial in image registration tasks. Second, heterogeneous modal images differ in features. Visible images have rich texture features, while infrared images have clear contours. Addressing these differences is essential to effectively improve the accuracy of bimodal image registration. Third, there is a lack of UAV bimodal registration datasets available for training and evaluation. These issues, coupled with the current lack of directly available UAV bimodal registration datasets, hinder the development of UAV bimodal registration algorithms.

[0073] In one embodiment, such as Figure 1 As shown, a dual-modal image registration method for UAVs is provided, including the following steps:

[0074] Step S100: Obtain a real-time bimodal image pair dataset, which consists of multiple one-to-one corresponding visible light-infrared bimodal image pairs obtained by UAV and sorted by time.

[0075] Step S110: A feature extraction network is used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain visible light multi-scale features and infrared multi-scale features.

[0076] Step S120: For visible light features and infrared features at the same scale, a feature aggregation network is used to fuse the common features between the two features to obtain the common fused features of visible light and infrared features at the corresponding scale.

[0077] Step S130: The encoder is used to fuse the common fusion features of visible light and infrared light at multiple scales to obtain the common fusion features of visible light and infrared light at multiple scales.

[0078] Step S140: A decoder constructed from a deformable convolutional network and sparse sampling is used to decode based on the multi-scale fusion features of visible light and the multi-scale fusion features of infrared light to obtain the offset vector based on the sampling points.

[0079] Step S150: The corner coordinates are obtained by using a corner regression network based on the offset vector, and the transformation parameters are obtained by using the regression network based on the corner coordinates.

[0080] Step S160: Transform the infrared image in the visible-infrared dual-modal image pair according to the transformation parameters to achieve registration with the corresponding visible image.

[0081] In this embodiment, the visible light-infrared dual-modal image pairs acquired in real time by the UAV are sequentially registered online to provide better pre-service for subsequent tasks such as target recognition.

[0082] In step S100, the images to be registered include multiple pairs, which are actually dual-modal image pairs consisting of a set of visible light images and an infrared image corresponding to each moment acquired in real time by the UAV. During processing, one image pair is processed at a time, and sequential processing is performed. Steps S110 and S160 are executed for each dual-modal image pair to achieve online registration of the image pairs.

[0083] Before step S110, the two images in the bimodal image pair are preprocessed separately. The preprocessing process includes, but is not limited to, ...

[0084] In step S110, two feature extraction networks are used to extract multi-scale features from the visible light image and the infrared image respectively, resulting in features at multiple different scales.

[0085] In this embodiment, the feature extraction network uses a ResNet network, the structure of which is as follows: Figure 2 As shown, in order to extract multi-scale image features, the last connection layer in the original ResNet network was removed, and the output of the last three convolutional layers was used as the output features. A convolution was used to transform the features to a preset size, and finally, image features at three different scales were obtained for visible light images and infrared images respectively.

[0086] Specifically, the three image features are at scales of 60*60*256, 30*30*256, and 15*15*256, with corresponding strides of 8, 16, and 32, respectively. The feature image scale is not limited to these given scales and can be other scales.

[0087] In this embodiment, since visible light images and infrared images are images of different modalities with different features, and the differences between these two modalities are significant, two feature extraction networks are needed to extract visible light features and infrared features respectively.

[0088] In step S120, the visible light feature image and the infrared feature image at the same scale are respectively subjected to feature aggregation to obtain the visible light common fusion feature and the infrared common fusion feature after heterogeneous feature aggregation.

[0089] Because visible light and infrared imaging mechanisms are different, the image features extracted by the feature extraction network are also different. Dual-modal image registration uses the correlation between image features for registration. Here, dual-modal fusion of visible light and infrared features is required to enhance the common features in different modalities.

[0090] In this embodiment, multiple convolutions are used to implement a dual-modal fusion network, and the specific network structure is as follows: Figure 3 Place.

[0091] Specifically, since multiple image features at different scales are extracted in step S110, in order to fuse the corresponding scale dual-modal information, it is necessary to perform modal information fusion on the visible light-infrared dual-modal image at each scale. The fusion process can be represented as follows:

[0092] F C =Conv C (Cat((Conv rgb (F rgb [i]),Conv ir (F ir [i])))

[0093] In the above formula, F rgb [i], F ir [i] represents the visible light multi-scale feature and the infrared multi-scale feature at the i-th scale, respectively. Conv(·) represents the convolution operation, and Cat(·) represents the dimension concatenation operation.

[0094] Next, regarding F C Performing a dimension separation operation is represented as:

[0095]

[0096] In the above formula, Split(·) represents the dimension separation operation, which separates the features F obtained above. C Separate the images to obtain visible light image features. and infrared image features

[0097] Furthermore, the visible light image features were analyzed separately. and infrared image features Perform the following operations:

[0098]

[0099] In the above formula, convolutional layers Conv1 and Conv2 are used to process the visible light image features, respectively. and infrared image features After performing convolution, the Sigmoid activation function is applied for further processing, targeting the features of visible light images. get Targeting infrared image features get Here, the visible and infrared features share two identical convolutions to enhance the common features across different modalities.

[0100] Finally, skip connections are used to obtain the fused bimodal features at the i-th scale, and common and characteristic features from different modalities are fused to form the aggregated bimodal features at the i-th scale. The fusion process is represented as follows:

[0101]

[0102] Feature maps at other scales are aggregated in the same way. Therefore, the feature aggregation network includes three branch networks that handle feature fusion at different scales.

[0103] After passing through the feature aggregation network, visible light multi-scale fusion features and corresponding infrared multi-scale fusion features can be obtained for each scale.

[0104] Following the bimodal fusion network, i.e., the feature aggregation network, bimodal features are obtained after modal information interaction. These bimodal features are then fed into the feature fusion network to model feature correlations, thereby estimating the variation parameters between images. Next, a Transformer is used to construct the feature fusion network. Since this method uses multi-scale features from visible light-infrared bimodal images, the concatenated feature size is b*(60*60+30*30+15*15)*256. Using a traditional Transformer structure would result in a massive computational burden, impacting network speed. Considering that sparse spatial sampling can reduce redundant computations and significantly improve network efficiency, effectively solving the problem of redundant computations in multi-scale images, deformable convolution is employed. By learning to determine convolution parameters, the network can better capture subtle features of the target, effectively improving the feature relationship modeling capability.

[0105] In step S130, the encoder includes two encoder branches that process visible light common fusion features and infrared common fusion features respectively. In each encoder branch, the visible light common fusion features or infrared common fusion features of multiple scales are spliced ​​together, and then the spliced ​​features are enhanced by an attention mechanism to obtain visible light multi-scale fusion features or infrared multi-scale fusion features. Finally, the visible light multi-scale fusion features and infrared multi-scale fusion features are spliced ​​together and used as the output of the encoder.

[0106] In this embodiment, the encoder consists of a multi-head self-attention module and a feedforward network. The feedforward network contains a two-layer multilayer perceptron, using GELU as the activation function. Layer normalization is used in the encoder to avoid overfitting. Residual structures are used in both the multi-head self-attention module and the feedforward network module to facilitate gradient backpropagation. Taking the processing of visible light features as an example, the encoder process can be expressed as follows:

[0107] Stitching together image features from multiple scales:

[0108] F = Cat(F) r ' gb [0],F r ' gb [1],F r ' gb [2])

[0109] In the above formula, F r ' gb [0],F r ' gb [1],F r ' gb [2] represent the common fusion features of visible light at different scales.

[0110] The stitched features are enhanced using an attention mechanism:

[0111] F MSA =F+MSA(LN(F))

[0112] f rgb =MLP(LN(F MSA ))+F MSA

[0113] In the above formula, LN(·) represents a linear layer, MSA(·) represents a multi-head self-attention network, and MLPA(·) represents a multi-layer perceptual layer. Similarly, the infrared image features f after passing through the self-attention network can be obtained. ir Here f rgb and f irThese represent the visible light-infrared image features after the encoder, namely the visible light multi-scale fusion features and the infrared multi-scale fusion features.

[0114] Finally, the optical multi-scale fusion features and the infrared multi-scale fusion features are stitched together using the following formula to obtain the encoder's output data:

[0115] f encoder =Cat(f rgb ,f ir )

[0116] In step S140, since multi-scale feature maps are used in this method, which can cause memory explosion, deformable convolutional networks and sparse sampling are used for decoding. By using multi-scale deformable attention, computational efficiency can be improved and the ability to model small target features can be enhanced.

[0117] In this embodiment, in the decoder: the query vector is autocorrelated using a multi-head self-attention network to obtain the autocorrelated query vector. Based on the autocorrelated query vector, the encoder's output features, and sparse sampling points, the offset vector is obtained by sequentially processing the data using a multi-scale deformable attention network and a feedforward network.

[0118] Specifically, the decoder consists of multiple networks including multi-head self-attention (MSA), multi-scale deformable attention (MSDA), and feedforward networks (FFN). In the multi-scale deformable attention module, the query can focus on the positions of all objects on the feature map, thereby predicting the coordinates of four robust corner points.

[0119] First, self-attention is applied to the query, expressed by the following formula:

[0120] f q ' uery =F query +MSA(LN(F query ))

[0121] In the above formula, F query f represents the input query vector, which is autocorrelated after self-attention. q ' uery In this embodiment, the coefficient sampling points are obtained using the following formula:

[0122] R' point =Sig(Linear(R) point ))

[0123] Multiscale Deformable Attention Module (MSDA) is a sparse sampling process that requires determining the locations of sparse sampling points. In the above equation, R here... pointThis represents the offset vector of sparse sampling points in the x and y directions, with a size of n*2. `Linear(·)` indicates a linear layer with an input of 256 and an output of 2. Where R... point It is the initial offset vector with a size of n*256.

[0124] The query vector and the encoder's output features are then decoded. The processing flow is as follows:

[0125] F MSDA =f q ' uery +MSDA(LN(f q ' uery ),R' point ,f encoder ))

[0126] f decoder =FFN(LN(f MSDA ))+f MSDA

[0127] In the above formula, f q ' uery R' represents the query vector after autocorrelation. point f represents sparse sampling points encoder F represents the output characteristics of the encoder. MSDA f MSDA This represents the output data of a multi-scale deformable attention network, f. decoder LN(·) represents the output data of the decoder, i.e., the offset vector. LN(·) represents a linear layer, MSDA(·) represents a multi-scale deformable attention network, and FFN(·) represents a feedforward network.

[0128] Furthermore, the decoder output is an offset vector f based on the sample points. decoder What is needed is the coordinates of the four corner points of the query vector. To obtain the actual x and y coordinates of the corner points, the offset vector needs to be processed, and the coordinates of the sampling points are added. In step S150, a corner regression network is used to obtain the corner coordinates based on the offset vector. The processing flow is as follows:

[0129] f output =MLP(f decoder )+Sig inv (R' point )

[0130] In the above formula, f decoder 、R' point Sig represents the decoder's output data, i.e., the offset vector and the sparse sampling points, respectively. inv (·) indicates the inverse sigmoid transformation, f output Represents the coordinates of the corner point.

[0131] Next, a regression network is used to perform regression based on the intersection coordinates to obtain the transformation parameters. In this embodiment, the regression network can use three linear layers as the parametric regression network. Since the value range of the transformation parameters is (-2 to 2), the Sigmoid or ReLU activation function is not used here; the transformation parameters are obtained by directly regressing the parameters.

[0132] In this embodiment, the structures of the encoder, decoder, and corner regression network are as follows: Figure 4 As shown.

[0133] In this embodiment, the aforementioned feature extraction network, feature aggregation network, encoder, decoder, corner regression network, and regression network can be used to construct a parameter transformation network. The visible light-infrared bimodal image pair is then input into the trained parameter transformation network to directly obtain the corresponding transformation parameters. After transforming the infrared image in the bimodal image pair according to these parameters, registration with the visible light image can be achieved.

[0134] In this embodiment, when training the network by transforming the parameters: First, a bimodal image pair training dataset is acquired, which includes multiple training image pairs, each of which includes a visible light image and a corresponding infrared image. Next, each training image in the bimodal image pair training dataset undergoes preprocessing, including size unification, grayscale variation, and pixel normalization. A transformation matrix is ​​constructed based on preset transformation parameters. Then, a spatial transformation is performed on the preprocessed infrared image using a two-dimensional network and the transformation matrix. Finally, the spatially transformed infrared image and the corresponding visible light image are used to train the network obtained by transforming the parameters, and the transformation matrix is ​​used as the ground truth label.

[0135] Specifically, the images in the training dataset of the bimodal image pairs are first preprocessed, and data augmentation is performed to enhance the generalization ability of the algorithm. The images are first uniformly scaled to ensure that the obtained infrared and visible light images have the same scale. Then, grayscale changes with a probability of 5% are performed on the image pairs to enhance the algorithm's generalization ability across various scenarios. The pixels of the visible light-infrared image pairs are normalized using the mean [0.485, 0.456, 0.406] and variance [0.229, 0.224, 0.225] obtained statistically from the ImageNet dataset.

[0136] Specifically, when constructing the transformation matrix, appropriate transformation parameters, including rotation, translation, and scaling, are selected to construct a transformation matrix that meets the requirements.

[0137] Furthermore, when transforming an infrared image using a transformation matrix, a two-dimensional grid of the same size as the infrared image is first constructed. The transformation matrix is ​​then used to spatially transform the two-dimensional grid. The transformed two-dimensional grid is then used to resample the infrared image to obtain the spatially transformed infrared image.

[0138] Furthermore, the transformed infrared image and the corresponding visible light image are used as transformation parameters to obtain the network input, and the corresponding transformation matrix is ​​used as the ground truth label to train the network until it converges, thus obtaining the trained transformation parameters and the network.

[0139] Specifically, when training the network using the transformed infrared image and the corresponding visible light image, the data processing process in the network is the same as the data processing flow in steps S110 to S150 above. First, multi-scale visible light-infrared features are extracted, and the visible light-infrared branch features are aggregated to obtain a multi-scale bimodal aggregated feature map. The multi-scale bimodal aggregated feature map of the aggregated visible light-infrared branch features is input into the multi-scale encoder and decoder to obtain the correspondence between features, regress the x and y coordinates of the four corner points, and finally obtain the transformation parameters of the infrared-visible image pair. Based on the transformation parameters of the infrared-visible image pair, the image pair is spatially transformed to obtain the registered infrared-visible image pair. The above process is repeated to continuously infer the image sequence, thereby achieving uninterrupted registration of UAV image pairs.

[0140] like Figure 5 The diagram shows the overall process of training the network by transforming the parameters.

[0141] In this paper, the effectiveness of the proposed method is also demonstrated through experiments. The experiments were conducted on a dataset collected by an experimental device mounted on an UAV with an optoelectronic pod. The robustness and registration accuracy of the proposed method compared to similar methods were analyzed through experimental comparison.

[0142] The dataset used in the experiment was actually acquired by a drone equipped with an electro-optical pod. The test sequence includes image sequence pairs acquired by the drone in various scenarios, including common drone mission scenarios such as changes in lighting, occlusion, shadows, and rapid drone movement. By preprocessing the dataset, unregistered bimodal image pairs and ground truth values ​​of the transformation parameters can be obtained. Inputting the unregistered bimodal image pairs into the transformation parameters of the proposed method's network yields the network's estimated transformation parameters. By comparing these estimates with the ground truth values, the registration effectiveness of the proposed method can be tested.

[0143] In the comparative experiments and analysis, the root mean square error (RMSE), average endpoint error (AEE), and average probability of correct keypoints (PCK) were used as evaluation metrics. PCK was used to measure correct keypoints at thresholds of 3, 5, and 7 pixels. A comparative experiment was conducted between the visible-infrared dual-modal image registration algorithm and mainstream registration algorithms (DHN, Cnngeo, unsupervise, PFNeT). The experimental results are shown in Table 1.

[0144] Table 1 Comparison Results of Algorithms

[0145]

[0146] Table 1 presents the statistical results of the comparative experiments of the above algorithms. As can be seen from Table 1, the proposed method outperforms other registration algorithms in terms of results, demonstrating a significant advantage across multiple metrics. Overall, the proposed method achieves good results in the experimental data, as shown in the specific registration performance figures. Figure 6 As shown.

[0147] Next, ablation experiments were conducted on each module of the proposed method to verify the effectiveness of the algorithm. The results of the ablation experiments are shown in Table 2.

[0148] Table 2 Ablation experiments of the algorithm

[0149]

[0150] In this experiment, five ablation experiments were conducted to compare and analyze the results: baseline algorithm, using multi-scale features, both encoder and decoder using multi-scale deformable attention mechanism, encoder using Transformer structure and decoder using multi-scale deformable attention mechanism, and the method proposed in this paper.

[0151] The baseline algorithm is inspired by target tracking algorithms. It mainly consists of three parts: a backbone network, a feature fusion network, and a regression network. The feature fusion network comprises a classic encoder and decoder structure. As shown in Table 2, the baseline's RMSE, AEE, pck_7, pck_5, and pck_3 are 21.19%, 19.9%, 48.1%, 33.1%, and 12.9%, respectively.

[0152] First, multi-scale features (Multi) are utilized on top of the baseline. In this case, the network uses multi-scale features to compute transformation parameters between image pairs. As the number of input features increases, training and inference of the network become more challenging, leading to greater display memory requirements. The batch size is limited to 1, resulting in an inference speed of 20fps. However, compared to the baseline algorithm, performance is significantly improved. RMSE is reduced by 8.73, AEE by 7.79, and PCK_7, PCK_5, and PCK_3 are increased by 11.7%, 19.1%, and 20.3%, respectively. Experiments show that using multi-scale features helps the network obtain more detailed feature representations and achieves better performance on small targets.

[0153] To reduce memory explosion caused by multi-scale processing, we replace traditional transformer layers with deformable attention and replace traditional dense sampling with sparse sampling. This significantly reduces network memory usage, resulting in a 1.43% reduction in RMSE, a 1.28% reduction in AEE, and increases in PCK_7 and PCK_5 by 4.9% and 2.9%, respectively, while PCK_3 decreases by 4.9%. These results demonstrate that deformable attention layers and sparse sampling reduce feature redundancy and improve computational efficiency. However, the use of sparse sampling in the encoder and decoder leads to the neglect of some important features, resulting in some performance degradation.

[0154] To improve the network's ability to capture details, a classic Transformer layer is used as the encoder. Since the encoder is insensitive to feature additions, this avoids memory overflow. Compared to the Deformable network, the network's ability to extract details is improved by 4.9% and 2.9% for PCK_7 and PCK_5 respectively, while PCK_7 is reduced by 1%. The results show that using a densely sampled encoder helps capture detailed image features, but feature redundancy is unavoidable.

[0155] Finally, a bimodal feature aggregation network was added to the network (the method presented in this paper). This network effectively addresses the heterogeneous feature differences between visible and infrared modalities. Using this network, information from the visible and infrared modalities is exchanged, and information from different modalities is aggregated, eliminating the differences in bimodal features. Compared to the encoder network, its performance is significantly improved. RMSE is reduced by 7.71, AEE by 7.71, and PCK_7, PCK_5, and PCK_3 are increased by 27.8%, 25.2%, and 27.0%, respectively. This demonstrates that the bimodal feature aggregation network can aggregate visible and infrared modal information, effectively solving the feature difference problem between different modes. This also highlights the importance of feature aggregation for images of different modalities.

[0156] To more clearly demonstrate the algorithm's performance in bimodal image registration, we visualized the algorithm using images from the dataset. The specific results are as follows: Figure 7 As shown in the figure, the infrared and visible light images represent infrared and visible light image pairs in the dataset, respectively. It can be seen that the dataset contains image pairs from multiple scenes, and the visible light sensor cannot function properly under low-light conditions. The ground truth visible light image represents the ground truth visible light image in the synthetic dataset, which is obtained by changing the visible light image with varying parameters. The network's input image pairs are infrared and visible light ground truth image pairs, and its output is the varying parameters. Using the network's output varying parameters to process the input infrared image yields the changed infrared image (the network-predicted image). Comparison images of the changed image with the original visible light image and the ground truth visible light image are then shown (where the visible light image forms the R and G channels, and the changed infrared image forms the B channel). Through the comparison images, it can be seen that the parameters estimated by the proposed network can achieve high-quality registration of visible light-infrared image pairs. Furthermore, the proposed network can effectively utilize dual-modal image information and achieve good results even under low-light conditions.

[0157] The aforementioned dual-modal image registration method for UAVs constructs a fully Transformer-based dual-modal image registration network, utilizing multi-scale image features to enhance the network's registration performance for detailed features. Simultaneously, a dual-modal feature aggregation network is proposed, extracting common features from heterogeneous modalities and combining them with characteristic features from each modality to generate aggregated dual-modal features. A deformable attention module is used for sparse sampling, reducing parameter redundancy caused by multi-scale features and improving the network's training and inference speed.

[0158] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0159] In one embodiment, such as Figure 8As shown, a dual-modal image registration device for unmanned aerial vehicles (UAVs) is provided, comprising: a dual-modal image pair dataset acquisition module 200, a feature extraction module 210, a common feature fusion module 220, an encoding fusion module 230, a decoding module 240, a transform parameter acquisition module 250, and a dual-modal image registration module 260, wherein:

[0160] The bimodal image pair dataset acquisition module 200 is used to acquire a real-time bimodal image pair dataset, wherein the bimodal image pair dataset consists of multiple one-to-one corresponding visible light-infrared bimodal image pairs obtained by UAV and sorted by time.

[0161] Feature extraction module 210 is used to perform multi-scale feature extraction on the visible light image and infrared image in the visible light-infrared dual-modal image pair using a feature extraction network, so as to obtain visible light multi-scale features and infrared multi-scale features.

[0162] The common feature fusion module 220 is used to perform common feature fusion between visible light features and infrared features at the same scale using a feature aggregation network to obtain visible light common fusion features and infrared common fusion features at the corresponding scale.

[0163] The encoding fusion module 230 is used to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales using an encoder, respectively, to obtain the multi-scale fusion features of visible light and the multi-scale fusion features of infrared light.

[0164] The decoding module 240 is used to decode the sample point-based offset vector by employing a decoder constructed from a deformable convolutional network and sparse sampling, based on the visible light multi-scale fusion features and the infrared multi-scale fusion features.

[0165] The transformation parameter acquisition module 250 is used to obtain corner coordinates based on the offset vector using a corner regression network, and to obtain transformation parameters based on the corner coordinates using a regression network.

[0166] The dual-modal image registration module 260 is used to transform the infrared image in the visible-infrared dual-modal image pair according to the transformation parameters, so as to achieve registration with the corresponding visible light image.

[0167] Specific limitations regarding the dual-modal image registration device for UAVs can be found in the limitations of the dual-modal image registration method for UAVs mentioned above, and will not be repeated here. Each module in the aforementioned dual-modal image registration device for UAVs can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0168] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a dual-modal image registration method for unmanned aerial vehicles (UAVs). The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0169] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0170] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0171] Acquire a real-time bimodal image pair dataset, which consists of multiple one-to-one corresponding visible light-infrared bimodal image pairs ordered by time from drone images;

[0172] A feature extraction network was used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain the visible light multi-scale features and the infrared multi-scale features;

[0173] For visible light features and infrared features at the same scale, a feature aggregation network is used to fuse the common features between the two features to obtain the common fused features of visible light and infrared features at the corresponding scale.

[0174] An encoder is used to fuse the common fusion features of visible light and infrared light at multiple scales to obtain the multi-scale fusion features of visible light and infrared light.

[0175] A decoder constructed from a deformable convolutional network and sparse sampling is used to decode based on the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling points.

[0176] A corner regression network is used to obtain the corner coordinates based on the offset vector, and a regression network is used to obtain the transformation parameters based on the corner coordinates.

[0177] The infrared image in the visible-infrared dual-modal image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible image.

[0178] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0179] Acquire a real-time bimodal image pair dataset, which consists of multiple one-to-one corresponding visible light-infrared bimodal image pairs ordered by time from drone images;

[0180] A feature extraction network was used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain the visible light multi-scale features and the infrared multi-scale features;

[0181] For visible light features and infrared features at the same scale, a feature aggregation network is used to fuse the common features between the two features to obtain the common fused features of visible light and infrared features at the corresponding scale.

[0182] An encoder is used to fuse the common fusion features of visible light and infrared light at multiple scales to obtain the multi-scale fusion features of visible light and infrared light.

[0183] A decoder constructed from a deformable convolutional network and sparse sampling is used to decode based on the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling points.

[0184] A corner regression network is used to obtain the corner coordinates based on the offset vector, and a regression network is used to obtain the transformation parameters based on the corner coordinates.

[0185] The infrared image in the visible-infrared dual-modal image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible image.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A dual-modal image registration method for unmanned aerial vehicles (UAVs), characterized in that, The method includes: Acquire a real-time bimodal image pair dataset, which consists of multiple one-to-one corresponding visible light-infrared bimodal image pairs ordered by time from drone images; A feature extraction network was used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain the visible light multi-scale features and the infrared multi-scale features; For visible light features and infrared features at the same scale, a feature aggregation network is used to fuse the common features between the two features to obtain the common fused features of visible light and infrared features at the corresponding scale. An encoder is used to fuse the common fusion features of visible light and infrared light at multiple scales to obtain the multi-scale fusion features of visible light and infrared light. A decoder constructed from a deformable convolutional network and sparse sampling is used to decode based on the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling points. A corner regression network is used to obtain the corner coordinates based on the offset vector, and a regression network is used to obtain the transformation parameters based on the corner coordinates. The infrared image in the visible-infrared dual-modal image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible image.

2. The dual-modal image registration method according to claim 1, characterized in that, The encoder includes two encoder branches that respectively process visible light common fusion features and infrared common fusion features; In each of the encoder branches, after splicing the common fusion features of visible light or infrared light at multiple scales, an attention mechanism is then used to enhance the spliced ​​features to obtain the multi-scale fusion features of visible light or infrared light. The visible light multi-scale fusion feature and the infrared multi-scale fusion feature are stitched together and used as the output of the encoder.

3. The dual-modal image registration method according to claim 2, characterized in that, In the decoder: The query vector is autocorrelated using a multi-head self-attention network to obtain the autocorrelated query vector. The offset vector is obtained by sequentially processing the autocorrelation-derived query vector, the encoder's output features, and sparse sampling points using a multi-scale deformable attention network and a feedforward network.

4. The dual-modal image registration method according to claim 3, characterized in that, In the decoder, the processing flow is represented as follows: F MSDA =f q ' uery +MSDA(LN(f q ' uery ),R' point ,f encoder )) f decoder =FFN(LN(f MSDA ))+f MSDA In the above formula, f q ' uery R' represents the autocorrelation-adjusted query vector. point f represents the sparse sampling points. encoder F represents the output characteristics of the encoder. MSDA f MSDA This represents the output data of a multi-scale deformable attention network, f. decoder The output data of the decoder is the offset vector, LN(·) represents a linear layer, MSDA(·) represents a multi-scale deformable attention network, and FFN(·) represents a feedforward network.

5. The dual-modal image registration method according to claim 3, characterized in that, The corner point regression network obtains the corner point coordinates based on the offset vector, and the processing flow is as follows: f output =MLP(f decoder )+Sig inv (R' point ) In the above formula, f decoder 、R' point These represent the output data of the decoder, i.e., the offset vector, and the sparse sampling points, respectively. Sig inv (·) indicates the inverse sigmoid transformation, f output This represents the coordinates of the corner point.

6. The dual-modal image registration method according to any one of claims 1-5, characterized in that, The method includes: The network is obtained by constructing parameter transformation based on the feature extraction network, feature aggregation network, encoder, decoder, corner regression network, and regression network. The visible light-infrared dual-modal image pair is input into the trained parameter transformation network to obtain the corresponding transformation parameters.

7. The dual-modal image registration method according to claim 6, characterized in that, Training the network by the parameter transformation includes: Obtain a bimodal image pair training dataset, which includes multiple training image pairs, each of which includes a visible light image and a corresponding infrared image; The bimodal images are preprocessed for each training image in the training dataset, including size unification, grayscale variation, and pixel normalization. A transformation matrix is ​​constructed based on preset transformation parameters, and the preprocessed infrared image is spatially transformed using a two-dimensional network and the transformation matrix. The parameters are transformed by comparing the spatially transformed infrared image with the corresponding visible light image to train the network, and the transformation matrix is ​​used as the ground truth label.

8. A dual-modal image registration device for unmanned aerial vehicles (UAVs), characterized in that, The device includes: The dual-modal image pair dataset acquisition module is used to acquire a real-time dual-modal image pair dataset, which consists of multiple one-to-one corresponding visible light-infrared dual-modal image pairs obtained by UAV and sorted by time. The feature extraction module is used to perform multi-scale feature extraction on the visible light image and the infrared image in the visible light-infrared dual-modal image pair using a feature extraction network, so as to obtain visible light multi-scale features and infrared multi-scale features. The common feature fusion module is used to perform common feature fusion between visible light features and infrared features at the same scale using a feature aggregation network to obtain visible light common fusion features and infrared common fusion features at the corresponding scale. The encoding fusion module is used to fuse the common fusion features of visible light and infrared light at multiple scales using an encoder, respectively, to obtain the multi-scale fusion features of visible light and the multi-scale fusion features of infrared light. The decoding module is used to employ a decoder constructed from a deformable convolutional network and sparse sampling to decode based on the visible light multi-scale fusion features and infrared multi-scale fusion features to obtain an offset vector based on the sampling points. The transformation parameter acquisition module is used to obtain corner coordinates based on the offset vector using a corner regression network, and to obtain transformation parameters based on the corner coordinates using a regression network. The dual-modal image registration module is used to transform the infrared image in the visible-infrared dual-modal image pair according to the transformation parameters, so as to achieve registration with the corresponding visible light image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images