Dual-mode image registration method and device for unmanned aerial vehicle, equipment and medium

Abstract

The invention relates to a dual-mode image registration method and device for an unmanned aerial vehicle, equipment and a medium. A feature extraction network is adopted to perform multi-scale feature extraction on a visible light image and an infrared image in a visible light-infrared dual-mode image pair; for visible light features and infrared ray features under the same scale, common feature fusion between the two features is carried out by using a feature aggregation network, and then visible light common fusion features and infrared ray common fusion features under multiple scales are fused by using an encoder. A decoder constructed by a deformable convolutional network and sparse sampling is utilized to decode the output of an encoder to obtain an offset vector based on sampling points, and finally a corner regression network and a regression network are adopted to obtain transformation parameters for registering the corresponding bimodal image pair according to the offset vector. By adopting the method, online bimodal registration can be quickly and effectively carried out.

Description

Technical Field

[0001] The present application relates to the technical field of image registration, and in particular to a dual-modal image registration method, device, equipment and medium for unmanned aerial vehicles. Background Art

[0002] With the widespread use of multiple types of sensors, multimodal images have attracted more and more attention in the fields of autonomous driving and drone image processing due to their strong anti-interference ability and large amount of information compared to traditional visible light images. Drones equipped with multiple sensors provide a new way to obtain comprehensive information on the scene. By processing the multi-sensor image information obtained by drones, richer perception information can be obtained, effectively expanding the use scenarios of drones. At present, visible light-infrared images have been widely used in many fields such as drone target detection and target tracking.

[0003] However, the current work on UAV dual-modal image processing only focuses on the fusion task of dual-modal images, assuming that different modal images have been registered. However, the visible light / infrared images obtained by the UAV are limited by the image acquisition equipment, and the obtained dual-modal images are not registered. They need to be pre-registered offline, which greatly limits the application of dual-modal algorithms. At the same time, for UAVs, the visible light payload includes wide-angle and zoom cameras. The wide-angle camera has a fixed focal length and a large imaging range, but the target is small and cannot be observed at a long distance. The zoom camera can magnify the target by changing the focal length, which is suitable for close-range observation of the target. The infrared payload is a fixed-focus camera with a fixed imaging area. In the dual-modal ground target detection and tracking tasks of UAVs, both wide-angle and zoom cameras need to be used simultaneously to realize the detection and tracking tasks of UAVs. Therefore, an algorithm directly used for dual-modal image registration of UAVs is very important for the dual-modal visual perception tasks of UAVs. Summary of the invention

[0004] Based on this, it is necessary to provide a dual-modal image registration method, device, equipment and medium for drones that can be directly used to address the above technical problems.

[0005] A dual-modal image registration method for an unmanned aerial vehicle, the method comprising:

[0006] Acquire a real-time dual-modal image pair data set, wherein the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs taken by a drone and ordered in time;

[0007] A feature extraction network is used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain visible light multi-scale features and infrared multi-scale features;

[0008] A feature aggregation network is used to fuse the common features of visible light features and infrared features at the same scale, respectively, to obtain the common fusion features of visible light and infrared at the corresponding scales.

[0009] An encoder is used to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales to obtain visible light multi-scale fusion features and infrared multi-scale fusion features;

[0010] A decoder constructed by a deformable convolutional network and sparse sampling is used to decode the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling point;

[0011] Using a corner point regression network to obtain corner point coordinates according to the offset vector, and using a regression network to obtain transformation parameters according to the corner point coordinates;

[0012] The infrared image in the visible light-infrared dual-modality image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible light image.

[0013] In one embodiment, the encoder includes two encoder branches for processing visible light commonality fusion features and infrared commonality fusion features respectively;

[0014] In each of the encoder branches, after splicing the common fusion features of visible light or the common fusion features of infrared light at multiple scales, an attention mechanism is used to enhance the spliced ​​features to obtain a visible light multi-scale fusion feature or an infrared multi-scale fusion feature;

[0015] The visible light multi-scale fusion features and the infrared multi-scale fusion features are concatenated and used as the output of the encoder.

[0016] In one embodiment, in the decoder:

[0017] Use a multi-head self-attention network to autocorrelate the query vector to obtain the autocorrelated query vector;

[0018] According to the query vector after autocorrelation, the output features of the encoder and the sparse sampling points, a multi-scale deformable attention network and a feedforward network are used in sequence to obtain the offset vector.

[0019] In one embodiment, in the decoder, the processing flow is represented as follows:

[0020] F MSDA =f q ' uery+MSDA(LN(f q ' uery ),R' point ,f encoder ))

[0021] f decoder =FFN(LN(f MSDA ))+f MSDA

[0022] In the above formula, f q ' uery represents the query vector after the autocorrelation, R' point represents the sparse sampling points, f encoder represents the output feature of the encoder, F MSDA 、f MSDA represents the output data of the multi-scale deformable attention network, f decoder represents the output data of the decoder, i.e., the offset vector, the LN(·) represents a linear layer, the MSDA(·) represents a multi-scale deformable attention network, and the FFN(·) represents a feedforward network.

[0023] In one embodiment, the corner point coordinates are obtained according to the offset vector using a corner point regression network, and the processing flow is expressed as follows:

[0024] f output =MLP(f decoder )+Sig inv (R' point )

[0025] In the above formula, f decoder , R' point Respectively represent the output data of the decoder, i.e., the offset vector, and the sparse sampling point, Sig inv (·) indicates anti-sigmod change processing, f output Represents the coordinates of the corner point.

[0026] In one embodiment, the method comprises:

[0027] Constructing a network by parameter transformation according to the feature extraction network, feature aggregation network, encoder, decoder, corner regression network and regression network;

[0028] The visible light-infrared dual-modal image pair is input into the trained parameter transformation network to obtain corresponding transformation parameters.

[0029] In one embodiment, training the parameter transformation to obtain a network includes:

[0030] Acquire a dual-modal image pair training data set, wherein the dual-modal image pair training data set includes a plurality of training image pairs, each of the training image pairs includes a visible light image and a corresponding infrared image;

[0031] Performing preprocessing including size unification, grayscale change and pixel normalization on each training image in the dual-modal image training data set;

[0032] A transformation matrix is ​​constructed according to preset transformation parameters, and a preprocessed infrared image is spatially transformed by using a two-dimensional network and the transformation matrix;

[0033] The infrared image after spatial transformation and the corresponding visible light image are used to transform the parameters to obtain the network for training, and the transformation matrix is ​​used as the true value label.

[0034] The present application provides a dual-modal image registration device for a drone, the device comprising:

[0035] A dual-modal image pair data set acquisition module is used to acquire a real-time dual-modal image pair data set, wherein the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs captured by a drone and sorted by time;

[0036] A feature extraction module is used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair using a feature extraction network to obtain visible light multi-scale features and infrared multi-scale features;

[0037] The common feature fusion module is used to use a feature aggregation network to fuse the common features of the visible light feature and the infrared feature at the same scale, and obtain the common fusion features of the visible light and the infrared at the corresponding scale;

[0038] The encoding fusion module is used to use an encoder to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales to obtain multi-scale fusion features of visible light and multi-scale fusion features of infrared light;

[0039] A decoding module, configured to use a decoder constructed by a deformable convolutional network and sparse sampling to decode the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling point;

[0040] A transformation parameter obtaining module, used to obtain corner point coordinates according to the offset vector using a corner point regression network, and to obtain transformation parameters according to the corner point coordinates using a regression network;

[0041] The dual-modal image registration module is used to transform the infrared image in the visible light-infrared dual-modal image pair according to the transformation parameters to achieve registration with the corresponding visible light image.

[0042] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0043] Acquire a real-time dual-modal image pair data set, wherein the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs taken by a drone and ordered in time;

[0044] A feature extraction network is used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain visible light multi-scale features and infrared multi-scale features;

[0045] A feature aggregation network is used to fuse the common features of visible light features and infrared features at the same scale, respectively, to obtain the common fusion features of visible light and infrared at the corresponding scales.

[0046] An encoder is used to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales to obtain visible light multi-scale fusion features and infrared multi-scale fusion features;

[0047] A decoder constructed by a deformable convolutional network and sparse sampling is used to decode the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling point;

[0048] Using a corner point regression network to obtain corner point coordinates according to the offset vector, and using a regression network to obtain transformation parameters according to the corner point coordinates;

[0049] The infrared image in the visible light-infrared dual-modality image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible light image.

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0051] Acquire a real-time dual-modal image pair data set, wherein the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs taken by a drone and ordered in time;

[0052] A feature extraction network is used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain visible light multi-scale features and infrared multi-scale features;

[0053] A feature aggregation network is used to fuse the common features of visible light features and infrared features at the same scale, respectively, to obtain the common fusion features of visible light and infrared at the corresponding scales.

[0054] An encoder is used to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales to obtain visible light multi-scale fusion features and infrared multi-scale fusion features;

[0055] A decoder constructed by a deformable convolutional network and sparse sampling is used to decode the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling point;

[0056] Using a corner point regression network to obtain corner point coordinates according to the offset vector, and using a regression network to obtain transformation parameters according to the corner point coordinates;

[0057] The infrared image in the visible light-infrared dual-modality image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible light image.

[0058] The above-mentioned dual-modal image registration method, device, equipment and medium for unmanned aerial vehicles, after using a feature extraction network to perform multi-scale feature extraction on the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, use a feature aggregation network to fuse the common features between the visible light features and the infrared features at the same scale, and then use an encoder to fuse the common fusion features of the visible light and the common fusion features of the infrared at multiple scales, respectively, use a decoder constructed by a deformable convolutional network and sparse sampling to decode the output of the encoder to obtain an offset vector based on the sampling point, and finally use a corner regression network and a regression network to obtain the transformation parameters for registering the corresponding dual-modal image pair according to the offset vector. The present method can quickly and effectively perform online dual-modal registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of a flow chart of a dual-modal image registration method for a drone in one embodiment;

[0060] Figure 2 A schematic diagram of the structure of a feature extraction network in one embodiment;

[0061] Figure 3A schematic diagram of the structure of a feature aggregation network in one embodiment;

[0062] Figure 4 Schematic diagram of the structure of an encoder, a decoder and a corner regression network in one embodiment;

[0063] Figure 5 A schematic diagram of a training process of a network obtained by parameter transformation in one embodiment;

[0064] Figure 6 This is a schematic diagram of the effects of image registration using multiple traditional methods and this method in an experiment;

[0065] Figure 7 A schematic diagram showing the visualization of the heat map of each module in the network for an image pair obtained by parameter transformation in an experiment;

[0066] Figure 8 is a structural block diagram of a dual-modal image registration device for a drone in one embodiment;

[0067] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] Image registration is a basic task in image processing. Its core is to match images from different perspectives, different sensors, and different times based on the correspondence between images, that is, to solve the spatial variation relationship between images, so that points at the same spatial position in different images correspond one to one. There are two main methods for aligning visible light and infrared images, namely traditional methods and deep methods.

[0070] Among them, the traditional method extracts the feature point pairs of the image and calculates the change parameters of the image according to the feature relationship between the images, such as the SIFT algorithm, etc. At the same time, in order to reduce the error of the extracted feature point pairs, the RANSAC algorithm is used to remove abnormal feature point pairs. The depth-based image registration method has now become the mainstream.

[0071] Among them, the depth-based image registration method uses the CNN feature extraction network as the skeleton, and calculates the correlation between features through the extracted features, and then obtains the homography matrix between images. This method is a dense matching method that only considers the relationship between feature pairs, without considering the global feature relationship. LOFTR uses Transformer to model the feature relationship between features, and avoids being affected by abnormal feature pairs by considering the global feature relationship. However, LOFTR is a two-stage matching process, which is computationally intensive and inefficient.

[0072] Compared with visible image registration, there are currently three main challenges in dual-modal image registration. First, similar to the drone target detection task, the targets in the images captured from the drone's perspective are of different scales. Therefore, in the image registration task, it is crucial to fully extract multi-scale features for image representation. Second, heterogeneous modal images differ in features. Visible light images have rich texture features, and infrared images have clear contours. This is crucial to address the differences in heterogeneous modalities and effectively improve the accuracy of dual-modal image registration. Third, there is a lack of drone dual-modal registration datasets available for training and evaluation. These problems have led to the lack of a directly available drone dual-modal registration dataset, which has hindered the development of drone dual-modal registration algorithms.

[0073] In one embodiment, Figure 1 As shown, a dual-modal image registration method for a UAV is provided, comprising the following steps:

[0074] Step S100, obtaining a real-time dual-modal image pair data set, where the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs taken by a drone and sorted in time.

[0075] Step S110, using a feature extraction network to perform multi-scale feature extraction on the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain visible light multi-scale features and infrared multi-scale features.

[0076] Step S120 , using a feature aggregation network to fuse common features between the visible light features and the infrared features at the same scale, to obtain common fused features of visible light and common fused features of infrared at corresponding scales.

[0077] Step S130 , using an encoder to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales, respectively, to obtain multi-scale fusion features of visible light and multi-scale fusion features of infrared light.

[0078] Step S140, using a decoder constructed by a deformable convolutional network and sparse sampling, decoding is performed according to the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling point.

[0079] Step S150, using a corner point regression network to obtain corner point coordinates according to the offset vector, and using a regression network to obtain transformation parameters according to the corner point coordinates.

[0080] Step S160 , transforming the infrared image in the visible light-infrared dual-modality image pair according to the transformation parameters to achieve registration with the corresponding visible light image.

[0081] In this embodiment, a sequence of visible light-infrared dual-modal images acquired by a drone in real time is sequentially registered online to provide better front-end services for subsequent tasks such as target recognition.

[0082] In step S100, the images to be registered include multiple pairs, which are actually dual-module image pairs consisting of a group of visible light images and infrared images corresponding to each moment acquired by the drone in real time. When processing them, one image pair is processed at a time and processed sequentially. The operations of steps S110 to S160 are implemented for each dual-modal image pair to achieve online registration of the image pair.

[0083] Before performing step S110, the two images in the dual-modality image pair are preprocessed respectively, and the preprocessing process includes but is not limited to. . .

[0084] In step S110, two feature extraction networks are used to perform multi-scale feature extraction on the visible light image and the infrared image respectively to obtain features of multiple different scales.

[0085] In this embodiment, the feature extraction network adopts the ResNet network, whose structure is as follows: Figure 2 As shown in the figure, in order to extract multi-scale image features, the last connection layer in the original ResNet network is removed, and the outputs of the last three convolutional layers are used as output features. A convolution is used to change the features to a preset size, and finally, image features at three different scales are obtained for visible light images and infrared images respectively.

[0086] Specifically, the scales of the image features of three different scales are: 60*60*256, 30*30*256, 15*15*256, and the corresponding steps are 8, 16, and 32. The scales of the feature images are not limited to the given scales, but can also be other scales.

[0087] In this embodiment, since the visible light image and the infrared image are images of different modalities and have different features, and the images of these two modalities are quite different, two feature extraction networks are required here to extract visible light features and infrared features respectively.

[0088] In step S120, feature aggregation is performed on the visible light feature image and the infrared feature image at the same scale to obtain visible light common fusion features and infrared common fusion features after heterogeneous feature aggregation.

[0089] Due to the different imaging mechanisms of visible light and infrared, the image features extracted by the feature extraction network are also different. Bimodal image registration uses the correlation between image features for registration. Here, it is necessary to perform bimodal fusion of visible light-infrared features to enhance the common features in different modalities.

[0090] In this embodiment, multiple convolutions are used to implement a dual-modal fusion network. The specific network structure is as follows: Figure 3 Place.

[0091] Specifically, since multiple image features of different scales are extracted in step S110, in order to fuse the corresponding scale dual-modal information, it is necessary to perform modal information fusion on the visible light-infrared dual-modal image of each scale. The fusion process can be expressed as:

[0092] F C =Conv C (Cat((Conv rgb (F rgb [i]),Conv ir (F ir [i])))

[0093] In the above formula, F rgb [i],F ir [i] represents the visible light multi-scale features and infrared multi-scale features at the i-th scale, Conv(·) represents the convolution operation, and Cat(·) represents the dimension concatenation operation.

[0094] Next, for F C Perform dimension separation operation, expressed as:

[0095]

[0096] In the above formula, Split(·) represents the dimension separation operation, which converts the feature F obtained above into C Separate and obtain visible light image features and infrared image features

[0097] Furthermore, the visible light image features and infrared image features Do the following:

[0098]

[0099] In the above formula, convolutional layers Conv1 and Conv2 are used to respectively transform the visible light image features and infrared image features After the convolution operation, the Sigmod activation function is used for processing, targeting the visible light image features get For infrared image features get Here, the visible light and infrared features share two identical convolutions to enhance the common features in each modality.

[0100] Finally, the skip connection is used to obtain the fused bimodal features at the i-th scale, and the common features and characteristic features under different modes are fused to form the aggregated bimodal features at the i-th scale. The fusion process is expressed as:

[0101]

[0102] The feature maps at other scales are also aggregated in the same way. Therefore, in the feature aggregation network, three branch networks are included to process feature fusion at different scales respectively.

[0103] After passing through the feature aggregation network, visible light multi-scale fusion features and corresponding infrared multi-scale fusion features can be obtained for each scale.

[0104] After the bimodal fusion network, i.e., the feature aggregation network, the bimodal features after the interaction of modal information are obtained, and then the bimodal features are sent to the feature fusion network to model the feature correlation, thereby estimating the change parameters between images. Next, Transformer is used to construct the feature fusion network. Since the multi-scale features of the visible light-infrared bimodal image are used in this method, the feature splicing is b*(60*60+30*30+15*15)*256. If the traditional Transformer structure is used, the amount of calculation is huge, which will affect the operation speed of the network. Considering that sparse spatial sampling can reduce the redundant calculation of the network, it greatly improves the operation efficiency of the network and effectively solves the problem of redundant calculation of multi-scale images. At the same time, deformable convolution is used to determine the convolution parameters through learning, which can better capture the tiny features of the target and effectively improve the ability to model feature relationships.

[0105] In step S130, the encoder includes two encoder branches for processing visible light common fusion features and infrared common fusion features respectively. In each encoder branch, after the common fusion features of visible light or infrared common fusion features of multiple scales are spliced, the attention mechanism is used to enhance the spliced ​​features to obtain visible light multi-scale fusion features or infrared multi-scale fusion features. Finally, the visible light multi-scale fusion features and the infrared multi-scale fusion features are spliced ​​as the output of the encoder.

[0106] In this embodiment, the encoder consists of a multi-head self-attention module and a feedforward network. The feedforward network includes a two-layer multi-layer perceptron. GELU is used as the activation function. Layer normalization is used in the encoder to avoid overfitting. A residual structure is used in the multi-head self-attention module and the feedforward network module to facilitate the return of the gradient. Taking the processing of visible light features as an example, the encoder process can be expressed as follows:

[0107] Concatenate multi-scale image features:

[0108] F=Cat(F r ' gb [0],F r ' gb [1],F r ' gb [2])

[0109] In the above formula, F r ' gb [0],F r ' gb [1],F r ' gb [2] represent the common fusion features of visible light at different scales.

[0110] The concatenated features are enhanced using the attention mechanism:

[0111] F MSA =F+MSA(LN(F))

[0112] f rgb =MLP(LN(F MSA ))+F MSA

[0113] In the above formula, LN(·) represents the linear layer, MSA(·) represents the multi-head self-attention network, and MLPA(·) represents the multi-layer perception layer. Similarly, the infrared image feature f after the self-attention network can be obtained ir , where f rgb and f irThey respectively represent the visible light-infrared image features after passing through the encoder, namely, the visible light multi-scale fusion features and the infrared multi-scale fusion features.

[0114] Finally, the following formula is used to concatenate the optical multi-scale fusion features and the infrared multi-scale fusion features to obtain the output data of the encoder:

[0115] f encoder =Cat(f rgb ,f ir )

[0116] In step S140, since a multi-scale feature map is used in this method, this will cause a memory explosion. Here, a deformable convolutional network and sparse sampling are used for decoding. By using multi-scale deformable attention, the computational efficiency can be improved and the modeling capability of small target features can be improved.

[0117] In this embodiment, in the decoder: a multi-head self-attention network is used to perform autocorrelation on the query vector to obtain an autocorrelated query vector, and according to the autocorrelated query vector, the output features of the encoder, and the sparse sampling points, a multi-scale deformable attention network and a feedforward network are used in sequence to obtain an offset vector.

[0118] Specifically, the decoder consists of multiple components including a multi-head self-attention (MSA) network, a multi-scale deformable attention (MSDA) network, and a feed-forward network (FFN). In the multi-scale deformable attention module, the query can focus on the locations of all objects on the feature map, thereby predicting four robust corner coordinates.

[0119] First, self-attention is performed on the query, which is expressed as follows:

[0120] f q ' uery =F query +MSA(LN(F query ))

[0121] In the above formula, F query Represents the input query vector, which is obtained by self-attention and autocorrelation. q ' uery In this embodiment, the coefficient sampling point is obtained using the following formula:

[0122] R' point =Sig(Linear(R point ))

[0123] The multi-scale deformable attention module (MSDA) is a sparse sampling process, which requires determining the location of sparse sampling points. In the above formula, R pointRepresents the offset vector of the sparse sampling point in the x and y directions, with a size of n*2. Linear(·) indicates that the input of a linear layer is 256 and the output is 2. point is the initial offset vector, size is n*256.

[0124] The query vector is then decoded with the output features of the encoder, and the processing flow is expressed as:

[0125] F MSDA =f q ' uery +MSDA(LN(f q ' uery ),R' point ,f encoder ))

[0126] f decoder =FFN(LN(f MSDA ))+f MSDA

[0127] In the above formula, f q ' uery represents the query vector after autocorrelation, R' point represents sparse sampling points, f encoder represents the output feature of the encoder, F MSDA 、f MSDA represents the output data of the multi-scale deformable attention network, f decoder represents the output data of the decoder, i.e., the offset vector, LN(·) represents the linear layer, MSDA(·) represents the multi-scale deformable attention network, and FFN(·) represents the feedforward network.

[0128] Furthermore, the output of the decoder is an offset vector f based on the sampling point decoder , what is needed is the coordinates of the four corner points of the query vector. In order to obtain the x, y coordinates of the actual corner points, the offset vector needs to be processed and the coordinates of the sampling points are added. In step S150, the corner point regression network is used to obtain the corner point coordinates according to the offset vector. The processing flow is expressed as:

[0129] f output =MLP(f decoder )+Sig inv (R' point )

[0130] In the above formula, f decoder , R' point Respectively represent the output data of the decoder, namely the offset vector and the sparse sampling point, Sig inv (·) indicates anti-sigmod change processing, f output Represents the coordinates of the corner point.

[0131] Next, a regression network is used to regress the intersection coordinates to obtain the transformation parameters. In this embodiment, the regression network can use three linear layers as a parameter regression network. Since the value range of the transformation parameter is (-2 to 2), the Sigmod or ReLu activation function is not used here, and the transformation parameters are obtained by directly regressing the parameters.

[0132] In this embodiment, the structures of the encoder, decoder and corner regression network are as follows: Figure 4 shown.

[0133] In this embodiment, the above-mentioned feature extraction network, feature aggregation network, encoder, decoder, corner regression network and regression network can be used to construct a parameter transformation network, and the visible light-infrared dual-modal image pair is input into the trained parameter transformation network to directly obtain the corresponding transformation parameters. After the infrared image in the dual-modal image pair is transformed according to the transformation parameters, the registration with the visible light image can be achieved.

[0134] In this embodiment, when the training parameter transformation is used to obtain the network: first, a dual-modal image pair training data set is obtained, wherein the dual-modal image pair training data set includes multiple training image pairs, each training image pair includes a visible light image and a corresponding infrared image. Next, each training image in the dual-modal image pair training data set is preprocessed including size unification, grayscale change, and pixel normalization, a transformation matrix is ​​constructed according to preset transformation parameters, and the preprocessed infrared image is spatially transformed by using a two-dimensional network and a transformation matrix. Finally, the infrared image after spatial transformation and the corresponding visible light image are transformed to obtain a network for parameter transformation, and the transformation matrix is ​​used as a true value label.

[0135] Specifically, the images in the dual-modal image pair training data set are first preprocessed, and the data is augmented to enhance the generalization of the algorithm. The images are first uniformly scaled so that the infrared images and visible light images obtained have the same scale. Then, a grayscale change with a probability of 5% is performed on the image pairs to enhance the generalization of the algorithm for a variety of scenarios. The pixels of the visible-infrared image pair are normalized using the mean [0.485, 0.456, 0.406] and variance [0.229, 0.2240.225] obtained by statistics on the ImageNet dataset.

[0136] Specifically, when constructing a transformation matrix, appropriate change parameters including rotation, translation, scaling and other parameters are selected to construct a transformation matrix that meets the requirements.

[0137] Furthermore, when performing image transformation on the infrared image according to the transformation matrix, a two-dimensional grid of the same size as the infrared image is first constructed, the two-dimensional grid is spatially transformed using the transformation matrix, and the transformed two-dimensional grid is used to resample the infrared image to obtain a spatially transformed infrared image.

[0138] Furthermore, the transformed infrared image and the corresponding visible light image are used as transformation parameters to obtain the input of the network, and the corresponding transformation matrix is ​​used as the true value label to train the network until it converges, and the trained transformation parameters are obtained to obtain the network.

[0139] Specifically, when the network is trained using the transformed infrared image and the corresponding visible light image, the data processing process in the network is the same as the data processing flow in the above-mentioned steps S110 to S150. First, multi-scale visible light-infrared features are extracted, and the visible light-infrared branch features are aggregated to obtain a multi-scale bimodal aggregate feature map. The multi-scale bimodal aggregate feature map of the aggregated visible light-infrared branch features is input into the multi-scale encoder and decoder to obtain the correspondence between the features, regress the x, y coordinates of the four corner points, and finally obtain the change parameters of the infrared-visible light image pair. According to the change parameters of the infrared-visible light image pair, the image pair is spatially changed to obtain the registered infrared-visible light image pair. Repeat the above process and continuously infer the image sequence to achieve uninterrupted registration of the drone image pair.

[0140] like Figure 5 As shown in FIG. 1 , it is the overall process of training the network by transforming the parameters.

[0141] In this paper, the effectiveness of this method is also demonstrated through experiments. The experiment was tested on a data set collected by an optoelectronic pod experimental device carried by a UAV, and the robustness and registration accuracy of this method compared with similar methods were analyzed through experimental comparison.

[0142] The data set used in the experiment is actually collected by an optoelectronic pod mounted on a drone. The test sequence contains image sequence pairs collected by drones in various scenarios, including multiple common drone mission scenarios such as illumination changes, occlusions, shadows, and rapid movement of drones. By preprocessing the data set, the unregistered bimodal image pairs and the true values ​​of the change parameters can be obtained. The unregistered bimodal image pairs are input into the transformation parameter obtaining network in this method to obtain the change parameters estimated by the network. By comparing them with the true values, the registration effect of this method can be tested.

[0143] In the comparative experiments and analysis, the root mean square error (RMSE), average endpoint error (AEE) and average probability of correct key points (PCK) are used as evaluation indicators, where the PCK indicator measures the correct key points under 3, 5, and 7 pixel thresholds respectively. The visible light-infrared dual-modal image registration algorithm is compared with the mainstream registration algorithms (DHN, Cnngeo, unsupervise, PFNeT). The experimental results are shown in Table 1:

[0144] Table 1 Comparison results of algorithms

[0145]

[0146] Table 1 is a statistical comparison of the above algorithms. From Table 1, we can see that the proposed method is superior to other registration algorithms in terms of results. In terms of multiple indicators, the proposed method has shown a huge advantage. In general, in the measured data, the proposed method has achieved a good effect. The specific registration effect is as follows: Figure 6 shown.

[0147] Next, ablation experiments are performed on each module of the proposed method to verify the effectiveness of the algorithm. The ablation experiment results are shown in Table 2.

[0148] Table 2 Ablation experiments of the algorithms

[0149]

[0150] In this experiment, five experiments were conducted to conduct ablation experiment comparative analysis, including the baseline algorithm, the use of multi-scale features, the use of multi-scale deformable attention mechanism in both the encoder and decoder, the use of Transformer structure in the encoder and the decoder and the multi-scale deformable attention mechanism in the decoder, and the method in this paper.

[0151] The baseline algorithm is inspired by the target tracking algorithm. It can be divided into three parts: backbone network, feature fusion network and regression network. The feature fusion network consists of a classic encoder and decoder structure. As shown in Table 2, the RMSE, AEE, pck_7, pck_5, and pck_3 of the baseline are 21.19, 19.9, 48.1%, 33.1%, and 12.9% respectively.

[0152] First, multi-scale features (Multi) are utilized on the basis of the baseline. In this case, the network uses multi-scale features to calculate the transformation parameters between image pairs. As the input features increase, the training and reasoning of the network become more challenging, resulting in larger display memory requirements. The batch size is limited to 1, and the reasoning speed is 20fps. However, the performance is greatly improved compared to the baseline algorithm. RMSE is reduced by 8.73, AEE is reduced by 7.79, and PCK_7, PCK_5, and PCK_3 increase by 11.7%, 19.1%, and 20.3%, respectively. Experiments show that using multi-scale features can help the network obtain more detailed information feature representation and have better performance on small objects.

[0153] In order to reduce the memory explosion caused by multi-scale, we use deformable attention to replace the traditional transformer layer and sparse sampling to replace the traditional dense sampling (Deformable). At this time, the network's memory usage will decrease rapidly. Compared with Multi, RMSE is reduced by 1.43, AEE is reduced by 1.28, PCK_7 and PCK_5 are increased by 4.9% and 2.9% respectively, and PCK_3 is reduced by 4.9%. The results show that deformable attention layers and sparse sampling reduce feature redundancy and improve computational efficiency. Due to the use of sparse sampling in the encoder and decoder, some important features are ignored, resulting in some performance degradation.

[0154] In order to improve the network's ability to capture details, the classic Transformer layer is used as the encoder. Since the encoder is insensitive to the increase of features, the insensitivity to the increase of features will not cause the video memory to explode. Compared with the Deformable network, the network's ability to extract details has been improved. PCK_7 and PCK_5 increased by 4.9% and 2.9% respectively, and pck7 decreased by 1%. The results show that the use of densely sampled encoders helps capture detailed features of the image, but feature redundancy is inevitable.

[0155] Finally, the bimodal feature aggregation network is added to the network (method in this paper). The bimodal feature aggregation network can effectively solve the heterogeneous feature differences of the visible-infrared modalities. Using the bimodal feature aggregation network, the information of the visible-infrared modalities is exchanged, and the information of different modalities is aggregated to eliminate the differences in the bimodal features. Compared with the Encoder network, its performance has been greatly improved. RMSE is reduced by 7.71, AEE is reduced by 7.71, and PCK_7, PCK_5, and PCK_3 increase by 27.8%, 25.2%, and 27.0%, respectively. This shows that the bimodal feature aggregation network can aggregate visible-infrared modal information and effectively solve the problem of feature differences between different modes. This also highlights the importance of feature aggregation for images of different modalities.

[0156] In order to more clearly see the performance of the algorithm for dual-modal image registration, we use the image pairs in the dataset to visualize the algorithm. The specific results are as follows: Figure 7 As shown in the figure. The infrared image and visible light image in the figure represent the infrared and visible light image pairs in the dataset, respectively. It can be seen that the dataset contains image pairs in multiple scenes, and the visible light sensor can no longer work properly under dark light conditions. The visible light true value image represents the visible light true value image in the synthetic dataset, which is obtained by changing the visible light image with the changing parameters. The input image pair of the network is the infrared image and the visible light true value image pair, and the output is the changing parameter. The input infrared image is processed using the changing parameters output by the network to obtain the changed infrared image (network predicted image), and then the comparison diagrams of the changed image, the original visible light image and the true value visible light image are shown respectively (where the visible light image constitutes the R and G channels, and the changed infrared image constitutes the B channel). Through the comparison diagram, it can be seen that the parameters estimated by the network proposed in this paper can complete the registration of visible light-infrared image pairs with high quality. At the same time, the network proposed in this paper can effectively utilize the dual-modal image information and obtain a good effect under dark light conditions.

[0157] In the above-mentioned dual-modal image registration method for drones, a dual-modal image registration network based entirely on Transformer is constructed, and multi-scale image features are used to enhance the network's registration effect on detail features. At the same time, a dual-modal feature aggregation network is proposed to extract the common features of heterogeneous modal features and the characteristic features of each modality, and then combine and aggregate the dual-modal features. The deformable attention module is used for sparse sampling to reduce the parameter redundancy problem caused by multi-scale in the network and improve the training and reasoning speed of the network.

[0158] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0159] In one embodiment, Figure 8As shown, a dual-modal image registration device for unmanned aerial vehicles is provided, comprising: a dual-modal image pair data set acquisition module 200, a feature extraction module 210, a common feature fusion module 220, a coding fusion module 230, a decoding module 240, a transformation parameter acquisition module 250 and a dual-modal image registration module 260, wherein:

[0160] The dual-modal image pair data set acquisition module 200 is used to acquire a real-time dual-modal image pair data set, wherein the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs captured by a drone and sorted by time;

[0161] A feature extraction module 210 is used to perform multi-scale feature extraction on the visible light image and the infrared image in the visible light-infrared dual-modal image pair using a feature extraction network to obtain visible light multi-scale features and infrared multi-scale features;

[0162] The common feature fusion module 220 is used to use a feature aggregation network to fuse the common features of the visible light feature and the infrared feature at the same scale, so as to obtain the common fusion features of the visible light and the infrared at the corresponding scales;

[0163] The encoding fusion module 230 is used to use an encoder to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales to obtain a visible light multi-scale fusion feature and an infrared multi-scale fusion feature;

[0164] A decoding module 240 is used to use a decoder constructed by a deformable convolutional network and sparse sampling to decode the visible light multi-scale fusion feature and the infrared multi-scale fusion feature to obtain an offset vector based on the sampling point;

[0165] A transformation parameter obtaining module 250, configured to obtain corner point coordinates according to the offset vector using a corner point regression network, and obtain transformation parameters according to the corner point coordinates using a regression network;

[0166] The dual-modality image registration module 260 is used to transform the infrared image in the visible light-infrared dual-modality image pair according to the transformation parameters to achieve registration with the corresponding visible light image.

[0167] For the specific definition of the dual-modal image registration device for drones, please refer to the definition of the dual-modal image registration method for drones above, which will not be repeated here. Each module in the above-mentioned dual-modal image registration device for drones can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0168] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a dual-modal image registration method for a drone is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0169] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0170] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0171] Acquire a real-time dual-modal image pair data set, wherein the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs taken by a drone and ordered in time;

[0172] A feature extraction network is used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain visible light multi-scale features and infrared multi-scale features;

[0173] A feature aggregation network is used to fuse the common features of visible light features and infrared features at the same scale, respectively, to obtain the common fusion features of visible light and infrared features at the corresponding scales;

[0174] An encoder is used to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales to obtain visible light multi-scale fusion features and infrared multi-scale fusion features;

[0175] A decoder constructed by a deformable convolutional network and sparse sampling is used to decode the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling point;

[0176] Using a corner point regression network to obtain corner point coordinates according to the offset vector, and using a regression network to obtain transformation parameters according to the corner point coordinates;

[0177] The infrared image in the visible light-infrared dual-modality image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible light image.

[0178] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0179] Acquire a real-time dual-modal image pair data set, wherein the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs taken by a drone and ordered in time;

[0180] A feature extraction network is used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain visible light multi-scale features and infrared multi-scale features;

[0181] A feature aggregation network is used to fuse the common features of visible light features and infrared features at the same scale, respectively, to obtain the common fusion features of visible light and infrared features at the corresponding scales;

[0182] An encoder is used to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales to obtain visible light multi-scale fusion features and infrared multi-scale fusion features;

[0183] A decoder constructed by a deformable convolutional network and sparse sampling is used to decode the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling point;

[0184] Using a corner point regression network to obtain corner point coordinates according to the offset vector, and using a regression network to obtain transformation parameters according to the corner point coordinates;

[0185] The infrared image in the visible light-infrared dual-modality image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible light image.

[0186] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0187] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A dual-modal image registration method for unmanned aerial vehicles, characterized in that: The method comprises: Acquire a real-time dual-modal image pair data set, wherein the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs taken by a drone and ordered in time; A feature extraction network is used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair, respectively, to obtain visible light multi-scale features and infrared multi-scale features; A feature aggregation network is used to fuse the common features of visible light features and infrared features at the same scale, respectively, to obtain the common fusion features of visible light and infrared at the corresponding scales. An encoder is used to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales to obtain visible light multi-scale fusion features and infrared multi-scale fusion features; A decoder constructed by a deformable convolutional network and sparse sampling is used to decode the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling point; Using a corner point regression network to obtain corner point coordinates according to the offset vector, and using a regression network to obtain transformation parameters according to the corner point coordinates; The infrared image in the visible light-infrared dual-modality image pair is transformed according to the transformation parameters to achieve registration with the corresponding visible light image.

2. The dual-modality image registration method according to claim 1, characterized in that: The encoder includes two encoder branches for processing visible light common fusion features and infrared common fusion features respectively; In each of the encoder branches, after splicing the common fusion features of visible light or the common fusion features of infrared light at multiple scales, an attention mechanism is used to enhance the spliced ​​features to obtain a visible light multi-scale fusion feature or an infrared multi-scale fusion feature; The visible light multi-scale fusion features and the infrared multi-scale fusion features are concatenated and used as the output of the encoder.

3. The dual-modality image registration method according to claim 2, characterized in that: In the decoder: Use a multi-head self-attention network to autocorrelate the query vector to obtain the autocorrelated query vector; According to the query vector after autocorrelation, the output features of the encoder and the sparse sampling points, a multi-scale deformable attention network and a feedforward network are used in sequence to obtain the offset vector.

4. The dual-modality image registration method according to claim 3, characterized in that: In the decoder, the processing flow is expressed as: F MSDA =f q ' uery +MSDA(LN(f q ' uery ),R' point ,f encoder )) f decoder =FFN(LN(f MSDA ))+f MSDA In the above formula, f q ' uery represents the query vector after the autocorrelation, R' point represents the sparse sampling points, f encoder represents the output feature of the encoder, F MSDA 、f MSDA represents the output data of the multi-scale deformable attention network, f decoder represents the output data of the decoder, i.e., the offset vector, the LN(·) represents a linear layer, the MSDA(·) represents a multi-scale deformable attention network, and the FFN(·) represents a feedforward network.

5. The dual-modality image registration method according to claim 3, characterized in that: The corner point regression network is used to obtain the corner point coordinates according to the offset vector, and the processing flow is expressed as follows: f output =MLP(f decoder )+Sig inv (R' point ) In the above formula, f decoder , R' point Respectively represent the output data of the decoder, i.e., the offset vector, and the sparse sampling point, Sig inv (·) indicates anti-sigmod change processing, f output Represents the coordinates of the corner point.

6. The dual-modality image registration method according to any one of claims 1 to 5, characterized in that: The method comprises: Constructing a network by parameter transformation according to the feature extraction network, feature aggregation network, encoder, decoder, corner regression network and regression network; The visible light-infrared dual-modal image pair is input into the trained parameter transformation network to obtain corresponding transformation parameters.

7. The dual-modality image registration method according to claim 6, characterized in that: Training the parameter transformation to obtain a network includes: Acquire a dual-modal image pair training data set, wherein the dual-modal image pair training data set includes a plurality of training image pairs, each of the training image pairs includes a visible light image and a corresponding infrared image; Performing preprocessing including size unification, grayscale change and pixel normalization on each training image in the dual-modal image training data set; A transformation matrix is ​​constructed according to preset transformation parameters, and a preprocessed infrared image is spatially transformed by using a two-dimensional network and the transformation matrix; The infrared image after spatial transformation and the corresponding visible light image are used to transform the parameters to obtain the network for training, and the transformation matrix is ​​used as the true value label.

8. A dual-modal image registration device for unmanned aerial vehicles, characterized in that: The device comprises: A dual-modal image pair data set acquisition module is used to acquire a real-time dual-modal image pair data set, wherein the dual-modal image pair data set is a plurality of one-to-one corresponding visible light-infrared dual-modal image pairs captured by a drone and sorted by time; A feature extraction module is used to extract multi-scale features from the visible light image and the infrared image in the visible light-infrared dual-modal image pair using a feature extraction network to obtain visible light multi-scale features and infrared multi-scale features; The common feature fusion module is used to use a feature aggregation network to fuse the common features of the visible light feature and the infrared feature at the same scale, and obtain the common fusion features of the visible light and the infrared at the corresponding scale; The encoding fusion module is used to use an encoder to fuse the common fusion features of visible light and the common fusion features of infrared light at multiple scales to obtain multi-scale fusion features of visible light and multi-scale fusion features of infrared light; A decoding module, configured to use a decoder constructed by a deformable convolutional network and sparse sampling to decode the visible light multi-scale fusion features and the infrared multi-scale fusion features to obtain an offset vector based on the sampling point; A transformation parameter obtaining module, used to obtain corner point coordinates according to the offset vector using a corner point regression network, and to obtain transformation parameters according to the corner point coordinates using a regression network; The dual-modal image registration module is used to transform the infrared image in the visible light-infrared dual-modal image pair according to the transformation parameters to achieve registration with the corresponding visible light image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images