Method for registering visible and infrared images based on data simulation and consistent feature extraction

Abstract

The application discloses a visible-infrared image registration method based on data simulation and a consistent feature extraction module. The method relies on a visible-infrared image registration model based on consistent feature extraction to achieve. In the construction process of the model: a random combination of multiple data enhancement methods is used to realize data simulation, rich multi-modal information is added to the visible light image as model training data; a consistent feature extraction module is designed, the encoding and decoding module and the Transformer module of which realize feature enhancement and can effectively extract consistent feature information of multi-modal images; and the CFEM is inserted before feature extraction of the registration network, so that the registration model is obtained. Based on the method provided in the application, model training only relies on visible light data, and the registration capability can be generalized to infrared and visible light images, thereby providing an effective method for an infrared and visible light image registration scene with high generalization requirement.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a visible light infrared image registration method based on data simulation and consistent feature extraction. Background Technology

[0002] Infrared and visible light images have become the most commonly used cross-modal image pairs in industrial and everyday applications because they can provide complementary information about the same scene. However, due to factors such as imaging principles and the movement of imaging devices, infrared and visible light images are often not aligned. Furthermore, real-world applications typically involve differences in camera models, shooting scenes, and other factors, resulting in significant modal differences between the captured infrared and visible light images. Therefore, developing highly generalizable registration algorithms is crucial.

[0003] Current mainstream traditional infrared-visible light image registration algorithms extract image features manually before registration, and are generally only applicable to specific application scenarios. Given sufficient data, deep learning methods can learn a highly generalized model. However, to obtain a highly generalized model, it is necessary to cover as many scenarios as possible. Using various visible light and infrared camera models to capture images in various indoor and outdoor scenes and manually annotating the data requires significant manpower and resources, resulting in extremely high data acquisition costs. Currently, there is a large amount of visible light data available. One direction for solving this problem is to find a way to train a registration model using visible light images that can generalize to the infrared-visible light image domain. Summary of the Invention

[0004] To address the shortcomings of the aforementioned background technologies, this invention provides a visible-infrared image registration method based on data simulation and consistent feature extraction. This method relies on a visible-infrared image registration model based on consistent feature extraction. During model construction, a random combination of various data augmentation methods is used to achieve data simulation, adding rich multimodal information to the visible-infrared image as training data. This solves the problem of poor model generalization caused by data inconsistencies due to camera model, weather conditions, scene type, etc. The model's consistent feature extraction module (CFEM) combines an encoding / decoding module with a Transformer to extract consistent feature maps from the multimodal image. Inserting the CFEM before feature extraction in the registration network yields the registration model. The registration model of this invention can generalize to registration in the visible-infrared domain.

[0005] This invention is achieved using the following technical solution:

[0006] A visible-infrared image registration method based on data simulation and consistent feature extraction relies on a visible-infrared image registration model based on consistent feature extraction. The construction of the registration model specifically includes the following steps:

[0007] (1) Use data simulation to obtain multimodal training image pairs; the specific method is: each training image pair consists of two multimodal images obtained by processing a visible light image through a random combination data augmentation method.

[0008] (2) Design a CFEM, which includes an encoder, a decoder and a Transformer. The CFEM is used to extract consistent feature maps of training image pairs. The specific method is as follows: input the two images generated by data simulation into the encoder with shared weights to obtain the encoded feature vector. After the encoded feature vector is enhanced by the Transformer, it is restored to the dimension of the input image by the decoder to obtain the consistent feature map.

[0009] (3) The obtained consistent feature map is processed by the registration network to output the registration parameters. The parameters of CFEM and the registration network are updated simultaneously using the loss function. After multiple rounds of training, the trained registration model is obtained.

[0010] In the above technical solution, a further step is to use data simulation methods to obtain multimodal training image pairs. Specific steps include:

[0011] Data simulation is achieved through a random combination of various data augmentation methods, thereby adding rich multimodal information to visible light images. The data augmentation methods used include: contrast-limited adaptive histogram equalization, color dithering, channel shuffling, image inversion, channel removal, weather simulation, sharpening, brightness gradation, and light source simulation. Specifically, the random combination of data augmentation methods consists of a subset randomly selected from the set of all data augmentation methods; this subset may be empty.

[0012] Furthermore, consistent feature maps of training image pairs are extracted using CFEM, specifically including the following steps:

[0013] (1) Use an encoder to extract the encoded feature vector of the input image.

[0014] The training image pairs are derived from the source image I s and target image I T Composition. With I s For example, the feature vector is first extracted through a 7×7 convolutional layer, followed by a 3×3 convolutional layer. After the initial feature extraction, it is processed through three basic modules to obtain the source encoded feature vector. It contains high-level semantic information extracted from the input image; the basic module consists of a 2×2 convolution with a stride of 2 and a basic residual block, where the 2×2 convolution is used to achieve downsampling, and the basic residual block is used to achieve deep feature extraction after downsampling by the 2×2 convolution. Target image I T Similarly, the target encoded feature vector is obtained by extracting it through the same encoder.

[0015] (2) Use Transformer to enhance the features of the encoded feature vector

[0016] The Transformer used has a symmetric structure with two paths for simultaneously inputting the source encoded feature vectors. and target encoded feature vector It can enhance features based on the relationship between the two. The two paths are the same, each containing three stages: first, the self-attention stage; second, the cross-attention stage; and finally, the feedforward layer processing stage, which ultimately outputs the enhanced feature vector.

[0017] by Taking the relevant pathway as an example:

[0018] During the self-attention stage First, the query Q is obtained through three linear transformation layers. S Key K S Sum V S By utilizing the self-attention mechanism, a first-order enhanced feature vector can be obtained. The self-attention mechanism can be represented as:

[0019]

[0020] Where d is K s In the channel direction dimension, norm represents the normalization operation.

[0021] During the cross-attention phase, The query Q′ is also obtained through three linear transformation layers. S 、key K′ S Sum V′ S In cross-attention calculation, the key-value pairs used come from... K′ calculated from the path T The cross-attention mechanism is used to obtain the feature vector with secondary enhancement. Specifically, it can be represented as

[0022]

[0023] at last After processing by the feedforward layer, the source encoded feature vector with enhanced features can be obtained. Feedforward layer processing can be represented as

[0024]

[0025] Where linear represents linear operation and relu represents activation function operation.

[0026] The pathway used and Using the same structure and the same network parameters, the enhanced target encoding feature vector is calculated. In the cross-attention phase, the key-value pairs used come from... The path in question was calculated.

[0027] (3) Use a decoder to restore the enhanced feature vectors to the size of the input image.

[0028] The decoder will enhance the source encoded feature vector and the enhanced target encoded feature vector By restoring the original input image to its original dimensions, a consistent feature map I′ of the source image is obtained. S Consistent feature map I′ with the target image T The decoder consists of three cascaded basic modules and a 1×1 convolution. The enhanced target encoded feature vector passes through all the basic modules, and then a 1×1 convolution is used to reduce the channel dimension to 3, outputting a consistent feature map. Each basic module includes a bilinear interpolation upsampling layer and a basic residual block. The input to the first basic module of the decoder is the enhanced feature vector from the Transformer. The inputs to the other two basic modules are obtained by concatenating the outputs of the previous level decoder's basic modules and the outputs of the encoder's basic modules with the same output dimension along the channel dimension.

[0029] Furthermore, step (3) specifically includes:

[0030] For existing registration networks Its function is to extract features and estimate parameters from the input image pair to obtain the registration parameters D. Simply insert the CFEM into... This can be done before feature extraction. Simultaneously, the data simulation method described above is applied to the visible light data used for training to obtain simulation data, which is then input into the CFEM and... In the network, the registration parameters D are predicted, and the true values ​​of the registration parameters D are used to determine the parameters. gt The loss L can be calculated using a loss function, and then the network parameters can be updated using stochastic gradient descent. After multiple rounds of training, the model can be applied to the registration of infrared and visible light images. The formula for calculating the loss function is as follows:

[0031] L = |DD gt |,

[0032] The advantages of this invention are:

[0033] This invention proposes a visible-infrared image registration method based on data simulation and consistent feature extraction. By training the network using only a visible-infrared image dataset, the network can achieve visible-infrared image registration capabilities across multiple scenarios. The proposed data simulation method acquires sufficiently rich modal training data from visible-infrared images, thereby improving model performance. The proposed consistent feature extraction module (CFEM) effectively extracts consistent features from infrared-visible images with significant modal differences. Experiments demonstrate that the model exhibits excellent generalization ability, avoids the economic costs associated with acquiring large amounts of infrared-visible data, and solves the limitation of deep learning in the application scenarios of visible-infrared registration. Attached Figure Description

[0034] Figure 1 A training method for registration models of visible and infrared images.

[0035] Figure 2 The encoder and decoder structures for the consistent feature extraction module.

[0036] Figure 3 This is the Transformer structure for the consistent feature extraction module.

[0037] Figure 4 The images used for partial testing are infrared and visible light images to be registered, consistent feature maps extracted by CFEM, and registered visible light images predicted by the registration model. Detailed Implementation

[0038] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0039] This invention provides a visible-infrared image registration method based on data simulation and consistent feature extraction. The method relies on a visible-infrared image registration model based on consistent feature extraction. Figure 1 The training method for the registration model is shown below, with the specific steps as follows:

[0040] (1) First, input a visible light image I. Then, use data simulation to obtain multimodal training image pairs. Specifically, each training image pair consists of two multimodal images obtained by processing a visible light image using randomly combined data augmentation methods. These data augmentation methods include contrast-limited adaptive histogram equalization, color jittering, channel shuffling, image inversion, channel removal, weather simulation, sharpening, brightness gradation, and light source simulation. The randomly combined data augmentation methods are specifically composed of a subset randomly selected from the set of all data augmentation methods. This subset may be empty, allowing for random simulation and addition of multimodal information. Performing two data simulations on I yields two multimodal images: the source image I... s and target image I T , where I T Remain unchanged, I S A random perspective transformation was performed, and the true value of the corresponding registration parameters is D. gt This allows us to simulate multimodal images captured from different perspectives in real-world scenarios.

[0041] (2)I S and I T The corresponding source-encoded feature vectors are obtained by passing them through the encoder. and target encoded feature vector The encoded feature vector represents the high-level semantic information of the input image. This feature vector is then fed into the Transformer, where self-attention and cross-attention mechanisms are applied to enhance the features, resulting in an enhanced source-encoded feature vector. and target encoded feature vector The enhanced feature vector is input into the decoder, which restores the size of the input image to obtain a consistent feature map I′ of the source image. S Consistent feature map I′ with the target image T It contains consistent features from both images.

[0042] (3) Consistent feature maps are input into the registration network. The forward propagation is performed to predict the registration parameters D between the two images. The predicted registration parameters D and the true registration parameters D are then used to determine the final registration parameters. gt The loss L can be calculated based on the loss function, and then backpropagation can be performed using stochastic gradient descent to update the network parameters.

[0043] After multiple rounds of training, a well-trained registration model is obtained.

[0044] like Figure 2 The diagram shows the encoding and decoding structure in the consistent feature extraction module proposed in this invention.

[0045] For input image I SFirst, preliminary feature extraction is performed using 7×7 and 3×3 convolutions. Each convolutional layer includes a batch normalization layer and a ReLU activation layer, in addition to the convolution operation. Next, three basic modules consisting of downsampling convolutions and basic residual blocks are used. Each basic module downsamples the feature map by a factor of 2, thereby reducing the feature map dimension, ultimately yielding the source encoded feature vector. Similarly, the target encoding feature vector can be obtained.

[0046] The enhanced source-encoded feature vector is obtained after passing through the Transformer. and the enhanced target encoded feature vector The code then enters the decoder. The decoder first passes through three basic modules consisting of bilinear interpolation and basic residual blocks, undergoing three 2x upsampling passes. Then, a 1×1 convolution is used to increase the channel dimension to 3, outputting a consistent feature map. During decoder propagation, the input to the first basic module is the Transformer-enhanced feature vector. The inputs to the other two basic modules are obtained by concatenating the outputs of the previous level decoder's basic module and the outputs of the encoder's basic module (which has the same output dimension) along the channel dimension; this utilizes skip connections.

[0047] like Figure 3 The diagram shows the Transformer structure in the consistent feature extraction module proposed in this invention.

[0048] The source encoded feature vector after encoder encoding and target encoded feature vector Simultaneously input a Transformer with a symmetrical structure. and First, each path performs its own self-attention calculation. Then, cross-attention calculation is performed based on K obtained from another path. Finally, the enhanced features are obtained by processing through a feedforward layer with nonlinear transformation capabilities.

[0049] like Figure 4 The image shows the prediction performance of the registration model trained according to this invention.

[0050] The consistent feature maps extracted by CFEM are shown in rows 3 and 4. These consistent feature maps effectively overcome the modal inconsistencies of the input image and extract consistent features. The registration network predicts registered images on real visible infrared datasets in multiple scenes. With target image I T Basic registration indicates that the model trained by this invention has the ability to generalize from registration in the visible light domain to registration in the visible light infrared image domain.

[0051] The above are merely specific embodiments of the present invention and should not be construed as limiting the scope of the present invention. Equivalent variations made by those skilled in the art based on this invention, as well as changes well-known to those skilled in the art, should still fall within the scope of the present invention.

Claims

1. A visible-infrared image registration method based on data simulation and consistent feature extraction, characterized in that, The method relies on a visible light infrared image registration model based on consistent feature extraction. The construction of the registration model includes the following steps: (1) Use data simulation to obtain multimodal training image pairs. Specifically, each training image pair consists of two multimodal images obtained by processing a visible light image through a random combination of data augmentation methods. (2) Construct a consistent feature extraction module, which includes an encoder, a decoder, and a Transformer. The consistent feature extraction module is used to extract consistent feature maps of training image pairs. Specifically, the two images generated by data simulation are input into a weight-shared encoder to obtain encoded feature vectors, and the encoded feature vectors are enhanced using a Transformer. The enhanced feature vectors are decoded to restore the dimensions of the input image, resulting in a consistent feature map; (3) The obtained consistent feature map is processed by the registration network to output the registration parameters. The parameters of the consistent feature extraction module and the registration network are updated simultaneously using the loss function. After multiple rounds of training, the trained registration model is obtained. In step (2), the Transformer used adopts a symmetrical structure with two paths. The two paths are the same and each contains three stages: first, the self-attention stage; second, the cross-attention stage; and finally, the feedforward layer processing stage. The final output is the enhanced feature vector.

2. The visible light infrared image registration method based on data simulation and consistent feature extraction as described in claim 1, characterized in that, In step (1): The data augmentation methods used include: contrast-limited adaptive histogram equalization, color dithering, channel shuffling, image inversion, channel removal, weather simulation, sharpening, brightness gradation, and light source simulation. The data augmentation method of random combination is specifically composed of a subset randomly selected from the set formed by all data augmentation methods, and the selected subset may be empty.

3. The visible light infrared image registration method based on data simulation and consistent feature extraction as described in claim 1, characterized in that, In step (2), the two images are input into an encoder with shared weights to obtain encoded feature vectors. The specific steps are as follows: The training image pairs are derived from the source image I s and target image I T Composition; for I s and I T The following processing steps are performed: First, a 7×7 convolutional layer is applied, followed by a 3×3 convolutional layer for preliminary feature extraction. After preliminary feature extraction, the feature vector is processed through three basic modules to obtain the source encoded feature vector. and target encoded feature vector The basic module consists of a 2×2 convolution with a stride of 2 and a basic residual block. The 2×2 convolution is used to implement downsampling, and the basic residual block is used to implement deep feature extraction after downsampling by the 2×2 convolution.

4. The visible light infrared image registration method based on data simulation and consistent feature extraction as described in claim 3, characterized in that, In step (2), the feature enhancement of the encoded feature vector using Transformer is performed using the following method: The Transformer used has a symmetric structure with two paths for simultaneously inputting the source encoded feature vectors. and target encoded feature vector It can enhance features based on the relationship between the two; by Taking the relevant pathway as an example: During the self-attention stage First, the query Q is obtained through three linear transformation layers. S Key K S Sum V S A first-order enhanced feature vector is obtained using a self-attention mechanism. Specifically represented as Where d is K S In the channel direction dimension, norm represents the normalization operation; During the cross-attention phase, The query Q′ is also obtained through three linear transformation layers. S 、key K′ S Sum V′ S However, when calculating cross-attention, the key values ​​used come from... K′ calculated from the path T The cross-attention mechanism is used to obtain the feature vector with secondary enhancement. Specifically represented as at last After processing by the feedforward layer, the source-encoded feature vector with enhanced features is obtained. Feedforward layer processing is represented as Where linear represents linear transformation operation and relu represents activation function operation; The pathway used and Using the same structure and the same network parameters, the enhanced target encoding feature vector is calculated. In the cross-attention phase, the key-value pairs used come from... The path in question was calculated.

5. The visible light infrared image registration method based on data simulation and consistent feature extraction as described in claim 4, characterized in that, In step (2), the decoder is used to restore the enhanced feature vector to the dimension of the input image. The specific method is as follows: The decoder will enhance the source encoded feature vector and the enhanced target encoded feature vector By restoring the original input image to its original dimensions, a consistent feature map I′ of the source image is obtained. S Consistent feature map I′ with the target image T ; The decoder consists of three cascaded basic modules and a 1×1 convolution. The enhanced target encoded feature vector passes through all the basic modules and then uses a 1×1 convolution to change the channel dimension to 3, outputting a consistent feature map. Each basic module includes a bilinear interpolation upsampling layer and a basic residual block. The input to the first basic module of the decoder is the enhanced feature vector from the Transformer. The inputs to the other two basic modules of the decoder are obtained by concatenating the outputs of the basic modules of the previous level decoder and the outputs of the basic modules of the encoder with the same output dimension along the channel dimension.

6. The visible light infrared image registration method based on data simulation and consistent feature extraction as described in claim 1, characterized in that, Step (3) specifically involves: For any existing registration network Its function is to perform feature extraction and parameter estimation on the input image pair to obtain the registration parameters D; and to insert the aforementioned consistent feature extraction module into... Before the feature extraction module, the data simulation method described above is simultaneously applied to the visible light data used for training, and then input into the consistent feature extraction module and... In a cascaded network, the registration parameter D is predicted, and the true value of the registration parameter D is then used. gt The loss L is calculated using a loss function, and then the network parameters are updated using stochastic gradient descent. After multiple rounds of training, the registration model can be applied to the registration of infrared and visible light images. The loss function is: L=|D-D gt |。

Images