Visible light infrared image registration method based on data simulation and consistent feature extraction

Abstract

The invention discloses a visible light infrared image registration method based on data simulation and a consistent feature extraction module. The method is realized by depending on a consistent feature extraction-based visible light infrared image registration model, and in a model construction process, data simulation is realized by utilizing random combination of multiple data enhancement modes, and rich multi-modal information is added to a visible light image to serve as model training data; a consistent feature extraction module is designed, feature enhancement is achieved through an encoding and decoding module and a Transform module of the consistent feature extraction module, and consistent feature information of the multi-modal image can be effectively extracted; a registration model can be obtained by inserting the CFEM before feature extraction of a registration network. Based on the method provided by the invention, the registration capability can be generalized to the infrared and visible light images by model training only depending on visible light data, and an effective method is provided for an infrared and visible light image registration scene with a high generalization requirement.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a visible light infrared image registration method based on data simulation and consistent feature extraction. Background Art

[0002] Infrared and visible light images have become the most commonly used cross-modal image pairs in industrial applications and life scenarios because they can provide complementary information in the same scene. Due to factors such as imaging principles and image device movement, infrared and visible light images are usually not aligned, and real-world applications usually have differences in camera models, shooting scenes, etc., and the captured infrared and visible light images usually have large modality differences. Therefore, it is very important to develop a highly generalizable registration algorithm.

[0003] The current mainstream traditional infrared-visible light image registration algorithm extracts image features through manual methods and then registers them, which is usually only applicable to individual application scenarios. If enough data is provided, the deep learning method can learn a model with high generalization. However, in order to obtain a model with high generalization, it is necessary to cover as many scenes as possible. Using a variety of visible light and infrared camera models to shoot in various indoor and outdoor scenes and manually annotate the data requires a lot of manpower and material resources, and the cost of data acquisition is extremely high. There is currently a large amount of visible light data. How to use visible light images to train models and obtain a registration model that can be generalized to the infrared-visible light image domain is a direction to solve the problem. Summary of the invention

[0004] In view of the defects existing in the above-mentioned background technology, the present invention provides a visible light infrared image registration method based on data simulation and consistent feature extraction. The method relies on a visible light infrared image registration model based on consistent feature extraction. In the model construction process: data simulation is realized by using a random combination of multiple data enhancement methods, and rich multimodal information is added to the visible light image as model training data, which can solve the problem of poor model generalization caused by data inconsistency caused by camera models, weather conditions, scene types, etc. The consistent feature extraction module (Consistent Feature Extraction Module, CFEM) of the model adopts a combination of encoding and decoding modules and Transformer, which can extract consistent feature maps of multimodal images. The CFEM is inserted before the feature extraction of the registration network to obtain the registration model. The registration model of the present invention can be generalized to the registration in the visible light infrared domain.

[0005] The present invention is implemented by the following technical solutions:

[0006] A visible light infrared image registration method based on data simulation and consistent feature extraction is implemented by relying on a visible light infrared image registration model based on consistent feature extraction. The construction of the registration model specifically includes the following steps:

[0007] (1) A data simulation method is used to obtain multimodal training image pairs; the specific method is: each training image pair consists of two multimodal images obtained by processing a visible light image with a randomly combined data enhancement method.

[0008] (2) Designing a CFEM, wherein the CFEM includes an encoder, a decoder, and a Transformer; the CFEM is used to extract a consistent feature map of a training image pair, wherein the specific method is as follows: two images generated by data simulation are input into a weight-sharing encoder to obtain a coded feature vector, the obtained coded feature vector is enhanced by the Transformer and then restored to the dimension of the input image by the decoder to obtain a consistent feature map.

[0009] (3) The obtained consistent feature map is passed through the registration network to output the registration parameters, and the loss function is used to simultaneously update the parameters of the CFEM and the registration network. After multiple rounds of training, a trained registration model is obtained.

[0010] In the above technical solution, further, a data simulation method is used to obtain a multimodal training image pair, and the specific steps include:

[0011] Data simulation is achieved based on a random combination of multiple data enhancement methods, thereby adding rich multimodal information to the visible light image. The data enhancement methods used include: limited contrast adaptive histogram equalization, color jitter, channel shuffling, image inversion, channel removal, weather simulation, sharpening, brightness gradient and light source simulation. The randomly combined data enhancement method is specifically composed of a subset randomly taken from the set formed by all data enhancement methods, and the taken subset may be empty.

[0012] Furthermore, CFEM is used to extract consistent feature maps of training image pairs. The specific steps include:

[0013] (1) Use the encoder to extract the encoded feature vector of the input image

[0014] The training image pair consists of the source image I s and the target image I T Composition. s For example, it first passes through a 7×7 convolution layer, and then passes through a 3×3 convolution layer to achieve preliminary feature extraction. After preliminary feature extraction, it is processed by three basic modules to obtain the source coding feature vector It contains high-level semantic information extracted from the input image; the basic module consists of a 2×2 convolution with a stride of 2 and a basic residual block, where the 2×2 convolution is used to achieve downsampling and the basic residual block is used to achieve deep feature extraction after 2×2 convolution downsampling. Target image I T Similarly, the target encoding feature vector is extracted through the same encoder

[0015] (2) Use Transformer to enhance the encoded feature vector

[0016] The Transformer used has a symmetrical structure with two paths for simultaneously inputting the source encoding feature vector and the target encoding feature vector The two pathways are identical and consist of three stages: the self-attention stage, the cross-attention stage, and the feedforward layer processing stage, which ultimately outputs the enhanced feature vector.

[0017] by Take the path where you are as an example:

[0018] In the self-attention stage, First, the query Q is obtained through three linear transformation layers S , key K S Sum value V S . Using the self-attention mechanism, we can get an enhanced feature vector The self-attention mechanism can be expressed as

[0019]

[0020] Where d is K s In the channel dimension, norm represents the normalization operation.

[0021] In the cross-attention phase, Also after 3 linear transformation layers, we get the query Q′ S , key K′ S Sum value V′ S , when calculating cross attention, the key value used comes from K′ calculated from the path T , using the cross attention mechanism to obtain the secondary enhanced feature vector Specifically, it can be expressed as

[0022]

[0023] at last After the feedforward layer processing, the feature-enhanced source encoding feature vector can be obtained The feedforward layer processing can be expressed as

[0024]

[0025] Among them, linear represents linear operation, and relu represents activation function operation.

[0026] The channel adopts and The same structure and the same network parameters are used to calculate the enhanced target encoding feature vector Among them, in the cross-attention stage, the key values ​​used come from The path is calculated.

[0027] (3) Use the decoder to restore the enhanced feature vector to the size of the input image

[0028] The decoder converts the enhanced source encoding feature vector And the enhanced target encoding feature vector Restore to the size of the original input image and obtain the consistent feature map I′ of the source image S and the consistent feature map I′ of the target image T . The decoder consists of three cascaded basic modules and 1×1 convolutions. After the enhanced target encoded feature vector passes through all basic modules, a 1×1 convolution is used to change the channel dimension to 3, and a consistent feature map is output. Each basic module includes a bilinear interpolation upsampling layer and a basic residual block. The input of the first basic module of the decoder is the feature vector enhanced by the Transformer. The input of the other two basic modules of the decoder is the output of the basic module of the previous decoder and the output of the basic module of the encoder with the same output dimension, which is spliced ​​in the channel dimension.

[0029] Furthermore, the step (3) is specifically as follows:

[0030] For the existing registration network The role of is to extract features and estimate parameters of the input image pair to obtain the registration parameter D. Simply insert the CFEM into At the same time, the data simulation method is applied to the visible light data used for training to obtain simulation data, which is input into CFEM and In the network composed of gt , the loss L can be calculated through the loss function, and then the network parameters are updated using the stochastic gradient descent method. After multiple rounds of training, the model can be applied to the registration of infrared images and visible light images. The calculation formula of the loss function is:

[0031] L=|DD gt |,.

[0032] The advantages of the present invention are:

[0033] The present invention proposes a visible light infrared image registration method based on data simulation and consistent feature extraction. Only by using visible light image data sets for network training, the network can have the ability to register visible light infrared images in various scenarios. In the present invention, the proposed data simulation method can obtain training data with sufficient rich modalities based on visible light images, thereby improving the model effect; the proposed consistent feature extraction module CFEM can effectively extract consistent features from infrared visible light images with great modal differences. Experiments have shown that the model has good generalization, avoids the economic cost of acquiring a large amount of infrared visible light data, and solves the problem of scene application limitations of deep learning in the field of visible light infrared registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A model training method for visible light infrared image registration.

[0035] Figure 2 The encoder and decoder structures of the consistent feature extraction module.

[0036] Figure 3 Transformer structure for the consistent feature extraction module.

[0037] Figure 4 The infrared and visible light images to be registered for some tests, the consistent feature maps extracted by CFEM, and the registered visible light images predicted by the registration model. DETAILED DESCRIPTION

[0038] The specific implementation modes of the present invention are further described below in conjunction with the accompanying drawings.

[0039] The present invention provides a visible light infrared image registration method based on data simulation and consistent feature extraction, and the method relies on a visible light infrared image registration model based on consistent feature extraction. Figure 1 The training method of the registration model is shown, and the specific steps are as follows:

[0040] (1) First, input a visible light image I, and use a data simulation method to obtain a multimodal training image pair. The specific method is as follows: each training image pair consists of two multimodal images obtained by processing a visible light image with a randomly combined data enhancement method; the data enhancement method includes limited contrast adaptive histogram equalization, color jitter, channel shuffling, image inversion, channel removal, weather simulation, sharpening, brightness gradient, and light source simulation; the randomly combined data enhancement method is specifically composed of a subset randomly taken from the set formed by all data enhancement methods, and the taken subset may be empty, so as to realize random simulation to add multimodal information. Performing data simulation on I twice, two multimodal images can be obtained: source image I s and the target image I T , where I T Remain unchanged, S A random perspective transformation is performed, and the corresponding true value of the registration parameter is D gt , thereby simulating multimodal images taken from different perspectives in actual situations.

[0041] (2)I S and I T The corresponding source encoding feature vectors are obtained by passing through the encoder respectively and the target encoding feature vector The encoded feature vector represents the high-level semantic information of the input image. The feature vector is then input into the Transformer, and the self-attention mechanism and cross-attention mechanism are applied to enhance the feature to obtain the enhanced source encoded feature vector. and the target encoding feature vector The enhanced feature vector is input into the decoder and restored to the size of the input image to obtain the consistent feature map I′ of the source image. S and the consistent feature map I′ of the target image T , contains the consistent features of the two images.

[0042] (3) The consistent feature map is input into the registration network The forward propagation is performed in the final prediction to obtain the registration parameter D of the two images. gt , the loss L can be calculated according to the loss function, and then the network parameters can be updated by back propagation according to the stochastic gradient descent method.

[0043] After multiple rounds of training, a trained registration model is obtained.

[0044] like Figure 2 The figure shows the encoding and decoding structure in the consistent feature extraction module proposed in the present invention.

[0045] For the input image I SFirst, the initial feature extraction is performed through 7×7 convolution and 3×3 convolution. In each convolution layer, in addition to the convolution operation, it also includes a batch normalization layer and a Relu activation layer. Then, after three basic modules consisting of downsampling convolution and basic residual blocks, each basic module downsamples the feature map by 2 times, thereby reducing the dimension of the feature map, and finally obtains the source coding feature vector Similarly, the target encoding feature vector can be obtained

[0046] After the Transformer, the enhanced source encoding feature vector is obtained And the enhanced target encoding feature vector Enter the decoder. The decoder first passes through three basic modules consisting of bilinear interpolation and basic residual blocks, performs three times of 2-fold upsampling, and then uses a 1×1 convolution to change the channel dimension to 3, and outputs a consistent feature map. During the decoder propagation process, the first basic module input of the decoder is the feature vector enhanced by the Transformer, and the input of the other two basic modules of the decoder is the output of the basic module of the previous decoder and the output of the basic module of the encoder with the same output dimension, which is the application of jump connection.

[0047] like Figure 3 The figure shows the Transformer structure in the consistent feature extraction module proposed in the present invention.

[0048] Source encoded feature vector encoded by the encoder and the target encoding feature vector At the same time, input the Transformer with a symmetrical structure, and First, each performs self-attention calculation, then performs cross-attention calculation based on K calculated by another pathway, and finally obtains enhanced features through a feedforward layer with nonlinear transformation capability.

[0049] like Figure 4 Shown is the prediction effect of the registration model trained by the present invention.

[0050] The consistent feature maps extracted by CFEM are shown in the 3rd and 4th rows. The obtained consistent feature maps have better overcome the inconsistency of the input image in terms of modality and extracted consistent features. The registered images predicted by the registration network on the real visible infrared dataset in multiple scenes With the target image I T The basic registration indicates that the model trained by the present invention has the ability to generalize from the registration in the visible light domain to the registration in the visible light infrared image domain.

[0051] The above are only specific implementation methods of the present invention, which cannot be used to limit the scope of the present invention. Equivalent changes made by ordinary technicians in this technical field based on this creation, as well as changes known to technicians in this field, should still fall within the scope of the present invention.

Claims

1. A visible light infrared image registration method based on data simulation and consistent feature extraction, characterized in that: The method relies on a visible light infrared image registration model based on consistent feature extraction, and the construction of the registration model includes the following steps: (1) Using a data simulation method to obtain multimodal training image pairs, the specific method is: each training image pair consists of two multimodal images obtained by processing a visible light image with a randomly combined data augmentation method; (2) constructing a consistent feature extraction module, the consistent feature extraction module comprising an encoder, a decoder and a Transformer; the consistent feature extraction module is used to extract a consistent feature map of a training image pair, and the specific method is as follows: inputting two images generated by data simulation into a weight-sharing encoder to obtain an encoded feature vector, and using a Transformer to perform feature enhancement on the encoded feature vector, The enhanced feature vector is restored to the dimension of the input image through the decoder to obtain a consistent feature map; (3) The obtained consistent feature map is passed through the registration network to output the registration parameters. The loss function is used to simultaneously update the parameters of the consistent feature extraction module and the registration network. After multiple rounds of training, a trained registration model is obtained.

2. A visible light infrared image registration method based on data simulation and consistent feature extraction as claimed in claim 1, characterized in that: In the step (1): The data augmentation methods used include: contrast-constrained adaptive histogram equalization, color jittering, channel shuffling, image inversion, channel removal, weather simulation, sharpening, brightness gradient, and light source simulation; The randomly combined data enhancement methods are specifically composed of a subset randomly taken out from a set formed by all data enhancement methods, and the taken out subset may be empty.

3. A visible light infrared image registration method based on data simulation and consistent feature extraction as claimed in claim 1, characterized in that: In step (2), the two images are input into a weight-shared encoder to obtain an encoded feature vector, and the specific steps are as follows: The training image pair consists of the source image I s and the target image I T Composition; s and I T The following processing is performed respectively: first, it passes through a 7×7 convolution layer, and then a 3×3 convolution layer to achieve preliminary feature extraction; after preliminary feature extraction, it passes through 3 basic modules to obtain the source coding feature vector and the target encoding feature vector The basic module consists of a 2×2 convolution with a stride of 2 and a basic residual block, where the 2×2 convolution is used to achieve downsampling and the basic residual block is used to achieve deep feature extraction after 2×2 convolution downsampling.

4. A visible light infrared image registration method based on data simulation and consistent feature extraction as claimed in claim 3, characterized in that: In step (2), the feature enhancement of the encoded feature vector is performed using Transformer, and the specific method is as follows: The Transformer used has a symmetrical structure with two paths for simultaneously inputting the source encoding feature vector and the target encoding feature vector The features can be enhanced based on the connection between the two. The two pathways are the same, both of which contain three stages: the first is the self-attention stage, the second is the cross-attention stage, and the last is the feedforward layer processing stage, and the final output is the enhanced feature vector. by Take the path where you are as an example: In the self-attention stage, First, the query Q is obtained through three linear transformation layers S , key K S Sum value V S ; Use the self-attention mechanism to obtain an enhanced feature vector Specifically expressed as Where d is K S In the channel dimension, norm represents the normalization operation; In the cross-attention phase, Also after 3 linear transformation layers, we get the query Q ′ S , key K ′ S Sum value V S ′ , but in the cross attention calculation, the key value used comes from K calculated from the path ′ T , using the cross attention mechanism to obtain the secondary enhanced feature vector Specifically expressed as at last After the feedforward layer processing, the source encoding feature vector with feature enhancement is obtained The feed-forward layer processing is expressed as Where linear represents the linear transformation operation, and relu represents the activation function operation; The channel adopts and The same structure and the same network parameters are used to calculate the enhanced target encoding feature vector Among them, in the cross-attention stage, the key values ​​used come from The path is calculated.

5. A visible light infrared image registration method based on data simulation and consistent feature extraction as claimed in claim 4, characterized in that: In step (2), the decoder is used to restore the enhanced feature vector to the dimension of the input image. The specific method is: The decoder converts the enhanced source encoding feature vector And the enhanced target encoding feature vector Restore to the size of the original input image and obtain the consistent feature map I of the source image ′ S and the consistent feature map I of the target image ′ T ; The decoder consists of three cascaded basic modules and 1×1 convolutions. After the enhanced target encoded feature vector passes through all basic modules, a 1×1 convolution is used to change the channel dimension to 3, and a consistent feature map is output; each basic module includes a bilinear interpolation upsampling layer and a basic residual block; the first basic module input of the decoder is the Transformer enhanced feature vector, and the input of the other two basic modules of the decoder is the output of the previous decoder basic module and the output of the encoder basic module with the same output dimension, which is spliced ​​in the channel dimension.

6. The visible light infrared image registration method based on data simulation and consistent feature extraction as claimed in claim 1, characterized in that: The step (3) is specifically: For any existing registration network The function of is to extract features and estimate parameters of the input image pair to obtain the registration parameter D; insert the consistent feature extraction module into Before the feature extraction module, the data simulation method is applied to the visible light data used for training, and input into the consistent feature extraction module and In the cascade network, the registration parameter D is predicted, and the true value D of the registration parameter is gt , the loss L is calculated through the loss function, and then the network parameters are updated using the stochastic gradient descent method. After multiple rounds of training, the registration model can be applied to the registration of infrared images and visible light images; the loss function is: L=|D-D gt |。

Images