Bidirectional registration method for infrared and visible light images

Abstract

The invention belongs to the field of computer vision and image processing, and particularly relates to an infrared and visible light image bidirectional registration method, which comprises the following steps of: taking a source image pair and a deformed image pair as input images, and predicting deformation fields from an infrared light image to a visible light image and from the visible light image to the infrared light image by using an infrared and visible light image bidirectional registration network; therefore, the registration of the infrared light image and the visible light image is realized. In the registration process, semantic features of an input image are extracted, and part of the semantic features are used as implicit features; according to the method, cross-modal features and same-modal features for learning are obtained by extracting common features of an input image, and a multi-scale similarity matrix is constructed on the basis of the cross-modal features and the same-modal features so as to be used for querying common features of an infrared light image and a visible light image in forward deformation field prediction and reverse deformation field prediction, so that multi-modal collaborative supervision is realized, and the learning efficiency is improved. And the stability and the accuracy of registration of the infrared light image and the visible light image are enhanced.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and image processing, and in particular relates to a two-way registration method for infrared and visible light images. Background Art

[0002] With the rapid development of computer vision and image processing technology, infrared and visible light registration technology has attracted much attention. The purpose of this technology is to construct an accurate pixel mapping relationship between the source image and the target image, make full use of the complementary information of the two modal images, and improve the performance of downstream tasks. Visible light images are widely used due to their rich texture information, but their imaging quality is largely restricted by ambient lighting conditions. In contrast, infrared images can provide thermal radiation information that is not affected by changes in lighting and show the essential characteristics of the scene. To effectively combine these two types of information, the first challenge is to accurately register images of different modalities.

[0003] Although significant achievements have been made in the field of single-modality image registration, when it is applied to multi-modality images such as infrared and visible light, the registration accuracy drops drastically. This is mainly due to the inherent differences between different image modalities and the one-way mapping method used in single-modality registration technology. This strategy is effective in single-modality, but when processing multi-modality images, it becomes a challenge to effectively utilize the common information of different modalities, which in turn affects the stability of registration. Summary of the invention

[0004] The purpose of the present invention is to provide a method for bidirectional registration of infrared and visible light images to solve the problems of low precision and poor stability of multimodal image registration.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A method for bidirectional registration of infrared and visible light images comprises the following steps:

[0007] Step 1, obtaining and processing training samples, the training samples include source image pairs, deformed image pairs and true deformation fields, the deformed image pairs and true deformation fields are both obtained by transforming the source image pairs; the source image pairs are composed of visible light images and infrared light images of the same scene, and the true deformation fields include forward deformation fields and reverse deformation fields generated during the transformation of the source image pairs;

[0008] Step 2: construct an infrared and visible light image bidirectional registration network based on the source image pair and the deformed image pair to register the infrared image and the visible light image; the infrared and visible light image bidirectional registration network includes a common feature encoder, a semantic encoder, and a bidirectional iteration module, and the registration implementation method includes:

[0009] 2.1. Based on infrared images and visible light images, a common feature encoder is used to extract features F for learning cross-modal and homomodal 1 and F 2 , use the semantic encoder to extract semantic features and divide the semantic features into two parts, one part is still the semantic feature H 1 and H 2 , and the other part is used as the initial hidden state and

[0010] 2.2. Using inner product to calculate feature F 1 and F 2 The cosine similarity between them is F 1 and F 2 The similarity matrix between 1 and F 2 The similarity matrix between them is average pooled to obtain a multi-scale similarity matrix for bidirectional query;

[0011] 2.3. Set the predicted deformation field The initial value of is 0, and the predicted deformation field is used to query the common features related to the multi-scale pixels in the multi-scale similarity matrix. The common features, semantic features, predicted deformation field, and implicit features are used as inputs, and a bidirectional iterative module is used to perform N times of forward deformation field prediction and reverse deformation field prediction between the infrared image and the visible light image to complete the registration of the infrared image and the visible light image.

[0012] Step 3: Set the loss function Use loss function to train and optimize the infrared and visible light image bidirectional registration network; loss function The calculation formula is as follows:

[0013]

[0014]

[0015] in, represents the endpoint loss, represents the bidirectional consistency loss, represents feature similarity loss, γ and β both represent weights, represents the true forward deformation field, represents the true inverse deformation field, represents the predicted i-th forward deformation field, represents the predicted i-th inverse deformation field, represents the two-dimensional grid coordinates, Indicates interpolation sampling at the corresponding coordinates, symbol ‖·‖ 1 Indicates the calculation of the mean absolute error;

[0016] Step 4: Input the infrared image and visible light image to be registered into the trained infrared and visible light image bidirectional registration network to complete the registration of the visible light image and the infrared image.

[0017] Furthermore, the method for generating a forward deformation field and a reverse deformation field during the transformation of the source image pair in step 1 includes:

[0018] 1.1. Perform a rigid transformation on the source image pair, and calculate the affine transformation matrix of the source image according to the transformation parameters. The calculation formula of the affine transformation matrix is:

[0019]

[0020] Where M represents the affine transformation matrix, θ represents the rotation angle, s represents the scaling factor, and t x and t y Respectively represent the pixel translation distance in the x and y directions of the image;

[0021] 1.2. Introduce random deformation field into the affine transformation matrix to obtain the forward deformation field ,

[0022]

[0023] in, represents the forward deformation field, represents the two-dimensional grid coordinates, φ rand According to Δ x ,Δ y ∈[-1,1] and standard deviation σ∈[16,32];

[0024] 1.3. Based on the forward deformation field, the reverse deformation field is calculated using the interpolation sampling method

[0025] Furthermore, the deformed image pair is generated by sampling the source image pair based on the forward deformation field obtained in step 1.2.

[0026] Furthermore, the implementation method of step 2.3 includes:

[0027] 2.3.1. Define the query range Δφ and add the defined query range to the forward deformation field through the broadcast mechanism and the inverse deformation field

[0028] 2.3.2. Assume the predicted deformation field The initial value of is 0, and the predicted deformation field is used to interpolate and search in the multi-scale similarity matrix to obtain the common features related to multi-scale pixels;

[0029] 2.3.3. Taking the common features, semantic features, predicted deformation fields, and implicit features related to multi-scale pixels as input, and concatenating the input features of the forward deformation field and the input features of the reverse deformation field, using a bidirectional iterative algorithm to perform N iterations to optimize the forward deformation field and the reverse deformation field, and obtaining N predicted forward deformation fields and predicted reverse deformation fields;

[0030] 2.3.4. Concatenate N predicted forward deformation fields to form a forward deformation field sequence Splice N predicted reverse deformation fields to form a reverse deformation field sequence

[0031] 2.3.5. Forward deformation field sequence With the inverse deformation field sequence By acting on the infrared image and visible light image to be registered, the registration of the infrared image and the visible light image can be achieved.

[0032] Furthermore, in step 2.3.3, the N forward deformation fields and the reverse deformation fields are predicted using the parallel computing method of the GPU.

[0033] The present invention provides a method for bidirectional registration of infrared and visible light images. By constructing a bidirectional registration network of infrared and visible light images, bidirectional registration of infrared images to visible light images and visible light images to infrared images is realized. By extracting the semantic features of the input image and using some features as implicit features, the expressiveness of the features is optimized; at the same time, since the common features of the input image are extracted, cross-modal features and homomodal features for learning are obtained, and a multi-scale similarity matrix is ​​constructed based on them; based on the multi-scale similarity matrix, the ability to effectively query common features in forward and reverse deformation field prediction is realized, thereby realizing multi-modal collaborative supervision and significantly improving the stability and accuracy of the registration of infrared images and visible light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the method for bidirectional registration of infrared and visible light images of an embodiment;

[0035] Figure 2 Schematic diagram of the transformation between infrared light image and visible light image in the embodiment;

[0036] Figure 3 A diagram of a network structure for bidirectional registration of infrared and visible light images constructed for the embodiment;

[0037] Figure 4 4 is a multimodal collaborative supervision framework diagram of an embodiment;

[0038] Figure 5 It is a schematic diagram of the similarity matrix result of the embodiment;

[0039] Figure 6 Schematic diagram of the change in registration accuracy during iterative optimization of the embodiment. DETAILED DESCRIPTION

[0040] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0041] like Figure 1 As shown, the present embodiment provides a method for bidirectional registration of infrared and visible light images, comprising the following steps:

[0042] Step 1, obtain and process training samples, the training samples include source image pairs, deformed image pairs and true deformation fields, the deformed image pairs and true deformation fields are both obtained by transforming the source image pairs; the source image pairs are composed of visible light images and infrared light images of the same scene, and the true deformation field includes the forward deformation field and the reverse deformation field generated during the transformation of the source image pairs.

[0043] In this embodiment, the training samples and test samples are obtained from the FLIR, MSRS, and RoadScene datasets. The obtained training samples have a total of 5911 pairs of images, of which 4627 pairs of images are obtained from the FLIR dataset, 1103 pairs of images are obtained from the MSRS dataset, and 181 pairs of images are obtained from the RoadScene dataset. The obtained test set has a total of 896 pairs of images, of which 515 pairs of images are obtained from the FLIR dataset, 361 pairs of images are obtained from the MSRS test set, and 20 pairs of images are obtained from the RoadScene test set.

[0044] The forward deformation field and the reverse deformation field calculation method generated during the source image transformation process are as follows: Figure 2 As shown, including:

[0045] 1.1. Perform rigid transformation on the infrared light source image and the visible light source image respectively, and calculate the affine transformation matrix of the infrared light source image and the visible light source image according to the transformation parameters. The calculation formula of the affine transformation matrix is:

[0046]

[0047] Among them, M represents the affine transformation matrix, θ represents the rotation angle, s represents the scaling ratio, and t x and t y Respectively represent the pixel translation distance in the x and y directions of the image. In this embodiment, the rotation angle θ∈[-10°,10°], the scaling ratio s∈[0.8,1.2], and the pixel translation distance t x ,t y ∈[-0.05,0.5]

[0048] 1.2. Introduce random deformation field into the affine transformation matrix to obtain the forward deformation field ,

[0049]

[0050] in, represents the forward deformation field, represents the two-dimensional grid coordinates, φ rand According to Δ x ,Δ y ∈[-1,1] and standard deviation σ∈[16,32];

[0051] 1.3. Based on the forward deformation field, the reverse deformation field is calculated using the interpolation sampling method

[0052]

[0053] in, represents the forward deformation field; Represents a 2D grid of coordinates for an arbitrary image.

[0054] The method for generating the transformed image is:

[0055] Using the forward transformation field For the infrared light source image I ir With visible light source image I vi Sampling is performed to generate the deformed image I i ' r and I v ' i , where I ir ,I vi ,I i ' r ,I v ' i ∈R 3×H×W .

[0056] Step 2: construct an infrared and visible light image bidirectional registration network based on the source image pair and the deformed image pair; the infrared and visible light image bidirectional registration network constructed in this embodiment adopts a multimodal collaborative supervision method to realize the registration of infrared and visible light images. The infrared and visible light image bidirectional registration network structure is as follows: Figure 3 As shown in , it includes two common feature encoders, two semantic encoders, a multi-scale similarity matrix, and a bidirectional iterative module. The registration process is as follows Figure 4 As shown below:

[0057] 2.1. Based on infrared images and visible light images, a common feature encoder is used to extract features F for learning cross-modal and homomodal 1 and F 2 , semantic features are extracted using a semantic encoder.

[0058] The two common feature encoders of this embodiment are both composed of the first three blocks of the ResNet18 structure; the first block of the common feature encoder is a convolution layer, whose convolution kernel size is 7×7, stride is 2, input channels are 3, and output channels are 64; the second block is composed of two 3×3 residual blocks, whose output channels are 64; the third block is also composed of two 3×3 residual blocks, whose output channels are 128. The two semantic feature encoders are also composed of the first three blocks of the ResNet18 structure; the difference from the shared feature encoder is that the output channel of the second block of the semantic feature encoder is 128, and the output channel of the third block is 256.

[0059] The feature F 1 and F 2 The extraction method is:

[0060] Using common feature encoder The common features F of the source image pair and the deformed image pair are extracted according to the following formula ir , F′ ir , F vi , F′ vi :

[0061]

[0062] Among them, I ir represents infrared image, I vi Represents the visible light image, I′ ir represents the infrared light deformation image, I′ vi represents the visible light deformation image, I ir ,I vi ,i i ' r ,I v ' i ∈R 3×H×W ;

[0063] For common features F ir ,F i ' r ,F vi ,F v ' i Concatenate to obtain the learning feature F for learning cross-modality and same-modality 1 and F 2 ,F 1 ,F 2 ∈R 4×D×H / 8×W / 8 , D = 128 is the feature encoding dimension, F 1 and F 2 The internal features correspond one to one.

[0064] F1 =cat(F ir ,F vi ,F ir , F vi )

[0065] F 2 =cat(F′ vi ,F′ ir , F′ ir ,F′ vi )

[0066] Among them, F ir represents the shared features of infrared light source images, F vi represents the shared features of visible light source images, F i ' r represents the shared features of infrared light deformation images, F v ' i Representing shared features of visible light deformable images.

[0067] The method for extracting the semantic features is:

[0068] Using semantic feature encoder The semantic features of the source image pair and the deformed image pair are extracted according to the following formula:

[0069]

[0070] Among them, H ir is the semantic feature of the infrared light source image, H i ' r is the semantic feature of infrared light deformation image, H vi is the semantic feature of the visible light source image, H v ' i is the semantic feature of the visible light deformed image; here, H ir , H′ ir , H vi , H′ vi ∈R 2D ×H / 8×Wf8 , the semantic features are divided into two parts, one part is still the semantic feature H 1 and H 2 , and the other part is used as the initial hidden state and

[0071] The semantic feature H 1 and H 2 The expression is as follows:

[0072] H 1 =cat(H ir [:D],H vi[:D],H ir [:D],H vi [:D])

[0073] H 2 =cat(H′ vi [:D],H′ ir [:D],H′ ir [:D],H′ vi [:D])

[0074] Initial hidden state and The expression is as follows:

[0075]

[0076] Therefore, the size of semantic features and initial latent features is 4×D×H / 8×W / 8.

[0077] 2.2. Using inner product to calculate feature F 1 and F 2 The cosine similarity between them is F 1 and F 2 The similarity matrix between 1 and F 2 The similarity matrix between them is averaged and pooled to obtain a multi-scale similarity matrix for bidirectional query; the expression of the multi-scale similarity matrix is:

[0078]

[0079] Among them, C 0 represents the similarity matrix, represents the similarity matrix between the infrared light source image and the visible light deformed image, represents the similarity matrix between the visible light source image and the infrared light deformed image, represents the similarity matrix between the infrared light source image and the infrared light deformation image, Represents the similarity matrix between the visible light source image and the visible light deformed image.

[0080] This embodiment has the common feature F ir and F′ vi As an example, construct a similarity matrix Taking the similarity matrix as an example, the construction process of the multi-scale similarity matrix is ​​described in detail: The calculation formula at the coordinates [i,j,k,l] is as follows:

[0081]

[0082] Among them, (F ir )dij represents the features of the infrared image at coordinates [d,i,j], (F′ vi ) dkl Represents the features of the visible light deformed image at coordinates [d, k, l].

[0083] Eigenvectors of the similarity matrix The first two dimensions represent the characteristic size of the infrared image, and the last two dimensions represent the characteristic size of the visible light image. The multi-scale similarity matrix is ​​obtained by repeating average pooling. The average pooling is completed by an average pooling layer with a pooling kernel size and a stride of 2. When the average pooling is applied to the last two dimensions, the feature size of the infrared image remains unchanged, while the size of the visible light feature image becomes smaller, thus achieving the I-based ir Query I′ vi The similarity matrix C ir→vi′ Construction; When average pooling is applied to the first two dimensions, the feature size of the infrared image is small, and the feature size of the visible light image remains unchanged, achieving the I′-based vi Query I ir The similarity matrix C vi′→ir Construct, so as to obtain Figure 5 The multi-scale similarity matrix for bidirectional query is shown in Figure 1. The expression of this matrix is ​​as follows:

[0084]

[0085] Similarly, other feature pairs use the same method to calculate the multi-scale similarity matrix of bidirectional query. Finally, the feature F is constructed 1 and F 2 Multi-scale similarity matrix between them.

[0086] 2.3. Obtain the common features related to multi-scale pixels. Take the common features related to multi-scale pixels, semantic features, predicted deformation field, and implicit features as input, and use a bidirectional iterative module to perform N forward deformation field predictions and reverse deformation field predictions between the infrared image and the visible light image to complete the registration of the infrared image and the visible light image, specifically including:

[0087] Step 2.3.1, obtaining common features related to multi-scale pixels. To ensure that the common features of the local fine-grained and global maximum range are queried, this embodiment adopts the following method to obtain the common features:

[0088] The defined query range Δφ is added to the forward deformation field through the broadcast mechanism and the inverse deformation field

[0089] Δφ={(u,v)|u,v∈{-r,-r-1,…0,…,r-1,r}}

[0090] Wherein, Δφ represents the query range, u represents the horizontal query coordinate, v represents the vertical query coordinate, and r represents the query radius. In this embodiment, the query radius r=4.

[0091] Interpolation query is performed in the multi-scale similarity matrix to obtain the common features related to the pixels of the infrared image and the visible light image;

[0092]

[0093] 2.3.3. Take the common features, semantic features, predicted deformation field, and implicit features related to multi-scale pixels as input, and concatenate the input features of the forward deformation field and the input features of the reverse deformation field. The concatenated features are:

[0094]

[0095] The bidirectional iteration module is used to perform N iterations to optimize the forward deformation field and the reverse deformation field, and obtain N predicted forward deformation fields and predicted reverse deformation fields. To increase the calculation rate, this embodiment uses the parallel computing method of the GPU to predict the N forward deformation fields and the reverse deformation fields.

[0096] The bidirectional iteration module of this embodiment is composed of a gated cycle unit and a convolution prediction module, wherein the gated cycle unit is represented by the following formula:

[0097]

[0098] Among them, Conv 3×3 (·) represents the convolution operation, the convolution kernel size is 3×3, W z ,W r ,W h Represents the parameters of the convolution kernel. ⊙ represents the element-by-element product operation. After combining the gated recurrent unit with the implicit state, the difference deformation field is calculated and the predicted deformation field is updated according to the following formula:

[0099]

[0100] Repeat the iteration and output N predicted deformation fields. Similarly, the other registered image pairs are the same as the above process.

[0101] 2.3.3. Concatenate the predicted forward deformation fields of N different registered image pairs to form a forward deformation field sequence The predicted inverse deformation fields of N different registered image pairs are spliced ​​to form an inverse deformation field sequence

[0102] 2.3.4. Forward deformation field sequence With the inverse deformation field sequence By acting on the infrared image and visible light image to be registered, the registration of the infrared image and the visible light image can be achieved.

[0103] Step 3: To improve the registration accuracy and robustness of infrared and visible light images, this embodiment designs a loss function The loss function is used to train and optimize the infrared and visible light image bidirectional registration network. The loss function of this embodiment The calculation formula is as follows:

[0104]

[0105] in, To predict the endpoint loss between the deformation field and the true deformation field, To predict the bidirectional consistency loss between the deformation field and the true deformation field, is the feature similarity loss.

[0106] Endpoint loss Used to regress the flow in the estimator for the entire prediction sequence and Supervision, combined with real deformation field and End point loss The calculation formula is as follows:

[0107]

[0108] Among them, γ represents the weight, γ = 0.8, represents the true forward deformation field, represents the true inverse deformation field, represents the predicted i-th forward deformation field, represents the predicted i-th inverse deformation field, symbolized by ‖·‖ 1 It means to calculate the mean absolute error.

[0109] Bidirectional consistency loss Used to ensure the consistency and robustness of predictions. Bidirectional consistency loss The calculation formula is as follows:

[0110]

[0111] Among them, β represents the weight, β = 0.65, represents the two-dimensional grid coordinates of the image, Indicates interpolation sampling at the corresponding coordinates.

[0112] Similarity loss Used to find common features between two images, improve the accuracy of multimodal image registration, feature similarity loss The formula is as follows:

[0113]

[0114] Weight γ=0.8, β=0.65, temperature coefficient τ=0.07.

[0115] Step 4: Input the infrared image and visible light image to be registered into the trained infrared and visible light image bidirectional registration network to complete the registration of the visible light image and the infrared image.

[0116] like Figure 6 As shown, this embodiment tests the above infrared and visible light image bidirectional registration method on the FLIR, MSRS, and RoadScene test sets, and the registration endpoint errors obtained by the test are 2.680, 2.557, and 1.510, respectively, indicating that the method has a stronger registration accuracy.

[0117] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A method for bidirectional registration of infrared and visible light images, characterized in that: The following steps are involved: Step 1, obtaining and processing training samples, the training samples include source image pairs, deformed image pairs and true deformation fields, the deformed image pairs and true deformation fields are both obtained by transforming the source image pairs; the source image pairs are composed of visible light images and infrared light images of the same scene, and the true deformation fields include forward deformation fields and reverse deformation fields generated during the transformation of the source image pairs; Step 2: constructing a bidirectional registration network for infrared and visible light images based on the source image pair and the deformed image pair, so as to register the infrared image and the visible light image; The infrared and visible light image bidirectional registration network includes a common feature encoder, a semantic encoder, and a bidirectional iteration module. The registration implementation method includes: 2.

1. Based on infrared images and visible light images, the common feature encoder is used to extract the features F1 and F2 for learning cross-modal and homomodal features. The semantic encoder is used to extract semantic features and divide the semantic features into two parts, one of which is still the semantic features H1 and H2, and the other is used as the initial hidden state. and 2.

2. The cosine similarity between features F1 and F2 is calculated using the inner product to obtain the similarity matrix between F1 and F2; the similarity matrix between F1 and F2 is average pooled to obtain a multi-scale similarity matrix for bidirectional query; 2.

3. Set the predicted deformation field The initial value of is 0, and the predicted deformation field is used to query the common features related to the multi-scale pixels in the multi-scale similarity matrix. The common features, semantic features, predicted deformation field, and implicit features are used as inputs, and a bidirectional iterative module is used to perform N times of forward deformation field prediction and reverse deformation field prediction between the infrared image and the visible light image to complete the registration of the infrared image and the visible light image. Step 3: Set the loss function Use loss function to train and optimize the infrared and visible light image bidirectional registration network; loss function The calculation formula is as follows: in, represents the endpoint loss, represents the bidirectional consistency loss, represents feature similarity loss, γ and β both represent weights, represents the true forward deformation field, represents the true inverse deformation field, represents the predicted i-th forward deformation field, represents the predicted i-th inverse deformation field, represents the two-dimensional grid coordinates, It indicates interpolation sampling at the corresponding coordinates, and the symbol ‖·‖1 indicates the calculation of the mean absolute error; Step 4: Input the infrared image and visible light image to be registered into the trained infrared and visible light image bidirectional registration network to complete the registration of the visible light image and the infrared image.

2. The method for bidirectional registration of infrared and visible light images according to claim 1, characterized in that: The method for generating a forward deformation field and a reverse deformation field during the transformation of the source image pair in step 1 includes: 1.

1. Perform a rigid transformation on the source image pair, and calculate the affine transformation matrix of the source image according to the transformation parameters. The calculation formula of the affine transformation matrix is: Where M represents the affine transformation matrix, θ represents the rotation angle, s represents the scaling ratio, and t x and t y Respectively represent the pixel translation distance in the x and y directions of the image; 1.

2. Introduce random deformation field into the affine transformation matrix to obtain the forward deformation field in, represents the forward deformation field, represents the two-dimensional grid coordinates, φ rand According to Δ x ,Δ y ∈[-1,1] and standard deviation σ∈[16,32]; 1.

3. Based on the forward deformation field, the reverse deformation field is calculated using the interpolation sampling method 3. The method for bidirectional registration of infrared and visible light images according to claim 2, characterized in that: The deformed image pair is generated by sampling the source image pair based on the forward deformation field obtained in step 1.

2.

4. The method for bidirectional registration of infrared and visible light images according to claim 2, characterized in that: The implementation method of step 2.3 includes: 2.3.1、Define the query range Δφ and add the defined query range to the forward deformation field through the broadcast mechanism and the inverse deformation field 2.3.2, Assume the predicted deformation field The initial value of is 0, and the predicted deformation field is used to interpolate and search in the multi-scale similarity matrix to obtain the common features related to multi-scale pixels; 2.3.

3. Taking the common features, semantic features, predicted deformation fields, and implicit features related to multi-scale pixels as input, and concatenating the input features of the forward deformation field and the input features of the reverse deformation field, using a bidirectional iterative algorithm to perform N iterations to optimize the forward deformation field and the reverse deformation field, and obtaining N predicted forward deformation fields and predicted reverse deformation fields; 2.3.

4. Concatenate N predicted forward deformation fields to form a forward deformation field sequence Splice N predicted reverse deformation fields to form a reverse deformation field sequence 2.3.

5. Forward deformation field sequence With the inverse deformation field sequence By acting on the infrared image and visible light image to be registered, the registration of the infrared image and the visible light image can be achieved.

5. The method for bidirectional registration of infrared and visible light images according to claim 4, characterized in that: In step 2.3.3, the forward deformation field and the reverse deformation field are predicted using the parallel computing method of the GPU.

Images