Infrared and visible image registration method and system considering modal difference

Abstract

The present application belongs to the technical field of image processing, and provides an infrared and visible light image registration method and system considering modal difference, which maps the infrared image and the visible light image into radiation invariant images, pre-processes the infrared image and the visible light image, reduces the modal difference between the two images, and then calculates the local features of the obtained radiation invariant images; and then uses the progressive consistent sampling algorithm to realize the registration of the infrared image and the visible light image. By mapping the infrared image and the visible light image into radiation invariant images, the problem of losing effective matching when calculating and matching the local features of the two modal images with large nonlinear pixel difference is avoided, the registration result is more accurate, and more effective matching of the infrared image and the visible light image is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an infrared and visible light image registration method and system considering modal differences. BACKGROUND

[0002] Fusion of infrared and visible light images can utilize temperature information of the infrared modal and detail appearance information of the visible light modal, and is helpful to improve the accuracy of electrical equipment positioning and fault diagnosis. However, due to the differences in position, focal length, distortion parameters and the like between infrared and visible light lenses, there are inevitably deformation problems such as offset and scaling between infrared and visible light images collected by the same infrared thermal imager. In order to fuse and analyze the infrared image and the visible light image, the images of the two modals need to be accurately registered first. The cross-modal registration process generally includes mapping the original infrared and visible light image pair into a radiation-invariant image pair, then calculating pixel point features, and matching the features.

[0003] The inventors found that the method of image local feature calculation and matching based on deep self-attention learning can overcome more image changes compared to traditional methods, and can cope with sparse texture and repetitive texture situations, easily generate dense matching point pairs, and to a certain extent, can generate more effective and stable matching pairs. However, directly calculating and matching local features of infrared images and visible light images, due to the large nonlinear pixel difference between the two modal images, directly using the deep self-attention learning method for local feature calculation and matching may lose many effective matches, which may interfere with subsequent registration and fusion. SUMMARY

[0004] In order to solve the above problems, the application provides an infrared and visible light image registration method and system considering modal differences. First, a wavelet transform-based method is used to convert infrared and visible images into radiation-invariant images, and the infrared and visible light images are preprocessed to reduce the modal difference between the two images. On this basis, a LoFTR local feature calculation and matching method based on deep self-attention learning is used. Finally, a PROSAC method is used to estimate the transformation parameters between the infrared and visible light images, realizing the registration and fusion of the infrared and visible light images. Compared with traditional methods, the method of the application can achieve more accurate registration of infrared and visible light images. Moreover, for different images taken at different angles, images with sparse texture and repetitive texture can also achieve accurate registration results.

[0005] In order to achieve the above purpose, the application is implemented by the following technical solutions:

[0006] In a first aspect, the present application provides an infrared and visible image registration method considering modal difference, comprising:

[0007] Obtaining an infrared image and a visible image;

[0008] Mapping the infrared image and the visible image into a radiation invariant image;

[0009] Performing local feature calculation and local point matching on the radiation invariant image to obtain matched feature point pairs;

[0010] According to the pixel coordinates of the matched feature point pairs in the infrared image and the visible image, using a progressive consistent sampling algorithm, obtaining the transformation parameters of the infrared image to the visible image;

[0011] According to the transformation parameters, transforming the coordinates of the infrared image into the visible image coordinate system to realize the registration of the infrared image and the visible image.

[0012] Further, for the infrared image and the visible image, first perform Fourier transform to map to the frequency domain, and perform two-dimensional log-Gabor filtering in the frequency domain.

[0013] Further, based on wavelet transform, using the spatial odd-symmetric part and the spatial even-symmetric part of the two-dimensional log-Gabor filter, performing spatial filtering on the infrared image and the visible image according to multiple scales and multiple directions;

[0014] Calculating the real part and the imaginary part of the results after filtering of different scales and different directions, and calculating the amplitude of each pixel;

[0015] Calculating the superposition results of the spatial pixel amplitudes of the same direction and different scales to obtain a log-Gabor graph of multiple directions;

[0016] Each pixel value is the maximum value of the corresponding pixels of the log-Gabor graph of different directions, to obtain a radiation invariant image.

[0017] Further, using a deep self-attention learning network, performing local feature calculation and local point matching on the radiation invariant image to obtain matched feature point pairs.

[0018] Further, inputting the radiation invariant image into the deep self-attention learning network, a local feature convolutional neural network extracts two coarse feature maps with a size of 1 / 8 of the original image size, and two fine feature maps with a size of 1 / 2 of the original image size;

[0019] Flattening the coarse feature map into a one-dimensional vector and using position information for encoding; the one-dimensional feature vector passes through a self-attention layer and a cross-attention layer to capture the global context within and between images to enhance the features; further inputting into a matching module to obtain coarse matching;

[0020] The matching module is realized by a double Softmax operator, a matching confidence matrix is calculated to select a matching item according to a confidence level, and the matching obtained on the coarse feature map is refined to a fine matching with sub-pixel accuracy.

[0021] Further, after obtaining the matched feature point pairs in the infrared image and the visible light image, transformation parameters between the images are calculated according to pixel coordinates of the feature point pairs.

[0022] Further, according to the transformation parameters, coordinates of the infrared image are transformed to a coordinate system of the visible light image, and a final fusion image is obtained by averaging values of corresponding position pixels.

[0023] In a second aspect, the present application further provides an infrared and visible light image registration system considering modal difference, comprising:

[0024] A data acquisition module is configured to acquire an infrared image and a visible light image.

[0025] A mapping module is configured to map the infrared image and the visible light image into radiation-invariant images.

[0026] A feature point matching module is configured to perform local feature calculation and local point matching on the radiation-invariant images to obtain matched feature point pairs.

[0027] A transformation parameter calculation module is configured to obtain transformation parameters from the infrared image to the visible light image by using a progressive consistent sampling algorithm according to pixel coordinates of the matched feature point pairs in the infrared image and the visible light image.

[0028] A registration module is configured to transform coordinates of the infrared image to a coordinate system of the visible light image according to the transformation parameters to realize registration of the infrared image and the visible light image.

[0029] In a third aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to realize steps of the infrared and visible light image registration method considering modal difference according to the first aspect.

[0030] In a fourth aspect, the present application further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the program to realize steps of the infrared and visible light image registration method considering modal difference according to the first aspect.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] 1. In the present application, the infrared image and the visible light image are first mapped into radiation invariant images, the infrared image and the visible light image are preprocessed, the modal difference of the two images is reduced, and the local features of the obtained radiation invariant image are calculated; then the registration of the infrared image and the visible light image is realized by using the progressive consistent sampling algorithm; by mapping the infrared image and the visible light image into radiation invariant images, the problem of losing effective matching when the local feature calculation and matching of two modal images with large nonlinear pixel difference is avoided, the registration result is more accurate, and more effective matching of the infrared image and the visible light image is realized.

[0033] 2. Compared with the traditional image registration method, on the one hand, the two-dimensional log-Gabor filter based on the wavelet transform idea can map the original image pair with large modal difference into an image pair with greatly reduced modal difference; on the other hand, more reliable local feature description can be extracted based on the deep self-attention learning network, and the feature point position information and context information are effectively utilized to realize more effective matching; on this basis, the progressive sampling consistency method can obtain more accurate transformation parameter estimation result, and accurate registration result can be obtained for images taken at different angles, images with large modal difference and images with relatively sparse texture. BRIEF DESCRIPTION OF DRAWINGS

[0034] The drawings constituting a part of this embodiment are used to provide further understanding of the embodiment, and the illustrative embodiments of the embodiment and the description thereof are used to explain the embodiment, and do not constitute improper limitation on the embodiment.

[0035] Figure 1 It is a flowchart of the embodiment 1 of the present application;

[0036] Figure 2 It is a schematic diagram of obtaining a radiation invariant image of the embodiment 2 of the present application;

[0037] Figure 3 It is an effect diagram of mapping the visible light image into a radiation invariant image by using the two-dimensional log-Gabor filter of the embodiment 2 of the present application;

[0038] Figure 4 It is a schematic diagram of calculating the local features of the image and performing local point matching by using the deep self-attention learning network of the embodiment 2 of the present application;

[0039] Figure 5 It is an effect diagram of calculating the local features of the obtained radiation invariant image and performing local point matching of the embodiment 2 of the present application;

[0040] Figure 6 It is a final fused image effect diagram of the embodiment 2 of the present application. DETAILED DESCRIPTION

[0041] The application will be further described below with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as would be understood by one of ordinary skill in the art to which this application belongs.

[0043] Embodiment 1:

[0044] Traditional methods of directly extracting corner points, calculating features and matching features on the original image are not robust enough to changes in different modalities such as infrared and visible light. The Best Bin First (BBF) feature matching method ignores the context information of neighboring feature points when matching, which can also lead to matching errors. Methods such as LoFTR for image local feature calculation and matching based on deep self-attention learning can overcome more image changes compared to traditional methods, and can also deal with sparse texture and repetitive texture situations, easily generating dense matching point pairs, and to some extent, generating more effective and stable matching pairs. However, directly performing local feature calculation and matching on infrared and visible light images, due to the large nonlinear pixel difference between the two modalities, directly using LoFTR for local feature calculation and matching can potentially lose many effective matches, which can interfere with subsequent registration and fusion.

[0045] To solve the above problems, as shown in Figure 1 The embodiment provides a modal-robust infrared and visible light image registration method, which comprises the following steps:

[0046] obtaining an infrared image and a visible light image;

[0047] mapping the infrared image and the visible light image into radiation-invariant images;

[0048] performing local feature calculation and local point matching on the radiation-invariant images to obtain matched feature point pairs;

[0049] According to the pixel coordinates of the matched feature point pairs in the infrared image and the visible light image, using the progressive consistent sampling algorithm, the transformation parameters of the infrared image to the visible light image are obtained;

[0050] According to the transformation parameters, the coordinates of the infrared image are transformed into the visible light image coordinate system, and the registration of the infrared image and the visible light image is realized.

[0051] In the embodiment, the process of converting the infrared and visible images into the radiation invariant map by using the wavelet transform-based method is as follows:

[0052] The infrared image and the visible image are spatially filtered according to multiple scales and multiple directions by using the odd-symmetric part and the even-symmetric part of the two-dimensional log-Gabor filter;

[0053] The real part and the imaginary part of the filtered results in different scales and different directions are calculated, and the amplitude of each pixel is calculated;

[0054] The superposition results of the amplitudes of the spatial pixels in the same direction and different scales are calculated to obtain the log-Gabor maps in multiple directions;

[0055] Each pixel value is the maximum value of the corresponding pixels in the log-Gabor maps in different directions to obtain the radiation invariant map.

[0056] In the embodiment, the local feature calculation and matching process of the obtained radiation invariant map are performed by using the deep self-attention learning network, and the process is as follows:

[0057] When the two radiation invariant maps are input, the local feature convolutional neural network extracts two coarse feature maps with a size of 1 / 8 of the original image size and two fine feature maps with a size of 1 / 2 of the original image size;

[0058] The coarse feature maps are flattened into one-dimensional vectors and encoded using position information. The one-dimensional feature vectors pass through a self-attention layer and a cross-attention layer, which capture the global context within and between images to enhance the features. They are further input into the matching module to obtain coarse matching;

[0059] The matching module is realized by a double-Softmax operator, which calculates the matching confidence matrix and selects the matching item with high confidence. Then, it is input into the coarse-to-fine module to refine the matching obtained on the coarse feature map into fine matching with sub-pixel accuracy. The entire network is trained using the coarse matching negative log-likelihood loss and the fine matching L2 loss.

[0060] In the embodiment, after obtaining the matching feature point pairs in the infrared image and the visible image, the transformation parameters between the images are calculated according to the pixel coordinates of the feature point pairs; wherein, the quality of all matching point pairs is evaluated to obtain Q values, and the Q values are arranged in descending order, and random sampling is performed according to the Q value descending order result in each iteration, and model hypothesis and verification are performed.

[0061] In the embodiment, the coordinates of the infrared image are transformed into the coordinate system of the visible image according to the transformation parameters, and the average value of the corresponding position pixels is obtained to obtain the final fusion image.

[0062] Embodiment 2:

[0063] This embodiment provides a modally robust infrared and visible light image registration method. The method is detailed by applying it to images of power equipment; the specific solution is as follows:

[0064] First, infrared and visible light images of the power equipment are acquired; specifically, existing technologies are used for image acquisition.

[0065] In this embodiment, a wavelet transform-based method is used to convert infrared and visible images into radiometric invariant maps;

[0066] Given that existing methods directly extract and match feature points from the original image, it is difficult to overcome the nonlinear modal differences between infrared and visible light images, resulting in large differences in the calculated local features or making matching difficult.

[0067] like Figure 2 As shown, the method for constructing the Log-Gabor sequence in this example is as follows: For the input infrared and visible light images, a Fourier transform is first performed to map them to the frequency domain, and then Log-Gabor filtering is performed in the frequency domain. The filtering transformation formula is:

[0068]

[0069] Where (ρ, θ) represent logarithmic polar coordinates; s and o represent scale and direction; (ρ s θ so ) is the center frequency of the 2D-LGF; σ ρ and σ θ This represents the bandwidth along the coordinates ρ and θ. The 2D-LGF is a filter defined in the frequency domain. The corresponding filter in the spatial domain is obtained through the inverse Fourier transform and can be expressed as:

[0070] L(x,y,s,o)=L even (x,y,s,o)+iL odd (x,y,s,o) (2)

[0071] Among them, L even (x,y,s,o) and L odd (x, y, s, o) represent the even-symmetric and odd-symmetric log-Gabor wavelets, respectively, which also correspond to the real and imaginary parts of the image convolution result in the spatial domain. Next, convolving the image with the even-symmetric and odd-symmetric log-Gabor wavelets respectively yields:

[0072]

[0073] Then, the magnitude A of the image I(x,y) at scale s and direction o. so(x,y) is calculated by the following formula:

[0074]

[0075] For direction o, the magnitude of all scales ( Ns The total number of scales is added together to generate a log-Gabor graph A. o (x,y):

[0076]

[0077] Finally, we take the log-Gabor plot A in all directions for each pixel. o The radiation invariant map is obtained by maximizing (x,y).

[0078]

[0079] in, Ns and No These represent the total number of scales and directions, respectively. Figure 3 The image shows the results of the radiometric invariant transformation of the infrared and visible light images. From the image, we can observe that the generated radiometric invariant map preserves the geometric structure well in both modes, and the image differences caused by the different modes are reduced in the transformed image.

[0080] In this embodiment, a deep self-attention learning network is used to calculate and match the local features of the acquired radiative invariant map;

[0081] The resulting radiance-invariant map is input into a Local Feature Transformer (LoFTR) for feature matching. LoFTR is a detector-free method that utilizes self-attention techniques to achieve robust matching results. Compared to traditional detector-based methods, LoFTR produces higher-quality matching results in low-texture or repetitive regions. Figure 4 The framework of LoFTR is explained. It consists of four main modules: a Local Feature Convolutional Neural Network (CNN), a coarse-level local feature transformation module, a matching module, and a coarse-to-fine module. Figure 4 As shown, in this embodiment, the input radiation invariant diagram M ir and M rgb Local feature convolutional neural networks (CNNs) extract two coarse feature maps. and The image is 1 / 8 the size of the original image, and includes two fine-level feature maps. and The size is 1 / 2 of the original image size. Then, the coarse-level features are unfolded into one-dimensional vectors and encoded using position information, which are input into the coarse-level local feature transformation module. The coarse-level local feature transformation module contains a self-attention layer and a cross-attention layer, which capture the global context within and between images, respectively, to enhance the features. The enhanced features are represented as and They are further input into the matching module and obtain coarse matching. The matching module is implemented by a double Softmax operator, which maps the confidence matrix as:

[0082] P c (i,j)=Softmax(S(i,·)) j ·Softmax(S(·,j)) i (7)

[0083] where i and j represent the pixel indices of the two images. And the matching items are selected with high confidence. The fine-to-module refines the matching obtained at the coarse level to fine-level matching with sub-pixel accuracy. The entire network is trained using the negative log-likelihood loss calculated by the coarse-level network and the L2 loss calculated by the fine-level network.

[0084] In this embodiment, the LoFTR module is used, and the obtained feature point matching result is as shown in Figure 5 .

[0085] In this embodiment, the transformation parameters from the infrared image to the visible light image are estimated using the PROSAC algorithm according to the pixel coordinates of the matched feature point pairs in the two images.

[0086] Specifically, after obtaining the matched feature point pairs in the infrared image and the visible light image, in this embodiment, the transformation parameters between the images are estimated according to the pixel coordinates of the feature point pairs; since the offset between the infrared lens and the visible light lens is relatively small compared to the distance of the object being photographed, it can be approximated as a common optical center, and the homography transformation matrix H is used to transform the infrared image into the coordinate system of the visible light image; in the homogeneous coordinate system, the pixel coordinates in the two images can be represented by the following relationship:

[0087]

[0088] where h 33 =1 in the homography transformation matrix H. Therefore, the parameter degrees of freedom of the transformation matrix are 8, which can be estimated using four pairs or more of feature point pairs.

[0089] Due to the existence of matching noise or even false matching outliers, a least square method or a RANSAC method is usually used for parameter estimation; however, in RANSAC, each pair of feature points is treated equally, and a sample is randomly selected from the entire set of feature point pairs, and there are problems such as randomness of estimation results and slow convergence speed. Therefore, in the embodiment, a PROSAC algorithm is used for estimation of transformation parameters.

[0090] In the embodiment, the PROSAC algorithm designs a semi-random method, calculates the Q value by evaluating the quality of all matching point pairs, and then arranges the Q values in descending order, preferentially samples in high-quality point pairs in each iteration, performs model hypothesis and verification, thereby reducing the algorithm complexity, improving the efficiency, and avoiding the situation that RANSAC random algorithm cannot guarantee convergence.

[0091] In the embodiment, according to the transformation parameters, the coordinates of the infrared image are transformed into the coordinate system of the visible light image to realize registration.

[0092] Specifically, after the transformation model is calculated, the coordinates of the infrared image are transformed into the coordinate system of the visible light image, then the average value of the corresponding position pixels is taken to obtain the final fusion image, as shown in Figure 6 .

[0093] In view of the difference between infrared and visible light modalities, the two-dimensional log-Gabor filter based on the wavelet transform idea is used to convert the infrared and visible images into radiation invariant images, the infrared and visible images are preprocessed to reduce the difference between the two image modalities. On this basis, the LoFTR local feature calculation and matching method based on deep self-attention learning is used, and finally the PROgressive SAmple Consensus (PROSAC) method is used to estimate the transformation parameters between the infrared and visible light images to realize the registration and fusion of the infrared and visible light images. Through the method in the present disclosure, more accurate registration of infrared and visible light images can be realized compared with traditional methods. And for different images taken at different angles, images with sparse texture and repeated texture can also obtain accurate registration results.

[0094] Embodiment 2:

[0095] The embodiment provides a modal difference considering infrared and visible light image registration system, comprising:

[0096] The data acquisition module is configured to acquire infrared images and visible light images.

[0097] The mapping module is configured to map the infrared images and the visible light images into radiation invariant images.

[0098] a feature point matching module configured to perform local feature calculation and local point matching on the radiation-invariant image to obtain matched feature point pairs;

[0099] a transformation parameter calculation module configured to obtain transformation parameters from the infrared image to the visible light image by using a progressive consistent sampling algorithm according to pixel coordinates of the matched feature point pairs in the infrared image and the visible light image;

[0100] a registration module configured to transform coordinates of the infrared image to a coordinate system of the visible light image according to the transformation parameters to implement registration of the infrared image and the visible light image.

[0101] The working method of the system is the same as the infrared and visible light image registration method considering modal difference of Embodiment 1, and thus will not be described here.

[0102] Embodiment 4:

[0103] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement steps of the infrared and visible light image registration method considering modal difference of Embodiment 1.

[0104] Embodiment 5:

[0105] The embodiment provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements steps of the infrared and visible light image registration method considering modal difference of Embodiment 1 when executing the program.

[0106] The above merely describes preferred embodiments of the present embodiment and is not intended to limit the present embodiment. The present embodiment can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A method for infrared and visible light image registration considering modal differences, characterized in that, include: Acquire infrared and visible light images; Map infrared and visible light images to a radiation-invariant image; Local feature calculation and local point matching are performed on the radiation-invariant image to obtain matched feature point pairs; By using a deep self-attention learning network, local feature calculation and local point matching are performed on the radiation-invariant image to obtain matched feature point pairs. The radiation-invariant image is input into a deep self-attention learning network. The local feature convolutional neural network extracts two coarse feature maps, each 1 / 8 the size of the original image, and two fine feature maps, each 1 / 2 the size of the original image. The coarse feature map is flattened into a one-dimensional vector and encoded using positional information; One-dimensional feature vectors are passed through a self-attention layer and a cross-attention layer to capture global context within and between images to enhance features; they are then further input into a matching module to obtain a coarse match. The matching module is implemented using dual Softmax operators to calculate the matching confidence matrix and select matching items based on the confidence; it refines the matching obtained on the coarse feature map into fine matching with sub-pixel precision. Based on the pixel coordinates of the matched feature point pairs in the infrared and visible light images, the transformation parameters from the infrared image to the visible light image are obtained using the progressive consistent sampling algorithm. Based on the transformation parameters, the coordinates of the infrared image are transformed to the visible light image coordinate system, thereby achieving registration between the infrared image and the visible light image.

2. The infrared and visible light image registration method considering modal differences as described in claim 1, characterized in that, For both infrared and visible light images, a Fourier transform is first performed to map them to the frequency domain, and then a two-dimensional logarithmic Gabor filter is applied in the frequency domain.

3. The infrared and visible light image registration method considering modal differences as described in claim 2, characterized in that, Based on wavelet transform, the spatial odd-symmetric and even-symmetric parts of the two-dimensional logarithmic Gapper filter are used to perform spatial filtering on infrared and visible light images according to multiple scales and multiple directions. Calculate the real and imaginary parts of the filtered results at different scales and in different directions, and calculate the amplitude of each pixel; The magnitude superposition results of spatial pixels at different scales in the same direction are calculated to obtain log-Gauber plots in multiple directions; Each pixel is assigned the maximum value of the corresponding pixel in the logarithmic Gabor plot for different directions, resulting in a radiation-invariant map.

4. The infrared and visible light image registration method considering modal differences as described in claim 1, characterized in that, After obtaining the matching feature point pairs in the infrared and visible light images, the transformation parameters between the images are calculated based on the pixel coordinates of the feature point pairs.

5. The infrared and visible light image registration method considering modal differences as described in claim 1, characterized in that, Based on the transformation parameters, the coordinates of the infrared image are transformed to the coordinate system of the visible light image, and the average value of the corresponding pixels is taken to obtain the final fused image.

6. An infrared and visible light image registration system considering modal differences, characterized in that, include: The data acquisition module is configured to acquire infrared and visible light images; The mapping module is configured to map infrared and visible light images into radiation-invariant images. The feature point matching module is configured to perform local feature calculation and local point matching on the radiation-invariant image to obtain matched feature point pairs. By using a deep self-attention learning network, local feature calculation and local point matching are performed on the radiation-invariant image to obtain matched feature point pairs. The radiation-invariant image is input into a deep self-attention learning network. The local feature convolutional neural network extracts two coarse feature maps, each 1 / 8 the size of the original image, and two fine feature maps, each 1 / 2 the size of the original image. The coarse feature map is flattened into a one-dimensional vector and encoded using positional information; One-dimensional feature vectors are passed through a self-attention layer and a cross-attention layer to capture global context within and between images to enhance features; they are then further input into a matching module to obtain a coarse match. The matching module is implemented using dual Softmax operators to calculate the matching confidence matrix and select matching items based on the confidence; it refines the matching obtained on the coarse feature map into fine matching with sub-pixel precision. The transformation parameter calculation module is configured to: obtain the transformation parameters from the infrared image to the visible light image based on the pixel coordinates of the matched feature point pairs in the infrared image and the visible light image using the progressive consistent sampling algorithm; The registration module is configured to transform the coordinates of the infrared image to the visible light image coordinate system according to the transformation parameters, thereby achieving registration between the infrared image and the visible light image.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps of the infrared and visible light image registration method considering modal differences as described in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the infrared and visible light image registration method considering modal differences as described in any one of claims 1-5.

Images