Two-section infrared and visible light image registration method, system and device

Abstract

The invention discloses a two-stage infrared and visible light image registration method, system and device. The method comprises the following steps of image preprocessing, contour extraction, angular point detection, feature description and matching, affine transformation estimation, multi-scale optical flow estimation, optical flow constraint and loss function, reverse resampling and fusion and error evaluation. According to the invention, rough registration is carried out by using contour angular point features, so that a preliminary alignment result can be quickly obtained; refined alignment is carried out in combination with an unsupervised optical flow network, and sub-pixel-level registration precision is achieved. The contour angular points are based on shape information of an image target, are natural and are not influenced by spectral differences, and the matching stability is enhanced through main direction and angle features. The unsupervised depth optical flow model estimates a pixel displacement field by learning consistency characteristics of an input image, and does not need to depend on annotation data. The combination can effectively eliminate the difference between infrared light and visible light, and improves the robustness and adaptability of registration.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly to a two-stage infrared and visible light image registration method, system and device. Background Art

[0002] Visible light and infrared images each have their own unique advantages and characteristics in the field of imaging. Fusion of infrared and visible light images helps to obtain more comprehensive information, enhance image recognition, and improve the accuracy and efficiency of target detection and recognition.

[0003] Currently, visible light and infrared images are generally registered based on regions, features, and deep learning. Region-based registration methods are insensitive to feature differences, are highly robust, and can handle grayscale inversions and complex grayscale relationships without requiring complex feature extraction algorithms. Feature-based registration methods are relatively insensitive to global grayscale differences and inversions, are computationally efficient, and can provide precise point correspondences. Deep learning-based registration methods achieve high-precision registration when sufficient annotated data is available, and perform particularly well in scenes with complex radiation differences and large deformations.

[0004] Although the above registration method achieves good registration effect, it still has the following problems:

[0005] 1. Feature matching failure caused by cross-modal spectral differences: The imaging mechanisms and spectral responses of infrared and visible light images are completely different. This results in traditional gradient-based descriptors (such as SIFT and SURF) having extremely low similarity between heterogeneous images, easily resulting in numerous mismatches or even failure to match.

[0006] 2. The difficulty in unified modeling of large-scale geometric deformations and local nonlinear displacements: Multi-source imaging platforms often have significant rotation, scale, parallax, and even lens distortion. Regional or frequency domain methods can only handle small displacements, and deep optical flow methods lack global constraints. Both have their limitations.

[0007] 3. The contradiction between high precision and real-time performance and the lack of training data: Although traditional deep end-to-end registration can output dense flow fields, training relies on large-scale labeled data. Moreover, the model is large and the inference process is time-consuming, making it difficult to deploy on platforms with limited computing power, such as power inspection robots, drones, and vehicle-mounted terminals.

[0008] Therefore, it is necessary to propose a new solution to solve the above problems. Summary of the Invention

[0009] The purpose of the present invention is to overcome the above-mentioned deficiencies of the prior art and to provide a two-stage infrared and visible light image registration method, system and device.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A two-stage infrared and visible light image registration method comprises the following steps:

[0012] Step S1, image preprocessing: grayscale conversion and Gaussian smoothing are performed on the input infrared image and visible light image;

[0013] Step S2, contour extraction: extract edge contours from the pre-processed infrared image and visible light image to obtain an infrared image contour map and a visible light contour map, respectively;

[0014] Step S3, corner point detection: extracting corner points with significant geometric structures on each contour image;

[0015] Step S4, feature description and matching: define a joint descriptor and use the nearest neighbor search strategy to perform preliminary point pair matching to obtain a set of candidate matching pairs;

[0016] Step S5, affine transformation estimation: screening candidate matching pairs to eliminate false matches and estimating the affine transformation matrix to obtain a coarse registration image;

[0017] Step S6, multi-scale optical flow estimation: the coarsely registered visible light image and the infrared original image are stitched together and fed into an unsupervised optical flow estimation network to obtain a fine-scale optical flow field;

[0018] Step S7, optical flow constraint and loss function: adaptive training is performed by jointly minimizing photometric consistency loss, smoothness loss, structural similarity loss function and distillation loss function;

[0019] Step S8, reverse resampling: performing pixel-level reverse mapping on the coarsely registered visible light image and combining it with the fine-scale optical flow field to obtain the final optical flow field;

[0020] Step S9, fusion and error evaluation: calculate the error map and mean square error, and evaluate the calculated registration results.

[0021] Furthermore, step S3 includes the following steps:

[0022] Step S301: Set infrared image The contour point sequence is , visible light image The contour point sequence is , using the boundary tracking algorithm to extract the contour connected domain into an ordered chain, for each point on the chain and , and define its normal direction as:

[0023] (1)

[0024] In formula (1), is the centroid coordinate of the current contour end;

[0025] Step S302, in order to measure the point and The degree of angle mutation is defined as the angle response function and for:

[0026] (2)

[0027] In formula (2), is the local window width of the corner response;

[0028] like Exceeding the set threshold , then it is believed that is a candidate corner point; if Exceeding the set threshold , then it is believed that is the candidate corner point.

[0029] Furthermore, step S4 includes the following steps:

[0030] Step S401: define the infrared image to take into account both local structure and direction information Descriptor and visible light images Descriptor :

[0031] (3)

[0032] In formula (3), is the contour direction angle of the infrared image, is the corner response intensity of the infrared image; is the contour direction angle of the visible light image, is the corner point response intensity of visible light;

[0033] Step S402: infrared image The set of corner points , and visible light images The set of corner points Construct the corresponding descriptor set and calculate the similarity by Euclidean distance:

[0034] , , (4);

[0035] Step S403: using the nearest neighbor search strategy, The set of corner points For each corner point, search the visible light image with the closest Euclidean distance Perform preliminary point pair matching on the corner points to obtain a set of candidate matching pairs .

[0036] Furthermore, step S5 includes the following steps:

[0037] Step S501: Based on the RANSAC algorithm, assume that a candidate affine transformation matrix H satisfies the following transformation relationship:

[0038] ,

[0039] Where a, b, c, and d are the coefficients of linear transformation; 、 is the translation amount; For visible light images The midpoint coordinates of For visible light images The midpoint coordinates after transformation, , ;

[0040] Step S502: determine the final affine transformation parameters by minimizing the following residual sum of squares:

[0041] ;

[0042] Step S503: After the affine transformation matrix H is obtained, it is used to transform the visible light image Perform preliminary registration to obtain a coarse registration image :

[0043] ,

[0044] Where H is the affine transformation matrix defined in step S501,

[0045] Coarsely register the image Serves as the input basis for unsupervised optical flow registration.

[0046] Furthermore, step S6 includes the following steps:

[0047] Step S601: Multi-scale optical flow estimation is used to roughly register the visible light image. And the original infrared image Splice to , R represents the number of channels of the image, L represents the height of the image, and W represents the width of the image;

[0048] Step S602: Send it to the unsupervised optical flow estimation network and let each layer iteratively output :

[0049] , ,i=1,2,3

[0050] in, , and upsample by scale

[0051] ;

[0052] Finally, we get the fine-scale optical flow field

[0053] ;

[0054] in, Represents the optical flow field of the iterative output of the i-th layer; Represents the fusion mask of the iterative output of the i-th layer; Represents the optical flow update amount of the i-th layer; represents the update amount of the fusion mask of the i-th layer; Represents the two components of optical flow; Indicates the height of the i-th layer; represents the width of the i-th layer; 、 represent the displacement in the x-axis and y-axis directions respectively.

[0055] Furthermore, in step S7, the comprehensive goal of the combined photometric consistency loss, smoothness loss, structural similarity loss, and distillation loss function is:

[0056] ;

[0057] in, 、 、 is any positive real number; is the photometric consistency loss function; is the structural similarity loss function; is the distillation loss function; is the smoothness loss function.

[0058] Furthermore, in step S8, the coarse registration of the visible light image Perform pixel-level reverse mapping to obtain the final optical flow field F:

[0059] ;

[0060] Select bilinear:

[0061] ;

[0062] in, , ;

[0063] Weight ; m and n represent integer offsets in the x-axis and y-axis directions respectively.

[0064] Furthermore, in step S9, the error graph is: ;

[0065] The mean square error is: ;

[0066] If the error value of each pixel in the error map is ≤ , the registration is considered successful; otherwise, it prompts manual compounding or re-running; µ is the set threshold;

[0067] The final output is the high-resolution visible light image after registration , fusion mosaic graph and error heat map visualization results.

[0068] The present invention also provides a two-stage infrared and visible light image registration system, the system comprising:

[0069] Image acquisition module: used to acquire infrared images and visible light images;

[0070] Preprocessing module: used for grayscale and Gaussian smoothing preprocessing of infrared images and visible light images;

[0071] Two-stage registration processing module: including the contour corner matching module and the unsupervised optical flow module; the contour corner matching module is used to extract significant corners on the image contour and perform feature matching based on local geometric direction information to complete the preliminary alignment of the infrared image and the visible light image; the unsupervised optical flow module uses an unsupervised optical flow network based on deep learning to accurately align the infrared image and the visible light image;

[0072] Registration result output module: used to output the infrared and visible light image pair results after two-stage registration.

[0073] The present invention also provides a two-stage infrared and visible light image registration device, comprising:

[0074] Memory, used to store computer programs and data;

[0075] The processor is configured to implement the steps of the above-mentioned two-stage infrared and visible light image registration method when executing a computer program.

[0076] The beneficial effects of the present invention are:

[0077] 1. This method uses contour corner features for coarse registration, quickly obtaining preliminary alignment results. It then combines this with an unsupervised optical flow network for refined alignment, achieving sub-pixel registration accuracy. This method is capable of high-precision registration of images with any rotation angle between 0° and 90°, as well as any translational offset.

[0078] 2. In this invention, contour corners are based on the shape information of the image target, which is naturally unaffected by spectral differences. The main direction and angle features enhance matching stability. The unsupervised deep optical flow model estimates the pixel displacement field by learning consistent features of the input image, without relying on labeled data. This combination effectively eliminates the differences between infrared and visible light, improving the robustness and adaptability of the registration.

[0079] 3. The registration method presented in this paper is highly robust to noise, brightness variations, partial occlusion, and slight deformation. Furthermore, the algorithm design has a moderate computational complexity, and its parameters can be adjusted both online and offline, meeting the practical requirements for fast and stable registration.

[0080] 4. The registration method presented in this paper is applicable to a variety of devices and scenarios and can be deployed on general-purpose computing devices, embedded systems, or edge computing platforms. Its high-precision and stable registration performance can significantly improve applications such as power equipment fault diagnosis, remote sensing data analysis, security monitoring and identification, and intelligent traffic environment perception. It is also of great significance for multimodal image fusion and subsequent intelligent analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 is a flow chart of the two-stage infrared and visible light image registration method in this embodiment;

[0082] Figure 2 1 is a structural framework diagram of the two-stage infrared and visible light image registration system in this embodiment;

[0083] Figure 3 This is a structural framework diagram of the two-stage infrared and visible light image registration device in this embodiment.

[0084] Reference numerals: image acquisition module 1 , preprocessing module 2 , two-stage registration processing module 3 , contour corner matching module 301 , unsupervised optical flow module 302 , registration result output module 4 , memory 5 , processor 6 . DETAILED DESCRIPTION

[0085] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0086] Example: A two-stage infrared and visible light image registration method, such as Figure 1As shown in the figure, the registration includes coarse registration based on contour corner matching and fine registration based on unsupervised optical flow. During registration, contour information is first extracted from the preprocessed infrared and visible light images, and corners and their main directions on the contours are detected. This can be done using the curvature scale space (CSS) corner detection operator or other corner detectors. Next, the spatial distribution and directional characteristics of the contour corners are used to perform point-to-point matching and estimate a rough geometric transformation (such as affine or transmission transformation). The preliminary registered structure is then input into the unsupervised deep optical flow estimation model. By optimizing the grayscale or structural consistency between images, the pixel-level displacement field is estimated, thereby finely correcting residual misalignment. This two-stage registration strategy effectively combines coarse matching based on shape features with dense optimization based on learning, which not only improves the registration convergence speed but also ensures high-precision alignment.

[0087] Among them, in the coarse registration based on contour corner matching, the infrared image and the visible light image are preliminarily aligned by extracting the significant corners on the image contour and combining them with the local geometric direction information for feature matching. The coarse registration based on contour corner matching includes five key steps: image preprocessing, contour extraction, corner detection, feature description and matching, and affine transformation estimation. Specifically:

[0088] Step S1, image preprocessing: input infrared image With visible light images Grayscale conversion and Gaussian smoothing are performed to improve the robustness of edge detection.

[0089] Step S2, contour extraction: Use the Canny operator to extract the pre-processed infrared image With visible light images Extract edge contours and obtain infrared image contour maps and visible light profiles .

[0090] Step S3, corner point detection: extracting corner points with significant geometric structures on each contour image E;

[0091] Furthermore, step S3 specifically includes:

[0092] Step S301: Set infrared image The contour point sequence is , visible light image The contour point sequence is , using the boundary tracking algorithm to extract the contour connected domain into an ordered chain, for each point on the chain and , and define its normal direction as:

[0093] (1)

[0094] In formula (1), The coordinates of the center of mass of the current contour end are used to unify the reference system of angles;

[0095] Step S302, in order to measure the point and The degree of angle mutation is defined as the angle response function and for:

[0096] (2)

[0097] In formula (2), is the local window width of the corner response;

[0098] like Exceeding the set threshold , then it is believed that is a candidate corner point; if Exceeding the set threshold , then it is believed that is the candidate corner point.

[0099] Step S4, feature description and matching: define a joint descriptor and use the nearest neighbor search strategy to perform preliminary point pair matching to obtain a set of candidate matching pairs;

[0100] Furthermore, step S4 specifically includes:

[0101] Step S401: define the infrared image to take into account both local structure and direction information Descriptor and visible light images Descriptor :

[0102] (3)

[0103] In formula (3), is the contour direction angle of the infrared image, is the corner response intensity of the infrared image; is the contour direction angle of the visible light image, is the corner response intensity of visible light;

[0104] Step S402: infrared image The set of corner points , and visible light images The set of corner points Construct the corresponding descriptor set and calculate the similarity by Euclidean distance:

[0105] , , (4);

[0106] Step S403: using the nearest neighbor search strategy, The set of corner points For each corner point, search the visible light image with the closest Euclidean distance Perform preliminary point pair matching on the corner points to obtain a set of candidate matching pairs .

[0107] The main direction descriptor of contour corners is designed to enhance the cross-modal consistency of feature matching.

[0108] Step S5, affine transformation estimation: screening candidate matching pairs to eliminate false matches and estimating the affine transformation matrix to obtain a coarse registration image;

[0109] Furthermore, step 5 specifically includes:

[0110] Step S501: Based on the RANSAC algorithm, assume that a candidate affine transformation matrix H satisfies the following transformation relationship:

[0111] ,

[0112] Where a, b, c, and d are the coefficients of the linear transformation, which describe the rotation, scaling, and shearing of the image; 、 is the translation amount, which describes the translation of the image in the x-axis and y-axis directions; For visible light images The midpoint coordinates of For visible light images The midpoint coordinates after transformation, , ;

[0113] Step S502: determine the final affine transformation parameters by minimizing the following residual sum of squares:

[0114] ;

[0115] Step S503: After the affine transformation matrix H is obtained, it is used to transform the visible light image Perform preliminary registration to obtain a coarse registration image :

[0116] ,

[0117] Where H is the affine transformation matrix defined in step S501, and the parameters of the affine transformation matrix H are solved in step S502.

[0118] Coarsely register the image Serves as the input basis for unsupervised optical flow registration.

[0119] In the unsupervised optical flow-based precise registration, an unsupervised optical flow network based on deep learning is used to further accurately align image pixels between infrared images and visible light images. Specifically:

[0120] Step S6, multi-scale optical flow estimation: The coarsely registered visible light image and the infrared original image are spliced ​​and sent to the unsupervised optical flow estimation network to obtain a fine-scale optical flow field.

[0121] Furthermore, step 6 specifically includes:

[0122] Step S601: Multi-scale optical flow estimation is used to roughly register the visible light image. And the original infrared image Splice to , R represents the number of channels of the image, L represents the height of the image, and W represents the width of the image;

[0123] Step S602: Send it to the unsupervised optical flow estimation network and let each layer iteratively output :

[0124] , ,i=1,2,3

[0125] in, , and upsample by scale, as shown in the following formula, the upsampling scale is 23-i times, that is, as the number of iterative layers increases, the upsampling scale decreases and gradually recovers to the resolution of the original image;

[0126] ;

[0127] Finally, we get the fine-scale optical flow field

[0128] ;

[0129] in, Represents the optical flow field of the iterative output of the i-th layer; Represents the fusion mask of the iterative output of the i-th layer; Represents the optical flow update amount of the i-th layer; represents the update amount of the fusion mask of the i-th layer; Represents the two components of optical flow (horizontal and vertical displacement); Indicates the height of the i-th layer; represents the width of the i-th layer; 、 represent the displacement in the x-axis and y-axis directions respectively.

[0130] In the multi-scale optical flow estimation step, a pyramid or multi-resolution deep optical flow network structure is adopted to improve the matching ability under large displacement.

[0131] The fine-scale optical flow field represents the motion relationship between the pixels of the visible light image and the infrared image after coarse registration, and is a form of registration. Its function is to be used for fine registration and correction of infrared and visible light images.

[0132] Step S7, optical flow constraint and loss function: adaptive training is performed by jointly minimizing photometric consistency loss, smoothness loss, structural similarity loss and distillation loss function.

[0133] Furthermore, the photometric consistency loss function is:

[0134] ,

[0135] ,

[0136] Where, It is a commonly used photometric consistency measure used to measure the similarity between two pixel values; is the error term; is a small constant used to avoid division by zero;

[0137] The smoothness loss function is:

[0138] ;

[0139] or ;

[0140] is a regularization parameter used to control the intensity of the gradient penalty; Represents the gradient of image I, which is usually used to measure how fast the pixel values ​​in the image change;

[0141] The distillation loss function is:

[0142] ;

[0143] Where, For if there is a teacher model.

[0144] The combined goal of photometric consistency, smoothness, and distillation loss functions is:

[0145] ;

[0146] in, 、 、 is any positive real number; is the photometric consistency loss function; is the structural similarity loss function; is the distillation loss function; is the smoothness loss function.

[0147] In the optical flow constraint and loss function step, adaptive training is performed by jointly minimizing loss functions such as photometric consistency and flow field smoothing. This allows the model to achieve both accuracy and generalization performance without the need for labeled data. The overall model is compact, capable of efficient feature extraction and fast displacement prediction.

[0148] Step S8, reverse resampling: perform pixel-level reverse mapping on the coarsely registered visible light image and combine it with the fine-scale optical flow field to obtain the final optical flow field, that is, the final registration result.

[0149] Furthermore, for coarse registration of visible light images Perform pixel-level reverse mapping to obtain the final optical flow field F:

[0150] ;

[0151] Select bilinear:

[0152] ;

[0153] in, , ;

[0154] Weight m and n represent integer offsets in the x-axis and y-axis directions respectively.

[0155] Step S9, fusion and error evaluation: calculate the error map and mean square error, and evaluate the registration result calculated in step S8.

[0156] The error graph is: ;

[0157] The mean square error is: ;

[0158] If the error value of each pixel in the error map is ≤ , the registration is considered successful; otherwise, it prompts manual compounding or re-running; µ is the set threshold;

[0159] The final output is the high-resolution visible light image after registration , fusion mosaic graph and error heat map visualization results.

[0160] In the technical solution provided in the embodiment of the present application, the contour corner features are used for coarse registration to quickly obtain preliminary alignment results; the unsupervised optical flow network is combined for refined alignment to achieve sub-pixel registration accuracy.

[0161] In the technical solution provided by the embodiments of this application, contour corners are based on the shape information of the image target and are naturally unaffected by spectral differences. The main direction and angle features enhance matching stability. The unsupervised deep optical flow model estimates the pixel displacement field by learning the consistent features of the input image, without relying on labeled data. This combination effectively eliminates the differences between infrared and visible light, improving the robustness and adaptability of the registration.

[0162] The technical solution provided in the embodiments of this application demonstrates a registration method that is highly robust to noise, brightness variations, partial occlusion, and slight deformation. Furthermore, the algorithm design has a moderate computational complexity, and its parameters can be adjusted online or offline, meeting the requirements of practical engineering projects for fast and stable registration.

[0163] See also Figure 2 , which is a structural diagram of a two-stage infrared and visible light image registration system provided in an embodiment of the present application; the system includes an image acquisition module 1, a preprocessing module 2, a two-stage registration processing module 3, and a registration result output module 4.

[0164] Among them, the image acquisition module 1 is used to acquire infrared images and visible light images;

[0165] Preprocessing module 2: used for preprocessing infrared images and visible light images by grayscale conversion and Gaussian smoothing;

[0166] Two-stage registration processing module 3: includes a contour corner matching module 301 and an unsupervised optical flow module 302; the contour corner matching module 301 is used to extract significant corners on the image contour and perform feature matching based on local geometric direction information to complete the preliminary alignment of the infrared image and the visible light image; the unsupervised optical flow module 302 uses an unsupervised optical flow network based on deep learning to accurately align the infrared image and the visible light image;

[0167] Registration result output module 4: used to output the infrared and visible light image pair results after two-stage registration.

[0168] In the technical solution provided in the embodiments of the present application, the registration system adopts a coarse-to-fine, global-first-local registration strategy, first estimating the global affine transformation using sparse shape features, and then using dense optical flow to perform pixel-level compensation for residual nonlinear misalignment to achieve high-precision alignment.

[0169] See also Figure 3 , a two-stage infrared and visible light image registration device provided in an embodiment of the present application; the device includes a memory 5 and a processor 6;

[0170] The memory 5 is used to store computer programs and data; the processor 6 is used to implement the steps of the above-mentioned two-stage infrared and visible light image registration method when executing the computer program.

[0171] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A two-stage infrared and visible light image registration method, characterized in that: The steps include: Step S1, image preprocessing: grayscale conversion and Gaussian smoothing are performed on the input infrared image and visible light image; Step S2, contour extraction: extract edge contours from the pre-processed infrared image and visible light image to obtain an infrared image contour map and a visible light contour map, respectively; Step S3, corner point detection: extracting corner points with significant geometric structures on each contour image; Step S4, feature description and matching: define a joint descriptor and use the nearest neighbor search strategy to perform preliminary point pair matching to obtain a set of candidate matching pairs; Step S5, affine transformation estimation: screening candidate matching pairs to eliminate false matches and estimating the affine transformation matrix to obtain a coarse registration image; Step S6, multi-scale optical flow estimation: the coarsely registered visible light image and the infrared original image are stitched together and fed into an unsupervised optical flow estimation network to obtain a fine-scale optical flow field; Step S7, optical flow constraint and loss function: adaptive training is performed by jointly minimizing photometric consistency loss, smoothness loss, structural similarity loss and distillation loss function; Step S8, reverse resampling: performing pixel-level reverse mapping on the coarsely registered visible light image and combining it with the fine-scale optical flow field to obtain the final optical flow field; Step S9, fusion and error evaluation: calculate the error map and mean square error, and evaluate the calculated registration results.

2. A two-stage infrared and visible light image registration method according to claim 1, characterized in that: Step S3 includes the following steps: Step S301: Set infrared image The contour point sequence is , visible light image The contour point sequence is , using the boundary tracking algorithm to extract the contour connected domain into an ordered chain, for each point on the chain and , and define its normal direction as: (1) In formula (1), is the centroid coordinate of the current contour end; Step S302, in order to measure the point and The degree of angle mutation is defined as the angle response function and for: (2) In formula (2), is the local window width of the corner response; like Exceeding the set threshold , then it is believed that is a candidate corner point; if Exceeding the set threshold , then it is believed that is the candidate corner point.

3. The two-stage infrared and visible light image registration method according to claim 1, characterized in that: Step S4 includes the following steps: Step S401: define the infrared image to take into account both local structure and direction information Descriptor and visible light images Descriptor : (3) In formula (3), is the contour direction angle of the infrared image, is the corner response intensity of the infrared image; is the contour direction angle of the visible light image, is the corner point response intensity of visible light; Step S402: infrared image The set of corner points , and visible light images The set of corner points Construct the corresponding descriptor set and calculate the similarity by Euclidean distance: ， ， (4)； Step S403: using the nearest neighbor search strategy, The set of corner points For each corner point, search the visible light image with the closest Euclidean distance Perform preliminary point pair matching on the corner points to obtain a set of candidate matching pairs .

4. The two-stage infrared and visible light image registration method according to claim 1, characterized in that: Step S5 includes the following steps: Step S501: Based on the RANSAC algorithm, assume that a candidate affine transformation matrix H satisfies the following transformation relationship: ， Where a, b, c, and d are the coefficients of linear transformation; 、 is the translation amount; For visible light images The midpoint coordinates of For visible light images The midpoint coordinates after transformation, , ; Step S502: determine the final affine transformation parameters by minimizing the following residual sum of squares: ； Step S503: After the affine transformation matrix H is obtained, it is used to transform the visible light image Perform preliminary registration to obtain a coarse registration image : ， Where H is the affine transformation matrix defined in step S501, Coarsely register the image Serves as the input basis for unsupervised optical flow registration.

5. The two-stage infrared and visible light image registration method according to claim 1, characterized in that: Step S6 includes the following steps: Step S601: Multi-scale optical flow estimation is used to roughly register the visible light image. And the original infrared image Splice to , R represents the number of channels of the image, L represents the height of the image, and W represents the width of the image; Step S602: Send it to the unsupervised optical flow estimation network and let each layer iteratively output : ， ，i=1,2,3 in, , and upsample by scale ； Finally, we get the fine-scale optical flow field ； in, Represents the optical flow field of the iterative output of the i-th layer; Represents the fusion mask of the iterative output of the i-th layer; Represents the optical flow update amount of the i-th layer; represents the update amount of the fusion mask of the i-th layer; Represents the two components of optical flow; Indicates the height of the i-th layer; represents the width of the i-th layer; 、 represent the displacement in the x-axis and y-axis directions respectively.

6. The two-stage infrared and visible light image registration method according to claim 1, characterized in that: In step S7, the comprehensive goal of the combined photometric consistency loss, smoothness loss, structural similarity loss, and distillation loss function is: ； in, 、 、 is any positive real number; is the photometric consistency loss function; is the structural similarity loss function; is the distillation loss function; is the smoothness loss function.

7. The two-stage infrared and visible light image registration method according to claim 1, characterized in that: In step S8, the coarse registration of the visible light image Perform pixel-level reverse mapping to obtain the final optical flow field F: ； Select bilinear: ； in, , ; Weight ; m and n represent integer offsets in the x-axis and y-axis directions respectively.

8. The two-stage infrared and visible light image registration method according to claim 1, characterized in that: In step S9, the error graph is: ; The mean square error is: ; If the error value of each pixel in the error map is ≤ , then the registration is judged to be successful; Otherwise, prompt manual compound or re-run; µ is the set threshold; The final output is the high-resolution visible light image after registration , fusion mosaic graph and error heat map visualization results.

9. A two-stage infrared and visible light image registration system for implementing the method of claim 1, characterized in that: The system comprises: Image acquisition module (1): used for acquiring infrared images and visible light images; Preprocessing module (2): used for preprocessing infrared images and visible light images by grayscale conversion and Gaussian smoothing; The two-stage registration processing module (3) includes a contour corner matching module (301) and an unsupervised optical flow module (302); the contour corner matching module (301) is used to extract significant corners on the image contour and perform feature matching in combination with local geometric direction information to complete the preliminary alignment of the infrared image and the visible light image; the unsupervised optical flow module (302) uses an unsupervised optical flow network based on deep learning to accurately align the infrared image and the visible light image; Registration result output module (4): used for outputting the infrared and visible light image pairing results after two-stage registration.

10. A two-stage infrared and visible light image registration device, characterized in that: include: Memory (5) for storing computer programs and data; A processor (6) is configured to implement the steps of the two-stage infrared and visible light image registration method according to any one of claims 1 to 8 when executing a computer program.

Images