Infrared and visible light image sequence registration method based on optical flow matching

Abstract

The invention discloses an image sequence registration method based on optical flow matching, which comprises the following steps of: firstly extracting a moving object in an image and a moving optical flow of a background, then evaluating the significance of the optical flow, then constructing a homography estimation network based on the optical flow to calculate a homography matrix, and finally adaptively updating the homography matrix by using a smoothing algorithm. Therefore, a more stable image sequence registration result is obtained. The result shows that the method can obtain an accurate and robust registration result.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an infrared and visible light image sequence registration method based on optical flow matching. Background Art

[0002] With the steady development of hardware and software industries, the ability to collect information using sensors, as well as transmit and process information, is also increasing. In this context, vision-based sensors are widely used because they can provide rich environmental information. Image registration technology can solve the alignment problem between multiple images, making the images consistent in space, providing a basis for subsequent analysis and processing, and greatly broadening the information expression of multiple images.

[0003] For the multi-source sensor system of vision, infrared and visible light images can be obtained through relatively simple equipment, and the most typical one is the fusion of infrared and visible light images. Due to the different imaging mechanisms of the two, visible images usually have higher spatial resolution and image contrast, which are suitable for human visual perception, but they are easily affected by harsh conditions, such as insufficient brightness, heavy rain, haze and other special climates. However, infrared images happen to have better scene anti-interference ability, and can more significantly display targets with higher temperatures than the environment, such as pedestrians. However, infrared images usually have low resolution and poor image details. The registration of these two types of images can be studied to better fuse the two, so that multiple information can be displayed on one image, highlighting the target, with richer details than a single image and the ability to resist harsh environments. Therefore, the registration of infrared and visible light images is to optimally align the infrared and visible light images in the same scene in space, so that the final fused image is more information-rich, which is more conducive to human eye recognition and automatic detection of machines, human observation and aesthetics, and subsequent image processing by computers.

[0004] Compared with image registration, image sequence registration adds information on the time scale and has broad application prospects in monitoring systems, medical imaging, film and television production, augmented reality and virtual reality, but it also brings challenges in computational complexity and robustness.

[0005] The general steps of image registration can be roughly divided into feature detection, feature matching, model estimation and coordinate interpolation. According to the different registration principles, traditional image registration methods are mainly divided into grayscale information-based and image feature-based methods. Grayscale information-based methods aim to use the grayscale information of the source image to estimate the transformation model by measuring similarity. This type of method has high accuracy but large computational complexity. The image feature-based method is the most widely used and mature method. This method uses significant features in the image such as points, lines, contours, etc. for registration. The computational complexity is low, but the image quality requirements are high.

[0006] In recent years, deep learning has become an increasingly important technology in the field of medical image registration and homologous image registration. DeTone et al. first proposed a neural network-based supervised scheme to estimate homography. In order to avoid artificially constructing a large number of data sets, Nguyen et al. proposed an unsupervised homography estimation method that combines direct linear transform (DLT) and spatial transformer network for image registration. In order to deal with depth differences and moving objects in real-world data, Zhang et al. proposed a new unsupervised deep homography method and introduced an outlier mask to focus on reliable registration areas. However, most of these methods are applied to homologous image registration tasks, and there is little research on the registration of infrared and visible light images.

[0007] Current image registration has gradually expanded to the registration of image sequences. The trajectory and optical flow of moving targets in image sequences are two commonly used motion information in registration tasks. Torabi et al. solved the problem of thermal visible image sequence registration based on the matching of moving target trajectories that track the target's centroid or vertex. However, this method tracks local image features and produces very sparse information representation for the registration task, which easily leads to poor robustness. Zhang et al. used optical flow information for infrared and visible light image sequence registration, matching a pair of pixels by measuring the similarity of the optical flow vector sequence at each pixel, thus obtaining a homography matrix for a pair of image sequences. However, this method cannot handle more complex situations, such as the situation where the depth of the scene where the target is located changes over time in the image sequence captured by the mobile platform. Summary of the invention

[0008] In order to address the deficiencies of the above-mentioned prior art, the present invention aims to propose a homography estimation method based on motion representation, which is suitable for end-to-end unsupervised learning paradigm to deal with the problem of registration of infrared and visible light image sequences in complex scenes. This method significantly improves the accuracy and reliability of registration by using dense optical flow information extracted by FlowNet as the basis for registration, which is superior to traditional features. At the same time, the present invention designs an optical flow-based homography estimation network for calculating the homography matrix, which can evaluate the importance of optical flow from local and global scales, thereby retaining the key information required for the registration task. In addition, the present invention also introduces a smooth update algorithm based on optical flow map loss to optimize the homography matrix, thereby significantly improving the stability of the registration results.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A method for registering infrared and visible light image sequences based on optical flow matching, comprising:

[0011] Step 1: extract the optical flow of the moving target, and further obtain the motion optical flow of the moving target relative to the background to eliminate the influence of the motion of the imaging platform.

[0012] Step 1.1: Read the infrared image frame sequence V captured by the multi-sensor imaging platform T =(f T,1 ,f T,2 ,…,f T,N ) and visible light image frame sequence V V =(f V,1 ,f V,2 ,…,f V,N ), the nth infrared image frame in the sequence is represented by f T,n , the nth visible light image frame is represented as f V,n , f V,n With f T,n correspond.

[0013] Step 1.2: Input two adjacent image frames of the infrared and visible light image sequences into the FlowNet model to extract the optical flow and obtain the infrared and visible light optical flow field sequences OF T =(of T,1 ,of T,2 ,…,of T,N-1 ) and OF V =(of V,1 ,of V,2 ,…,of V,N-1 ), where the optical flow field of the nth infrared image frame is represented by T,n , the optical flow field of the nth visible light image frame is expressed as V,n , each pixel in the optical flow field corresponds to a two-dimensional optical flow vector v of , each pixel in the optical flow field corresponds to a two-dimensional optical flow vector v of .

[0014] Step 1.3: Convert the optical flow vector v of Convert to polar coordinates, that is, v of =(r of ,θ of ) T , based on the phase angle θ of the optical flow vector of and amplitude r of Color code to get T,n and of V,n The significant optical flow map in RGB color space is denoted as Iof T,n and Iof V,n , the corresponding sequence is represented as IOF T =(Iof T,1 ,Iof T,2,…,Iof T,N-1 ) and IOF V =(Iof V,1 ,Iof V,2 ,…,Iof V,N-1 ).

[0015] Step 1.4: Estimate the optical flow vector v of The distribution of the corresponding optical flow field in an image frame.

[0016] Step 1.5: Identify the optical flow distribution of the background.

[0017] Step 1.6: Extract the foreground optical flow, subtract the background optical flow, and obtain the motion optical flow relative to the background.

[0018] Step 2: Evaluate the optical flow saliency and retain image frames containing significant optical flow information.

[0019] Step 2.1: Introduce the local optical flow saliency metric (LocalS) to evaluate the infrared and visible optical flows in the local area.

[0020] Step 2.2: Introduce the global optical flow saliency measure (GlobalS) to evaluate a pair of infrared and visible light optical flows. If both are significant, they will be used for subsequent calculations.

[0021] Step 3: Construct a homography estimation network based on optical flow and calculate the homography matrix.

[0022] Step 3.1: Since the resolution and image size of infrared and visible light image frames are different, it is necessary to perform preprocessing such as cropping and resizing the input to make them consistent in size. V,n Apply center cropping to get the same resolution as Iof T,n Same size H T ×W T Then Iof T,n and Iof V,n Synchronous adjustment to H of ×W of .

[0023] Step 3.2: Build the OF-HGEN network structure and input the significant optical flow map Iof T,n and Iof V,n , output four coordinate offsets, used to construct the 4-point displacement parameter matrix H 4pt,n .

[0024] Step 3.3: Obtain the corresponding Iof through linear transformation T,n and Iof V,n The homography matrix H between of,n .

[0025] Step 3.4: According to the homography matrix H of,n , Iof T,n Convert to Iof V,n In the coordinate system, we get Iof′ T,n =Warp(Iof T,n ,H of,n ), Warp(·,·) represents the homography transformation.

[0026] Step 3.5: Construct the loss function At the same time, avoid the feature map approaching zero and construct The final loss function is defined as where λ is a hyperparameter.

[0027] Step 3.6: Repeat steps 3.2 to 3.5 until the number of calculations on the entire registration image training set reaches the set value, and the trained registration network is obtained. The output is represented by H of,n The corresponding f obtained by conversion relationship T,n and f V,n The homography matrix H between n .

[0028] Step 4: Apply a smoothing algorithm to adaptively update the homography matrix, and realize the registration of infrared and visible light image sequences based on the updated homography matrix.

[0029] This step uses a smoothing algorithm to estimate H frame by frame. of,n Make adjustments and judge according to step 2 T,n and of V,n If it is considered insignificant, it will not be included in the calculation of OF-HGEN and will be directly taken If it is considered significant, it is input into the OF-HGEN network to obtain H of,n ,pass Update η is a parameter that controls smoothness, usually η = 0.9; H of,n Replace with Based on step 3.6, the updated

[0030] In one embodiment, the calculation of step 1 is as follows:

[0031] In the scene captured by the mobile platform, the background and foreground objects are usually in motion, and the proportion of foreground objects is relatively low. Therefore, the optical flow corresponding to the background area occupies a major proportion, and its optical flow is consistent. The present invention clusters the optical flow through a clustering algorithm to distinguish the optical flow of the background area and the foreground area. Given the optical flow vector v at each pixel position of ∈R2 , K-means clustering algorithm is used to cluster all optical flow vectors into K clusters. K is determined by the complexity of the scene and can generally be 3 to 5. Assume that the kth cluster contains n k Optical flow vectors, denoted as set Ω k , then the weight of the kth cluster is calculated as The mean is The variance is

[0032] In one embodiment, the background optical flow is identified according to ω k / |Σ k The clusters are sorted by the value of |, and the mean μ1 of the first cluster is selected as the background optical flow distribution.

[0033] In one embodiment, the motion optical flow of the foreground relative to the background is obtained by subtracting the mean optical flow of the background from each optical flow vector, that is,

[0034] In one embodiment, the calculation steps of step 2 are as follows:

[0035] (1) Optical flow field of T,n and of V,n Perform denoising. If the optical flow vector v corresponding to a pixel point of =(r of ,θ of ) T The amplitude r of <γ, where γ is the threshold, then the optical flow vector is set to zero.

[0036] (2) Optical flow field of T,n and of V,n The denoised result is then segmented into multiple connected regions using connected domain detection. Specifically, adjacent non-zero optical flow vectors are represented to form m T and m V connected regions, denoted as {R T,i |i=1,2,…,m T} and {R V,i |i=1,2,…,m V}, each region corresponds to a moving foreground object.

[0037] (3) Calculate each region R T,i and R V,i The average amplitude of the optical flow is denoted as like All below the threshold Then the optical flow of the area is set to zero; otherwise, the isotropy of the optical flow in the area is analyzed and H E express.

[0038] (4) The optical flow vector v of =(r of ,θ of ) T The phase angle θ of ∈[0°,360°) is divided into B intervals, and each area R is calculated T,i and R V,i The phase angle histogram of the internal optical flow vector is denoted as [h(1),h(2),…,h(B)]; definition Where |R| represents the region R T,i and R V,i The number of pixels in H. E Above threshold It is believed that the optical flow is randomly distributed in this area and cannot provide sufficient evidence for registration, so the optical flow vector of this area is set to zero; if H E Below threshold This area is considered important.

[0039] (5) GlobalS is defined as the sum of pixels in the image frame that have non-zero optical flow vectors in LocalS evaluation. gof If S gof Above threshold Consider it significant.

[0040] In one embodiment, in step 3.1, since the resolution and image size of infrared and visible light images are different, it is necessary to perform preprocessing such as cropping and resizing on the input. V,n Apply center cropping to get the same resolution as Iof T,n Same size H T ×W T Then Iof T,n and Iof V,n Synchronous adjustment to H of ×W of OF-HGEN consists of three parts: FeatNet, MaskNet and the backbone network ResNet. The FeatNet network is used to extract features, and the input is the optical flow map Iof T,n and Iof V,n , the output is the corresponding feature map F T,n and F V,n ; MaskNet is used to generate weight maps, and the input is the optical flow map Iof T,n and Iof V,n , the output is the corresponding weight matrix M T,n and M V,n , which measures the importance of the feature map area; the backbone network ResNet is used for calculation, and the input is in Indicates that the elements are multiplied, and the output is a 4-point displacement parameter matrix H 4pt,n .

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] First, we propose an end-to-end unsupervised learning method for infrared and visible image sequence registration. We use optical flow information as the basis for registration to register infrared and visible image sequences of static and mobile platforms. The motion feature representation based on dense optical flow eliminates the huge difference between infrared and visible light, thus producing accurate and robust registration results.

[0043] Second, the optical flow saliency evaluation and smoothing algorithm are applied. The former enhances the robustness of the optical flow calculation effect, and the latter improves the stability of the homography registration results in the temporal direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The overall framework diagram of the given scheme. The input is the infrared and visible light image sequence to be registered, and the output is the registered image sequence. It consists of four parts, namely, a dense optical flow estimation module for calculating the motion information of moving targets, an evaluation module for measuring the importance of optical flow at local and global scales, and an optical flow-based homography estimation network (OF-HGEN) for calculating the homography matrix, where the input is a pair of optical flow images and the output is the homography matrix. The smoothing algorithm based on the optical flow map error update is used to update the homography matrix along time.

[0045] Figure 2 This is the registration result of the infrared and visible light image sequences in the embodiment of the present invention. DETAILED DESCRIPTION

[0046] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0047] Infrared and visible light image registration aims to estimate the transformation of the best spatial alignment between two types of images taken at the same scene, so that the scene information represented by the images at the same pixel position of the two types of images is consistent, which is conducive to subsequent image fusion and other tasks. As an important process in multi-image modality fusion, image registration is to find a set of geometric transformations, usually homography matrices, to transform the image to be registered to the target's spatial coordinate system, so that it has the same content representation as the target image at the same position. Image sequence registration can be regarded as the registration of a series of image sequences, and can be registered according to the registration method for a single image. The advantage of image sequences over single images is that the former contains information in the time dimension. Because the motion characteristics they represent are highly consistent for infrared and visible light, they are often used as the basis for registration.

[0048] Due to the different imaging mechanisms, the content expression of thermal imaging and visible light imaging has significant differences in color and texture, which is more obvious in low-light environments. Therefore, the field of image registration between thermal imaging and visible light imaging is still very challenging. Some methods try to estimate the registration parameters from sparse evidence, but the results obtained are less robust. At present, most of the deep learning-based methods are used in the task of homologous image registration. Based on this, the present invention provides an infrared and visible light image fusion method based on optical flow matching. The overall architecture is as follows: Figure 1 , the steps are as follows:

[0049] Step 1: Extract the optical flow of the moving target and further obtain its motion optical flow relative to the background to eliminate the influence of the imaging platform motion.

[0050] Step 1.1: Read a pair of infrared image frame sequences V captured by a multi-sensor imaging platform in time synchronization T =(f T,1 ,f T,2 ,…,f T,n ,…,f T,N ) and visible light image frame sequence V V =(f V,1 ,f V,2 ,…,f V,n ,…,f V,N ), f T,n represents the nth infrared image frame, f V,n represents the nth visible light image frame, and f T,n Correspondingly, N is the total number of image frames in a single sequence, and the image size in the infrared image frame sequence is H T ×W T , the image size in the visible light image frame sequence is H V ×W V For image registration research, based on the "almost planar" assumption, the homography transformation can be simplified to: where u′ and v′ represent the horizontal and vertical coordinates of the pixel in the target image, respectively; u and v represent the horizontal and vertical coordinates of the pixel in the source image, respectively; and h 11 、h 13 、h 22 、h 23 Represents the parameters of the homography transformation.

[0051] Step 1.2: Input two adjacent image frames in each sequence into the FlowNet model to extract the optical flow, input V T With V V The two adjacent frames in the image are sent to the FlowNet model, and the infrared optical flow field sequences OF corresponding to the infrared image frames are output respectively. T =(ofT,1 ,of T,2 ,…,of T,n ,…,of T,N-1 ) and the visible light optical flow field sequence OF corresponding to the visible light image frame V =(of V，1 ,of V，2 ,…,of V,n ,…,of V，N-1 ), where of T,n and of V,n They represent the optical flow fields of the nth infrared image frame and the nth visible light image frame respectively. The optical flow field contains the two-dimensional optical flow vector v at each pixel position. of ;

[0052] Step 1.3: Convert the optical flow vector v of Convert to polar coordinates, that is, v of =(r of ,θ of ) T , based on the optical flow phase angle θ of and amplitude r of Color coding is performed to obtain the optical flow field of T,n and of V,n The significant optical flow map in RGB color space is denoted as Iof T,n and Iof V,n , as a graphical representation of the optical flow calculated for the nth and n+1th image frames. The overall infrared salient optical flow map is denoted as IOF T =(Iof T,1 ,Iof T,2 ,…,Iof T,n ,…,Iof T,N ), the overall visible light salient optical flow map is denoted as IOF V =(Iof V,1 ,Iof V,2 ,…,Iof V,n ,…,Iof V,N ).

[0053] Step 1.4: Estimate the optical flow vector v of The distribution of the corresponding optical flow field in an image frame, given the optical flow vector v at each pixel position of ∈R 2 , K-means clustering algorithm is used to cluster all optical flow vectors into K clusters. K is determined by the complexity of the scene and can generally be 3 to 5. Assume that the kth cluster contains n k Optical flow vectors, denoted as set Ω k , then the weight of the kth cluster is calculated as The mean is The variance is

[0054] Step 1.5: Identify the optical flow distribution of the background. Generally, the optical flow vector at the background pixel is uniform and occupies a large proportion in the entire frame. k / |Σ k The value of | sorts the clusters after clustering, and selects the mean μ1 of the first cluster as the background optical flow;

[0055] Step 1.6: Extract the foreground optical flow, subtract the background optical flow from each optical flow vector, and obtain the motion optical flow of the foreground relative to the background, that is,

[0056] Step 2: Evaluate the optical flow saliency and retain the image frames containing significant optical flow information

[0057] Step 2.1: Introduce the local optical flow saliency measure (LocalS) to evaluate the infrared and visible optical flows in the local area. First, the optical flow field of T,n and of V,n Perform denoising and set the optical flow vectors with amplitudes below the threshold γ to zero; perform connected domain detection on the optical flow field and divide it into multiple connected regions; calculate the average amplitude of the optical flow in each region If the value is below the threshold Then the optical flow of the area is set to zero; analyze the isotropy H of the optical flow in each area E , if H E Below threshold Then the area is considered important;

[0058] Step 2.2: Introduce the global optical flow saliency measure (GlobalS), which is defined as the sum of pixels in the image that have non-zero optical flow vectors in the LocalS evaluation. gof If S gof Above threshold It is considered significant. When applied, a pair of infrared and visible light flows are evaluated separately. If both are significant, they are used for subsequent calculations;

[0059] Step 3: Construct a homography estimation network based on optical flow and calculate the homography matrix. The specific process can be described as follows.

[0060] Step 3.1: Since the resolution and image size of infrared and visible light images are different, the input needs to be preprocessed by cropping and resizing. Apply center cropping to obtain the same Same size H T ×W T Then and Synchronous adjustment to Hof ×W of ;

[0061] Step 3.2: Construct the OF-HGEN network structure, which consists of three parts: FeatNet, MaskNet and backbone network ResNet. The FeatNet network is used to extract features, and the input is the salient optical flow map Iof T,n and Iof V,n , the output is the feature map F T,n and F V,n ; MaskNet is used to generate weight maps, and the input is the optical flow map Iof T,n and Iof V,n , the output is the weight matrix M T,n and M V,n , which measures the importance of the feature map area; the backbone network ResNet is used for calculation, and the input is in Indicates that the elements are multiplied, and the output is a 4-point displacement parameter matrix H 4pt .

[0062] Step 3.3: Input salient optical flow map Iof T,n and Iof V,n , output four coordinate offsets, used to construct the 4-point displacement parameter matrix H 4pt,n , H 4pt,n Expressed as in Among them, Δu k =u′ k -u k and Δv k =v′ k -v k , where (u k ,v k ) T and(u′ k ,v′ k ) T Represent the vertex coordinates of the rectangle in the visible light image and infrared image, k = 1, 2, 3, 4. Through DLT linear transformation, the corresponding Iof is obtained T,n and Iof V,n The homography matrix H between of,n ;

[0063] Step 3.4: According to the homography matrix H of,n , will Iof T,n Convert to Iof V,n In the coordinate system, we get Iof′ T,n =Warp(Iof T,n ,H of,n), Warp(·,·) represents homography transformation;

[0064] Step 3.5: Construct the loss function At the same time, avoid the feature map approaching zero and construct The final loss function is defined as Where λ is a hyperparameter;

[0065] Step 3.6: Repeat steps 3.3 to 3.6 until the number of calculations on the entire registration image training set reaches the set value, and the trained registration network is obtained. Since the optical flow map is cropped and resized in the preprocessing stage, H is obtained. of,n Then, we get the corresponding f T,n and f V,n The homography matrix H between n .

[0066]

[0067] in: is the matrix H of,n The (i,j)th item, Δx=(W V -W T ) / 2 and Δy=(H V -H T ) / 2, are the offsets of the upper left corner of the cropped optical flow image relative to the initial image, and k x =W T / W of , k y =H T / H of Respectively represent the scaling ratio of the optical flow map in the horizontal and vertical directions;

[0068] Step 4: Apply a smoothing algorithm based on the error update of the optical flow map to adaptively update the homography matrix. of,n Make adjustments and judge according to step 2 T,n and of V,n If it is considered insignificant, it will not be included in the calculation of OF-HGEN and will be directly taken If it is considered significant, it is input into the OF-HGEN network to obtain H of,n ,pass Update η is a parameter that controls smoothness, usually η = 0.9; H of,n Replace with Based on step 3.6, the updated

[0069] Figure 2The registration results of infrared and visible light image sequences are shown. The original infrared and visible light image sequences are from drone footage, with resolutions of 512×640 and 1080×1920 respectively. The resolution of the visible light image sequence is adjusted to 512×910, i.e. H T =512,W T =640, H V =512,W V =910; Input the infrared and visible light image sequences into FlowNet, and after optical flow saliency evaluation, obtain the salient optical flow map Iof T,n and Iof V,n , its resolution is adjusted to 360×480, that is, H of =360,W of =480. According to the above steps, the homography matrix after smoothing is obtained Figure 2 According to the homography matrix The infrared and visible light images are superimposed, and the results show that the present invention can well complete the registration task of infrared and visible light image sequences.

Claims

1. A method for registering infrared and visible light image sequences based on optical flow matching, characterized in that: The steps include: Step 1: extract the optical flow of the moving target and obtain its motion optical flow relative to the background to eliminate the influence of the motion of the imaging platform; Step 2, evaluate the optical flow saliency and retain the image frames containing significant optical flow information; Step 3, construct a homography estimation network based on optical flow and calculate the homography matrix; Step 4: Adaptively update the homography matrix using a smoothing algorithm, and realize registration of infrared and visible light image sequences based on the updated homography matrix.

2. The infrared and visible light image sequence registration method based on optical flow matching according to claim 1, characterized in that: The step 1 extracts the optical flow of the moving target and obtains its motion optical flow relative to the background. The implementation method is as follows: Step 1.1, read a pair of infrared image sequences and visible light image sequences captured synchronously by a multi-sensor imaging platform, the nth infrared image frame in the sequence is represented as f T,n , the nth visible light image frame is represented as f V,n , f V,n With f T,n correspond; Step 1.2: Input two adjacent image frames in each sequence into the FlowNet model to extract the optical flow, and obtain the infrared optical flow field sequence and the visible light optical flow field sequence, where the optical flow field of the nth infrared image frame is represented by T,n , the optical flow field of the nth visible light image frame is expressed as V,n , each pixel in the optical flow field corresponds to a two-dimensional optical flow vector v of ; Step 1.3, the optical flow vector v of Convert to polar coordinates (r of ,θ of ) T , based on the optical flow phase angle θ of and amplitude r of Color code to get T,n and of V,n The significant optical flow map in RGB color space is denoted as Iof T,n and Iof V,n ; Step 1.4, estimate the optical flow vector v of The distribution of the corresponding optical flow field in an image frame; Step 1.5, identify the optical flow distribution of the background; Step 1.6, extract the foreground optical flow, subtract the background optical flow, and obtain the motion optical flow relative to the background.

3. The infrared and visible light image sequence registration method based on optical flow matching according to claim 2, characterized in that: The step 1.4 estimates the distribution of the optical flow field corresponding to the optical flow vector in an image frame, and the implementation method is as follows: K-means clustering algorithm is used to cluster all optical flow vectors into K clusters. The kth cluster contains n k optical flow vectors, denoted as set Ω k , the weight of the kth cluster is The mean is The variance is The step 1.5, identifying the optical flow distribution of the background, is implemented as follows: According to ω k / |Σ k The value of | sorts the clusters after clustering, and selects the mean μ1 of the first cluster as the background optical flow; In step 1.6, the background optical flow is subtracted from each optical flow vector to obtain the motion optical flow of the foreground relative to the background, that is, 4. The infrared and visible light image sequence registration method based on optical flow matching according to claim 1, characterized in that: The step 2, evaluating the optical flow saliency and retaining the image frames containing significant optical flow information, is implemented as follows: Step 2.1, introduce the local optical flow saliency metric (LocalS) to evaluate the infrared and visible optical flows in the local area; In step 2.2, the global optical flow saliency measure (GlobalS) is introduced to evaluate a pair of infrared and visible light optical flows separately. If both are significant, they will be used for subsequent calculations.

5. The infrared and visible light image sequence registration method based on optical flow matching according to claim 4, characterized in that: In step 2.1, firstly, the optical flow field of T,n and of V,n De-noising is performed, and the optical flow vectors with amplitudes lower than the threshold γ are set to zero; then the denoised optical flow field is detected for connected domains and divided into multiple connected regions; and the average amplitude of the optical flow in each region is calculated. like Below threshold Then the optical flow of the area is set to zero; otherwise, the isotropic nature of the optical flow in the area is analyzed. E , if H E Below threshold Then the area is considered important. In step 2.2, the global optical flow saliency is defined as the sum of the pixels of the image frame with non-zero optical flow vectors in the local optical flow saliency evaluation. gof , if S gof Above threshold Consider it significant.

6. The infrared and visible light image sequence registration method based on optical flow matching according to claim 1, characterized in that: The step 3 is to construct a homography estimation network based on optical flow and calculate the homography matrix. The implementation method is as follows: Step 3.1, adjusting the sizes of the infrared image frame and the visible light image frame to be consistent; Step 3.2, Iof T,n and Iof V,n Input OF-HGEN network structure and output four coordinate offsets to construct the 4-point displacement parameter matrix H 4pt,n ; Step 3.3, through linear transformation, obtain the corresponding n-th infrared salient optical flow map Iof T,n And the nth visible light salient optical flow map Iof V,n The homography matrix H between of,n ; Step 3.4, according to the homography matrix H of,n , Iof T,n Convert to Iof V,n In the coordinate system, we get Iof T ' ,n =Warp(Iof T,n ,H of,n ), Warp(·,·) represents homography transformation; Step 3.5, construct the loss function At the same time, avoid the feature map approaching zero and construct The final loss function is defined as Where λ is a hyperparameter; Step 3.6, repeat steps 3.2 to 3.5 until the number of calculations on the entire registration image training set reaches the set value, and the trained registration network is obtained. The output is represented by H of,n The corresponding f obtained by conversion relationship T,n and f V,n The homography matrix H between n .

7. The infrared and visible light image sequence registration method based on optical flow matching according to claim 6, characterized in that: In step 3.1, Apply center cropping to obtain the same Same size H T ×W T , then and The size is adjusted to H of ×W of .

8. The infrared and visible light image sequence registration method based on optical flow matching according to claim 6, characterized in that: In step 3.2, the OF-HGEN network structure consists of three parts: FeatNet, MaskNet and the backbone network ResNet. The FeatNet network is used to extract features, and the input is Iof T,n and Iof V,n , the output is the corresponding feature map F T,n and F V,n ; MaskNet is used to generate a weight map to measure the importance of the feature map area, and the input is Iof T,n and Iof V,n , the output is the corresponding weight matrix M T,n and M V,n ; The backbone network ResNet is used for calculation, and the input is in Indicates that the elements are multiplied, and the output is a 4-point displacement parameter matrix H 4pt,n , expressed as: Where Δu k =u′ k -u k and Δv k =v′ k -v k ,(u k ,v k ) T and(u′ k ,v′ k ) T Represent the vertex coordinates of the rectangle in the visible light image and infrared image, k = 1, 2, 3, 4; In step 3.6, the homography matrix H is obtained n It is expressed as: in: is the matrix H of,n The (i,j)th item, Δx=(W V -W T ) / 2 and Δy=(H V -H T ) / 2, are the offsets of the upper left corner of the cropped optical flow image relative to the initial image, and k x =W T / W of , k y =H T / H of Respectively represent the scaling ratio of the optical flow map along the horizontal and vertical directions.

9. The infrared and visible light image sequence registration method based on optical flow matching according to claim 6, characterized in that: In step 3.3, the linear transformation adopts the DLT algorithm.

10. The infrared and visible light image sequence registration method based on optical flow matching according to claim 1, characterized in that: The step 4 uses a smoothing algorithm to adaptively update the homography matrix, and the implementation method is as follows: The smoothing algorithm is used to estimate H frame by frame. of,n Make adjustments and judge according to step 2 T,n and of V,n If it is considered insignificant, it will not be included in the calculation of OF-HGEN and will be directly taken If it is considered significant, it is input into the OF-HGEN network to obtain H of,n ,pass Update η is a parameter that controls smoothness; H of,n Replace with Updated based on step 3.6

Images