An image registration method based on the calculation of the external parameters of multispectral images

Abstract

The present invention discloses an image registration method based on multispectral image extrinsic parameter calculation, which relates to the field of computer vision. Aiming at the problem of low accuracy of cross-spectral feature matching in multi-sensor extrinsic parameter estimation of multispectral cameras and multispectral image registration, a multispectral optical flow feature matching method is proposed. A multispectral image sequence is shot in a motion scene, and the optical flow information of each spectral acquisition subsystem imaging is calculated using the adjacent two frames of the same spectrum image sequence. The optical flow information of different spectra is used to calculate the extrinsic parameters between different spectral image acquisition subsystems, and further realize multispectral image registration. The method of the present invention adopts multispectral image sequence acquisition, optical flow calculation, optical flow feature matching, error screening, matching point screening, radiation matrix or flexible deformation field estimation to realize image registration. By converting cross-spectral into homomodal optical flow, the complexity of cross-spectral feature matching is avoided, and the effect of extrinsic parameter estimation and image registration is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision. Specifically, it relates to a method for estimating extrinsic parameters and performing image registration for multispectral cameras. This method, which incorporates optical flow information as a new clue, is of great significance for improving the accuracy of extrinsic parameter estimation and image registration for multispectral cameras. Background Art

[0002] Multispectral image acquisition systems often consist of multiple single-spectral cameras. Estimating extrinsic parameters and registering images between different single-spectral cameras are important preprocessing steps for multispectral image processing. Traditional multispectral camera extrinsic parameter estimation and image registration schemes directly process multispectral images (J.Li, Q.Hu and M.Ai, "RIFT: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform," in IEEE Transactions on Image Processing, vol. 29, pp. 3296-3310, 2020, doi:10.1109 / TIP.2019.2959244.). Due to the radiometric differences between images with different spectra, their image features vary significantly. Traditional single-modal feature matching algorithms, such as those based on SIFT (DGLowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004.) and ORB features (Rublee E, Rabaud V, Konolige K, et al. ORB: An efficient alternative to SIFT or SURF[C] / / 2011 International conference on computer vision. Ieee, 2011: 2564-2571.), cannot accurately perform feature matching, resulting in poor extrinsic parameter estimation and image registration results. However, using radiation-invariant features involves more tedious calculations, and its effect is often only effective in specific scenarios, such as E. Shechtman and M. Irani, "Matching local self-similarities across images and videos," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2007, p. 3. Summary of the Invention

[0003] To address the low accuracy of cross-spectral feature matching in multispectral camera multisensor extrinsic parameter estimation and multispectral image registration, a multispectral optical flow feature matching method is proposed. A multispectral image sequence is captured in a moving scene. The optical flow information of each spectral acquisition subsystem is calculated using the adjacent frames of the same spectrum. This optical flow information of different spectra is then used to calculate the extrinsic parameters between the different spectral image acquisition subsystems, further enabling multispectral image registration.

[0004] The technical solution of the present invention is an image registration method based on multispectral image extrinsic parameter calculation, which includes the following steps:

[0005] Step 1: Calibrate the intrinsic parameter K of each single spectrum camera separately m ,m∈{1,2,…,M}, M is the total number of spectral components;

[0006] Step 2: Acquire a multispectral image sequence Where: I represents the image, the subscripts represent different spectra, j represents the image sequence number, ranging from 1 to N, and N is the sequence length;

[0007] Step 3: Use the optical flow method to calculate the optical flow of each spectral component image separately

[0008] Step 4: Optical Flow Perform feature point matching to obtain matching point pairs in represents the i-th matching point in the j-th optical flow map calculated from the j-th and j+1-th images in the m-th spectral image sequence;

[0009] The feature matching method used uses the amplitude information and angle information of the optical flow, and establishes the feature descriptor D of the amplitude G and angle A information for each optical flow. g and D a :

[0010] D g =S(G)

[0011] D a =S(A)

[0012] S(G) represents the feature descriptor calculation algorithm, which is a mature feature extraction algorithm in the industry, such as the scale-invariant feature transform method SIFT or the ORB method (Oriented FAST and Rotated BRIEF). The SIFT method literature includes DGLowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004.), and the ORB method literature includes Rublee E, Rabaud V, Konolige K, et al. ORB: An efficient alternative to SIFT or SURF [C] / / 2011 International conference on computer vision. Ieee, 2011: 2564-2571.); the feature descriptor D is used as the input. g and D a Direct cascade:

[0013] D=[D g ,D a ]

[0014] For optical flow and optical flow Its feature point sets are P m 、P n , using Euclidean distance as a metric to obtain matching feature point pairs The specific calculation is for P m Each point in Statistical P n The eigenvectors and The Euclidean distance of the feature vector of the point, the point with the smallest distance is taken as P n Neutralization Matched points

[0015] Step 5: Match point pairs using features and K m ,m∈{1,2,…,M} calculates the camera extrinsic parameters Represents the camera extrinsics from single-spectrum camera n to single-spectrum camera m;

[0016] Step 6: Using multiple external references Eliminate the influence of time-unsynchronized image sequences and improve the accuracy of external parameter estimation; Step 7: Use the matched point pairs Calculate the affine transformation matrix or flexible deformation field between different spectral images to achieve registration between spectral images.

[0017] Furthermore, the screening method of step 6 is as follows: first, the candidate external parameter Match point pairs The point in the mth spectral component is projected into the camera coordinate system of the nth spectral component to obtain the reprojected point Calculate the reprojection error:

[0018]

[0019] When the target motion amplitude is less than the set threshold, the depth of the camera and the target is considered unchanged, and the correlation model between the image extrinsic parameters and the optical flow is as follows:

[0020]

[0021] Represents optical flow The center is at the point Optical flow information, K n It is represented by the internal parameter of the nth single-spectrum camera obtained by step 1, R m,n represents the rotation matrix between single-spectrum cameras m and n;

[0022] According to the correlation model between image extrinsic parameters and optical flow, the optical flow deviation is calculated:

[0023]

[0024] Represents the optical flow deviation corresponding to the i-th matching feature point in the j-th optical flow map; the other component of the cost function is the feature deviation that is unrelated to radiation distortion

[0025]

[0026] Where M is a radiation-independent feature extraction method, such as RIFT (J. Li, Q. Hu and M. Ai, "RIFT: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform," in IEEE Transactions on Image Processing, vol. 29, pp. 3296-3310, 2020, doi: 10.1109 / TIP.2019.2959244.); represents the jth image in spectral channel n, Represents the i-th matching feature point in the j-th optical flow map in spectral channel n;

[0027] Sum all the reprojection costs to get Fixed m, n, changed j, selected different Calculate the sum of the reprojection errors E, then the optimal extrinsic parameter estimate is the solution [R, T] that minimizes E n,m ; Using the [R,T] obtained by screening n,m Combined feature bias Filter out the feature point pairs that meet the requirements and remove the inaccurate points. Specifically, calculate the deviation e corresponding to the feature point pairs respectively. i :

[0028]

[0029] The deviation e i Sort and remove point pairs whose deviation is greater than the set threshold.

[0030] Furthermore, the specific calculation step of the optical flow in step 3 is to use the reference frame I r and input frame I i Input them into the encoder respectively to get the image features (L r ,L i ), L r represents the reference frame image features, L i Represents the input frame image features, and then takes the image block with a fixed window of w×h centered on different pixels (T r,k ,T i,k ),k∈{1,2,…,WH}, W and H are the width and height of the entire image, T r,k represents the reference frame image block, T i,k Represents the input frame image block;

[0031]

[0032] The cosine approximation is used to calculate the similarity between each image block and other image blocks in the neighborhood; the offset corresponding to the neighborhood block with the largest similarity is taken as the optical flow.

[0033] Furthermore, the optical flow calculation method in step 3 is the RAFT method or the Lucas-Kanade method. (The RAFT method is a recursive all-pairs field transform for optical flow, Teed Z, Deng J. Raft: Recurrent all-pairs field transforms for optical flow [C] / / Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020: 402-419; the Lucas-Kanade method is the LK optical flow method, Barron JL, Fleet DJ, Beauchemin SS. Performance of optical flow techniques [J]. International journal of computer vision, 1994, 12: 43-77.

[0034] Furthermore, the camera extrinsic parameter estimation method in step 5 is the eight-point method.

[0035] This invention discloses a method for extrinsic parameter estimation and image registration for multispectral images. The method includes multispectral image sequence acquisition, optical flow calculation, optical flow feature matching, error screening, matching point selection, and radiation matrix or flexible deformation field estimation to achieve image registration. By converting cross-spectral optical flow into homomodal optical flow, the complexity of cross-spectral feature matching is avoided, effectively improving the effectiveness of extrinsic parameter estimation and image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the whole method;

[0037] Figure 2 To demonstrate the effect of the visible light-infrared image registration using this patented method, the images are interlaced with visible light and infrared squares for easy comparison. At the edges of the squares, it can be seen that the visible light and infrared images are aligned. DETAILED DESCRIPTION

[0038] For image sequences captured by a single-spectral camera, when there is scene or object motion between frames, the motion vector, or optical flow, of each pixel between adjacent frames under the same spectrum can be calculated. At the same moment in time, the scene or object motion vector is identical, and the optical flow information calculated using image sequences from different single-spectral cameras is similar, resulting in similar image features from the optical flows of different cameras. Using the optical flow information from different single-spectral cameras for feature matching can avoid the difficulty in accurately matching features due to significant feature differences between spectral images. Using established single-modal feature matching algorithms to match the optical flow features of different spectral cameras enables extrinsic parameter estimation for multispectral cameras. Furthermore, since optical flow and images have a one-to-one correspondence, pairs of optical flow feature points can be used to generate a guideline matrix or deformation field data for image registration.

[0039] like Figure 1 As shown in the figure, unlike the traditional multispectral camera extrinsic parameter estimation and image registration that directly uses cross-spectral feature matching, this solution collects image sequences from each single-spectral camera and uses the image sequences to solve the optical flow of the single-spectral camera. It then calculates the image features of the cross-spectral optical flow and converts the cross-spectral image feature matching with huge image feature differences into spectral feature matching with similar image features, thus simplifying the complexity of cross-spectral feature matching. The specific steps are as follows:

[0040] Step 1: Calibrate the internal reference K of each single spectrum camera m ,m∈{1,2,…,M}, taking the visible-infrared multispectral camera as an example, it contains two spectral components, visible light and infrared. The Zhang Zhengyou calibration method can be used to calibrate the visible light camera and the infrared camera to obtain K1 and K2. The Zhang Zhengyou calibration method is a well-known and widely used data calibration method in the industry.

[0041] Step 2: Collect multispectral image sequences. Taking a visible-infrared multispectral camera as an example, the sequence collected is j∈{1,2,3,…,N}, where: I represents the image, the subscripts represent different spectra, j represents the image sequence number, ranging from 1 to N, and N is the sequence length, for example, N=3;

[0042] Step 3: Use the optical flow method to calculate the optical flow of each spectral component image. Taking the visible-infrared multispectral camera as an example, the optical flow calculated using the Lucas-Kanade optical flow algorithm is

[0043] Step 4: Taking the visible-infrared multispectral camera as an example, Perform feature point matching to obtain matching point pairs in represents the i-th matching point in the optical flow calculated from the j-th and j+1-th images in the m-th spectral image sequence;

[0044] Step 5: Use the Eight Point Method and K m ,m∈{1,2} calculate the camera external parameters

[0045] Step 6: Use the RANSAC algorithm to calculate the external parameters Perform screening to eliminate the influence of time-unsynchronized image sequences;

[0046] Step 7: Use the matching point pairs Calculate the affine transformation matrix or flexible deformation field between different spectral images to achieve registration between spectral images;

[0047] In step 2 of the present invention, the registration uses an image sequence rather than a single image. In step 4, the feature point matching uses an image pair other than a cross-modal one. But the optical flow pair of the same modality In step 5, in order to obtain more accurate external parameter estimation, N-1 optical flows are used at the same time The estimated results.

[0048] When using the RANSAC algorithm in step 6, the correlation model F between image extrinsic parameters and optical flow is used. n =f(F m ,[R,T] n,m ) to evaluate the accuracy of the estimation; the accuracy of the estimation is evaluated using features that are independent of radiation distortion between multispectral images. The matching point pairs between the spectral images and the optical flow images of the present invention are the same.

[0049] Using the above steps, the visible light-infrared camera was calibrated and registered, and the reprojection error was statistically used as the accuracy measure of the extrinsic parameter estimation, as shown in the following table

[0050] Table 1. Analysis results of reprojection error of visible light and infrared extrinsic parameter estimation, per pixel

[0051]

[0052] It can be seen that the reprojection error is small, which shows the effectiveness of the external parameter estimation method of this patent. Figure 2 As shown, for the convenience of comparison, the picture is placed alternately with visible light and infrared squares. At the edges of the squares, it can be seen that the visible light and infrared images are aligned, which illustrates the effectiveness of the image registration method of this patent.

[0053] The multispectral image extrinsic parameter estimation and image registration method of the present invention utilizes the optical flow information that is often avoided and ignored in multispectral camera extrinsic parameter estimation and image registration, uses the optical flow method to convert cross-spectral feature matching into same-spectral feature matching, and uses optical flow correlation and optical flow feature matching deviation to perform extrinsic parameter estimation and image registration, which can effectively improve the accuracy of extrinsic parameter estimation and image registration.

Claims

1. An image registration method based on multispectral image extrinsic parameter calculation, the method comprising the following steps: Step 1: Calibrate the intrinsic parameter K of each single spectrum camera separately m ,m∈{1,2,…,M}, M is the total number of spectral components; Step 2: Acquire a multispectral image sequence Where: I represents the image, the subscripts represent different spectra, j represents the image sequence number, ranging from 1 to N, and N is the sequence length; Step 3: Use the optical flow method to calculate the optical flow of each spectral component image separately Step 4: Optical Flow Perform feature point matching to obtain matching point pairs in represents the i-th matching point in the j-th optical flow map calculated from the j-th and j+1-th images in the m-th spectral image sequence; Step 5: Match point pairs using features and K m ,m∈{1,2,…,M} calculates the camera extrinsic parameters Represents the camera extrinsics from single-spectrum camera n to single-spectrum camera m; Step 6: Using multiple external references Eliminate the influence of time-unsynchronized image sequences and improve the accuracy of extrinsic parameter estimation, including: first using candidate extrinsic parameters Match point pairs The point in the mth spectral component is projected into the camera coordinate system of the nth spectral component to obtain the reprojected point Calculate reprojection error Calculate the optical flow deviation based on the correlation model between camera extrinsics and optical flow Another component of the cost function is the characteristic deviation that is independent of radiation distortion Sum all the reprojection costs to get Choose different Calculate E, then the optimal external parameter estimate is the solution [R, T] that minimizes E n,m ; Using the [R,T] obtained by screening n,m Combined feature bias Filter out the feature point pairs that meet the requirements and remove inaccurate points; Step 7: Using Matched Point Pairs Calculate the affine transformation matrix or flexible deformation field between different spectral images to achieve registration between spectral images.

2. The image registration method based on multispectral image extrinsic parameter calculation according to claim 1, characterized in that: The feature matching method used in step 4 uses the amplitude information and angle information of the optical flow, and establishes the feature descriptor D of the amplitude G and angle A information for each optical flow. g and D a :The feature descriptor calculation algorithm is the scale invariant feature transform method SIFT or ORB method; D g =S(G) D a =S(A) S(G) represents the feature descriptor calculation algorithm; The feature descriptor D g and D a Direct cascade: D=[D g ,D a ]; For optical flow and optical flow Its feature point sets are P m 、P n , using Euclidean distance as a metric to obtain matching feature point pairs The specific calculation is for P m Each point in Statistical P n The eigenvectors and The Euclidean distance of the feature vector of the point, the point with the smallest distance is taken as P n Neutralization Matched points .

3. The image registration method based on multispectral image extrinsic parameter calculation according to claim 1, characterized in that: The elimination method of step 6 is as follows: first, the candidate external parameters are used Match point pairs The point in the mth spectral component is projected into the camera coordinate system of the nth spectral component to obtain the reprojected point Calculate the reprojection error: When the target motion amplitude is less than the set threshold, the depth of the camera and the target is considered unchanged. The correlation model between the camera extrinsic parameters and the optical flow is as follows: Represents optical flow The center is at the point Optical flow information, K n It is represented by the internal parameter of the nth single-spectrum camera obtained by step 1, R m,n represents the rotation matrix between single-spectrum cameras m and n; According to the correlation model between camera extrinsics and optical flow, the optical flow deviation is calculated: Represents the optical flow deviation corresponding to the i-th matching feature point in the j-th optical flow map; the other component of the cost function is the feature deviation that is unrelated to radiation distortion Where M is a feature extraction method that is independent of radiation distortion, represents the jth image in spectral channel n, Represents the i-th matching feature point in the j-th optical flow map in spectral channel n; Sum all the reprojection costs to get Fixed m, n, changed j, selected different Calculate the sum of the reprojection costs E, then the optimal extrinsic parameter estimate is the solution [R, T] that minimizes E n,m ; Using the [R,T] obtained by screening n,m Combined feature bias Filter out the feature point pairs that meet the requirements and remove the inaccurate points. Specifically, calculate the deviation e corresponding to the feature point pairs respectively. i : The deviation e i Sort and remove point pairs whose deviation is greater than the set threshold.

4. The image registration method based on multispectral image extrinsic parameter calculation according to claim 1, characterized in that: The specific calculation steps of the optical flow in step 3 are: The reference frame I r and input frame I i Input them into the encoder respectively to get the image features (L r ,L i ), L r represents the reference frame image features, L i Represents the input frame image features, and then takes the image block with a fixed window of w×h centered on different pixels (T r,k ,T i,k ),k∈{1,2,…,WH}, W and H are the width and height of the entire image, T r,k represents the reference frame image block, T i,k Represents the input frame image block; The cosine approximation is used to calculate the similarity between each image block and other image blocks in the neighborhood; the offset corresponding to the neighborhood block with the largest similarity is taken as the optical flow.

5. The image registration method based on multispectral image extrinsic parameter calculation according to claim 1, characterized in that: The optical flow calculation method in step 3 is the RAFT method or the Lucas-Kanade method.

6. The image registration method based on multispectral image extrinsic parameter calculation according to claim 1, characterized in that: The camera extrinsic parameter estimation method in step 5 is the eight-point method.

Images