Image registration method based on multi-spectral image extrinsic parameter calculation

Abstract

The invention discloses an image registration method based on multi-spectral image extrinsic parameter calculation, and relates to the field of computer vision. In order to solve the problem of low cross-spectral feature matching accuracy in multi-spectral camera multi-sensor external parameter estimation and multi-spectral image registration, the invention provides a multi-spectral optical flow feature matching method. A multi-spectral image sequence is shot in a motion scene, optical flow information of imaging of each spectrum acquisition subsystem is calculated by using adjacent front and back frames of same spectral image sequences, external parameters among different spectrum image acquisition subsystems are calculated by using optical flow information of different spectrums, and multi-spectral image registration is further realized. According to the method, image registration is achieved through multi-spectral image sequence collection, optical flow calculation, optical flow feature matching, error screening, matching point screening and radiation matrix or flexible deformation field estimation. By converting the cross-spectrum into the same-mode optical flow, the complexity of cross-spectrum feature matching is avoided, and the effects of external parameter estimation and image registration are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and specifically to a multi-spectral camera extrinsic parameter estimation and image registration method. The method additionally introduces optical flow information as a new clue, which is of great significance for improving the accuracy of multi-spectral camera extrinsic parameter estimation and image registration. Background Art

[0002] Multispectral image acquisition systems are often composed of multiple single-spectral cameras. External parameter estimation and image registration between different single-spectral cameras are important preprocessing steps for multispectral image processing. Traditional multispectral camera external parameter estimation and image registration schemes directly process multispectral images (J.Li, Q.Hu and M.Ai, "RIFT: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform," in IEEE Transactions on Image Processing, vol. 29, pp. 3296-3310, 2020, doi: 10.1109 / TIP.2019.2959244.). Due to the radiation differences between images with different spectra, their image features are significantly different. Traditional single-modal feature matching algorithms, such as those based on SIFT (DGLowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004.) and ORB features (Rublee E, Rabaud V, Konolige K, et al. ORB: An efficient alternative to SIFT or SURF [C] / / 2011 International conference on computer vision. Ieee, 2011: 2564-2571.), cannot accurately perform feature matching, resulting in poor extrinsic parameter estimation and image registration effects. However, the use of radiation invariant features involves more cumbersome calculations, and its effect is often only effective in specific scenarios, such as E. Shechtman and M. Irani, "Matching local self-similarities across images and videos," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2007, p. 3. Summary of the invention

[0003] Aiming at the problem of low accuracy of cross-spectral feature matching in multi-spectral camera multi-sensor extrinsic parameter estimation and multi-spectral image registration, a multi-spectral optical flow feature matching method is proposed. Multi-spectral image sequences are shot in motion scenes, and the optical flow information of each spectral acquisition subsystem imaging is calculated using the adjacent two frames of the same spectrum image sequence. The optical flow information of different spectra is used to calculate the extrinsic parameters between different spectral image acquisition subsystems, and further realize multi-spectral image registration.

[0004] The technical solution of the present invention is an image registration method based on multispectral image extrinsic parameter calculation, the method comprising the following steps:

[0005] Step 1: Calibrate the intrinsic parameters K of each single-spectrum camera m ,m∈{1,2,…,M}, M is the total number of spectral components;

[0006] Step 2: Acquire a multispectral image sequence Where: I represents the image, the subscripts represent different spectra, j represents the image sequence number, ranging from 1 to N, and N is the sequence length;

[0007] Step 3: Use the optical flow method to calculate the optical flow of each spectral component image separately

[0008] Step 4: Optical Flow Perform feature point matching to obtain matching point pairs in represents the i-th matching point in the j-th optical flow map calculated from the j-th and j+1-th images in the m-th spectral image sequence;

[0009] The feature matching method used uses the amplitude information and angle information of the optical flow, and establishes the feature descriptor D of the amplitude G and angle A information for each optical flow. g and D a :

[0010] D g =S(G)

[0011] D a =S(A)

[0012] S(G) represents the feature descriptor calculation algorithm, which is the scale-invariant feature transformation method SIFT or ORB method (Oriented FAST and Rotated BRIEF). The SIFT method literature includes DGLowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004.), and the ORB method literature includes Rublee E, Rabaud V, Konolige K, et al. ORB: An efficient alternative to SIFT or SURF [C] / / 2011 International conference on computer vision. Ieee, 2011: 2564-2571.) and other mature feature extraction algorithms in the industry; the feature descriptor D g and D a Direct cascade:

[0013] D=[D g ,D a ]

[0014] For optical flow and optical flow Its feature point sets are P m , P n , using Euclidean distance as a metric to obtain matching feature point pairs The specific calculation is for P m Each point in Statistical P n The eigenvectors and The Euclidean distance of the feature vector of the point, the point with the smallest distance is taken as P n Neutralization Matched points

[0015] Step 5: Use features to match point pairs and K m ,m∈{1,2,…,M} calculates the camera extrinsic parameters Represents the camera extrinsics from single-spectrum camera n to single-spectrum camera m;

[0016] Step 6: Using multiple external references Eliminate the influence of time-unsynchronized image sequences and improve the accuracy of external parameter estimation; Step 7: Use the matched point pairs Calculate the matrix or flexible deformation field of the affine transformation between different spectral images to achieve registration between spectral images.

[0017] Further, the screening method of step 6 is as follows: first, the candidate external parameters are used Matching point pairs The point in the mth spectral component is projected into the camera coordinate system of the nth spectral component to obtain the reprojected point Calculate the reprojection error:

[0018]

[0019] When the target motion amplitude is less than the set threshold, the depth of the camera and the target is considered unchanged, and the correlation model between the image extrinsic parameters and the optical flow is as follows:

[0020]

[0021] Representing optical flow Center at point The optical flow information, K n It is represented as the internal parameter of the nth single-spectrum camera calibrated in step 1, R m,n represents the rotation matrix between single-spectrum cameras m and n;

[0022] According to the correlation model between image extrinsics and optical flow, the optical flow deviation is calculated:

[0023]

[0024] Represents the optical flow deviation corresponding to the i-th matching feature point in the j-th optical flow map; the other component of the cost function is the feature deviation that is unrelated to radiation distortion

[0025]

[0026] Where M is a radiation-independent feature extraction method, such as RIFT (J. Li, Q. Hu and M. Ai, "RIFT: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform," in IEEE Transactions on Image Processing, vol. 29, pp. 3296-3310, 2020, doi: 10.1109 / TIP.2019.2959244.); represents the jth image in spectral channel n, represents the i-th matching feature point in the j-th optical flow map in spectral channel n;

[0027] Sum all the reprojection costs to get Fix m, n, change j, choose different Calculate the sum of the reprojection errors E, then the optimal extrinsic parameter estimate is the solution [R, T] that minimizes E n,m ; Using the [R,T] obtained by screening n,m Combining feature bias Filter out the feature point pairs that meet the requirements and remove the inaccurate points. Specifically, calculate the deviation e corresponding to the feature point pairs respectively. i :

[0028]

[0029] The deviation e i Sort and remove point pairs whose deviation is greater than the set threshold.

[0030] Furthermore, the specific calculation step of the optical flow in step 3 is to convert the reference frame I r and input frame I i Input them into the encoder respectively to obtain the image features (L r ,L i ), L r represents the reference frame image feature, L i Represents the input frame image features, and then takes image blocks with a fixed window size of w×h centered on different pixels (T r,k ,T i,k ),k∈{1,2,…,WH}, W and H are the width and height of the entire image, T r,k represents the reference frame image block, T i,k Represents the input frame image block;

[0031]

[0032] The cosine approximation is used to calculate the similarity between each image block and other image blocks in the neighborhood; the offset corresponding to the neighborhood block with the greatest similarity is taken as the optical flow.

[0033] Furthermore, the optical flow calculation method in step 3 is the RAFT method or the Lucas-Kanade method. (The RAFT method is a recursive all-pairs field transform for optical flow, Teed Z, Deng J. Raft: Recurrent all-pairs field transforms for optical flow [C] / / Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020: 402-419; the Lucas-Kanade method is the LK optical flow method, Barron JL, Fleet DJ, Beauchemin S S. Performance of optical flow techniques [J]. International journal of computer vision, 1994, 12: 43-77.

[0034] Furthermore, the camera extrinsic parameter estimation method in step 5 is the eight-point method.

[0035] The present invention discloses a multispectral image extrinsic parameter estimation and image registration method, which includes multispectral image sequence acquisition, optical flow calculation, optical flow feature matching, error screening, matching point screening, radiation matrix or flexible deformation field estimation, and image registration. By converting cross-spectral into homomodal optical flow, the complexity of cross-spectral feature matching is avoided, and the effect of extrinsic parameter estimation and image registration is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the whole method;

[0037] Figure 2 In order to take advantage of the effect of the registered visible light-infrared image by the patented method, the image is placed alternately with visible light and infrared squares for easy comparison. At the edges of the squares, it can be seen that the visible light and infrared images are aligned. DETAILED DESCRIPTION

[0038] For image sequences captured by a single-spectrum camera, when there is movement of the scene or target between different frames, the motion vector of each pixel between adjacent frames under the same spectrum, i.e., the optical flow, can be calculated. At the same moment, the motion vector of the scene or target is the same, and the optical flow information calculated using image sequences of different single-spectrum cameras is similar, and the image features of the optical flows of different spectral cameras are similar. Using the optical flow information of different single-spectrum cameras for feature matching can avoid the problem of difficulty in accurately matching features due to significant feature differences between different spectral images. Using a mature single-modal feature matching algorithm to match the optical flow features of different spectral cameras can achieve external parameter estimation of multi-spectral cameras. Furthermore, since the optical flow and image have a one-to-one correspondence, the optical flow feature point pairs can be used to generate a guideline matrix or deformation field data for image registration.

[0039] like Figure 1 As shown in the figure, unlike the traditional multi-spectral camera external parameter estimation and image registration that directly uses cross-spectral feature matching, this scheme collects image sequences of each single-spectral camera, and uses the image sequence to solve the optical flow of the single-spectral camera, calculates the image features of the cross-spectral optical flow, and converts the cross-spectral image feature matching with huge image feature differences into spectral feature matching with similar image features, which simplifies the complexity of cross-spectral feature matching. The specific steps are:

[0040] Step 1: Calibrate the internal reference K of each single-spectrum camera m ,m∈{1,2,…,M}, taking the visible light-infrared multispectral camera as an example, it contains two spectral components, namely visible light and infrared. The Zhang Zhengyou calibration method can be used to calibrate the visible light camera and the infrared camera to obtain K1 and K2. The Zhang Zhengyou calibration method is a well-known and widely used data calibration method in the industry.

[0041] Step 2: Collect multispectral image sequences. Taking a visible-infrared multispectral camera as an example, the collected sequence is: j∈{1,2,3,…,N}, where: I represents the image, the subscripts represent different spectra, j represents the image sequence number, ranging from 1 to N, and N is the sequence length, for example, N=3;

[0042] Step 3: Use the optical flow method to calculate the optical flow of each spectral component image. Taking the visible light-infrared multispectral camera as an example, the optical flow calculated using the Lucas-Kanade optical flow algorithm is:

[0043] Step 4: Taking the visible light-infrared multispectral camera as an example, Perform feature point matching to obtain matching point pairs in represents the i-th matching point in the optical flow calculated from the j-th and j+1-th images in the m-th spectral image sequence;

[0044] Step 5: Use the Eight Point Method and K m ,m∈{1,2} calculates the camera extrinsic parameters

[0045] Step 6: Use the RANSAC algorithm to calculate the external parameters Screening is performed to eliminate the influence of image sequences that are not synchronized in time;

[0046] Step 7: Use the matching point pairs Calculate the matrix or flexible deformation field of the affine transformation between different spectral images to achieve registration between spectral images;

[0047] In step 2 of the present invention, the registration uses an image sequence instead of a single image. In step 4, the feature point matching uses a non-cross-modal image pair. But the optical flow pair of the same mode In step 5, in order to obtain a more accurate external parameter estimation, N-1 optical flows are used at the same time The estimated results.

[0048] When using the RANSAC algorithm in step 6, the correlation model F between image extrinsics and optical flow is used. n =f(F m ,[R,T] n,m ) to evaluate the accuracy of the estimation; the features that are independent of radiation distortion between multispectral images are used to evaluate the accuracy of the estimation. The matching point pairs between the spectral images and the optical flow images of the present invention are the same.

[0049] Using the above steps, the visible light-infrared camera was calibrated and registered, and the reprojection error was statistically used as the accuracy measure of the external parameter estimation, as shown in the following table

[0050] Table 1. Analysis results of reprojection errors of visible light and infrared extrinsic parameter estimation, per pixel

[0051]

[0052] It can be seen that the reprojection error is small, which shows the effectiveness of the external parameter estimation method of this patent. Figure 2 As shown, for the convenience of comparison, the picture is placed alternately with visible light and infrared squares. At the edges of the squares, it can be seen that the visible light and infrared images are aligned, which illustrates the effectiveness of the image registration method of this patent.

[0053] By utilizing the multispectral image extrinsic parameter estimation and image registration method of the present invention, utilizing the optical flow information that is often avoided and ignored in multispectral camera extrinsic parameter estimation and image registration, utilizing the optical flow method to convert cross-spectral feature matching into same-spectral feature matching, utilizing optical flow correlation and optical flow feature matching deviation to perform extrinsic parameter estimation and image registration, the accuracy of extrinsic parameter estimation and image registration can be effectively improved.

Claims

1. An image registration method based on multispectral image extrinsic calculation, the method comprising the following steps: Step 1: Calibrate the intrinsic parameters K of each single-spectrum camera m ,m∈{1,2,…,M}, M is the total number of spectral components; Step 2: Acquire a multispectral image sequence Where: I represents the image, the subscripts represent different spectra, j represents the image sequence number, ranging from 1 to N, and N is the sequence length; Step 3: Use the optical flow method to calculate the optical flow of each spectral component image separately Step 4: Optical Flow Perform feature point matching to obtain matching point pairs in represents the i-th matching point in the j-th optical flow map calculated from the j-th and j+1-th images in the m-th spectral image sequence; The feature matching method used uses the amplitude information and angle information of the optical flow, and establishes the feature descriptor D of the amplitude G and angle A information for each optical flow. g and D a : D g =S(G) D a =S(A) S(G) represents the feature descriptor calculation algorithm; The feature descriptor D g and D a Direct cascade: D=[D g ,D a ] For optical flow and optical flow Its feature point sets are P m , P n , using Euclidean distance as a metric to obtain matching feature point pairs The specific calculation is for P m Each point in Statistical P n The eigenvectors and The Euclidean distance of the feature vector of the point, the point with the smallest distance is taken as P n Neutralization Matched points Step 5: Use features to match point pairs and K m ,m∈{1,2,…,M} calculates the camera extrinsic parameters Represents the camera extrinsics from single-spectrum camera n to single-spectrum camera m; Step 6: Using multiple external references Eliminate the influence of time-unsynchronized image sequences and improve the accuracy of extrinsic parameter estimation; Step 7: Use the matching pairs Calculate the matrix or flexible deformation field of the affine transformation between different spectral images to achieve registration between spectral images.

2. The image registration method based on multispectral image extrinsic parameter calculation as claimed in claim 1, characterized in that: The feature descriptor calculation algorithm in step 4 is the scale-invariant feature transformation method SIFT or ORB method.

3. The image registration method based on multispectral image extrinsic parameter calculation as claimed in claim 1, characterized in that: The screening method of step 6 is as follows: first, the candidate external parameters are used Matching point pairs The point in the mth spectral component is projected into the camera coordinate system of the nth spectral component to obtain the reprojected point Calculate the reprojection error: When the target motion amplitude is less than the set threshold, the depth of the camera and the target is considered unchanged, and the correlation model between the image extrinsic parameters and the optical flow is as follows: Representing optical flow Center at point The optical flow information, K n It is represented as the internal parameter of the nth single-spectrum camera calibrated in step 1, R m,n represents the rotation matrix between single-spectrum cameras m and n; According to the correlation model between image extrinsics and optical flow, the optical flow deviation is calculated: Represents the optical flow deviation corresponding to the i-th matching feature point in the j-th optical flow map; the other component of the cost function is the feature deviation that is unrelated to radiation distortion Where M is a feature extraction method that is independent of radiation distortion, represents the jth image in spectral channel n, represents the i-th matching feature point in the j-th optical flow map in spectral channel n; Sum all the reprojection costs to get Fix m, n, change j, choose different Calculate the sum of the reprojection errors E, then the optimal extrinsic parameter estimate is the solution [R, T] that minimizes E n,m ; Using the [R,T] obtained by screening n,m Combining feature bias Filter out the feature point pairs that meet the requirements and remove the inaccurate points. Specifically, calculate the deviation e corresponding to the feature point pairs respectively. i : The deviation e i Sort and remove point pairs whose deviation is greater than the set threshold.

4. The image registration method based on multispectral image extrinsic parameter calculation as claimed in claim 1, characterized in that: The specific calculation step of the optical flow in step 3 is to convert the reference frame I r and input frame I i Input them into the encoder respectively to obtain the image features (L r ,L i ), L r represents the reference frame image feature, L i Represents the input frame image features, and then takes image blocks with a fixed window size of w×h centered on different pixels (T r,k ,T i,k ),k∈{1,2,…,WH}, W and H are the width and height of the entire image, T r,k represents the reference frame image block, T i,k Represents the input frame image block; The cosine approximation is used to calculate the similarity between each image block and other image blocks in the neighborhood; the offset corresponding to the neighborhood block with the greatest similarity is taken as the optical flow.

5. The image registration method based on multispectral image extrinsic parameter calculation as claimed in claim 1, characterized in that: The optical flow calculation method in step 3 is the RAFT method or the Lucas-Kanade method.

6. The image registration method based on multispectral image extrinsic parameter calculation as claimed in claim 1, characterized in that: The camera extrinsic parameter estimation method in step 5 is the eight-point method.

Images