A Terahertz Photoelectric Image Registration Method Utilizing Video Feature Tracking

Abstract

This invention discloses a terahertz photoelectric image registration method using video feature following, comprising the following steps: image preprocessing; SURF-based first-frame image registration: calculating feature points of the matching first-frame radar image and first-frame optical image based on SURF features; video following feature extraction; and sequence image registration. This invention utilizes a video feature following method to transform the complex problem of heterogeneous image registration into a homogeneous image registration problem, reducing the problem complexity and improving registration accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a terahertz photoelectric image registration method that utilizes video feature tracking. Background Technology

[0002] In the complex battlefield environment of modern times, a single sensor source is insufficient for reconnaissance and situational awareness. Meanwhile, the emergence of multi-source sensor systems and the increasing number of multiple sensor images of a single area have given rise to the emerging discipline of multi-source data fusion. SAR (Radar and Optical Array) offers all-weather, all-day imaging capabilities and good penetration through clouds, fog, and dust; optical images often have higher imaging resolution and are more consistent with human vision. Fusion processing of radar and optical images can significantly improve the system's fault tolerance, robustness, and reconfiguration capabilities.

[0003] Image registration is fundamental to multi-source data fusion. In particular, unlike traditional microwave radar image registration with visible light images, terahertz video radar image registration presents new technical challenges due to its inherent characteristics. First, compared to traditional microwave frequencies, the high resolution of terahertz radar clearly reveals the shape, structure, and texture of targets. However, it also makes phenomena such as top-to-bottom inversion and forward-looking / backward-looking distortion more pronounced in radar images, and the strong differences between heterogeneous images demand higher registration accuracy. Second, terahertz radar has a fast imaging speed, requiring registration algorithms to achieve the same video-level processing speed. Finally, the unavoidable jitter during aircraft movement demands extremely high stability from the algorithm.

[0004] Classic image registration methods include pixel intensity-based and image feature point-based methods. Pixel intensity-based methods utilize pixel correlations between the same regions in heterogeneous images to calculate image similarity, phase mutual information (M. Mellor and M. Brady, “Phase mutual information as a similarity measure for registration,” Med. Image Anal., vol. 9, no. 4, pp. 330–343, 2005.), and mutual information (S. Suri and P. Reinartz, “Mutual-information-based registration of TerraSAR-X and Ikonosimagery in urban areas,” IEEE Trans. Geosci. Remote Sens., vol. 48, no. 2, pp. 939–949, Feb. 2010.), and iteratively estimate the registration transformation matrix. Methods based on image feature points calculate metrics such as Harris (J. Chen, J. Tian, ​​N. Lee, J. Zheng, RTSmith, and AFLaine, “A partial intensity invariant feature descriptor for multimodal retinal image registration,” IEEE...). Using key feature points such as Trans. Biomed. Eng., vol. 57, no. 7, pp. 1707–1718, Jul. 2010., SIFT (D. GLowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004.), and SURF (D. Zhao, Y. Yang, Z. Ji, and X. Hu, “Rapid multimodality registration based on MM-SURF,” Neurocomputing, vol. 131, pp. 87–97, May 2014.), the transformation matrix is ​​obtained by matching the corresponding feature points, which is relatively more efficient.In recent years, feature-based methods have been applied to the registration of microwave radar and optical images (Y. Xiang, F. Wang, H. You, “OS-SIFT: A Robust SIFT-Like Algorithm for High-Resolution Optical-to-SAR Image Registration in Suburban Areas”, IEEE Trans. Geosci. Remote Sens., 2017; J. Fan, Y. Wu, Ming. L, “SAR and Optical Image Registration Using NonlinearDiffusion and Phase Congruency Structural Descriptor”, IEEE Trans. Geosci. Remote Sens., 2018), but there is still no research on the registration of terahertz video SAR images with optical images, which remains a rather challenging problem. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a terahertz photoelectric image registration method utilizing video feature tracking, comprising the following steps:

[0006] Image preprocessing: The radar image is preprocessed using a slant-to-ground transform, and the optical image is de-jittered using cross-correlation.

[0007] First frame image registration based on SURF: Calculate the feature points of the matching first frame radar image and first frame optical image based on SURF features;

[0008] Video tracking feature extraction: The optical image stream and radar image stream are processed separately. The gradient descent method is used to minimize the root mean square error of the matching between the frame image to be registered and the first frame image, so as to obtain the feature points and transformation matrix of each frame image except the first frame image; the frame image includes each radar image and photoelectric image.

[0009] Sequence image registration: Calculate the corresponding photoelectric transformation matrix based on the feature points of each frame of the radar image and the photoelectric image in the video, and register each frame of the radar image and the photoelectric image to obtain the registered image sequence.

[0010] Specifically, the video following feature extraction includes the following steps:

[0011] The first and second radar images are registered. The root mean square error of the intensity of the two images is used as the optimization target. The gradient descent method is used to iteratively estimate the transformation matrix P2 of the second radar image.

[0012] The feature points of the second radar image can be calculated from the feature points of the first radar image and the transformation matrix P2 as X2 = P2X1; similarly, the same operation can be performed on the optical image to obtain the feature points Y2 = Q2Y1 in the optical image; where X1 is the feature point of the first radar image obtained by registering the first image based on the SURF feature, Y1 is the feature point of the first optical image obtained by registering the first image based on the SURF feature, Y2 is the feature point of the second optical image, and Q2 is the transformation matrix P2 of the second optical image;

[0013] For each subsequent optical and radar image to be registered, the corresponding feature points are obtained by registering them with the first optical and radar images, respectively.

[0014] Specifically, the SURF-based first frame image registration includes the following steps:

[0015] The first frame of optical image and the first frame of radar image are registered using a SURF feature-based method to obtain the feature points of the first frame of optical image and the first frame of radar image, which are X1 and Y1 respectively;

[0016] Calculate the Hessian matrix of each pixel in the first frame of the optical image and the first frame of the radar image;

[0017] Each pixel processed by the Hessian matrix is ​​compared with its 3*3 neighborhood. If the pixel is the maximum or minimum value, it is retained and treated as the original feature point. If the pixel is not the maximum or minimum value, it is skipped.

[0018] By extracting Haar wavelet features in the neighborhood of the original feature points and calculating and determining the main direction of the original feature points, a SURF feature point descriptor vector is constructed.

[0019] The Euclidean distance between the SURF feature point description operator vectors is used as a similarity metric for the original feature points in the two images. Coarse matching is performed on the original feature points to obtain the initial matching point pairs between the first optical image and the first radar image.

[0020] The RANSAC algorithm is used to filter the initial matching point pairs to obtain the final matching feature point pairs.

[0021] Specifically, the SURF feature point descriptor vector expression uses υ subregion Specifically, it means:

[0022] υ subregion =[∑dx,∑dy,∑|dx|,∑|dy|]

[0023] Where ∑dx represents the sum of horizontal values ​​of the Haar wavelet features along the main direction of the original feature points, ∑|dx| represents the sum of absolute horizontal values ​​of the Haar wavelet features along the main direction of the original feature points, ∑dy represents the sum of vertical values ​​of the Haar wavelet features along the main direction of the original feature points, and ∑|dy| represents the sum of absolute vertical values ​​of the Haar wavelet features along the main direction of the original feature points.

[0024] Specifically, the coarse matching of the original feature points includes the following steps: for each original feature point in the optical image, its nearest neighbor and second nearest neighbor are obtained by querying the original feature point set of the radar image; if the ratio of the nearest neighbor distance to the second nearest neighbor distance is less than a given threshold, the match between the feature point and its nearest neighbor is retained; otherwise, the match is discarded, thereby obtaining the initial matching point pair of the radar image and the optical image.

[0025] Specifically, the shaking removal process for the optical image using cross-correlation includes the following steps:

[0026] The correlation coefficient matrix ρ between two adjacent optical images is calculated using Fast Fourier Transform:

[0027] ρ=IFFT2[FFT2(A1)·conj(FFT2(B2))]

[0028] Where FFT2 and IFFT2 represent two-dimensional Fourier transform and two-dimensional inverse Fourier transform, respectively, conj() represents conjugate calculation, A1 represents the previous optical image in two adjacent optical images, and B2 represents the thicker optical image in two adjacent optical images.

[0029] The row and column positions corresponding to the maximum value in ρ are calculated as follows:

[0030] [a, b] = arc max(ρ)

[0031] Where a represents the row position of the maximum value in the correlation coefficient matrix ρ, and b represents the column position of the maximum value in the correlation coefficient matrix ρ;

[0032] Let M r and M c These represent the row position deviation and column position deviation of the corresponding image matching, respectively, and are calculated using the following formula:

[0033]

[0034] Where, N ak and N bk These represent the total number of rows and columns of pixels in the image, respectively.

[0035] The jitter of the optical images is corrected by circular shifting based on the row and column position deviations of the corresponding image matching.

[0036] Compared with the prior art, the significant advantages of this invention are:

[0037] (1) By using the video feature following method, the complex problem of registering heterogeneous images is transformed into the problem of registering homogeneous images, which reduces the complexity of the problem and improves the registration accuracy;

[0038] (2) By utilizing the continuity of the video image stream, the transformation matrix between each frame image and the first frame image can be solved quickly, and the algorithm is fast.

[0039] (3) Correlation correction is applied to optical images captured on the airborne platform to remove jitter, thereby improving the stability of image registration. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the terahertz video photoelectric image registration process according to an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the video feature following method according to an embodiment of the present invention.

[0042] Figure 3 This is the optical image correlation stabilization result of an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of SURF feature point matching in the first frame image of an embodiment of the present invention.

[0044] Figure 5 This is the video feature tracking result of an embodiment of the present invention.

[0045] Figure 6 This is the terahertz photoelectric image registration result for the helipad in an embodiment of the present invention. Detailed Implementation

[0046] The purpose of this invention is to provide a terahertz optoelectronic video image registration method utilizing video feature tracking. For the optoelectronic image registration problem of video SAR, a high-precision, real-time, and robust processing framework is proposed. This framework fully utilizes the continuous input characteristic of video images, and by performing distortion correction and feature point estimation on the radar image stream and optical image stream respectively, it can accurately, quickly, and stably process online data.

[0047] The following describes in detail, with reference to the accompanying drawings, a specific embodiment of the terahertz photoelectric image registration method of the present invention that utilizes video feature tracking.

[0048] This embodiment provides a terahertz optoelectronic video image registration method utilizing video feature tracking. The overall process is as follows: Figure 1 As shown, it includes the following steps:

[0049] (10) Image preprocessing

[0050] For large-scene images, downsampling is performed first to improve the speed of subsequent steps. Before registering radar and optical images, preprocessing is required to obtain an input video image stream suitable for registration. In this embodiment, the radar image is specifically a SAR image.

[0051] For radar images, since they are imaged on a slant-range plane, a slant-range transformation is required to obtain a planar image that matches the camera's photograph.

[0052] For optical images, severe jitter between consecutive frames is caused by the unstable motion of the carrier aircraft. Here, image cross-correlation is used to calculate the jitter offset between each frame. After correction, a stable, jitter-free video stream is obtained. The correlation coefficient matrix ρ between two adjacent optical images is calculated using Fast Fourier Transform (FFT).

[0053] ρ=IFFT2[FFT2(A1)·conj(FFT2(B2))] (1)

[0054] Where FFT2 and IFFT2 represent the two-dimensional Fourier transform and the two-dimensional inverse Fourier transform, respectively; conj() represents conjugate calculation; A1 represents the previous optical image in two adjacent optical images; and B2 represents the thicker optical image in two adjacent optical images. In the correlation coefficient matrix ρ, the row and column positions corresponding to the maximum value are related to the row and column position deviations of the two image blocks. The row and column positions corresponding to the maximum value in ρ are calculated as follows:

[0055] [a, b] = arc max(ρ) (2)

[0056] Where a represents the row position of the maximum value in the correlation coefficient matrix ρ, and b represents the column position of the maximum value in the correlation coefficient matrix ρ.

[0057] Let M r and M c Let these represent the row position deviation and column position deviation of the corresponding image matching, respectively, which can be calculated as follows:

[0058]

[0059] Where, N ak and N bk These represent the total number of rows and columns of pixels in the image, respectively.

[0060] The jitter of the optical images is corrected by circular shifting based on the row and column position deviations of the corresponding image matching.

[0061] In this embodiment, terahertz optoelectronic video image registration is completed using measured data from the helipad. The input image size is 1200*1200. First, the image is downsampled by a factor of 4. Cross-correlation is used to remove jitter from the optical image, such as... Figure 3 As shown in the second row of the image, the original image has been circularly shifted by a certain amount to make the entire scene image continuous and stable.

[0062] (20) First frame image registration based on SURF features

[0063] The SURF operator has been widely used due to its good real-time processing performance and strong matching ability. Here, the first frame of optical image and radar image are registered using a SURF feature-based method to obtain the feature points in the first frame of radar image and the first frame of optical image, denoted by X1 and Y1 respectively. The specific steps are as follows:

[0064] Calculate the SURF feature vectors for each of the two images. First, calculate the Hessian matrix for each pixel:

[0065]

[0066] Here, f(x, y) represents the image, and (x, y) represents the pixel in the image.

[0067] Each pixel processed by the Hessian matrix is ​​compared with its 3x3 neighborhood. If the pixel is the maximum or minimum value, it is retained and treated as the original feature point; otherwise, it is skipped.

[0068] The main direction of the feature points is determined by extracting Haar wavelet features from the neighborhood of the original feature points and calculating them; a SURF feature point description operator vector is constructed, the expression of which is v subregion Specifically, it means:

[0069] v subregion =[∑dx,∑dy,∑|dx|,∑|dy|] (5)

[0070] Where ∑dx represents the sum of horizontal values ​​of the Haar wavelet features along the main direction of the original feature points, ∑|dx| represents the sum of absolute horizontal values ​​of the Haar wavelet features along the main direction of the original feature points, ∑dy represents the sum of vertical values ​​of the Haar wavelet features along the main direction of the original feature points, and ∑|dy| represents the sum of absolute vertical values ​​of the Haar wavelet features along the main direction of the original feature points.

[0071] The Euclidean distance between the SURF feature point description operator vectors is used as a similarity metric for the original feature points in the two images. A coarse matching is performed on the original feature points: for each original feature point in the optical image, its nearest and second nearest neighbors are retrieved from the point set of original feature points in the radar image. If the ratio of the nearest neighbor distance to the second nearest neighbor distance is less than a given threshold, the match between the feature point and its nearest neighbor is retained; otherwise, the match is discarded, thus obtaining the initial matching point pairs between the radar and optical images.

[0072] Even after coarse matching, a large number of erroneous matches still exist in the initial matching point pairs. Therefore, further processing of these matching points is needed to eliminate erroneous matching point pairs as much as possible. In this embodiment, the RANSAC (Random Sample Consensus) algorithm, which has good reliability, high accuracy, and strong robustness, is used to filter the initial matching point pairs to obtain the final matched feature point pairs, thus obtaining the first photoelectric conversion matrix M1.

[0073] In the specific implementation process, SURF feature points are calculated and matched for both radar and optical images, such as... Figure 4 As shown in the figure. In the experiment, the coarse matching threshold was set to 0.85, and the number of RANSAC fine matching iterations was 5000. The positions of the feature points in the figure were calculated and saved. These feature points are the basis for subsequent feature point following and need to have a fairly high accuracy.

[0074] (30) Video following feature extraction

[0075] For both radar and optical image streams, the algorithm needs to estimate the positions of initial feature points in the new image and then calculate the corresponding transformations; this step is called "video following feature extraction." Due to the inherent similarity of photoelectric images, an intensity-based algorithm is first used to register the first frame image with subsequent images, thereby obtaining the feature points of the subsequent images.

[0076] like Figure 2 As shown, the first and second radar images are registered. Using the root mean square error of the intensity of the two images as the optimization objective, the gradient descent method is used to iteratively estimate the transformation matrix, resulting in the second radar image transformation matrix P2. The second radar image transformation matrix P2 is an affine matrix, incorporating changes in image scale, rotation, scaling, and translation.

[0077] Then, the feature points of the second radar image can be calculated from the feature points of the first radar image and the transformation matrix P2 as X2 = P2X1; similarly, the same operation can be performed on the optical image to obtain the feature points Y2 = Q2Y1 in the optical image. Here, X1 represents the feature points of the first radar image obtained by registering the first image based on SURF features, Y1 represents the feature points of the first optical image obtained by registering the first image based on SURF features, Y2 represents the feature points of the second optical image, and Q2 represents the transformation matrix P2 of the second optical image.

[0078] For each subsequent frame of optical and radar image, registration is performed with the first frame of optical and radar images, respectively, to obtain the feature points of the corresponding images, such as... Figure 5 As shown, the registration results and feature point positions of the optical images (first frame), second frame, 40th frame, and 80th frame, and the registration results and feature point positions of the radar image are displayed respectively (SURF feature points are simplified for clarity). In the experiment, the maximum number of iterations was set to 10, and the maximum search step size was 0.001. In the registration results, the mismatched parts after transformation are only visible at the edges; the main scene parts in the middle match well, and the positions of each feature point are accurate.

[0079] (40) Image sequence registration

[0080] The second photoelectric transformation matrix M2 can be calculated from feature point X2 of the second radar image and feature point Y2 of the second optical image. For each image frame other than the first frame, to obtain accurate new feature point positions, it is still necessary to register the nth frame image (including radar and optical images) with the corresponding first frame image to solve for the feature points and corresponding transformation matrix of the nth frame image. The nth radar image transformation matrix corresponding to the nth radar image is P. n The transformation matrix of the nth optical image corresponding to the nth frame is Q. n Where n∈[1,2,…,N], and N is a positive integer. Here, the continuity of the video image stream is cleverly utilized to transform the radar image matrix P of the (n-1)th frame. n-1 The initial solution is used to find the transformation matrix P of the nth frame radar image. n The optical image transformation matrix Q of the (n-1)th frame n-1 The initial solution is used to find the optical image transformation matrix Q of the nth frame. n This greatly reduces the number of iterations and improves the solution accuracy.

[0081] Based on the feature points and transformation matrix of each frame of the video (excluding the first frame), the corresponding photoelectric image transformation matrix is ​​calculated, specifically from the second photoelectric transformation matrix M1 to the Nth photoelectric transformation matrix M. NThis allows for the registration of each frame of radar and optical images, resulting in a registered image sequence. The actual implementation results are as follows: Figure 6 As shown. The registration result uses the radar image as a background and overlays the transformed optical image for display. The algorithm runs at a speed of 5Hz. The photoelectric transformation matrix is ​​specifically an affine matrix.

[0082] Compared with the prior art, the significant advantages of this invention are:

[0083] (1) By using the video feature following method, the complex problem of registering heterogeneous images is transformed into the problem of registering homogeneous images, which reduces the complexity of the problem and improves the registration accuracy;

[0084] (2) By utilizing the continuity of the video image stream, the transformation matrix between each frame image and the first frame image can be solved quickly, and the algorithm is fast.

[0085] (3) Correlation correction is applied to optical images captured on the airborne platform to remove jitter, thereby improving the stability of image registration.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A terahertz photoelectric image registration method using video feature following, characterized by, The photoelectric images are optical images and radar images, and the method comprises the following steps: Image preprocessing: using slant transform to preprocess the radar images, and using cross-correlation to de-jitter the optical images; SURF-based first-frame image registration: calculating the matching feature points of the first-frame radar image and the first-frame optical image based on SURF features; Video following feature extraction: The first frame radar image and the second frame radar image are registered, a gradient descent method is used for iterative estimation of the transformation matrix to obtain a second radar image transformation matrix P2 with a root mean square error of two image intensities as an optimization target; the feature points of the second frame radar image can be calculated from the feature points of the first frame radar image and the transformation matrix P2, i.e. X2 = P2X1; similarly, the same operation is performed on the optical image to obtain the feature points in the optical image, i.e. Y2 = Q2Y1; wherein X1 is the feature points of the first frame radar image obtained by first frame image registration based on SURF features, Y1 is the feature points of the first frame optical image obtained by first frame image registration based on SURF features, Y2 is the feature points of the second frame optical image, P2 is the second optical image transformation matrix; the optical image and the radar image of each subsequent frame to be registered are registered with the first frame optical image and the first frame radar image respectively to obtain the corresponding feature points thereof; Sequence image registration: calculating the corresponding photoelectric transformation matrix according to the feature points of each frame of radar image and photoelectric image of the video and the transformation matrix, registering each frame of radar image and optical image to obtain the registered image sequence.

2. The terahertz electro-optical image registration method using video feature following according to claim 1, characterized in that, The SURF-based first-frame image registration comprises the following steps: Registering the first-frame optical image and the first-frame radar image based on SURF features to obtain the feature points of the first-frame optical image and the first-frame radar image, respectively, which are X1 and Y1; Calculating the Hessian matrix of each pixel point of the first-frame optical image and the first-frame radar image; Comparing each pixel point processed by the Hessian matrix with the neighborhood points in its 3*3 neighborhood; if the pixel point is the maximum or minimum value, it is reserved and regarded as the original feature point, and if the pixel point is not the maximum or minimum value, the pixel point is skipped; Constructing the SURF feature point descriptor vector by extracting the haar wavelet features in the neighborhood of the original feature point and calculating the main direction of the original feature point; Using the Euclidean distance between the SURF feature point descriptor vectors as the similarity judgment measure of the original feature points in the two images to coarsely match the original feature points to obtain the initial matching point pairs of the first-frame optical image and the first-frame radar image: Using the RANSAC algorithm to screen the initial matching point pairs to obtain the final matching feature point pairs.

3. The terahertz electro-optical image registration method using video feature following according to claim 2, characterized in that, The SURF feature point description operator vector expression is represented by and specifically is: wherein, denotes the sum of the horizontal values of the haar wavelet features in the main direction of the original feature point, denotes the sum of the absolute horizontal values of the haar wavelet features in the main direction of the original feature point, denotes the sum of the vertical values of the haar wavelet features in the main direction of the original feature point, denotes the sum of the absolute vertical values of the haar wavelet features in the main direction of the original feature point.

4. The terahertz electro-optical image registration method using video feature following according to claim 3, characterized in that, The coarse matching of the original feature points comprises the following steps: for each original feature point in the optical image, querying its nearest neighbor point and second nearest neighbor point in the original feature point set of the radar image; if the ratio of the nearest neighbor distance to the second nearest neighbor distance is less than a given threshold, the matching of the feature point and its nearest neighbor is reserved, otherwise, the matching pair is removed, thereby obtaining the initial matching point pairs of the radar image and the optical image.

5. The terahertz photoelectric image registration method using video feature following according to claim 2, characterized by, The de-jittering of the optical images by cross-correlation comprises the following steps: A correlation coefficient matrix of two adjacent optical images is calculated using a fast Fourier transform : wherein and respectively denote a two-dimensional Fourier transform and a two-dimensional inverse Fourier transform, denotes a conjugate computation, denotes a preceding one of the two adjacent optical images, denotes a following one of the two adjacent optical images; Computing The row and column positions corresponding to the maximum value in the center are as follows: wherein, denotes the row position of the maximum value in the correlation coefficient matrix denotes the column position of the maximum value in the correlation coefficient matrix denotes the row position of the maximum value in the correlation coefficient matrix denotes the column position of the maximum value in the correlation coefficient matrix Let and respectively represent the row position deviation and the column position deviation corresponding to image matching, and the calculation formula is: wherein, and respectively the total number of rows and columns of pixels of the image; Correcting the jitter of the optical images by circular shift according to the matching row position deviation and column position deviation of the corresponding images.

Images