Interframe tiny offset high-speed video measurement phase correlation matching method

Abstract

The invention discloses an inter-frame tiny offset high-speed video measurement phase correlation matching method. The method comprises the following steps: acquiring a high-frame-frequency image sequence; dynamically adjusting the image sequence through an adaptive window to obtain a neighborhood sub-image; performing gradient real-virtual dual-channel Gaussian filtering on the neighborhood sub-image to obtain a normalized cross power spectrum matrix; performing plane fitting on the normalized cross power spectrum matrix, dynamically adjusting a sampling subset through a noise estimation index and an interior point probability, and performing iterative optimization to obtain an initial offset value; and updating a normalized cross-power spectrum calculation residual offset value based on the initial offset value, iterating until the residual offset value is lower than a preset threshold value, and summing all iteration offset values to obtain a final sub-pixel offset value. According to the method, the robustness advantages of frequency domain phase correlation matching on illumination variation and noise are utilized, accurate identification and tracking of inter-frame tiny offset sequences of target points on the surface of a high-speed moving body under the influence of illumination, angles and the like are achieved, and therefore the accuracy of dynamic monitoring is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of high-speed video measurement matching technology, and particularly relates to a frame-to-frame micro-offset high-speed video measurement phase correlation matching method. BACKGROUND

[0002] High-speed video measurement records the spatial position and state of a moving object in the form of a video or image sequence, and obtains high-precision three-dimensional spatial coordinates of feature points of the moving object through photogrammetric analysis and processing, so as to qualitatively and quantitatively analyze the features of the moving object and describe the motion trajectory and motion parameters of the object. Unlike ordinary video measurement with a camera, high-speed video measurement can reach hundreds or even thousands of frame frequencies, thereby meeting the video measurement requirements of high-speed moving objects. Sequence image target tracking is one of the core technologies of high-precision high-speed video measurement. High-speed video measurement can obtain a large amount of image sequence data in a short time, so that the displacement of target points between image sequences is not more than a pixel level or even reaches a sub-pixel level. To obtain the motion information of a high-speed moving object from the image sequence, the key problem is high-precision tracking of the micro-offset of high-frame-frequency sequence images.

[0003] Sequence image micro-shift tracking technology is mainly used for detecting and tracking the micro-shift changes of targets in sequence images, mainly including optical flow method, template matching method, deep learning method and phase correlation method. The optical flow tracking method estimates the continuous motion of the target by solving the optical flow equation (such as Horn-Schunck equation or Lucas-Kanade equation) to calculate the motion vector of the pixel points in the sequence image, and the tracking accuracy in complex background can be further improved by combining with deep learning algorithms such as FlowNet and PWC-Net, but this method is sensitive to image noise and illumination changes, and is only suitable for micro-shift tracking in simple scenes with little illumination change. The template matching method tracks the target by matching the target template with the region in the sequence image, and the search method, range and threshold must be reasonably selected, which is suitable for scenes with small changes in target shape and texture, but is sensitive to target deformation and rotation, and has limitations in capturing micro-shift. The deep learning method uses convolutional neural network (CNN) or recurrent neural network (RNN) to learn the motion features of the target, which can realize high-precision tracking of the target in complex background, but it relies on a large amount of data for training, and the difference between the training data and the actual scene will lead to insufficient recognition ability of micro-shift. The phase correlation method is a frequency domain-based image registration technology, which can improve the registration accuracy to sub-pixel level (such as 1 / 100 pixel level) by introducing peak estimation technology, significantly improving the detection ability of micro-shift, and it is not sensitive to image gray scale change, noise and illumination change, and is suitable for large field of view airborne moving platform high-speed video measurement with large illumination change. However, the existing method lacks effective measures to deal with aliasing noise and other interference, and relies too much on parameter optimization to ensure robustness. Therefore, a phase correlation matching method for inter-frame micro-shift high-speed video measurement is needed. SUMMARY

[0004] The purpose of the present application is to provide a phase correlation matching method for inter-frame micro-shift high-speed video measurement to solve the problems existing in the prior art.

[0005] To achieve the above purpose, the present application is implemented according to the following technical solutions:

[0006] On the one hand, the present application comprises the following steps:

[0007] obtaining a high-frame-rate image sequence;

[0008] obtaining a neighborhood sub-image by dynamically adjusting the image sequence through an adaptive window;

[0009] obtaining a normalized cross power spectrum matrix by performing gradient real and imaginary double-channel Gaussian filtering on the neighborhood sub-image;

[0010] Fitting a plane to the normalized cross power spectrum matrix, dynamically adjusting the initial offset value by noise estimation index and inner point probability after iteration optimization of the sampling subset;

[0011] Based on the initial offset value, update the normalized cross power spectrum to calculate the residual offset value, and iterate until the residual offset value is lower than the preset threshold. Sum all the iteration offset values to get the final sub-pixel offset value.

[0012] Further, the neighborhood sub-image obtained by dynamically adjusting the image sequence by adaptive windowing specifically includes: extracting the feature point set of the first frame of the high-speed moving body target region using the SuperPoint feature network, extracting the feature point neighborhood based on the reference window with the feature point as the center, and quantifying the local features by combining the Sobel gradient, taking the standard deviation of the gradient value in the target region as the threshold to dynamically adjust the window size, wherein the high gradient region with gradient value exceeding the threshold uses a small window, and the smooth region with gradient value lower than the threshold uses a large window, and the segmentation obtains several neighborhood sub-images centered on the feature points as matching templates.

[0013] Further, in order to retain the frequency response of the significant features of the neighborhood sub-image, the image expression form adopts the gradient complex G(x m,n ,y m,n ) instead of the original image intensity function g(x m,n ,y m,n ):

[0014] G(x m,n ,y m,n )=G x (x m,n ,y m,n )+jG y (x m,n ,y m,n ) (1)

[0015] Wherein, G x (x m,n ,y m,n ) and G y (x m,n ,y m,n ) represent the gradients in x and y directions respectively, x m,n and y m,n represent image coordinates, m represents frame number, and n represents point number.

[0016] Construct a normalized cross power spectrum matrix two-dimensional Gaussian filter Q filtered (u m,n ,v m,n ) of real and imaginary parts, and the specific formula is:

[0017]

[0018] Q filtered (u m,n ,v m,n )=Re(Q filtered (u m,n ,v m,n ))+i·Im(Q filtered (u m,n ,v m,n ))(3)

[0019] Where, Q(u) m,n ,v m,n ) represents the normalized cross-power spectrum matrix, F denotes the Fourier transform, * denotes complex conjugate, and u m,n ,v m,n Represents frequency domain coordinates, and The offsets in the horizontal and vertical directions to be calculated, Re(Q) filtered (u m,n ,v m,n )) represents filtering the real part, Im(Q filtered (u m,n ,v m,n )) indicates filtering of the imaginary part.

[0020] Furthermore, the robust estimation of the initial offset value specifically includes plane fitting based on the filtered normalized cross power spectrum matrix using an improved HMSS robust estimation algorithm. The improved HMSS robust estimation algorithm dynamically adjusts the size of the sampling subset through noise estimation index and interior point probability, and then obtains the initial offset value through iterative optimization using the LKOS cost function.

[0021] Furthermore, to ensure that each sampled large subset M has a sufficient number of inliers, the size of the subset is dynamically adjusted using noise estimation σ and inlier probability p:

[0022] M=m+[λ·f(σ,p)] (5)

[0023] Where m is the minimum subset size, f(σ,p) is an adjustment function designed based on the noise estimation and the proportion p of interior points, and λ is the adjustment coefficient. If the noise level σ is low and p is high, then f(σ,p) takes a smaller value; if the noise is high, a larger M is chosen to compensate for the error risk. The parameters of the plane model are finally accurately estimated through iterative optimization using the LKOS cost function.

[0024]

[0025] in, This represents the estimated initial offset value, r. (t) This represents the residual iterative model, where t represents the number of iterations. This indicates the termination condition: the average residual of the data sample set from the first two iterations is less than the residual of the k-th sorting. This indicates the phase angle.

[0026] Furthermore, in the iterative offset value summation, a stop condition of 0.01 pixels is used as the preset threshold, and the final offset value is the sum of all iterative offset values:

[0027]

[0028] in, is the final offset value between frames of the image sequence, and s is the number of robust iterations.

[0029] The beneficial effects of this invention are:

[0030] This invention is a phase correlation matching method for high-speed video measurement with small inter-frame offsets. Compared with the prior art, this invention has the following technical advantages:

[0031] This invention leverages the robustness of frequency domain phase correlation matching to changes in illumination and noise. Through three key technologies—adaptive window neighborhood sub-image extraction, real and virtual dual-channel Gaussian filtering of neighborhood sub-image gradients, and robust estimation and iteration—it achieves accurate identification and tracking of minute inter-frame offset sequences of target points on the surface of high-speed moving objects under the influence of illumination, angle, and other factors, thereby ensuring the accuracy of dynamic monitoring. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the system framework for a high-speed video measurement phase correlation matching method with small inter-frame offsets according to the present invention. Detailed Implementation

[0033] The present invention will be further described below through specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0034] like Figure 1 As shown, a phase correlation matching method for high-speed video measurement with small inter-frame offsets includes the following steps:

[0035] Acquire high frame rate image sequences;

[0036] Neighborhood sub-images are obtained by dynamically adjusting the image sequence using an adaptive window.

[0037] Gradient real and imaginary dual-channel Gaussian filtering is performed on the neighborhood sub-image to obtain the normalized cross power spectrum matrix;

[0038] The normalized cross power spectrum matrix is ​​fitted to a plane, and the initial offset value is obtained by iterative optimization after dynamically adjusting the sampling subset through noise estimation index and interior point probability.

[0039] Based on the initial offset value, update the normalized cross power spectrum to calculate the residual offset value, iterate to the residual offset value below the preset threshold, and sum all the iteration offset values to obtain the final sub-pixel offset value.

[0040] (1) Phase correlation matching method for inter-frame micro-offset high-speed video measurement

[0041] High-speed video measurement can obtain massive image sequence data in a short time, but to obtain the motion information of high-speed moving objects, it is necessary to accurately and stably track the micro-offset sequence. This stage is a key step of high-speed video measurement system of large field of view airborne moving platform, and has important influence on the accuracy and efficiency of subsequent camera pose calibration and three-dimensional coordinate calculation. In actual monitoring process, the targets on the surface of high-speed moving objects have differences in imaging contrast and texture due to the influence of light, angle, etc., resulting in the problem of low tracking accuracy of inter-frame micro-offset of high-speed moving objects. In order to solve this problem, a phase correlation matching method for inter-frame micro-offset high-speed video measurement is proposed, which overcomes the key technologies such as adaptive window neighborhood sub-image extraction, neighborhood sub-image gradient real and imaginary double channel Gaussian filtering, micro-offset robust estimation and robust iteration, and realizes accurate identification and tracking of inter-frame micro-offset of sequence images. The specific technical methods are as follows:

[0042] 1) Adaptive window neighborhood sub-image extraction: define P1=(x 1,1 ,y 1,1 ),...,(x 1,n ,y 1,n ) as the feature point set of the target area of high-speed moving objects extracted by SuperPoint feature network, and n represents the point number. Taking P1=(x 1,1 ,y 1,1 ),...,(x 1,n ,y 1,n ) as the center, an adaptive window adjustment strategy is established to segment several sub-images as matching templates. The feature points neighborhood is extracted by the reference window (such as 32×32), and the local features are quantified by combining Sobel gradient. The standard deviation of gradient value in the target area is used as the threshold to dynamically adjust the window size: the high gradient area exceeds the threshold, and a small window (16×16) is used to improve the calculation speed and suppress interference; the smooth area is below the threshold, and a large window (64×64) is used to enhance the robustness.

[0043] Adaptive window neighborhood sub-image extraction adjusts the template size dynamically according to the local texture, contrast or gradient distribution around the feature points by establishing an adaptive window adjustment strategy, enhancing the robustness of phase correlation matching to noise and illumination changes. Only the necessary high-information area around the target point is extracted as the template to reduce the number of redundant pixels involved in the correlation operation, which not only maintains the matching accuracy, but also avoids the computational burden of traditional global search. This adaptability enables the system to adapt to various complex monitoring environments and requirements.

[0044] 2) Real and imaginary dual-channel Gaussian filtering of neighborhood sub-image gradient: In order to preserve the frequency response of the significant features of the neighborhood sub-image, the gradient complex G(x m,n ,y m,n ) is used instead of the original image intensity function g(x m,n ,y m,n ):

[0045] G(x m,n ,y m,n ) = G x (x m,n ,y m,n ) + jG y (x m,n ,y m,n ) (1)

[0046] where G x (x m,n ,y m,n ) and G y (x m,n ,y m,n ) represent the gradients in the x and y directions, respectively, x m,n and y m,n represent the image coordinates, m represents the frame number, and n represents the point number.

[0047] A two-dimensional Gaussian filter Q filtered (u m,n ,v m,n ) is constructed for the normalized cross-power spectrum matrix of the real and imaginary channels to solve the problem of phase wrapping discontinuity caused by direct smoothing of phase angle data due to aliasing noise from high-frame-rate camera shooting:

[0048]

[0049] Q filtered (u m,n ,v m,n ) = Re(Q filtered (u m,n ,v m,n )) + i·Im(Q filtered (u m,n ,v m,n)) (3)

[0050] where Q(u m,n ,v m,n ) is the normalized cross-power spectrum matrix, F denotes the Fourier transform, * denotes the complex conjugate, u m,n ,v m,n denotes the frequency domain coordinate, and are the horizontal and vertical direction offsets to be solved, Re(Q filtered (u m,n ,v m,n )) denotes the real part filtering, and Im(Q filtered (u m,n ,v m,n )) denotes the imaginary part filtering.

[0051] 3) Micro-shift robust estimation and robust iteration: the shift between the sequence neighborhood sub-images is expressed as the linear phase difference of the normalized cross-power spectrum in the frequency domain:

[0052]

[0053] where denotes the phase angle. In theory, the phase angle is defined by the shift parameters and in the Cartesian coordinate system u-v.

[0054] The neighborhood sub-image gradient real-imaginary dual-channel Gaussian filtering method and the micro-shift robust estimation and robust iteration method use gradient complex numbers to express images, retain the frequency response of the significant features of the neighborhood sub-images, and construct a two-dimensional Gaussian filtering of the normalized cross-power spectrum matrix of the real and imaginary dual channels, effectively solving the problem of phase wrapping discontinuity caused by the direct smoothing of the phase angle data due to the aliasing noise caused by high frame rate shooting of the high-speed camera, and ensuring the accuracy of micro-shift identification. At the same time, the micro-shift robust estimation and robust iteration method uses the improved HMSS robust estimation to calculate the initial shift value and the robust iteration of the residual shift value calculated by continuously updating the normalized cross-power spectrum to obtain the final sub-pixel shift value, thereby improving the tracking accuracy.

[0055] ① Initial shift value robust estimation: the improved HMSS robust estimation algorithm is used for plane fitting, and the adaptive sampling subset size is used to replace the fixed selection of the subset size larger than the minimum subset for plane fitting:

[0056] The local residual distribution of the randomly extracted N points is calculated to obtain a noise estimation index σ. The probability of each data point being an inlier is p. This p value represents the probability of most points in the data being inliers, which is usually determined by the quality of the data and the noise level. If the noise in the data is less, the probability of inliers p will be higher. If the noise is more, the probability of inliers p will be lower. In order to ensure that the large subset M sampled each time has enough inliers, an adjustment mechanism is designed to dynamically adjust the size of the subset using the noise estimation σ and the inlier probability p:

[0057] M = m + [λ·f(σ, p)] (5)

[0058] Where m is the minimum subset size, f(σ, p) is an adjustment function designed according to the noise estimation and inlier probability p, and λ is the adjustment coefficient. If the noise level σ is low and p is high, f(σ, p) takes a smaller value. If the noise is high, a larger M is selected to compensate for the error risk. Finally, the parameters of the estimated plane model are iteratively optimized by the LKOS cost function:

[0059]

[0060] Where, represents the initial offset value estimated, r (t) represents the residual iteration model, t represents the iteration number, represents the termination condition, i.e. the average residual of the data sample set of the previous two iterations is less than the kth sorted residual.

[0061] ② Final offset value sub-pixel phase correlation robust iteration: after obtaining the initial offset value for the first time, the normalized cross power spectrum is updated to calculate the residual offset value, and the iteration is performed until the residual offset value is lower than the threshold (such as 0.01 pixels). The final offset value is the sum of all iteration offset values:

[0062]

[0063] Where, is the final offset value between frames of the sequence image, and s is the number of robust iterations.

[0064] The present application precisely acquires a plurality of neighborhood sub-images centered on the target point, uses image gradient complex instead of original image intensity function as image expression, preserves the frequency response of the significant features of the neighborhood sub-image; constructs a two-dimensional Gaussian filter of the normalized cross power spectrum matrix of the real part and the imaginary part, solves the problem of phase wrapping discontinuity caused by direct smoothing of phase angle data due to aliasing noise caused by high frame rate shooting of high-speed cameras, and obtains the final sub-pixel offset value through robust iteration of updating the residual offset value calculated by the normalized cross power spectrum, so as to realize accurate identification and tracking of the micro offset of the target of the sequence image.

[0065] The above merely describes preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A frame-to-frame micro-shift high-speed video measurement phase correlation matching method, characterized in that, The method comprises the following steps: obtaining a high-frame-rate image sequence; dynamically adjusting the image sequence through an adaptive window to obtain a neighborhood sub-image; performing gradient real and imaginary double-channel Gaussian filtering on the neighborhood sub-image to obtain a normalized cross power spectrum matrix; performing plane fitting on the normalized cross power spectrum matrix, and dynamically adjusting a sampling subset through a noise estimation index and an inlier probability to obtain an initial offset value after iterative optimization; based on the initial offset value, updating the normalized cross power spectrum to calculate a residual offset value, and iterating until the residual offset value is lower than a preset threshold, and summing all the iterative offset values to obtain a final sub-pixel offset value.

2. The frame-to-frame fractional shift high speed video measurement phase correlation matching method of claim 1, wherein, The dynamically adjusting the image sequence through an adaptive window to obtain a neighborhood sub-image specifically comprises: extracting a first-frame feature point set of a high-speed moving object region by using a SuperPoint feature network, extracting a feature point neighborhood based on a reference window with the feature point as the center, quantifying local features by combining a Sobel gradient, and dynamically adjusting a window size by taking a standard deviation of gradient values in the target region as a threshold, wherein a high-gradient region with a gradient value exceeding the threshold uses a small window, and a smooth region with a gradient value lower than the threshold uses a large window, and a plurality of neighborhood sub-images centered on the feature points are segmented as matching templates.

3. The frame-to-frame fractional shift high speed video measurement phase correlation matching method of claim 1, wherein, To preserve the frequency response of the local image salient features, the image representation is adopted to replace the original image intensity function g(x m,n ,y m,n ) with the gradient complex G(x m,n ,y m,n ): G(x m,n ,y m,n ) = G x (x m,n ,y m,n )+jG y (x m,n ,y m,n ) (1) where G x (x m,n ,y m,n ) and G y (x m,n ,y m,n ) represent gradients in the x and y directions, respectively, x m,n and y m,n represent image coordinates, m represents a frame number, and n represents a point number; Constructing real and imaginary dual-channel normalized cross-power spectrum matrix two-dimensional Gaussian filter Q filtered (u m,n ,v m,n ), the specific formula is: Q filtered (u m,n ,v m,n ) = Re(Q filtered (u m,n ,v m,n ))+i·Im(Q filtered (u m,n ,v m,n ))(3) where Q(u m,n ,v m,n ) is a normalized cross-power spectrum matrix, F denotes a Fourier transform, * denotes a complex conjugate, u m,n ,v m,n denote frequency domain coordinates, and are horizontal and vertical direction offsets to be solved, Re(Q filtered (u m,n ,v m,n )) denotes filtering on a real part, and Im(Q filtered (u m,n ,v m,n )) denotes filtering on an imaginary part.

4. The frame-to-frame fractional shift high speed video measurement phase correlation matching method of claim 1, wherein, The robust estimation of the initial offset value specifically comprises: based on the filtered normalized cross power spectrum matrix, performing plane fitting by using an improved HMSS robust estimation algorithm, the improved HMSS robust estimation algorithm dynamically adjusts the size of the sampling subset through a noise estimation index and an inlier probability, and then iteratively optimizes the initial offset value through an LKOS cost function.

5. The frame-to-frame fractional shift high speed video measurement phase correlation matching method of claim 4, wherein, In order to ensure that each sampling large subset M has enough inliers, the size of the subset is dynamically adjusted by using a noise estimation σ and an inlier probability p: M = m + [λ·f(σ, p)] (5) wherein m is the minimum subset size, f(σ, p) is an adjustment function designed according to the noise estimation and the inlier probability p, λ is an adjustment coefficient, if the noise level σ is low and the inlier probability p is high, f(σ, p) takes a smaller value, if the noise is high, a larger M is selected to compensate for the error risk, and the parameters of the plane model are finally accurately estimated through the LKOS cost function iterative optimization: wherein, represents an estimated initial offset value, r (t) represents a residual iterative model, t represents the number of iterations, represents a termination condition, i.e. the average residual of the data sample set of the previous two iterations is less than the kth ordered residual, represents a phase angle.

6. The frame-to-frame fractional shift high speed video measurement phase correlation matching method of claim 1, wherein, In the summing of the iterative offset values, 0.01 pixels is taken as a preset threshold stop condition, and the final offset value is the sum of all the iterative offset values: wherein, is the final offset value between sequence video frames, s is the robust iteration number.

Images