LK optical flow target tracking method based on improved corner detection

Abstract

The invention discloses an LK optical flow target tracking method based on improved corner detection, and the method comprises the following steps: firstly, obtaining a current frame of a video, and extracting candidate corners; angular point responses of the candidate angular points are calculated through a Shii-Tomasi algorithm, angular point responses of non-candidate angular points are set to be zero, and an angular point response matrix R is obtained; s3, calculating a threshold value by using a histogram entropy algorithm and screening the angular point response matrix R; s4, searching a local maximum value for the screened angular point response matrix R and performing non-maximum suppression, wherein a non-zero point of the angular point response matrix R after suppression is the angular point of the current frame; s5, estimating the position of the corner of the current frame in the next frame by using an LK optical flow algorithm, obtaining the behavior track of the corner, and realizing target tracking; according to the method, the Shii-Tomasi corner detection is improved, so that the accuracy of corner detection can be ensured, the detection time can be greatly shortened, and the real-time performance and the accuracy of LK optical flow target tracking are further improved.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to an LK optical flow target tracking method based on improved corner point detection. Background Art

[0002] In the field of computer vision, target tracking is an extremely important technology, which is widely used in video analysis, human-computer interaction and autonomous driving. LK optical flow method is an effective means to achieve target tracking. It analyzes the displacement changes of pixel points in adjacent frames of the video stream based on the assumptions of grayscale invariance, spatial consistency and small motion, and obtains the target motion trajectory, thereby achieving the purpose of target tracking. In order to meet the real-time and accuracy of the optical flow method, the corner point is usually combined with the optical flow method, that is, the target tracking is achieved by tracking the corner points of the target. Therefore, the speed and accuracy of corner point detection will directly affect the effect of LK optical flow target tracking.

[0003] Common corner detection algorithms include Harris corner detection, FAST corner detection, and Shi-Tomasi corner detection. Harris corner detection determines whether a pixel is a corner by calculating the corner response value R of each pixel in the image. Although the algorithm has high recognition accuracy, the process of calculating R is very cumbersome, resulting in slow recognition speed and failure to meet real-time requirements. FAST corner detection establishes a circular window with each pixel as the center, and determines whether it is a corner by judging the difference in brightness between the pixel on the circle and the center pixel. Although the recognition speed is fast, the accuracy is not high and pseudo corners are easily identified. Shi-Tomasi corner detection is based on the improvement of Harris corner detection, and the formula for calculating R is simplified, which greatly shortens the calculation time. Therefore, Shi-Tomasi corner detection is often combined with LK optical flow method to achieve target tracking.

[0004] However, Shi-Tomasi corner detection still has the following shortcomings: first, the algorithm calculates R for each pixel in the image, which takes a long time; second, the threshold needs to adjust the empirical coefficient k, which is inefficient; third, non-maximum suppression uses a fixed window, which is easy to suppress real corners and identify pseudo corners. These shortcomings will directly affect the real-time performance and accuracy of LK optical flow target tracking. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes an LK optical flow target tracking method based on improved corner point detection, which not only shortens the corner point detection time but also improves the corner point detection quality, thereby meeting the real-time and accuracy of LK optical flow target tracking.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] An LK optical flow target tracking method based on improved corner point detection includes the following steps:

[0008] S1, obtain the current frame of the video and extract candidate corner points;

[0009] S2. Use Shi-Tomasi algorithm to calculate the corner point response for the candidate corner points, set the corner point response of non-candidate corner points to 0, and obtain the corner point response matrix R;

[0010] S3, using the histogram entropy algorithm to calculate the threshold and screen the corner point response matrix R;

[0011] S4, searching for the local maximum value of the corner point response matrix R after screening and performing non-maximum suppression, and the non-zero point of the corner point response matrix R after suppression is the corner point of the current frame;

[0012] S5. Use the LK optical flow algorithm to estimate the position of the corner point of the current frame in the next frame, obtain the corner point behavior trajectory, and achieve target tracking;

[0013] Preferably, the step S1 includes the following sub-steps:

[0014] S1-1, use the Canny edge detection algorithm to detect the edge points of the current frame;

[0015] S1-2, use the improved FAST corner detection algorithm to extract candidate corner points from edge points;

[0016] Preferably, the improved FAST corner point detection algorithm extracts candidate corner points in the following specific process:

[0017] A circular window with a radius of 3 is established with each edge point as the center. The four azimuth points on the circle, namely the top, bottom, left, and right, are compared with the center point of the circle. If two or more azimuth points satisfy the formula: k -I center |>Th, the edge point is considered to be a candidate corner point, where I k is the gray value of a certain point, I center is the gray value of the center point of the circle, Th is the threshold value set to 0.2*I center .

[0018] Preferably, step S3 includes the following sub-steps:

[0019] S3-1. Sort the non-zero values ​​of the diagonal point response matrix R from small to large and merge the same values ​​to obtain nonzeroR = [r0, r1, ..., r n-1 ], where r iis the i-th non-zero value, n is the number of non-zero values, and r0 <r1<...<r n-1 ;

[0020] S3-2. Calculate the number of each item of nonzeroR in the corner point response matrix R, and obtain the non-zero value histogram of the corner point response matrix R Hist = [m0,m1,...,m n-1 ], where m i Yes i The number of corner point responses in the matrix R;

[0021] S3-3. Calculate the probability histogram normHist = [p0, p1, ..., p n-1 ], the calculation formula is: p i =m i / nonzeronum(0≤i≤n-1), where p i Yes i The probability distribution of , nonzeronum is the total number of non-zero values ​​of the corner point response matrix R;

[0022] S3-4. Calculate the cumulative probability histogram cumunormHist = [z0, z1, ..., z n-1 ], the calculation formula is: where z i Yes i The cumulative probability distribution of

[0023] S3-5, calculate the entropy of each non-zero value entropy = [e0, e1, ..., e n-1 ], the calculation formula is: where e i Yes i Entropy of

[0024] S3-6. Calculate the sum of entropy before and after the segmentation with index i as the boundary f = [f0,f1,...,f n-1 ], the calculation formula is: where f i It is the sum of entropy before and after the segmentation with i as the boundary;

[0025] S3-7, traverse each item of f, find the index i of the maximum value, and obtain the threshold nonzeroR[i];

[0026] S3-8. Compare the non-zero values ​​of the corner point response matrix R with the threshold, and discard the values ​​less than the threshold as 0.

[0027] Preferably, step S4 includes the following sub-steps:

[0028] S4-1. Establish a 3x3 window with each non-zero point of the corner point response matrix R as the center. If the center point is the maximum value in the window, it is the local maximum and is retained. Otherwise, it is discarded and set to 0.

[0029] S4-2. Determine the size of the circular window for non-maximum suppression. Centered on the local maximum point of the corner point response matrix R, establish a circular window with a radius of 2 to calculate the corner point density ρ. The calculation formula is: Where num is the number of local maximum points in a circular window with a radius of 2; if If the corner point density is low, a large circular window with a radius of 4 is used for suppression; otherwise, if the corner point density is high, a small circular window with a radius of 2 is used for suppression;

[0030] S4-3, taking each local maximum point of the corner point response matrix R as the center, and performing non-maximum suppression according to the circular window size determined in step S4-2, the suppression method is: only retaining the center point of the circular window, and discarding the remaining points in the window and setting them to 0;

[0031] S4-4. The non-zero point position of the corner point response matrix R after non-maximum suppression is the corner point position of the current frame, thereby obtaining the corner point.

[0032] The present invention has the following characteristics and beneficial effects:

[0033] By adopting the above technical scheme, the present invention is based on the LK optical flow target tracking method of improved corner point detection, adds a candidate corner point extraction link to the Shi-Tomasi corner point detection algorithm, changes the threshold selection and non-maximum suppression methods, so that the corner point detection speed is faster and the quality is higher, thereby meeting the real-time and accuracy of LK optical flow target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 The present invention is a flowchart of an LK optical flow target tracking method based on improved corner point detection according to an embodiment of the present invention.

[0036] Figure 2 This is an experimental comparison diagram of a corner detection algorithm of an LK optical flow target tracking method based on improved corner detection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0040] The embodiment of the present invention provides a LK optical flow target tracking method based on improved corner point detection, such as Figure 1 As shown, the following steps are included:

[0041] S1, obtain the current frame of the video and extract candidate corner points;

[0042] The step S1 comprises the following sub-steps:

[0043] S1-1, convert the current frame to grayscale, and then use Gaussian filtering to smooth and denoise, where the Gaussian kernel size is 11x11; then perform Canny edge detection on the filtered image to obtain the image edge points, where the low threshold is 10 and the high threshold is 70;

[0044] S1-2, use the improved FAST corner detection algorithm to extract candidate corner points from edge points;

[0045] Furthermore, the improved FAST corner point detection algorithm extracts candidate corner points in the following specific process:

[0046] A circular window with a radius of 3 is established with each edge point as the center. The four azimuth points on the circle, namely the top, bottom, left, and right, are compared with the center point of the circle. If two or more azimuth points satisfy the formula: k -I center |>Th, the edge point is considered to be a candidate corner point, where I k is the gray value of a certain point, I center is the gray value of the center point of the circle, Th is the threshold value set to 0.2*I center .

[0047] S2. Use Shi-Tomasi algorithm to calculate the corner point response for the candidate corner points, set the corner point response of non-candidate corner points to 0, and obtain the corner point response matrix R;

[0048] Specifically, in this embodiment, step S2 includes the following sub-steps:

[0049] S2-1. Create a 3x3 window with each candidate corner point as the center, and use the Sobel operator to calculate the gradient I of each point in the window along the x direction. x and the gradient I along the y direction y , and then multiply it by the coefficient Scale, I x ,I y Normalized to [0,1]; the coefficient scale calculation formula is: Where sobel_size is the window size when the Sobel operator calculates the gradient, which is set to 3;

[0050] S2-2, I obtained according to step S2-1 x ,I y , calculate I for each point in the 3x3 window xx ,I yy and I xy , the calculation formula is: xx =I x *I x , I yy =I y *I y , I xy =I x *I y ;

[0051] S2-3, according to step S2-2 obtained I xx ,I yy and I xy , calculate the elements A, B and C of the autocorrelation matrix M of each candidate corner point, where The calculation formula is: A = ∑ w＝3x3 I xx , B=∑ w＝3x3 I yy , C = ∑ w＝3x3 I xy , where w = 3x3 refers to a 3x3 window centered at each candidate corner point;

[0052] S2-4. Calculate the corner point response R of each candidate corner point based on A, B and C obtained in step S2-3. i , the calculation formula is: Among them, λ1 and λ2 are the two characteristic roots of the autocorrelation matrix;

[0053] S2-5. Set the corner point responses of non-candidate corner points to 0, and combine them with the corner point responses of candidate corner points obtained in step S2-4 to obtain a corner point response matrix R with the same height and width as the current frame of the video.

[0054] S3, using the histogram entropy algorithm to calculate the threshold and screen the corner point response matrix R;

[0055] The step S3 comprises the following sub-steps:

[0056] S3-1, sort the non-zero values ​​of the diagonal point response matrix R from small to large and merge the same values ​​to obtain nonzeroR = [r0, r1, ..., r n-1 ], where r i is the i-th non-zero value, n is the number of non-zero values, and r0 <r1<...<r n-1 ;

[0057] S3-2. Calculate the number of each item of nonzeroR in the corner point response matrix R, and obtain the non-zero value histogram of the corner point response matrix R Hist = [m0,m1,...,m n-1 ], where m i Yes i The number of corner point responses in the matrix R;

[0058] S3-3. Calculate the probability histogram normHist = [p0, p1, ..., p n-1 ], the calculation formula is: p i =m i / nonzeronum(0≤i≤n-1), where p i Yes i The probability distribution of , nonzeronum is the total number of non-zero values ​​of the corner point response matrix R;

[0059] S3-4. Calculate the cumulative probability histogram cumunormHist = [z0, z1, ..., z n-1 ], the calculation formula is: where z i Yes i The cumulative probability distribution of

[0060] S3-5, calculate the entropy of each non-zero value entropy = [e0, e1, ..., e n-1 ], the calculation formula is: where e i Yes i Entropy of

[0061] S3-6. Calculate the sum of entropy before and after the segmentation with index i as the boundary f = [f0,f1,...,f n-1 ], the calculation formula is: where f i It is the sum of entropy before and after the segmentation with i as the boundary;

[0062] S3-7, traverse each item of f, find the index i of the maximum value, and obtain the threshold nonzeroR[i];

[0063] S3-8. Compare the non-zero values ​​of the corner point response matrix R with the threshold nonzeroR[i], and discard the values ​​less than the threshold nonzeroR[i] and set them to 0.

[0064] S4, searching for the local maximum value of the corner point response matrix R after screening and performing non-maximum suppression, and the non-zero point of the corner point response matrix R after suppression is the corner point of the current frame;

[0065] The step S4 comprises the following sub-steps:

[0066] S4-1. Establish a 3x3 window with each non-zero point of the corner point response matrix R as the center. If the center point is the maximum value in the window, it is the local maximum and is retained. Otherwise, it is discarded and set to 0.

[0067] S4-2. Determine the size of the circular window for non-maximum suppression. Centered on the local maximum point of the corner point response matrix R, establish a circular window with a radius of 2 to calculate the corner point density ρ. The calculation formula is: Where num is the number of local maximum points in a circular window with a radius of 2; if If the corner point density is low, a large circular window with a radius of 4 is used for suppression; otherwise, if the corner point density is high, a small circular window with a radius of 2 is used for suppression;

[0068] S4-3, taking each local maximum point of the corner point response matrix R as the center, and performing non-maximum suppression according to the circular window size determined in step S4-2, the suppression method is: only retaining the center point of the circular window, and discarding the remaining points in the window and setting them to 0;

[0069] S4-4. The non-zero point position of the corner point response matrix R after non-maximum suppression is the corner point position of the current frame, thereby obtaining the corner point.

[0070] S5. Use the LK optical flow algorithm to estimate the position of the corner point of the current frame in the next frame, obtain the corner point behavior trajectory, and achieve target tracking;

[0071] The step S5 comprises the following sub-steps:

[0072] S5-1. Based on the grayscale invariance assumption, establish the grayscale equation between the current frame and the next frame: I(x+dx,y+dy,t+dt)=I(x,y,t), where I(x,y,t) represents the grayscale value of the corner point at the position (x,y) of the current frame, and I(x+dx,y+dy,t+dt) represents the grayscale value of the corner point at the position (x,y) of the current frame after one frame time dt moves to the position (x+dx,y+dy) of the next frame;

[0073] S5-2. Perform a first-order Taylor expansion on I(x+dx,y+dy,t+dt) and retain the first-order terms to obtain I(x,y,t)+I x dx+I y dy+I t dt, and substitute it into the equation established in step S5-1 to obtain I x dx+I y dy+I t dt=0, divide both sides by dt and move to get Will Denoted as u, Recorded as v, further get I x u+I y v=-I t , where I x is the gradient along the x direction, I y is the gradient along the y direction, I t is the gradient along the t direction, u is the instantaneous velocity of the corner point along the x direction, and v is the instantaneous velocity of the corner point along the y direction;

[0074] S5-3, based on the spatial consistency assumption, a 15x15 window is established with each corner point as the center, and each pixel point in the window satisfies the formula I derived in step S5-2 x u+I y v=-I t , substituting into the matrix equation Using the least squares method, we can obtain in

[0075] S5-4, the instantaneous speed of the corner point of the current frame Multiply it by the frame time dt to get the position of the corner point of the current frame in the next frame, so as to obtain the corner point behavior trajectory and realize target tracking.

[0076] The following is the experimental part of the present invention:

[0077] In order to verify that the improved corner detection algorithm proposed in this paper can effectively improve the running speed and effect of LK optical flow target tracking, it is compared with the classic corner detection algorithms Shi-Tomasi and Harris. The comparative experimental method is to use the above three algorithms to perform corner detection on the same image respectively, and compare them based on the detection time and the proportion of accurate corner points. The detection results of the three algorithms are shown in Figure 2. Figure 2 And as shown in Table 1.

[0078] Table 1 Comparison of experimental results of various algorithms

[0079]

[0080] from Figure 2 As can be seen from Table 1, Shi-Tomasi and Harris will extract more pseudo corner points, while the improved corner detection algorithm proposed in the present invention can effectively remove pseudo corner points and improve the accuracy of corner detection by calculating the screening threshold through the histogram entropy algorithm and determining the circular window size of non-maximum suppression according to the corner density. Secondly, Shi-Tomasi and Harris calculate the corner response R for each pixel of the image, which makes the detection time longer, while the improved corner detection algorithm proposed in the present invention calculates the corner response R only for the candidate corner points by extracting candidate corner points, which effectively shortens the detection time. In summary, the improved corner detection algorithm proposed in the present invention has more advantages in corner detection quality and detection speed. Using the detected corner points as the target feature points of the LK optical flow can further improve the running rate and effect of the target tracking algorithm.

[0081] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A LK optical flow target tracking method based on improved corner point detection, characterized in that: The following steps are involved: S1, obtain the current frame of the video and extract candidate corner points; S2, using the Shi-Tomasi algorithm to calculate the corner point response for the candidate corner points, and setting the corner point response of the non-candidate corner points that have not been extracted to 0, thereby obtaining the corner point response matrix R; S3, using the histogram entropy algorithm to calculate the threshold and screen the corner point response matrix R; S3-1. Sort the non-zero values ​​of the diagonal point response matrix R from small to large and merge the same values ​​to obtain nonzeroR = [r0, r1, ..., r n-1 ], where r i is the i-th non-zero value, n is the number of non-zero values, and r0<r1<…<r n-1 ; S3-2. Calculate the number of each item of nonzeroR in the corner point response matrix R, and obtain the non-zero value histogram of the corner point response matrix R: Hist = [m0, m1, ..., m n-1 ], where m i Yes i The number of corner point responses in the matrix R; S3-3. Calculate the probability histogram normHist = [p0, p1, ..., p n-1 ], the calculation formula is: p i =m i / nonzeronum(0≤i≤n-1), where p i Yes i The probability distribution of , nonzeronum is the total number of non-zero values ​​of the corner point response matrix R; S3-4. Calculate the cumulative probability histogram cumunormHist = [z0, z1, ..., z n-1 ], the calculation formula is: where z i Yes i The cumulative probability distribution of S3-5. Calculate the entropy of each non-zero value entropy = [e0, e1, ..., e n-1 ], the calculation formula is: where e i Yes i Entropy of S3-6. Calculate the sum of entropies before and after the segmentation with index i as the boundary, f = [f0, f1, ..., f n-1 ], the calculation formula is: where f i It is the sum of entropy before and after the segmentation with i as the boundary; S3-7, traverse each item of f, find the index i of the maximum value, and obtain the threshold nonzeroR[i]; S3-8, compare the non-zero values ​​of the corner point response matrix R with the threshold nonzeroR[i], and discard the values ​​less than the threshold nonzeroR[i] as 0; S4, searching for the local maximum value of the corner point response matrix R after screening and performing non-maximum suppression, and the non-zero point of the corner point response matrix R after suppression is the corner point of the current frame; S4-1. Establish a 3x3 window with each non-zero point of the corner point response matrix R as the center. If the center point is the maximum value in the window, it is the local maximum and is retained. Otherwise, it is discarded and set to 0. S4-2, taking each local maximum point of the corner point response matrix R as the center, a circular window is established according to the corner point density, and then only the center point of the circular window is retained, and the remaining points in the window are discarded and set to 0, thereby completing non-maximum suppression; S4-3, the non-zero point position of the corner point response matrix R after non-maximum suppression is the corner point position of the current frame, thereby obtaining the corner point; S5. Use the LK optical flow algorithm to estimate the position of the corner point of the current frame in the next frame, obtain the corner point behavior trajectory, and realize target tracking.

2. The LK optical flow target tracking method based on improved corner detection according to claim 1, characterized in that: The step S1 comprises the following sub-steps: S1-1, use the Canny edge detection algorithm to detect the edge points of the current frame; S1-2. Use the corner detection algorithm to extract candidate corner points from edge points.

3. The LK optical flow target tracking method based on improved corner detection according to claim 2, characterized in that: In step S1-2, an improved FAST corner point detection algorithm is used to extract candidate corner points. The improved FAST corner point detection algorithm is as follows: a circular window with a radius of 3 is established with each edge point as the center, and the four azimuth points of the upper, lower, left, and right on the circle are compared with the center point of the circle respectively. If there are two or more azimuth points that satisfy the formula: k -I center |>Th, the edge point is considered to be a candidate corner point, where I k is the gray value of a certain point, I center is the gray value of the center point of the circle, Th is the threshold value set to 0.2*I center .

4. The LK optical flow target tracking method based on improved corner detection according to claim 1, characterized in that: In step S4-2, the method for establishing a circular window according to the corner point density is as follows: taking the local maximum point of the corner point response matrix R as the center, establishing a circular window with a radius of 2 to calculate the corner point density ρ, if If the corner point density is low, a large circular window with a radius of 4 is used to suppress it. Otherwise, if the corner point density is high, a small circular window with a radius of 2 is used to suppress it.

5. The LK optical flow target tracking method based on improved corner detection according to claim 4 is characterized in that: The corner point density calculation formula is: Where num is the number of local maximum points in a circular window with a radius of 2.

Images