A video flow measurement method and device based on LK tracking and grayscale statistical feature method

Abstract

The present invention relates to a video flow measurement method and device based on LK tracking and grayscale statistical feature methods, including using the LK tracking algorithm to track speed measurement points to form speed measurement point tracking trajectories for the first ten frames of a video; enhancing the texture of the spatiotemporal image trajectory and suppressing background noise and stripe direction noise using a Gaussian directional stretching filter, a noise suppression function, and a directional filtering function; using a random Hough transform to detect the straight angles of the spatiotemporal image to obtain an approximate rotation range; statistically calculating the grayscale histogram of each angle of the rotated spatiotemporal image; calculating the eigenvalue of the grayscale histogram; and drawing a characteristic curve to obtain the optimal texture angle. The present invention can automatically mark speed measurement lines, effectively remove noise from the spatiotemporal image and enhance the trajectory, identify clear spatiotemporal image texture, and thereby improve the robustness, applicability, and accuracy of the spatiotemporal image measurement method, effectively overcome the influence of the complex environment in natural river channels on the measurement results, and improve detection accuracy.

Description

Technical Field

[0001] The invention relates to a video flow measurement method and device based on LK tracking and grayscale statistical feature method, belonging to the technical field of river flow measurement. Background Art

[0002] Flow velocity is a fundamental hydrological element of rivers. Measuring river flow velocity is beneficial for water resource management, flood prevention, river morphology research, and water resource protection. Accurately measuring river flow velocity is a crucial task in hydrological information monitoring, particularly given the numerous rivers in my country. River flow velocity measurements in my country often require complex environmental conditions, which can complicate instrument placement and require high operator skill levels, making traditional contact measurement methods impractical. With the development of water conservancy informatization and the widespread adoption of network video surveillance systems, many key hydrological observatories in my country are equipped with these systems, providing favorable conditions for video-based river flow velocity measurement.

[0003] Spatiotemporal image velocimetry offers significant advantages for real-time monitoring of river surface velocity and flow, due to its high spatial resolution, strong real-time performance, and safety and simplicity. Existing spatiotemporal image velocimetry methods typically adjust the camera angle so that the riverbanks appear parallel to the frame. Velocimetry lines are then set along the direction of the measured flow, generating a spatiotemporal image. Velocimetry is then performed to identify the principal texture direction to calculate the flow velocity. However, the vast majority of natural rivers in my country have complex and diverse environmental morphologies and unstable flow characteristics, with winding banks often present. Adjusting the camera to achieve parallel riverbanks in the frame makes setting velocimetry lines difficult and highly subjective. The complex and variable water surface imaging environment of the synthesized spatiotemporal images introduces significant background noise and fringe noise, which can affect texture detection. Traditional methods for identifying principal texture directions, including the tensor gradient method, the two-dimensional autocorrelation function method, and the edge spectrum analysis method, are highly sensitive to noise, which can easily lead to errors or even failure in identification. These methods lack robust and reliable features, making them unsuitable for use in complex natural river environments. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a video flow measurement method and device based on LK tracking and grayscale statistical feature method, which can automatically generate speed measurement lines, effectively remove noise from spatiotemporal images, accurately find the optimal texture angle, and accurately measure the flow rate of rivers in natural rivers in complex environments.

[0005] The technical solution of the present invention is as follows: In a first aspect, the present invention provides a video flow measurement method based on LK tracking and grayscale statistical feature method, comprising the following process:

[0006] Convert each frame image in the video stream into a grayscale image;

[0007] Get the starting point and end point of the cross section line, and determine the speed measurement point and the corresponding block area based on the starting point and end point of the cross section line;

[0008] Calculate the inter-frame displacement of each speed measurement point based on the LK tracking algorithm, and obtain the displacement coordinates of the speed measurement points in the first ten frames;

[0009] According to the displacement coordinates of the first ten frames of each speed measurement point, the least squares straight line fitting algorithm is used to fit the displacement coordinates of the first ten frames of each speed measurement point to obtain the straight line equation and slope;

[0010] Draw a speed measurement line at the corresponding speed measurement point according to the straight line equation and slope corresponding to each speed measurement point, and generate a corresponding space-time image diagram based on the speed measurement line;

[0011] Gaussian directional stretching filter is used to preprocess the spatiotemporal image;

[0012] Construct noise suppression function and directional filtering function to suppress background noise and filter horizontal stripe noise of preprocessed spatiotemporal images;

[0013] Random Hough transform is used to detect straight lines in the space-time image after noise removal and obtain the average angle to determine the approximate range of the trajectory direction;

[0014] The processed spatiotemporal image is rotated at different angles with a certain step size within the approximate range of the trajectory direction to obtain the grayscale histogram of each angle;

[0015] The grayscale histogram of each angle is calculated using the feature statistics method to obtain the eigenvalue and draw the feature curve, and the peak value is found to obtain the texture angle corresponding to the optimal trajectory direction;

[0016] The flow velocity and flow rate of the velocity measurement line are calculated based on the texture angle corresponding to the optimal trajectory direction.

[0017] Furthermore, the determining of the speed measurement point and the corresponding block area according to the starting point and the end point of the cross-section line includes:

[0018] According to the starting point and end point of the cross-section line, the line connecting the starting point to the end point of the cross-section line is segmented, and the segmentation points on the cross-section line are used as speed measurement points;

[0019] For any speed measurement point P is (x,y) T , the corresponding block area G(P) is (xr; x+r, yr; y+r), where r is the larger of the absolute values ​​of half the distance in the x direction and half the distance in the y direction of each segment after segmentation.

[0020] Furthermore, the segmentation adopts the principle of equal division, and the number of segments is determined by the number of cross-sectional data.

[0021] Furthermore, the inter-frame displacement of each speed measurement point is calculated based on the LK tracking algorithm, and the displacement coordinates of the speed measurement points of the first ten frames are obtained, including:

[0022] Align the block area I(P) of the next frame image with the block area G(P) of the speed measurement point of the current frame image. The mapping from I(P) to G(P) is w(P;q), where q is the transformation parameter (q1, q2, ..., q n ) T ,q1,q2,…,q n is the transformation vector, then w(P;q) is as shown in formula (1):

[0023]

[0024] The transformed image of w(P;q) is I(w(P;q)), and the displacement between I(w(P;q)) and G(P) is minimized by Δq, as shown in formula (2):

[0025]

[0026] Minimizing Δq is nonlinear, and the vector q is iterated, as shown in formula (3):

[0027]

[0028] Given a threshold ε, stop when ||Δq||≤ε, find the vector q, and then find the image coordinates of the tracking point in the next frame, and get the tracking point coordinates (x i ,y i ) T .

[0029] Furthermore, the least squares straight line fitting algorithm is used to fit the displacement coordinates of the first ten frames of each speed measurement point to obtain the straight line equation and slope, including:

[0030] The ten coordinates of the trajectory points are fitted with a straight line using the least squares method, as shown in formula (4):

[0031]

[0032] When the minimum value is taken in formula (4), the equation and slope of the straight line are obtained.

[0033] Furthermore, drawing a speed measurement line at each speed measurement point according to the linear equation and slope corresponding to each speed measurement point includes:

[0034] At the speed measurement point coordinate P = (x, y) TTake xn / 2 and x+n / 2 as the horizontal coordinates of the starting point and end point of the speed measurement line respectively and substitute them into the above straight line equation to obtain the starting point coordinates and end point coordinates of the speed measurement line, where n is the required length of the speed measurement line. Then draw the speed measurement line according to the starting point coordinates and end point coordinates.

[0035] Furthermore, generating a corresponding spatiotemporal image diagram according to the speed measurement line includes:

[0036] By plotting the changes in the brightness of the pixels on the velocity line over time on an image, the horizontal axis represents the length of the velocity line and the vertical axis represents the time axis. The resulting image is called a space-time image.

[0037] Furthermore, the preprocessing of the spatiotemporal image using a Gaussian directional stretching filter includes:

[0038] The linear structure of the spatiotemporal image is enhanced using the Gaussian directional stretching filter l(x,y), as shown in formula (5):

[0039]

[0040] Among them, σ represents the filter factor that controls the filter size; γ represents the filter factor that controls the filter stretch length.

[0041] The filter is rotated to detect trajectories in different directions, so the rotation matrix R is defined as shown in formula (6):

[0042]

[0043] Convolve l(x,y) with the rotation matrix R. The direction θ corresponding to the pixel with the maximum response in the convolution result is * When the image is enhanced, the enhanced image is recorded as As shown in formula (7):

[0044]

[0045] Furthermore, the construction of the noise suppression function and the directional filtering function to suppress background noise and filter horizontal stripe noise on the preprocessed spatiotemporal image includes:

[0046] 1) Taking advantage of the fact that the enhanced image still obeys the Gaussian distribution, the probability density function Q(X) of the pixel point X in the image is defined as shown in formula (8):

[0047] Q(X)=w n Q n (X)+(1-w)Q t (X) (8)

[0048] Among them, Qn (X) is the probability density function of noise-related pixels; Q t (X) is the probability density function of trajectory-related pixels, w n is the weighting coefficient 0<w<1. n (X) is shown in formula (9):

[0049]

[0050] Among them, u n represents the mean value of the Gaussian distribution function, σ n Represents the variance of the Gaussian distribution function.

[0051] By controlling the weight coefficient, high weights protect pixels related to the trajectory, while low weights weaken and filter out pixels related to noise. Therefore, the noise suppression function N(X) is shown in formula (10):

[0052]

[0053] By expanding N(X) into a two-dimensional function and applying it to the spatiotemporal image to be processed, the noise-suppressed image is obtained.

[0054] 2) Calculate the main direction of the image texture. First, construct the direction filter h θ (x,y), and perform Hilbert transform to H θ (x,y), as shown in formula (11):

[0055] H θ (x,y)=Hibert{h θ (x,y)} (11)

[0056] image Directional texture energy E θ (x,y), as shown in formula (12):

[0057]

[0058] Take E θ The angle direction at the maximum value of (x,y) is the main direction of the texture θ max , as shown in formula (13):

[0059] θ max =argmax{E θ (x,y)|θ∈(0,π]} (13)

[0060] Therefore, the directional filter function N2(X) is as shown in formula (14):

[0061]

[0062] Among them, σ represents the scaling function; θ * Indicates the angle of the enhanced image; 0<g2(x,y)<1. Use the directional filter to max The direction of max The direction is removed, so that the noise can be filtered out and the final processed spatiotemporal image is obtained.

[0063] Furthermore, the method of using random Hough transform to detect straight lines in the space-time image after noise removal and obtaining an average angle to determine the approximate range of the trajectory direction includes:

[0064] Random Hough transform is used to detect the trajectory direction of the processed spatiotemporal image, and ρ is used to represent the detected straight line, as shown in formula (15):

[0065] ρ=x i cosθ i +y i sinθ i (15)

[0066] Among them, (x i ,y i ) represents the coordinates of the trajectory line in the space-time image; θ i represents the angle between the trajectory line and the vertical vector line; ρ represents the length of the trajectory line.

[0067] After obtaining K fuzzy straight lines, calculate the average value of the inclination angles of all straight lines Get a rough direction of the trajectory direction.

[0068] Furthermore, the processed spatiotemporal image is rotated at different angles with a certain step length within the approximate range of the trajectory direction to obtain a grayscale histogram at each angle, including:

[0069] Set a threshold ε to determine the rotation range of the projection transformation

[0070] The enhanced denoised spatiotemporal image is rotated within the range The rotation was performed in 0.5° steps.

[0071] For each space-time image after rotation angle θ, the grayscale values ​​of each column of pixels in the vertical direction are accumulated to draw the grayscale histogram f θ , for the vertical projection f of n columns θn , as shown in formula (16):

[0072] f θn =∑ M c nm(16)

[0073] Among them, c nm represents the grayscale value of the mth pixel in the projection direction of the nth column. In this way, all grayscale histograms of the spatiotemporal image are obtained for every 0.5° rotation within the rotation range.

[0074] Furthermore, the method of calculating the grayscale histogram of each angle using feature statistics to obtain feature values ​​and draw a feature curve includes:

[0075] Calculate the Renyi entropy H of the grayscale histogram at each angle θα (V), as shown in formula (17):

[0076]

[0077] Where V represents a discrete random variable and α represents the order of Renyi entropy.

[0078] Calculate the variance D of the grayscale histogram at each angle θ , as shown in formula (18):

[0079]

[0080] The weighted sum of Renyi entropy and variance is calculated using weighted coefficients w1 and w2 to obtain the characteristic statistical value S for each angle. θ , as shown in formula (19):

[0081] S θ =w1*H θα (V)+w2*D θ (V) (19)

[0082] The characteristic statistics S calculated from all angles θ Draw characteristic statistical curves.

[0083] Furthermore, the best texture angle is the angle corresponding to the peak position in the statistical characteristic curve, which is the best spatiotemporal image texture trajectory direction angle.

[0084] Furthermore, the flow velocity and flow rate of the speed measurement line are calculated according to the texture angle corresponding to the optimal trajectory direction, including:

[0085] The surface flow characteristics of the river move along the velocity measurement line for a distance D within the time T, which corresponds to a movement of d pixels within the τ frame in the image coordinate system. The velocity vector of the velocity measurement line is shown in formula (20):

[0086]

[0087] Among them Sx Indicates the actual distance represented by each pixel (unit: m / pixel), fps indicates the camera frame rate (unit: pixel / s), Represents the tangent value of the slope of the space-time image.

[0088] Furthermore, the flow rate is calculated using the velocity-area method, as shown in formula (21):

[0089]

[0090] Where η is the surface velocity coefficient, V i represents the surface velocity of the interval, A i Represents the area of ​​the interval.

[0091] On the other hand, a video flow measurement device based on LK tracking and grayscale statistical feature method is provided, comprising:

[0092] The frame image acquisition module is used to convert each frame image in the video stream into a grayscale image;

[0093] The block region displacement calculation module is used to calculate the average value of the inter-frame displacement of the block region between all adjacent frames in the video stream as the inter-frame displacement of the block region;

[0094] The straight line fitting module is used to fit the tracking point trajectory coordinates into a straight line to obtain the straight line equation and slope;

[0095] The image processing module is used to enhance the trajectory texture of the spatiotemporal image through the Gaussian pull-up filter, and to suppress the background noise and stripe direction noise using the noise suppression function and directional filter function respectively;

[0096] Line detection module, used to detect the angle of the line in the spatiotemporal image and obtain the approximate area of ​​rotation;

[0097] Grayscale statistics module, used to count the grayscale value of each angle to obtain its corresponding grayscale histogram;

[0098] The feature statistics calculation module is used to calculate the grayscale histogram feature value of each angle and statistically compile it into a feature curve to obtain the optimal texture angle;

[0099] and a flow calculation module for calculating the flow velocity at each measuring point according to the optimal texture angle.

[0100] Compared with the prior art, the present invention has the following beneficial effects:

[0101] 1) Compared with the existing technology, the completely non-contact video flow measurement is adopted. The LK tracking algorithm is combined to realize automatic marking of speed measurement lines, which enables the subsequently generated spatiotemporal images to retain more complete texture information.

[0102] 2) Using Gaussian second-order stretching filter to perform image enhancement preprocessing, and constructing noise suppression filter function and directional filter function, can effectively improve the contrast and robustness of spatiotemporal images, enhance trajectory texture, and suppress background and horizontal stripe noise.

[0103] 3) The texture of spatiotemporal images is detected using the projection transformation feature statistical algorithm, which can accurately measure the texture angle of spatiotemporal images, significantly shorten the measurement time, and improve work efficiency.

[0104] 4) Compared with the existing widely used flow measurement methods, the present invention is more applicable to the detection of natural river channels. It can automatically mark the speed measurement line according to the river flow direction, effectively filter out the interference of unstable factors in the natural environment on the spatiotemporal image speed measurement method, greatly improve the measurement accuracy, enhance the robustness, improve the algorithm efficiency, and the video acquisition equipment is relatively simple to maintain. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments and in conjunction with the accompanying drawings, wherein:

[0106] Figure 1 Schematic diagram of the processing flow of the method of the present invention;

[0107] Figure 2 This is a schematic diagram of the marking points and section lines in step 2;

[0108] Figure 3 This is the speed measurement point tracking trajectory diagram in step 3;

[0109] Figure 4 The first ten frames of speed measurement point tracking trajectory diagram in step 3;

[0110] Figure 5 Fit a straight line graph to the coordinates of the tracking points for the first ten frames in step 4;

[0111] Figure 6 Draw a graph for the speed measurement line in step 5;

[0112] Figure 7 is the spatiotemporal image of step 6;

[0113] Figure 8 Enhance the pre-processed image with the Gaussian directional stretch filter in step 7;

[0114] Figure 9 is the background noise suppression map of step 8;

[0115] Figure 10 This is the stripe noise filter image of step 8;

[0116] Figure 11 It is the random Hough detection straight line graph of step 9;

[0117] Figure 12 This is the statistical principle diagram of the grayscale histogram in step 10;

[0118] Figure 13 This is the characteristic statistical curve diagram of step 12. DETAILED DESCRIPTION

[0119] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0120] The technical concept of the present invention is: combining the LK tracking algorithm to track the speed measurement point to form the motion trajectory of the speed measurement point, fitting the motion trajectory of the first ten frames to obtain the straight line equation and slope, arranging the speed measurement line at the speed measurement point according to the straight line equation and slope and generating a space-time image, using the Gaussian directional stretching filter, noise suppression function and directional filtering function to enhance the space-time image preprocessing and denoising, and finally using the grayscale statistical feature method to perform texture angle detection on the processed space-time image to calculate the river surface flow rate.

[0121] Example 1

[0122] The present invention is a video flow measurement method based on LK tracking and grayscale statistical feature method, see Figure 1 As shown, the following process is included:

[0123] Convert each frame image in the video stream into a grayscale image;

[0124] Get the starting point and end point of the cross section line, and determine the speed measurement point and the corresponding block area based on the starting point and end point of the cross section line;

[0125] Calculate the inter-frame displacement of each speed measurement point based on the LK tracking algorithm, and obtain the displacement coordinates of the speed measurement points in the first ten frames;

[0126] According to the displacement coordinates of the first ten frames of each speed measurement point, the least squares straight line fitting algorithm is used to fit the displacement coordinates of the first ten frames of each speed measurement point to obtain the straight line equation and slope;

[0127] Draw a speed measurement line at the corresponding speed measurement point according to the straight line equation and slope corresponding to each speed measurement point, and generate a corresponding space-time image diagram based on the speed measurement line;

[0128] Gaussian directional stretching filter is used to preprocess the spatiotemporal image;

[0129] Construct noise suppression function and directional filtering function to suppress background noise and filter horizontal stripe noise of preprocessed spatiotemporal images;

[0130] Random Hough transform is used to detect straight lines in the space-time image after noise removal and obtain the average angle to determine the approximate range of the trajectory direction;

[0131] The processed spatiotemporal image is rotated at different angles with a certain step size within the approximate range of the trajectory direction to obtain the grayscale histogram of each angle;

[0132] The grayscale histogram of each angle is calculated using the feature statistics method to obtain the eigenvalue and draw the feature curve, and the peak is found to obtain the texture angle corresponding to the optimal trajectory direction;

[0133] The flow velocity and flow rate of the velocity measurement line are calculated based on the texture angle corresponding to the optimal trajectory direction.

[0134] In this embodiment, a hydrological station is taken as an example to apply the above method, as follows:

[0135] Step 1: Select video data from a hydrological station. The video length is 20 seconds, the video frame rate is 30 frames / s, and the size of each frame is 1920*1080.

[0136] First, cut the video into frames, and then convert each frame into a grayscale image.

[0137] Step 2: See Figure 2 The coordinate information of the four marked points on the map is read as A(227,790), B(1674,850), C(1835,119), and D(1160,48); the coordinate of the starting point of the section line on the map is E(155,1059), and the coordinate of the end point on the map is F(1695,111); the actual coordinates between the four marked points are AB=9.49, BC=50.39, CD=20.43, DA=51.63, AC=53.63, and BD=56.81.

[0138] According to the principle of equally dividing the section line, a speed measuring point is set on the section line every 5 meters, the coordinates of the speed measuring point (x, y) are obtained, and the corresponding block area (xr; x+r, yr; y+r) is obtained. The size of the block area is 2r*2r.

[0139] Step 3: See Figure 3 As shown, one of the speed measurement points and the corresponding block area is selected, and the block area I(P) of the next frame image is aligned with the block area G(P) of the speed measurement point of the current frame image. The mapping from I(P) to G(P) is w(P;q):

[0140]

[0141] q is the transformation parameter (q1,q2,…,q n ) T ,q1,q2,…,qn is the transformation vector, the transformed image of w(P;q) is I(w(P;q)), and the displacement between I(w(P;q)) and G(P) is minimized by Δq:

[0142]

[0143] Minimizing Δq is nonlinear, and it is iterated to find the vector q:

[0144]

[0145] Given a threshold ε, stop when ||Δq||≤ε, find the vector q, and then find the image coordinates of the tracking point in the next frame, see Figure 4 Get the tracking point coordinates (x i ,y i ) T .

[0146] Step 4: See Figure 5 Use the least squares method to fit the ten coordinates of the trajectory points to a straight line:

[0147]

[0148] Take the minimum value and get the straight line equation y i =bx i +a with slope b.

[0149] Step 5: At the speed measurement point coordinate P = (x, y) T Take x+n / 2 and xn / 2 as the horizontal coordinates of the starting point and end point of the speed measurement line respectively and substitute them into the straight line equation obtained above:

[0150]

[0151] Get the starting coordinates (x+n / 2, y1) and the ending coordinates (xn / 2, y2) of the speed measurement line, where n is the required speed measurement line length, see Figure 6 Then draw the speed measurement line according to the starting point coordinates and the end point coordinates.

[0152] Step 6: See Figure 7 By plotting the changes in the brightness of the pixels on the velocity line over time on an image, the horizontal axis represents the length of the velocity line and the vertical axis represents the time axis. The resulting image is called a space-time image.

[0153] Step 7: The Gaussian directional stretch filter can easily construct a slender ridge structure similar to the trajectory line, which can better detect the trajectory line in a specific direction in the ST image. Therefore, the Gaussian directional stretch filter l(x,y) is used to enhance the linear structure of the spatiotemporal image:

[0154]

[0155] Among them, σ represents the filter factor that controls the filter size; γ represents the filter factor that controls the filter stretch length.

[0156] The filter is rotated to detect trajectories in different directions, so the rotation matrix R is defined:

[0157]

[0158] Convolve l(x,y) with the rotation matrix R. The direction θ corresponding to the pixel with the maximum response in the convolution result is * When the image is enhanced, the enhanced image is referred to Figure 8 , recorded as

[0159]

[0160] Step 8: Although the low-contrast trajectories are enhanced in the enhanced spatiotemporal image, the background noise and streak noise are also enhanced. The enhanced image can be regarded as an image composed of trajectories and noise. Therefore, it is necessary to suppress the noise and retain the trajectories. The specific processing process includes:

[0161] (1) Taking advantage of the fact that the enhanced image still obeys the Gaussian distribution, the probability density function Q(X) of the pixel point X in the image is defined as:

[0162] Q(X)=w n Q n (X)+(1-w)Q t (X)

[0163] Among them, Q n (X) is the probability density function of noise-related pixels; Q t (X) is the probability density function of trajectory-related pixels, w n is the weighting coefficient 0<w<1. n (X) is:

[0164]

[0165] Among them, u n represents the mean value of the Gaussian distribution function, σ n Represents the variance of the Gaussian distribution function.

[0166] By controlling the weight coefficient, high weights protect pixels related to the trajectory, while low weights weaken and filter out pixels related to noise. Therefore, the noise suppression function N(X) is:

[0167]

[0168] By expanding N(X) into a two-dimensional function and applying it to the spatiotemporal image to be processed, see Figure 9 Get the image after background noise suppression

[0169] (2) Then calculate the main direction of the image texture.

[0170] First, construct the directional filter h θ (x,y), and perform Hilbert transform to H θ (x,y):

[0171] H θ (x,y)=Hibert{h θ (x,y)}

[0172] image The directional texture energy is expressed as E θ (x,y) means:

[0173]

[0174] Take E θ The angle direction at the maximum value of (x,y) is the main direction of the texture θ max :

[0175] θ max =argmax{E θ (x,y)|θ∈(0,π]}

[0176] Therefore, the directional filter function N2(X) is:

[0177]

[0178] Among them, σ represents the scaling function; θ * Indicates the angle of the enhanced image; 0<g2(x,y)<1. Use the directional filter to max The direction of max direction is removed, so that the noise can be filtered out and the Figure 10 Get the final processed spatiotemporal image

[0179] Step 9: See Figure 11 Random Hough transform is used to detect the direction of the trajectory line for the processed spatiotemporal image, and ρ is used to represent the detected straight line:

[0180] ρ=x i cosθ i +y i sinθ i

[0181] Among them, (x i ,y i ) represents the coordinates of the trajectory line in the space-time image; θ i represents the angle between the trajectory line and the vertical vector line; ρ represents the length of the trajectory line.

[0182] After obtaining K fuzzy straight lines, calculate the average value of the inclination angles of all straight lines θ to get the approximate direction of the trajectory

[0183] Step 10: Set a threshold ε to determine the rotation range of the projection transformation

[0184] The enhanced denoised spatiotemporal image is rotated within the range The rotation was performed in 0.5° steps.

[0185] For each space-time image after rotation angle θ, the grayscale values ​​of each column of pixels in the vertical direction are accumulated to draw the grayscale histogram f θ , for the vertical projection f of n columns θn have:

[0186] f θn =∑ M c nm

[0187] Among them, c nm Represents the grayscale value of the mth pixel in the projection direction of the nth column. In this way, all grayscale histograms of the spatiotemporal image are obtained for every 0.5° rotation within the rotation range. Figure 12 As shown in Figure 2, when the grayscale histogram shows more peaks, the trajectory lines on the spatiotemporal image are more vertical.

[0188] Step 11: Calculate the Renyi entropy H of the grayscale histogram at each angle θα (V), as shown below:

[0189]

[0190] Where V represents a discrete random variable and α represents the order of Renyi entropy.

[0191] Calculate the variance D of the grayscale histogram at each angle θ , as shown below:

[0192]

[0193] The weighted sum of Renyi entropy and variance is calculated using weighted coefficients w1 and w2 to obtain the characteristic statistical value S for each angle. θ , as shown below:

[0194] S θ =w1*H θα (V)+w2*D θ (V)

[0195] See also Figure 13 As shown, the characteristic statistics S calculated from all angles θ Draw the characteristic statistical curve and find the angle corresponding to the peak position in the statistical characteristic curve, which is the optimal spatiotemporal image texture trajectory direction angle

[0196] Repeat steps 3 to 11 to calculate the optimal texture trajectory direction angle of the spatiotemporal image of the speed measurement line at all speed measurement points.

[0197] Step 12: The river surface flow feature moves a distance D along the velocity measurement line within time T, which corresponds to a movement of d pixels within the τ frame in the image coordinate system. The velocity vector magnitude V of the velocity measurement line is:

[0198]

[0199] Among them, S x Indicates the actual distance represented by each pixel (unit: m / pixel), fps indicates the camera frame rate (unit: pixel / s), Represents the tangent value of the slope of the space-time image.

[0200] Finally, the flow rate Q is calculated using the velocity-area method:

[0201]

[0202] Where η represents the surface velocity coefficient, V i represents the surface velocity of the interval, A i Represents the area of ​​the interval.

[0203] This method can be used to calculate the velocity vector V of the velocity line at each velocity measuring point, and then the river flow Q can be calculated using the velocity area method.

[0204] Example 2

[0205] Based on the same inventive concept as Example 1, this example introduces a video flow measurement device based on LK tracking and grayscale statistical feature method, including:

[0206] The frame image acquisition module is used to convert each frame image in the video stream into a grayscale image;

[0207] The block region displacement calculation module is used to calculate the average value of the inter-frame displacement of the block region between all adjacent frames in the video stream as the inter-frame displacement of the block region;

[0208] The straight line fitting module is used to fit the tracking point trajectory coordinates into a straight line to obtain the straight line equation and slope;

[0209] The image processing module is used to enhance the trajectory texture of the spatiotemporal image through the Gaussian pull-up filter, and to suppress the background noise and stripe direction noise using the noise suppression function and directional filter function respectively;

[0210] Line detection module, used to detect the angle of the line in the spatiotemporal image and obtain the approximate area of ​​rotation;

[0211] Grayscale statistics module, used to count the grayscale value of each angle to obtain its corresponding grayscale histogram;

[0212] The feature statistics calculation module is used to calculate the grayscale histogram feature value of each angle and statistically compile it into a feature curve to obtain the optimal texture angle;

[0213] and a flow calculation module for calculating the flow velocity at each measuring point according to the optimal texture angle.

[0214] For the specific functional implementation of the above functional modules, please refer to the relevant method content in Example 1.

[0215] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0216] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0217] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0218] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0219] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A video flow measurement method based on LK tracking and grayscale statistical feature method, characterized by: The following processes are included: Convert each frame image in the video stream into a grayscale image; Get the starting point and end point of the cross section line, and determine the speed measurement point and the corresponding block area based on the starting point and end point of the cross section line; Calculate the inter-frame displacement of each speed measurement point based on the LK tracking algorithm, and obtain the displacement coordinates of the speed measurement points in the first ten frames; According to the displacement coordinates of the first ten frames of each speed measurement point, the least squares straight line fitting algorithm is used to fit the displacement coordinates of the first ten frames of each speed measurement point to obtain the straight line equation and slope; Draw a speed measurement line at the corresponding speed measurement point according to the straight line equation and slope corresponding to each speed measurement point, and generate a corresponding space-time image diagram based on the speed measurement line; Gaussian directional stretching filter is used to preprocess the spatiotemporal image; Construct noise suppression function and directional filtering function to suppress background noise and filter horizontal stripe noise of preprocessed spatiotemporal images; Random Hough transform is used to detect straight lines in the space-time image after noise removal and obtain the average angle to determine the approximate range of the trajectory direction; The processed spatiotemporal image is rotated at different angles with a certain step size within the approximate range of the trajectory direction to obtain the grayscale histogram of each angle; The grayscale histogram of each angle is calculated using the feature statistics method to obtain the eigenvalue and draw the feature curve, and the peak is found to obtain the texture angle corresponding to the optimal trajectory direction; Calculate the flow velocity and flow rate of the velocity measurement line according to the texture angle corresponding to the optimal trajectory direction; The LK tracking algorithm is used to calculate the inter-frame displacement of each speed measurement point and obtain the displacement coordinates of the speed measurement points in the first ten frames, including: Align the block area I(P) of the next frame image with the block area G(P) of the speed measurement point of the current frame image. The mapping from I(P) to G(P) is w(P;q), where q is the transformation parameter (q1, q2, ..., q n ) T ,q1,q2,…,q n is the transformation vector, then w(P;q) is as shown in formula (1): The transformed image of w(P;q) is I(w(P;q)), and the displacement between I(w(P;q)) and G(P) is minimized by △q, as shown in formula (2): △q=argmin∑ x ‖I(w(P;q))-G(P)‖ 2 (2) Minimizing △q is nonlinear, and the vector q is iterated, as shown in formula (3): Given a threshold ε, stop when ‖△q‖≤ε, find the vector q, and then find the image coordinates of the tracking point in the next frame, and get the tracking point coordinates (x i ,y i ) T ; The least squares straight line fitting algorithm is used to fit the displacement coordinates of the first ten frames of each speed measurement point to obtain the straight line equation and slope, including: The ten coordinates of the trajectory points are fitted with a straight line using the least squares method, as shown in formula (4): When the minimum value is taken in formula (4), the equation and slope of the straight line are obtained; Drawing a speed measurement line at a corresponding speed measurement point according to a straight line equation and a slope corresponding to each speed measurement point includes: At the speed measurement point coordinate P = (x, y) T Substitute xn / 2 and x+n / 2 as the horizontal coordinates of the starting and ending points of the speed measurement line into the above straight line equation to obtain the starting and ending coordinates of the speed measurement line, where n is the required length of the speed measurement line. Then draw the speed measurement line based on the starting and ending coordinates. Generating a corresponding spatiotemporal image according to the speed measurement line includes: By plotting the changes in the brightness of the pixels on the velocity line over time on an image, the horizontal axis represents the length of the velocity line and the vertical axis represents the time axis. The resulting image is called a space-time image.

2. The video flow measurement method based on LK tracking and grayscale statistical feature method according to claim 1, characterized in that: The determining of the speed measurement point and the corresponding block area according to the starting point and the end point of the cross-section line includes: According to the starting point and end point of the cross-section line, the line connecting the starting point to the end point of the cross-section line is segmented, and the segmentation points on the cross-section line are used as speed measurement points; For any speed measurement point P is (x,y) T , the corresponding block area G(P) is (xr; x+r, yr; y+r), where r is the larger of the absolute value of half the distance in the x direction and half the distance in the y direction of each segment after segmentation; The segmentation is performed in an equal division principle, and the number of segments is determined by the number of cross-sectional data.

3. The video flow measurement method based on LK tracking and grayscale statistical feature method according to claim 1 is characterized in that: The preprocessing of the spatiotemporal image by using a Gaussian directional stretching filter includes: The linear structure of the spatiotemporal image is enhanced using the Gaussian directional stretching filter l(x,y), as shown in formula (5): Among them, σ represents the filter factor that controls the size of the filter; γ represents the filter factor that controls the stretch length of the filter; The filter is rotated to detect trajectories in different directions, so the rotation matrix R is defined as shown in formula (6): Convolve l(x,y) with the rotation matrix R. The convolution result shows that when the pixel has the maximum corresponding direction θ * When the image is enhanced, the enhanced image is recorded as As shown in formula (7):

4. The video flow measurement method based on LK tracking and grayscale statistical feature method according to claim 3 is characterized in that: The construction of the noise suppression function and the directional filtering function to suppress background noise and filter horizontal stripe noise on the preprocessed spatiotemporal image includes: 1) Taking advantage of the fact that the enhanced image still obeys the Gaussian distribution, the probability density function Q(X) of the pixel point X in the image is defined as shown in formula (8): Q(X)=w n Q n (X)+(1-w)Q t (X)(8) Among them, Q n (X) is the probability density function of noise-related pixels; Q t (X) is the probability density function of trajectory-related pixels, w n is the weighting coefficient 0<w<1; Q n (X) is shown in formula (9): Among them, u n represents the mean value of the Gaussian distribution function, σ n represents the variance of the Gaussian distribution function; By controlling the weight coefficient, high weights protect pixels related to the trajectory, while low weights weaken and filter out pixels related to noise; therefore, the noise suppression function N(X) is shown in formula (10): By expanding N(X) into a two-dimensional function and applying it to the enhanced image The noise-suppressed image is obtained 2) Calculate the main direction of the image texture; first construct the direction filter h θ (x,y), and perform Hilbert transform to H θ (x,y), as shown in formula (11): H θ (x,y)=Hibert{h θ (x,y)}(11) image Directional texture energy E θ (x,y), as shown in formula (12): Take E θ The angle direction at the maximum value of (x,y) is the main direction of the texture θ max , as shown in formula (13): i max =arg max{E θ (x,y)|θ∈(0,π]} (13) Therefore, the directional filter function N2(X) is as shown in formula (14): Among them, σ represents the scaling function; θ * Represents the angle of the enhanced image; using the directional filter to max The direction of max The direction is removed, so that the noise can be filtered out and the final processed spatiotemporal image is obtained.

5. The video flow measurement method based on LK tracking and grayscale statistical feature method according to claim 1 is characterized in that: The method of using random Hough transform to detect straight lines in the space-time image after noise removal and obtaining an average angle to determine the approximate range of the trajectory direction includes: Random Hough transform is used to detect the trajectory direction of the processed spatiotemporal image, and ρ is used to represent the detected straight line, as shown in formula (15): p=x i cosθ i +y i sinth i (15) Among them, (x i ,y i ) represents the coordinates of the trajectory line in the space-time image; θ i Represents the angle between the trajectory line and the vertical vector line; After obtaining K fuzzy straight lines, calculate the average value of the inclination angles of all straight lines Get a rough direction of the trajectory direction.

6. The video flow measurement method based on LK tracking and grayscale statistical feature method according to claim 5 is characterized in that: The processing of rotating the spatiotemporal image at different angles with a certain step length within the approximate range of the trajectory direction to obtain a grayscale histogram at each angle includes: Set a threshold ε to determine the rotation range of the projection transformation The enhanced denoised spatiotemporal image is rotated within the range The rotation was performed in 0.5° steps; For each space-time image after rotation angle θ, the grayscale values ​​of each column of pixels in the vertical direction are accumulated to draw the grayscale histogram f θ , for the vertical projection f of n columns θn , as shown in formula (16): f θn =∑ M c nm (16) Among them, c nm Represents the grayscale value of the mth pixel in the projection direction passing through the nth column, thereby obtaining all grayscale histograms of the spatiotemporal image rotated every 0.5° within the rotation range.

7. The video flow measurement method based on LK tracking and grayscale statistical feature method according to claim 6 is characterized in that: The method of using feature statistics to calculate the grayscale histogram of each angle to obtain the feature value and draw the feature curve includes: Calculate the Renyi entropy H of the grayscale histogram at each angle θα (V), as shown in formula (17): Where V represents a discrete random variable; α represents the order of Renyi entropy; Calculate the variance D of the grayscale histogram at each angle θ , as shown in formula (18): The weighted sum of Renyi entropy and variance is calculated using weighted coefficients w1 and w2 to obtain the characteristic statistical value S for each angle. θ , as shown in formula (19): S θ =w1*H θα (V)+w2*D θ (V) (19) The characteristic statistics S calculated from all angles θ Draw the characteristic statistical curve; the best texture angle, find the angle corresponding to the peak position in the statistical characteristic curve, which is the best spatiotemporal image texture trajectory direction angle The calculating the flow velocity and flow rate of the speed measurement line according to the texture angle corresponding to the optimal trajectory direction includes: The surface flow characteristics of the river move along the velocity measurement line for a distance D within the time T, which corresponds to a movement of d pixels within the τ frame in the image coordinate system. The velocity vector of the velocity measurement line is shown in formula (20): Among them S x Indicates the actual distance represented by each pixel (unit: m / pixel), fps indicates the camera frame rate (unit: pixel / s), Represents the tangent value of the slope of the space-time image; The flow rate is calculated using the velocity-area method, as shown in formula (21): Where η is the surface velocity coefficient, V i represents the surface velocity of the interval, A i Represents the area of ​​the interval.

8. A video flow measurement device based on LK tracking and grayscale statistical feature method designed according to any one of claims 1 to 7, characterized in that: include: The frame image acquisition module is used to convert each frame image in the video stream into a grayscale image; The block area displacement calculation module is used to calculate the inter-frame displacement of each speed measurement point and obtain the displacement coordinates of the speed measurement points in the first ten frames; The straight line fitting module is used to fit the tracking point trajectory coordinates into a straight line to obtain the straight line equation and slope; The image processing module is used to enhance the trajectory texture of the spatiotemporal image through the Gaussian pull-up filter, and to suppress the background noise and stripe direction noise using the noise suppression function and directional filter function respectively; Line detection module, used to detect the angle of the line in the spatiotemporal image and obtain the approximate area of ​​rotation; Grayscale statistics module, used to count the grayscale value of each angle to obtain its corresponding grayscale histogram; The feature statistics calculation module is used to calculate the grayscale histogram feature value of each angle and statistically compile it into a feature curve to obtain the optimal texture angle; And a flow calculation module is used to calculate the flow velocity and flow rate of each measuring point according to the optimal texture angle.

Images