A method for measuring river surface velocity based on tracking floating objects on the water surface

Abstract

The present invention discloses a method for measuring river surface velocity based on surface floating object tracking, comprising: performing distortion correction on an original video image sequence and setting a ROI region; fusing a pixel-level adaptive segmentation algorithm that introduces a foreground counting mechanism and a mean subtraction image method to perform multi-target segmentation on the distortion-corrected image; screening candidate targets and removing false targets; reconstructing the movement trajectory of floating objects using a target matching method that uses multi-feature fusion, and storing the image coordinate information of the center of mass of the floating object target on each trajectory; converting the image coordinates on each trajectory into world coordinates using a direct linear transformation, and calculating the velocity value based on the world coordinates of adjacent points on each trajectory; and gridding all the velocity values using an inverse distance weighted interpolation method to obtain a uniform two-dimensional velocity vector field. The present invention is suitable for measuring river velocity in scenarios with multiple floating objects, and can improve the accuracy of the flow measurement algorithm in such measurement scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent water conservancy video monitoring, and relates to a method for measuring river surface flow velocity, in particular to a river surface flow velocity measurement method based on water surface floating object target tracking. Background Art

[0002] In recent years, with the rapid development of video surveillance and computer vision technology, image-based flow measurement technology has received widespread attention and application in the field of hydrological monitoring due to its advantages of non-contact, low cost and high safety. A series of image-based flow measurement technologies have been derived, including large-scale particle image velocimetry (LSPIV), particle tracking velocimetry (PTV), space-time image velocimetry (STIV) and optical tracking velocimetry (OTV). Large-scale particle image velocimetry divides the area to be measured in the image into several regions, and performs correlation analysis on the regions in the previous and next frame images. The displacement of the two windows with the largest correlation coefficient corresponds to the average velocity of the region. It requires the presence of floating objects or a stable water surface pattern on the water surface to obtain a two-dimensional surface velocity field. Particle tracking velocimetry identifies and tracks the movement of tracer particles on the river surface and reconstructs the movement trajectory of the particles to obtain a two-dimensional velocity field. It requires the presence of floating objects or artificially spread tracer particles on the water surface. The measured flow field velocity has a high spatial resolution. Space-time image velocimetry utilizes the continuity of the movement of water tracers to obtain a two-dimensional velocity field. Using the velocity measurement line parallel to the downstream direction as the analysis area, the texture direction features related to the tracer movement are detected in the space-time image composed of image space and sequence time, and the one-dimensional time-averaged motion vector in the specified spatial direction is directly estimated. It only requires the presence of stable natural water surface patterns such as ripples and bubbles on the water surface, and has the advantages of high spatial resolution and strong real-time performance. Optical tracking velocity measurement technology detects the corner point features of the target on the water surface, combines the tracking algorithm to reconstruct the motion trajectory of the feature points, and then obtains a sparse velocity field. It requires the presence of floating objects or artificially spread tracer particles on the water surface to provide stable feature points, and can obtain a two-dimensional velocity field with higher spatial resolution.

[0003] In scenarios where river surface velocities exceed 0.5 m / s, natural water surface models, floating objects, and tracer particles can effectively represent water motion, and therefore the aforementioned imaging-based flow measurement techniques all achieve good results. However, in scenarios with lower flow rates and the influence of wind and waves on the river surface, natural water surface models are susceptible to environmental noise such as ripples caused by wind interference, rendering them incapable of accurately representing water motion. In contrast, using rigid floating objects to represent water motion aligns with the measurement principles of the buoy method used in hydrographic surveys and is therefore more suitable for flow velocity measurements under these conditions. However, large-scale particle image velocimetry and spatiotemporal image velocimetry cannot identify the effective velocity vector estimated from the motion of rigid floating targets in their measurement results, and the feature points tracked by optical tracking velocimetry cannot filter out the influence of water ripple fluctuations. The existing particle tracking velocimetry technology focuses on the study of issues such as particle density and particle occlusion when artificially spreading tracer particles. Its particle detection method is mainly aimed at artificial tracer particles with clear targets and good backgrounds, and it is difficult to solve the problem of detecting floating objects in natural scenes. Its particle tracking method is mainly aimed at particle matching problems under conditions of high particle density and particle occlusion. It is relatively complex and computationally intensive, and is not applicable to the assumption that the density of floating objects in natural scenes is generally sparse and there is usually no occlusion problem. Summary of the Invention

[0004] Purpose of the invention: To address the problem of river surface flow velocity measurement, a river surface flow velocity measurement method based on surface floating object tracking is provided. The surface flow velocity is estimated by using the motion trajectory of floating object targets in natural scenes. The method is suitable for measuring river surface flow velocity in scenes such as low flow velocity and wind and waves.

[0005] Technical solution: To achieve the above-mentioned purpose, the present invention provides a method for measuring river surface flow velocity based on surface floating object tracking, comprising the following steps:

[0006] S1: Perform distortion correction on the original video image sequence and set the ROI area;

[0007] S2: The pixel-level adaptive segmentation algorithm (PBAS) with foreground counting mechanism and the mean subtraction image method are integrated to perform multi-target segmentation on the distortion-corrected image.

[0008] S3: For the segmented targets, an adaptive pipeline filtering method is used to screen candidate targets using the motion characteristics of the targets between adjacent frames to remove false targets;

[0009] S4: Use the target matching method of multi-feature fusion to reconstruct the movement trajectory of the floating object, filter it according to the length and angle of the trajectory, and save the image coordinate information of the center of mass of the floating object on each trajectory;

[0010] S5: The image coordinates on each trajectory are converted into world coordinates using direct linear transformation (DLT), and the flow velocity value is calculated based on the world coordinates of the adjacent points on each trajectory;

[0011] S6: All velocity values are gridded using the inverse distance weighted interpolation method to obtain a uniform two-dimensional velocity vector field.

[0012] Furthermore, in step S1, the camera is calibrated with internal parameters in the laboratory according to the Zhang Zhengyou calibration method. After calibration, the pixel size s, the internal parameter matrix K and the distortion parameter matrix D are obtained as follows:

[0013]

[0014] D=[k1 k2 p1 p2] (18)

[0015] Among them, C x Represents the horizontal coordinate of the principal point, C y represents the vertical coordinate of the principal point, f x Indicates the equivalent focal length of the camera on the x-axis of the image plane, f y It represents the equivalent focal length of the camera on the y-axis of the image plane. The focal length f of the camera is calculated based on the pixel size s of the image sensor:

[0016] f=(f x +f y )·s / 2 (19)

[0017] k1 represents the first radial distortion coefficient, p1 represents the first tangential distortion coefficient, k2 represents the second radial distortion coefficient, and p2 represents the second tangential distortion coefficient. The image is corrected for distortion using the following formula:

[0018]

[0019] The camera coordinates (x, y) of the undistorted image are converted to image coordinates (u, v) using the following formula:

[0020]

[0021] The camera coordinates (x′, y′) of the distorted image are converted to image coordinates (u′, v′) in the same way:

[0022]

[0023] After completing the image distortion correction, set the ROI area for subsequent measurement.

[0024] Furthermore, in step S2, the pixel-level adaptive segmentation algorithm (PBAS) with a foreground counting mechanism and the mean subtraction image method are integrated to perform multi-target segmentation, and the process is as follows:

[0025] First, the PBAS algorithm is used to segment the floating object foreground and the river surface background of the video image sequence. The video image sequence is input and the foreground and background classification is determined by comparing the image background model with the current pixel point. The background model is defined by an array of N recently observed pixel values:

[0026] B(x i )={B1(x i ),...,B k (x i ),...,B N (x i )} (twenty three)

[0027] At the same time, the PBAS algorithm determines the classification of foreground and background by comparing the current pixel with the background model. i The pixel value I(x i ) and at least #min of the N background values, whose distance is less than the judgment threshold R(x i ), then determine x i As the background, the calculation process is:

[0028]

[0029] Among them, F = 0 and F = 1 respectively indicate that the pixel point is a background point and a foreground point, dist(I(x i ),Bk(x i )) represents the distance between the current point and the background model; for the pixel currently determined to be the background, a random and uniform selection of an index k∈1,...N, the corresponding background model B k (x i ) with probability P = 1 / T(x i ) is replaced by the current pixel value I(x i ); At the same time, with probability P = 1 / T(x i ) Randomly select adjacent pixels y i ∈N(x i ), the background model B at the adjacent pixel k (y i ) is replaced by its current pixel value I(y i )replace;

[0030] In addition to the background model B(x i ) saves the array of recently observed pixel values and creates an array D(x i )={D1(xi ),...,D N (x i )}, each execution of B k (x i ) is updated, the currently observed minimum distance d min (x i )=min k dist(I(x i ),B k (x i ))Write to this array, thereby creating the historical values of the minimum decision distance, the average of these historical values It is used to measure the degree of dynamic change of the background; thus, the decision threshold R(x i ):

[0031]

[0032] where R inc / dec and R scale is a fixed parameter used to i ) dynamic control;

[0033] For the update probability T(x i ), also with Take control:

[0034]

[0035] Where T inc , T dec is a fixed parameter; at the same time, the upper and lower limits of the update probability are set to T lower <T<T upper ;

[0036] At the same time, in order to suppress the water surface background from being frequently detected as the foreground due to the large noise interference in the area with severe water ripple fluctuations, a foreground counting mechanism COUNT is introduced. t (x i ):

[0037]

[0038] F(xi)=0,if COUNT t (x i )=COUNT max (28)

[0039] where COUNT maxThe maximum number of times a pixel is detected as foreground, set to twice the frame rate. The foreground counting mechanism ensures that in areas with violent water ripples, the water surface background is not detected as foreground for too long. In other normal areas, only floating objects passing by are considered foreground, and the foreground counting mechanism is not triggered.

[0040] Then, the subtraction mean image method is used to segment the floating object target and calculate the average image of the video image sequence:

[0041]

[0042] Where N is the total number of frames in the video image sequence, and S is a fixed parameter;

[0043] Subtract the average image from each frame to get the saliency map:

[0044] I s =I k -I m (30)

[0045] Use the maximum inter-class variance method (OTSU) to segment the target and background on the saliency map;

[0046] Finally, the segmentation detection results of the two methods are fused:

[0047] I mask =I pbas ∩I mean (31)

[0048] Among them I pbas The segmentation result of the PBAS algorithm with the foreground counting mechanism is introduced. mean is the segmentation result of the mean-subtracted image method, I mask The final output multi-target segmentation result.

[0049] Furthermore, in step S3, an adaptive pipeline filtering method is used to screen candidate targets using the motion characteristics of the target between adjacent frames to remove false targets. The process is as follows:

[0050] For the multi-target segmentation result finally output in step S2, the centroid (x i ,y i ) is regarded as the center of the pipeline, and a spatial pipeline is established that runs through the multi-frame image sequence. The pipeline length L represents the time span of the required continuous detection, and the pipeline diameter d represents the comparison range between two adjacent frames. According to the target size setting, when the number of times the target appears in the pipeline reaches the threshold requirement, the target in the corresponding first frame image is determined to be a real foreground target. Otherwise, it is regarded as a false detection result and is eliminated. When a new target that does not belong to the existing pipeline appears, a new pipeline is created for the target;

[0051] At the same time, for the traditional pipeline filtering, the pipeline created is a fixed rectangle with the pipeline as the center. When the target's motion in the multi-frame image sequence is not a standard linear motion, a larger pipeline diameter d is required to ensure that the target is in the pipeline. When there are many targets, it is difficult to ensure that there is only one moving target in a pipeline, which makes the pipeline difficult to distinguish. Therefore, the center of the pipeline is adaptive. When the target located in the pipeline in the kth frame is found through the pipeline created in the k-1th frame, the pipeline center is corrected by the center of the pipeline in the k-1th frame and the center of mass of the target in the kth frame:

[0052]

[0053]

[0054] in is the center coordinate of the k-th frame pipeline, is the center coordinate of the k-1th frame pipeline, x k ,y k is the position coordinate of the target in the kth frame, Δx and Δy are the correction values of the pipeline center.

[0055] Furthermore, in step S4, the target matching method of multi-feature fusion is used to reconstruct the movement trajectory of the floating object, filter according to the length and angle of the trajectory, and save the image coordinate information of the center of mass of the floating object on each trajectory. The process is as follows:

[0056] For the target segmentation result image sequence that has been processed by pipeline filtering in step S3, a fixed frame interval is selected to count the perimeter L, area S, and circularity e of the floating object target in the selected image. The perimeter L is obtained by counting the number of pixels of the target contour line in the segmentation result, the area S is obtained by counting the number of pixels in the target area, and the circularity e is calculated based on the perimeter and area characteristics of the target, with a maximum value of 1, as follows:

[0057]

[0058] The target to be matched in the previous frame image is regarded as the parent target, and the target in the next frame image is regarded as the child target. The above three features are integrated to calculate the relative error between the parent target and the child target to estimate the similarity between the two:

[0059]

[0060] Among them, λ1, λ2, λ3 represent the weights of the target perimeter, area and circularity respectively, and satisfy λ1+λ2+λ3=1. Considering that the influence of each feature of each target in the continuous image sequence and each image is different, each feature is given the same weight, that is, δ is the relative error between the target to be matched in the previous frame image and the target in the next frame image, and δ∈[0,1). The higher the target similarity, the closer δ is to 0. The relative error between the parent target and the child target should satisfy δ<δ min , δ min is the relative error threshold; and the sub-target should be located within the rectangular search window of the parent target, and the size of the search window is determined by the prior maximum flow rate;

[0061] In the subsequent frame image, all floating objects that meet the above conditions are found as candidate targets, and their distances to the parent target in the previous frame image are calculated. The target with the smallest distance is the sub-target that matches the parent target. All floating objects in the selected images are traversed, and the sub-targets that meet the matching relationship with the parent target in the previous frame image have their center of mass coordinates added to the same trajectory. Sub-targets that do not meet the matching relationship are considered as new targets and added to the newly created trajectory. The motion trajectory of each floating object target is a collection of center of mass coordinates, which is saved in the form of a two-dimensional array.

[0062] Filtering is performed based on the length and angle of the trajectory. Based on the prior information of the river's flow direction, each trajectory should have a certain directionality, that is, satisfying:

[0063]

[0064] where x start ,y start are the horizontal and vertical coordinates of the starting point of the trajectory, x end ,y end are the horizontal and vertical coordinates of the end point of the trajectory, and θ is the angle threshold; at the same time, shorter trajectories are eliminated:

[0065]

[0066] L is the preset shortest trajectory length;

[0067] At this point, the reconstruction of the floating object's trajectory is completed.

[0068] Furthermore, step S5 converts the image coordinates on each trajectory into world coordinates using direct linear transformation (DLT):

[0069]

[0070] where z c are arbitrary parameters, (u,v) are image coordinates, (x w ,y w ,z w ) is the world coordinate, the L matrix is the perspective projection matrix, and is solved by a set of control points with known space point coordinates and image point coordinates;

[0071] The actual physical distance is calculated from the world coordinates of two adjacent points on each trajectory:

[0072]

[0073] x1, y1 and x2, y2 are the world coordinates of two adjacent points on the trajectory. The time interval t between the two adjacent points is calculated by the video frame rate. According to the speed formula:

[0074] v=s / t (40)

[0075] Calculate the actual velocity of two adjacent points on each trajectory; calculate the direction of the actual velocity based on the image coordinate information of the two adjacent points:

[0076]

[0077] Save the actual flow velocity magnitude v, direction ang, and world coordinates (x1, y1); traverse all trajectories, calculate the actual flow velocity of all adjacent points on each trajectory, and save the calculation results.

[0078] Furthermore, in step S6, all velocity values are gridded using the inverse distance weighted interpolation method to obtain a uniform two-dimensional velocity vector field. The process is as follows:

[0079] The video image is divided into uniform rectangular speed measurement grids; the flow velocity value in each grid is interpolated to the center of the grid using the inverse distance weighted interpolation method:

[0080]

[0081] Where N is the number of velocity vectors in each grid, d i is the velocity vector The distance between the interpolation point j and w is the weight coefficient;

[0082] The above interpolation calculation is performed on each uniformly divided velocity measurement grid to obtain a gridded velocity field.

[0083] This paper assumes that the application scenario is a natural river with natural floating objects on its surface. Based on this premise, the camera intrinsic parameters are first calibrated in the laboratory and used to correct image distortion. Then, a pixel-level adaptive segmentation algorithm (PBAS) with a foreground counting mechanism and the mean subtraction image method are integrated for multi-target segmentation. Target screening is performed using a pipeline filtering method to remove false targets. The target segmentation results are then matched, and the floating object motion trajectory is reconstructed. The trajectory reconstruction results are then filtered. The image coordinates on each trajectory are then converted to world coordinates using a direct linear transformation (DLT) and the flow velocity value is calculated. Finally, gridding is performed using the inverse distance weighted interpolation method to obtain a uniform two-dimensional flow velocity vector field.

[0084] Beneficial effects: Compared with the existing technology, the present invention uses floating targets on the surface of natural rivers as tracers to characterize the movement of river water bodies, reconstructs the movement trajectory by tracking the movement of floating targets, and then obtains the surface flow velocity of the river without the need for artificial spreading of tracer particles. This solves the problem that the existing flow measurement algorithm cannot accurately identify the effective flow velocity vector estimated by the movement of rigid floating targets in its measurement results in scenarios where natural water surface patterns such as low flow rate and wind and waves cannot accurately characterize the movement of water bodies, thereby improving the accuracy of the flow measurement algorithm in such measurement scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 is a flow chart of the method of the present invention;

[0086] Figure 2 is the original image to be corrected for distortion;

[0087] Figure 3 is the distortion-corrected image;

[0088] Figure 4 It is a flowchart for multi-target segmentation of a single frame image;

[0089] Figure 5 It is a schematic diagram of the pipeline filtering method;

[0090] Figure 6 This is a specific example diagram of pipeline filtering;

[0091] Figure 7 It is a schematic diagram of the flow field grid;

[0092] Figure 8 This is a specific example diagram of the gridded flow field. DETAILED DESCRIPTION

[0093] The present invention will be further explained below with reference to the accompanying drawings. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0094] The present invention provides a method for measuring river surface velocity based on tracking of floating objects on the water surface. Figure 1 As shown, it includes the following steps:

[0095] S1: Yes Figure 2 The original video image sequence shown is subjected to distortion correction. The camera is calibrated with internal parameters in the laboratory according to the Zhang Zhengyou calibration method. After calibration, the pixel size s, the internal parameter matrix K and the distortion parameter matrix D are obtained as follows:

[0096]

[0097] D=[k1 k2 p1 p2] (44)

[0098] Among them, C x Represents the horizontal coordinate of the principal point, C y represents the vertical coordinate of the principal point, f x Indicates the equivalent focal length of the camera on the x-axis of the image plane, f y It represents the equivalent focal length of the camera on the y-axis of the image plane. The focal length f of the camera is calculated based on the pixel size s of the image sensor:

[0099] f=(f x +f y )·s / 2 (45)

[0100] k1 represents the first radial distortion coefficient, p1 represents the first tangential distortion coefficient, k2 represents the second radial distortion coefficient, and p2 represents the second tangential distortion coefficient. The image is corrected for distortion using the following formula:

[0101]

[0102] The camera coordinates (x, y) of the undistorted image are converted to image coordinates (u, v) using the following formula:

[0103]

[0104] The camera coordinates (x′, y′) of the distorted image are converted to image coordinates (u′, v′) in the same way:

[0105]

[0106] At this point, the image distortion correction is completed, and we get Figure 3 The undistorted image is shown, and the ROI area is set in the undistorted image for subsequent measurement.

[0107] In the present invention, the ROI area is manually selected and set. Since the shooting angle of the camera does not change, the ROI only needs to be set once. The purpose of setting the ROI is to determine the measurement area.

[0108] S2: Fusion of the pixel-level adaptive segmentation algorithm (PBAS) with foreground counting mechanism and the mean subtraction image method for multi-target segmentation. The process is as follows Figure 4 As shown:

[0109] First, the PBAS algorithm is used to segment the floating object foreground and the river surface background of the video image sequence. The video image sequence is input and the foreground and background classification is determined by comparing the image background model with the current pixel point. The background model is defined by an array of N recently observed pixel values:

[0110] B(x i )={B1(x i ),...,B k (x i ),...,B N (x i )} (49)

[0111] At the same time, the PBAS algorithm determines the classification of foreground and background by comparing the current pixel with the background model. i The pixel value I(x i ) and at least #min of the N background values, whose distance is less than the judgment threshold R(x i ), then determine x i As the background, the calculation process is:

[0112]

[0113] Among them, F = 0 and F = 1 respectively indicate that the pixel point is a background point and a foreground point, dist(I(xi), Bk(xi)) represents the distance between the current point and the background model; for the pixel point currently determined to be the background, a certain index k∈1,...N is randomly and uniformly selected, and the corresponding background model B k (x i ) with probability P = 1 / T(x i ) is replaced by the current pixel value I(x i ); At the same time, with probability P = 1 / T(x i ) Randomly select adjacent pixels y i ∈N(x i ), the background model B at the adjacent pixel k (y i ) is replaced by its current pixel value I(y i )replace;

[0114] In addition to the background model B(x i ) saves the array of recently observed pixel values and creates an array D(x i )={D1(x i ),...,D N (x i )}, each execution of B k (x i ) is updated, the currently observed minimum distance d min (x i )=min k dist(I(x i ),B k (x i))Write to this array, thereby creating the historical values of the minimum decision distance, the average of these historical values It is used to measure the degree of dynamic change of the background; thus, the decision threshold R(x i ):

[0115]

[0116] where R inc / dec and R scale is a fixed parameter used to i ) dynamic control;

[0117] For the update probability T(x i ), also with Take control:

[0118]

[0119] Where T inc , T dec is a fixed parameter; at the same time, the upper and lower limits of the update probability are set to T lower <T<T upper ;

[0120] At the same time, in order to suppress the water surface background from being frequently detected as the foreground due to the large noise interference in the area with severe water ripple fluctuations, a foreground counting mechanism COUNT is introduced. t (x i ):

[0121]

[0122] F(xi)=0,if COUNT t (x i )=COUNT max (54)

[0123] where COUNT max The maximum number of times a pixel is detected as foreground, set to twice the frame rate. The foreground counting mechanism ensures that in areas with violent water ripples, the water surface background is not detected as foreground for too long. In other normal areas, only floating objects passing by are considered foreground, and the foreground counting mechanism is not triggered.

[0124] Then, the subtraction mean image method is used to segment the floating object target and calculate the average image of the video image sequence:

[0125]

[0126] Where N is the total number of frames in the video image sequence, and S is a fixed parameter;

[0127] Subtract the average image from each frame to get the saliency map:

[0128] I s =I k -I m (56)

[0129] Use the maximum inter-class variance method (OTSU) to segment the target and background on the saliency map;

[0130] Finally, the segmentation detection results of the two methods are fused:

[0131] I mask =I pbas ∩I mean (57)

[0132] Among them I pbas The segmentation result of the PBAS algorithm with the foreground counting mechanism is introduced. mean is the segmentation result of the mean-subtracted image method, I mask The final output multi-target segmentation result.

[0133] S3: Use the adaptive pipeline filtering method to screen candidate targets using the motion characteristics of the target between adjacent frames and remove false targets. The process is as follows: Figure 5 As shown:

[0134] For the multi-target segmentation result finally output in step S2, the centroid (x i ,y i ) is regarded as the center of the pipeline, and a spatial pipeline is established that runs through the multi-frame image sequence. The pipeline length L represents the time span of the required continuous detection, and the pipeline diameter d represents the comparison range between two adjacent frames. According to the target size setting, when the number of times the target appears in the pipeline reaches the threshold requirement, the target in the corresponding first frame image is determined to be a real foreground target. Otherwise, it is regarded as a false detection result and is eliminated. When a new target that does not belong to the existing pipeline appears, a new pipeline is created for the target;

[0135] At the same time, for the traditional pipeline filtering, the pipeline created is a fixed rectangle with the pipeline as the center. When the target's motion in the multi-frame image sequence is not a standard linear motion, a larger pipeline diameter d is required to ensure that the target is in the pipeline. When there are many targets, it is difficult to ensure that there is only one moving target in a pipeline, which makes the pipeline difficult to distinguish. Therefore, the center of the pipeline is adaptive. When the target located in the pipeline in the kth frame is found through the pipeline created in the k-1th frame, the pipeline center is corrected by the center of the pipeline in the k-1th frame and the center of mass of the target in the kth frame:

[0136]

[0137]

[0138] in is the center coordinate of the k-th frame pipeline, is the center coordinate of the k-1th frame pipeline, x k ,y k is the position coordinate of the target in the kth frame, Δx and Δy are the correction values of the pipeline center; the processing result of pipeline filtering is as follows Figure 6 shown.

[0139] S4: Use the target matching method of multi-feature fusion to reconstruct the movement trajectory of the floating object, filter it according to the length and angle of the trajectory, and save the image coordinate information of the center of mass of the floating object on each trajectory. The process is as follows:

[0140] For the target segmentation result image sequence that has been processed by pipeline filtering in step S3, a fixed frame interval is selected to count the perimeter L, area S, and circularity e of the floating object target in the selected image. The perimeter L is obtained by counting the number of pixels of the target contour line in the segmentation result, and the area S is obtained by counting the number of pixels in the target area. The circularity e is calculated based on the perimeter and area characteristics of the target, and its maximum value is 1. It is calculated as follows:

[0141]

[0142] The target to be matched in the previous frame image is regarded as the parent target, and the target in the next frame image is regarded as the child target. The above three features are integrated to calculate the relative error between the parent target and the child target to estimate the similarity between the two:

[0143]

[0144] Among them, λ1, λ2, λ3 represent the weights of the target perimeter, area and circularity respectively, and satisfy λ1+λ2+λ3=1. Considering that the influence of each feature of each target in the continuous image sequence and each image is different, each feature is given the same weight, that is, δ is the relative error between the target to be matched in the previous frame image and the target in the next frame image, and δ∈[0,1). The higher the target similarity, the closer δ is to 0. The relative error between the parent target and the child target should satisfy δ<δ min , δ min is the relative error threshold; and the sub-target should be located within the rectangular search window of the parent target, and the size of the search window is determined by the prior maximum flow rate;

[0145] In the subsequent frame image, all floating objects that meet the above conditions are found as candidate targets, and their distances to the parent target in the previous frame image are calculated. The target with the smallest distance is the sub-target that matches the parent target. All floating objects in the selected images are traversed, and the sub-targets that meet the matching relationship with the parent target in the previous frame image have their center of mass coordinates added to the same trajectory. Sub-targets that do not meet the matching relationship are considered as new targets and added to the newly created trajectory. The motion trajectory of each floating object target is a collection of center of mass coordinates, which is saved in the form of a two-dimensional array.

[0146] Filtering is performed based on the length and angle of the trajectory. Based on the prior information of the river's flow direction, each trajectory should have a certain directionality, that is, satisfying:

[0147]

[0148] where x start ,y start are the horizontal and vertical coordinates of the starting point of the trajectory, x end ,y end are the horizontal and vertical coordinates of the end point of the trajectory, and θ is the angle threshold; at the same time, shorter trajectories are eliminated:

[0149]

[0150] L is the preset shortest trajectory length;

[0151] At this point, the reconstruction of the floating object's trajectory is completed.

[0152] S5: Use direct linear transformation (DLT) to convert the image coordinates on each trajectory into world coordinates:

[0153]

[0154] where z c are arbitrary parameters, (u,v) are image coordinates, (x w ,y w ,z w ) is the world coordinate, the L matrix is the perspective projection matrix, and is solved by a set of control points with known space point coordinates and image point coordinates;

[0155] The actual physical distance is calculated from the world coordinates of two adjacent points on each trajectory:

[0156]

[0157] x1, y1 and x2, y2 are the world coordinates of two adjacent points on the trajectory. The time interval t between the two adjacent points is calculated by the video frame rate. According to the speed formula:

[0158] v=s / t (66)

[0159] Calculate the actual velocity of two adjacent points on each trajectory; calculate the direction of the actual velocity based on the image coordinate information of the two adjacent points:

[0160]

[0161] Save the actual flow velocity magnitude v, direction ang, and world coordinates (x1, y1); traverse all trajectories, calculate the actual flow velocity of all adjacent points on each trajectory, and save the calculation results.

[0162] S6: All velocity values are gridded using the inverse distance weighted interpolation method to obtain a uniform two-dimensional velocity vector field. The process is as follows: Figure 7 As shown:

[0163] The video image is divided into uniform rectangular speed measurement grids; the flow velocity value in each grid is interpolated to the center of the grid using the inverse distance weighted interpolation method:

[0164]

[0165] Where N is the number of velocity vectors in each grid, d i is the velocity vector The distance between the interpolation point j and w is the weight coefficient;

[0166] The above interpolation calculation is performed on each uniformly divided velocity measurement grid. The grid with track gets the correct interpolated velocity as shown by the solid vector arrow, and the grid without track is marked with the dotted vector arrow as the missing mark. Finally, Figure 8 The gridded velocity field is shown.

[0167] The grid calculation results of the velocity field are shown in Table 1. The grid numbers start from the upper left corner of the image and are 1-16 from left to right and from top to bottom. The X-direction velocity represents the horizontal rightward velocity component, and the Y-direction velocity represents the vertical downward velocity component. The absence of velocity indicates that there is no trajectory in the grid.

[0168] Table 1

[0169]

Claims

1. A method for measuring river surface velocity based on tracking of floating objects on the water surface, characterized in that: The steps include: S1: Perform distortion correction on the original video image sequence and set the ROI area; S2: The pixel-level adaptive segmentation algorithm (PBAS) with foreground counting mechanism and the mean subtraction image method are integrated to perform multi-target segmentation on the distortion-corrected image. S3: For the segmented targets, an adaptive pipeline filtering method is used to screen candidate targets using the motion characteristics of the targets between adjacent frames to remove false targets; S4: Use the target matching method of multi-feature fusion to reconstruct the movement trajectory of floating objects, filter according to the length and angle of the trajectory, and save the image coordinate information of the center of mass of the floating object on each trajectory; S5: The image coordinates on each trajectory are converted into world coordinates using direct linear transformation (DLT), and the flow velocity value is calculated based on the world coordinates of the adjacent points on each trajectory; S6: All velocity values are gridded using the inverse distance weighted interpolation method to obtain a uniform two-dimensional velocity vector field; In step S4, the target matching method of multi-feature fusion is used to reconstruct the movement trajectory of the floating object, filter it according to the length and angle of the trajectory, and save the image coordinate information of the center of mass of the floating object on each trajectory. The process is as follows: For the target segmentation result image sequence that has been processed by pipeline filtering in step S3, a fixed frame interval is selected to count the perimeter L, area S, and circularity e of the floating object target in the selected image. The perimeter L is obtained by counting the number of pixels of the target contour line in the segmentation result, the area S is obtained by counting the number of pixels in the target area, and the circularity e is calculated based on the perimeter and area characteristics of the target, with a maximum value of 1. The calculation formula is as follows: The target to be matched in the previous frame image is regarded as the parent target, and the target in the next frame image is regarded as the child target. The above three features are integrated to calculate the relative error between the parent target and the child target to estimate the similarity between the two: Among them, λ1, λ2, λ3 represent the weights of the target perimeter, area and circularity respectively, and satisfy λ1+λ2+λ3=1. Considering that the influence of each feature of each target in the continuous image sequence and each image is different, each feature is given the same weight, that is, δ is the relative error between the target to be matched in the previous frame image and the target in the next frame image, and δ∈[0,1); the relative error between the parent target and the child target satisfies δ<δ min , δ min is the relative error threshold; and the sub-target should be located within the rectangular search window of the parent target, and the size of the search window is determined by the prior maximum flow rate; In the subsequent frame image, all floating objects that meet the above conditions are found as candidate targets, and their distances to the parent target in the previous frame image are calculated. The target with the smallest distance is the child target that matches the parent target. Traverse all floating objects in the selected image. The center of mass coordinates of the sub-objects that match the parent object in the previous frame image are added to the same trajectory. Sub-objects that do not meet the matching relationship are considered as new objects and added to the newly created trajectory. The motion trajectory of each floating object is a collection of center of mass coordinates, which is saved in the form of a two-dimensional array. Filtering is performed based on the length and angle of the trajectory. Based on the prior flow direction information of the river, each trajectory satisfies: where x start , ystart is the horizontal and vertical coordinates of the starting point of the trajectory, x end , yend is the horizontal and vertical coordinates of the end point of the trajectory, θ is the angle threshold; at the same time, shorter trajectories are eliminated: L is the preset shortest trajectory length; At this point, the reconstruction of the floating object's trajectory is completed.

2. A method for measuring river surface velocity based on surface floating object tracking according to claim 1, characterized in that: In step S1, the camera is calibrated with internal parameters in the laboratory according to the Zhang Zhengyou calibration method to obtain the corresponding internal parameter matrix and distortion parameter matrix. The original video image sequence is subjected to distortion correction based on the internal parameter matrix and the distortion parameter matrix, and the ROI area is set.

3. The method for measuring river surface velocity based on surface floating object tracking according to claim 1, characterized in that: In step S2, the pixel-level adaptive segmentation algorithm (PBAS) with foreground counting mechanism and the mean subtraction image method are integrated to perform multi-target segmentation, and the process is as follows: First, the PBAS algorithm is used to segment the video image sequence into floating objects and the river surface background. The PBAS algorithm starts by creating a background model from the first frame of the image. During the segmentation process, the floating object judgment threshold and the background model update rate are dynamically adjusted. The foreground and background classification are determined by comparing the current pixel with the background model. The background model is defined by an array of N recently observed pixel values: B(x i )={B1(x i ),...,B k (x i ),...,B N (x i )} If pixel x i The pixel value I(x i ) and at least #min of the N background values, whose distance is less than the judgment threshold R(x i ), then determine x i As the background, the calculation process is: Among them, F = 0 and F = 1 respectively indicate that the pixel point is a background point and a foreground point, and dist(I(xi), Bk(xi)) represents the distance between the current point and the background model. At the same time, in order to suppress the water surface background from being frequently detected as the foreground due to the large noise interference in the area with severe water ripple fluctuations, a foreground counting mechanism COUNT is introduced. t (x i ): where COUNT max The maximum number of times a pixel is detected as foreground, set to twice the frame rate. The foreground counting mechanism ensures that in areas with violent water ripples, the water surface background is not detected as foreground for too long. In other normal areas, only floating objects passing by are considered foreground, and the foreground counting mechanism is not triggered. Then, the subtraction mean image method is used to segment the floating object target and calculate the average image of the video image sequence: Where N is the total number of frames in the video image sequence, and S is a fixed parameter; Subtract the average image from each frame to get the saliency map: I s =I k -I m (5) Use the maximum inter-class variance method (OTSU) to segment the target and background on the saliency map; Finally, the segmentation detection results of the two methods are fused: I mask =I pbas ∩I mean (6) Among them I pbas The segmentation result of the PBAS algorithm with the foreground counting mechanism is introduced. mean is the segmentation result of the mean-subtracted image method, I mask The final output multi-target segmentation result.

4. The method for measuring river surface velocity based on surface floating object tracking according to claim 1, characterized in that: In step S3, an adaptive pipeline filtering method is used to screen candidate targets using the motion characteristics of the target between adjacent frames to remove false targets. The process is as follows: For the multi-target segmentation result finally output in step S2, the centroid (x i ,y i ) is regarded as the center of the pipeline, and a spatial pipeline is established that runs through the multi-frame image sequence. The pipeline length L represents the time span of the required continuous detection, and the pipeline diameter d represents the comparison range between two adjacent frames. According to the target size setting, when the number of times the target appears in the pipeline reaches the threshold requirement, the target in the corresponding first frame image is determined to be a real foreground target. Otherwise, it is regarded as a false detection result and is eliminated. When a new target that does not belong to the existing pipeline appears, a new pipeline is created for the target; At the same time, for the traditional pipeline filtering, which creates a fixed rectangle with the pipeline as the center, when the target's motion in the multi-frame image sequence is not a standard linear motion, a pipeline diameter d is required to ensure that the target is within the pipeline. The center of the pipeline is adapted. When the target located in the pipeline in the kth frame is found through the pipeline created in the k-1th frame, the pipeline center is corrected using the center of the pipeline in the k-1th frame and the center of mass of the target in the kth frame: in is the center coordinate of the k-th frame pipeline, is the center coordinate of the k-1th frame pipeline, x k ,y k is the position coordinate of the target in the kth frame, Δx and Δy are the correction values of the pipeline center.

5. The method for measuring river surface velocity based on surface floating object tracking according to claim 1, characterized in that: In step S5, the image coordinates on each track are converted into world coordinates using direct linear transformation (DLT), and the actual physical distance is calculated from the world coordinates of two adjacent points on each track: x1, y1 and x2, y2 are the world coordinates of two adjacent points on the trajectory. The time interval t between the two adjacent points is calculated by the video frame rate. According to the speed formula: v = s / t (14) calculates the actual velocity of two adjacent points on each trajectory; based on the image coordinate information of the two adjacent points, the direction of the actual velocity is calculated: Save the actual flow velocity magnitude v, direction ang, and world coordinates (x1, y1); traverse all trajectories, calculate the actual flow velocity of all adjacent points on each trajectory, and save the calculation results.

6. The method for measuring river surface velocity based on surface floating object tracking according to claim 1, characterized in that: The step S6 performs gridding processing on all velocity values using the inverse distance weighted interpolation method to obtain a uniform two-dimensional velocity vector field. The process is as follows: The video image is divided into uniform rectangular speed measurement grids; the flow velocity value in each grid is interpolated to the center of the grid using the inverse distance weighted interpolation method: Where N is the number of velocity vectors in each grid, d i is the velocity vector The distance between the interpolation point j and w is the weight coefficient; The above interpolation calculation is performed on each uniformly divided velocity measurement grid to obtain a gridded velocity field.

7. The method for measuring river surface velocity based on surface floating object tracking according to claim 2, characterized in that: The specific process of step S1 is: The camera is calibrated in the laboratory according to Zhang Zhengyou's calibration method. After calibration, the pixel size s, the intrinsic parameter matrix K and the distortion parameter matrix D are obtained as follows: D=[k1 k2 p1 p2] (18)where C x Represents the horizontal coordinate of the principal point, C y represents the ordinate of the principal point, fx represents the equivalent focal length of the camera on the x-axis of the image plane, fy represents the equivalent focal length of the camera on the y-axis of the image plane, and the focal length f of the camera is calculated based on the pixel size s of the image sensor: f=(f x +f y )·s / 2 (19) k1 represents the first radial distortion coefficient, p1 represents the first tangential distortion coefficient, k2 represents the second radial distortion coefficient, and p2 represents the second tangential distortion coefficient. The image is corrected for distortion using the following formula: The camera coordinates (x, y) of the undistorted image are converted to image coordinates (u, v) using the following formula: The camera coordinates (x′, y′) of the distorted image are converted to image coordinates (u′, v′) in the same way: After completing the image distortion correction, set the ROI area for subsequent measurement.

8. The method for measuring river surface velocity based on surface floating object tracking according to claim 3, characterized in that: The specific method for dynamically adjusting the floating object judgment threshold and the background model update rate during the foreground and background segmentation process in step S2 is: For the pixel currently determined to be the background, randomly and uniformly select an index k∈1,...N, and the corresponding background model B k (x i ) with probability P = 1 / T(x i ) is replaced by the current pixel value I(x i ); At the same time, with probability P = 1 / T(x i ) Randomly select adjacent pixels y i ∈N(x i ), the background model B at the adjacent pixel k (y i ) is replaced by its current pixel value I(y i )replace; In addition to the background model B(x i ) saves the array of recently observed pixel values and creates an array D(x i )={D1(x i ),...,D N (x i )}, each execution of B k (x i ) is updated, the currently observed minimum distance d min (x i )=min k dist(I(x i ),B k (x i ))Write to this array, thereby creating the historical values of the minimum decision distance, the average of these historical values It is used to measure the degree of dynamic change of the background; thus, the decision threshold R(x i ): where R inc / dec and R scale is a fixed parameter used to i ) dynamic control; For the update probability T(x i ), also with Take control: Where T inc , T dec is a fixed parameter; at the same time, the upper and lower limits of the update probability are set to T lower <T<T upper .

Images