Pixel calibration based optical flow speed measurement method and system

Abstract

The present application provides a kind of pixel calibration-based optical flow velocity measurement method and system, by introducing external calibration strategy, the algorithm prediction output can be synchronized with the dynamic change of water flow in time without increasing too much delay, the application of algorithm in actual project is optimized.Further, the present application also introduces the distance parameter from camera to lake surface to correct the output prediction value in combination with pixel displacement in external calibration, so that the result is more accurate, thereby solving the uncertainty output problem of calibration deep learning flow measurement and the problem of large flow velocity error caused by dynamic change of flow velocity.

Description

Technical Field

[0001] This invention relates to the field of hydrological flow measurement technology, and more specifically, to a method and system for optical flow velocity measurement based on pixel calibration. Background Technology

[0002] In the prior art, patent document CN113781528A discloses a method for measuring river surface velocity based on optical flow calculation. This method first preprocesses the acquired dynamic video of river flow to generate a river flow dataset; then, it combines particle image velocimetry with a convolutional neural network model, training and testing the convolutional neural network model on the image dataset to improve the estimation effect and robustness of complex flows; finally, it inputs real-time river flow images into the trained convolutional neural network model to calculate the river surface velocity and outputs a visualization.

[0003] However, when the water flow changes dynamically, the existing deep learning flow measurement algorithm has certain synchronization errors due to the uncertainty of the output, which leads to the inability to provide timely warnings when the water flow increases dramatically. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the purpose of this invention is to provide an optical flow velocity measurement method and system based on pixel calibration.

[0005] A pixel-calibrated optical flow velocimetry method according to the present invention includes:

[0006] Step S1: Select the Region of Interest (ROI) from the water flow video captured by camera W, and denote it as the b-th ROI region W of the a-th camera. ab ;

[0007] Step S2: Collect multiple multi-angle water flow videos in real-world conditions, calculate the distance parameter dis from each camera to the water surface, and then test to obtain the external calibration pixel distance parameter Pid for each water flow video. o Pid o The mean is used as the benchmark value δ;

[0008] Step S3: Perform off-pixel calibration based on the reference value δ to obtain W. ab External calibration pixel distance parameter Pid o Pix out-of-pixel calibration parameters o ;

[0009] Step S4: Perform detection and extract the video ROI region W. ab The i-th frame and the (i+1)-th frame are denoted as F. i F i+1 ; F i and F i+1Convert to HSV image, multiply the luminance channel of the HSV image by the luminance parameter br to enhance the luminance, and then convert it to the original image color space;

[0010] Step S5: Convert the F color space of the original image to F i and F i+1 As input to the Flownet optical flow network, the output is a two-channel optical flow graph, which is then converted into a three-channel RGB image with motion information.

[0011] Step S6: Use the RGB format three-channel image with motion information as input to the trained detection and regression model, and output the video W. ab F i+1 The initial predicted flow rate value V of the frame pre ;

[0012] Step S7: Based on the out-of-pixel calibration parameter Pix o Initial predicted flow velocity value V pre Dynamic flow velocity calibration is performed to obtain the predicted flow velocity value V. cal2 , as a predictive flow rate.

[0013] Preferably, before flow rate detection, corner point detection is performed on each ROI region of each camera, and then the detected corner points are tracked to obtain the difference in pixel displacement between the corner points in the previous and next frames. After traversing the previous and next frames multiple times, a list of pixel displacement differences is obtained. After filtering out outliers from the list, calibration is performed to obtain the calibration pixel displacement parameters for each ROI region of each camera. Then, the calibration parameters for out-of-pixel calibration are obtained by combining the distance parameters from the camera to the ROI region.

[0014] When outputting the flow velocity after the first two frames of input, corner detection is performed on the first frame to obtain the corner points for the next detection used for sparse optical flow tracking. Starting from the second detection, the corner pixel displacement difference between the current frame and the previous frame is obtained, and the obtained corner pixel displacement difference is added to the displacement difference list. After traversing multiple frames, the nearest neighbor density of the data points is used to filter out the outliers in the displacement difference list to obtain the final average displacement difference. The difference is then compared with the pixel external calibration distance parameter to determine the change in the difference and to calibrate the flow velocity value.

[0015] Preferably, step S3 includes:

[0016] Step S3.1: Define the video ROI region W ab Extract the i-th frame, denoted as F. i , will F i Convert to HSV image, multiply the luminance channel of HSV by the luminance parameter br to enhance luminance, and then convert to the original image color space (F). i ;

[0017] Step S3.2: Enhance the brightness of the F i Convert to grayscale image F i ', Select a movable window with a side length of m in F i The process involves moving the pointer up to obtain the partial derivative matrix M. After finding the smallest eigenvalue of M, it's determined whether it exceeds a threshold. If it does, the detected corner point in the current sliding window is obtained; otherwise, no detected corner point in the current sliding window is obtained. This process continues until the entire grayscale image is traversed, resulting in the set C of all invertible corner points in the image. i ={C0,C1,C2,...,C s}, where the subscript indicates the sequence number;

[0018] Step S3.3: Traverse video W ab F in frame i+1 i ' +1 Converted to grayscale image F through brightness enhancement operation i ' +1 Then combine with the previous frame F i 'Set of detected irreversible corner points C' i Under the assumptions of the LK optical flow algorithm, the estimated optical flow value is obtained by constructing a Gaussian pyramid and solving it layer by layer, thus obtaining the set of irreversible corner points C. i In F i ' +1 The position on the frame is C i+1 :

[0019] C i+1 ST = LK(F i ',F i ' +1 C i (Equation 1)

[0020] Step S3.4: Filter out F based on the corner state variables ST output by the LK optical flow algorithm. i ' +1 The corresponding tracked F i The set of corner points C i This yields the set of corner points from the previous frame after filtering. and the set of corner points in the current frame

[0021]

[0022] Step S3.5: Calculation and The pixel distances between the nodes are sorted from largest to smallest. A distance threshold is set, and the nodes are filtered again to obtain the top k pixel distances. The average of these top k pixel distances is then calculated to obtain the average pixel displacement of the current ROI. The F-axis is then used to calculate the average pixel displacement. i ' +1 The average pixel displacement of the frame is added to the pixel displacement list Z;

[0023] Iterate through steps S3.1 to S3.5 until n times to obtain the pixel displacement list Z = {Z0, Z1, ..., Zn}. n Outliers in Z are filtered out using the 2-sigma criterion, and the average is used to obtain the external calibration pixel distance parameter Pid for the current ROI. o :

[0024]

[0025]

[0026] Among them, Z mean Z represents the mean of Z, and Mean represents the mean function. std The standard deviation of Z is represented by std, which represents the standard deviation function; thr is an algebraic expression.

[0027] Step S3.6: Put Pid o Divide the distance parameter dis from the camera to the ROI by the reference value δ to obtain the calibration difference, and then multiply it by the scaling factor β to obtain the out-of-pixel calibration parameter Pix. o :

[0028] Pix o =(Pid) o / dis-δ)×β (Equation 5)

[0029] Wherein, the baseline value δ is the Pid of multiple water flow videos during the actual test. o Mean.

[0030] Preferably, step S7 includes:

[0031] Step S7.1: Obtain the initial predicted flow velocity value V of the ROI through the detection model. pre Next, out-of-pixel calibration is performed, using a nearest neighbor density algorithm to filter out outliers. This algorithm determines whether a pixel is an outlier by comparing the density of each value in the pixel displacement list Z with the density of its neighboring points. The lower the density of the pixel displacement value, the more likely it is to be considered an outlier.

[0032] res = LOF k (Z) (Equation 5)

[0033] Among them, LOF k is the nearest neighbor density function, k represents the k-th neighbor of the input point; res is the output result of the nearest neighbor density function, which has the same length as the input pixel displacement list Z, and contains 1 and -1, with a normal value of 1 and an abnormal value of -1;

[0034] Step S7.2: Based on the output result res of the nearest neighbor density function, outliers are filtered out from the pixel displacement list Z, and the average value is taken to obtain the pixel position value of the detected part.

[0035] Pid d =mean(Z[res]) (Equation 6)

[0036] Where Z[res] represents the value at position 1 in the saved pixel displacement list, and removes the value at position -1; Pid d This represents the pixel displacement value output by the detection section;

[0037] Step S7.3: Measure the Pid of multiple water flow videos during the actual test. o The baseline flow velocity V corresponding to the mean o The pixel calibration parameters Pix for the detected portion are obtained. d Then obtain the flow rate value V from the first calibration. cal1 V pre To detect the external calibration parameters of the output pixels (Pix) o The predicted flow rate value output by the detection section at that time;

[0038] V cal1 =mean(V pre V o +Pix o (Equation 7)

[0039] Step S7.4: Detect the Pid output from the detection section. d Pid corresponding to the ROI pixel external calibration output o After comparison, then V cal1 The second calibration yields the final flow rate value V output by the detection algorithm. cal2 To adapt to dynamic changes in actual flow velocity:

[0040]

[0041] According to the present invention, a pixel-calibrated optical flow velocimetry system includes:

[0042] Module M1: Selects the Region of Interest (ROI) from the water flow video captured by camera W, denoted as the b-th ROI region W of the a-th camera. ab ;

[0043] Module M2: Collects multiple multi-angle water flow videos in real-world conditions, calculates the distance parameter dis from each camera to the water surface, and then tests to obtain the external calibration pixel distance parameter Pid for each water flow video. o Pid o The mean is used as the benchmark value δ;

[0044] Module M3: Performs off-pixel calibration based on the reference value δ to obtain W. ab External calibration pixel distance parameter Pid o Pix out-of-pixel calibration parameters o ;

[0045] Module M4: Performs detection and extracts the video ROI region W. ab The i-th frame and the (i+1)-th frame are denoted as F. i F i+1 ; F i and F i+1 Convert to HSV image, multiply the luminance channel of the HSV image by the luminance parameter br to enhance the luminance, and then convert it to the original image color space;

[0046] Module M5: Converts the original image's color space to F... i and F i+1 As input to the Flownet optical flow network, the output is a two-channel optical flow graph, which is then converted into a three-channel RGB image with motion information.

[0047] Module M6: Takes a three-channel RGB image with motion information as input to a trained detection and regression model, and outputs a video W. ab F i+1 The initial predicted flow rate value V of the frame pre ;

[0048] Module M7: Based on the Pix out-of-pixel calibration parameter o Initial predicted flow velocity value V pre Dynamic flow velocity calibration is performed to obtain the predicted flow velocity value V. cal2 , as a predictive flow rate.

[0049] Preferably, before flow rate detection, corner point detection is performed on each ROI region of each camera, and then the detected corner points are tracked to obtain the difference in pixel displacement between the corner points in the previous and next frames. After traversing the previous and next frames multiple times, a list of pixel displacement differences is obtained. After filtering out outliers from the list, calibration is performed to obtain the calibration pixel displacement parameters for each ROI region of each camera. Then, the calibration parameters for out-of-pixel calibration are obtained by combining the distance parameters from the camera to the ROI region.

[0050] When outputting the flow velocity after the first two frames of input, corner detection is performed on the first frame to obtain the corner points for the next detection used for sparse optical flow tracking. Starting from the second detection, the corner pixel displacement difference between the current frame and the previous frame is obtained, and the obtained corner pixel displacement difference is added to the displacement difference list. After traversing multiple frames, the nearest neighbor density of the data points is used to filter out the outliers in the displacement difference list to obtain the final average displacement difference. The difference is then compared with the pixel external calibration distance parameter to determine the change in the difference and to calibrate the flow velocity value.

[0051] Preferably, the module M3 includes:

[0052] Module M3.1: Transforms the video ROI region W ab Extract the i-th frame, denoted as F. i , will F i Convert to HSV image, multiply the luminance channel of HSV by the luminance parameter br to enhance luminance, and then convert to the original image color space (F). i ;

[0053] Module M3.2: Enhances the brightness of the F i Convert to grayscale image F i ', Select a movable window with a side length of m in F i The process involves moving the pointer up to obtain the partial derivative matrix M. After finding the smallest eigenvalue of M, it's determined whether it exceeds a threshold. If it does, the detected corner point in the current sliding window is obtained; otherwise, no detected corner point in the current sliding window is obtained. This process continues until the entire grayscale image is traversed, resulting in the set C of all invertible corner points in the image. i ={C0,C1,C2,...,C s}, where the subscript indicates the sequence number;

[0054] Module M3.3, Traversing Video W ab F in frame i+1 i ' +1 Converted to grayscale image F through brightness enhancement operation i ' +1 Then combine with the previous frame F i 'Set of detected irreversible corner points C' i Under the assumptions of the LK optical flow algorithm, the estimated optical flow value is obtained by constructing a Gaussian pyramid and solving it layer by layer, thus obtaining the set of irreversible corner points C. i In F i ' +1 The position on the frame is C i+1 :

[0055] C i+1 ST = LK(F i ',F i '+1 C i (Equation 1)

[0056] Module M3.4: Filters out F based on the corner state variable ST output by the LK optical flow algorithm. i ' +1 The corresponding tracked F i The set of corner points C i This yields the set of corner points from the previous frame after filtering. and the set of corner points in the current frame

[0057]

[0058] Module M3.5: Calculation and The pixel distances between the nodes are sorted from largest to smallest. A distance threshold is set, and the nodes are filtered again to obtain the top k pixel distances. The average of these top k pixel distances is then calculated to obtain the average pixel displacement of the current ROI. The F-axis is then used to calculate the average pixel displacement. i ' +1 The average pixel displacement of the frame is added to the pixel displacement list Z;

[0059] Traverse the loop from module M3.1 to module M3.5 up to n times to obtain the pixel displacement list Z = {Z0, Z1, ..., Zn}. n Outliers in Z are filtered out using the 2-sigma criterion, and the average is used to obtain the external calibration pixel distance parameter Pid for the current ROI. o :

[0060]

[0061]

[0062] Among them, Z mean Z represents the mean of Z, and Mean represents the mean function. std The standard deviation of Z is represented by std, which represents the standard deviation function; thr is an algebraic expression.

[0063] Module M3.6: This will include Pid... o Divide the distance parameter dis from the camera to the ROI by the reference value δ to obtain the calibration difference, and then multiply it by the scaling factor β to obtain the out-of-pixel calibration parameter Pix. o :

[0064] Pix o =(Pid) o / dis-δ)×β (Equation 5)

[0065] Wherein, the baseline value δ represents the Pid of multiple water flow videos during the actual test.o Mean.

[0066] Preferably, the module M7 includes:

[0067] Module M7.1: Obtains the initial predicted flow velocity value V of the ROI through the detection model. pre Next, out-of-pixel calibration is performed, using a nearest neighbor density algorithm to filter out outliers. This algorithm determines whether a pixel is an outlier by comparing the density of each value in the pixel displacement list Z with the density of its neighboring points. The lower the density of the pixel displacement value, the more likely it is to be considered an outlier.

[0068] res = LOF k (Z) (Equation 5)

[0069] Among them, LOF k is the nearest neighbor density function, k represents the k-th neighbor of the input point; res is the output result of the nearest neighbor density function, which has the same length as the input pixel displacement list Z, and contains 1 and -1, with a normal value of 1 and an abnormal value of -1;

[0070] Module M7.2: Based on the nearest neighbor density function output result res, outliers are filtered from the pixel displacement list Z, and the average value is taken to obtain the pixel position value of the detected part.

[0071] Pid d =mean(Z[res]) (Equation 6)

[0072] Where Z[res] represents the value at position 1 in the saved pixel displacement list, and removes the value at position -1; Pid d This represents the pixel displacement value output by the detection section;

[0073] Module M7.3: Measured the Pid of multiple water flow videos during the actual test. o The baseline flow velocity V corresponding to the mean o The pixel calibration parameters Pix for the detected portion are obtained. d Then obtain the flow rate value V from the first calibration. cal1 V pre To detect the external calibration parameters of some output pixels (Pix) o The predicted flow rate value output by the detection section at that time;

[0074] V cal1 =mean(V pre V o +Pix o (Equation 7)

[0075] Module M7.4: This module will output the Pid from the detection section.d Pid corresponding to the ROI pixel external calibration output o After comparison, then V cal1 The second calibration yields the final flow rate value V output by the detection algorithm. cal2 To adapt to dynamic changes in actual flow velocity:

[0076]

[0077] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the steps of the pixel-calibrated optical flow velocimetry method are implemented.

[0078] An electronic device according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the computer program, when executed by the processor, implements the steps of the pixel-calibrated optical flow velocity measurement method.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] 1. The out-of-pixel calibration algorithm used in this invention first performs corner detection on each ROI region of each camera before flow velocity detection. Then, a sparse optical flow tracking algorithm is used to track the detected corners to obtain the difference in pixel displacement between the corners of two consecutive frames. After traversing the previous and next frames multiple times, a list of pixel displacement differences is obtained. The list is then filtered for outliers using the 2-sigma criterion before calibration to obtain the calibration pixel displacement parameters for each ROI region of each camera. These parameters are then combined with the distance parameters from the camera to the ROI region to obtain the calibration parameters for out-of-pixel calibration. The calibration parameters for out-of-pixel calibration can effectively reflect the corner displacement of the current ROI, avoiding errors caused by image quality and water surface occlusion in some frames.

[0081] 2. The dynamic flow velocity calibration algorithm used in this invention is suitable for situations where the actual flow velocity changes suddenly, but the algorithm's output flow velocity cannot be synchronized in time. When detecting and outputting the flow velocity in the first two frames before and after the initial input, corner point detection is performed on the first frame simultaneously to obtain the corner points used for sparse optical flow tracking in the next detection. Starting from the second detection, the corner pixel displacement difference between the current frame and the previous frame can be obtained, and this difference is added to the displacement difference list. After iterating through multiple frames, the nearest neighbor density of the data points is used to filter out outliers in the displacement difference list to obtain the final average displacement difference. This average displacement difference is then compared with the distance parameter from the camera to the ROI region and the out-of-pixel calibration parameter output by the out-of-pixel calibration algorithm to determine the change in the difference, thereby calibrating the flow velocity value output by the detection algorithm. Attached Figure Description

[0082] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0083] Figure 1 This is a schematic diagram illustrating the overall principle of the present invention.

[0084] Figure 2 This is a schematic diagram of the external calibration process for a single ROI pixel.

[0085] Figure 3 This is a schematic diagram of the dynamic flow rate calibration process for a single ROI. Detailed Implementation

[0086] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0087] This invention introduces an external calibration strategy to enable the algorithm's predicted output to synchronize with the dynamic changes in water flow in a timely manner without adding excessive latency, thus optimizing the algorithm's application in practical projects. Furthermore, this invention also incorporates the distance parameter from the camera to the lake surface into the pixel displacement during external calibration to correct the output prediction value, making the results more accurate. This solves the problems of uncertain output in calibrating deep learning flow measurement and large flow velocity errors caused by dynamic changes in flow velocity.

[0088] like Figure 1 As shown, a pixel-calibrated optical flow velocity measurement method and system according to the present invention includes:

[0089] Step S1: Select the Region of Interest (ROI) from the water flow video captured by camera W, denoted as W. ab Where 'a' represents the camera index, 'b' represents the ROI index, and W... ab This represents the b-th ROI region of the a-th camera.

[0090] Step S2: Collect multiple multi-angle water flow videos in real-world conditions, calculate the distance parameter dis from each camera to the water surface, and then test to obtain the external calibration pixel distance parameter Pid for each water flow video. o Pid o The mean is used as the benchmark value δ.

[0091] Step S3: Perform out-of-pixel calibration to obtain W. ab External calibration pixel distance parameter Pid o and out-of-pixel calibration parameters Pixo Specifically, step S3 includes:

[0092] Step S3.1: Determine the video ROI region W ab Extract the i-th frame, denoted as F. i , will F i Convert to HSV image, multiply the luminance channel of HSV by the luminance parameter br to enhance luminance, and then convert to the original image color space.

[0093] Step S3.2: Enhance the brightness of F i Convert to grayscale image F i ', Select a movable window with a side length of m in F i The partial derivative matrix M is obtained by moving the slider. The smallest eigenvalue of M is then calculated, and its value is checked against a minimum threshold to determine the detected corner point within the current sliding window. This process continues until the entire grayscale image is traversed, resulting in the set C of all invertible corner points in the image. i ={C0,C1,C2,...,C s}

[0094] Step S3.3: Traverse video W ab F in frame i+1 i ' +1 Repeat the brightness enhancement operation in step 3.1 and then convert it to grayscale image F. i ' +1 Then combine with the previous frame F i 'Set of detected irreversible corner points C' i Under the premise of satisfying the assumptions of the LK optical flow estimation algorithm, the estimated optical flow value is obtained by constructing a Gaussian pyramid and solving it layer by layer, and then the set of irreversible corner points C is obtained. i In F i ' +1 The position on the frame is C i+1 As shown in equation (1).

[0095] Step S3.4: Then, based on the LK algorithm, output the corner state variables ST to filter out F. i ' +1 The corresponding tracked F i The set of corner points C i This yields the set of corner points from the previous frame after filtering. and the set of corner points in the current frame As shown in equation (2).

[0096] C i+1 ST = LK(F i ',F i ' +1 C i (1)

[0097]

[0098] Step S3.5, Calculation and The pixel distances between the nodes are sorted from largest to smallest. A distance threshold is set, and the nodes are filtered again to obtain the top k pixel distances. The average of the top k pixel distances is then calculated to obtain the average pixel displacement of the current ROI. The F-value is then used to calculate the average pixel displacement of the current ROI. i ' +1 The average pixel displacement of the frame is added to the pixel displacement list Z.

[0099] Iterate through steps 3.1 to 3.5 until n times, where n is between 20 and 30. This will give you the pixel displacement list Z = {Z0, Z1, ..., Zn}. n Outliers in Z are filtered out using the 2-sigma criterion, and the average is used to obtain the external calibration pixel distance parameter Pid for the current ROI. o As shown in equations (3) and (4).

[0100]

[0101]

[0102] Step S3.6, Pid o Divide the distance parameter dis from the camera to the ROI by the reference value δ to obtain the calibration difference, and then multiply it by the scaling factor β to obtain the out-of-pixel calibration parameter Pix. o As shown in equation (5).

[0103] Pix o =(Pid) o / dis-δ)×β (5)

[0104] Wherein, the baseline value δ represents the Pid of multiple water flow videos during the actual test. o The mean is typically 1.56. The dynamic adjustment range for β is 0.1-0.5.

[0105] Step S4: Proceed to the detection algorithm section to extract the video ROI region W. ab The i-th frame and the (i+1)-th frame are denoted as F. i and F i+1 F i and F i+1 Convert to HSV image, multiply the luminance channel of HSV by the luminance parameter br to enhance luminance, and then convert to the original image color space.

[0106] Step S5: F i and F i+1As input to the optical flow network Flownet, the weight file of Flownet is a pre-trained model on the kitti dataset. The output is a two-channel optical flow map, which is then converted into a three-channel RGB image with motion information.

[0107] Step S6: Use the RGB format three-channel image with motion information as input to the detection and regression model, which has already been trained using the RGB image with motion information and its corresponding labels. Load the corresponding weight file, and output the video W. ab F i+1 The initial predicted flow rate value V of the frame pre .

[0108] Step S7: Proceed to the dynamic flow rate calibration algorithm section, such as... Figure 3 As shown, the predicted flow velocity value V is obtained. cal2 This is used as a detection algorithm to predict flow velocity. Step S7 includes:

[0109] Step S7.1: After obtaining the predicted flow rate value of the ROI through the detection model, the images of the previous and next frames input into the detection model are used to traverse steps 3.1 to 3.5 of the pixel out-of-pixel calibration algorithm. At this time, n is generally set to 5-10, and the nearest neighbor density algorithm is used to filter out outliers. The nearest neighbor density algorithm determines whether a point is an outlier by comparing the density of each value in the pixel displacement list Z with the density of its neighboring points. If the density of the pixel displacement value is lower, the pixel displacement is determined to be an outlier, as shown in Equation (6).

[0110] res = LOF k (Z) (6)

[0111] Among them LOF k This is the nearest neighbor density algorithm, where k represents the k-th neighbor of the input point, typically n / / 2. res is the algorithm output, with the same length as the input pixel displacement list Z, containing 1 and -1, where a normal value is 1 and an abnormal value is -1.

[0112] Step S7.2: After filtering out the outliers from the pixel displacement list Z according to res, take the average to obtain the pixel position value of the detected part, as shown in Equation (7).

[0113] Pid d =mean(Z[res]) (7)

[0114] Where Z[res] represents the value at position 1 in the saved pixel displacement list, and removes the value at position -1. Pid d This represents the pixel displacement value output by the detection section.

[0115] Step S7.3: Measure the Pid of multiple water flow videos during the actual test process. o The baseline flow velocity V corresponding to a mean of 1.56 o =0.31m / s. Step 3.6 of the off-pixel calibration algorithm is performed to obtain the pixel calibration parameters Pix for the detected portion. o Then, the flow rate value V of the first calibration is obtained through equation (7). cal1 V pre To detect the output pixel calibration parameters Pix o The predicted flow rate value output by the detection section at that time.

[0116] V cal1 =mean(V pre V o +Pix o (7)

[0117] Step S7.4: Detect the Pid output by the detection section. d Pid corresponding to the ROI pixel external calibration output o After comparison, then V cal1 The second calibration yields the final flow rate value V output by the detection algorithm. cal2 To adapt to the dynamic changes in actual flow rate, the specific process is shown in equation (8).

[0118]

[0119] The present invention also provides a pixel-calibrated optical flow velocity measurement system. The pixel-calibrated optical flow velocity measurement system can be implemented by executing the process steps of the pixel-calibrated optical flow velocity measurement system method. Those skilled in the art can understand the pixel-calibrated optical flow velocity measurement method as a preferred embodiment of the pixel-calibrated optical flow velocity measurement system.

[0120] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0121] Specifically, according to the present invention, a pixel-calibrated optical flow velocimetry system includes:

[0122] Module M1: Selects the Region of Interest (ROI) from the water flow video captured by camera W, denoted as the b-th ROI region W of the a-th camera. ab ;

[0123] Module M2: Collects multiple multi-angle water flow videos in real-world conditions, calculates the distance parameter dis from each camera to the water surface, and then tests to obtain the external calibration pixel distance parameter Pid for each water flow video. o Pid o The mean is used as the benchmark value δ;

[0124] Module M3: Performs off-pixel calibration based on the reference value δ to obtain W. ab External calibration pixel distance parameter Pid o Pix out-of-pixel calibration parameters o ;

[0125] Module M4: Performs detection and extracts the video ROI region W. ab The i-th frame and the (i+1)-th frame are denoted as F. i F i+1 ; F i and F i+1 Convert to HSV image, multiply the luminance channel of the HSV image by the luminance parameter br to enhance the luminance, and then convert it to the original image color space;

[0126] Module M5: Converts the original image's color space to F... i and F i+1 As input to the Flownet optical flow network, the output is a two-channel optical flow graph, which is then converted into a three-channel RGB image with motion information.

[0127] Module M6: Takes a three-channel RGB image with motion information as input to a trained detection and regression model, and outputs a video W. ab F i+1 The initial predicted flow rate value V of the frame pre ;

[0128] Module M7: Based on the Pix off-pixel calibration parameter o Initial predicted flow velocity value V pre Dynamic flow velocity calibration is performed to obtain the predicted flow velocity value V. cal2 , as a predictive flow rate.

[0129] Preferably, before flow rate detection, corner point detection is performed on each ROI region of each camera, and then the detected corner points are tracked to obtain the difference in pixel displacement between the corner points in the previous and next frames. After traversing the previous and next frames multiple times, a list of pixel displacement differences is obtained. After filtering out outliers from the list, calibration is performed to obtain the calibration pixel displacement parameters for each ROI region of each camera. Then, the calibration parameters for out-of-pixel calibration are obtained by combining the distance parameters from the camera to the ROI region.

[0130] When outputting the flow velocity after the first two frames of input, corner detection is performed on the first frame to obtain the corner points for the next detection used for sparse optical flow tracking. Starting from the second detection, the corner pixel displacement difference between the current frame and the previous frame is obtained, and the obtained corner pixel displacement difference is added to the displacement difference list. After traversing multiple frames, the nearest neighbor density of the data points is used to filter out the outliers in the displacement difference list to obtain the final average displacement difference. The difference is then compared with the pixel external calibration distance parameter to determine the change in the difference and to calibrate the flow velocity value.

[0131] The module M3 includes:

[0132] Module M3.1: Transforms the video ROI region W ab Extract the i-th frame, denoted as F. i , will F i Convert to HSV image, multiply the luminance channel of HSV by the luminance parameter br to enhance luminance, and then convert to the original image color space (F). i ;

[0133] Module M3.2: Enhances the brightness of the F i Convert to grayscale image F i ', Select a movable window with a side length of m in F i The process involves moving the pointer up to obtain the partial derivative matrix M. After finding the smallest eigenvalue of M, it's determined whether it exceeds a threshold. If it does, the detected corner point in the current sliding window is obtained; otherwise, no detected corner point in the current sliding window is obtained. This process continues until the entire grayscale image is traversed, resulting in the set C of all invertible corner points in the image. i ={C0,C1,C2,...,C s}, where the subscript indicates the sequence number;

[0134] Module M3.3, Traversing Video W ab F in frame i+1 i ' +1 Converted to grayscale image F through brightness enhancement operation i ' +1 Then combine with the previous frame F i 'Set of detected irreversible corner points C' iUnder the assumptions of the LK optical flow algorithm, the estimated optical flow value is obtained by constructing a Gaussian pyramid and solving it layer by layer, thus obtaining the set of irreversible corner points C. i In F i ' +1 The position on the frame is C i+1 :

[0135] C i+1 ST = LK(F i ',F i ' +1 C i (Equation 1)

[0136] Module M3.4: Filters out F based on the corner state variable ST output by the LK optical flow algorithm. i ' +1 The corresponding tracked F i The set of corner points C i This yields the set of corner points from the previous frame after filtering. and the set of corner points in the current frame

[0137]

[0138] Module M3.5: Calculation and The pixel distances between the nodes are sorted from largest to smallest. A distance threshold is set, and the nodes are filtered again to obtain the top k pixel distances. The average of these top k pixel distances is then calculated to obtain the average pixel displacement of the current ROI. The F-axis is then used to calculate the average pixel displacement. i ' +1 The average pixel displacement of the frame is added to the pixel displacement list Z;

[0139] Traverse the loop from module M3.1 to module M3.5 up to n times to obtain the pixel displacement list Z = {Z0, Z1, ..., Zn}. n Outliers in Z are filtered out using the 2-sigma criterion, and the average is used to obtain the external calibration pixel distance parameter Pid for the current ROI. o :

[0140]

[0141]

[0142] Among them, Z mean Z represents the mean of Z, and Mean represents the mean function. std The standard deviation of Z is represented by std, which represents the standard deviation function; thr is an algebraic expression.

[0143] Module M3.6: This will include Pid... oDivide the distance parameter dis from the camera to the ROI by the reference value δ to obtain the calibration difference, and then multiply it by the scaling factor β to obtain the out-of-pixel calibration parameter Pix. o :

[0144] Pix o =(Pid) o / dis-δ)×β (Equation 5)

[0145] Wherein, the baseline value δ represents the Pid of multiple water flow videos during the actual test. o Mean.

[0146] The module M7 includes:

[0147] Module M7.1: Obtains the initial predicted flow velocity value V of the ROI through the detection model. pre Next, out-of-pixel calibration is performed, using a nearest neighbor density algorithm to filter out outliers. This algorithm determines whether a pixel is an outlier by comparing the density of each value in the pixel displacement list Z with the density of its neighboring points. The lower the density of the pixel displacement value, the more likely it is to be considered an outlier.

[0148] res = LOF k (Z) (Equation 5)

[0149] Among them, LOF k is the nearest neighbor density function, k represents the k-th neighbor of the input point; res is the output result of the nearest neighbor density function, which has the same length as the input pixel displacement list Z, and contains 1 and -1, with a normal value of 1 and an abnormal value of -1;

[0150] Module M7.2: Based on the nearest neighbor density function output result res, outliers are filtered from the pixel displacement list Z, and the average value is taken to obtain the pixel position value of the detected part.

[0151] Pid d =mean(Z[res]) (Equation 6)

[0152] Where Z[res] represents the value at position 1 in the saved pixel displacement list, and removes the value at position -1; Pid d This represents the pixel displacement value output by the detection section;

[0153] Module M7.3: Measured the Pid of multiple water flow videos during the actual test. o The baseline flow velocity V corresponding to the mean o The pixel calibration parameters Pix for the detected portion are obtained. d Then obtain the flow rate value V from the first calibration. cal1 V preTo detect the external calibration parameters of the output pixels (Pix) o The predicted flow rate value output by the detection section at that time;

[0154] V cal1 =mean(V pre V o +Pix o (Equation 7)

[0155] Module M7.4: This module will output the Pid from the detection section. d Pid corresponding to the ROI pixel external calibration output o After comparison, then V cal1 The second calibration yields the final flow rate value V output by the detection algorithm. cal2 To adapt to dynamic changes in actual flow velocity:

[0156]

[0157] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A pixel-calibrated optical flow velocimetry method, characterized in that, include: Step S1: Place the camera The captured water flow video is used to select a Region of Interest (ROI), denoted as the [number missing]. The first camera ROI regions ; Step S2: Collect multiple multi-angle water flow videos in real time and calculate the distance parameters from each camera to the water surface. Then, the external calibration pixel distance parameter of each water flow video was obtained through testing. , and thus The mean is used as a benchmark. ; Step S3: Based on the baseline value Perform out-of-pixel calibration to obtain External calibration pixel distance parameters Pixel external calibration parameters ; Step S4: Perform detection and extract the video ROI region. No. Frame, First Frames, respectively denoted as , ;Will and Convert to HSV image, multiply the luminance channel of the HSV image by the luminance parameter br to enhance the luminance, and then convert it to the original image color space; Step S5: Convert the original image's color space... and As input to the Flownet optical flow network, the output is a two-channel optical flow graph, which is then converted into a three-channel RGB image with motion information. Step S6: Use the RGB format three-channel image containing motion information as input to the trained detection and regression model, and output as video. of Initial predicted flow rate value of the frame ; Step S7: Based on off-pixel calibration parameters Initial predicted flow velocity value Dynamic flow velocity calibration is performed to obtain the predicted flow velocity value. , as a predictive flow velocity; Step S3 includes: Step S3.1: Determine the video ROI region Extract the first Frame, denoted as ,Will Convert to HSV image, multiply the luminance channel of HSV by the luminance parameter br to enhance luminance, and then convert back to the original image color space. ; Step S3.2: Enhance the brightness Convert to grayscale Select a moving window with a side length of m. The partial derivative matrix M is obtained by moving the pointer upwards. After finding the smallest eigenvalue of M, it is determined whether it is greater than a threshold. If it is greater than the threshold, the detected corner point in the current sliding window is obtained; otherwise, the detected corner point in the current sliding window is not obtained. This process continues until the entire grayscale image is traversed, resulting in the set of all invertible corner points in the image. ={ , , ,..., }, where the subscript indicates the sequence number; Step S3.3: Traverse the video No. frame Convert to grayscale image through brightness enhancement operation Then combine with the previous frame Set of detected irreversible corner points Under the assumptions of the LK optical flow algorithm, the estimated optical flow value is obtained by constructing a Gaussian pyramid and solving it layer by layer, thus obtaining the set of irreversible corner points. exist Position on the frame : (Equation 1) Step S3.4: Output corner state variables according to the LK optical flow algorithm. Filter out The corresponding tracking Corner set This yields the set of corner points from the previous frame after filtering. and the set of corner points in the current frame : (Equation 2) Step S3.5: Calculation and The pixel distances between the nodes are sorted from largest to smallest. A distance threshold is set, and the nodes are filtered again to obtain the top k pixel distances. The average of these top k pixel distances is then calculated to obtain the average pixel displacement of the current ROI. The average pixel displacement of the frame is added to the pixel displacement list. middle; Iterate through steps S3.1 to S3.5 until... Next, obtain the pixel displacement list. ,right Outliers were filtered out using the 2-sigma criterion, and the average value was used to obtain the external calibration pixel distance parameter of the current ROI. : (Equation 3) (Equation 4) in, Z represents the mean, and Mean represents the mean function. The standard deviation of Z is represented by std, which represents the standard deviation function; thr is an algebraic expression. Step S3.6: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Divide by the distance parameter from the camera to the ROI Compared with the benchmark value Subtract the values ​​to get the calibration difference, then multiply by the proportionality factor. Obtain out-of-pixel calibration parameters : (Equation 5) Among them, the benchmark value For multiple water flow videos during actual testing Mean.

2. The optical flow velocimetry method based on pixel calibration according to claim 1, characterized in that, Before flow rate detection, corner point detection is performed on each ROI region of each camera. Then, the detected corner points are tracked to obtain the difference in pixel displacement between the corner points in the previous and next frames. After traversing the previous and next frames multiple times, a list of pixel displacement differences is obtained. After filtering out outliers from the list, calibration is performed to obtain the calibration pixel displacement parameters for each ROI region of each camera. Then, the calibration parameters for out-of-pixel calibration are obtained by combining the distance parameters from the camera to the ROI region. When the flow velocity is detected and output during the first input of two frames, corner point detection is performed on the first frame to obtain the corner points for the next detection for sparse optical flow tracking. Starting from the second detection, the corner point pixel displacement difference between the current frame and the previous frame is obtained, and the obtained corner point pixel displacement difference is added to the displacement difference list. After iterating through multiple frames, the nearest neighbor density of the data points is used to filter out outliers in the displacement difference list to obtain the final average displacement difference. Then, the difference between the two is compared with the pixel-outside calibration distance parameter to determine the change in the difference between the two, and the flow velocity value is calibrated.

3. The optical flow velocimetry method based on pixel calibration according to claim 1, characterized in that, Step S7 includes: Step S7.1: Obtain the initial predicted flow rate value of the ROI through the detection model. Next, out-of-pixel calibration is performed, and the nearest neighbor density algorithm is used to filter out outliers; the nearest neighbor density algorithm compares the pixel displacement list. The density of each pixel displacement value and its neighboring points is used to determine whether the pixel displacement is an outlier. If the density of the pixel displacement value is lower, the pixel displacement is considered an outlier. (Equation 6) in, The nearest neighbor density function, The first one representing the input point Neighborhood; The output of the nearest neighbor density function, along with the input pixel displacement list. The lengths are the same, and they contain 1 and -1. The normal value is 1, and the abnormal value is -1. Step S7.2: Output the result based on the nearest neighbor density function. From pixel displacement list After filtering out outliers, the average value is taken to obtain the pixel position value of the detected part: (Equation 7) in, Indicates the corresponding pixel displacement in the saved pixel displacement list For a value at position 1, remove the corresponding... The value at position -1; This represents the pixel displacement value output by the detection section; Step S7.3: Measure multiple water flow videos during the actual test process. The baseline flow rate corresponding to the mean The calibration parameters of the detected pixels were obtained. Then obtain the flow rate value from the first calibration. ,in To detect the external calibration parameters of some output pixels The predicted flow rate value output by the detection section at that time; (Equation 8) Step S7.4: Output from the detection section With the corresponding ROI pixel external calibration output After comparison, then... The second calibration yields the final flow rate value output by the detection algorithm. To adapt to dynamic changes in actual flow velocity: (Equation 9).

4. A pixel-calibrated optical flow velocimetry system, characterized in that, include: Module M1: Includes the camera The captured water flow video is used to select a Region of Interest (ROI), denoted as the [number missing]. The first camera ROI regions ; Module M2: Collects multiple multi-angle water flow videos in real-world conditions and calculates the distance parameters from each camera to the water surface. Then, the external calibration pixel distance parameter of each water flow video was obtained through testing. , and thus The mean is used as a benchmark. ; Module M3: Based on the baseline value Perform out-of-pixel calibration to obtain External calibration pixel distance parameters Pixel external calibration parameters ; Module M4: Performs detection and extracts the ROI region from the video. No. Frame, First Frames, respectively denoted as , ;Will and Convert to HSV image, multiply the luminance channel of the HSV image by the luminance parameter br to enhance the luminance, and then convert it to the original image color space; Module M5: Converts the original image's color space... and As input to the Flownet optical flow network, the output is a two-channel optical flow graph, which is then converted into a three-channel RGB image with motion information. Module M6: Takes a three-channel RGB image with motion information as input to a trained detection and regression model, and outputs a video. of Initial predicted flow rate value of the frame ; Module M7: Based on out-of-pixel calibration parameters Initial predicted flow velocity value Dynamic flow velocity calibration is performed to obtain the predicted flow velocity value. , as a predictive flow velocity; The module M3 includes: Module M3.1: Identifies the video ROI region Extract the first Frame, denoted as ,Will Convert to HSV image, multiply the luminance channel of HSV by the luminance parameter br to enhance luminance, and then convert back to the original image color space. ; Module M3.2: Enhances the brightness Convert to grayscale Select a moving window with a side length of m. The partial derivative matrix M is obtained by moving the pointer upwards. After finding the smallest eigenvalue of M, it is determined whether it is greater than a threshold. If it is greater than the threshold, the detected corner point in the current sliding window is obtained; otherwise, the detected corner point in the current sliding window is not obtained. This process continues until the entire grayscale image is traversed, resulting in the set of all invertible corner points in the image. ={ , , ,..., }, where the subscript indicates the sequence number; Module M3.3, Traversing Videos No. frame Convert to grayscale image through brightness enhancement operation Then combine with the previous frame Set of detected irreversible corner points Under the assumptions of the LK optical flow algorithm, the estimated optical flow value is obtained by constructing a Gaussian pyramid and solving it layer by layer, thus obtaining the set of irreversible corner points. exist Position on the frame : (Equation 1) Module M3.4: Outputs corner state variables based on the LK optical flow algorithm. Filter out The corresponding tracking Corner set This yields the set of corner points from the previous frame after filtering. and the set of corner points in the current frame : (Equation 2) Module M3.5: Calculation and The pixel distances between the nodes are sorted from largest to smallest. A distance threshold is set, and the nodes are filtered again to obtain the top k pixel distances. The average of these top k pixel distances is then calculated to obtain the average pixel displacement of the current ROI. The average pixel displacement of the frame is added to the pixel displacement list. middle; Iterate through modules M3.1 to M3.5 until... Next, obtain the pixel displacement list. ,right Outliers were filtered out using the 2-sigma criterion, and the average value was used to obtain the external calibration pixel distance parameter of the current ROI. : (Equation 3) (Equation 4) in, Z represents the mean, and Mean represents the mean function. The standard deviation of Z is represented by std, which represents the standard deviation function; thr is an algebraic expression. Module M3.6: will Divide by the distance parameter from the camera to the ROI Compared with the benchmark value Subtract the values ​​to get the calibration difference, then multiply by the proportionality factor. Obtain out-of-pixel calibration parameters : (Equation 5) Among them, the benchmark value For multiple water flow videos during actual testing Mean.

5. The pixel-calibrated optical flow velocimetry system according to claim 4, characterized in that, Before flow rate detection, corner point detection is performed on each ROI region of each camera. Then, the detected corner points are tracked to obtain the difference in pixel displacement between the corner points in the previous and next frames. After traversing the previous and next frames multiple times, a list of pixel displacement differences is obtained. After filtering out outliers from the list, calibration is performed to obtain the calibration pixel displacement parameters for each ROI region of each camera. Then, the calibration parameters for out-of-pixel calibration are obtained by combining the distance parameters from the camera to the ROI region. When the flow velocity is detected and output during the first input of two frames, corner point detection is performed on the first frame to obtain the corner points for the next detection for sparse optical flow tracking. Starting from the second detection, the corner point pixel displacement difference between the current frame and the previous frame is obtained, and the obtained corner point pixel displacement difference is added to the displacement difference list. After iterating through multiple frames, the nearest neighbor density of the data points is used to filter out outliers in the displacement difference list to obtain the final average displacement difference. Then, the difference between the two is compared with the pixel-outside calibration distance parameter to determine the change in the difference between the two, and the flow velocity value is calibrated.

6. The pixel-calibrated optical flow velocimetry system according to claim 4, characterized in that, The module M7 includes: Module M7.1: Obtains the initial predicted flow velocity value of the ROI through the detection model. Next, out-of-pixel calibration is performed, and the nearest neighbor density algorithm is used to filter out outliers; the nearest neighbor density algorithm compares the pixel displacement list. The density of each pixel displacement value and its neighboring points is used to determine whether the pixel displacement is an outlier. If the density of the pixel displacement value is lower, the pixel displacement is considered an outlier. (Equation 6) in, The nearest neighbor density function, The first one representing the input point Neighborhood; The output of the nearest neighbor density function, along with the input pixel displacement list. The lengths are the same, and they contain 1 and -1. The normal value is 1, and the abnormal value is -1. Module M7.2 outputs results based on the nearest neighbor density function. From pixel displacement list After filtering out outliers, the average value is taken to obtain the pixel position value of the detected part: (Equation 7) in, Indicates the corresponding pixel displacement in the saved pixel displacement list For a value at position 1, remove the corresponding... The value at position -1; This represents the pixel displacement value output by the detection section; Module M7.3: Measured multiple water flow videos during the actual test process. The baseline flow rate corresponding to the mean The calibration parameters of the detected pixels were obtained. Then obtain the flow rate value from the first calibration. ,in To detect the external calibration parameters of some output pixels The predicted flow rate value output by the detection section at that time; (Equation 8) Module M7.4: Outputs from the detection section With the corresponding ROI pixel external calibration output After comparison, then... The second calibration yields the final flow rate value output by the detection algorithm. To adapt to dynamic changes in actual flow velocity: (Equation 9).

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the pixel-calibrated optical flow velocimetry method according to any one of claims 1 to 3.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the pixel-calibrated optical flow velocimetry method according to any one of claims 1 to 3.

Images