A method and system for video flow measurement

Abstract

The application provides a video flow measurement method and system, relates to the technical field of hydrological monitoring, and solves the technical problem that the prior art cannot effectively eliminate camera jitter artifacts while realizing automatic processing, adapt to weak water surface texture scenes, and cause insufficient water surface flow speed measurement accuracy and robustness. The method specifically comprises the following steps: acquiring continuous video frames of a water surface to be measured; performing motion compensation processing on adjacent frames in the continuous video frames to obtain a stabilized video frame aligned with a previous frame coordinate system; automatically selecting a ROI in the stabilized video frame based on the ROI to screen and determine an optimal speed measurement area ROI; performing adaptive texture enhancement processing on the optimal speed measurement area ROI based on a CLAHE algorithm to generate an enhanced ROI; and inputting the enhanced ROI into a water surface flow speed calculation model to determine the water surface flow speed. The application is used for video flow measurement.

Description

Technical Field

[0001] This application relates to the field of hydrological monitoring technology, and in particular to a video flow measurement method and system. Background Technology

[0002] Water surface velocity measurement is a crucial link in hydrological monitoring, directly providing core data support for water resource allocation, flood control and disaster reduction, and water environment management. Current water surface velocity measurement technologies are mainly divided into two categories: contact and non-contact. Contact methods require equipment to be installed in the water, making them susceptible to water flow impact and sediment deposition, inconvenient to deploy and maintain, and difficult to adapt to complex flow fields. Non-contact methods include radar flow measurement, sonar flow measurement, and early video flow measurement. Radar and sonar equipment are expensive, while early video flow measurement methods often rely on manual subjective selection of the measurement area (ROI), resulting in low automation. Some methods do not effectively correct for non-fluid motion artifacts introduced by camera shake, leading to interference information in pixel displacement calculations. Furthermore, in low-light, gentle water flow, and other scenarios with weak surface textures, there is a lack of adaptive texture enhancement processing, resulting in insufficient extraction of water flow features and decreased accuracy in optical flow calculations or spatiotemporal imaging (STI) analysis, making it difficult to balance automation, robustness, and accuracy in measurement. Therefore, existing technologies suffer from the technical problem that current video flow measurement methods cannot effectively eliminate camera shake artifacts and adapt to scenes with weak water surface textures while achieving automated processing, resulting in insufficient accuracy and robustness in water surface flow velocity measurement. Summary of the Invention

[0003] This application provides a video flow measurement method and system, which solves the technical problem that existing video flow measurement methods are difficult to automate while effectively eliminating camera shake artifacts and adapting to scenes with weak water surface textures, resulting in insufficient accuracy and robustness in water surface flow velocity measurement.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] Firstly, a video flow measurement method is provided, comprising: acquiring continuous video frames of the surface of the water body to be measured; performing motion compensation processing on adjacent frames in the continuous video frames to obtain a stable video frame aligned with the coordinate system of the previous frame; automatically selecting and determining the optimal velocity measurement region (ROI) in the stable video frames based on the ROI; performing adaptive texture enhancement processing on the optimal velocity measurement region (ROI) based on the CLAHE algorithm to generate an enhanced ROI; inputting the enhanced ROI into a water surface velocity calculation model to determine the water surface velocity; the water surface velocity calculation model includes an STI analysis branch and a CNN optical flow estimation branch; the STI analysis branch is used to generate an STI based on the enhanced ROI, detect the main texture direction of the STI, and perform spatial calibration from the image plane to the object plane in combination with the first homography matrix to calculate the water surface velocity; the CNN optical flow estimation branch is used to construct a dense optical flow field using the encoder-decoder structure of the CNN, and determine the water surface velocity based on the dense optical flow field.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, motion compensation processing is performed on adjacent frames in a continuous video frame to obtain a stable video frame aligned with the coordinate system of the previous frame. This includes: extracting the adjacent previous and current frames from the continuous video frames; converting the previous and current frames into grayscale images respectively; using the ORB algorithm to detect feature points in the static background region of the grayscale image; matching the feature point descriptors of the previous and current frames using a brute-force matching method combined with Hamming distance to obtain feature point matching pairs; using cross-validation to remove erroneous matching point pairs from the feature point matching pairs and retaining high-quality matching point pairs; using the RANSAC algorithm to determine a 3×3 second homography matrix based on the coordinates of the high-quality matching point pairs; the second homography matrix is ​​used to describe the camera translation, rotation, scaling, and perspective transformation motion from the image plane of the previous frame to the image plane of the current frame; calculating the inverse matrix of the second homography matrix, and applying the inverse matrix to the current frame based on the perspective transformation function to perform perspective transformation, generating a stable video frame aligned with the coordinate system of the previous frame.

[0007] In conjunction with the first aspect mentioned above, one possible implementation involves automatically selecting and determining the optimal velocity measurement region (ROI) in the stabilized video frame based on the ROI. This includes: dividing the stabilized video frame into a preset number of grids, with each grid serving as a candidate ROI; calculating texture feature evaluation metrics and motion feature evaluation metrics for each candidate ROI; calculating the gradient magnitude of all pixels within the candidate ROI using the Sobel or Scharr operator, and then averaging all gradient magnitudes to obtain the average gradient magnitude, which serves as the texture feature evaluation metric; calculating the motion vectors of feature points within the candidate ROI using the Lucas-Kanade sparse optical flow algorithm, and then determining the consistency score of the motion vectors in both direction and magnitude, which serves as the motion feature evaluation metric; weighted summing of the texture feature evaluation metric and motion feature evaluation metric for each candidate ROI to obtain a comprehensive score for each candidate ROI; selecting the candidate ROI with the highest comprehensive score as the optimal velocity measurement region (ROI) in the stabilized video frame, and performing the step of determining the optimal velocity measurement region (ROI) for each stabilized video frame.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the motion vectors of feature points within the candidate ROI are calculated using the Lucas-Kanade sparse optical flow algorithm, and then the consistency score of the motion vectors in direction and magnitude is determined. This includes: selecting feature points in the previous frame of two consecutive stabilized video frames corresponding to the candidate ROI using the Harris corner detection algorithm; defining a local window of a preset size centered on each selected feature point; and within the local window, calculating the spatial gradient of the image in the x-direction, the spatial gradient in the y-direction, and the temporal gradient between the two frames to construct the basic optical flow equation. The basic optical flow equation satisfies the following formula:

[0009]

[0010] in, The optical flow component in the x-direction. For the optical flow component in the y-direction, The spatial gradient of the image in the x-direction. This represents the spatial gradient of the image in the y-direction. The gradient of the image in the time dimension is calculated. Based on the fundamental equation of optical flow, a linear equation is constructed for each pixel within a local window to form an overdetermined system of equations. The least squares method is used to solve the overdetermined system of equations to obtain the motion vector (u,v) of each feature point in the x and y directions. The average direction and average magnitude of all motion vectors are calculated, and then the standard deviation of the direction and the coefficient of variation of the magnitude are calculated respectively. The consistency score is calculated by weighted summation of the standard deviation of the direction and the coefficient of variation of the magnitude.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, adaptive texture enhancement processing is performed on the optimal velocity measurement region (ROI) based on the CLAHE algorithm to generate an enhanced ROI. This includes: dividing the optimal velocity measurement region (ROI) into multiple non-overlapping rectangular sub-blocks; the size of the rectangular sub-blocks is preset according to the ROI size and the water surface texture scale; calculating the pixel grayscale histogram of each sub-block, setting a clipping threshold for each histogram, and evenly distributing the sum of the portion of any grayscale level count value in the histogram that exceeds the clipping threshold to all grayscale levels in the entire histogram; performing standard histogram equalization on each sub-block of the adjusted histogram to obtain the grayscale mapping function of each sub-block; and performing biline interpolation on the grayscale mapping functions of the four adjacent sub-blocks of each sub-block to obtain the enhanced ROI.

[0012] In conjunction with the first aspect mentioned above, in one possible implementation, an STI (Sequential Texture Indicator) is generated based on an enhanced ROI, the main texture direction of the STI is detected, and spatial calibration from the image plane to the object plane is performed using the first homography matrix to calculate the surface velocity of the water body. This includes: setting a velocity line with a single pixel width and a preset length of n in the water flow direction of the enhanced ROI; synthesizing an n×m corresponding STI for a sequence of continuously acquired m frames of enhanced ROI images, with the water flow spatial position as the x-axis and the acquisition time as the t-axis, and performing noise reduction processing on the STI; determining the eigenvalues ​​and eigenvectors of the local gradient covariance matrix of the STI using the gray-level gradient tensor method, and detecting the main texture direction of the STI; determining the first homography matrix based on the perspective transformation relationship between 2D planes, and converting pixel displacement into physical distance by accumulating the object-image scale factors of the pixels related to the velocity line in the image plane;

[0013] The surface velocity of the water body is calculated by combining the main direction of the texture with the calibrated physical distance.

[0014] In conjunction with the first aspect mentioned above, in one possible implementation, a first homography matrix is ​​determined based on the perspective transformation relationship between 2D planes, and pixel displacement is converted into physical distance by accumulating the object-image scale factors of related pixels of the velocity line in the image plane. This includes: determining the first homography matrix based on the perspective transformation relationship between 2D planes; the transformation relationship satisfies the following formula:

[0015]

[0016] in, Let x and y be the first homography matrix, and x and y be the two-dimensional coordinates of a pixel in the image plane. , Let be the two-dimensional coordinates in the object plane corresponding to the image plane, and s be a non-zero scaling factor. (i, j=1,2,3) are the elements of the first homography matrix; for pixel displacement on the velocity measurement line, after converting the pixel coordinates to physical coordinates on the object plane through the first homography matrix, the physical distance between adjacent pixels on the object plane is calculated and accumulated to obtain the total physical distance in the x and y directions; the total physical distance in the x and y directions satisfies the following formula:

[0017]

[0018] in, This represents the total physical distance in the x-direction. This represents the total physical distance in the y-direction. Let x be the object scale factor of the pixel on the velocity measurement line in the x-direction. Let be the object scale factor of the pixel on the velocity measurement line in the y-direction. This is the starting pixel index of the velocity line in the x-direction of the image plane. This is the index of the terminating pixel of the velocity line in the x-direction of the image plane. This is the starting pixel index of the velocity line in the y-direction of the image plane. This is the index of the terminating pixel of the velocity line in the y-direction of the image plane.

[0019] In conjunction with the first aspect mentioned above, in one possible implementation, by combining the principal direction of the texture with the calibrated physical distance, the surface velocity of the water body is calculated to satisfy the following formula:

[0020]

[0021] in, The surface velocity of the water body. for , The composite physical distance along the main direction of the texture, where T is the water flow motion. The corresponding time, Let S be the pixel displacement of the water flow along the principal direction of the texture in the image coordinate system, where S is the spatial calibration coefficient and M is the number of video frames corresponding to the water flow movement of r pixels. The time interval for a single frame of an image. The main direction of the STI texture. This represents the pixel rate of the water flow in the image.

[0022] In conjunction with the first aspect mentioned above, one possible implementation involves constructing a dense optical flow field using the encoder-decoder structure of a CNN, and determining the surface velocity of the water body based on the dense optical flow field. This includes: acquiring two consecutive frames of enhanced ROI images as input image pairs, and inputting them into a CNN model containing an encoder-decoder structure; the CNN model includes a feature extraction encoder, a correlation layer, and an optical flow estimation decoder; extracting multi-scale depth feature maps from low to high levels from the input image pairs using the feature extraction encoder; calculating the pixel-wise correlation between the two frame feature maps using the correlation layer to generate a 4D correlation volume encoding pixel displacement information; performing deconvolution upsampling and convolution operations on the multi-scale feature maps and the 4D correlation volume based on the optical flow estimation decoder to progressively optimize and obtain a dense optical flow field matching the size of the enhanced ROI; multiplying the pixel displacement vectors in the dense optical flow field by spatial calibration coefficients and the video frame rate to convert them into velocity vectors; and performing median filtering on the velocity vectors to obtain the surface velocity of the water body.

[0023] Secondly, a video flow measurement system is provided, comprising: a video acquisition module, a preprocessing module, a velocity measurement module, and an output module; wherein, the video acquisition module is used to acquire continuous video frames of the surface of the water body to be measured; the preprocessing module includes a motion compensation submodule, an automatic ROI selection submodule, and an image enhancement submodule; the motion compensation submodule is used to perform motion compensation processing on adjacent frames in the continuous video frames to obtain a stable video frame aligned with the coordinate system of the previous frame; the automatic ROI selection submodule is used to automatically select and determine the optimal velocity measurement region ROI in the stable video frames based on ROI selection; the image enhancement submodule is used to perform adaptive texture enhancement processing on the optimal velocity measurement region ROI based on the CLAHE algorithm to generate an enhanced ROI; the velocity measurement module is used to input the enhanced ROI into the water surface velocity calculation model to determine the water surface velocity; and the output module is used to receive the water surface velocity results output by the velocity measurement module and present the water surface velocity data on a display screen.

[0024] Thirdly, this application provides a video flow measurement device, including: a communication unit and a processing unit; the communication unit is used to acquire continuous video frames of the surface of the water body to be measured; the processing unit is used to perform motion compensation processing on adjacent frames in the continuous video frames to obtain a stable video frame aligned with the coordinate system of the previous frame; automatically select and determine the optimal velocity measurement region ROI in the stable video frames based on ROI; perform adaptive texture enhancement processing on the optimal velocity measurement region ROI based on the CLAHE algorithm to generate an enhanced ROI; input the enhanced ROI into the water surface velocity calculation model to determine the water surface velocity; the water surface velocity calculation model includes an STI analysis branch and a CNN optical flow estimation branch; the STI analysis branch is used to generate STI based on the enhanced ROI, detect the main texture direction of the STI and perform spatial calibration from the image plane to the object plane in combination with the first homography matrix, and calculate the water surface velocity; the CNN optical flow estimation branch is used to construct a dense optical flow field using the encoder-decoder structure of CNN, and determine the water surface velocity based on the dense optical flow field.

[0025] Fourthly, this application provides a video flow measurement device, comprising: a processor and a storage medium; the storage medium includes instructions, and the processor is configured to execute the instructions to implement the methods described in the first aspect and any possible implementation thereof. The video flow measurement device may be an electronic device or a chip within an electronic device.

[0026] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a video flow measurement device, cause the video flow measurement device to perform the methods described in the first aspect and any possible implementation thereof.

[0027] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a video flow measurement device, cause the video flow measurement device to perform the methods described in the first aspect and any possible implementation thereof.

[0028] This application provides a video flow measurement method and system that eliminates non-fluid motion artifacts through motion compensation processing, ensuring that pixel displacement reflects real water flow. Automatic ROI selection eliminates the need for manual intervention, improving processing efficiency and objectivity. CLAHE adaptive enhancement strengthens weak textures, broadening the method's applicability. The dual-branch velocimetry mechanism combining STI and CNN optical flow leverages the reliability of traditional algorithms with the high precision of deep learning. This achieves high automation, robustness, and accuracy in water surface velocity measurement, solving the technical problem that existing video flow measurement methods struggle to effectively eliminate camera shake artifacts and adapt to scenes with weak water surface textures while achieving automated processing, resulting in insufficient accuracy and robustness in water surface velocity measurement.

[0029] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0030] Figure 1 A system architecture diagram of a video flow measurement system provided in this application embodiment;

[0031] Figure 2 A flowchart illustrating a video flow measurement method provided in an embodiment of this application;

[0032] Figure 3 A comparative schematic diagram illustrating the adaptive texture enhancement effect provided in the embodiments of this application;

[0033] Figure 4 A flowchart illustrating another video flow measurement method provided in this application embodiment;

[0034] Figure 5 A schematic diagram of STI provided for an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of calibration point selection provided in an embodiment of this application;

[0036] Figure 7 A flowchart illustrating another video flow measurement method provided in this application embodiment;

[0037] Figure 8 A CNN network structure diagram provided for an embodiment of this application;

[0038] Figure 9 A schematic diagram of a CNN architecture provided for an embodiment of this application;

[0039] Figure 10 This is a schematic diagram of the structure of a video flow measurement device provided in an embodiment of this application;

[0040] Figure 11This is a schematic diagram of the hardware structure of a video flow measurement device provided in an embodiment of this application. Detailed Implementation

[0041] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0042] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0043] The video flow measurement method provided in this application embodiment can be applied to, for example... Figure 1 The video flow measurement system shown includes: a video acquisition module 101, a preprocessing module 102, a speed measurement module 103, and an output module 104.

[0044] The video acquisition module 101 is used to acquire continuous video frames of the surface of the water body to be measured; the preprocessing module 102 includes a motion compensation submodule 102A, an automatic ROI selection submodule 102B, and an image enhancement submodule 102C; the motion compensation submodule 102A is used to perform motion compensation processing on adjacent frames in the continuous video frames to obtain a stable video frame aligned with the coordinate system of the previous frame; the automatic ROI selection submodule 102B is used to filter and determine the optimal velocity measurement region ROI in the stable video frames based on the automatic ROI selection; the image enhancement submodule 102C is used to perform adaptive texture enhancement processing on the optimal velocity measurement region ROI based on the CLAHE algorithm to generate an enhanced ROI; the velocity measurement module 103 is used to input the enhanced ROI into the water surface velocity calculation model to determine the water surface velocity; the output module 104 is used to receive the water surface velocity results output by the velocity measurement module and present the water surface velocity data on the display screen.

[0045] To address the technical problem that existing video flow measurement methods struggle to achieve automated processing while effectively eliminating camera shake artifacts and adapting to scenes with subtle water surface textures, resulting in insufficient accuracy and robustness in water surface velocity measurement, this application provides a video flow measurement method. This method includes: acquiring continuous video frames of the water surface to be measured; performing motion compensation on adjacent frames to obtain stable video frames; automatically selecting the optimal velocity measurement region (ROI); enhancing the ROI using the CLAHE algorithm; and then inputting the enhanced ROI into a model containing STI analysis and CNN optical flow estimation branches to determine the flow velocity. Based on this, motion compensation eliminates non-fluid motion interference, automatic ROI improves the automation of video flow measurement, CLAHE adapts the flow measurement to scenes with subtle textures, and finally, the water surface velocity is determined through a dual-branch approach, ensuring measurement accuracy and robustness. This solves the technical problem that existing video flow measurement methods struggle to achieve automated processing while effectively eliminating camera shake artifacts and adapting to scenes with subtle water surface textures, resulting in insufficient accuracy and robustness in water surface velocity measurement.

[0046] Figure 2 This is a flowchart illustrating the video flow measurement method provided in the embodiments of this application, as shown below. Figure 2 As shown, the method includes:

[0047] S201. Acquire continuous video frames of the surface of the water body to be tested.

[0048] Among them, continuous video frames refer to the sequence of images of the surface of the water body to be tested, collected sequentially at different time points, which must ensure temporal continuity and spatial coverage.

[0049] In this embodiment of the application, the video acquisition module acquires continuous video frames of the surface of the water body to be tested. During the acquisition process, it is necessary to ensure that the equipment is stably set up to avoid spatial misalignment of the frame sequence due to initial position offset.

[0050] It should be noted that the resolution and frame rate of the captured video frames need to be adapted to the subsequent processing requirements. Usually, a resolution of 1080P or higher and a frame rate of 25fps or higher are selected.

[0051] As an example, a high-definition industrial camera is deployed on a fixed bracket on the riverbank to capture continuous video frames of the river section at a resolution of 1920×1080 and a frame rate of 60fps. The acquisition time is set to 5-10 minutes according to the flow measurement requirements.

[0052] Based on the above steps, stable raw image data required for flow measurement is directly obtained, ensuring that subsequent motion compensation and ROI selection processing have a complete and reliable input foundation.

[0053] S202. Perform motion compensation processing on adjacent frames in a series of video frames to obtain a stable video frame aligned with the coordinate system of the previous frame.

[0054] Motion compensation processing refers to the process of eliminating non-fluid motion artifacts introduced by camera shake, translation, rotation, etc.; stabilized video frames refer to video frames that are aligned with the coordinate system of the previous frame and retain only the pixel changes caused by water flow.

[0055] In this embodiment, the motion compensation submodule extracts adjacent previous and current frames from consecutive video frames; converts the previous and current frames into grayscale images; uses the ORB algorithm to detect feature points in static background areas (such as riverbanks, buildings, and fixed landmarks) of the grayscale images; matches the feature point descriptors of the previous and current frames using a brute-force matching method combined with Hamming distance to obtain feature point matching pairs; employs cross-validation to eliminate erroneous matching point pairs and retain high-quality matching point pairs; and uses the RANSAC algorithm to determine a 3×3 second homography matrix based on the coordinates of the high-quality matching point pairs. , Used to describe camera translation, rotation, scaling, and perspective changes from the previous frame image plane to the current frame image plane; calculates... The inverse matrix is ​​obtained, and the inverse matrix is ​​applied to the current frame for perspective transformation based on a perspective transformation function (such as the warpPerspective function in OpenCV) to generate a stable video frame aligned with the coordinate system of the previous frame.

[0056] Optionally, solve the matrix The inverse matrix satisfies the following standard formula for finding the matrix inversion:

[0057]

[0058] in, To perform operations for calculating the determinant of a matrix, To compute the adjoint matrix operation.

[0059] It should be noted that static background areas should avoid water flow areas to ensure that the feature point matching reflects the camera movement rather than the water flow movement; ORB combines the advantages of FAST corner detector and BRIEF descriptor, and has the characteristics of fast calculation speed and good invariance to rotation and scale changes; the brute-force matching method combined with Hamming distance is used to match the feature point descriptors of the previous frame and the current frame to obtain feature point matching pairs, and the specific process of cross-validation is not specifically limited in this application.

[0060] As an example, if a video of a lake is captured, the system detects 200 feature points in the bank slope area in the grayscale images of the previous and current frames. After brute-force matching and Hamming distance calculation, 180 pairs of matching points are obtained. Then, 30 pairs of incorrect matching points are removed through cross-validation, retaining 150 high-quality matching points. Finally, the RANSAC algorithm is used to fit the data to obtain... After calculating its inverse matrix, a perspective transformation is performed on the current frame to finally generate a stable frame aligned with the coordinate system of the previous frame, eliminating background displacement caused by slight camera shake.

[0061] Based on the above steps, interference introduced by camera movement can be effectively eliminated, ensuring that the pixel displacement calculated subsequently is caused only by water flow, thus improving the accuracy of flow velocity calculation.

[0062] S203. Based on ROI, automatically select and determine the optimal speed measurement region ROI in the stabilized video frames.

[0063] The ROI (Region of Interest) refers to a specific rectangular area in a video frame used for flow rate calculation.

[0064] In this embodiment, the ROI automatic selection submodule divides the stabilized video frames into a preset number of grids (e.g., 5×5), with each grid serving as a candidate ROI. For each candidate ROI, texture feature evaluation metrics and motion feature evaluation metrics are calculated: the texture feature evaluation metrics are calculated using the Sobel or Scharr operator to determine the gradient magnitude of all pixels within the candidate ROI, and then the average gradient magnitude is obtained by averaging all gradient magnitudes. A higher average gradient magnitude indicates richer texture. The motion feature evaluation metrics are calculated using the Lucas-Kanade sparse optical flow algorithm: in two consecutive stabilized video frames corresponding to the candidate ROI, feature points are selected in the previous frame using the Harris corner detection algorithm; for each selected feature point, a local window of a preset size (e.g., 15×15 pixels) centered on the feature point is defined; within the local window, the spatial gradient of the image in the x-direction is calculated. Spatial gradient in the y-direction and the temporal gradient between two frames The process involves constructing the fundamental equations of optical flow; based on these equations, constructing linear equations for each pixel within a local window to form an overdetermined system of equations; solving the overdetermined system of equations using the least squares method to obtain the motion vectors (u, v) of each feature point in the x and y directions; calculating the average direction and average magnitude of all motion vectors, and then calculating the standard deviation of the direction and the coefficient of variation of the magnitude; calculating the consistency score by weighted summation of the standard deviation of the direction and the coefficient of variation of the magnitude; weighted summation of the texture feature evaluation index and motion feature evaluation index of each candidate ROI to obtain the comprehensive score of each candidate ROI; selecting the candidate ROI with the highest comprehensive score as the optimal velocity measurement region ROI in the stabilized video frame, and performing the step of determining the optimal velocity measurement region ROI for each stabilized video frame.

[0065] Optionally, the fundamental equation of optical flow satisfies the following formula:

[0066]

[0067] in, The optical flow component in the x-direction. For the optical flow component in the y-direction, The spatial gradient of the image in the x-direction. This represents the spatial gradient of the image in the y-direction. This represents the gradient of the image over time.

[0068] It should be noted that the number of grid divisions needs to be adjusted according to the water area. Large water areas can be divided into 8×8 grids, while small water areas can be divided into 3×3 grids to avoid having too many or too few candidate ROIs, which would affect the efficiency and accuracy of the selection.

[0069] As an example, if the stabilized frame is divided into a 5×5 grid (25 candidate ROIs), a candidate ROI, calculated by the Sobel operator, has an average gradient magnitude of 85 (high texture score); using the Lucas-Kanade algorithm to select 12 feature points, the average direction of the motion vector is found to be 135°, the average size is 5 pixels / frame, the standard deviation of the direction is 12°, and the coefficient of variation of the size is 0.15. A weighted sum (texture weight 0.4, motion weight 0.6) yields a comprehensive score of 82.1, the highest among all candidate ROIs, and it is selected as the optimal velocity measurement region ROI; if a certain local window... =4、 =3、 =-22 (taking a negative value indicates that the gray value of the current frame pixel is lower than that of the previous frame), substituting into the basic equation of optical flow, we get 4u+3v=22. Combining the linear equations of the other 12 pixels in the window, we use the least squares method to solve for u=4 and v=2, that is, the motion vector of the feature point is (4,2).

[0070] Based on the above steps, the system automatically selects velocity measurement areas with rich textures and stable water flow without human intervention, thereby improving the automation level and objectivity of flow measurement results.

[0071] S204. Based on the CLAHE algorithm, adaptive texture enhancement processing is performed on the optimal velocity measurement region ROI to generate an enhanced ROI.

[0072] Among them, the CLAHE algorithm (contrast-limited adaptive histogram equalization) is an algorithm that enhances the local texture of an image by processing the histogram in blocks and limiting contrast overflow; the enhanced ROI refers to the optimal velocity measurement area after texture detail enhancement and contrast improvement.

[0073] In this embodiment, the image enhancement submodule divides the optimal velocity measurement region (ROI) into multiple non-overlapping rectangular sub-blocks. The size of the rectangular sub-blocks is preset based on the ROI size and the water surface texture scale (e.g., when the ROI is 256×256 pixels, the sub-block is set to 32×32 pixels). A pixel grayscale histogram for each sub-block is calculated, and a cropping threshold (usually 2-4 times the histogram mean) is set for each histogram. The sum of any grayscale count value exceeding the cropping threshold in the histogram is evenly distributed across all grayscale levels in the entire histogram. Standard histogram equalization is performed on each sub-block of the adjusted histogram to obtain the grayscale mapping function for each sub-block. Bilinear interpolation is performed on the grayscale mapping functions of the four adjacent sub-blocks of each sub-block to eliminate the blockiness caused by the segmentation, resulting in an enhanced ROI.

[0074] It should be noted that the cropping threshold needs to be adjusted according to the water surface brightness. In low-light scenes, the threshold can be appropriately reduced to avoid excessive noise enhancement. The grayscale mapping function is the correspondence between the "original grayscale value and the enhanced grayscale value" generated after histogram adjustment and equalization of each sub-block: first, the grayscale histogram of each sub-block is cropped to remove the count value exceeding the threshold and evenly distributed. Then, the cumulative distribution function is calculated through standard histogram equalization to map the probability distribution of the original grayscale level to a uniform grayscale distribution. Each sub-block has an independent mapping function due to its different grayscale characteristics, thus achieving local adaptive texture enhancement. Bilinear interpolation is used for target pixels in the enhanced ROI. First, four adjacent sub-blocks are selected, and the mapping function of each sub-block is called to obtain the initial enhanced grayscale value. Then, weights are assigned according to the distance from the pixel to the center of the sub-block (the closer the distance, the greater the weight). The final enhanced grayscale value of the pixel is obtained by weighted summation, ensuring smooth grayscale transition at the edge of the sub-block and generating a natural enhanced ROI. The specific process of constructing the grayscale mapping function and bilinear interpolation is not specifically limited in this application.

[0075] As an example, such as Figure 3 The comparative diagram of the adaptive texture enhancement effect is shown. If the optimal ROI is 128×128 pixels, it is divided into 16×16 pixel sub-blocks (64 sub-blocks in total). A clipping threshold of 45 is set for the histogram of each sub-block. The count value of gray level 30 in a certain sub-block is 68, which exceeds the threshold of 23. The 23 is evenly distributed to 256 gray levels (each gray level increases by about 0.09). The gray level mapping function of the sub-block is obtained by histogram equalization. Then, the mapping functions of the four adjacent sub-blocks are bi-line interpolated to finally generate the enhanced ROI.

[0076] Based on the above steps, the subtle water surface textures in low-light and gentle water flow scenarios can be enhanced, avoiding calculation errors caused by texture blurring.

[0077] S205. Input the enhanced ROI into the water surface velocity calculation model to determine the water surface velocity.

[0078] The water surface velocity calculation model is a two-branch model that includes an STI analysis branch and a CNN optical flow estimation branch. It can select single-branch or dual-branch collaborative calculation according to the water flow scenario.

[0079] In this embodiment, the velocity measurement module inputs the enhanced ROI into the water surface velocity calculation model. The STI analysis branch is used to generate STI (spatial-temporal image) based on the enhanced ROI, detect the main texture direction of the STI, and perform spatial calibration from the image plane to the object plane in combination with the first homography matrix to calculate the water surface velocity. The CNN optical flow estimation branch is used to construct a dense optical flow field using the encoder-decoder structure of the CNN, and determine the water surface velocity based on the dense optical flow field.

[0080] It should be noted that the two branches can output flow rate results independently, or the final flow rate can be obtained through weighted fusion, thus improving the reliability of the results.

[0081] As an example, for an enhanced ROI of a turbulent river, the CNN optical flow estimation branch can be enabled to quickly output a dense optical flow field and determine the flow velocity; for an enhanced ROI of a calm lake, the STI analysis branch can be enabled to calculate the flow velocity through the principal direction of the STI texture and spatial calibration.

[0082] Based on the above steps, the surface velocity of the water body is determined by a dual-branch machine, which takes into account the reliability of traditional algorithms and the high accuracy of deep learning, ensuring the flexibility and accuracy of velocity calculation.

[0083] Based on the above steps, a high degree of automation in water surface velocity measurement is achieved. This effectively eliminates camera shake artifacts and is adaptable to scenes with weak textures, such as low light and gentle water flow. At the same time, dual-branch computation ensures measurement accuracy and robustness. This solution specifically addresses the technical problem that existing video flow measurement methods are unable to effectively eliminate camera shake artifacts and adapt to scenes with weak water surface textures while achieving automated processing, resulting in insufficient accuracy and robustness in water surface velocity measurement.

[0084] In one possible approach, combining the above... Figure 2 ,like Figure 4 As shown, the specific process of generating STI based on enhanced ROI, detecting the principal direction of the STI texture, and performing spatial calibration from the image plane to the object plane by combining the first homography matrix in S205 above, and calculating the surface velocity of the water body, can be implemented through the following S401-S405:

[0085] S401. Set a velocity measurement line with a single pixel width and a preset length of n in the direction of water flow in the enhanced ROI.

[0086] Among them, the velocity measurement line is a linear region with a single pixel width and a fixed length, used to accurately capture the movement trajectory of water flow along the spatial direction. The length n needs to be adapted to the enhanced ROI size and the water flow scale.

[0087] In this embodiment, the STI analysis branch first determines the water flow direction based on the water flow trend (such as texture direction) initially identified within the enhanced ROI, and then sets a single-pixel-wide velocity measurement line along this direction. The preset length n is set according to the horizontal / vertical dimensions of the ROI to ensure coverage of the main water flow area without exceeding the ROI range.

[0088] It should be noted that the speed measurement line should avoid background interference areas within the ROI.

[0089] As an example, if the enhanced ROI is 200×150 pixels and the water flow is horizontal, the system sets a velocity measuring line with a single pixel width and a length of n=100 pixels along the horizontal direction, located vertically centered in the ROI.

[0090] Based on the above steps, a precise spatial range is defined for subsequent water flow displacement detection, avoiding aimless global calculations and improving the efficiency of flow velocity calculation.

[0091] S402. For the continuously acquired m-frame enhanced ROI image sequence, synthesize the corresponding n×m STI with the water flow spatial position as the x-axis and the acquisition time as the t-axis, and perform noise reduction processing on the STI.

[0092] STI is a two-dimensional image generated by combining the spatial water flow information of multiple ROIs with the time dimension. The x-axis reflects the spatial location and the t-axis reflects the acquisition time sequence.

[0093] In this embodiment of the application, m frames of enhanced ROI image sequence are continuously acquired, and an n×m STI is synthesized according to the rule that "the gray values ​​of n pixels on the speed measurement line in each frame are used as a row of STI, and m frames correspond to m rows". Then, Gaussian filtering or median filtering is used to denoise the STI to eliminate texture interference caused by illumination fluctuations and pixel noise.

[0094] It should be noted that the value of m needs to balance time resolution and computational load, and is usually set to 20-30 frames.

[0095] Based on the above steps, the time-series water flow motion is transformed into a visualized image texture, providing a clear analytical basis for subsequent texture main direction detection.

[0096] S403. The gray-level gradient tensor method is used to determine the eigenvalues ​​and eigenvectors of the local gradient covariance matrix of STI, and the main texture direction of STI is detected.

[0097] Among them, the gray-level gradient tensor method is an algorithm that constructs a gradient covariance matrix by calculating the gradient information of local pixels in the image, and uses the matrix eigenvalues ​​and eigenvectors to represent the texture direction. The direction of the eigenvector corresponding to the largest eigenvalue is the main texture direction.

[0098] In this embodiment, the grayscale gradients of each pixel in the STI image in the x-direction (spatial axis) and y-direction (time axis) are first calculated; then, the STI is traversed with a sliding window of a preset size, and a local gradient covariance matrix is ​​constructed for the gradient values ​​of the pixels within the window; the eigenvalues ​​and eigenvectors of the covariance matrix are solved, and the direction of the eigenvector corresponding to the largest eigenvalue is selected as the texture direction of the window; finally, the texture directions of all windows are counted, and the direction with the highest frequency of occurrence is taken as the main texture direction of the STI.

[0099] It should be noted that the size of the sliding window needs to be adapted to the STI texture density to avoid the window being too large and causing directional blurring.

[0100] As an example, such as Figure 5 The STI diagram shown illustrates the oblique texture pattern of STI. The system traverses the STI using a 3×3 window. The direction of the eigenvector corresponding to the maximum eigenvalue of most windows aligns with the main texture direction marked in the diagram. Consistent, the final detected STI texture main direction matches the direction in the image.

[0101] S404. Determine the first homography matrix based on the perspective transformation relationship between 2D planes, and convert the pixel displacement into physical distance by accumulating the object-image scale factor of the related pixels of the velocity line in the image plane.

[0102] The first homography matrix is ​​a 3×3 non-singular matrix used to describe the perspective transformation relationship from the image plane (pixel coordinates) to the object plane (physical coordinates); the object-image scale factor is the actual physical length of a single pixel on the object plane, reflecting the mapping ratio between the pixel and the physical space.

[0103] In the embodiments of this application, such as Figure 6 The diagram showing the selection of calibration points illustrates the process of selecting calibration points with known physical coordinates on the object plane (such as fixed markers on the shore), obtaining their pixel coordinates on the image plane, and forming calibration point pairs. Based on the perspective transformation relationship between 2D planes, the RANSAC algorithm is used to robustly fit the calibration point pairs to obtain a 3×3 first homography matrix. Then, for each consecutive pixel on the velocity measurement line, its image plane coordinates are substituted into the first homography matrix to calculate the corresponding object plane coordinates. The distance between two adjacent pixels on the object plane is calculated as the object-image scale factor for a single pixel. Finally, the object-image scale factors of all adjacent pixels on the velocity measurement line are sequentially accumulated to obtain the total physical distance of the velocity measurement line in the x and y directions.

[0104] Optionally, the perspective transformation relationship of the first homography matrix satisfies the following formula:

[0105]

[0106] in, Let x and y be the first homography matrix, and x and y be the two-dimensional coordinates of a pixel in the image plane. , Let be the two-dimensional coordinates in the object plane corresponding to the image plane, and s be a non-zero scaling factor. (i, j=1,2,3) are the elements of the first homography matrix.

[0107] Optionally, the total physical distance in the x and y directions satisfies the following formula:

[0108]

[0109] in, This represents the total physical distance in the x-direction. This represents the total physical distance in the y-direction. Let x be the object scale factor of the pixel on the velocity measurement line in the x-direction. Let be the object scale factor of the pixel on the velocity measurement line in the y-direction. This is the starting pixel index of the velocity line in the x-direction of the image plane. This is the index of the terminating pixel of the velocity line in the x-direction of the image plane. This is the starting pixel index of the velocity line in the y-direction of the image plane. This is the index of the terminating pixel of the velocity line in the y-direction of the image plane.

[0110] It should be noted that the more calibration points selected, the higher the fitting accuracy of the RANSAC algorithm.

[0111] As an example, if the start and end indices of the speed measuring line in the x-direction... =10, =60, each The average value is 0.1m / pixel, and the cumulative values ​​are... =5m; Start and end indices in the y direction =5, =30, each The average value is 0.12m / pixel, and the cumulative result is... =3m.

[0112] Based on the above steps, a precise conversion from image plane pixel displacement to object plane physical distance is achieved, eliminating flow velocity calculation errors caused by the mismatch between pixel scale and actual spatial scale.

[0113] S405. By combining the main direction of the texture with the calibrated physical distance, the surface velocity of the water body is calculated.

[0114] The calibrated physical distance is obtained from S404. , The vector composite distance reflects the actual displacement of the water flow along the true direction.

[0115] In this embodiment of the application, firstly , The calibrated physical distance is obtained through vector synthesis calculation. Combined with water flow The corresponding time T (determined by the number of acquisition frames M and the single-frame interval) calculate, The surface velocity of water is calculated using the velocity formula.

[0116] Alternatively, the flow velocity formula satisfies the following formula:

[0117]

[0118] in, The surface velocity of the water body. According to , The calibrated physical distance, where T represents the water flow motion. The corresponding time, Let S be the pixel displacement in the image coordinate system, S be the spatial calibration coefficient (i.e., the object scale factor on the velocity measurement line), and M be the number of video frames corresponding to r pixels of water flow motion. The time interval for a single frame of an image. The main direction of the STI texture. This represents the pixel rate of the water flow in the image.

[0119] As an example, if r = 5m and S = 0.9m / pixel, calculate... =4.5m, the number of frames M is 25, and the single frame interval is... =0.04s, T=25×0.04=1s, calculate V=4.5m / s.

[0120] It should be noted that the derivation of the flow velocity formula is based on multi-dimensional equivalence relations.

[0121] Based on the above technical solution, through the progressive process of spatial positioning of velocity lines, STI temporal texture conversion, accurate detection of main direction, spatial scale calibration, and flow velocity calculation, the water flow information in the image domain is gradually converted into flow velocity data in the physical domain. This not only ensures the accuracy of the water flow direction, but also achieves accurate conversion from pixel to physical distance, and finally efficiently outputs reliable water surface flow velocity.

[0122] In one possible approach, combining the above... Figure 2 ,like Figure 7 As shown, in S205 above, a dense optical flow field is constructed using the encoder-decoder structure of a CNN. The specific process of determining the surface velocity of the water body based on the dense optical flow field can be implemented through the following S701-S706:

[0123] S701. Obtain two consecutive frames of enhanced ROI images as input image pairs, and input them into a CNN model with an encoder-decoder structure.

[0124] The input image pair refers to two temporally consecutive enhanced ROI images, requiring the continuity of water flow motion between frames; for example... Figure 8 The CNN network structure diagram shown is... Figure 9 The diagram shown illustrates a CNN architecture. The CNN model with an encoder-decoder structure is a neural network specifically designed for dense optical flow estimation, containing a feature extraction encoder, a correlation layer, and an optical flow estimation decoder.

[0125] In this embodiment, two consecutive adjacent frames are extracted from the preprocessed enhanced ROI sequence as input image pairs. After checking the size consistency of the two frames, they are input into a preset CNN model with an encoder-decoder structure to ensure that the input format meets the model requirements.

[0126] Based on the above steps, qualified input data is provided for the CNN model, ensuring that the model can analyze water flow motion based on continuous inter-frame information.

[0127] S702. Through the feature extraction encoder, multi-scale depth feature maps from low to high level are extracted from the input image pairs.

[0128] Among them, multi-scale deep feature maps refer to feature representations from low-level (edges, textures) to high-level (semantics, overall motion trends) of an image, with different levels of features corresponding to different resolutions.

[0129] In this embodiment, the feature extraction encoder extracts low-level features (such as water surface texture edges), mid-level features (such as local water flow patterns), and high-level features (such as overall water flow trends) of the image step by step through multiple sets of convolutional layers (3×3 kernel size) and downsampling layers (max pooling, stride 2), generating multiple sets of depth feature maps with different resolutions.

[0130] As an example, the feature extraction encoder processes two 256×256 pixel ROI images and generates four sets of multi-scale feature maps with resolutions of 128×128, 64×64, 32×32, and 16×16, corresponding to features from low to high level.

[0131] Based on the above steps, water flow features in different dimensions of the image were captured, providing multi-dimensional feature support for subsequent pixel displacement information encoding.

[0132] S703. Calculate the pixel-wise correlation between the feature maps of two frames through the correlation layer to generate a 4D correlation volume that encodes pixel displacement information.

[0133] The 4D correlation volume refers to a data structure with dimensions of "feature map height × feature map width × horizontal displacement range × vertical displacement range", which is used to encode the similarity of each pixel in the feature maps of two frames under different displacements.

[0134] In this embodiment, the relevant layer takes the multi-scale feature maps of two frames as input, calculates the feature similarity between each pixel in the current frame and the pixels with different displacements in the previous frame for each level of feature map, organizes the similarity results according to "height, width, x-direction displacement, y-direction displacement", generates the 4D correlation volume corresponding to each level, and completely encodes the pixel displacement information.

[0135] As an example, for a feature map with a resolution of 64×64, the preset displacement range is -3 to 3 pixels. After the correlation layer is calculated, a 4D correlation volume of 64×64×7×7 is generated, and the value at each position represents the pixel similarity under the corresponding displacement.

[0136] Based on the above steps, the displacement correlation information of pixels between two frames is accurately encoded.

[0137] S704: Based on the optical flow estimation decoder, deconvolution upsampling and convolution operations are performed on the multi-scale feature map and the 4D correlation volume to gradually optimize and obtain a dense optical flow field that matches the size of the enhanced ROI.

[0138] Among them, the dense optical flow field refers to the motion vector field covering each pixel of the enhanced ROI, and each vector contains pixel displacement information in the x and y directions; deconvolution upsampling is used to restore low-resolution features to the original ROI size.

[0139] In this embodiment, the optical flow estimation decoder receives multi-scale feature maps and 4D correlation volumes. It first performs deconvolution upsampling on the low-resolution 4D correlation volumes (3×3 deconvolution kernel, stride 2) to gradually restore the resolution. At the same time, it fuses with the multi-scale feature maps of the corresponding level and optimizes the optical flow information through convolution operations (1×1 convolution kernel). The upsampling and fusion process is repeated to finally obtain a dense optical flow field that perfectly matches the size of the enhanced ROI.

[0140] It should be noted that the upsampling process must correspond to the encoder downsampling step to ensure accurate size matching.

[0141] As an example, the optical flow estimation decoder processes multiple sets of data and progressively optimizes the optical flow field from 16×16 resolution to 256×256 pixels, consistent with the size of the input enhanced ROI, with each pixel corresponding to a (x,y) displacement vector.

[0142] Based on the above steps, high-precision displacement information covering the entire ROI is generated, providing pixel-level motion data for flow velocity calculation.

[0143] S705: Multiply the pixel displacement vector in the dense optical flow field by the spatial calibration coefficient and the video frame rate to convert it into a velocity vector.

[0144] Among them, the spatial calibration coefficient refers to the conversion coefficient between pixel displacement and actual physical distance (unit: m / pixel), which needs to be calibrated in advance through the homography matrix; the video frame rate refers to the number of frames per second (unit: fps) of image acquisition, which is used to convert the inter-frame time into actual time.

[0145] In this embodiment, the system first calls the pre-stored spatial calibration coefficient S (calculated from the first homography matrix) and the video frame rate f; then it traverses the displacement vector (u, v) of each pixel in the dense optical flow field, multiplies u and v by the spatial calibration coefficient and the video frame rate respectively, and obtains the physical velocity component (u×S×f, v×S×f) corresponding to each pixel, which is then combined to form a velocity vector.

[0146] As an example, the spatial calibration coefficient S = 0.05m / pixel, the video frame rate f = 25fps, the displacement vector of a certain pixel is (4,2) pixels / frame, and the converted velocity vector is (4×0.05×25,2×0.05×25)=(5,2.5).

[0147] Based on the above steps, a precise conversion from image domain pixel displacement to physical domain velocity is achieved, directly obtaining quantifiable water flow velocity data.

[0148] S706. Perform median filtering on the velocity vector to obtain the surface velocity of the water body.

[0149] In this embodiment, the system sets a sliding window of a preset size (e.g., 3×3) and traverses all velocity vectors; for each velocity vector in the window, the median of its magnitude (resultant velocity) is calculated, and the median is used to replace the magnitude of the velocity vector at the center of the window; after traversal, all processed velocity vector magnitudes are statistically analyzed, and the average value is taken as the final surface velocity of the water body.

[0150] It should be noted that the sliding window size needs to be adjusted according to the ROI size to avoid the window being too large and causing the speed information to become blurred.

[0151] Based on the above steps, the interference of abnormal velocity values ​​is eliminated, and the stability and reliability of the surface velocity results of the water body are improved.

[0152] Based on the above technical solution, through the process of input image pair preparation, multi-scale feature extraction, displacement information encoding, dense optical flow generation, velocity conversion, and outlier filtering, and relying on the advantages of the CNN encoder-decoder structure, end-to-end calculation from image features to physical flow velocity is realized. This not only captures pixel-level fine water flow motion, but also ensures flow velocity accuracy through calibration and filtering.

[0153] The foregoing mainly describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as a video flow measurement device, includes at least one of the hardware structures and software modules corresponding to the execution of each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0154] This application embodiment can divide the video flow measurement device into functional units according to the above method example. For example, each function can be divided into separate functional units, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0155] When using integrated units, Figure 10 A possible structural schematic diagram of the video flow measurement device (referred to as video flow measurement device 100) involved in the above embodiments is shown. The video flow measurement device 100 includes a processing unit 1001 and a communication unit 1002, and may also include a storage unit 1003. Figure 10 The structural diagram shown can be used to illustrate the structure of the video flow measurement device involved in the above embodiments.

[0156] when Figure 10The structural diagram shown is used to illustrate the structure of the video flow measurement device involved in the above embodiments. The processing unit 1001 is used to control and manage the operation of the video flow measurement device, the communication unit 1002 is used for the video flow measurement device to communicate with other devices, and the storage unit 1003 is used to store the program code and data of the video flow measurement device.

[0157] For example, communication unit 1002 is used to acquire continuous video frames of the surface of the water body to be measured;

[0158] The processing unit 1001 is used to perform motion compensation processing on adjacent frames in continuous video frames to obtain stable video frames aligned with the coordinate system of the previous frame; automatically select and determine the optimal velocity measurement region (ROI) in the stable video frames based on ROI; perform adaptive texture enhancement processing on the optimal velocity measurement region (ROI) based on the CLAHE algorithm to generate an enhanced ROI; input the enhanced ROI into the water surface velocity calculation model to determine the water surface velocity; the water surface velocity calculation model includes an STI analysis branch and a CNN optical flow estimation branch; the STI analysis branch is used to generate STI based on the enhanced ROI, detect the main texture direction of the STI and perform spatial calibration from the image plane to the object plane in combination with the first homography matrix, and calculate the water surface velocity; the CNN optical flow estimation branch is used to construct a dense optical flow field using the encoder-decoder structure of the CNN, and determine the water surface velocity based on the dense optical flow field.

[0159] The processing unit 1001 can be a processor or a controller, and the communication unit 1002 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 1003 can be a memory. When the video flow measurement device 100 is a chip, the processing unit 1001 can be a processor or a controller, and the communication unit 1002 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 1003 can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., read-only memory (ROM), random access memory (RAM, etc.).

[0160] The communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the video flow measurement device 100 can be considered as the communication unit 1002 of the video flow measurement device 100, and the processor with processing functions can be considered as the processing unit 1001 of the video flow measurement device 100. Optionally, the device in the communication unit 1002 that implements the receiving function can be considered as the communication unit. The communication unit is used to execute the receiving steps in the embodiments of this application, and the communication unit can be a receiver, a receiver circuit, etc. The device in the communication unit 1002 that implements the transmitting function can be considered as the transmitting unit. The transmitting unit is used to execute the transmitting steps in the embodiments of this application, and the transmitting unit can be a transmitter, a transmitter, a transmitting circuit, etc.

[0161] Figure 10 If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0162] Figure 10 The units in the process can also be called modules; for example, a processing unit can be called a processing module.

[0163] This application also provides a hardware structure diagram of a video flow measurement device (referred to as video flow measurement device 110), see [link to diagram]. Figure 11 The video flow measurement device 110 includes a processor 1101, and optionally, a memory 1102 connected to the processor 1101.

[0164] In the first possible implementation, see Figure 11The video flow measurement device 110 also includes a transceiver 1103. The processor 1101, memory 1102, and transceiver 1103 are connected via a bus. The transceiver 1103 is used to communicate with other devices or communication networks. Optionally, the transceiver 1103 may include a transmitter and a receiver. The device in the transceiver 1103 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 1103 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.

[0165] Based on the first possible implementation method Figure 11 The structural diagram shown can be used to illustrate the structure of the video flow measurement device involved in the above embodiments.

[0166] in, Figure 11 The system chip in the video flow measurement device can also be illustrated. In this case, the actions performed by the aforementioned video flow measurement device can be implemented by this system chip. The specific actions performed can be found above and will not be repeated here.

[0167] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0168] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a separate semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may be integrated with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a SoC (System-on-a-Chip), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0169] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto.

[0170] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0171] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0172] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0173] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0174] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0175] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A video flow measurement method, characterized in that, include: Acquire continuous video frames of the surface of the water body to be tested; Motion compensation processing is performed on adjacent frames in the continuous video frames to obtain a stable video frame aligned with the coordinate system of the previous frame. The optimal speed measurement region (ROI) is automatically selected and determined from the stabilized video frames based on the ROI. The optimal velocity measurement region (ROI) is adaptively textured using the CLAHE algorithm to generate an enhanced ROI. The enhanced ROI is input into the water surface velocity calculation model to determine the water surface velocity; the water surface velocity calculation model includes an STI analysis branch and a CNN optical flow estimation branch. The STI analysis branch is used to generate STI based on the enhanced ROI, detect the main texture direction of the STI and perform spatial calibration from the image plane to the object plane in combination with the first homography matrix, and calculate the surface velocity of the water body; the CNN optical flow estimation branch is used to construct a dense optical flow field using the encoder-decoder structure of the CNN, and determine the surface velocity of the water body based on the dense optical flow field. The surface velocity of the water body output by the STI analysis branch and the CNN optical flow estimation branch is weighted and fused to obtain the final surface velocity of the water body.

2. The method according to claim 1, characterized in that, Motion compensation processing is performed on adjacent frames in the continuous video frames to obtain stable video frames aligned with the coordinate system of the previous frame, including: Extract the adjacent previous frame and the current frame from the consecutive video frames; Convert the previous frame and the current frame into grayscale images respectively; The ORB algorithm is used to detect feature points in the static background region of the grayscale image; By combining brute-force matching with Hamming distance, feature point descriptors of the previous frame and the current frame are matched to obtain feature point matching pairs. Cross-validation is used to remove incorrect matching point pairs from the feature point matching pairs and retain high-quality matching point pairs; Using the RANSAC algorithm, a 3×3 second homography matrix is ​​determined based on the coordinates of the high-quality matching point pairs. The second homography matrix Used to describe camera translation, rotation, scaling, and perspective changes from the previous frame image plane to the current frame image plane; Calculate the second homography matrix The inverse matrix is ​​obtained, and the inverse matrix is ​​applied to the current frame for perspective transformation based on the perspective transformation function to generate a stable video frame aligned with the coordinate system of the previous frame.

3. The method according to claim 1, characterized in that, The automatic selection of the optimal speed measurement region (ROI) based on ROI in the stabilized video frames includes: The stabilized video frames are divided into a preset number of grids, with each grid serving as a candidate ROI; For each candidate ROI, a texture feature evaluation index and a motion feature evaluation index are calculated. The texture feature evaluation index is calculated by using the Sobel operator or the Scharr operator to calculate the gradient magnitude of all pixels in the candidate ROI, and then averaging all gradient magnitudes to obtain the average gradient magnitude, which is used as the texture feature evaluation index. The motion feature evaluation index is calculated by using the Lucas-Kanade sparse optical flow algorithm to calculate the motion vector of the feature points in the candidate ROI, and then determining the consistency score of the motion vector in terms of direction and magnitude, which is used as the motion feature evaluation index. The texture feature evaluation index and motion feature evaluation index of each candidate ROI are weighted and summed to obtain the comprehensive score of each candidate ROI; The candidate ROI with the highest comprehensive score is selected as the optimal speed measurement region ROI in the stabilized video frame, and the step of determining the optimal speed measurement region ROI is performed for each stabilized video frame.

4. The method according to claim 3, characterized in that, The process of calculating motion vectors for feature points within candidate ROIs using the Lucas-Kanade sparse optical flow algorithm, and then determining the consistency score of the motion vectors in both direction and magnitude, includes: In the two consecutive stabilized video frames corresponding to the candidate ROI, feature points are selected in the previous frame using the Harris corner detection algorithm. For each selected feature point, define a local window of a preset size centered on the feature point; Within the local window, the spatial gradient of the image in the x-direction, the spatial gradient in the y-direction, and the temporal gradient between two frames are calculated respectively to construct the fundamental optical flow equation; the fundamental optical flow equation satisfies the following formula: in, The optical flow component in the x-direction. For the optical flow component in the y-direction, The spatial gradient of the image in the x-direction. This represents the spatial gradient of the image in the y-direction. The gradient of the image over time; Based on the fundamental equations of optical flow, a linear equation is constructed for each pixel within the local window to form an overdetermined system of equations. The least squares method is then used to solve the overdetermined system of equations to obtain the motion vector (u,v) of each feature point in the x and y directions. Calculate the average direction and average magnitude of all motion vectors, and then calculate the standard deviation of the direction and the coefficient of variation of the magnitude respectively; The consistency score is calculated by weighted summation of the directional standard deviation and the magnitude coefficient of variation.

5. The method according to claim 1, characterized in that, The adaptive texture enhancement processing of the optimal velocity measurement region ROI based on the CLAHE algorithm to generate an enhanced ROI includes: The optimal velocity measurement area (ROI) is divided into multiple non-overlapping rectangular sub-blocks; the size of the rectangular sub-blocks is preset based on the ROI size and the water surface texture scale. Calculate the pixel grayscale histogram for each sub-block, set a clipping threshold for each histogram, and evenly distribute the sum of the grayscale counts of any grayscale level in the histogram that exceed the clipping threshold to all grayscale levels in the entire histogram. Perform standard histogram equalization on each sub-block of the adjusted histogram to obtain the grayscale mapping function for each sub-block; The grayscale mapping function of each sub-block is bilinearly interpolated to obtain the enhanced ROI.

6. The method according to claim 1, characterized in that, Based on the enhanced ROI, a surface texture index (STI) is generated. The principal direction of the STI is detected, and spatial calibration from the image plane to the object plane is performed using the first homography matrix. The surface velocity of the water body is then calculated, including: A velocity measuring line with a single pixel width and a preset length of n is set in the water flow direction of the enhanced ROI; For an enhanced ROI image sequence of m consecutively acquired frames, an n×m STI is synthesized with the spatial location of water flow as the x-axis and the acquisition time as the t-axis, and the STI is then denoised. The gray-level gradient tensor method is used to determine the eigenvalues ​​and eigenvectors of the local gradient covariance matrix of STI, and the main texture direction of STI is detected. The first homography matrix is ​​determined based on the perspective transformation relationship between 2D planes. The pixel displacement is converted into physical distance by accumulating the object-image scale factor of the related pixels of the velocity line in the image plane. The surface velocity of the water body is calculated by combining the main direction of the texture with the calibrated physical distance.

7. The method according to claim 6, characterized in that, The first homography matrix is ​​determined based on the perspective transformation relationship between 2D planes. The pixel displacement is converted into physical distance by accumulating the object-image scale factors of the relevant pixels of the velocity measurement lines within the image plane, including: The first homography matrix is ​​determined based on the perspective transformation relationship between 2D planes; the transformation relationship satisfies the following formula: in, Let x and y be the first homography matrix, and x and y be the two-dimensional coordinates of a pixel in the image plane. , Let be the two-dimensional coordinates in the object plane corresponding to the image plane, and s be a non-zero scaling factor. (i, j=1,2,3) are the elements of the first homography matrix; For pixel displacement on the speed measuring line, after converting the pixel coordinates into physical coordinates on the object plane using the first homography matrix, the physical distances between adjacent pixels on the object plane are calculated and accumulated to obtain the total physical distance in the x and y directions; the total physical distance in the x and y directions satisfies the following formula: in, This represents the total physical distance in the x-direction. This represents the total physical distance in the y-direction. Let x be the object scale factor of the pixel on the velocity measurement line in the x-direction. Let be the object scale factor of the pixel on the velocity measurement line in the y-direction. This is the starting pixel index of the velocity line in the x-direction of the image plane. This is the index of the terminating pixel of the velocity line in the x-direction of the image plane. This is the starting pixel index of the velocity line in the y-direction of the image plane. This is the index of the terminating pixel of the velocity line in the y-direction of the image plane.

8. The method according to claim 7, characterized in that, Combining the principal direction of the texture with the calibrated physical distance, the calculated surface velocity of the water body satisfies the following formula: in, The surface velocity of the water body. for , The composite physical distance along the main direction of the texture, where T is the water flow motion. The corresponding time, Let S be the pixel displacement of the water flow along the principal direction of the texture in the image coordinate system, where S is the spatial calibration coefficient and M is the number of video frames corresponding to the water flow movement of r pixels. The time interval for a single frame of an image. The main direction of the STI texture. This represents the pixel rate of the water flow in the image.

9. The method according to claim 1, characterized in that, The method of constructing a dense optical flow field using a CNN encoder-decoder structure and determining the surface flow velocity of the water body based on the dense optical flow field includes: Two consecutive frames of enhanced ROI images are acquired as input image pairs and input into a CNN model with an encoder-decoder structure; the CNN model includes a feature extraction encoder, a correlation layer, and an optical flow estimation decoder. The feature extraction encoder extracts multi-scale depth feature maps from low to high levels from the input image pairs. The correlation layer is used to calculate the pixel-wise correlation between the feature maps of two frames and generate a 4D correlation volume that encodes pixel displacement information. Based on the optical flow estimation decoder, deconvolution upsampling and convolution operations are performed on multi-scale feature maps and 4D correlation volumes to gradually optimize and obtain a dense optical flow field that matches the size of the enhanced ROI; The pixel displacement vector in the dense optical flow field is multiplied by the spatial calibration coefficient and the video frame rate to convert it into a velocity vector. The velocity vector is subjected to median filtering to obtain the surface velocity of the water body.

10. A video flow measurement system for implementing the method according to any one of claims 1-9, characterized in that, The system includes: a video acquisition module, a preprocessing module, a speed measurement module, and an output module; The video acquisition module is used to acquire continuous video frames of the surface of the water body to be tested; The preprocessing module includes a motion compensation submodule, an automatic ROI selection submodule, and an image enhancement submodule. The motion compensation submodule performs motion compensation processing on adjacent frames in the continuous video frames to obtain stable video frames aligned with the coordinate system of the previous frame. The automatic ROI selection submodule automatically selects and filters the optimal speed measurement region (ROI) in the stable video frames based on ROI selection. The image enhancement submodule performs adaptive texture enhancement processing on the optimal speed measurement region (ROI) based on the CLAHE algorithm to generate an enhanced ROI. The velocity measurement module is used to input the enhanced ROI into the water surface velocity calculation model to determine the water surface velocity, and to perform weighted fusion of the water surface velocity output by the STI analysis branch and the CNN optical flow estimation branch to obtain the final water surface velocity. The output module is used to receive the water surface velocity results output by the velocity measurement module and display the water surface velocity data on the display screen.

Images