A river surface water flow velocity detection method based on video stream image processing

Abstract

The application discloses a river surface water flow velocity detection method based on video stream image processing, and comprises the following steps: acquiring a water flow video image; pre-processing the image; water body segmentation of the image; inputting the pre-processed image into a pre-trained convolutional neural network for feature extraction; obtaining representative tracers in a window according to the features and the segmentation result; establishing a displacement mapping relationship between the image and pixels by using an optical flow estimation model; filtering the optical flow field by using the tracers, and clustering the filtered vectors by using a clustering method to obtain a river mainstream direction; constructing a depth estimation dataset, and establishing a mapping relationship between the image and the depth by using the dataset; and obtaining the river surface water flow velocity according to the displacement mapping relationship, the river mainstream direction and the mapping relationship between the image and the depth. The application can effectively detect the river surface water flow velocity of a target region.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a method for detecting the surface water flow velocity of a river based on video stream image processing. Background Technology

[0002] River surface flow velocity is an important monitoring indicator. Both basic hydrological research and water conservancy engineering applications require measured river data as a foundation. Traditional water conservancy research directions such as river channel diversion, slope stability, and floodplain inundation, as well as emerging eco-hydrological engineering projects such as aquatic habitat protection, fishway construction, and large-scale relocation, all require high-precision data on several aspects such as flow velocity distribution and flow rate change.

[0003] Traditional water flow velocity measurement mainly uses contact methods such as propeller velocimeters, current meters, flumes, and weirs to directly or indirectly measure water flow velocity. However, these instruments require manual operation at fixed times and locations, resulting in high labor costs. Furthermore, the suddenness of floods during severe weather makes it difficult to guarantee the safety of flow velocity measurement personnel, which is detrimental to the implementation of modern flood control work.

[0004] Current methods for measuring water flow velocity include non-contact methods such as buoy tracking and acoustic Doppler current profilers. These methods offer high accuracy and can approximate the actual water flow velocity, but they also have limitations. Buoy tracking requires casting a buoy into the water and using a camera to track the pixel displacement of the buoy in the image. This is combined with parameters such as the distance between the camera and the river surface, the height difference between the camera and the river surface, the camera angle, and camera intrinsic parameters to calculate the surface velocity. Buoys often require secondary retrieval, and some are made of disposable biodegradable materials, which increases their cost, and measuring various parameters is time-consuming and labor-intensive. Acoustic Doppler current profilers rely on the Doppler effect, using the frequency changes of sound waves at different water flow velocities to estimate the water flow velocity. They offer high accuracy, but as high-precision devices, these current profilers are expensive and have poor maintainability. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a method for detecting the surface water flow velocity in a river channel based on video stream image processing. The method includes the following steps:

[0006] S1. Acquire video images of river flow;

[0007] S2. Preprocess the video image to obtain a preprocessed image; perform water segmentation on the video image to obtain water segmentation results;

[0008] S3. Input the preprocessed image into a pre-trained convolutional neural network for feature extraction to obtain features of different dimensions;

[0009] S4. Based on the characteristics of different dimensions and the water body segmentation results, a sliding window is used to calculate the cross-correlation, and max pooling is used to obtain representative tracers within the window.

[0010] S5. Create an optical flow dataset based on the video images, and use an optical flow estimation model to establish a displacement mapping relationship between images and pixels on the optical flow dataset;

[0011] S6. Use tracers to filter the optical flow field into vectors, and use clustering methods to cluster the filtered vectors. The average direction of the vectors in the category with the most vectors is taken as the main direction of the river channel.

[0012] S7. Construct a depth estimation dataset based on the video images and radar depth measurement data, and establish a mapping relationship between images and depths on the depth estimation dataset using an image depth estimation model;

[0013] S8. Based on the displacement mapping relationship between the image and pixels, the main direction of the river channel, and the mapping relationship between the image and depth, the surface water flow velocity of the river channel is obtained by using a coordinate projection transformation method and a mean method.

[0014] A storage device that stores instructions and data for implementing a method for detecting the surface water flow velocity of a river based on video stream image processing.

[0015] A river surface water flow velocity detection device based on video stream image processing includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement a river surface water flow velocity detection method based on video stream image processing.

[0016] The beneficial effects provided by this invention are as follows: This invention uses ResNet for depth extraction to obtain feature vectors of image pixels; a sliding window is used on these feature vectors to calculate mutual information to obtain tracers; a loss function combining EPE loss, velocity consistency loss, and brightness consistency loss is proposed to train the optical flow estimation model (RAFT network); clustering methods are used to obtain the main flow direction of the river channel; and Monodepth is used to obtain the distance between the automatically measured camera and the water surface. These methods not only improve the automation level of water flow velocity detection but also achieve high detection accuracy. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention;

[0018] Figure 2 This is a schematic diagram illustrating the transformation between image coordinates and real-world coordinates;

[0019] Figure 3This is a schematic diagram of the hardware device of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0021] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the method flow of the present invention;

[0022] This invention provides a method for detecting the surface water flow velocity of a river based on video stream image processing, comprising:

[0023] S1. Acquire video images of river flow;

[0024] S2. Preprocess the video image to obtain a preprocessed image; perform water segmentation on the video image to obtain water segmentation results;

[0025] It should be noted that in step S2, the image preprocessing specifically involves: applying mean filtering and wavelet denoising to the image, and then using adaptive histogram equalization to reduce the differences between different bright areas, resulting in the preprocessed image.

[0026] S3. Input the preprocessed image into a pre-trained convolutional neural network for feature extraction to obtain features of different dimensions;

[0027] It should be noted that step S3 specifically involves: inputting the preprocessed image into the ResNet residual network for convolution, and using diagonal maximum value for pooling to obtain the feature vector of each frame pixel of the image.

[0028] S4. Based on the characteristics of different dimensions and the water body segmentation results, a sliding window is used to calculate the cross-correlation, and max pooling is used to obtain representative tracers within the window.

[0029] As one embodiment, this invention uses a Residual Network (ResNet) to convolve the first frame of the image and performs pooling using the diagonal maximum value to obtain the feature vector of each pixel in the image. It defines the image at time t as imgt, the image at time t+1 as imgt+1, and defines the window size w and stride, starting from img. t Starting from the top left corner, the feature vector of each pixel within the window is compared with the image. t+1 The correlation of feature vectors of all pixels within the same window is calculated using a dot product formula.

[0030]

[0031] Max pooling is performed on each window to obtain the maximum dot product value within the window. The image containing the maximum dot product value within the window is then mapped. t The coordinates of the pixels in the vector are used as the tracer corresponding to the window. i represents the i-th dimension of the vector, and n represents the length of the vector; a i Represents the i-th dimension of vector a; b i This represents the i-th dimension of the vector b.

[0032] S5. Create an optical flow dataset based on the video images, and use an optical flow estimation model to establish a displacement mapping relationship between images and pixels on the optical flow dataset;

[0033] In step S5, the optical flow estimation model is optimized during training by using the final loss function obtained by weighting EPE loss, velocity consistency loss and brightness consistency loss.

[0034] Specifically, an optical flow dataset is created based on consecutive frames of water flow video images. A computer image processing-based optical flow estimation model (RAFT-Recurrent All-Pairs Field Transforms) is trained on this dataset to establish a mapping relationship between images and pixel displacements. When two consecutive frames are input into the model, the model outputs the displacement distance of pixels from the previous frame in the second frame. The model's loss function includes:

[0035] 1. Endpoint Error (EPE) Loss: EPE loss measures the Euclidean distance between the predicted optical flow vector and the true optical flow vector. This loss is used to measure the accuracy of optical flow. The formula for EPE loss is as follows:

[0036]

[0037] Among them, u i It is the predicted optical flow vector, u gt,i It is the true optical flow vector, and N is the number of samples.

[0038] 2. Velocity Consistency Loss: Water flow in a river exhibits relatively significant velocity consistency. Therefore, velocity consistency loss measures the degree of dispersion between predicted optical flow vectors, encouraging predicted optical flows to have as similar a velocity as possible. The formula for velocity consistency loss is as follows:

[0039]

[0040] Where, x i It is the predicted optical flow vector. is the average Manhattan distance of the predicted optical flow vector, and N is the number of samples.

[0041] 3. Brightness Consistency Loss: Brightness consistency loss measures the degree to which the optical flow result preserves image brightness. It is based on the assumption of image brightness consistency, encouraging the pixel values ​​after optical flow transformation to remain consistent with the corresponding reference frame pixel values. The formula for brightness consistency loss is as follows:

[0042]

[0043] Among them, I t (p i ) is the pixel position p at time t. i The pixel value at I t+1 (p i +u i ) represents the pixel value at time t in the image at time t+1, based on the displacement result of the predicted optical flow, where N is the number of samples, and u i It is the predicted optical flow vector.

[0044] These losses are weighted and summed to obtain the final loss function. Backpropagation and optimization algorithms are then used to update the model parameters, minimizing the difference between the predicted and actual optical flow. Based on this loss function, a RAFT network is trained. Ultimately, by inputting two image frames into the model, the model outputs the absolute values ​​of the displacements of each pixel in the first frame in the x and y directions of the second frame.

[0045] The optical flow is extracted from the image of the target area using the model, and finally the optical flow vector field is obtained.

[0046] S6. Use tracers to filter the optical flow field into vectors, and use clustering methods to cluster the filtered vectors. The average direction of the vectors in the category with the most vectors is taken as the main direction of the river channel.

[0047] Specifically, the x and y coordinates of the tracer obtained in step S4 are used to extract the corresponding tracer optical flow (i.e., the displacement vector of the tracer in the second frame relative to the first frame) from the optical flow vector field obtained in step S5. Based on these optical flows, k-means clustering is performed, and the class with the most in-class vectors in the clustering results is selected as the main class. The optical flow in the main class represents the flow direction and magnitude of the water flow on the river surface.

[0048] S7. Construct a depth estimation dataset based on the video images and radar depth measurement data, and establish a mapping relationship between images and depths on the depth estimation dataset using an image depth estimation model;

[0049] Specifically, a depth estimation dataset is created using images and actual radar bathymetry data. This dataset is then trained using the Monodepth monocular bathymetry model to establish a mapping relationship between images and depth. The first frame from the aforementioned image set is input into Monodepth to obtain the image's bathymetry field, i.e., the depth map. The corresponding depth value in the depth map is selected from the tracer coordinates in the aforementioned main category.

[0050] S8. Based on the displacement mapping relationship between the image and pixels, the main direction of the river channel, and the mapping relationship between the image and depth, the surface water flow velocity of the river channel is obtained by using a coordinate projection transformation method and a mean method.

[0051] Perspective projection coordinate system as follows Figure 2 A camera coordinate system O was established. c -X c Y c Z c And the image coordinate system O-XY. In the figure, M(x) c ,y c ,z c Let O be a point in the camera coordinate system, and let N(x,y) be its projection point in the image coordinate system. c For the camera's optical center, Z c Let f be the camera's optical axis and f be the focal length. The following equation can be derived from the similarity of triangles.

[0052]

[0053] The transformation relationship between the image coordinate system and the pixel coordinate system can be obtained from the following formula.

[0054]

[0055] Where dx and dy represent the length units occupied by a pixel in the horizontal and vertical directions on the imaging plane, respectively, and are used to connect the image coordinate system and the real coordinate system.

[0056] The depth value (Z) obtained from the aforementioned steps c By combining the camera intrinsic parameters (f) and the corresponding pixel displacement optical flow (u,v), the velocity of the tracer in the real world can be calculated. By averaging these tracer velocities, the average velocity and direction of the water flow on the river surface can be obtained.

[0057] Please see Figure 3 , Figure 3 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: a river surface water flow velocity detection device 401 based on video stream image processing, a processor 402, and a storage device 403.

[0058] A fossil size measurement and contour restoration device 401 based on image processing: The river surface water flow velocity detection device 401 based on video stream image processing implements the river surface water flow velocity detection method based on video stream image processing.

[0059] Processor 402: The processor 402 loads and executes the instructions and data in the storage device 403 to implement the method for detecting the surface water flow velocity of a river based on video stream image processing.

[0060] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the method for detecting the surface water flow velocity of a river based on video stream image processing.

[0061] The beneficial effects of this invention are as follows: This invention uses ResNet for depth extraction to obtain feature vectors of image pixels; a sliding window is used on these feature vectors to calculate mutual information to obtain tracers; a loss function combining EPE loss, velocity consistency loss, and brightness consistency loss is proposed to train the optical flow estimation model (RAFT network); clustering methods are used to obtain the main flow direction of the river channel; and Monodepth is used to obtain the distance between the automatically measured camera and the water surface. These methods not only improve the automation level of water flow velocity detection but also achieve high detection accuracy.

[0062] Overall:

[0063] 1. This invention enables the acquisition of river surface water flow velocity simply by inputting a video of the river surface water flow, thereby reducing the cost of monitoring river surface water flow velocity and improving the efficiency of modern flood control work.

[0064] 2. This invention improves the model's accuracy in identifying the main water flow direction in the river by using feature point extraction and improves the accuracy of optical flow extraction by using deep learning networks, thus effectively detecting the surface water flow velocity in the target area.

[0065] 3. This method can be deployed on devices such as surveillance cameras, which can reduce the hardware cost of water flow velocity measurement and the risk of human measurement, while also helping the water conservancy and hydropower industry to carry out real-time monitoring and information processing of river flow.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A river surface water flow velocity detection method based on video stream image processing, characterized in that: The method comprises the following steps: S1, acquiring a river water flow video image; S2, preprocessing the video image to obtain a preprocessed image; S3, performing water body segmentation on the video image to obtain a water body segmentation result; S4, inputting the preprocessed image into a pre-trained convolutional neural network for feature extraction to obtain features of different dimensions; S5, calculating cross-correlation using a sliding window according to the features of different dimensions and the water body segmentation result, and obtaining a representative tracer in the window using max pooling; S6, preparing an optical flow dataset according to the video image, and establishing a displacement mapping relationship between images and pixels on the optical flow dataset using an optical flow estimation model; S7, performing vector screening on the optical flow field using the tracer, and clustering the screened vectors using a clustering method, and taking the average direction of the vector class with the most vectors as the river mainstream direction; S8, constructing a depth estimation dataset according to the video image and the radar sounding data, and establishing a mapping relationship between images and depth on the depth estimation dataset using an image depth estimation model; In step S5, the optical flow estimation model is optimized by the final loss function obtained by weighting the EPE loss, the velocity consistency loss and the brightness consistency loss during training; The EPE loss measures the Euclidean distance between the predicted optical flow vector and the true optical flow vector; The velocity consistency loss measures the dispersion degree of the predicted optical flow vector; The brightness consistency loss is used to measure the degree of keeping the image brightness by the optical flow result, which is based on the brightness consistency assumption of the image and encourages the pixel value after the optical flow transformation to keep consistent with the corresponding reference frame pixel value. In step S2, the image preprocessing specifically comprises: image denoising using mean filtering and wavelet denoising, and then using adaptive histogram equalization to reduce the difference between different bright areas to obtain the preprocessed image.

2. The river surface water flow velocity detection method based on video stream image processing according to claim 1, characterized in that: Step S3 specifically comprises: inputting the preprocessed image into a residual network ResNet for convolution, and using diagonal maximum value for pooling to obtain a feature vector of each frame of pixels of the image.

3. The river surface water flow velocity detection method based on video stream image processing according to claim 1, characterized in that: Step S4 specifically comprises:

4. The river surface water flow velocity detection method based on video stream image processing according to claim 3, characterized in that: In step S5, the EPE loss is as follows: Define the image at time t as img t, The image at time t+1 as img t+1 Define the window size w, and the step size, from img t Starting from the top left corner, calculate the cross-correlation of the feature vector of each pixel in the window with the feature vector of all pixels in the same window in img t+1 The cross-correlation calculation uses the dot product The maximum pooling is performed on each window to obtain a maximum dot product value in the window, and coordinates of a pixel in the imgt where the maximum dot product value is located are taken as a tracer corresponding to the window, where i represents the i-th dimension in the vector, and n represents the length of the vector; a i represents the i-th dimension data in the a vector; b i represents the i-th dimension data in the b vector.

5. The river surface water flow velocity detection method based on video stream image processing according to claim 1, characterized in that: In step S5, the velocity consistency loss is as follows: wherein, is a predicted optical flow vector, is a true optical flow vector, is a number of samples.

6. The river surface water flow velocity detection method based on video stream image processing according to claim 1, characterized in that: In step S5, the brightness consistency loss is as follows: wherein, is the predicted optical flow vector, is the average of the Manhattan distance of the predicted optical flow vector, is the number of samples.

7. The river surface water flow velocity detection method based on video stream image processing according to claim 1, characterized in that: The storage device stores instructions and data for implementing the river surface water flow velocity detection method based on video stream image processing according to any one of claims 1-7. wherein is the pixel value at the pixel position at time instant , is the pixel value at the pixel position at time instant according to the displacement result of the predicted optical flow, is the number of samples, is the predicted optical flow vector.

8. A storage device, characterized by: It comprises:

9. A river surface water flow velocity detection device based on video stream image processing, characterized in that: A processor and a storage device; the processor loads and executes the instructions and data in the storage device to implement the river surface water flow velocity detection method based on video stream image processing according to any one of claims 1-7. ​

Images