A water surface flow velocity detection method based on deep learning optical flow estimation
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2025-05-12
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies for water surface velocity detection suffer from high hardware costs, difficult maintenance, and measurement accuracy greatly affected by the environment. Traditional image-based methods are unstable in complex flow conditions and uneven lighting, while deep learning-based optical flow estimation models are not adaptable to complex environments.
An improved deep learning optical flow estimation model, RAFT, is adopted. The correlation between adjacent frames is calculated through a feature extraction network with a multi-scale feature fusion structure and patch convolution. The KPA attention mechanism is introduced into the iterative update module to construct a multi-scale correlation pyramid, thereby improving the model's recognition accuracy in uneven lighting and complex motion environments.
It achieves low-cost and accurate surface velocity detection, applicable to clean rivers without tracers, enhances the model's anti-interference ability and detection accuracy in complex environments, and alleviates feature confusion problems in uneven illumination and complex motion scenarios.
Smart Images
Figure CN120543595B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to optical flow estimation tasks in the field of computer vision technology, and more particularly to a method for detecting water surface velocity based on deep learning optical flow estimation. Background Technology
[0002] Currently, the field of water surface velocity detection mainly relies on hardware devices such as current meters, which determine the water surface velocity by comparing the frequency difference between transmitted and reflected signals (microwaves). This method of velocity measurement requires significant manpower and maintenance costs, and its measurement accuracy is greatly affected by environmental conditions and equipment parameters.
[0003] With the rapid development of intelligent video processing technology, non-contact image-based velocimetry offers advantages over flow meter velocimetry in terms of low cost and high efficiency. Currently, image-based velocimetry methods for water surface velocity detection include Particle Image Velocimetry (PIV), Space-Time Image Velocimetry (STIV), and Optical Flow Velocimetry (OFV). PIV estimates water surface velocity by tracking tracer particles on the water surface, but it is not suitable for clean water flows without tracers. STIV obtains the temporal variation texture of the water flow by synthesizing spatiotemporal images and uses the direction of this texture to calculate the water surface velocity; however, this method has poor noise resistance and poor detection performance in complex flow conditions. OFV, based on the traditional optical flow method, estimates water flow motion through the principle of brightness consistency. Although this method does not rely on tracers, traditional OFV is sensitive to noise and its detection performance is unstable. Deep learning-based OFV methods, however, greatly alleviate these problems by utilizing feature extraction and iterative update networks in deep learning.
[0004] RAFT is a high-efficiency deep learning-based optical flow estimation model that combines feature extraction and volumetric approaches from other optical flow models with an iterative update unit to refine the optical flow field. The model comprises a feature extraction module, a visual similarity pyramid, and an iterative update module. The feature extraction module extracts spatial features from adjacent frames and contextual features from a single frame. The visual similarity pyramid is the multi-scale result of calculating the correlation between spatial feature maps of adjacent frames. The iterative update module fuses and performs inter-calculation with the feature information from the contextual feature map, the multi-scale correlation information from the correlation pyramid, and the initial optical flow information to obtain pixel-level motion estimation results. The iterative update unit then uses an RNN module to continuously update and refine this motion estimation prediction result.
[0005] However, the direct application of the RAFT model to water surface velocity suffers from significant inconsistencies in detection across different light intensities within the same watershed when dealing with uneven illumination, resulting in substantial discrepancies in detection results. Furthermore, in complex motion scenarios such as cluttered backgrounds and interference from other moving objects, it is prone to incorrectly including non-target detection areas in the detection range. To improve the model's detection accuracy and adaptability to complex environments, the RAFT feature extraction network is modified into a feature extraction network with a multi-scale feature fusion structure. Patch convolution calculations are introduced into the visual similarity pyramid module, and a KPA attention mechanism is incorporated into the iterative update module. Summary of the Invention
[0006] The technical problem this invention aims to solve is to propose a precise, efficient, and low-cost method for detecting water surface velocity. Traditional current meters are limited by instrumentation and environmental factors, while the PIV method in image-based water surface velocity measurement relies on surface tracers. The STIV method is not suitable for complex flow regimes such as turbulence and eddies. The traditional OFV method is significantly affected by noise, and the basic OFV method combined with deep learning needs further improvement in adapting to complex environmental interferences such as uneven lighting and complex motion environments.
[0007] To address these issues, this invention proposes a water surface velocity detection method based on deep learning optical flow estimation. It optimizes the design of the RAFT deep learning optical flow estimation model by using a multi-scale fusion feature extraction network, calculating the multi-scale correlation between adjacent frames using patch convolution, and introducing the KPA attention mechanism to improve the model's recognition accuracy in complex interference environments such as uneven illumination and complex motion environments.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] A method for detecting water surface velocity based on deep learning optical flow estimation includes the following steps:
[0010] Step 1: Construct an optical flow estimation network for water surface velocity;
[0011] The deep learning optical flow estimation network for detecting water surface velocity uses RAFT as the baseline model, which includes a feature extraction module, a visual similarity pyramid module, and an iterative update module. To improve the model's detection accuracy and adaptability to complex environments, the feature extraction network is improved to a feature extraction network with a multi-scale feature fusion structure. Patch convolution calculation is introduced in the visual similarity pyramid module, and the KPA attention mechanism is introduced in the iterative update module.
[0012] The feature extraction network with a multi-scale feature fusion structure is mainly used to extract large-scale features such as water areas and small-scale features such as water ripples from consecutive frames. Its structure is shown in Table 1. The convolutional layer consists of two consecutive convolutional blocks, the downsampling layer consists of one max-pooling layer and one convolutional layer, and the upsampling layer takes two feature maps of different sizes as input and consists of upsampling operations, concatenation operations, and convolutional layers. This feature extraction network ultimately outputs a feature map pyramid, which consists of four feature maps of different sizes but with the same number of channels.
[0013] Table 1 Feature Extraction Network Structure
[0014] Network layer Input dimensions Output size Input Channel Output Channel Convolutional layer 1 640×360 640×360 3 64 Downsampling layer 1 640×360 320×180 64 64 Downsampling layer 2 320×180 160×90 64 128 Downsampling layer 3 160×90 80×45 128 128 Downsampling layer 4 80×45 40×22 128 256 downsampling layer 5 40×22 20×11 256 256 downsampling layer 6 20×11 10×5 256 512 Convolutional layer 7 10×5 10×5 512 256 Upsampling layer 1 [10×5,20×11] 20×11 [256,256] 256 Upsampling layer 2 [20×11,40×22] 40×22 [256,256] 256 Upsampling layer 3 [40×22,80×45] 80×45 [256,128] 256
[0015] This invention uses patch convolution to calculate the correlation between two images in consecutive frames. All layers of the feature map pyramid obtained by the feature extraction module of the subsequent frame image are unfolded into 3×3 convolution kernels (patches). The unfolded result of each layer is convolved with the last layer of the feature map pyramid obtained by the feature extraction module of the previous frame. The correlation between each position in the previous frame and the corresponding position in the subsequent frame is obtained by using the convolution operation.
[0016] The iterative update module is used to iteratively refine the optical flow field. This module takes the last layer of the feature map pyramid obtained from the feature extraction module of the previous frame as the context feature map, and inputs the query results of the visual similarity pyramid, as well as the possible initial optical flow field. It iteratively optimizes the predicted value of the optical flow field based on the context features and the correlation information between adjacent frames. The iterative operation is implemented using the GRU unit of the RNN module. The GRU unit considers the current input and the output of the previous time step, which is beneficial for continuous iterative refinement. This invention introduces the KPA attention mechanism into the iterative update module, updating the weights of the context feature map before GRU iteration. By utilizing the local similarity of context features and considering the dynamic relationship between context features and correlation, the accuracy of optical flow estimation can be improved.
[0017] Step 2: Construct an optical flow dataset for water surface velocity estimation;
[0018] Deep learning-based optical flow estimation models require image pairs and their optical flow annotations as inputs for training. This invention uses a dense, non-rigid optical flow generation algorithm based on real video to construct the dataset, comprising five parts: region extraction, image matching, image deformation and optical flow generation, image warping, and background replacement. In region extraction, this invention uses annotated masks to extract water flow regions, and adds more than 100 background images in the background replacement. Ultimately, 887 image pairs and their optical flow annotation files are generated, with the optical flow annotation files in '.flo' format, containing the horizontal and vertical motion data for each pixel.
[0019] Step 3: Train the optical flow estimation network model for water surface velocity;
[0020] The optical flow estimation network model for water surface velocity was trained using the training set from a self-built optical flow dataset of water flow scenes. The input image size was adjusted to 640×360 to match the aspect ratio of 1920×1080 resolution images. Images were randomly flipped before being input into the training model for enhancement. The learning rate was set to 0.00002, the number of iterations to 100,000, the batch size to 4, the number of iteration units to 12, and the L1 loss function was used.
[0021] Step 4: Test the optical flow estimation network model for water surface velocity;
[0022] The test video is segmented into frames, with each frame fed into a trained water surface velocity detection model. The network model outputs a 2D tensor, representing the lateral and longitudinal displacements at each pixel. The displacement at each pixel is calculated using vector calculation formulas. Multiplying this displacement by the video's FPS (frames per second) and the distance ratio yields the flow velocity at each pixel. Comparing this flow velocity with the standard flow velocity obtained from a radar velocity detector reveals the model error.
[0023] Compared with the prior art, the present invention has the following advantages:
[0024] This method abandons velocimeters and uses image recognition to detect flow velocity, effectively solving the problems of high cost, difficult maintenance, and limited detection range associated with hardware-based flow velocity identification. Furthermore, it no longer relies on surface tracers for flow velocity detection, but instead uses details such as surface ripples and changes in optical flow to determine flow velocity, making it suitable for clean rivers without tracers.
[0025] Based on the RAFT deep learning optical flow estimation model as the baseline, this paper enhances the model's generalization performance for real-world application scenarios by building a self-developed dataset. Furthermore, it strengthens the model's robustness against non-river targets and mitigates feature confusion in complex motion scenarios by constructing a multi-scale correlation pyramid using a multi-scale fusion feature extraction network and patch convolutions. In addition, using patch convolutions to calculate correlations improves the model's perception of neighborhoods and alleviates intra-domain differences caused by uneven illumination. Moreover, the introduction of the KPA attention mechanism in the iterative update module effectively improves the accuracy of optical flow prediction and alleviates the problem of zero optical flow values in some shadow areas. Attached Figure Description
[0026] Figure 1 This is a flowchart illustrating the water surface velocity detection method based on deep learning optical flow estimation provided by the present invention.
[0027] Figure 2 This is a schematic diagram of the overall structure of the water surface velocity detection model based on deep learning optical flow estimation provided by the present invention.
[0028] Figure 3 This is a schematic diagram of the feature extraction network structure using multi-scale feature fusion in this invention. Detailed Implementation
[0029] This invention primarily implements a water surface velocity detection method based on deep learning optical flow estimation. The specific method employed in this invention will be described in detail below with reference to the accompanying drawings.
[0030] Specifically, the process of the water surface velocity detection method based on deep learning optical flow estimation is as follows: Figure 1 As shown, the process includes the following steps: S1: Construct an optical flow estimation network for water surface velocity. S2: Construct an optical flow dataset for water surface velocity estimation. S3: Train the optical flow estimation network model for water surface velocity. S4: Test the optical flow estimation network model for water surface velocity.
[0031] For S1: Construct an optical flow estimation network for water surface velocity.
[0032] In this invention, the optical flow estimation network structure for water surface velocity includes a feature extraction module, a correlation pyramid construction module, and an iterative update module. The overall network structure diagram is shown below. Figure 2 As shown below. Detailed explanations will follow.
[0033] Feature Extraction Module: The feature extraction module of the optical flow estimation model greatly alleviates image noise interference by extracting target features. This invention designs a multi-scale fusion feature extraction structure, as shown in Table 1. This structure enables the network to simultaneously extract deep semantic features and shallow detail features. These features are crucial for water surface velocity detection, as the task requires more accurate estimation of optical flow data based on the semantic features of the water surface and detailed features such as water ripples. The feature extraction network structure from shallow to deep is as follows: 1 convolutional layer, 6 downsampling layers, 1 convolutional layer, and 3 upsampling layers. The convolutional layers consist of two consecutive convolutional blocks, the downsampling layers consist of one max pooling layer and one convolutional layer, and the upsampling layers take two feature maps of different sizes as input and consist of upsampling operations, concatenation operations, and convolutional layers. The detailed network structure is as follows: Figure 3 As shown, the feature extraction module of the optical flow estimation model includes a spatial feature extraction module and a contextual feature extraction module. The former is used to extract feature maps of the preceding and following frames and calculate their correlation, while the latter is used to extract the contextual features of the previous frame and input them along with the correlation into the iterative update module for refining the optical flow. For the spatial feature extraction module, the outputs of the last three upsampling layers and one convolutional layer are stacked for output, resulting in a multi-scale feature map pyramid containing four feature layers of increasing size. For the contextual feature extraction module, only the result of the last upsampling layer of the feature extraction network is output.
[0034] Correlation Pyramid Construction Module: The correlation pyramid is used to calculate the multi-scale correlation between feature maps of two consecutive frames. The original RAFT model uses inner product to calculate correlation and performs multiple average pooling operations on the correlation volume to obtain a multi-scale correlation pyramid; or it uses multiplication to calculate the multi-scale correlation volume between the feature map of the previous frame and the multi-scale feature map obtained by multiple average pooling operations of the next frame, thereby constructing the correlation pyramid. However, these methods using inner product or multiplication cannot fully consider spatial features and have limited perception of neighborhood. Therefore, this invention uses patch convolution to calculate the correlation between two images of consecutive frames. All layers of the feature map pyramid obtained by the feature extraction module of the next frame image are unfolded into 3×3 convolution kernels (patches), and the unfolded result of each layer is convolved with the last layer of the feature map pyramid obtained by the feature extraction module of the previous frame. The correlation between each position in the previous frame and the corresponding position in the next frame is obtained by using the convolution operation.
[0035] The iterative update module: Its function is to input the correlated sampled values, initial optical flow, motion features obtained by fusing preceding and following features, and contextual features into multiple cascaded GRU update units, continuously refining the predicted optical flow values iteratively. Before these features are input into the GRU units, this invention employs the KPA attention mechanism to enhance the model's attention to both static and dynamic features of the detected target, thereby improving detection accuracy. Specifically, this attention mechanism splits the contextual features into channels, dividing one part into several local patches and expanding the other part into several 3×3 local patches with a central patch as the unit. The attention weights are updated by calculating the similarity between these two parts; this is called static features. Since the features represented by a single frame are limited, KPA also expands the correlated feature maps of consecutive frames into 3×3 local patches with a central patch as the unit, and calculates the correlation between the expanded content and the attention weights to further update the attention weights; this is called dynamic features. Updating the attention weights by calculating the similarity of local patches improves the accuracy of optical flow estimation.
[0036] For S2: Construct an optical flow dataset for water surface velocity estimation.
[0037] This invention employs an existing algorithm for constructing optical flow datasets. This algorithm is a dense, non-rigid optical flow generation algorithm based on real video, comprising five parts: region extraction, image matching, image deformation and optical flow generation, image warping, and background replacement. In region extraction, this invention uses a labeled mask to extract the water flow region, and adds more than 100 background images for background replacement. In image matching, the DeepMatching method is used. This method divides consecutive frames into several 4×4 grids, calculates the 4×4 correlation using convolution operations for corresponding grids, and constructs 8×8 and 16×16 correlations through operations such as max pooling. Matching between pixel blocks is achieved using top-down maximum correlation value retrieval. Image matching performs dense matching on pixel blocks in the water flow region to obtain quasi-dense matching correspondences. In image warping and optical flow map generation, the ARAP image mesh warping method is used. Guided by the image matching results, the moving object is warped while adhering to physical feasibility. This is achieved by minimizing energy so that the lines connecting the mesh vertices are rotated as much as possible, avoiding translational transformations. The warped results are then interpolated to obtain a dense optical flow field. Due to potential errors introduced by the matching and warping algorithms, the generated optical flow map is only an approximation of the real flow field. Therefore, in image warping and background replacement, image warping techniques are used, guided by the optical flow map. The previous frame is warped, and the background is replaced in both the preceding and following frames to obtain a correct triplet. Finally, 887 image pairs and their optical flow annotation files are generated. The optical flow annotation files are in '.flo' format and contain the horizontal and vertical motion data of each pixel.
[0038] For S3: Train the optical flow estimation network model for water surface velocity.
[0039] The optical flow estimation network model for water surface velocity was trained using the training set from a self-built optical flow dataset of water flow scenes. The input image size was adjusted to 640×360 to match the aspect ratio of 1920×1080 resolution images. Images were randomly flipped before being input into the training model for enhancement. The input dataset used consecutive frame image pairs as input images and the optical flow annotations of the image pairs as ground truth. During training, the model was iteratively optimized using the L1 loss function. The training parameters included a learning rate of 0.00002, 100,000 iterations, a batch size of 4, and 12 update units.
[0040] For S4: Optical flow estimation network model for testing water surface velocity.
[0041] The test video is segmented into frames, with each frame fed into a trained water surface velocity detection model. The network model outputs a 2D tensor, representing the lateral and longitudinal displacements at each pixel. The displacement at each pixel is calculated using vector calculation formulas. Multiplying this displacement by the video's FPS (frames per second) and the distance ratio yields the flow velocity at each pixel. Comparing this flow velocity with the standard flow velocity obtained from a radar velocity detector reveals the model error.
[0042] The above specific embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all similar technical solutions obtained by equivalent substitution or equivalent transformation are within the protection scope of the present invention.
Claims
1. A method for detecting water surface velocity based on deep learning optical flow estimation, characterized in that: Includes the following steps, Step 1: Construct an optical flow estimation network for water surface velocity; The deep learning optical flow estimation network for detecting water surface velocity uses RAFT as the baseline model, which includes a feature extraction module, a visual similarity pyramid module, and an iterative update module. The feature extraction network is improved into a feature extraction network with a multi-scale feature fusion structure. Patch convolution calculation is introduced in the visual similarity pyramid module, and KPA attention mechanism is introduced in the iterative update module. A feature extraction network with a multi-scale feature fusion structure is used to extract large-scale features such as water areas and small-scale features such as water ripples from consecutive frames. The convolutional layer consists of two consecutive convolutional blocks, the downsampling layer consists of one max pooling layer and one convolutional layer, and the upsampling layer takes two feature maps of different sizes as input and consists of upsampling operation, concatenation operation and convolutional layer. The feature extraction network finally outputs a feature map pyramid, which consists of four feature maps of different sizes but the same number of channels. Patch convolution is used to calculate the correlation between two images in consecutive frames. All layers of the feature map pyramid obtained by the feature extraction module of the subsequent frame image are unfolded into 3×3 convolution kernels. The unfolded result of each layer is convolved with the last layer of the feature map pyramid obtained by the feature extraction module of the previous frame. The correlation between each position in the previous frame and the corresponding position in the subsequent frame is obtained by using the convolution operation. The iterative update module is used to iteratively refine the optical flow field. The iterative update module takes the last layer of the feature map pyramid obtained by the feature extraction module of the previous frame image as the context feature map, and takes the query results of the visual similarity pyramid as well as the possible initial optical flow field as input. Based on the context features and the correlation information between adjacent frames, the predicted value of the optical flow field is continuously optimized iteratively. The iterative operation is implemented using the GRU unit of the RNN module. The GRU unit considers the current input and the output of the previous time step, which is beneficial for continuous iterative refinement. The KPA attention mechanism is introduced in the iterative update module. The weights of the context feature map are updated before the GRU iteration. By utilizing the local similarity of the context features and considering the dynamic relationship between the context features and the correlation, the accuracy of optical flow estimation can be improved. The KPA attention mechanism splits the context features into channels, divides one part into several local patches, and expands the other part into several 3×3 local patches with the central patch as the unit. The attention weights are updated by calculating the similarity between the two parts, which is called static features. Since a single frame represents limited features, KPA expands the correlation feature maps of consecutive frames into 3×3 local patches with the center patch as the unit, and calculates the correlation between the expanded content and the attention weights to further update the attention weights, which is called dynamic features; by calculating the similarity of local patches, the attention weights are updated to improve the accuracy of optical flow estimation. Step 2: Construct an optical flow dataset for water surface velocity estimation; The deep learning-based optical flow estimation network model for water surface velocity requires image pairs and their optical flow labels as inputs for training. An existing algorithm for constructing an optical flow dataset is used to construct the dataset. This algorithm is a dense non-rigid optical flow generation algorithm based on real video, which includes five parts: region extraction, image matching, image deformation and optical flow generation, image warping, and background replacement. Step 3: Train the optical flow estimation network model for water surface velocity; The optical flow estimation network model for water surface velocity was trained using the training set in the self-built optical flow dataset of water flow scenes, and the input image size was adjusted to 640×360. Step 4: Test and use the optical flow estimation network model for water surface velocity; The test video is segmented into frames, and each frame is input into a trained optical flow estimation network model for water surface velocity. The output of the optical flow estimation network model for water surface velocity is a 2D tensor, representing the lateral and longitudinal displacements at each pixel. The displacement at each pixel is calculated using vector calculation formulas, and the velocity at each pixel is obtained by multiplying it by the video's frame rate (fps) and distance ratio. This velocity is then compared with the standard velocity obtained by a radar speed gun to determine the error of the optical flow estimation network model for water surface velocity.
2. The water surface velocity detection method based on deep learning optical flow estimation according to claim 1, characterized in that: In step 2, the region extraction was performed using a mask to extract the water flow region. More than 100 background images were added during the background replacement, generating 887 image pairs and their optical flow annotation files. The optical flow annotation files are in '.flo' format and contain the horizontal and vertical motion data of each pixel.
3. The method for detecting water surface velocity based on deep learning optical flow estimation according to claim 1, characterized in that: The images in step 2 are augmented by random flipping before being input into the training model; the learning rate is set to 0.00002, the number of iterations is 100000, the batch size is set to 4, the number of iteration units is set to 12, and the L1 loss function is used.
4. The water surface velocity detection method based on deep learning optical flow estimation according to claim 1, characterized in that: A simulated dataset was constructed using a dense non-rigid optical flow generation algorithm based on real video. Since the images in the dataset are obtained by computer simulation, the results of image deformation and background replacement are used as image pairs.
5. The method for detecting water surface velocity based on deep learning optical flow estimation according to claim 1, characterized in that: A feature extraction network using multi-scale feature fusion is used to extract multi-scale information from consecutive frames and output a multi-scale feature map pyramid to construct a multi-scale correlation pyramid.
6. The method for detecting water surface velocity based on deep learning optical flow estimation according to claim 1, characterized in that: Patch convolution is used to calculate the visual similarity between consecutive frames. This is achieved by unfolding the multi-scale feature map of the subsequent frame into a patch convolution kernel and performing a convolution operation with the previous frame, thereby obtaining a correlation value that preserves spatial features.
7. The method for detecting water surface velocity based on deep learning optical flow estimation according to claim 1, characterized in that: The KPA attention mechanism is introduced into the iterative update module of the optical flow estimation model. By utilizing the contextual local similarity of feature maps and the dynamic correlation between consecutive frames, the weights of the feature maps are reallocated, thereby improving the accuracy of model detection and local perception.
8. The method for detecting water surface velocity based on deep learning optical flow estimation according to claim 1, characterized in that: The method of use is to input a detection video, automatically perform video frame segmentation and model inference, and obtain pixel-level flow rate results based on the model output results according to the speed calculation formula.