A method for predicting surface flow velocity of water flow based on improved RAFT optical flow

Abstract

The application belongs to the technical field of river surface flow velocity measurement, and discloses a method for predicting water flow surface flow velocity based on improved RAFT optical flow. Firstly, in the feature extraction stage, the feature descriptor information output by the SuperPoint network is fused, the representation ability of key features is strengthened, the reliability of feature matching is improved, and problems such as ambiguous optical flow estimation results are solved. Secondly, after correlation calculation, an evaluation module is introduced, the evaluation mode is used to calculate a matching quality evaluation index according to a 4D correlation body and a context feature map, and whether to enable a moving window function is judged according to the size of the evaluation index and a preset threshold value, so that the 4D correlation body is kept or updated. By introducing the moving window function, the problem of false features caused by optical flow discontinuity and noise in the water flow scene is solved, and the evaluation mechanism is constructed to dynamically decide the enabling time of the moving window function, so that the balance between the estimation accuracy and efficiency of the optical flow is realized.

Description

Technical Field

[0001] This invention belongs to the field of river surface velocity measurement and image processing technology, and specifically relates to a method for predicting water surface velocity based on improved RAFT optical flow. Background Technology

[0002] With the continuous maturation of computer vision technology, the application scenarios of deep learning methods in the extraction and analysis of surface water flow characteristics in rivers are constantly expanding. Accurate and efficient measurement of river surface velocity is a key link in hydrological research and engineering practice, and also an important prerequisite for conducting river movement trend prediction and flood risk assessment.

[0003] The RAFT optical flow estimation network is one of the mainstream solutions in the field of deep learning optical flow. It extracts deep features from each pixel in an image to construct a high-dimensional feature representation space, and then uses a cyclic full-pair transform structure to complete the global feature correlation calculation, thereby achieving pixel-level motion vector estimation. This RAFT optical flow estimation network abandons the manual feature design of traditional optical flow algorithms. Relying on an end-to-end learning framework, it can adaptively capture naturally occurring weak features such as foam and ripples on the water surface. Therefore, this method is suitable for tracerless water flow scenarios, and can complete the tracking of water flow motion and velocity inversion on the river surface without the need for manual deployment of tracer particles.

[0004] However, current RAFT optical flow estimation networks have significant limitations in representing features extracted in complex scenes such as weakly textured regions and low-contrast areas. Under conditions of sparse texture and uneven illumination on natural water surfaces, the network struggles to acquire stable and reliable feature information, leading to blurred, offset, or even distorted optical flow estimation results. This directly causes feature trajectory tracking to fail, failing to accurately reflect the true state of water flow. Furthermore, optical flow field breaks often occur in areas with abrupt changes in flow velocity, such as river shallows, eddies, and around obstacles. Even if the actual optical flow is continuous, the lack of historical context can lead to incorrect matching. In actual measurement scenarios, noise factors such as water surface reflection, raindrop interference, and camera shake easily generate false features, which traditional optical flow methods struggle to effectively identify. These noises are often misjudged as real motion information, ultimately resulting in a large number of outliers in the calculated flow velocity data.

[0005] In summary, existing RAFT optical flow estimation methods still have shortcomings in predicting surface velocity of water flow. Summary of the Invention

[0006] To address the issues of optical flow ambiguity, discontinuity, and spurious features in the original RAFT model for measuring surface velocity in water flow, this invention proposes a method for predicting surface velocity in water flow based on improved RAFT optical flow. By improving the RAFT optical flow estimation model, the adaptability of the model to complex water flow scenarios and the accuracy of optical flow estimation are enhanced.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A method for predicting surface velocity of water flow based on improved RAFT optical flow includes the following steps: Step 1. Convert the acquired video of the river surface flow velocity into an image and preprocess the image; Step 2. Build an improved RAFT optical flow estimation model, which includes a bi-branch feature extraction module, a context feature extraction module, a multi-scale fusion module, a similarity calculation module, an evaluation module, and an optical flow iterative calculation module; The dual-branch feature extraction module includes a feature encoding branch and a SuperPoint network branch, which are used to obtain feature maps of the query frame and the reference frame, including the original RAFT features and the SuperPoint features, respectively. The context feature extraction module is used to extract the context feature map of the reference frame; The multi-scale fusion module is used to receive feature maps of query frames and reference frames respectively, and fuse them through a multi-strategy fusion mechanism to obtain new feature maps. Then, a self-attention mechanism is used for optimization to obtain high-quality feature maps. The similarity calculation module performs similarity calculations based on high-quality feature maps to obtain 4D related volumes; The evaluation module is used to calculate the matching quality evaluation index based on the 4D related volume and the context feature map, and to determine whether to enable the moving window function based on the magnitude of the index and the preset threshold, so as to maintain or update the 4D related volume. The optical flow iterative calculation module performs optical flow iterative estimation based on the 4D correlated volume output by the evaluation module; The pre-processed images of the first and last frames are input into the improved RAFT optical flow estimation model. The optical flow algorithm is used to estimate the displacement of the two images in pixel coordinates to obtain the pixel displacement between the two images. Step 3. Obtain the actual three-dimensional spatial coordinates of the pixels in the previous and next frames of the image through transmission transformation. Finally, combine the calculated actual displacement with the frame rate to obtain the actual flow velocity and flow direction information of the river surface.

[0008] Furthermore, based on the aforementioned method for predicting surface velocity of water flow using improved RAFT optical flow, this invention also proposes a corresponding system for predicting surface velocity of water flow using improved RAFT optical flow, which adopts the following technical solution: A system for predicting surface velocity of water flow based on improved RAFT optical flow includes a camera and a computer device, wherein the camera is used to acquire video and send the video to the computer device.

[0009] The computer device includes a memory and one or more processors. Executable code is stored in the memory; when the processor executes the executable code, it implements steps for a method of predicting surface flow velocity of water flow based on an improved RAFT optical flow.

[0010] The present invention has the following advantages: As described above, this invention discloses a method for predicting surface velocity of water flow based on improved RAFT optical flow. This method uses the RAFT model as its basic framework and introduces a SuperPoint network, a multi-scale fusion module, and a moving window function to construct an improved RAFT optical flow estimation model suitable for specific tasks. The aim is to improve the model's adaptability to complex water flow scenarios and the accuracy of optical flow estimation. Specifically, this invention first fuses the feature descriptor information output by the SuperPoint network during the feature extraction stage. Through multi-scale fusion of original RAFT features, weighted fusion features, and attention-gated fusion features, it achieves synergistic complementarity of features from different dimensions. Furthermore, a self-attention mechanism is employed to ensure that each location in the feature map pays attention to all other locations, thereby capturing long-distance dependencies and improving the coherence and physical plausibility of the optical flow field in occluded areas and large displacement scenarios. This strengthens the representation ability of key features (such as river tracers and abrupt changes in water flow texture), improving the reliability of feature matching and solving problems such as fuzzy optical flow estimation results. Secondly, this invention introduces an evaluation module after correlation calculation. This evaluation module calculates the matching quality evaluation index based on the 4D correlated volume and contextual feature map, and determines whether to enable the moving window function based on its magnitude with a preset threshold, thereby maintaining or updating the 4D correlated volume. By introducing the moving window function, the problem of false features caused by optical flow discontinuities and noise (such as water surface reflection) in water flow scenarios is solved. At the same time, the evaluation mechanism dynamically decides when to enable the moving window function, achieving a balance between optical flow estimation accuracy and efficiency. Overall, this invention progressively builds a high-precision optical flow estimation system suitable for complex water flow scenarios, from feature extraction, multi-scale fusion, temporal window enhancement to quality evaluation. It not only improves the model's adaptability to complex water flow scenarios but also achieves a balance between optical flow estimation accuracy and computational efficiency, providing core support for high-precision and robust water surface velocity prediction. Attached Figure Description

[0011] Figure 1 This is a flowchart of a method for predicting surface velocity of water flow based on improved RAFT optical flow in an embodiment of the present invention; Figure 2This is a network structure diagram of the improved RAFT optical flow estimation model built in an embodiment of the present invention; Figure 3 This is a network structure diagram of the feature encoding branch in an embodiment of the present invention; Figure 4 This is a network structure diagram of the multi-scale fusion module in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the multi-strategy fusion mechanism in an embodiment of the present invention. Figure 6 This is a schematic diagram of weighted fusion in an embodiment of the present invention; Figure 7 This is a network structure diagram of attention-gated fusion in an embodiment of the present invention; Figure 8 This is a network structure diagram of the self-attention mechanism in an embodiment of the present invention; Figure 9 This is a network structure diagram of the context feature extraction module in an embodiment of the present invention; Figure 10 This is a schematic diagram of the adaptive addition and fusion process of the moving window in an embodiment of the present invention; Figure 11 This is a schematic diagram illustrating the volume calculation related to the moving window function in an embodiment of the present invention; Figure 12 Multi-scale correlation pyramid diagram in the embodiments of the present invention; Figure 13 A schematic diagram of the multi-scale related pyramid channel dimensions in an embodiment of the present invention; Figure 14 A schematic diagram of the upsampling module in an embodiment of the present invention. Detailed Implementation

[0012] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 To address the problems of optical flow ambiguity, discontinuity, and spurious features in the original RAFT model for measuring surface velocity in water flow, this embodiment 1 describes a method for predicting surface velocity in water flow based on improved RAFT optical flow.

[0013] like Figure 1 As shown, the method for predicting surface velocity of water flow based on improved RAFT optical flow includes the following steps: Step 1. First, in the data preprocessing stage, the captured video of the river surface flow velocity is converted into images, and preprocessing such as image distortion correction and image orthorectification is performed to ensure the quality and applicability of the input data. Since the lens itself has radial and tangential distortion, correction is necessary. However, tangential distortion is usually very small and almost negligible due to the improved precision in modern lens manufacturing. Therefore, this embodiment mainly corrects radial distortion. Additionally, since a drone equipped with a gimbal camera is used to vertically hover and film the water flow, the drone's attitude can be affected by wind, causing image sequence jitter and flight path deviation. Therefore, affine transformation is used to achieve orthorectification of the images.

[0014] Step 2. Build an improved RAFT optical flow estimation model, the network structure of which is as follows: Figure 2 As shown.

[0015] Based on the RAFT network and combined with the task characteristics and practical needs of the water flow dataset, this invention makes targeted improvements to the structure and algorithm of the RAFT optical flow estimation model and proposes the RAFT-SW model.

[0016] Compared to the baseline RAFT network, the RAFT-SW model makes two key improvements in its network structure: First, it introduces SuperPoint feature descriptor information in the feature extraction stage to enhance the representation ability of key features, thereby improving the reliability of feature matching and solving problems such as fuzzy optical flow estimation results. Second, it introduces a moving window function after correlation calculation, utilizing the temporal memory of the correlation volume between consecutive frames to aggregate feature matching information from multiple frames, improving the matching stability of low-texture regions and alleviating the problem of single-frame error accumulation. However, globally enabling this function will smooth the peak of high-quality region matching, reduce optical flow accuracy, increase redundant computation, and reduce efficiency. Therefore, an evaluation mechanism is constructed to determine the triggering and use of this function.

[0017] Specifically, the improved RAFT optical flow estimation model includes a bi-branch feature extraction module, a context feature extraction module, a multi-scale fusion module, a similarity calculation module, an evaluation module, and an optical flow iterative calculation module.

[0018] The dual-branch feature extraction module includes a feature encoding branch and a SuperPoint network branch, which are used to obtain feature maps of the query frame and the reference frame, including the original RAFT features and the SuperPoint features, respectively.

[0019] The RAFT-SW model is defined as taking two frames of river images as input. and ,in As a reference frame, For query frames.

[0020] Reference Frame and query frames Each corresponds to a dual-branch feature extraction module.

[0021] The feature coding branch is used for the reference frame Or query frame Feature encoding is performed to obtain the original RAFT features.

[0022] The SuperPoint network branch is used to extract reference frames. and query frames The SuperPoint feature descriptors supplement the RAFT framework with rich local texture information, and these descriptors are highly robust to changes in lighting and viewpoint.

[0023] In addition, reference frame It also inputs a context feature extraction module to extract global information.

[0024] There are two multi-scale fusion modules, one for receiving feature maps of the query frame and the other for receiving feature maps of the reference frame. and query frames Each feature map has two parts: one is the original RAFT feature, and the other is the SuperPoint feature.

[0025] With corresponding reference frame Take one of the multi-scale fusion modules as an example: The original RAFT features and SuperPoint features are fed into a multi-scale fusion module and fused using a multi-strategy fusion mechanism to obtain a new feature map. Then, a self-attention mechanism is used for optimization to obtain the reference frame. High-quality feature maps.

[0026] Similarly, another multi-scale fusion module outputs a query frame. High-quality feature maps.

[0027] The similarity calculation module performs similarity calculations based on the two high-quality feature maps above to obtain the 4D related bodies.

[0028] The evaluation module is used to calculate the matching quality evaluation index based on the 4D related volume and the context feature map, and to determine whether to enable the moving window function based on the magnitude of the index and the preset threshold, so as to maintain or update the 4D related volume.

[0029] The optical flow iterative calculation module performs optical flow iterative estimation based on the 4D correlated volume output by the evaluation module.

[0030] The following section provides a detailed explanation of each module in the improved RAFT optical flow estimation model.

[0031] like Figure 3 As shown, the feature encoding branch uses instance normalization (IN) to enhance the model's robustness to the feature distribution of each image, effectively mitigating the interference of factors such as water surface reflection and local disturbances on optical flow estimation.

[0032] Input image Image size is Each pixel contains 3 channels, meaning the input dimension is... .

[0033] The initial part of the feature encoding branch uses a convolutional layer with a 7×7 kernel for the input image. Initial feature extraction is performed, followed by gradual reduction of the image's spatial size through multi-layer convolution operations.

[0034] Each spatial downsampling reduces the image's height and width to half of their original values, ultimately restoring the original resolution. Compress to Meanwhile, the number of feature channels gradually increased from 64 to 128, 192, and finally to 256.

[0035] This process enhances the receptive field, reduces computational overhead, and provides a more compact and semantically rich representation for subsequent modules.

[0036] The feature encoding branch contains three sets of residual modules, each containing two residual units (a total of six). The introduction of the residual structure effectively alleviates the gradient vanishing problem in deep networks and improves training stability.

[0037] Meanwhile, residual connections enable the model to extract high-level semantic features while retaining low-level detailed features, enhancing its ability to express complex structures such as water surface reflections and ripples, thereby improving the robustness of optical flow estimation in natural river scenarios.

[0038] The final output of the feature encoding branch is the feature map. Each spatial location corresponds to a 256-dimensional feature vector, containing rich semantic and spatial structural information. The function mapping relationship of the feature encoder network is shown in the following equation: .

[0039] The feature encoder outputs feature maps at 1 / 8 resolution. , The number of channels is D=256.

[0040] For images Its processing procedure is the same as the above process, and its output is the original RAFT feature. .

[0041] This invention introduces the Superpoint network into the RAFT model, such as... Figure 4 As shown, it mainly solves the inherent limitations of traditional optical flow estimation in sparse texture regions, large displacement scenes, and repetitive texture matching.

[0042] By extracting highly discriminative 256-dimensional SuperPoint feature descriptors, rich local texture information is added to the RAFT framework. These descriptors are highly robust to changes in illumination and viewpoint.

[0043] After multi-strategy fusion with global features from RAFT, the feature representation is enhanced, especially the matching of sparse texture regions is improved, the tracking accuracy of large displacement motion is improved, and the matching ambiguity caused by repeated textures is effectively alleviated through the high discriminativeness of local descriptors. Ultimately, optical flow estimation exhibits stronger robustness and accuracy in complex dynamic scenes.

[0044] Then, a self-attention mechanism was introduced. By modeling the global long-range dependencies within the feature map, the global consistency and semantic discriminability of the feature representation were enhanced, thereby improving the coherence and physical rationality of the optical flow field in occluded areas and large displacement scenarios, reducing the matching error caused by local feature blurring, and solving the problem of optical flow estimation blurring or drift.

[0045] like Figure 5 As shown, the multi-strategy fusion mechanism combines the original RAFT features. Weighted fusion characteristics and attention-gated fusion features The fusion formula for the three different features is as follows: .

[0046] in This is the new feature map obtained after fusion. , The learnable weights are obtained by combining learnable parameters with Softmax normalization. , For feature map size, This represents the number of channels.

[0047] This invention fully leverages the advantages of different fusion strategies by learning the optimal weight combination of these three features.

[0048] Let the RAFT feature be Superpoint feature descriptor is .

[0049] First, the two features are concatenated along the channel dimension to obtain the concatenated feature map. : ;in, For splicing operations; .

[0050] Weighted fusion achieves adaptive dynamic fusion of two types of features by enabling a neural network to autonomously learn the spatial weight distribution between RAFT base features and SuperPoint feature maps. Figure 6 As shown.

[0051] Weighted fusion features The weighted fusion method achieves adaptive dynamic fusion of the two types of features by autonomously learning the spatial weight distribution between the original RAFT features and SuperPoint features through a neural network. The process is as follows: First, the original RAFT features are... and SuperPoint features The images are stitched together to obtain a stitched feature map. Then pass through in sequence Convolutional layers, ReLU activation layers, and Convolutional layers yield weighted fused features. The formula is as follows: ;in, yes Convolution operation; .

[0052] On the one hand, the weighted fusion mechanism can fully leverage the complementary advantages of the two types of features, retaining the ability of RAFT features to represent global texture and motion trends, while using the high discriminativeness of SuperPoint features to enhance the feature expression of key local areas (such as river tracers), thereby improving the semantic discriminability and scene adaptability of the fused features.

[0053] On the other hand, its adaptive weight learning capability can dynamically adjust the contribution ratio of the two types of features in complex situations such as weak texture and sudden changes in flow velocity in river scenes, avoiding the interference of the limitations of a single feature on subsequent optical flow estimation and flow velocity calculation, and ultimately enhancing the robustness of the model to complex water flow environments, providing more reliable feature support for high-precision flow velocity measurement.

[0054] The attention-gated fusion mechanism, through the construction of an adaptive weight allocation module, deeply interacts and fuses the 256-dimensional basic features extracted by the original RAFT network with the 256-dimensional feature descriptors generated by SuperPoint. Figure 7 As shown.

[0055] First, the original RAFT features are concatenated with the SuperPoint features to obtain a concatenated 512-dimensional feature map. Then, after dimensionality reduction and activation function processing by convolutional layers, a refined attention weight map is generated through a sigmoid layer, expressed by the following formula: .

[0056] in Use the Sigmoid activation function; for Convolution operation, This is the activation function.

[0057] This weight graph The contribution weight of SuperPoint features at each pixel location is quantified using spatial location as the dimension. For regions with high feature recognition and strong correlation with flow velocity measurement tasks, the weight coefficient is higher to enhance the guiding role of SuperPoint features; for regions with blurred features and high redundancy, the weight coefficient is lower to reduce interference from invalid information.

[0058] Subsequently, the weighted graph The original SuperPoint features are modulated element-wise and then added to the original RAFT features to obtain the attention-gated fusion features. , Its formula is expressed as: .

[0059] in This indicates element-wise multiplication. This dynamic weight adjustment mechanism gives the model adaptive capabilities, allowing it to adjust the fusion ratio of the two types of features in real time according to the texture and scene complexity of the river image, achieving complementary advantages of features and improving the model's adaptability to complex water flow scenes and the robustness of feature representation.

[0060] The multi-strategy fusion mechanism, comprised of the three fusion methods described above, achieves synergistic complementarity of features across different dimensions. After obtaining the fused feature map, a self-attention mechanism is employed to ensure that each location in the feature map pays attention to all other locations, thereby capturing long-range dependencies and improving the coherence and physical plausibility of the optical flow field in occluded regions and large-displacement scenes. This contributes to understanding the motion relationships between different parts of the scene in optical flow estimation, such as... Figure 8 As shown.

[0061] I. Generate query (Q), key (K), and value (V).

[0062] First, in the newly obtained feature map after fusion... The above uses three 1×1 convolutional layers to generate Q, K, and V, as shown below: ; ; .

[0063] II. Generate the attention score matrix.

[0064] First, calculate the dot product of query Q and key K: Then, scaling is performed, and the formula is: ;in This is the scaling factor.

[0065] III. Normalization.

[0066] To convert attention scores into weights, a Softmax function is applied for normalization; Softmax ensures that the sum of all output weights is 1, thus enabling the model to learn the importance of each element pair.

[0067] ; IV. Weighted summation.

[0068] Normalized attention weights are used to perform a weighted summation of the value vector V: ;in This is a transpose operation.

[0069] It was then remodeled to restore its original spatial structure for use in subsequent missions, as shown below: ;in For reshaping operations.

[0070] V. Output; finally, the final feature map is obtained through residual connections: .

[0071] in It is a learnable scalar parameter used to balance the contribution ratio of self-attention-enhanced features to original features.

[0072] The context feature extraction module employs batch normalization (BN), which helps stabilize the distribution of hidden states during training and improves the consistency and generalization ability of features across samples.

[0073] like Figure 9 As shown, the image As its input, it is used to extract the initial contextual feature information required for optical flow estimation. Its network structure is almost the same as that of the feature encoder, consisting of multiple convolutional layers and residual modules.

[0074] The context feature extraction module has two outputs: a context feature map and a context feature map. and initial hidden feature map The number of channels is 128, of which Used for optical flow matching Used to initialize optical flow residual estimation.

[0075] The function mapping relationship of the context feature extraction module is shown in the following equation: The context feature extraction module outputs the initial hidden feature map at 1 / 8 resolution. and context feature map , Number of channels =128.

[0076] The similarity calculation module performs similarity calculations based on the two high-quality feature maps above to obtain the 4D related bodies.

[0077] Visual similarity is expressed using 4D correlation volume, describing the similarity between each pair of pixels in two images. It is calculated by taking the dot product of the feature vectors of the two images (generated through a feature extraction network), as shown in the formula: .

[0078] in, , .

[0079] Each pixel in the image Each corresponds to a two-dimensional response map, showing its relationship with the image. The correlation between all pixel locations. Quantizing this similarity as a dot product allows for efficient capture of the relationships between pixels.

[0080] The relevance module in the original RAFT is prone to ambiguity during feature matching, such as in scenes with sparse textures (e.g., skies, monotonous walls) and fast-moving scenes. Even if the actual optical flow is continuous, incorrect matching may occur due to the lack of historical context information. In actual measurement scenarios, noise factors such as water reflections, raindrop interference, and camera shake can easily generate false features, which traditional optical flow methods struggle to effectively identify. They often misjudge such noise as real motion information, ultimately resulting in a large number of outliers in the calculated flow velocity data.

[0081] To address these issues, this invention employs a moving window mechanism to perform temporal memory of the correlation volume between consecutive frames, thereby enhancing the robustness of feature matching. In order to ensure accuracy without wasting resources, this invention also designs an evaluation mechanism to determine whether the moving window function needs to be activated. When the scene features meet the matching stability requirements, the model will maintain the correlation calculation process of the original RAFT network, thus balancing algorithm performance and computational efficiency.

[0082] like Figure 10 The adaptive blending process for the moving window is illustrated, where the matching quality evaluation metrics are... The calculation formula is as follows: ;in, Indicates the response consistency score. Indicates the continuity score of the matching response. Represents the texture complexity score. , , for , , Differential weighting coefficients.

[0083] The calculated matching quality assessment index , and preset threshold Size determination.

[0084] When matching quality assessment indicators Greater than or equal to the preset threshold If the feature matching quality of the current region meets the accuracy requirements, the moving window function is not enabled, and the original 4D correlated volume is used directly for subsequent calculations.

[0085] When matching quality assessment indicators Below the preset threshold If the feature matching quality of the current region is deemed poor, a moving window function is triggered to fuse the relevant volume data stored in the 4D buffer, thereby optimizing the robustness of feature matching.

[0086] Using a moving window function for all input image frames can improve the matching robustness of low-quality regions, but it will smooth out the true matching peaks of high-quality texture regions, leading to a decrease in optical flow accuracy. At the same time, the global function call pattern will generate a lot of redundant calculations, resulting in a waste of computing resources and a reduction in the model's running efficiency.

[0087] Therefore, this invention introduces an evaluation mechanism to achieve on-demand optimization and avoid the drawbacks of globally enabling the moving window function. Through the three-dimensional quality judgment of the evaluation mechanism, high / low quality matching regions can be accurately distinguished. Moving window optimization is only enabled for low-quality regions, while the original matching accuracy is preserved for high-quality regions, ultimately improving the system robustness without sacrificing accuracy.

[0088] Response consistency assessment determines the reliability of feature matching by quantifying the local response distribution of the relevant volume.

[0089] In optical flow matching, the correlation volume is the feature similarity matrix of all pixel pairs in two frames of images (for example, the more similar the features of pixel A and pixel B are, the higher the corresponding correlation volume value). The response peak refers to the value of the most similar pixel in the correlation volume for a given pixel (i.e., the value with the highest matching degree).

[0090] The core of response consistency is to see if the response peak is prominent and concentrated enough.

[0091] If the peak is very sharp (only a few pixels have a high matching degree, while the matching degree of other pixels is low), it means that the matching of features between the two frames is clear; if the peak is very flat (the matching degree of many pixels is about the same), it means that there is ambiguity in the feature matching.

[0092] The calculation process for the response consistency score is as follows: The local matching response distribution of the 4D correlated volume is transformed into a response consistency score through two convolution operations, as shown in the following formula: .

[0093] in This represents the input 4D related volume; The number of channels for the 4D correlated volume; It is the number of channels in the hidden layer of the evaluation module, that is, the number of intermediate channels used to extract features in the evaluation mechanism, which can be manually set according to task requirements.

[0094] Indicates size The convolution kernel and the number of input channels are The number of output channels is ; Indicates size The convolution kernel and the number of input channels are The number of output channels is 1.

[0095] In real-world scenarios, the motion of adjacent pixels is usually spatially continuous, so their corresponding volumes should also be spatially smooth. If a sudden change in local gradient occurs in the corresponding volume, it is most likely caused by noise or spurious features.

[0096] The core objective of matching response continuity is to quantify the spatial smoothness of 4D correlated volumes, and the calculation process is as follows: I. 4D related volume folding.

[0097] Since 4D correlated volumes cannot be convolved or have gradient operations performed, therefore 4D correlated volumes... Folded into 3D related volume .

[0098] II. Spatial gradient calculation.

[0099] The absolute values ​​of the first-order differences in the height and width directions are calculated separately, as shown in the following formula, to obtain the gradient characteristics in the two directions. A larger gradient indicates a more drastic change in response.

[0100] Height gradient (differentiation between adjacent rows, quantifying changes in vertical response): ; Gradient in width direction (differentiation between adjacent columns, quantifying changes in the horizontal response): ; in Fill with zeros. .

[0101] When the value is 1, zero-padding is performed on this part, such as... This means adding a column of zeros on the left to ensure that the gradient dimension is consistent with the original folded 3D volume. Consistent; It is an absolute value.

[0102] IV. Gradient feature fusion.

[0103] The height and width gradient of the splicing are shown in the following formula, forming... channel gradient tensor It fully preserves the spatial variation information corresponding to all reference pixels. ;in It's a splicing operation.

[0104] V. Feature quantization is converted into a matching response continuity score; pass Convolution extracts local gradient patterns, then uses... Convolutional compression is converted into single-channel fractions, as shown in the following formula; .

[0105] Indicates size The convolution kernel and the number of input channels are The number of output channels is ; Indicates size The convolution kernel and the number of input channels are The number of output channels is 1.

[0106] Since regions with richer textures have higher feature discrimination and higher matching accuracy, while regions with sparser textures have lower feature discrimination and are more prone to ambiguity in matching, this invention uses texture complexity as a dimension of the evaluation mechanism.

[0107] The calculation process for the texture complexity score is as follows: I. First, extract the degree of change in local features.

[0108] Local statistical features of contextual features are extracted through convolution operations. This is used to quantify the richness of the texture.

[0109] .

[0110] in, It is the number of channels for contextual features. This is a context feature map.

[0111] II. Convert to single-channel scores.

[0112] Extracted local statistical features This is converted into a single-channel score, i.e., a texture complexity score.

[0113] .

[0114] in Indicates size The convolution kernel and the number of input channels are The number of output channels is 1.

[0115] After completing the quantitative calculations across the three dimensions, this invention assigns differentiated weights based on the contribution of each dimension to the feature matching quality, and then calculates the final matching quality evaluation index using a weighted average. As shown in the formula above.

[0116] When this indicator is greater than or equal to a preset threshold If the feature matching quality of the current region meets the accuracy requirements, the moving window function is not enabled, and the original 4D related volume is used directly for subsequent calculations.

[0117] When this indicator is below the preset threshold If the feature matching quality of the current region is deemed poor, a moving window function is triggered to fuse the relevant volume data stored in the 4D buffer, thereby optimizing the robustness of feature matching.

[0118] The above describes the evaluation mechanism. Next, we will introduce the moving window function, such as... Figure 11 As shown. This invention employs a moving window mechanism to perform temporal memory of the correlation volume between consecutive frames, thereby enhancing the robustness of feature matching. The specific details are as follows.

[0119] Define the moving window of the 4D related object as: .in It is the first The frame's 4D related volume window buffer has a fixed window length. ; It represents the total number of historical frames within the window.

[0120] The move window function is designed to remember each The correlation volume is calculated for consecutive frame pairs within the same frame. By fusing the correlation volumes within the window, temporal consistency is used to assess the degree of ambiguity in feature matching.

[0121] If the changes in feature correspondence within each time interval are consistent with historical patterns, the fused correlation volume can suppress erroneous matches caused by noise or sparse texture.

[0122] Continuously save all satisfied A historic moment Image frames and Calculated 4D related volume .like Insufficient memory If there are 4D related objects, continue caching; after several frames, when At that time, first calculate the correlation volume of the current frame. (from frame pairs) (Calculated) Push it to the end of the window, at which point the window just contains If the size increases to the point of window memory overflow, the oldest memory... It will be removed from the window's memory container to accommodate the new related volume. The fused 4D related volume is calculated by weighted summation of the 4D related volumes within the moving window, as follows:

[0123] To achieve robust feature matching by leveraging temporal consistency, the fused correlation volume is calculated by weighted summation of the correlation volumes within the moving window.

[0124] First, the evaluation metrics are calculated between each pair of adjacent frames. The weights are determined as follows: ; in For weights; Image frame and Matching quality assessment metrics. Updated 4D related volumes. for: ; in It is composed of frame pairs The calculated 4D correlation volume.

[0125] like Figure 2As shown, the optical flow iterative calculation module has a relatively conventional structure. This optical flow iterative estimation module consists of three parts: a multi-scale correlation pyramid, a correlation lookup module, and a gated recurrent unit (GRU). Its general processing procedure is as follows: First, a multi-scale correlation pyramid is used for feature processing. The 4D related volumes are downsampled to obtain 4D related blocks at different scales. The correlation lookup module extracts similarity features at different scales from the 4D related blocks.

[0126] This similarity feature provides matching information for subsequent optical flow estimation. This process is used to calculate pixel-level motion relationships between two frames, ensuring that the improved RAFT model can capture subtle changes in water flow.

[0127] The searched features are input into the GRU for optical flow estimation. The GRU combines the current optical flow estimate with the historical hidden state and continuously optimizes the optical flow calculation results through iteration. During the iteration process, the updated optical flow is fed back to the next iteration.

[0128] This invention improves the stability and accuracy of optical flow estimation by iteratively optimizing the optical flow calculation results. Furthermore, during the iteration process, the updated optical flow is fed back into the next iteration, significantly improving the convergence effect.

[0129] Although 4D correlated volume can effectively capture local matching information between images and has a good representation ability for small-range pixel displacements, its perception ability is relatively limited for regions with large displacements.

[0130] To enhance the model's adaptability to large-scale pixel motion, this invention employs a correlation pyramid structure, which simultaneously handles small and large displacement scenes through multi-scale correlation calculations.

[0131] The specific principles and structure of the relevant pyramids are as follows: Figure 12 As shown. The last two dimensions of the relevant volumes with pooling kernel sizes of 1, 2, 4, and 8, along with equal step sizes, are used to... Construct a 4-layer pyramid { , , , }

[0132] By maintaining four dimensions ( , The first two dimensions of the relevant information remain unchanged to ensure high-resolution information, thus allowing the simultaneous acquisition of global and local feature information of the image.

[0133] To reduce invalid relationships, correlation lookup is needed on the generated multi-scale image features. Simply put, this involves extracting local correlation features from the constructed correlation pyramid using the current optical flow estimate, which are then used for subsequent optical flow correction.

[0134] Correlation lookup first requires the current optical flow estimate as a guide, for coordinate location The search center is calculated using the current optical flow estimation. , These are the predicted search coordinates.

[0135] At the s-th level of the pyramid, the last two dimensions are downsampled. The actual search coordinates need to be scaled accordingly. ,by Centered on, with a radius of local neighborhood Perform a correlation search within the internal system.

[0136] The neighborhood form is shown below: .

[0137] in, The search radius (in this invention) Take 4, which is the balance point between accuracy and computational load. Indicates from the center position Starting pixel offset vector, Control the size of the search range. Because... The coordinates may be decimals. To avoid offset errors caused by sampling at non-integer positions, bilinear interpolation is used to extract features from the relevant map during the search process.

[0138] After completing the lookup operations at all scales, the relevance features from each level are concatenated into a unified vector using bilinear interpolation, such as... Figure 13 As shown.

[0139] The dimensions of each pyramid level are: There are 4 layers of pyramids, and the final dimensions after assembly are: .

[0140] Therefore, the 4D correlation volume obtained from the temporal similarity extraction and fusion module is input into the module, which performs multi-scale pyramid and correlation lookup operations to finally obtain a multi-scale correlation feature map. The function mapping relationship is shown in the following equation: ,That Number of channels .

[0141] This multi-scale correlated feature map serves as input to the subsequent gated recurrent unit update module, providing accurate matching information from the multi-scale context for optical flow estimation. This multi-level search method effectively enhances the model's robustness under large-scale motion estimation while maintaining accurate matching capabilities for image details and small-scale motion.

[0142] Optical flow estimation starts from an initial optical flow estimate. (Image frame) and The final optical flow after iterative updates, where the initial optical flow estimate for the first and second frames is 0, i.e. ) began by estimating a series of optical flow estimates.

[0143] In each iteration, the update operation generates a new optical flow update. and add it to Above, generate an updated optical flow estimate. The formula is shown below: .

[0144] in It refers to the first Second-rate( Optical flow estimation during iteration, when hour, , This represents the optical flow update calculated from the current state. Indicates the first The updated optical flow estimate at the next iteration.

[0145] This iterative process will be repeated until the fixed point can be approximated. ,Right now This fixed point This indicates that the optical flow estimate has converged to a stable value, i.e., the final optical flow estimate.

[0146] Optical flow estimation is primarily performed by the GRU. For the first... For the first estimation, the input to the GRU consists of context features, the current optical flow estimate, and correlation features (Corr) concatenated along the channel dimension; the second... Hidden state in the next iteration This is passed down from the previous iteration and dynamically encodes historical update information, with its initial value coming from the context encoder. Its output includes two parts: the updated hidden state. That is, the first The hidden state of the previous iteration is used to propagate to the next iteration; optical flow update direction. The GRU dynamically manages and optimizes the path by using reset and update gates to update the optical flow estimate. The hidden state output by the GRU is the optical flow estimate obtained after passing through two convolutional layers. This image is 1 / 8 of the input image resolution, which is a resolution of... Low-resolution optical flow field.

[0147] This update is fused with the current optical flow estimate to obtain the updated optical flow field. Once the optical flow estimation is stable or the error reaches the required level, the iteration stops, and a final optical flow update is performed to obtain the latest optical flow. .

[0148] because yes The low-resolution optical flow field needs to be upgraded to match the resolution of the input image. Therefore, for Upsampling is performed to obtain the final high-resolution optical flow field. The process is as follows Figure 14 As shown.

[0149] like Figure 14 As shown, the upsampling module generates the high-resolution optical flow field through the following steps: the input is a low-resolution optical flow estimate that eventually converges to a fixed value. Its resolution is .

[0150] First, a weight mask with size is generated through two convolutional layers. The dimension 9 of each weight mask represents the weight of the current pixel in the low-resolution neighborhood (3×3 grid). The Softmax activation function is used to ensure that the sum of the weights of each pixel is 1, thus forming a convex combination.

[0151] For each pixel coordinate at high resolution Its optical flow value is obtained by weighted summation of the optical flow values ​​of the low-resolution neighborhood. Assume the optical flow value in the low-resolution neighborhood is... , express If the weighted template is used, then the value of the high-resolution optical flow field is: .

[0152] The output is a high-resolution optical flow field after upsampling, with dimensions of . The optical flow value of each high-resolution pixel is calculated from a convex combination of the pixel values ​​in its low-resolution neighborhood.

[0153] The preprocessed first and second frame images are input into the improved RAFT optical flow estimation model, and the optical flow algorithm is used to calculate the pixel coordinates of the first and second frame images. Displacement estimation is performed below. This allows us to obtain the pixel displacement of a pixel between two frames of an image.

[0154] Step 3. Obtain the actual three-dimensional spatial coordinates of the pixels in the previous and next frames of the image through transmission transformation. Finally, combine the calculated actual displacement with the frame rate to obtain the actual flow velocity and flow direction information of the river surface.

[0155] Pixel displacement is achieved through transmission transformation. Displacement converted to world coordinate system : .

[0156] in Let be a rotation matrix. It is a translation matrix. This represents the actual height of the camera above the water surface. This is the camera's intrinsic parameter. Finally, the calculated actual displacement is multiplied by the frame rate to obtain the actual flow velocity and direction information of the river surface.

[0157] Example 2 This embodiment 2 describes a system for predicting the surface velocity of water flow based on improved RAFT optical flow. This system is based on the same inventive concept as the method for predicting the surface velocity of water flow based on improved RAFT optical flow in embodiment 1 above.

[0158] The system for predicting surface velocity of water flow based on improved RAFT optical flow in this embodiment includes a camera and a computer device, wherein the camera is used to acquire video and send the video to the computer device.

[0159] The computer device includes a memory and one or more processors. Executable code is stored in the memory; when the processor executes the executable code, it implements steps for a method of predicting surface flow velocity of water flow based on an improved RAFT optical flow.

[0160] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A method for predicting surface velocity of water flow based on improved RAFT optical flow, characterized in that, Includes the following steps: Step 1. Convert the acquired video of the river surface flow velocity into an image and preprocess the image; Step 2. Build an improved RAFT optical flow estimation model, which includes a bi-branch feature extraction module, a context feature extraction module, a multi-scale fusion module, a similarity calculation module, an evaluation module, and an optical flow iterative calculation module; The dual-branch feature extraction module includes a feature encoding branch and a SuperPoint network branch, which are used to obtain feature maps of the query frame and the reference frame, including the original RAFT features and the SuperPoint features, respectively. The context feature extraction module is used to extract the context feature map of the reference frame; The multi-scale fusion module is used to receive feature maps of query frames and reference frames respectively, and fuse them through a multi-strategy fusion mechanism to obtain new feature maps. Then, a self-attention mechanism is used for optimization to obtain high-quality feature maps. The similarity calculation module performs similarity calculations based on high-quality feature maps to obtain 4D related volumes; The evaluation module is used to calculate the matching quality evaluation index based on the 4D related volume and the context feature map, and to determine whether to enable the moving window function based on the magnitude of the index and the preset threshold, so as to maintain or update the 4D related volume. The optical flow iterative calculation module performs optical flow iterative estimation based on the 4D correlated volume output by the evaluation module; The pre-processed images of the first and last frames are input into the improved RAFT optical flow estimation model. The optical flow algorithm is used to estimate the displacement of the two images in pixel coordinates to obtain the pixel displacement between the two images. Step 3. Obtain the actual three-dimensional spatial coordinates of the pixels in the previous and next frames of the image through transmission transformation. Finally, combine the calculated actual displacement with the frame rate to obtain the actual flow velocity and flow direction information of the river surface.

2. The method for predicting surface velocity of water flow based on improved RAFT optical flow according to claim 1, characterized in that, The multi-strategy fusion mechanism combines the original RAFT features. Weighted fusion characteristics and attention-gated fusion features The fusion formula for the three different features is as follows: ； in This is the new feature map obtained after fusion. , The learnable weights are obtained by combining learnable parameters with Softmax normalization. , For feature map size, Number of channels; Weighted fusion features The weighted fusion method achieves adaptive dynamic fusion of the two types of features by autonomously learning the spatial weight distribution between the original RAFT features and SuperPoint features through a neural network. The process is as follows: First, the original RAFT features are... and SuperPoint features The images are stitched together to obtain a stitched feature map. Then pass through in sequence Convolutional layers, ReLU activation layers, and Convolutional layers yield weighted fused features. The formula is as follows: ;in, yes Convolution operation; ; Attention-gated fusion features The attention-gated fusion mechanism is obtained by constructing an adaptive weight allocation module to deeply interact with and fuse the original RAFT features and SuperPoint features. The process is as follows: First, the original RAFT features are concatenated with the SuperPoint features to obtain the concatenated feature map. Then, after dimensionality reduction and activation function processing by convolutional layers, a refined attention weight map is generated through a sigmoid layer, expressed by the following formula: ； in Use the Sigmoid activation function; for Convolution operation, For activation functions; Subsequently, the weighted graph The original SuperPoint features are modulated element-wise and then added to the original RAFT features to obtain the attention-gated fusion features. , Its formula is expressed as: ;in This indicates element-wise multiplication.

3. The method for predicting surface velocity of water flow based on improved RAFT optical flow according to claim 2, characterized in that, The self-attention mechanism processes the following steps: I. First, in the newly obtained feature map after fusion... The above uses three 1×1 convolutional layers to generate Q, K, and V, as shown below: ； ； ； II. Generate the attention score matrix; first, calculate the dot product of query Q and key K: Then, scaling is performed, and the formula is: ;in This is the scaling factor; III. Normalization; To convert attention scores into weights, the Softmax function is applied for normalization; Softmax ensures that the sum of all output weights is 1, thus allowing the model to learn the importance of each element pair: ； IV. Weighted Summation; Normalized attention weights are used to perform a weighted summation on the value vector V: ;in This is a transpose operation; It was then remodeled to restore its original spatial structure for use in subsequent missions, as shown below: ;in For reshaping operation; V. Output; finally, the final feature map is obtained through residual connections: ; in It is a learnable scalar parameter used to balance the contribution ratio of self-attention-enhanced features to original features.

4. The method for predicting surface velocity of water flow based on improved RAFT optical flow according to claim 1, characterized in that, The matching quality assessment indicators The calculation formula is as follows: ;in, Indicates the response consistency score. Indicates the continuity score of the matched response. Represents the texture complexity score. , , for , , Differential weighting coefficients; The calculated matching quality assessment index , and preset threshold Size determination; When matching quality assessment indicators Greater than or equal to the preset threshold When the feature matching quality of the current region is determined to meet the accuracy requirements, the moving window function is not enabled, and the original 4D related volume is used directly for subsequent calculations. When matching quality assessment indicators Below the preset threshold If the feature matching quality of the current region is deemed poor, a moving window function is triggered to fuse the relevant volume data stored in the 4D buffer, thereby optimizing the robustness of feature matching.

5. The method for predicting surface velocity of water flow based on improved RAFT optical flow according to claim 4, characterized in that, The calculation process for the response consistency score is as follows: The local matching response distribution of the 4D correlated volume is transformed into a response consistency score through two convolution operations, as shown in the following formula: ； in This represents the input 4D related volume; The number of channels for the 4D correlated volume; It is the number of channels in the hidden layer of the evaluation module, that is, the number of intermediate channels used to extract features in the evaluation mechanism; Indicates size The convolution kernel and the number of input channels are The number of output channels is ; Indicates size The convolution kernel and the number of input channels are The number of output channels is 1.

6. The method for predicting surface velocity of water flow based on improved RAFT optical flow according to claim 4, characterized in that, The calculation process for the continuity score of the matched response is as follows: I. 4D related volume folding; 4D related bodies Folded into 3D related volume ; II. Spatial gradient calculation; Calculate the absolute values ​​of the first-order differences in the height and width directions respectively, as shown in the following formula, to obtain the gradient features in the two directions; The gradient in the height direction is as follows: ； The gradient in the width direction is as follows: ； in Fill with zeros. It is the absolute value; IV. Gradient feature fusion; The height and width gradient of the splicing are shown in the following formula, forming... channel gradient tensor ; ;in It's a splicing operation; V. Feature quantization is converted into a matching response continuity score; pass Convolution extracts local gradient patterns, then uses... Convolutional compression is converted into single-channel fractions, as shown in the following formula; ； in Indicates size The convolution kernel and the number of input channels are The number of output channels is ; Indicates size The convolution kernel and the number of input channels are The number of output channels is 1.

7. The method for predicting surface velocity of water flow based on improved RAFT optical flow according to claim 4, characterized in that, The calculation process for the texture complexity score is as follows: I. First, extract the degree of change in local features; Local statistical features of contextual features are extracted through convolution operations. To quantify the richness of the texture; ； in, It is the number of channels for contextual features. For context feature maps; II. Extracted local statistical features Convert to a single-channel score, i.e., a texture complexity score; ； in Indicates size The convolution kernel and the number of input channels are The number of output channels is 1.

8. The method for predicting surface velocity of water flow based on improved RAFT optical flow according to claim 4, characterized in that, The process of updating the 4D related volume by fusing the relevant volume data stored in the 4D buffer using a moving window function is as follows: Define the moving window of the 4D related object as: ; in It is the first The frame's 4D related volume window buffer has a fixed window length. ; It represents the total number of historical frames within the window; Continuously save all satisfied A historic moment Image frames and Calculated 4D related volume ;like Insufficient memory If there are 4D related objects, continue caching; after several frames, when At that time, first calculate the correlation volume of the current frame. Push to the end of the window, at which point the window will contain exactly what you need. If the size increases to the point of window memory overflow, the oldest memory... It will be removed from the window's memory container to accommodate the new related size; The fused 4D related volumes are calculated by weighted summation of the 4D related volumes within the moving window, as follows: First, the final evaluation metric is calculated between each two adjacent frames. The weights are determined as follows: ； in For weights; Image frame and Matching quality assessment indicators; Updated 4D related objects for: ； in It is composed of frame pairs The calculated 4D correlation volume.

9. The method for predicting surface velocity of water flow based on improved RAFT optical flow according to claim 1, characterized in that, The optical flow iterative calculation module includes a multi-scale correlation pyramid, a correlation lookup module, and a gated loop unit (GRU). First, a multi-scale correlation pyramid is used for feature processing. The 4D related volumes are downsampled to obtain 4D related blocks at different scales. Then, the correlation lookup module extracts similarity features at different scales from the 4D related blocks. The searched features are input into the GRU for optical flow estimation. The GRU combines the current optical flow estimate with the historical hidden state and continuously optimizes the optical flow calculation results through iteration. During the iteration process, the updated optical flow is fed back to the next iteration.

10. A system for predicting surface velocity of water flow based on improved RAFT optical flow, comprising a camera and a computer device; wherein the camera is used to acquire video and send the video to the computer device; The computer device includes a memory and one or more processors; The memory stores executable code; its characteristic is that... When the processor executes the executable code, it is used to implement the steps of the method for predicting surface velocity of water flow based on the improved RAFT optical flow as described in any one of claims 1 to 9.

Images