River flow velocity fluctuation early warning method and system based on optical flow algorithm

Abstract

The invention discloses a river flow velocity fluctuation early warning method and system based on an optical flow algorithm. The method comprises the following steps: collecting continuous frame video data; processing the video data through a traditional optical flow algorithm and a deep learning optical flow algorithm, and performing densification processing on the sparse optical flow field data to obtain traditional optical flow densified data; calculating confidence mapping data of the sparse optical flow field data and uncertainty data of the deep learning optical flow field data; obtaining weight map data through a lightweight residual error optimized weight generation network; performing fusion processing on the traditional optical flow dense data and the deep learning optical flow field data according to the weight map data to obtain fused optical flow field data; and calculating flow velocity distribution data of the river surface by fusing the optical flow field data, and outputting flow velocity fluctuation early warning information data according to the flow velocity distribution data. According to the invention, the boundary retention capability of the traditional algorithm and the dense prediction advantage of the deep learning algorithm are fully exerted through the fusion of the two-path optical flow algorithm.

Description

Technical Field

[0001] This invention relates to the field of hydrological monitoring technology, and in particular to a method and system for early warning of river flow velocity fluctuations based on optical flow algorithms. Background Technology

[0002] Traditional optical flow methods (such as Lucas-Kanade and Farneback algorithms) calculate flow velocity by the displacement of pixels in adjacent frames. However, they are limited by the assumptions of constant brightness and small motion, and are prone to noise interference in complex water flow (such as turbulent flow and high flow velocity) and lighting changes, resulting in drastic fluctuations in the output flow velocity. For example, the Lucas-Kanade algorithm relies on local window feature matching, which is prone to misjudgment due to aperture issues; while Farneback's dense optical flow covers the entire surface, it has a large computational load and is difficult to process high-resolution video in real time.

[0003] Deep learning algorithms, exemplified by FlowNet 2.0, predict optical flow through end-to-end CNN networks, overcoming the reliance on handcrafted features in traditional methods. However, outliers still exist in the output when training data is insufficient or in complex environments (such as water surface reflection or floating object occlusion). Furthermore, the model's inference results require post-processing (such as optical flow filtering and coordinate transformation) to generate physically meaningful flow velocity data. Directly using the raw output can easily lead to false alarms due to insufficient model generalization ability. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0005] In a first aspect, the present invention provides a method for early warning of river flow velocity fluctuations based on optical flow algorithms, comprising:

[0006] Acquire continuous frame video data;

[0007] Video data is processed using both traditional optical flow algorithms and deep learning optical flow algorithms to obtain sparse optical flow field data and deep learning optical flow field data. The sparse optical flow field data is then densified to obtain traditional optical flow densified data.

[0008] Calculate the confidence mapping data of sparse optical flow field data and the uncertainty data of deep learning optical flow field data;

[0009] A lightweight residual-optimized weight generation network is used to process traditional optical flow densification data, deep learning optical flow field data, confidence mapping data, and uncertainty data to obtain weight map data.

[0010] Based on the weighted graph data, traditional optical flow densification data and deep learning optical flow field data are fused to obtain fused optical flow field data.

[0011] The velocity distribution data on the river surface is calculated using the fused optical flow field data, and velocity fluctuation early warning information is output based on the velocity distribution data.

[0012] As a preferred embodiment of the river flow velocity fluctuation early warning method based on optical flow algorithm described in this invention, the step of processing video data through the traditional optical flow algorithm module includes:

[0013] Two consecutive frames of video data are converted into grayscale image data; Gaussian pyramid data is constructed using the grayscale image data, with L=5 pyramid layers and each layer being half the size of the previous layer; the motion of feature points is tracked at each level of the Gaussian pyramid data using the pyramid Lucas-Kanade algorithm to obtain motion vector set data corresponding to key points; sparse optical flow field data is output based on the motion vector set data.

[0014] As a preferred embodiment of the river flow velocity fluctuation early warning method based on optical flow algorithm described in this invention, the step of densifying the sparse optical flow field data includes:

[0015] Initialize a zero-matrix data with dimension H×W×2; fill the corresponding coordinate positions with the effective optical flow vectors in the sparse optical flow field data; for blank pixel positions, find the nearest effective optical flow vector and fill it by nearest neighbor interpolation until the H×W matrix is ​​traversed; output traditional optical flow densification data with dimension H×W×2.

[0016] As a preferred embodiment of the river flow velocity fluctuation early warning method based on optical flow algorithm described in this invention, the step of processing video data using the deep learning optical flow algorithm includes:

[0017] Two consecutive frames of video data are converted into grayscale image data and preprocessed. The preprocessed grayscale image data is then used to perform end-to-end prediction on the FlowNet deep network model to obtain two-dimensional displacement component data for each pixel position. The two-dimensional displacement component data are then combined to form deep learning optical flow field data.

[0018] As a preferred embodiment of the river flow velocity fluctuation early warning method based on optical flow algorithm described in this invention, the step of calculating the confidence mapping data includes:

[0019] Image gradient information is calculated from the grayscale image data, and the horizontal gradient is obtained using the Sobel operator. Vertical gradient and time gradient ;

[0020] Based on the effective optical flow vector in the sparse optical flow field data and the image gradient information data, the tracking error data is defined as the sum of squares of the residuals of the optical flow constraint equations in the neighborhood, expressed by the formula:

[0021] ;

[0022] In the formula, (x,y) represents the optical flow tracking error at pixel position (x,y). For a neighborhood window centered at (x,y), Let be the horizontal grayscale gradient at pixel (i,j). Let be the vertical grayscale gradient at pixel (i,j). Let be the gray-level gradient in the time direction at pixel (i,j), i.e. u is the horizontal optical flow component at the pixel location, and v is the vertical optical flow component at the pixel location;

[0023] Through the exponential decay function The tracking error data is converted into confidence weights to obtain confidence mapping data with dimensions H×W×1;

[0024] in, The attenuation coefficient is... This represents the confidence mapping value at pixel (x, y). It is an exponential function.

[0025] The beneficial effects of this preferred technical solution are as follows: by calculating the residual of the optical flow constraint equation to evaluate the confidence level of traditional optical flow algorithms, the problem of difficulty in reliability assessment of traditional optical flow in areas with missing textures and occlusion is solved. This method can accurately identify regions with poor optical flow estimation quality, providing reliable quality guidance for subsequent fusion processing and improving the applicability of traditional optical flow algorithms in complex water surface environments.

[0026] As a preferred embodiment of the river flow velocity fluctuation early warning method based on optical flow algorithm described in this invention, the step of calculating uncertainty data includes:

[0027] The predicted deep learning optical flow field data is applied to the next frame of image data. The pixel difference between the distorted image data and the previous frame of image data is calculated, and the uncertainty data is output. The calculation formula is expressed as follows:

[0028] ;

[0029] In the formula, The uncertain data value at pixel position (x, y) represents the reliability of the prediction quality of the deep learning optical flow algorithm. Let (x, y) be the grayscale value of the reference frame image at pixel position (x, y). Let be the grayscale value of the target frame image at pixel position (x, y). Let x be the horizontal optical flow component at position (x, y) in the deep learning optical flow field data, representing the horizontal displacement of the pixel. This represents the vertical optical flow component at position (x, y) in the deep learning optical flow field data, indicating the displacement of the pixel in the vertical direction. This is the interpolated grayscale value at position (x,y) after the first frame image is inversely distorted based on the optical flow vector (u,v).

[0030] The beneficial effects of this preferred technical solution are: it assesses the uncertainty of deep learning algorithms based on time consistency reconstruction error, and effectively solves the problem of lack of interpretability and reliability assessment in deep learning methods.

[0031] As a preferred embodiment of the river flow velocity fluctuation early warning method based on optical flow algorithm described in this invention, the step of obtaining the weighted graph data includes:

[0032] Traditional optical flow densification data, deep learning optical flow field data, confidence mapping data, and uncertainty data are stitched together in four channels to obtain... As input data for the weight generation network;

[0033] The weighted graph data of dimension H×W×1 is obtained by processing through a weight generation network, specifically represented as follows:

[0034] ;

[0035] In the formula, F_{Dense} represents the traditional optical flow densification data. For deep learning optical flow field data, For confidence level mapping data, For uncertain data, Generate network functions for the weights. This is weighted graph data.

[0036] The beneficial effects of this preferred technical solution are as follows: by stitching the dual-path optical flow results with the corresponding quality assessment data in four channels, a multimodal feature input mechanism is constructed, enabling the weight generation network to simultaneously perceive the optical flow prediction results and quality assessment information. This design improves the network's ability to judge the optical flow quality in different regions and achieves pixel-level accurate weight allocation.

[0037] As a preferred embodiment of the river flow velocity fluctuation early warning method based on optical flow algorithm described in this invention, the fusion processing step includes:

[0038] Based on the weighted graph data, using the formula We perform weighted fusion of deep learning optical flow field data and traditional optical flow densification data to obtain fused optical flow field data with dimensions H×W×2;

[0039] In the formula, To fuse optical flow field data.

[0040] As a preferred embodiment of the river flow velocity fluctuation early warning method based on optical flow algorithm described in this invention, the lightweight residual optimization weight generation network includes:

[0041] A four-layer multi-scale feature pyramid data was constructed, and the original four-layer standard residual blocks were replaced with two layers of lightweight residual units.

[0042] Each lightweight residual unit contains a 1×1 convolutional channel reduced to C / 4, a 3×3 depthwise separable convolution, and a 1×1 convolutional channel restored to C;

[0043] Both input and output are feature map data of size H / 16×W / 16×128, and a lightweight pixel enhancement module is embedded at the end of the residual connection.

[0044] In the integral branch, a 3-stage parallel pooling is used instead of the original 4-stage serial pooling. Branch 1 performs 1×1 adaptive average pooling with the same input and output resolution. Branch 2 performs 2×2 average pooling and then upsamples to H / 64×W / 64 through bilinear interpolation. Branch 3 performs 4×4 average pooling and then upsamples to H / 64×W / 64 through bilinear interpolation.

[0045] The beneficial effects of this preferred technical solution are: by using lightweight residual optimization and a multi-scale parallel processing architecture, the computational complexity is significantly reduced while ensuring fusion accuracy.

[0046] Secondly, the present invention provides a river flow velocity fluctuation early warning system based on optical flow algorithm, comprising:

[0047] The video acquisition module is used to acquire continuous frame video data;

[0048] The traditional optical flow processing module is used to process video data using traditional optical flow algorithms to obtain sparse optical flow field data, perform densification processing on the sparse optical flow field data to obtain traditional optical flow densified data, and calculate confidence mapping data.

[0049] The deep learning optical flow processing module is used to process video data using deep learning optical flow algorithms to obtain deep learning optical flow field data and calculate uncertainty data.

[0050] The intelligent fusion module is used to process traditional optical flow densification data, deep learning optical flow field data, confidence mapping data and uncertainty data through a lightweight residual-optimized weight generation network to obtain weight map data, and to fuse traditional optical flow densification data and deep learning optical flow field data according to the weight map data to obtain fused optical flow field data.

[0051] The flow velocity early warning module is used to calculate the flow velocity distribution data on the river surface through the fused optical flow field data, and output flow velocity fluctuation early warning information data based on the flow velocity distribution data.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] Traditional optical flow algorithms tend to fail in areas lacking texture, while deep learning algorithms suffer from ambiguity in handling boundary details. This invention, by fusing dual-path optical flow algorithms, fully leverages the boundary preservation capabilities of traditional algorithms and the dense prediction advantages of deep learning algorithms, achieving complementary strengths and significantly improving the robustness and accuracy of optical flow estimation in complex water surface environments.

[0054] To address the lack of reliability evaluation for optical flow estimation results, this invention innovatively proposes a dual quality assessment mechanism combining confidence mapping and uncertainty assessment. By utilizing the residuals of the optical flow constraint equations and the temporal consistency reconstruction error, the reliable and unreliable regions of optical flow estimation can be accurately identified, providing a scientific quality guidance basis for intelligent fusion. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the overall process of a river flow velocity fluctuation early warning method based on optical flow algorithm according to an embodiment of the present invention. Detailed Implementation

[0057] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0058] Example 1, referring to Figure 1As an embodiment of the present invention, a method for early warning of river flow velocity fluctuations based on optical flow algorithm is provided, comprising:

[0059] S1. Acquire continuous frame video data;

[0060] In this embodiment, a high-resolution waterproof camera is used to collect video data at the river monitoring point. The specific implementation steps are as follows:

[0061] Equipment configuration: An industrial-grade waterproof camera with a resolution of 1920×1080 pixels is selected, equipped with autofocus and image stabilization. It is installed on a fixed bracket on the riverbank, with the camera 5-8 meters above the water surface and the shooting angle at 30-45 degrees to the water surface to ensure that clear water texture and flow characteristics can be captured.

[0062] Acquisition parameter settings: The video frame rate is set to 25fps to ensure sufficient temporal correlation between adjacent frames for optical flow calculation; the image format adopts RGB 24-bit true color mode; the exposure time is adaptively adjusted according to the ambient lighting conditions, set to 1 / 60 second during the day and 1 / 30 second at night or in low light conditions; the ISO sensitivity is controlled within the range of 100-800 to ensure image clarity.

[0063] Data preprocessing: The continuously acquired video streams are marked and stored according to timestamps. Each video segment is 10 seconds long and contains 250 frames of image data. The original RGB images are denoised using a Gaussian filter to remove sensor noise. Abnormal frames that are overexposed or underexposed are detected and removed to ensure the quality and stability of the input data.

[0064] S2. Process video data using traditional optical flow algorithm and deep learning optical flow algorithm respectively to obtain sparse optical flow field data and deep learning optical flow field data. Perform densification processing on the sparse optical flow field data to obtain traditional optical flow densification data.

[0065] The steps for processing video data using the traditional optical flow algorithm module include S2.1 to S2.4:

[0066] S2.1 Convert two consecutive frames of video data into grayscale image data;

[0067] Extract two adjacent frames from the continuous video stream acquired in step S1. and An RGB color image with dimensions of 1920×1080×3 was used. A weighted average method was employed for grayscale conversion: Gray = 0.299×R + 0.587×G + 0.114×B, resulting in a 1920×1080 single-channel grayscale image. Histogram equalization was then applied to the converted grayscale image to enhance image contrast and improve the accuracy of feature point detection.

[0068] S2.2 Construct Gaussian pyramid data using the grayscale image data. The number of pyramid layers L=5, and the size of each layer is reduced to half that of the previous layer.

[0069] Using the grayscale image obtained in step S2.1 as the base layer (L=0), a Gaussian kernel is applied. After convolutional filtering, a pyramid is constructed by downsampling at a 2:1 ratio. The specific layer dimensions are: L0 layer 1920×1080, L1 layer 960×540, L2 layer 480×270, L3 layer 240×135, and L4 layer 120×67. Each layer is preprocessed with a 5×5 Gaussian filter before downsampling to prevent aliasing.

[0070] S2.3. Track the motion of feature points at each level of the Gaussian pyramid data using the pyramid Lucas-Kanade algorithm to obtain the motion vector set data corresponding to the key points;

[0071] Initial feature points were extracted at the highest L4 layer using a Shi-Tomasi corner detector, with a detection threshold set to 0.01, a minimum feature point spacing of 10 pixels, and a maximum feature point count of 500. The Lucas-Kanade optical flow tracing algorithm was employed, with a search window size of 15×15 pixels. The iteration termination condition was a maximum of 30 iterations or a convergence accuracy of 0.03 pixels. Tracking results were propagated downwards layer by layer from the highest layer, with each layer's initial estimate being twice the magnification of the previous layer's result.

[0072] S2.4 Output sparse optical flow field data based on the motion vector set data.

[0073] The motion vector sets obtained from each tracking layer are integrated, and feature points that failed to track (points with tracking errors greater than 2.0 pixels) are removed, ultimately yielding motion vector data with 300-400 valid feature points. Each motion vector contains coordinates (x, y) and displacement components (u, v), forming a sparse optical flow field data structure. .

[0074] The steps for densifying the sparse optical flow field data include S2.41 to S2.44:

[0075] S2.41. Initialize a zero-matrix data with dimensions H×W×2;

[0076] Create a three-dimensional array with dimensions 1920×1080×2. The third dimension has two channels that store the optical flow components in the horizontal and vertical directions, respectively, with all elements initialized to 0. A 1920×1080 binary mask matrix (Mask) is also created to mark the positions of the filled pixels.

[0077] S2.42. Fill the effective optical flow vector in the sparse optical flow field data into the corresponding coordinate positions;

[0078] Iterate through the sparse optical flow field data obtained in step S2.4, and fill in the optical flow vector (u,v) of each effective feature point. The corresponding coordinates (x, y) of the array are set, and Mask(x, y) is marked as 1 to indicate that the position is filled.

[0079] S2.43. For blank pixel positions, find the nearest effective optical flow vector and fill it using nearest neighbor interpolation until the H×W matrix has been traversed.

[0080] For blank pixel locations with a Mask value of 0, the KD-tree algorithm is used to quickly search for the nearest filled pixel. The Euclidean distance is then calculated. The optical flow value of the filled point with the smallest distance is selected and assigned. To avoid error propagation caused by over-interpolation, the maximum search radius is set to 50 pixels, and pixels outside this range are set as zero vectors.

[0081] S2.44, Output traditional optical flow densification data with dimensions H×W×2.

[0082] After completing the full matrix traversal, the densification result is subjected to median filtering with a 3×3 filter window to remove outliers that may occur during interpolation. The final output is traditional optical flow densification data with dimensions of 1920×1080×2. .

[0083] The steps for processing video data using the deep learning optical flow algorithm include S2.5 to S2.7:

[0084] S2.5. Convert two consecutive frames of video data into grayscale image data and preprocess the grayscale image data.

[0085] The same method as in step S2.1 is used to convert two consecutive RGB images into grayscale images. The grayscale images are then normalized: pixel values ​​are normalized to the range of [0,1], the mean is centered (subtracted by 0.5), and the data format is adjusted to the tensor form [2, 1, 1080, 1920] required by the deep network.

[0086] S2.6. Perform end-to-end prediction on the preprocessed grayscale image data using the FlowNet deep network model to obtain the two-dimensional displacement component data of each pixel position;

[0087] Load the pre-trained FlowNet-C network model, which employs an encoder-decoder architecture and can handle inputs of arbitrary resolution. Input the pre-processed image to the network and compute the two-dimensional optical flow prediction value for each pixel location through forward propagation. The network output is a [1, 2, 1080, 1920]-dimensional tensor, where the two channels of the second dimension correspond to the horizontal and vertical optical flow components, respectively.

[0088] S2.7 Combine the two-dimensional displacement component data to form deep learning optical flow field data.

[0089] The network output tensor is converted to a standard optical flow field data format, rearranged to a dimension of 1920×1080×2. Post-processing is then performed on the output: the optical flow values ​​are restored from the network's normalized range to the actual pixel displacement range, and boundary pixel smoothing is applied. Finally, dense deep learning optical flow field data is obtained. This data naturally covers all pixel locations.

[0090] S3. Calculate the confidence mapping data of sparse optical flow field data and the uncertainty data of deep learning optical flow field data;

[0091] The steps for calculating the confidence mapping data include S3.1 to S3.3:

[0092] S3.1 Calculate image gradient information data using the grayscale image data, and obtain the horizontal gradient using the Sobel operator. Vertical gradient and time gradient ;

[0093] Based on the grayscale image data obtained in step S2.1, the Sobel operator is used to calculate the gradient information in three directions. The gradient in the horizontal direction is calculated using... Core: [[-1,0,1],[-2,0,2],[-1,0,1]]; Vertical gradient uses Kernel: [[-1,-2,-1],[0,0,0],[1,2,1]]. Convolution is performed on the grayscale image to obtain the horizontal gradient Ix(i,j) and the vertical gradient Iy(i,j). Temporal gradient... Calculated by the difference between corresponding pixels in two adjacent frames: Δt is the inter-frame time interval of 0.04 seconds. To avoid numerical instability in gradient calculation, the gradient values ​​are truncated and limited to the range [-255, 255].

[0094] S3.2. Based on the effective optical flow vector in the sparse optical flow field data and the image gradient information data, the tracking error data is defined as the sum of squares of the residuals of the optical flow constraint equations in the neighborhood, expressed by the formula:

[0095] ;

[0096] In the formula, (x,y) represents the optical flow tracking error at pixel position (x,y). For a neighborhood window centered at (x,y), Let be the horizontal grayscale gradient at pixel (i,j). Let be the vertical grayscale gradient at pixel (i,j). Let be the gray-level gradient in the time direction at pixel (i,j), i.e. u is the horizontal optical flow component at the pixel location, and v is the vertical optical flow component at the pixel location;

[0097] Based on the effective optical flow vector in the sparse optical flow field data obtained in step S2.4, and combined with the image gradient information calculated in step S3.1, the tracking error of each feature point is calculated. A 5×5 pixel neighborhood window W(x,y) is defined centered on each feature point, and the sum of squared residuals of the optical flow constraint equations is calculated within this window. Specifically, for a feature point at position (x,y), whose optical flow component is (u,v), the residuals of the constraint equations at each pixel (i,j) are calculated by traversing the 25 pixels within the neighborhood window. Then square and sum them up to get For feature points in the boundary region, a boundary filling strategy is used to expand the neighborhood window.

[0098] S3.3, through the exponential decay function The tracking error data is converted into confidence weights to obtain confidence mapping data with dimensions H×W×1;

[0099] in, The attenuation coefficient is... This represents the confidence mapping value at pixel (x, y). It is an exponential function;

[0100] An exponential decay function is used to convert tracking errors into confidence weights, with the decay coefficient λ set to 0.01. This parameter was optimized through extensive experimental data. For each feature point location (x, y), a confidence value is calculated. Since sparse optical flow only has values ​​at feature point locations, the confidence needs to be extended to the entire image. Using the same nearest neighbor interpolation method as S2.43, the confidence values ​​of the feature points are propagated to the entire 1920×1080 image region, generating confidence mapping data of dimension H×W×1. The interpolation results are Gaussian smoothed with a standard deviation σ=2.0 to ensure the continuity and reasonableness of the confidence distribution.

[0101] The steps for calculating uncertain data include:

[0102] The predicted deep learning optical flow field data is applied to the next frame of image data. The pixel difference between the distorted image data and the previous frame of image data is calculated, and the uncertainty data is output. The calculation formula is expressed as follows:

[0103] ;

[0104] In the formula, The uncertain data value at pixel position (x, y) represents the reliability of the prediction quality of the deep learning optical flow algorithm. Let (x, y) be the grayscale value of the reference frame image at pixel position (x, y). Let be the grayscale value of the target frame image at pixel position (x, y). Let x be the horizontal optical flow component at position (x, y) in the deep learning optical flow field data, representing the horizontal displacement of the pixel. This represents the vertical optical flow component at position (x, y) in the deep learning optical flow field data, indicating the displacement of the pixel in the vertical direction. This is the interpolated grayscale value at position (x,y) after the first frame image is inversely distorted based on the optical flow vector (u,v).

[0105] Based on the deep learning optical flow field data obtained in step S2.7 For reference frame image A reverse warping operation is performed. For each pixel position (x, y), the source pixel position (x+u, y+v) is calculated based on the optical flow vector (u(x, y), v(x, y)). Since the calculated source position is usually a non-integer coordinate, bilinear interpolation is used to obtain the interpolated grayscale value. For locations that extend beyond the image boundaries, a boundary copying strategy is used for filling.

[0106] Compare the distorted reference frame image with the target frame image. Perform pixel-by-pixel comparisons and calculate the L1 norm differences: To improve computational efficiency, vectorization is used to process the entire image simultaneously. Post-processing of the computation results is performed: first, outlier detection is performed, truncating pixel values ​​greater than 3 times the standard deviation to a reasonable range; then, 3×3 median filtering is used to remove isolated noise points; finally, the uncertainty values ​​are normalized to the [0,1] range for easier subsequent network processing.

[0107] The final output is confidence mapping data with dimensions of 1920×1080×1. Uncertainty data with dimensions of 1920×1080×1 Both datasets are stored in 32-bit floating-point format to ensure numerical precision. In the confidence mapping data, the closer the value is to 1, the more reliable the prediction of the traditional optical flow algorithm at that location; in the uncertainty data, the closer the value is to 0, the more accurate the prediction of the deep learning algorithm at that location.

[0108] S4. The traditional optical flow densification data, deep learning optical flow field data, confidence mapping data and uncertainty data are processed by a lightweight residual-optimized weight generation network to obtain weight map data.

[0109] The steps for obtaining weighted graph data include:

[0110] Traditional optical flow densification data, deep learning optical flow field data, confidence mapping data, and uncertainty data are stitched together in four channels to obtain... As input data for the weight generation network;

[0111] The weighted graph data of dimension H×W×1 is obtained by processing through a weight generation network, specifically represented as follows:

[0112] ;

[0113] In the formula, F_{Dense} represents the traditional optical flow densification data. For deep learning optical flow field data, For confidence level mapping data, For uncertain data, Generate network functions for the weights. This is weighted graph data.

[0114] In this embodiment, the four types of data obtained in steps S2 and S3 are preprocessed to ensure compatibility. Traditional optical flow densification data... The dimensions are 1920×1080×2, and its optical flow amplitude needs to be calculated as a single-channel input: This yielded data of 1920×1080×1. Deep learning optical flow field data. Similarly, the amplitude is calculated to obtain The dimensions are 1920×1080×1. Confidence mapping data. and uncertain data It is already in single-channel format, with dimensions of 1920×1080×1.

[0115] To ensure the stability and convergence of network training, the four types of data were normalized respectively: Normalized to the [0,1] range, the maximum optical flow amplitude is set to 50 pixels / frame; The same normalization strategy is used; The data is already within the range (0,1], and no additional processing is required; The data is mapped to the range [0,1] through min-max normalization.

[0116] The tensor concatenation operation is used to concatenate the four types of data along the channel dimension: This yields an input tensor with dimensions of 1920×1080×4. The concatenation order is arranged according to data importance: channel 0 is the traditional optical flow amplitude, channel 1 is the deep learning optical flow amplitude, channel 2 is the confidence mapping, and channel 3 is the uncertainty data.

[0117] The weight generation network adopts an encoder-decoder architecture, with the encoder containing 4 downsampling layers and the decoder containing 4 upsampling layers. The network input size is 1920×1080×4. To adapt to GPU memory limitations, the input data is divided into overlapping windows of 480×270×4 for block processing, with the overlapping area being 32 pixels wide.

[0118] Encoder Layer 1: Uses a 3×3 convolution kernel, 4 input channels, 32 output channels, stride of 2, padding of 1, and ReLU activation function; output size is 960×540×32. Encoder Layer 2: 3×3 convolution, 32 input channels, 64 output channels, stride of 2; output size is 480×270×64. Encoder Layers 3 and 4 output 128 and 256 channel feature maps respectively, with a final encoded feature size of 120×67×256.

[0119] Lightweight residual units are added after each layer of the encoder. The residual unit structure is as follows: a 1×1 convolution reduces the number of channels to C / 4; a 3×3 depthwise separable convolution is used for spatial feature extraction, which decomposes the standard convolution into depthwise convolution and pointwise convolution, significantly reducing the number of parameters; finally, a 1×1 convolution restores the original number of channels C. The residual connections use an identity mapping, and the output is the element-wise sum of the input and the output of the residual unit.

[0120] The decoder uses transposed convolutions for upsampling, with each layer upsampling by a factor of 2. The decoder layers have 128, 64, 32, and 1 channels respectively, and are designed symmetrically with the encoder. Skip connections are used during decoding to concatenate the feature maps of corresponding layers in the encoder with those in the decoder, enhancing the ability to reconstruct details.

[0121] The final layer of the network uses a 1×1 convolution to map the feature map to a single-channel output, and the sigmoid activation function ensures that the weight values ​​are within the range of (0,1). The results of the block processing are concatenated and recombined, and a weighted average fusion strategy is used for overlapping regions, with weights distributed according to their distance from the boundary. The final output is a weight map data w with dimensions of 1920×1080×1.

[0122] The weight map data is smoothed by using a 5×5 Gaussian kernel for convolution with a standard deviation σ=1.5 to eliminate block artifacts and outliers. The weight values ​​are dynamically adjusted: w_adjusted = 0.1 + 0.8 × w, ensuring the weight values ​​are within the range [0.1, 0.9] to avoid complete bias towards one algorithm's result. The final weight map data w indicates the fusion weight allocation of the traditional and deep learning algorithms at each pixel location.

[0123] S5. Based on the weighted graph data, the traditional optical flow densification data and the deep learning optical flow field data are fused to obtain fused optical flow field data.

[0124] The steps of the fusion process include:

[0125] Based on the weighted graph data, using the formula We perform weighted fusion of deep learning optical flow field data and traditional optical flow densification data to obtain fused optical flow field data with dimensions H×W×2;

[0126] In the formula, To fuse optical flow field data.

[0127] Based on the weighted graph data w (dimension 1920×1080×1) obtained in step S4, it is expanded to the same dimension as the optical flow data. Through a tensor copy operation, the weighted graph is copied twice along the third dimension to obtain the weight tensor. The dimensions are 1920×1080×2, which correspond to the weights of the horizontal and vertical optical flow components, respectively.

[0128] For traditional optical flow densification data and deep learning optical flow field data Pixel-wise weighted fusion is performed. To ensure the physical validity of the fusion result, constraints are imposed on the fusion process. The reasonable range for the optical flow component is set to [-50, 50] pixels / frame, and outliers outside this range are truncated. A local consistency check is used to calculate the optical flow difference between each pixel and its 8 neighboring pixels. If the difference exceeds a threshold of 10 pixels / frame, the median of the neighboring pixels is used for replacement.

[0129] Since image boundary regions lack sufficient neighborhood information, weight calculation may be unstable. For regions within 50 pixels of the image boundary, a conservative fusion strategy is adopted: the weight values ​​are converged towards 0.5 to ensure the stability of the fusion result in the boundary region.

[0130] After fusion, the results are post-processed and optimized. A 3×3 Gaussian filter is used to smooth the fused optical flow field, with a filter coefficient σ=0.8, to eliminate potential spatial discontinuities during the fusion process. The final output is fused optical flow field data with dimensions of 1920×1080×2, in 32-bit floating-point format. The two channels of the third dimension store the horizontal and vertical optical flow components, respectively.

[0131] S6. Calculate the flow velocity distribution data on the river surface using the fused optical flow field data, and output flow velocity fluctuation early warning information data based on the flow velocity distribution data.

[0132] The optical flow to actual flow velocity was calibrated and converted based on the camera's installation parameters. The camera was installed at a height of 6 meters, with a shooting angle of 37 degrees and a pixel resolution of 1920×1080. A mapping relationship between pixel displacement and actual displacement was established using geometric projection. Combined with an inter-frame time interval of 0.04 seconds, the actual flow velocity corresponding to each pixel was calculated. ,in, and These are the flow velocity components in the horizontal and vertical directions, respectively.

[0133] After converting the fused optical flow field data F_{fused} into velocity distribution data, a regular grid velocity distribution map was generated using spatial interpolation. A 1920×1080 pixel grid was mapped onto the actual river monitoring area (20 meters long, 12 meters wide) to generate a 200×120 velocity distribution grid, with each grid cell representing a 0.1m × 0.1m area. A weighted average of the velocity values ​​from multiple pixels within each grid cell was then calculated, with the weights determined based on the distance from the pixel to the grid center.

[0134] Flow velocity statistical feature extraction

[0135] Calculate key statistical features based on velocity distribution data: average velocity Flow velocity standard deviation Maximum flow rate and minimum flow rate Flow velocity gradient intensity The average value was calculated. Simultaneously, the spatial correlation of the velocity distribution was calculated, and Moran's I index was used to assess the spatial clustering characteristics of the velocity field.

[0136] Establish a multi-level early warning threshold system. Level 1 warning (green): Average flow velocity exceeds the historical average + 1 standard deviation, or the flow velocity gradient exceeds 0.5 m / s / m; Level 2 warning (yellow): Average flow velocity exceeds the historical average + 2 standard deviations, or the maximum flow velocity exceeds 3 m / s, or the flow velocity standard deviation exceeds 1.2 times the historical standard deviation; Level 3 warning (red): Average flow velocity exceeds the historical average + 3 standard deviations, or the maximum flow velocity exceeds 5 m / s, or a significant flow velocity mutation area appears (gradient > 1.0 m / s / m).

[0137] When an abnormal flow rate is detected, the system automatically generates early warning information. This information includes: timestamp, warning level, coordinates of the abnormal area, flow rate statistics, duration of the abnormality, and confidence score. The warning information is output in JSON format for easy integration with other monitoring systems. Simultaneously, a visual flow rate distribution map and a warning heatmap are generated, using pseudo-color coding to display flow rate magnitude, with abnormal areas marked with red borders.

[0138] Example 2 illustrates a schematic scheme for a river flow velocity fluctuation early warning method based on optical flow algorithms. It should be noted that the technical solution of this river flow velocity fluctuation early warning system based on optical flow algorithms is based on the same concept as the aforementioned river flow velocity fluctuation early warning method based on optical flow algorithms. Details not described in detail in the technical solution of the river flow velocity fluctuation early warning system based on optical flow algorithms in this embodiment can be found in the description of the aforementioned river flow velocity fluctuation early warning method based on optical flow algorithms.

[0139] This embodiment also provides a river flow velocity fluctuation early warning system based on optical flow algorithm, including:

[0140] The video acquisition module is used to acquire continuous frame video data;

[0141] The traditional optical flow processing module is used to process video data using traditional optical flow algorithms to obtain sparse optical flow field data, perform densification processing on the sparse optical flow field data to obtain traditional optical flow densified data, and calculate confidence mapping data.

[0142] The deep learning optical flow processing module is used to process video data using deep learning optical flow algorithms to obtain deep learning optical flow field data and calculate uncertainty data.

[0143] The intelligent fusion module is used to process traditional optical flow densification data, deep learning optical flow field data, confidence mapping data and uncertainty data through a lightweight residual-optimized weight generation network to obtain weight map data, and to fuse traditional optical flow densification data and deep learning optical flow field data according to the weight map data to obtain fused optical flow field data.

[0144] The flow velocity early warning module is used to calculate the flow velocity distribution data on the river surface through the fused optical flow field data, and output flow velocity fluctuation early warning information data based on the flow velocity distribution data.

[0145] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for early warning of river flow velocity fluctuations based on optical flow algorithm, characterized in that, include: Acquire continuous frame video data; Video data is processed using both traditional optical flow algorithms and deep learning optical flow algorithms to obtain sparse optical flow field data and deep learning optical flow field data. The sparse optical flow field data is then densified to obtain traditional optical flow densified data. Calculate the confidence mapping data of sparse optical flow field data and the uncertainty data of deep learning optical flow field data; A lightweight residual-optimized weight generation network is used to process traditional optical flow densification data, deep learning optical flow field data, confidence mapping data, and uncertainty data to obtain weight map data. Based on the weighted graph data, traditional optical flow densification data and deep learning optical flow field data are fused to obtain fused optical flow field data. The velocity distribution data on the river surface is calculated using the fused optical flow field data, and velocity fluctuation early warning information is output based on the velocity distribution data.

2. The river flow velocity fluctuation early warning method based on optical flow algorithm as described in claim 1, characterized in that, The steps for processing video data using the traditional optical flow algorithm module include: Convert two consecutive frames of video data into grayscale image data; Gaussian pyramid data is constructed using the grayscale image data, with L=5 pyramid layers and the size of each layer reduced to half that of the previous layer; The motion of feature points is tracked at each level of the Gaussian pyramid data using the pyramid Lucas-Kanade algorithm to obtain the set of motion vector data corresponding to the key points. Output sparse optical flow field data based on the set of motion vectors.

3. The river flow velocity fluctuation early warning method based on optical flow algorithm as described in claim 2, characterized in that, The steps for densifying the sparse optical flow field data include: Initialize a zero-based matrix of dimension H×W×2; Fill the effective optical flow vectors from the sparse optical flow field data into the corresponding coordinate positions; For blank pixel locations, find the nearest valid optical flow vector and fill it using nearest neighbor interpolation until the H×W matrix has been traversed; Output traditional optical flow densification data with dimensions H×W×2.

4. The river flow velocity fluctuation early warning method based on optical flow algorithm as described in claim 3, characterized in that, The steps for processing video data using the deep learning optical flow algorithm include: Convert two consecutive frames of video data into grayscale image data, and preprocess the grayscale image data; The FlowNet deep network model is used to perform end-to-end prediction on the preprocessed grayscale image data to obtain the two-dimensional displacement component data of each pixel position. The two-dimensional displacement component data are combined to form deep learning optical flow field data.

5. The river flow velocity fluctuation early warning method based on optical flow algorithm as described in claim 4, characterized in that, The steps for calculating the confidence mapping data include: Image gradient information is calculated from the grayscale image data, and the horizontal gradient is obtained using the Sobel operator. Vertical gradient and time gradient ; Based on the effective optical flow vector in the sparse optical flow field data and the image gradient information data, the tracking error data is defined as the sum of squares of the residuals of the optical flow constraint equations in the neighborhood, expressed by the formula: ； In the formula, (x,y) represents the optical flow tracking error at pixel position (x,y). For a neighborhood window centered at (x,y), Let be the horizontal grayscale gradient at pixel (i,j). Let be the vertical grayscale gradient at pixel (i,j). Let be the gray-level gradient in the time direction at pixel (i,j), i.e. u is the horizontal optical flow component at the pixel location, and v is the vertical optical flow component at the pixel location; Through the exponential decay function The tracking error data is converted into confidence weights to obtain confidence mapping data with dimensions H×W×1; in, The attenuation coefficient is... This represents the confidence mapping value at pixel (x, y). It is an exponential function.

6. The river flow velocity fluctuation early warning method based on optical flow algorithm as described in claim 5, characterized in that, The steps for calculating uncertain data include: The predicted deep learning optical flow field data is applied to the next frame of image data. The pixel difference between the distorted image data and the previous frame of image data is calculated, and the uncertainty data is output. The calculation formula is expressed as follows: ； In the formula, The uncertain data value at pixel position (x, y) represents the reliability of the prediction quality of the deep learning optical flow algorithm. Let (x, y) be the grayscale value of the reference frame image at pixel position (x, y). Let be the grayscale value of the target frame image at pixel position (x, y). Let x be the horizontal optical flow component at position (x, y) in the deep learning optical flow field data, representing the horizontal displacement of the pixel. This represents the vertical optical flow component at position (x, y) in the deep learning optical flow field data, indicating the displacement of the pixel in the vertical direction. This is the interpolated grayscale value at position (x,y) after the first frame image is inversely distorted based on the optical flow vector (u,v).

7. The river flow velocity fluctuation early warning method based on optical flow algorithm as described in claim 6, characterized in that, The steps for obtaining the weighted graph data include: Traditional optical flow densification data, deep learning optical flow field data, confidence mapping data, and uncertainty data are stitched together in four channels to obtain... As input data for the weight generation network; The weighted graph data of dimension H×W×1 is obtained by processing through a weight generation network, specifically represented as follows: ； In the formula, F_{Dense} represents the traditional optical flow densification data. For deep learning optical flow field data, For confidence level mapping data, For uncertain data, Generate network functions for the weights. This is weighted graph data.

8. The river flow velocity fluctuation early warning method based on optical flow algorithm as described in claim 7, characterized in that, The steps of the fusion process include: Based on the weighted graph data, using the formula We perform weighted fusion of deep learning optical flow field data and traditional optical flow densification data to obtain fused optical flow field data with dimensions H×W×2; In the formula, To fuse optical flow field data.

9. The river flow velocity fluctuation early warning method based on optical flow algorithm as described in claim 8, characterized in that, The lightweight residual optimization weight generation network includes: A four-layer multi-scale feature pyramid data was constructed, and the original four-layer standard residual blocks were replaced with two layers of lightweight residual units. Each lightweight residual unit contains a 1×1 convolutional channel reduced to C / 4, a 3×3 depthwise separable convolution, and a 1×1 convolutional channel restored to C; Both input and output are feature map data of size H / 16×W / 16×128, and a lightweight pixel enhancement module is embedded at the end of the residual connection. In the integral branch, a 3-stage parallel pooling is used instead of the original 4-stage serial pooling. Branch 1 performs 1×1 adaptive average pooling with the same input and output resolution. Branch 2 performs 2×2 average pooling and then upsamples to H / 64×W / 64 through bilinear interpolation. Branch 3 performs 4×4 average pooling and then upsamples to H / 64×W / 64 through bilinear interpolation.

10. A river flow velocity fluctuation early warning system based on optical flow algorithm, using the method described in any one of claims 1-9, characterized in that, include: The video acquisition module is used to acquire continuous frame video data; The traditional optical flow processing module is used to process video data using traditional optical flow algorithms to obtain sparse optical flow field data, perform densification processing on the sparse optical flow field data to obtain traditional optical flow densified data, and calculate confidence mapping data. The deep learning optical flow processing module is used to process video data using deep learning optical flow algorithms to obtain deep learning optical flow field data and calculate uncertainty data. The intelligent fusion module is used to process traditional optical flow densification data, deep learning optical flow field data, confidence mapping data and uncertainty data through a lightweight residual-optimized weight generation network to obtain weight map data, and to fuse traditional optical flow densification data and deep learning optical flow field data according to the weight map data to obtain fused optical flow field data. The flow velocity early warning module is used to calculate the flow velocity distribution data on the river surface through the fused optical flow field data, and output flow velocity fluctuation early warning information data based on the flow velocity distribution data.

Images