River flow velocity fluctuation early warning method and system based on optical flow algorithm

Abstract

The application discloses a river flow velocity fluctuation early warning method and system based on an optical flow algorithm, and comprises the following steps: collecting continuous frame video data; processing the video data through a traditional optical flow algorithm and a deep learning optical flow algorithm respectively, carrying out densification processing on sparse optical flow field data, and obtaining traditional optical flow densification data; calculating confidence mapping data of the sparse optical flow field data and uncertainty data of deep learning optical flow field data; obtaining weight map data through a lightweight residual optimization weight generation network; carrying out fusion processing on the traditional optical flow densification data and the deep learning optical flow field data according to the weight map data, and obtaining fusion optical flow field data; calculating river surface flow velocity distribution data through the fusion optical flow field data, and outputting flow velocity fluctuation early warning information data according to the flow velocity distribution data. Through the fusion of the double optical flow algorithms, the boundary keeping capability of the traditional algorithm and the dense prediction advantage of the deep learning algorithm are fully exerted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hydrological monitoring, and in particular to a river flow velocity fluctuation early warning method and system based on an optical flow algorithm. BACKGROUND

[0002] Traditional optical flow methods (such as Lucas-Kanade and Farneback algorithms) calculate flow velocity by pixel displacement between adjacent frames, but are limited by the constant brightness assumption and small motion assumption, and are prone to noise interference in complex river surface flow (such as turbulent flow and high flow velocity) and light change scenarios, resulting in dramatic fluctuations in output flow velocity. For example, the Lucas-Kanade algorithm relies on local window feature matching, which is prone to false positives due to aperture problems; although Farneback dense optical flow covers the entire global area, it has a large amount of calculation and is difficult to process high-resolution video in real time.

[0003] Deep learning algorithms represented by FlowNet2.0 predict optical flow through an end-to-end CNN network, overcoming the dependence on manual features of traditional methods, but still have abnormal output values in the case of insufficient training data or complex environments (such as water surface reflection and floating object obstruction). In addition, the model inference result needs to be combined with post-processing (such as optical flow filtering and coordinate conversion) to generate flow velocity data with physical meaning, and if the original output is used directly, it is easy to cause false alarms due to insufficient model generalization ability. SUMMARY

[0004] To solve the above technical problems, the present application provides the following technical solutions:

[0005] In a first aspect, the present application provides a river flow velocity fluctuation early warning method based on an optical flow algorithm, comprising:

[0006] collecting continuous frame video data;

[0007] processing the video data through a traditional optical flow algorithm and a deep learning optical flow algorithm respectively to obtain sparse optical flow field data and deep learning optical flow field data, and performing dense processing on the sparse optical flow field data to obtain traditional optical flow dense data;

[0008] calculating confidence mapping data of the sparse optical flow field data and uncertainty data of the deep learning optical flow field data;

[0009] processing the traditional optical flow dense data, the deep learning optical flow field data, the confidence mapping data and the uncertainty data through a lightweight residual optimized weight generation network to obtain weight map data;

[0010] performing fusion processing on the traditional optical flow dense data and the deep learning optical flow field data according to the weight map data to obtain fusion optical flow field data;

[0011] The fusion optical flow field data is used to calculate river surface flow velocity distribution data, and flow velocity fluctuation early warning information data is output according to the flow velocity distribution data.

[0012] As a preferred scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm, the step of processing the video data by the traditional optical flow algorithm module comprises:

[0013] The video data of the two continuous frames is converted into gray image data, the Gaussian pyramid data is constructed by using the gray image data, the number of layers L of the Gaussian pyramid is 5, and the size of each layer is reduced by half of the size of the previous layer; the motion of the feature points is tracked by using the pyramid Lucas-Kanade algorithm on each layer of the Gaussian pyramid data, and a set of motion vector data corresponding to the key points is obtained; and the sparse optical flow field data is output according to the set of motion vector data.

[0014] As a preferred scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm, the step of processing the video data by the traditional optical flow algorithm module comprises:

[0015] A zero matrix data with a dimension of HxWx2 is initialized; the effective optical flow vectors in the sparse optical flow field data are filled into the corresponding coordinate positions; for the blank pixel positions, the nearest effective optical flow vector is found to fill the blank pixel positions by using the nearest neighbor interpolation until the HxW matrix is traversed; and the traditional optical flow densification data with a dimension of HxWx2 is output.

[0016] As a preferred scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm, the step of processing the video data by the deep learning optical flow algorithm comprises:

[0017] The video data of the two continuous frames is converted into gray image data, and the gray image data is preprocessed; the two-dimensional displacement component data of each pixel position is obtained by using the FlowNet deep network model to perform end-to-end prediction on the preprocessed gray image data; and the deep learning optical flow field data is formed by combining the two-dimensional displacement component data.

[0018] As a preferred scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm, the step of calculating the confidence mapping data comprises:

[0019] The image gradient information data is calculated by using the gray image data, and the horizontal direction gradient , the vertical direction gradient and the time direction gradient are obtained by using the Sobel operator, respectively.

[0020] Based on the effective optical flow vector in the sparse optical flow field data and the image gradient information data, a tracking error data is defined as a sum of squares of residual errors of neighborhood optical flow constraint equations, and is expressed by a formula as follows:

[0021] ;

[0022] In the formula, (x,y) is a pixel position (x,y) optical flow tracking error, is a neighborhood window with (x,y) as the center, is a horizontal direction gray gradient at pixel (i,j), is a vertical direction gray gradient at pixel (i,j), is a time direction gray gradient at pixel (i,j), that is u is a horizontal optical flow component of the pixel position, and v is a vertical optical flow component of the pixel position;

[0023] The tracking error data is converted into a confidence weight by an exponential decay function The tracking error data is converted into a confidence weight by an exponential decay function

[0024] In the formula, is a decay coefficient, is a confidence mapping value at pixel (x,y), is an exponential function.

[0025] The beneficial effects of the preferred technical scheme are that the confidence of the traditional optical flow algorithm is evaluated by calculating the residual error of the optical flow constraint equation, and the problem of difficulty in reliability evaluation of the traditional optical flow in the texture missing and occlusion area is solved. The method can accurately identify the area with poor optical flow estimation quality, provide reliable quality guidance for subsequent fusion processing, and improve the applicability of the traditional optical flow algorithm in the complex water surface environment.

[0026] As a preferred scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm, the step of calculating the uncertainty data comprises:

[0027] The predicted deep learning optical flow field data is applied to the next frame image data, the pixel difference between the warped image data and the previous frame image data is calculated, and the uncertainty data is output, and the calculation formula is expressed as:

[0028] ;

[0029] In the formula, is a pixel position (x,y) uncertainty data value, representing the reliability degree of the prediction quality of the deep learning optical flow algorithm, is a gray value of the reference frame image at pixel position (x,y), is the gray value of the target frame image at the pixel position (x, y), is the horizontal optical flow component of the deep learning optical flow field data at position (x, y), representing the displacement amount of the pixel in the horizontal direction, is the vertical optical flow component of the deep learning optical flow field data at position (x, y), representing the displacement amount of the pixel in the vertical direction, is the interpolated gray value at position (x, y) after the first frame image is inversely warped according to the optical flow vector (u, v).

[0030] The beneficial effects of the preferred technical solution are that the uncertainty of the deep learning algorithm is evaluated based on the temporal consistency reconstruction error, effectively solving the problems of lack of explainability and reliability evaluation of the deep learning method.

[0031] As a preferred scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm, the weight map data acquisition step comprises:

[0032] The traditional optical flow densification data, the deep learning optical flow field data, the confidence mapping data and the uncertainty data are four-channel spliced to obtain as input data of the weight generation network;

[0033] The weight map data with the dimension of HxWx1 is obtained by processing through the weight generation network, and is specifically represented as:

[0034]

[0035] In the formula, is the traditional optical flow densification data, is the deep learning optical flow field data, is the confidence mapping data, is the uncertainty data, is the weight generation network function, is the weight map data.

[0036] The beneficial effects of the preferred technical solution are that the dual optical flow results and the corresponding quality evaluation data are four-channel spliced to construct a multi-modal feature input mechanism, so that the weight generation network can simultaneously perceive the optical flow prediction results and the quality evaluation information. This design improves the network's ability to judge the quality of optical flow in different regions and realizes accurate pixel-level weight distribution.

[0037] As a preferred scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm, the fusion processing step comprises:

[0038] According to the weight map data, the formula ​The deep learning optical flow field data and the traditional optical flow densification data are weighted and fused to obtain fusion optical flow field data with a dimension of H*W*2.

[0039] In the formula, The fusion optical flow field data is obtained.

[0040] As a preferred scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm, the lightweight residual optimization weight generation network comprises:

[0041] A 4-layer multi-scale feature pyramid data is constructed, and the original 4-layer standard residual block is replaced by a 2-layer lightweight residual unit.

[0042] Each lightweight residual unit comprises 1*1 convolution channel dimension reduction to C / 4, 3*3 depth separable convolution, and 1*1 convolution channel recovery to C.

[0043] The input and output are both feature map data with a size of H / 16*W / 16*128, and a lightweight pixel enhancement module is embedded at the end of the residual connection.

[0044] In the integral branch, 3-level parallel pooling is used to replace the original 4-level serial pooling, branch 1 performs 1*1 adaptive average pooling with the same input and output resolution, branch 2 performs 2*2 average pooling and then is up-sampled to H / 64*W / 64 through bilinear interpolation, and branch 3 performs 4*4 average pooling and then is up-sampled to H / 64*W / 64 through bilinear interpolation.

[0045] The preferred technical scheme has the beneficial effects that through lightweight residual optimization and multi-scale parallel processing architecture, the calculation complexity is greatly reduced under the premise of ensuring fusion accuracy.

[0046] In a second aspect, the application provides a river flow velocity fluctuation early warning system based on an optical flow algorithm, comprising:

[0047] A video acquisition module is configured to acquire continuous frame video data.

[0048] A traditional optical flow processing module is configured to process the video data through a traditional optical flow algorithm to obtain sparse optical flow field data, perform densification processing on the sparse optical flow field data to obtain traditional optical flow densification data, and calculate confidence mapping data.

[0049] A deep learning optical flow processing module is configured to process the video data through a deep learning optical flow algorithm to obtain deep learning optical flow field data, and calculate uncertainty data.

[0050] The intelligent fusion module is used for obtaining weight map data by processing traditional optical flow densification data, deep learning optical flow field data, confidence mapping data and uncertainty data through a lightweight residual optimization weight generation network, and fusing the traditional optical flow densification data and the deep learning optical flow field data according to the weight map data to obtain fused optical flow field data.

[0051] The flow rate warning module is used for calculating flow rate distribution data of a river surface through the fused optical flow field data, and outputting flow rate fluctuation warning information data according to the flow rate distribution data.

[0052] Compared with the prior art, the present application has the following beneficial effects:

[0053] The traditional optical flow algorithm is prone to failure in texture missing areas, and the deep learning algorithm has a blur problem in boundary detail processing. The present application fully utilizes the boundary preservation ability of the traditional algorithm and the dense prediction advantage of the deep learning algorithm through the fusion of the dual optical flow algorithm, realizes the complementary advantages, and significantly improves the robustness and accuracy of optical flow estimation under complex water surface environment.

[0054] In view of the problem that the optical flow estimation result lacks reliability evaluation, the present application innovatively proposes a dual quality evaluation mechanism of confidence mapping and uncertainty evaluation. Through the optical flow constraint equation residual and the time consistency reconstruction error, the reliable area and the unreliable area of the optical flow estimation can be accurately identified, and scientific quality guidance basis is provided for intelligent fusion. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Figure 1 The whole flow chart of the river flow rate fluctuation warning method based on the optical flow algorithm is shown. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0058] Embodiment 1, refer to Figure 1For an embodiment of the present application, a river flow velocity fluctuation early warning method based on an optical flow algorithm is provided, comprising:

[0059] S1, collecting continuous frame video data;

[0060] In this embodiment, high-resolution waterproof camera equipment is used for video data collection at the river monitoring point. The specific implementation steps are as follows:

[0061] Device configuration: an industrial-grade waterproof camera with a resolution of 1920x1080 pixels is selected, equipped with an automatic focusing function and an anti-shake mechanism, installed on a fixed support on the river bank, the camera distance from the water surface is 5-8 meters, the shooting angle is 30-45 degrees with the water surface, ensuring that the clear water surface texture and flow characteristics can be captured.

[0062] Collection parameter setting: the video frame rate is set to 25fps to ensure sufficient time correlation between adjacent frames for optical flow calculation; the image format uses RGB24-bit true color mode; the exposure time is adaptively adjusted according to the environmental light conditions, set to 1 / 60 seconds during the day and 1 / 30 seconds at night or in low light conditions; the ISO sensitivity is controlled within the range of 100-800 to ensure image clarity.

[0063] Data preprocessing: the continuously collected video stream is labeled and stored according to the timestamp, each video segment is 10 seconds long and contains 250 image data; the original RGB image is denoised using a Gaussian filter to remove sensor noise; abnormal frames with overexposure or underexposure are detected and removed to ensure the quality stability of the input data.

[0064] S2, processing the video data through a traditional optical flow algorithm and a deep learning optical flow algorithm respectively to obtain sparse optical flow field data and deep learning optical flow field data, and performing densification processing on the sparse optical flow field data to obtain traditional optical flow densification data;

[0065] The step of processing the video data through the traditional optical flow algorithm module includes S2.1-S2.4:

[0066] S2.1, converting the video data of two consecutive frames into gray image data;

[0067] Extracting two adjacent frames of images from the continuous video stream collected in step S1 and , the size of the RGB color image is 1920x1080x3. The weighted average method is used for gray scale conversion: Gray = 0.299xR + 0.587xG + 0.114xB, to obtain a 1920x1080 single-channel gray image. The converted gray image is subjected to histogram equalization processing to enhance the image contrast and improve the accuracy of feature point detection.

[0068] S2.2, constructing a Gaussian pyramid data by the grayscale image data, pyramid layer number L=5, and the size of each layer is reduced by half of the previous layer;

[0069] The grayscale image obtained in step S2.1 is taken as a base layer (L=0), and a Gaussian kernel After the convolution filtering, the pyramid is constructed by down-sampling at a ratio of 2:1. The specific layer size is: L0 layer 1920x1080, L1 layer 960x540, L2 layer 480x270, L3 layer 240x135, and L4 layer 120x67. Each layer image is preprocessed by a 5x5 Gaussian filter before down-sampling to prevent aliasing.

[0070] S2.3, tracking the motion of feature points on each level of the Gaussian pyramid data by a pyramid Lucas-Kanade algorithm to obtain a motion vector set data corresponding to key points;

[0071] In the L4 highest layer, an Shi-Tomasi corner detector is used to extract initial feature points, the detection threshold is set to 0.01, the minimum feature point spacing is set to 10 pixels, and the maximum number of feature points is limited to 500. A Lucas-Kanade optical flow tracking algorithm is used, the search window size is set to 15x15 pixels, and the iteration termination condition is that the maximum iteration number is 30 times or the convergence accuracy reaches 0.03 pixels. Starting from the highest layer, the tracking results are propagated layer by layer downward, and the initial estimate value of each layer is 2 times magnification of the result of the upper layer.

[0072] S2.4, outputting sparse optical flow field data according to the motion vector set data.

[0073] The motion vector sets obtained by tracking each layer are integrated, and the feature points that fail to track (the points with tracking error greater than 2.0 pixels) are removed, and finally the motion vector data of 300-400 effective feature points is obtained. Each motion vector contains coordinate position (x, y) and displacement component (u, v), forming a sparse optical flow field data structure: .

[0074] The steps of densifying the sparse optical flow field data include S2.41-S2.44:

[0075] S2.41, initializing a full zero matrix data with a dimension of HxWx2;

[0076] creating a three-dimensional array with a dimension of 1920x1080x2 , two channels of the third dimension store the horizontal and vertical components of the optical flow, respectively, and all elements are initialized to 0. At the same time, a 1920x1080 binary mask matrix Mask is created to mark the filled pixel positions.

[0077] S2.42, fill the valid optical flow vectors in the sparse optical flow field data into the corresponding coordinate positions;

[0078] S2.42, fill the valid optical flow vectors in the sparse optical flow field data into the corresponding coordinate positions; array, and mark Mask(x, y) as 1 to indicate that the position has been filled.

[0079] S2.43, for the blank pixel positions, find the nearest valid optical flow vector to fill by nearest neighbor interpolation until the HxW matrix is traversed.

[0080] For the blank pixel positions with Mask value of 0, the K-D tree algorithm is used to quickly search for the nearest filled pixel point. Calculate the Euclidean distance , select the optical flow value of the nearest filled point to assign. To avoid error propagation caused by excessive interpolation, set the maximum search radius to 50 pixels, and set the pixels beyond the range to zero vector.

[0081] S2.44, output the traditional optical flow densification data with dimensions HxWx2.

[0082] After completing the full matrix traversal, the densification result is subjected to median filtering processing, and the filtering window size is 3x3 to remove abnormal values that may be generated in the interpolation process. Finally, the traditional optical flow densification data with dimensions 1920x1080x2 is output .

[0083] The steps of processing video data by the deep learning optical flow algorithm include S2.5~S2.7:

[0084] S2.5, convert the continuous two frames of video data into grayscale image data, and pre-process the grayscale image data;

[0085] The same method as step S2.1 is used to convert the continuous two frames of RGB images into grayscale images. Standardize the grayscale images: normalize the pixel values to the range [0, 1], center the mean value (subtract 0.5), and adjust the data format to the tensor form required by the deep network [2, 1, 1080, 1920].

[0086] S2.6, perform end-to-end prediction on the pre-processed grayscale image data by the FlowNet deep network model to obtain two-dimensional displacement component data for each pixel position;

[0087] Load the pre-trained FlowNet-C network model, which adopts an encoder-decoder structure and can process input of arbitrary resolution. Input the preprocessed image pair into the network, and calculate the two-dimensional optical flow prediction value of each pixel position through forward propagation. The network output is a [1, 2, 1080, 1920] dimensional tensor, where the two channels of the second dimension correspond to the horizontal and vertical optical flow components, respectively.

[0088] S2.7, combine the two-dimensional displacement component data to form deep learning optical flow field data.

[0089] Convert the network output tensor to the standard optical flow field data format, and rearrange the dimensions to 1920x1080x2. Post-process the output results: restore the optical flow value from the network's normalized range to the actual pixel displacement range, and perform smoothing processing on the boundary pixels. Finally, obtain the dense deep learning optical flow field data , which naturally covers all pixel positions.

[0090] S3, calculate the confidence mapping data of the sparse optical flow field data and the uncertainty data of the deep learning optical flow field data;

[0091] The step of calculating the confidence mapping data includes S3.1-S3.3:

[0092] S3.1, calculate image gradient information data from the gray image data, and obtain horizontal direction gradient , vertical direction gradient and time direction gradient by Sobel operator respectively;

[0093] Based on the gray image data obtained in step S2.1, calculate the gradient information in three directions using Sobel operator. The horizontal direction gradient uses kernel: [[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]; the vertical direction gradient uses kernel: [[-1, -2, -1], [0, 0, 0], [1, 2, 1]]. Convolution operation is performed on the gray image to obtain horizontal direction gradient Ix(i,j) and vertical direction gradient Iy(i,j). The time direction gradient is calculated by the difference value of corresponding pixels of adjacent two frames: , where Δt is the inter-frame time interval 0.04 seconds. In order to avoid numerical instability in gradient calculation, the gradient value is truncated to be limited in the range of [-255, 255].

[0094] S3.2, based on the valid optical flow vectors in the sparse optical flow field data and the image gradient information data, define tracking error data as the sum of squares of residual errors of optical flow constraint equations in the neighborhood, expressed by the formula as follows:

[0095] ;

[0096] In the formula, (x, y) is the optical flow tracking error at pixel position (x, y), is the neighborhood window centered at (x, y), is the horizontal direction gray gradient at pixel (i, j), is the vertical direction gray gradient at pixel (i, j), is the time direction gray gradient at pixel (i, j), that is, u is the horizontal optical flow component of the pixel position, and v is the vertical optical flow component of the pixel position;

[0097] Based on the valid optical flow vectors in the sparse optical flow field data obtained in step S2.4, combined with the image gradient information calculated in step S3.1, the tracking error of each feature point is calculated. A 5x5 pixel neighborhood window W(x, y) is defined with each feature point as the center, and the sum of squares of residual errors of optical flow constraint equations is calculated in the window. In specific implementation, for the feature point at position (x, y), the optical flow component is (u, v), and the 25 pixel points in the neighborhood window are traversed, the constraint equation residual error at each pixel point (i, j) is calculated: Then square and accumulate to get For the feature points in the boundary region, the neighborhood window is extended by using the boundary padding strategy.

[0098] S3.3, convert the tracking error data into confidence weight by an exponential decay function to obtain the confidence mapping data with the dimension of HxWx1;

[0099] Wherein, is the decay coefficient, is the confidence mapping value at pixel (x, y), is the exponential function;

[0100] The tracking error is converted into confidence weight using an exponential decay function, and the decay coefficient λ is set to 0.01, which is optimized through a large amount of experimental data. For each feature point position (x, y), the confidence value is calculated. Since sparse optical flow only has values at feature point positions, it is necessary to extend the confidence to the full image. The same nearest neighbor interpolation method as S2.43 is used to propagate the confidence value of the feature point to the entire 1920x1080 image area, generating a confidence map data with dimensions HxWx1. The interpolation result is Gaussian smoothed with a standard deviation σ = 2.0 to ensure the continuity and reasonableness of the confidence distribution.

[0101] The step of calculating uncertainty data includes:

[0102] The predicted deep learning optical flow field data is applied to the next frame image data, and the pixel difference between the warped image data and the previous frame image data is calculated, and the uncertainty data is output, and the calculation formula is represented as:

[0103] ;

[0104] In the formula, is the uncertainty data value at pixel position (x, y), representing the reliability of the prediction quality of the deep learning optical flow algorithm, is the gray value of the reference frame image at pixel position (x, y), is the gray value of the target frame image at pixel position (x, y), is the horizontal optical flow component of the deep learning optical flow field data at position (x, y), representing the displacement amount of the pixel in the horizontal direction, is the vertical optical flow component of the deep learning optical flow field data at position (x, y), representing the displacement amount of the pixel in the vertical direction, is the interpolated gray value at position (x, y) after the first frame image is inversely warped according to the optical flow vector (u, v).

[0105] Based on the deep learning optical flow field data obtained in step S2.7 , the reference frame image is inversely warped. For each pixel position (x, y), the source pixel position (x+u, y+v) is calculated according to the optical flow vector (u(x, y), v(x, y)). Since the calculated source position is usually a non-integer coordinate, a bilinear interpolation method is used to obtain the interpolated gray value . For positions outside the image boundary, a boundary replication strategy is used for filling.

[0106] The warped reference frame image and the target frame image are compared pixel by pixel, and the L1 norm difference is calculated: To improve the computational efficiency, vectorization operation is used to process the whole image at the same time. Post-processing is performed on the calculation results: first, outlier detection is performed to truncate the pixel values greater than 3 times the standard deviation to a reasonable range; then, 3x3 median filtering is used to remove isolated noise points; finally, the uncertainty value is normalized to the range [0, 1] to facilitate subsequent network processing.

[0107] The final output dimension is 1920x1080x1 confidence map data and uncertainty data with dimension 1920x1080x1 Both data are stored in 32-bit floating-point format to ensure numerical accuracy. In the confidence map data, the closer the value is to 1, the more reliable the prediction of the traditional optical flow algorithm at that position; in the uncertainty data, the closer the value is to 0, the more accurate the prediction of the deep learning algorithm at that position.

[0108] S4, the weight generation network optimized by light residual generates weight map data by processing traditional optical flow densification data, deep learning optical flow field data, confidence map data and uncertainty data;

[0109] The steps for obtaining the weight map data include:

[0110] The traditional optical flow densification data, deep learning optical flow field data, confidence map data and uncertainty data are four-channel spliced to obtain as input data of the weight generation network;

[0111] The weight map data with dimension HxWx1 is obtained by processing the weight generation network, which is specifically represented as:

[0112]

[0113] In the formula, is the traditional optical flow densification data, is the deep learning optical flow field data, is the confidence map data, is the uncertainty data, is the weight generation network function, is the weight map data.

[0114] In this embodiment, the four kinds of data obtained by steps S2 and S3 are preprocessed to ensure compatibility. The traditional optical flow densification data has a dimension of 1920x1080x2, and the optical flow amplitude needs to be calculated as a single-channel input: , obtaining 1920x1080x1 data. The deep learning optical flow field data also calculates the amplitude to obtain ​, with dimension 1920x1080x1. Confidence map data and uncertainty data have been formatted into single-channel, with dimension 1920x1080x1.

[0115] To ensure the stability and convergence of network training, the four kinds of data are normalized respectively: normalized to the range [0, 1], and the maximum optical flow amplitude is set to 50 pixels / frame; The same normalization strategy is adopted; The data has been in the range (0, 1], and no additional processing is required; The data is mapped to the range [0, 1] by min-max normalization.

[0116] Tensor concatenation operation is used to concatenate the four kinds of data along the channel dimension: , resulting in an input tensor with dimension 1920x1080x4. The concatenation order is arranged according to the importance of the data: the 0th channel is the traditional optical flow amplitude, the 1st channel is the deep learning optical flow amplitude, the 2nd channel is the confidence map, and the 3rd channel is the uncertainty data.

[0117] The weight generation network adopts an encoder-decoder architecture, the encoder contains 4 layers of down-sampling modules, and the decoder contains 4 layers of up-sampling modules. The input size of the network is 1920x1080x4, to adapt to the GPU memory limit, the input data is divided into 480x270x4 overlapping windows for block processing, and the overlapping area is 32 pixels wide.

[0118] The first layer of the encoder: use 3x3 convolution kernel, input channel 4, output channel 32, step 2, padding 1, activation function ReLU; the output size is 960x540x32. The second layer of the encoder: 3x3 convolution, input 32 channels, output 64 channels, step 2; the output size is 480x270x64. The third and fourth layers of the encoder output 128 and 256 channel feature maps respectively, and the final encoding feature size is 120x67x256.

[0119] A lightweight residual unit is added after each layer of the encoder. The residual unit structure is: 1x1 convolution reduces the channel number to C / 4; 3x3 depth separable convolution is used for spatial feature extraction, which decomposes the standard convolution into depth convolution and pointwise convolution, greatly reducing the parameter amount; finally, 1x1 convolution is used to restore the original channel number C. The residual connection uses identity mapping, and the output is the element-wise addition of the input and the residual unit output.

[0120] The decoder uses transpose convolution for up-sampling, and each layer is up-sampled by a factor of 2. The number of channels of each layer of the decoder is 128, 64, 32, and 1, which is symmetrically designed with the encoder. In the decoding process, a skip connection is used to concatenate the feature maps of the corresponding layers of the encoder and the decoder to enhance the detail reconstruction capability.

[0121] The last layer of the network uses a 1x1 convolution to map the feature map to a single-channel output, and the Sigmoid activation function ensures that the weight value is in the range (0, 1). The results of the block processing are spliced and reorganized, and the overlapping area uses a weighted average fusion strategy, with the weight being allocated according to the distance from the boundary. The final output dimension is 1920x1080x1 weight map data w.

[0122] Smooth the weight map data: use a 5x5 Gaussian kernel for convolution, with a standard deviation σ = 1.5, to eliminate block effects and outliers. Adjust the dynamic range of the weight value: w_adjusted = 0.1 + 0.8 × w, to ensure that the weight value is in the range [0.1, 0.9], avoiding complete bias towards a certain algorithm result. The final weight map data w indicates the fusion weight distribution of the traditional algorithm and the deep learning algorithm at each pixel position.

[0123] S5, according to the weight map data, fusing the traditional optical flow densification data and the deep learning optical flow field data to obtain fused optical flow field data;

[0124] The step of fusing includes:

[0125] According to the weight map data, by formula weighting fusion of the deep learning optical flow field data and the traditional optical flow densification data to obtain fused optical flow field data with a dimension of HxWx2;

[0126] wherein, is the fused optical flow field data.

[0127] Based on the weight map data w (dimension 1920x1080x1) obtained in step S4, it is extended to the same dimension as the optical flow data. By tensor replication operation, the weight map is copied twice along the third dimension to obtain the weight tensor with a dimension of 1920x1080x2, corresponding to the weight of the horizontal and vertical optical flow components respectively.

[0128] The traditional optical flow densification data and the deep learning optical flow field data The pixel-by-pixel weighted fusion is performed. To ensure the physical rationality of the fusion result, the fusion process is constrained. The reasonable range of the optical flow component is set to [-50, 50] pixels / frame, and the abnormal values exceeding the range are truncated. The local consistency test is adopted to calculate the optical flow difference between each pixel and its 8-neighborhood pixels. If the difference exceeds the threshold of 10 pixels / frame, the median value in the neighborhood is used for replacement.

[0129] Due to the lack of sufficient neighborhood information in the image boundary region, the weight calculation may be unstable. For the region within a distance of 50 pixels from the image boundary, a conservative fusion strategy is adopted: the weight value converges to 0.5, ensuring the stability of the fusion result in the boundary region.

[0130] After the fusion is completed, the result is optimized by post-processing. A 3x3 Gaussian filter is used to smooth the fused optical flow field, with a filter coefficient σ = 0.8, to eliminate the spatial discontinuity that may occur during the fusion process. The final output dimension of the fused optical flow field data is 1920x1080x2, with a data format of 32-bit floating point numbers, and the two channels in the third dimension store the horizontal and vertical optical flow components, respectively.

[0131] S6, calculate the flow velocity distribution data of the river surface through the fused optical flow field data, and output the flow velocity fluctuation warning information data according to the flow velocity distribution data.

[0132] Calibration conversion of optical flow to actual flow velocity according to camera installation parameters. The camera installation height is 6 meters, the shooting angle is 37 degrees, and the pixel resolution is 1920x1080. The mapping relationship between pixel displacement and actual displacement is established through geometric projection relationship. Combined with the inter-frame time interval of 0.04 seconds, the actual flow velocity corresponding to each pixel is calculated: wherein, and are the horizontal and vertical flow velocity components, respectively.

[0133] After converting the fused optical flow field data into flow velocity distribution data, a spatial interpolation method is used to generate a regular grid flow velocity distribution map. The 1920x1080 pixel grid is mapped to the actual river monitoring area (20 meters long and 12 meters wide), generating a 200x120 flow velocity distribution grid, with each grid cell representing a 0.1 meter x 0.1 meter area. The multiple pixel flow velocity values in each grid cell are weighted and averaged, with the weight determined by the distance from the pixel to the grid center.

[0134] Flow velocity statistical feature extraction

[0135] Based on the flow velocity distribution data, the key statistical features are calculated: average flow velocity , flow velocity standard deviation , maximum flow velocity and minimum flow velocity average value of the flow velocity gradient The spatial correlation of the flow velocity distribution is calculated simultaneously, and the spatial aggregation characteristics of the flow velocity field are evaluated using Moran's I index.

[0136] A multi-level early warning threshold system is established. First-level warning (green): average flow velocity exceeds historical mean + 1 standard deviation, or flow velocity gradient exceeds 0.5 m / s / m; second-level warning (yellow): average flow velocity exceeds historical mean + 2 standard deviations, or maximum flow velocity exceeds 3 m / s, or flow velocity standard deviation exceeds 1.2 times the historical standard deviation; third-level warning (red): average flow velocity exceeds historical mean + 3 standard deviations, or maximum flow velocity exceeds 5 m / s, or there is an obvious flow velocity mutation area (gradient > 1.0 m / s / m).

[0137] When flow velocity anomalies are detected, the system automatically generates early warning information data. The early warning information includes: timestamp, warning level, abnormal area position coordinates, flow velocity statistical parameters, abnormal duration, confidence score, etc. The early warning information is output in JSON format, which is convenient for integration with other monitoring systems. At the same time, a visual flow velocity distribution map and early warning heat map are generated, and the flow velocity size is displayed using pseudo-color coding, and the abnormal area is marked with a red border.

[0138] Embodiment 2, the above is a schematic scheme of a river flow velocity fluctuation early warning method based on optical flow algorithm. It should be noted that the technical scheme of the river flow velocity fluctuation early warning system based on the optical flow algorithm belongs to the same concept as the technical scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm described above. The technical scheme of the river flow velocity fluctuation early warning system based on the optical flow algorithm in this embodiment is not described in detail, and can be referred to the description of the technical scheme of the river flow velocity fluctuation early warning method based on the optical flow algorithm.

[0139] The embodiment also provides a river flow velocity fluctuation early warning system based on an optical flow algorithm, comprising:

[0140] a video acquisition module for acquiring continuous frame video data;

[0141] a traditional optical flow processing module for processing video data through a traditional optical flow algorithm to obtain sparse optical flow field data, performing densification processing on the sparse optical flow field data to obtain traditional optical flow densification data, and calculating confidence mapping data;

[0142] a deep learning optical flow processing module for processing video data through a deep learning optical flow algorithm to obtain deep learning optical flow field data, and calculating uncertainty data;

[0143] The intelligent fusion module is configured to generate weight map data by processing traditional optical flow densification data, deep learning optical flow field data, confidence map data and uncertainty data through a lightweight residual optimization weight generation network, and to fuse the traditional optical flow densification data and the deep learning optical flow field data according to the weight map data to obtain fused optical flow field data.

[0144] The flow rate warning module is configured to calculate flow rate distribution data of a river surface by using the fused optical flow field data, and to output flow rate fluctuation warning information data according to the flow rate distribution data.

[0145] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. A river flow velocity fluctuation early warning method based on an optical flow algorithm, characterized in that, The method comprises the following steps: Collecting continuous frame video data; Processing the video data through a traditional optical flow algorithm and a deep learning optical flow algorithm respectively to obtain sparse optical flow field data and deep learning optical flow field data, and performing dense processing on the sparse optical flow field data to obtain traditional optical flow dense data; Calculating confidence mapping data of the sparse optical flow field data and uncertainty data of the deep learning optical flow field data; Processing the traditional optical flow dense data, the deep learning optical flow field data, the confidence mapping data and the uncertainty data through a lightweight residual optimized weight generation network to obtain weight map data; Fusing the traditional optical flow dense data and the deep learning optical flow field data according to the weight map data to obtain fused optical flow field data; Calculating river surface flow velocity distribution data through the fused optical flow field data, and outputting flow velocity fluctuation early warning information data according to the flow velocity distribution data.

2. The river flow velocity fluctuation early warning method based on the optical flow algorithm of claim 1, wherein, The step of processing the video data through the traditional optical flow algorithm module comprises the following steps: Converting the video data of two continuous frames into grayscale image data; Constructing Gaussian pyramid data through the grayscale image data, the number of pyramid layers L=5, and the size of each layer is reduced by half of the previous layer; Tracking feature point motion on each level of the Gaussian pyramid data through the pyramid Lucas-Kanade algorithm to obtain a set of motion vector data corresponding to the key points; Outputting sparse optical flow field data according to the set of motion vector data.

3. The river flow velocity fluctuation early warning method based on the optical flow algorithm of claim 2, wherein, The step of performing dense processing on the sparse optical flow field data comprises the following steps: Initializing a full zero matrix data with a dimension of HxWx2; Filling the effective optical flow vectors in the sparse optical flow field data into the corresponding coordinate positions; For blank pixel positions, finding the nearest effective optical flow vector to fill through nearest neighbor interpolation until the HxW matrix is traversed; Outputting traditional optical flow dense data with a dimension of HxWx2.

4. The river flow velocity fluctuation early warning method based on the optical flow algorithm of claim 3, wherein, The step of processing the video data through the deep learning optical flow algorithm comprises the following steps: Converting the video data of two continuous frames into grayscale image data, and preprocessing the grayscale image data; Performing end-to-end prediction on the preprocessed grayscale image data through a FlowNet deep network model to obtain two-dimensional displacement component data of each pixel position; Combining the two-dimensional displacement component data to form deep learning optical flow field data.

5. The river flow velocity fluctuation early warning method based on the optical flow algorithm of claim 4, wherein, The step of calculating the confidence mapping data comprises the following steps: Calculating image gradient information data from the gray scale image data, obtaining horizontal direction gradient , vertical direction gradient and time direction gradient by Sobel operator respectively; Based on the effective optical flow vectors in the sparse optical flow field data and the image gradient information data, defining tracking error data as the sum of squares of residual errors of optical flow constraints in the neighborhood, which is represented by the formula as follows: ； wherein (x,y) is the optical flow tracking error at pixel position (x,y), is a neighborhood window centered at (x,y), is the horizontal direction gray scale gradient at pixel (i,j), is the vertical direction gray scale gradient at pixel (i,j), is the temporal direction gray scale gradient at pixel (i,j), i.e. u is the horizontal optical flow component at this pixel position and v is the vertical optical flow component at this pixel position. by an exponential decay function convert the tracking error data into a confidence weight to obtain confidence map data with a dimension of HxWx1; wherein, is an attenuation coefficient, is a confidence map value at pixel (x, y), is an exponential function.

6. The river flow velocity fluctuation early warning method based on the optical flow algorithm of claim 5, wherein, The step of calculating the uncertainty data comprises the following steps: Applying the predicted deep learning optical flow field data to the next frame image data, calculating the pixel difference between the distorted image data and the previous frame image data, and outputting the uncertainty data, which is represented by the formula as follows: ； wherein, is the uncertainty data value at pixel position (x, y) representing the reliability degree of the prediction quality of the deep learning optical flow algorithm, is the gray value of the reference frame image at pixel position (x, y), is the gray value of the target frame image at pixel position (x, y), is the horizontal optical flow component of the deep learning optical flow field data at position (x, y) representing the displacement amount of the pixel in the horizontal direction, is the vertical optical flow component of the deep learning optical flow field data at position (x, y) representing the displacement amount of the pixel in the vertical direction, is the interpolated gray value at position (x, y) after the first frame image is inversely warped according to the optical flow vector (u, v).

7. The river flow velocity fluctuation early warning method based on the optical flow algorithm of claim 6, wherein, The step of obtaining the weight map data comprises the following steps: The four-channel splicing of traditional optical flow densification data, deep learning optical flow field data, confidence mapping data and uncertainty data is obtained as input data for the weight generation network Processing through the weight generation network to obtain weight map data with a dimension of HxWx1, which is specifically represented as follows: ； wherein, is traditional optical flow densification data, is deep learning optical flow field data, is confidence map data, is uncertainty data, is a weight generation network function, is weight map data.

8. The river flow velocity fluctuation early warning method based on the optical flow algorithm of claim 7, wherein, The step of fusing comprises the following steps: According to the weight map data, by formula The deep learning optical flow field data and the traditional optical flow densification data are weighted and fused to obtain the fused optical flow field data with a dimension of HxWx2. In the formula, is the fused optical flow field data.

9. The river flow velocity fluctuation early warning method based on the optical flow algorithm of claim 8, wherein, The lightweight residual optimized weight generation network comprises the following steps: Constructing a 4-layer multi-scale feature pyramid data, and replacing the original 4-layer standard residual block with a 2-layer lightweight residual unit; Each layer of lightweight residual unit contains 1x1 convolution channel dimension reduction to C / 4, 3x3 depth separable convolution, 1x1 convolution channel recovery to C; The input and output are both feature map data with a size of H / 16xW / 16x128, and a lightweight pixel enhancement module is embedded at the end of the residual connection; In the integral branch, 3-level parallel pooling is used to replace the original 4-level serial pooling, branch 1 performs 1x1 adaptive average pooling with the same input and output resolution, branch 2 performs 2x2 average pooling and is up-sampled to H / 64xW / 64 through bilinear interpolation, and branch 3 performs 4x4 average pooling and is up-sampled to H / 64xW / 64 through bilinear interpolation.

10. A river flow fluctuation warning system based on optical flow algorithm, applying the method of any one of claims 1-9, characterized in that, The method comprises the following steps: a video acquisition module is configured to acquire continuous frame video data; a traditional optical flow processing module is configured to process the video data through a traditional optical flow algorithm to obtain sparse optical flow field data, perform dense processing on the sparse optical flow field data to obtain traditional optical flow dense data, and calculate confidence mapping data; a deep learning optical flow processing module is configured to process the video data through a deep learning optical flow algorithm to obtain deep learning optical flow field data, and calculate uncertainty data; an intelligent fusion module is configured to process the traditional optical flow dense data, the deep learning optical flow field data, the confidence mapping data and the uncertainty data through a lightweight residual optimization weight generation network to obtain weight map data, and perform fusion processing on the traditional optical flow dense data and the deep learning optical flow field data according to the weight map data to obtain fusion optical flow field data; a flow velocity early warning module is configured to calculate river surface flow velocity distribution data through the fusion optical flow field data, and output flow velocity fluctuation early warning information data according to the flow velocity distribution data.

Images