River surface space-time image validity identification and speed measurement method based on deep learning

Abstract

The invention discloses a river surface space-time image validity identification and speed measurement method based on deep learning, and the method comprises the steps: collecting a natural river video in a model training stage, and synthesizing and screening out space-time images STI of different scenes; dividing the textural features into effective, unreliable and invalid three classes according to whether the textural features can represent the actual flow velocity, and further training a classification network model; semi-automatic marking of texture principal direction MOT is carried out on the effective STI, data expansion is carried out by adopting overturning, rotating and color jitter modes to construct an MOT regression data set, and a regression network model combining grouped convolution and a convolution attention module optimization residual module is trained; in the flow velocity measurement stage, the classification network model is identified as effective STI, MOT is detected by applying a regression network model and converted into flow velocity, and interpolation correction is performed on unreliable, invalid and blind area flow velocity in combination with a section flow velocity distribution law to obtain a section flow velocity field. According to the invention, after training is completed, accurate flow measurement and flow velocity correction can be realized without manual parameter adjustment and addition of an experimental river data set.

Description

Technical Field

[0001] The present invention relates to the technical field of flow measurement using an image method, and in particular to a method for validating and measuring the speed of a river surface spatiotemporal image based on deep learning. Background Art

[0002] Timely and accurate acquisition of hydrological information such as flow velocity and flow is crucial to reducing the losses caused by flood disasters. However, during the peak flood season, the river flow is fast, the sediment content is high, and there are many floating objects. The measuring equipment is easily damaged, and the traditional contact measurement technology is difficult to implement effectively due to the complex turbulent characteristics of natural rivers. Therefore, in recent years, non-contact fluid motion vector measurement technology has gradually emerged and has been widely used due to its high efficiency and safety. This technology records the water surface video through a camera, analyzes the motion vector of the tracer and converts it to the world coordinate system to achieve flow velocity measurement.

[0003] At present, common image-based velocity measurement methods include large-scale particle tracking velocimetry, large-scale particle image velocimetry, and space-time image velocimetry (STIV). Among them, STIV directly estimates the one-dimensional time-averaged velocity by detecting the main texture direction (MOT) of the space-time image (STI), which has the advantages of high spatial resolution and strong real-time performance. MOT detection is the core step of STIV, and its detection methods can be divided into spatial domain method and frequency domain method. Spatial domain methods such as the gradient tensor method are more sensitive to image area division, and frequency domain methods such as the fast Fourier transform method (FFT-STIV) have advantages in accuracy and efficiency, but are more sensitive to noise, especially in complex natural environments. Existing filtering technology is difficult to completely eliminate the influence of noise.

[0004] In recent years, with the rapid development of deep learning, researchers have tried to apply it to MOT detection of STIV to improve detection accuracy and robustness. However, although it performs well under ideal conditions, the accuracy is low in complex scenes, and real data sets need to be added for intensive training. These methods are mainly based on artificially constructed STI data sets and classification models. Although they perform well under ideal conditions, the accuracy is low in complex scenes, and real data sets need to be added for intensive training. In addition, the angles output by these classification models are discrete integer values, which makes it difficult to capture the continuity of the actual flow rate.

[0005] Validity identification is a key step in the STIV process, but current methods have not discussed this in depth, so there is a lack of effective means to identify and correct invalid values ​​in actual measurements. Summary of the invention

[0006] Purpose of the invention: The present invention provides a method for validating and measuring the spatiotemporal images of river surfaces based on deep learning, which can achieve accurate flow measurement and velocity correction after training without manual parameter adjustment and adding experimental river data sets.

[0007] Technical solution: The method for river surface spatiotemporal image validity recognition and speed measurement based on deep learning described in the present invention comprises the following steps:

[0008] Step 1: Collect natural river videos of different sections, water levels, flow rates, weather, lighting, and terrain, and arrange speed measurement lines to synthesize the basic spatiotemporal image STI dataset; divide the basic spatiotemporal image dataset into three categories: valid, unreliable, and invalid according to whether the main texture of the STI can represent the one-dimensional time-averaged flow rate of the water flow, and perform random dithering of random brightness, contrast, saturation, and hue to obtain the STI classification dataset; perform MOT semi-automatic annotation on the valid class STI, perform data cleaning according to whether it contains vertical texture, and then expand it separately: rotate the STI without vertical texture 180 times in 1° steps, and then perform maximum size cropping to maximize the retention of effective information; perform horizontal flipping, vertical flipping, horizontal and vertical flipping on the STI with vertical texture, and perform random dithering of random brightness, contrast, saturation, and hue on the expanded valid class STI to obtain the MOT regression dataset;

[0009] Step 2: Build an STI classification network model based on EfficientNetV2, input the STI classification data set into the model training, and save the training model; build a MOT regression network model based on the optimized residual module, input the MOT regression data set into the model training, and save the training model;

[0010] Step 3: Use the camera system installed at the hydrological station to monitor and obtain river surface videos in real time, arrange speed measurement lines to synthesize spatiotemporal images in real time, input the original spatiotemporal images into the trained STI classification network model, and obtain three types of spatiotemporal images: valid, unreliable, and invalid. Input the valid spatiotemporal images into the trained MOT regression network model to obtain the corresponding MOT.

[0011] Step 4: Convert the effective MOT into the flow velocity value at the corresponding position, and perform interpolation correction on the unreliable, invalid and blind area flow velocities in combination with the cross-sectional flow velocity distribution law to obtain the cross-sectional flow velocity field.

[0012] Furthermore, in step 1, the valid class is STI without strong environmental interference. At this time, MOT can correctly characterize the actual flow velocity, which includes normal, glare, noise, occlusion, turbulence, static shadows, standing waves, blur, and ripple scenes; the unreliable class is STI in scenes that strongly interfere with natural water flow or have textures that are confused with water flow. MOT is unreliable and it is difficult to correctly characterize the one-dimensional time-averaged flow velocity, which includes floodplain flow, dynamic shadows, backflow, and strong wind ripple scenes; the invalid class is STI in scenes that completely interfere with the camera's observation of the river surface, which includes strong light, dark noise, complete occlusion, strong shadows, heavy fog, and heavy rain scenes.

[0013] Furthermore, in step 1, the semi-automatic MOT annotation is performed by first using FFT-STIV for preliminary annotation, and then checking with the manual direct annotation value. When the interpolation value of the two is less than 1°, the FFT-STIV annotation value is used as the dataset MOT; otherwise, if it is determined to be STI in a complex scene, the manual annotation value is used as the dataset MOT.

[0014] Furthermore, in step 1, in order to maximize the size of the STI after rotation and cropping to retain more original information, the relationship between the cropping size l, the original image size h and the rotation angle θ is shown in the following formula:

[0015] l=hisin45° / sin(135°-θ%90°).

[0016] Furthermore, in step 2, the STI classification network model uses EfficientNetV2 as the backbone, and uses the FC fully connected layer to set the output nodes to 3, corresponding to three categories.

[0017] Furthermore, in step 3, the MOT regression network model is based on the residual network architecture. The convolution operation in the residual module is replaced by grouped convolution, and a convolutional attention module CBAM is added before the residual connection. The FC fully connected layer is used to limit the output node to 1, and the final activation function is modified to Sigmoid to adapt to regression convergence.

[0018] Furthermore, in step 4, the cross-sectional velocity distribution law is as follows: v(x) represents the water surface velocity at a starting point distance x, x0 is the starting point distance from the water edge, L is the water surface width in the cross-sectional direction, k is the coefficient to be determined, and the effective velocity set from the nearshore to the mid-channel area {v(x1), v(x2),…, v(x i )|x l ≤x i ≤x mid} and the corresponding starting point distance set {x1,x2,…,x i |x l ≤x i ≤x mid} is the known data, and the least square method is used to calculate [x l ,x mid ] area, similarly, the effective flow velocity set {v(x mid ),v(x mid+1 ),…,v(x i )|x mid ≤x i ≤x r} and the corresponding starting point distance set {x mid ,x mid+1 ,…,x i |x mid ≤x i≤x r} is the known data, and the least square method is used to calculate [x mid ,x r ] area, where x mid is the distance from the starting point of Zhonghong, x l is the starting distance of the nearshore waterline, x r is the distance from the starting point of the far shore water edge;

[0019]

[0020] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) By classifying the scenes and effectiveness of spatiotemporal images of river water surface, the boundary conditions of spatiotemporal image velocity measurement method applicable to complex and changing environments in the wild are clarified, which provides a basis for controlling the gross error of velocity measurement in online video flow measurement system under extreme conditions; (2) The lightweight EfficienctNet is used The V2 network model performs the validity identification of the three-class spatiotemporal image, which, on the one hand, avoids the deterioration of the overall detection accuracy caused by the interference of invalid and unreliable samples when training the MOT regression model, and on the other hand, avoids the computational overhead of subsequent MOT detection of invalid and unreliable STIs; (3) By semi-automatically annotating the valid STI with MOT and comprehensively adopting data cleaning, flipping, rotation and color jittering to expand the data, a multi-scene, wide-range, high-precision MOT regression dataset of real river water surface spatiotemporal images is constructed for model training. Compared with the existing artificially synthesized spatiotemporal image datasets, it can truly reflect the complexity and diversity of river water surface scenes under different weather, lighting and water flow conditions, making the model have stronger generalization ability and can be directly applied to sites with no or little data without the need for model transfer learning or enhanced training. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0022] Figure 2 Construct a flow chart for the dataset of the present invention.

[0023] Figure 3 This is a structural diagram of the MOT regression network model of the present invention.

[0024] Figure 4 The present invention is a flowchart of the method. DETAILED DESCRIPTION

[0025] like Figure 1 As shown, a method for validating and measuring the speed of river surface spatiotemporal images based on deep learning includes the following steps:

[0026] Step 1: Build a multi-scene, wide-range, high-precision real-time spatiotemporal image dataset, such as Figure 2 As shown in the figure, natural river videos with different sections, water levels, flow rates, weather, lighting, and terrain are collected, and speed measurement lines are arranged to synthesize the basic spatiotemporal image STI dataset.

[0027] According to whether the main texture of STI can represent the one-dimensional time-averaged velocity of water flow, the basic spatiotemporal image dataset is divided into three categories: valid, unreliable, and invalid. The valid category is STI without strong environmental interference. At this time, MOT can correctly represent the actual flow velocity, which includes normal, glare, noise, occlusion, turbulence, static shadows, standing waves, blur, ripples and other scenes; the unreliable category is STI in scenes that strongly interfere with natural water flow or have textures that confuse water flow. MOT is unreliable and difficult to correctly represent the one-dimensional time-averaged velocity, which includes floodplain flow, dynamic shadows, backflow, strong wind ripples and other scenes; the invalid category is STI in scenes that completely interfere with the camera's observation of the river surface, which includes strong light, dark noise, complete occlusion, strong shadows, heavy fog, heavy rain and other scenes. Random brightness, contrast, and saturation enhancement are performed to obtain the STI classification dataset.

[0028] The effective STI is semi-automatically annotated with MOT. The semi-automatic MOT annotation is to use FFT-STIV for preliminary annotation first, and then verify it with the manual direct annotation value. When the interpolation value of the two is less than 1°, the FFT-STIV annotation value is used as the dataset MOT; otherwise, it is identified as STI in a complex scene, and the manual annotation value is used as the dataset MOT. After data cleaning according to whether it contains vertical texture, it is expanded separately: the STI without vertical texture is rotated 180 times in 1° steps. In order to maximize the size of the STI after rotation and cropping to retain more original information, the relationship between the adaptive cropping size l, the original image size h and the rotation angle θ is shown in Equation 1; the STI with vertical texture is flipped horizontally, vertically, horizontally and vertically. The expanded effective STI is randomly jittered in brightness, contrast, saturation, and hue to obtain the MOT regression dataset.

[0029] l=hisin45° / sin(135°-θ%90°)

[0030] Step 2: Build an STI classification network model based on EfficientNetV2 and a MOT regression network model based on the optimized residual module.

[0031] The STI classification network model uses EfficientNetV2 as the backbone, and uses the FC fully connected layer to set the output nodes to 3, corresponding to three categories. The network structure is shown in Table 1: Conv2d is the convolution operation, SiLU is the activation function, which has the characteristics of no upper bound, lower bound, smoothness, and non-monotonicity compared to ReLU, and can provide better performance in deep models. Pooling is the pooling layer.

[0032] Table 1 STI classification network structure

[0033]

[0034] EfficientNet V2 uses the inverted residual structure (MBConv) and the channel attention mechanism (SE). The basic process of MBConv is: first, the low-dimensional input is increased in dimension through dilated convolution, then deep convolution is performed, and then the dimension is reduced through point convolution, and finally the residual is added. This structure effectively improves the gradient propagation capability and significantly reduces the computational cost by using deep separable convolution. Although deep convolution is less efficient in the early stages, it is effective in the later stages. Therefore, in the early stages of the model, ordinary 3x3 convolutions are used to replace the 1x1 dilated convolutions and 3x3 deep convolutions in MBConv, thus forming a Fused-MBConv structure. This design ensures that performance is maintained while improving model efficiency.

[0035] The SE module explicitly establishes dependencies between feature channels. It weights features by learning the importance of each channel, thereby highlighting important features and suppressing unimportant features. During operation, the SE module first compresses the spatial features of each channel into a global feature through global average pooling, thereby completing the fusion of feature information. Next, the module uses two fully connected layers to predict the importance of each channel. Among them, the first fully connected layer reduces the number of channels and dimensions by scaling hyperparameters, thereby reducing the amount of computation; the second fully connected layer restores the original dimension to ensure that the importance of the channel is exactly matched with the number of its feature maps. Finally, the calculated channel importance is used to weight the original feature map, multiplying the feature map of each channel with the corresponding weight to generate a new weighted feature map. This process effectively increases the network's attention to important features while suppressing unimportant features, thereby enhancing the model's representation ability and performance.

[0036] like Figure 3As shown in the figure, a MOT regression network model based on the optimized residual module is built. In order to cope with deep learning training and accurate MOT detection under huge data sets, a MOT regression network model combining group convolution and convolution attention mechanism is designed. Taking the first convolution layer of the structure diagram (Structure) as an example, the convolution kernel size used is 7×7, the number of output channels is 64, and the stride is / 2. Repetition is the number of times the module is repeated. Groups=32 in the BottleNeck module means that 32 groups of group convolutions are used. The last layer is a fully connected layer (FC), whose output node is set to 1, and the result is limited to the (0,1) interval through the Sigmoid activation function to achieve continuous angle regression prediction.

[0037] Grouped convolution divides the convolution into multiple groups of dimensionality reduction convolution operations, and then connects the output after dimensionality increase. This design retains the function of the residual module to avoid gradient disappearance / explosion while increasing the scalability of the module and improving computational efficiency. It is particularly suitable for environments with limited resources and large-scale data sets.

[0038] CBAM consists of two main submodules: Channel Attention Module (CAM) and Spatial Attention Module (SAM). The goal of CAM is to enhance the feature expression of each channel. Through global maximum pooling (MaxPool) and global average pooling (AvgPool), the maximum eigenvalue and average eigenvalue on each channel are calculated, and then the attention weight of each channel is learned through the shared multi-layer perceptron (SharedMLP) fully connected layer (FC). Finally, the Sigmoid activation function is applied to generate CAM weights, and these weights are applied to each channel of the original feature map. The spatial attention module focuses on the spatial position of the input data, performs global maximum pooling (ChannelMaxPool) and global average pooling (ChannelAvgPool) operations along the channel dimension, and finally applies the Sigmoid activation function to generate SAM weights, and finally generates spatial attention weights, and these weights are applied to the spatial dimension of the original feature map. This mechanism of paying attention to both channel and spatial information is particularly suitable for long texture detection tasks, which effectively improves the accuracy of feature extraction and the performance of the model.

[0039] The STI classification data set is used to train the STI classification network model, and the training parameters are shown in Table 2; the MOT regression data set is used to train the MOT regression network model, and the training parameters are shown in Table 3. The data set is standardized to improve the learning ability, and the standardization formula is as follows:

[0040]

[0041] Where input(c) represents the pixel value of the original data channel c, output(c) is the normalized pixel value, N represents the number of images, and H×W represents the image size. Use early stopping of training to avoid overfitting when the accuracy stops increasing. Monitor the memory situation and use the automatic reduction scheme of BatchSize to avoid video memory overflow. Save the training model.

[0042] Table 2 STI classification network model training parameters

[0043]

[0044] Table 3. Training parameters of MOT regression network model

[0045]

[0046] Step 3: If Figure 4 As shown in the figure, the video of the river surface is obtained by real-time monitoring through the camera system installed in the hydrological station, and the speed measurement line is arranged to obtain the real-time spatiotemporal image. The original spatiotemporal image is input into the saved STI classification network model to obtain three types of spatiotemporal images: valid, unreliable, and invalid. The valid spatiotemporal image is input into the saved MOT regression network model to obtain the effective position MOT.

[0047] Step 4: Convert the MOT of the valid class to the velocity value of the corresponding position according to Formula 5. In Formula 5, δ is the MOT value. Assume that the tracer in the object plane moves a distance D along a certain velocity line within time T, which is expressed as d pixels moving within τ frames in the image coordinate system. v (pixel / s) represents the optical flow motion vector, and its positive or negative value reflects the direction of movement; there is only a scaling relationship between V and v, which is represented by the object image scale factor Δs (m / pixel) on the velocity line, and its value is obtained by calibration of the monocular vision plane measurement method based on variable height homography.

[0048]

[0049] Combined with the cross-sectional velocity distribution law, the unreliable, invalid and blind area velocity is interpolated and corrected to obtain the cross-sectional velocity field. The cross-sectional velocity distribution law is shown in formula 6:

[0050]

[0051] v(x) represents the water surface velocity at the starting point x, x0 is the starting point distance from the water edge, L is the water surface width in the cross section, and k is the coefficient to be determined. The effective velocity set from the nearshore to the middle channel {v(x1), v(x2), …, v(x i )|x l ≤x i ≤x mid} and the corresponding starting point distance set {x1,x2,…,xi |x l ≤x i ≤x mid} is the known data, and the least square method is used to calculate [x l ,x mid ] area. Similarly, the effective flow velocity set {v(x mid ),v(x mid+1 ),…,v(x i )|x mid ≤x i ≤x r} and the corresponding starting point distance set {x mid ,x mid+1 ,…,x i |x mid ≤x i ≤x r} is the known data, and the least square method is used to calculate [x mid ,x r ] area. mid is the distance from the starting point of Zhonghong, x l is the starting distance of the nearshore waterline, x r It is the distance from the starting point of the far shore waterline.

Claims

1. A method for validating and measuring river surface spatiotemporal images based on deep learning, characterized in that: The steps include: Step 1: Collect natural river videos of different sections, water levels, flow rates, weather, lighting, and terrain, and arrange speed measurement lines to synthesize the basic spatiotemporal image STI dataset; divide the basic spatiotemporal image dataset into three categories: valid, unreliable, and invalid according to whether the main texture of the STI can represent the one-dimensional time-averaged flow velocity of the water flow, and obtain the STI classification dataset after processing it. After the effective class STI is semi-automatically annotated with MOT, the data is cleaned according to whether it contains vertical textures, and then expanded to obtain the MOT regression datasets; Step 2: Build an STI classification network model based on EfficientNetV2, input the STI classification data set into the model training, and save the training model; build a MOT regression network model based on the optimized residual module, input the MOT regression data set into the model training, and save the training model; Step 3: Use the camera system installed at the hydrological station to monitor and obtain river surface videos in real time, arrange speed measurement lines to synthesize spatiotemporal images in real time, input the original spatiotemporal images into the trained STI classification network model, and obtain three types of spatiotemporal images: valid, unreliable, and invalid. Input the valid spatiotemporal images into the trained MOT regression network model to obtain the corresponding MOT. Step 4: Convert the effective MOT into the flow velocity value at the corresponding position, and perform interpolation correction on the unreliable, invalid and blind area flow velocities in combination with the cross-sectional flow velocity distribution law to obtain the cross-sectional flow velocity field.

2. The method for river surface spatiotemporal image validity recognition and speed measurement based on deep learning as claimed in claim 1, characterized in that: In step 1, the valid category is STI without strong environmental interference. At this time, MOT can correctly characterize the actual flow velocity, which includes normal, glare, noise, occlusion, turbulence, static shadows, standing waves, blur, and ripple scenes; the unreliable category is STI in scenes that strongly interfere with natural water flow or have textures that are confused with water flow. MOT is unreliable and it is difficult to correctly characterize the one-dimensional time-averaged flow velocity, which includes floodplain flow, dynamic shadows, backflow, and strong wind ripple scenes; the invalid category is STI in scenes that completely interfere with the camera's observation of the river surface, which includes strong light, dark noise, complete occlusion, strong shadows, heavy fog, and heavy rain scenes.

3. The method for river surface spatiotemporal image validity recognition and speed measurement based on deep learning as claimed in claim 1, characterized in that: In step 1, the semi-automatic MOT annotation is to first use FFT-STIV for preliminary annotation, and then verify it with the manual direct annotation value. When the interpolation value of the two is less than 1°, the FFT-STIV annotation value is used as the MOT of the dataset; otherwise, it is determined to be STI in a complex scene, and the manual annotation value is used as the MOT of the dataset.

4. The method for river surface spatiotemporal image validity recognition and speed measurement based on deep learning as claimed in claim 1, characterized in that: In step 1, after the effective class STI is semi-automatically annotated with MOT, the data is cleaned according to whether it contains vertical texture, and then expanded separately: the STI without vertical texture is rotated 180 times in 1° steps, and then cropped to the maximum size to maximize the retention of effective information; the STI with vertical texture is flipped horizontally, vertically, horizontally and vertically, and the expanded effective class STI is randomly jittered in brightness, contrast, saturation, and hue to obtain the MOT regression data set.

5. The method for river surface spatiotemporal image validity recognition and speed measurement based on deep learning as claimed in claim 4, characterized in that: In step 1, in order to maximize the size of the STI after rotation and cropping to retain more original information, the relationship between the cropping size l, the original image size h and the rotation angle θ is shown in the following formula: l=hisin45° / sin(135°-θ%90°).

6. The method for river surface spatiotemporal image validity recognition and speed measurement based on deep learning as claimed in claim 1, characterized in that: In step 2, the STI classification network model uses EfficientNetV2 as the backbone, and uses the FC fully connected layer to set the output nodes to 3, corresponding to three categories.

7. The method for river surface spatiotemporal image validity recognition and speed measurement based on deep learning as claimed in claim 1, characterized in that: In step 3, the MOT regression network model is based on the residual network architecture. The convolution operation in the residual module is replaced by grouped convolution, and a convolutional attention module CBAM is added before the residual connection. The FC fully connected layer is used to limit the output node to 1, and the final activation function is modified to Sigmoid to adapt to regression convergence.

8. The method for river surface spatiotemporal image validity recognition and speed measurement based on deep learning as claimed in claim 1, characterized in that: In step 4, the cross-sectional velocity distribution law is as follows: v(x) represents the water surface velocity at a starting point distance x, x0 is the starting point distance from the water edge, L is the water surface width in the cross-sectional direction, k is the coefficient to be determined, and the effective velocity set from the nearshore to the mid-channel area {v(x1), v(x2), …, v(x i )|x l ≤x i ≤x mid } and the corresponding starting point distance set {x1,x2,…,x i |x l ≤x i ≤x mid } is the known data, and the least square method is used to calculate [x l ,x mid ] area, similarly, the effective flow velocity set {v(x mid ),v(x mid+1 ),…,v(x i )|x mid ≤x i ≤x r } and the corresponding starting point distance set {x mid ,x mid+1 ,…,x i |x mid ≤x i ≤x r } is the known data, and the least square method is used to calculate [x mid ,x r ] area, where x mid is the distance from the starting point of Zhonghong, x l is the starting distance of the nearshore waterline, x r The distance from the starting point of the far shore waterline;

Images