River video speed measurement method and device based on multi-feature fusion PSA-ResNet network and medium

Abstract

The invention relates to the technical field of video water flow velocity detection, and discloses a river video velocity measurement method and device based on a multi-feature fusion PSA-ResNet network, and a medium. The method comprises the following steps: constructing an STIA data set; extracting low-level textural features of the space-time image by using a CLBP multi-feature extraction algorithm, performing channel-level fusion on a feature map and an original RGB image to form a six-channel CLBP-STIA feature map, and constructing a CLBP-STIA data set according to the six-channel CLBP-STIA feature map; fusing a pyramid segmentation attention module PSA in a residual block of the residual network ResNet to construct an angle classification model, and training the model by using a CLBP-STIA data set; generating enhanced feature representation of the to-be-detected space-time image by referring to the above mode, then inputting the enhanced feature representation to the trained classification model, and outputting a corresponding texture principal direction angle; and acquiring the actual length of the velocity measurement line in the video image, and calculating the actual flow velocity of the surface flow feature on the velocity measurement line in combination with the texture main direction angle. The texture main direction is accurately estimated, so that the speed measurement precision and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video water flow velocity detection, and in particular to a river video velocity measurement method based on a multi-feature fusion PSA-ResNet network, as well as an electronic device and a computer-readable storage medium applying the method. Background Art

[0002] River velocity data is one of the most basic data in hydrological research and the cornerstone of various hydrological studies and applications. However, due to the complex water flow characteristics of natural rivers, accurately measuring the flow velocity of natural rivers has always been a difficult task in hydrological surveys. This challenge becomes more prominent under extreme conditions such as floods and droughts. In such cases, it is particularly difficult to obtain key data such as the temporal changes in river velocity and runoff. Currently, traditional methods for measuring river velocity include the current meter method, the buoy method, and the acoustic Doppler current profiler (ADCP) method.

[0003] Among them, (1) the velocity meter method involves using a velocity meter to measure the velocity distribution along a vertical line, and then using the flow area method to calculate the discharge. Although this is the most basic and widely used method, it is time-consuming, labor-intensive, and requires direct contact with the water body, posing a safety risk to surveyors, especially during periods of high flow or floods. (2) The buoy method is another commonly used technique, which estimates the surface velocity by tracking buoys on a river cross section, and then uses the cross-sectional data to infer the water surface velocity and adjust it by the buoy coefficient to estimate the total discharge. However, this method is susceptible to uncertainties, including buoy velocity measurement errors and empirical selection of the buoy coefficient. In addition, due to poor reliability and safety issues, this method is not applicable during floods. (3) ADCP can simultaneously measure the flow velocity at multiple depths along a vertical line by tracking the movement of suspended scatterers in the water body, and then calculate the discharge using the flow field method. Although ADCP has the advantages of non-contact and high-resolution measurement, it is easily entangled by floating objects and produces measurement errors. In addition, the high cost of ADCP equipment also limits its widespread application in conventional hydrological monitoring.

[0004] With the rapid development of information technology, flow measurement technologies based on imaging, optics, and radar have made significant progress. Image-based flow measurement methods, in particular, have attracted considerable attention due to their simplicity, efficiency, and safety. As a non-contact method, image-based flow measurement effectively overcomes the limitations of traditional contact methods, acquiring flow velocity and other fluid dynamics parameters in a simple, rapid, and safe manner. Therefore, it is particularly suitable for deployment under high-flow or flood conditions, where traditional methods may be impractical or dangerous. Furthermore, the widespread installation of river monitoring systems at hydrological stations across the country has provided a favorable foundation for the wider adoption of image-based methods, while also reducing operating costs. Image-based flow measurement methods can be divided into two main categories, depending on the motion vector estimation technique: large-scale particle image velocimetry (LSPIV) and space-time image velocimetry (STIV).

[0005] Large-scale particle image velocimetry (LSPIV), proposed by Fujita et al. in the 1990s, is an improvement on the particle image velocimetry (PIV) technique traditionally used in laboratory fluid dynamics. LSPIV extends PIV to field applications for observing surface flow fields and measuring river discharge. It employs naturally occurring surface features (such as floating debris and ripples) as tracers, uses ambient light as the primary illumination source, and captures image sequences using a digital or video camera. By tracking the motion of these tracers within image frames within a defined region of interest, LSPIV estimates the two-dimensional surface flow field through image matching techniques. Despite its advantages, LSPIV also has limitations. Its accuracy depends heavily on the visibility and distribution of surface tracers, which can be affected by factors such as low tracer density, uneven spatial and temporal distribution, and complex surface imaging conditions. These factors can lead to significant errors or failures in the reconstruction of the time-averaged flow field. Furthermore, the choice of analysis region size introduces further uncertainty: an excessively large analysis window can reduce spatial resolution due to spatial averaging, while an excessively small region increases the likelihood of mismatches due to insufficient tracer information. In addition, LSPIV requires a large amount of storage capacity and has relatively low computational efficiency.

[0006] Compared with LSPIV, the space-time image velocimetry (STIV) proposed by Fujita et al. in 2007 shows superior performance in cross-sectional flow measurement. Unlike LSPIV, which relies on a rectangular analysis window, STIV uses a single-pixel-wide velocimetry line as the analysis area. This design greatly improves spatial resolution and computational efficiency. The core of STIV technology lies in estimating the main direction of texture (MOT), which reflects the direction of motion encoded in the space-time image. Existing MOT estimation algorithms can be roughly divided into two categories according to the image processing domain: (1) spatial domain methods, such as the gradient tensor method (GTM) and other space-time methods; (2) frequency domain methods based on fast Fourier transform (FFT), such as FFT-STIV. However, traditional STIV is easily affected by interference factors when processing complex scenes, resulting in inaccurate estimation of the main direction of texture, which limits the speed measurement accuracy in complex environments and high noise conditions. Summary of the Invention

[0007] In order to solve the technical problem that the existing STIV method is difficult to accurately estimate the main direction of texture under complex environmental conditions, thereby limiting the speed measurement accuracy and efficiency, the present invention provides a river video speed measurement method, device and medium based on a multi-feature fusion PSA-ResNet network.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention discloses a river video speed measurement method based on a multi-feature fusion PSA-ResNet network, comprising:

[0010] S1. Construct the STIA dataset; the STIA dataset includes spatiotemporal images of rivers and their corresponding texture main direction angle labels;

[0011] S2. Use the CLBP multi-feature extraction algorithm to extract low-level texture features from the spatiotemporal images in the STIA dataset, generating three feature maps: center pixel encoding, sign component encoding, and amplitude component encoding. These three feature maps are then fused with the original RGB image at the channel level to form a 6-channel CLBP-STIA feature map, which is then used to construct the CLBP-STIA dataset.

[0012] S3. Integrate the pyramid segmentation attention module (PSA) into the residual block of the ResNet to build an angle classification model, and train the model using the CLBP-STIA dataset.

[0013] S4. Referring to step S2, an enhanced feature representation of the spatiotemporal image to be detected is generated, which is then input into the trained angle classification model to output the corresponding main texture direction angle;

[0014] S5. Obtain the actual length of the speed measurement line in the video image, and calculate the actual flow velocity of the surface flow feature on the speed measurement line in combination with the main direction angle of the texture.

[0015] As a further improvement of the above solution, step S1 includes the following specific steps:

[0016] S11. Record river video;

[0017] S12. Preprocess the recorded river video, including grayscale conversion, video frame extraction, and noise reduction;

[0018] S13. Set the size parameters of the speed measurement line and select the speed measurement area; wherein the length of the speed measurement line is L pixels and the width is 1 pixel;

[0019] S14. Based on the image sequence captured from the preprocessed river video, a velocity measurement line is set in each frame along the direction of water flow. The corresponding velocity measurement lines in m consecutive frames are arranged along the time axis to generate an L×m space-time image. In this image, the horizontal direction is the spatial dimension, i.e., the length of the velocity measurement line, and the vertical direction is the time dimension.

[0020] S15. Manually analyze the generated spatiotemporal images to identify and annotate the main texture direction angle in each spatiotemporal image, thereby forming the STIA dataset.

[0021] As a further improvement of the above solution, in step S11, multiple river videos are recorded under different lighting conditions, weather environments, flow rates or river channel shapes.

[0022] As a further improvement of the above scheme, in step S3, the residual network includes an input layer, a convolutional layer, a residual block, an average pooling layer, a fully connected layer and a softmax layer; the input layer is used to receive the 6-channel CLBP-STIA feature map; the convolutional layer is used to perform initial feature extraction using a 7×7 convolution kernel; there are multiple residual blocks, each of which is composed of two 3×3 convolutional layers, and the gradient is prevented from disappearing by skip connection; the global average pooling layer is used to pool the feature map to reduce the computational complexity; the fully connected layer is used to map the pooled features as the classifier input; the softmax layer is used to output the final texture main direction angle category probability;

[0023] Among them, the pyramid segmentation attention module extracts features of different spatial scales through multi-scale convolution kernels, and performs weighted fusion of features of different scales through the Softmax function, thereby enhancing features related to the classification task and suppressing irrelevant features.

[0024] As a further improvement to the above solution, in step S3, the training method of the angle classification model includes:

[0025] The CLBP-STIA dataset is divided into training set, test set and validation set according to the set ratio;

[0026] Iteratively training the angle classification model on the training set and adjusting hyperparameters to ensure model convergence; the hyperparameters include learning rate and batch size;

[0027] Evaluate model performance through validation sets and adjust network structure or training strategies to optimize classification accuracy;

[0028] The test set is input into the trained angle classification model to predict the main texture direction, and the model performance is evaluated by calculating indicators; the indicators include accuracy, correctness and recall.

[0029] As a further improvement to the above solution, in step S2, the method for generating the three feature maps includes:

[0030] For each pixel in the spatiotemporal image, calculate the grayscale difference between it and its n neighboring pixels to generate an n-dimensional difference vector;

[0031] Decomposing the difference vector into a sign component and an amplitude component; wherein the sign component is generated by comparing the grayscale difference between the central pixel and the surrounding pixels to determine the brightness relationship of each neighboring pixel relative to the central pixel, i.e., a binary sign vector; and the amplitude component is formed by calculating the absolute difference between the central pixel and the surrounding pixels;

[0032] The grayscale value, sign component and amplitude component of the central pixel are encoded to form a central pixel code, a sign component code and an amplitude component code respectively.

[0033] As a further improvement to the above solution, in step S5, the actual length of the speed measurement line is calculated as follows:

[0034] Get the mapping function between the phase plane coordinates (x, y) and the actual space rectangular coordinates (X, Y, Z):

[0035]

[0036] In the formula, (x p ,y p ) represents the phase plane coordinate of the principal point; (X p ,Y p ,Z p ) represents the actual space rectangular coordinate of the camera; f represents the focal length; (Δx, Δy) represents the lens distortion correction factor; r ij (i, j = 1 to 3) are the conversion coefficients between the phase plane coordinates and the actual space rectangular coordinates, both of which are unknown parameters;

[0037] At least six ground calibration signs are set up in advance at the river speed measurement site. The actual spatial rectangular coordinates of each calibration sign relative to the measurement station are measured using a total station. The phase plane coordinates corresponding to each calibration sign are obtained from the video image, and the known coordinates are used to solve the unknown parameters in the mapping function.

[0038] The actual spatial rectangular coordinates of any point on the space-time image are calculated according to the mapping function and combined with the water level value, thereby obtaining the actual length of the speed measurement line.

[0039] As a further improvement to the above solution, in step S5, the actual flow velocity v of the surface flow characteristic on the velocity measurement line is calculated according to the following formula:

[0040]

[0041] Where α represents the main direction angle of the texture; S x Indicates the actual distance represented by each pixel, the unit is: m / pixel; fps indicates the frame rate of the camera, the unit is: frames / second.

[0042] The present invention also discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the river video speed measurement method based on the multi-feature fusion PSA-ResNet network as described above are implemented.

[0043] The present invention also discloses a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the river video speed measurement method based on the multi-feature fusion PSA-ResNet network as described above are implemented.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention significantly improves the speed measurement accuracy in complex environments and high noise conditions by combining the pyramid segmentation attention residual network (PSA-ResNet) and the completely local binary pattern (CLBP) multi-feature extraction algorithm. The traditional STIV method is easily affected by interference factors when processing complex scenes, resulting in inaccurate estimation of the main direction of the texture. The present invention can more accurately identify texture features through the attention mechanism of PSA-ResNet and the multi-feature extraction capability of CLBP, thereby achieving more accurate flow rate measurement. By constructing a special spatiotemporal image angle (STIA) dataset and a CLBP-STIA multi-feature fusion dataset, a systematic experimental verification was carried out. The results show that compared with traditional methods, the present invention has significant advantages in speed measurement accuracy and computational efficiency. At the same time, comparative analysis in actual river scenes also confirms its accuracy and reliability.

[0046] 2. The proposed method is significantly more robust than traditional methods in complex river conditions. By focusing on key features using PSA-ResNet and enhancing texture features using CLBP-STIA, this method effectively suppresses the influence of noise and interference, maintaining stable speed measurement performance in a variety of complex scenarios.

[0047] 3. This invention significantly improves the computational efficiency of the velocity measurement method by optimizing the feature extraction and classification processes. The lightweight design of PSA-ResNet and the efficient feature description of CLBP-STIA enable it to meet the needs of real-time flow velocity monitoring, especially in large-scale flow field data processing.

[0048] 4. This invention can be applied not only to river flow velocity monitoring but also to other complex fluid environments, such as industrial pipeline flow velocity monitoring and atmospheric flow field analysis. Its versatility and flexibility significantly broaden the application areas of the STIV method. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flowchart of the river video speed measurement method based on the multi-feature fusion PSA-ResNet network in Example 1 of the present invention.

[0050] Figure 2 Schematic diagram of the spatiotemporal image synthesis process in Example 1 of the present invention.

[0051] Figure 3 This is the spatiotemporal image synthesized in Example 1 of the present invention.

[0052] Figure 4 This is the overall structural diagram of the angle classification model (i.e., the PSA-ResNet model) in Example 1 of the present invention.

[0053] Figure 5 for Figure 4 PSA-Bottleneck structure diagram in.

[0054] Figure 6 for Figure 5 PSA Module structure diagram in .

[0055] Figure 7 for Figure 6 Slice Pyramid (SPC) module structure diagram in .

[0056] Figure 8 This is a flowchart of the CLBP multi-feature extraction algorithm in Example 1 of the present invention.

[0057] Figure 9This is a graph showing the relevant indicator results (including precision, recall, and F1 score) of different algorithms in the STIA experiment in Example 1 of the present invention.

[0058] Figure 10 This is a graph showing the relevant indicator results (including precision, recall rate, and F1 score) of different algorithms in the CLBP-STIA experiment in Example 1 of the present invention.

[0059] Figure 11 This is a display diagram of the actual outdoor river test scene and speed measurement line setting in Example 1 of the present invention.

[0060] Figure 12 Schematic diagram of the structure of an electronic device in Example 2 of the present invention. DETAILED DESCRIPTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0062] Example 1

[0063] See also Figure 1 ,This embodiment provides a river video speed measurement method based on a multi-feature fusion PSA-ResNet network, including steps S1 to S5.

[0064] S1. Construct the STIA dataset; the STIA dataset includes spatiotemporal images of rivers and their corresponding texture main direction angle labels.

[0065] Step S1 includes the following specific steps:

[0066] S11. Recording River Videos: In this embodiment, an industrial-grade high-speed camera can be used to record real river videos, ensuring that the videos include a variety of natural and artificial river environments, covering different lighting conditions (e.g., sunny, overcast, and cloudy) and different weather conditions (e.g., rainy and foggy). Furthermore, the recorded videos should cover scenes with varying flow rates and different river channel morphologies (e.g., curved, straight, and turbulent areas) to ensure data diversity and representativeness.

[0067] S12. Preprocess the recorded river video, including grayscale conversion, video frame extraction, and noise reduction. Grayscale conversion converts color video files into grayscale images to reduce data redundancy and improve the efficiency of subsequent processing. Video frame extraction extracts video frames at a fixed frequency (e.g., 25 frames per second) and saves them as a series of continuous grayscale images. Noise removal performs preliminary noise reduction on the image using Gaussian filtering or other denoising algorithms to reduce the impact of noise on subsequent feature extraction.

[0068] S13. Set the size parameters of the speed measurement line and select the speed measurement area. The frame rate is set to 25 frames per second to ensure a sufficiently high temporal resolution for the video. Speed ​​measurement line parameters: Set the speed measurement line length to L = 50 pixels and the width to 1 pixel to ensure that sufficient spatial information is captured without introducing redundant data. Speed ​​measurement area: Based on the direction of the water flow, select an appropriate area as the speed measurement zone to ensure that the water flow movement within the speed measurement area is representative.

[0069] S14. Figure 2 and Figure 3 As shown in the figure, based on the image sequence collected from the preprocessed river video, a speed measurement line is set along the direction of water flow in each frame of the image. The corresponding speed measurement lines in the continuous m = 30 frames of grayscale images are arranged along the time axis to generate an L × m (50 × 30) space-time image; in this image, the horizontal direction is the spatial dimension, that is, the length direction of the speed measurement line, and the vertical direction is the time dimension.

[0070] S15. Manually analyze the generated spatiotemporal images to identify and label the main texture direction angles in each spatiotemporal image. Labeling the main texture direction angles can be considered a classification task, classifying them into specific angle ranges (e.g., "diagonally to the right 45° ± 5°" corresponds to category 45).

[0071] It's important to note that the water flow direction is consistent across all frames. The dominant texture direction refers to the most prominent arrangement or extension direction of the texture pattern in an image. It is the core feature that describes texture directionality. It reflects the overall orientation of texture elements (such as lines, spots, and periodic structures).

[0072] Afterwards, the annotated spatiotemporal images are stored in a standard image file format and their corresponding angle labels are recorded, eventually forming the Spatiotemporal Image Angle (STIA) dataset.

[0073] S2. The CLBP multi-feature extraction algorithm is used to extract low-level texture features of the spatiotemporal images in the STIA dataset, generating three feature maps: center pixel encoding, sign component encoding, and amplitude component encoding. These three feature maps are then fused with the original RGB image at the channel level to form a 6-channel CLBP-STIA feature map, which is then used to construct the CLBP-STIA dataset.

[0074] It should be noted that the original RGB image refers to a color photo set consisting of all video frames of the river video in the above S12 that have not been grayscaled.

[0075] Traditional CNN models (such as ResNet) excel at extracting high-level, global features from images, but struggle with texture feature extraction. Especially for complex texture structures, traditional CNNs are prone to losing low-level texture information, resulting in incomplete feature representation. Therefore, the CLBP multi-feature extraction algorithm in this embodiment is designed to extract low-level texture features that complement the high-level features extracted by CNN.

[0076] The generation methods of the above three feature maps include:

[0077] like Figure 8 As shown in Figure 1, for each pixel in the spatiotemporal image, the grayscale difference between it and the surrounding 8 neighboring pixels is calculated to generate an 8-dimensional difference vector;

[0078] Decomposing the difference vector into a sign component and an amplitude component; wherein the sign component is generated by comparing the grayscale difference between the central pixel and the surrounding pixels to determine the brightness relationship of each neighboring pixel relative to the central pixel, i.e., a binary sign vector; and the amplitude component is formed by calculating the absolute difference between the central pixel and the surrounding pixels;

[0079] The grayscale value (CLBP_C), sign component (CLBP_S) and amplitude component (CLBP_M) of the center pixel are encoded to form the center pixel code, sign component code and amplitude component code respectively.

[0080] In this embodiment, the CLBP-STIA dataset is expanded through geometric transformations such as rotation, scaling, and flipping to increase data diversity, thereby improving the generalization ability of the model.

[0081] S3. The pyramid segmentation attention module (PSA) is integrated into the residual block of the residual network (ResNet) to build an angle classification model, and the model is trained using the CLBP-STIA dataset.

[0082] See also Figures 4 to 7 , Figure 4 In the example, stride is the stride and padding is the padding. Figure 5 In the figure, Input and Output represent input and output respectively, BatchNorm represents batch normalization, ReLU is the activation function, and Conv is convolution. Figure 6 In the figure, C, H, and W represent the number of channels, height, and width, respectively. Softmax is the activation function, and element-wise product represents the multiplication of the elements of two matrices or vectors at corresponding positions. Figure 7 In the figure, X represents the input tensor, Split represents the segmentation process, X0 to X3 are the results of the segmentation process, K0 to K3 are the convolution kernels corresponding to the segmentation process results, G0 to G3 are the groups corresponding to the segmentation process results, F0 to F3 are the multi-scale feature kernel maps corresponding to the segmentation process results, and F is the fused multi-scale feature map.

[0083] The residual network includes an input layer, a convolutional layer, a residual block, an average pooling layer, a fully connected layer, and a Softmax layer. In this embodiment, the residual network adopts ResNet50.

[0084] Input layer: used to receive 6-channel CLBP-STIA feature maps;

[0085] Convolutional layer: used for initial feature extraction using a 7×7 convolution kernel; there are multiple residual blocks, each consisting of two 3×3 convolutional layers, with skip connections to prevent gradient vanishing;

[0086] Global average pooling layer: used to pool feature maps to reduce computational complexity; the fully connected layer is used to map the pooled features to the classifier input;

[0087] Softmax layer: used to output the final texture main direction angle category probability;

[0088] The pyramid segmentation attention module extracts features of different spatial scales using multi-scale convolution kernels (e.g., 1×1, 3×3, 5×5) and fuses the results. It also performs weighted fusion of features of different scales using the Softmax function, thereby enhancing features relevant to the classification task and suppressing irrelevant features.

[0089] In this embodiment, the cross entropy loss function can be used in combination with the Adam optimizer for model training. The training method of the angle classification model includes:

[0090] The CLBP-STIA dataset was divided into training set, test set, and validation set in a ratio of 8:1:1;

[0091] Iteratively training the angle classification model on the training set and adjusting hyperparameters to ensure model convergence; the hyperparameters include learning rate and batch size;

[0092] Evaluate model performance through validation sets and adjust network structure or training strategies to optimize classification accuracy;

[0093] The test set is input into the trained angle classification model to predict the main texture direction, and the model performance is evaluated by calculating indicators; the indicators include accuracy, correctness and recall.

[0094] S4. Referring to step S2, generate an enhanced feature representation of the spatiotemporal image to be detected, and then input it into the trained angle classification model to output the corresponding texture main direction angle.

[0095] S5. Obtain the actual length of the speed measurement line in the video image, and calculate the actual flow velocity of the surface flow feature on the speed measurement line in combination with the main direction angle of the texture.

[0096] The actual length of the speed measurement line is calculated as follows:

[0097] Get the mapping function between the phase plane coordinates (x, y) and the actual space rectangular coordinates (X, Y, Z):

[0098]

[0099] In the formula, (x p ,y p ) represents the phase plane coordinate of the principal point; (X p ,Y p ,Z p ) represents the actual space rectangular coordinate of the camera; f represents the focal length; (Δx, Δy) represents the lens distortion correction factor; r ij (i, j = 1 to 3) are the conversion coefficients between the phase plane coordinates and the actual space rectangular coordinates, both of which are unknown parameters;

[0100] The above mapping function contains 12 unknown parameters, so the phase plane coordinates and actual space coordinates of 6 points are required to solve the above unknown parameters (two equations can be listed for each point). To do this, it is necessary to pre-set at least six ground calibration signs at the river speed measurement site, use a total station to measure the actual spatial rectangular coordinates of each calibration sign relative to the measurement station, and obtain the phase plane coordinates corresponding to each calibration sign from the video image. Use the known coordinates to solve the unknown parameters in the mapping function.

[0101] After obtaining the above 12 unknown parameters, the actual spatial rectangular coordinates of any point on the space-time image can be calculated according to the mapping function and combined with the water level value, thereby obtaining the actual length of the speed measurement line.

[0102] After obtaining the texture angle and the actual length of the velocity line, assuming that in the actual space rectangular coordinate system, the distance that the surface flow feature such as ripples moves along the velocity line in time T is L0, correspondingly, in the phase plane coordinate system, it corresponds to the movement of i pixels in k frames. The flow velocity v on the velocity line is:

[0103]

[0104] Where α represents the main direction angle of the texture; S x Indicates the actual distance represented by each pixel, the unit is: m / pixel; St Indicates the duration of each frame. For example, if the video format is 30 frames per second, S t It is 1 / 30 second, and is multiplied by the number of frames k within the speed measurement range to obtain T; fps represents the frame rate of the camera, and the unit is: frames / second.

[0105] After calculating the flow velocity, the result is output to complete the speed measurement task.

[0106] This example further illustrates the effectiveness of the present invention through experiments on the STIA dataset and the CLBP-STIA dataset.

[0107] 1. Experimental Setup

[0108] To comprehensively evaluate the effectiveness of the proposed algorithm, this example compares the classification performance of PSA-ResNet with several baseline and advanced algorithms. These algorithms include traditional CNN, VGG, SE, CBAM, CA, STIV, and ResNet. The experimental settings for each algorithm are detailed below:

[0109] (1) Traditional CNN: A basic three-layer convolutional neural network is used to classify the STIA dataset. When experimenting with the CLBP-STIA dataset, the first convolutional layer is modified to accept six-channel input.

[0110] (2) VGG: The VGG-16 architecture was selected for performance comparison. The SE, CBAM, and CA modules were integrated into the standard CNN framework to evaluate their impact. For all three attention mechanisms, the compression ratio was set to 16. When applied to the CLBP-STIA dataset, the first convolutional layer of each model was adjusted to accommodate six-channel input.

[0111] (3) STIV: The gradient tensor method (GTM) is used to extract the main orientation of texture (MOT), which is consistent with the original STIV framework.

[0112] (4) ResNet: The ResNet-50 architecture is used as the baseline residual network. For the CLBP-STIA dataset, its first convolutional layer is adjusted to accommodate six-channel input.

[0113] (5) PSA-ResNet: The proposed PSA-ResNet integrates the Pyramid Segment Attention (PSA) module into the ResNet architecture. Specifically, the Bottleneck structure is replaced with the PSA-Bottleneck structure. The classification performance is evaluated using the STIA and CLBP-STIA datasets. For experiments involving the CLBP-STIA dataset, the first convolutional layer is modified to process six-channel input.

[0114] (6) Parameter setting: All the above methods are implemented based on PyTorch. All experiments are performed on NVIDIA3090 GPU. For all models, the step size of convolution layer and pooling layer is set to 1 and 2 respectively. The learning rate is fixed to 1×10 -4 , the batch size is 32 and the number of training rounds is 100.

[0115] 2. Results Analysis

[0116] The comparison results of model classification performance are shown in Table 1 and Table 2 respectively.

[0117] Table 1: Classification accuracy of main texture directions of spatiotemporal images of various methods under STIA dataset

[0118]

[0119] Table 2: Classification accuracy of main texture directions of spatiotemporal images of various methods under CLBP-STIA dataset

[0120]

[0121] Table 1 (STIA dataset) and Table 2 (CLBP-STIA dataset) show the classification accuracy of various algorithms for spatiotemporal images from different angle categories. As shown in both tables, the proposed PSA-ResNet consistently achieves higher classification accuracy than the other evaluated algorithms, at 96.87% and 98.05%, respectively, demonstrating its superior classification performance. Furthermore, when tested on the CLBP-STIA dataset, which includes enhanced texture features, all algorithms achieved improved accuracy. Notably, PSA-ResNet's performance on this dataset significantly improves, further validating its effectiveness and robustness in capturing spatiotemporal texture information.

[0122] This experiment uses accuracy ACC, precision, recall rate Recall, and F1-score to quantitatively analyze the method proposed in this invention and the comparative method. The results are as follows: Figure 9 and Figure 10 As shown, where:

[0123]

[0124] Where TP represents the spatiotemporal image with the correctly classified angle that meets the "Intersection over Union" (IoU) threshold, otherwise it is considered FP. FN means that there is indeed an angle, but the model did not discover it.

[0125] Figure 9 and Figure 10 The comparison of precision, recall and F1 score of several classification algorithms is shown. Figure 9 and Figure 10 As shown in the figure, the PSA-ResNet algorithm consistently achieves higher precision, recall, and F1 scores in both the STIA and CLBP-STIA experiments. Notably, PSA-ResNet achieves the best performance when tested on the CLBP-STIA dataset. This indicates that the PSA-ResNet algorithm outperforms other classification algorithms in terms of classification performance, with significant improvements observed in the CLBP-STIA experiments.

[0126] In order to evaluate the effectiveness of PSA-ResNet in an actual river environment, this embodiment chose to evaluate it on a real river video. In these experiments, the river video speed measurement method based on the multi-feature fusion PSA-ResNet network (PSA-ResNet-STIV for short) was comprehensively compared with the traditional STIV method and manual angle measurement. After PSA-ResNet-STIV and STIV detected the main texture directions, subsequent calculations were performed to obtain the corresponding flow velocity. In addition, in order to further verify the practical applicability of PSA-ResNet-STIV in flow velocity estimation, a flow velocity comparison test was also carried out using the standard flow meter method specified in GB 50179-2015 "River Flow Measurement Specification".

[0127] The test scenario uses a river video shot on a certain river under clear weather conditions on a certain day. The video is 30 seconds long and the flow measurement line is configured as follows: Figure 11 As shown, the river flow direction is marked as from right to left. Current tracking conditions are affected by various factors, including bank turbulence, obstruction by obstacles, and glare on the water surface. Each flow measurement line is 200 pixels long, and the starting distances of the five measurement lines range from 0.5 to 4.5 meters. Three methods are used to extract the dominant texture directions from the spatiotemporal image: traditional STIV, the proposed PSA-ResNet-STIV, and manual angle measurement. Flow velocity is then calculated from the dominant texture directions and compared with the ground truth obtained using a current meter. The comparison results are summarized in Table 3.

[0128] Table 3: Texture main direction detection results and flow velocity measurement results of real river scene test in the wild

[0129]

[0130] As shown in Table 3, the PSA-ResNet-STIV method achieved higher accuracy than the traditional STIV method for five flow measurement lines with varying levels of interference in the test scene. In particular, the STIV method exhibited greater error in conditions with significant interference, such as turbulence and glare. These results demonstrate the robustness and effectiveness of the PSA-ResNet-STIV method for accurately estimating flow velocity in clear river scenes with complex flow characteristics.

[0131] In general, the river video speed measurement method based on the multi-feature fusion PSA-ResNet network proposed in this paper achieves a better balance between water flow speed measurement efficiency, anti-interference and accuracy.

[0132] Example 2

[0133] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the river video speed measurement method based on the multi-feature fusion PSA-ResNet network as described in Example 1 are implemented.

[0134] like Figure 12 As shown, the computer terminal provided in this embodiment includes: at least one processor 101, and a memory 102 connected to the at least one processor 101. The specific connection medium between the processor 101 and the memory 102 is not limited in this embodiment. Figure 12 In the example, the processor 101 and the memory 102 are connected via the bus 100. Figure 12 The bus 100 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 12 The bus is represented by only one thick line, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 101 can also be called a controller, and there is no limitation on the name.

[0135] In this embodiment, the memory 102 stores instructions that can be executed by at least one processor 101 , and the at least one processor 101 can perform the aforementioned method by executing the instructions stored in the memory 102 .

[0136] Among them, the processor 101 is the control center of the device, which can use various interfaces and lines to connect the various parts of the entire control device, and monitor the device as a whole by running or executing instructions stored in the memory 102 and calling data stored in the memory 102, the various functions of the device and processing data.

[0137] In one possible design, processor 101 may include one or more processing units. Processor 101 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 101. In some embodiments, processor 101 and memory 102 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0138] Processor 101 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the river video speed measurement method based on the multi-feature fusion PSA-ResNet network disclosed in Example 1 can be directly implemented and executed by a hardware processor, or by a combination of hardware and software modules in processor 101.

[0139] The memory 102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 102 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM), a static random access memory (SRAM), a programmable read-only memory (PROM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 102 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 102 in this embodiment can also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.

[0140] By programming the processor 101, the code corresponding to the security verification method described in the above embodiment can be fixed into the chip, so that the chip can execute the security verification method when it is running. Figure 1The steps of the river video speed measurement method based on the multi-feature fusion PSA-ResNet network are shown. How to design and program the processor 101 is a technology well known to those skilled in the art and will not be repeated here.

[0141] Example 3

[0142] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the river video speed measurement method based on the multi-feature fusion PSA-ResNet network as described in Example 1 are implemented.

[0143] The computer-readable storage medium may include flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store the operating system and various application software installed on the computer device. In addition, the memory may also be used to temporarily store various types of data that have been output or are about to be output.

[0144] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A river video speed measurement method based on a multi-feature fusion PSA-ResNet network is characterized by: include: S1. Construct the STIA dataset; the STIA dataset includes spatiotemporal images of rivers and their corresponding texture main direction angle labels; S2. Use the CLBP multi-feature extraction algorithm to extract low-level texture features from the spatiotemporal images in the STIA dataset, generating three feature maps: center pixel encoding, sign component encoding, and amplitude component encoding. These three feature maps are then fused with the original RGB image at the channel level to form a 6-channel CLBP-STIA feature map, which is then used to construct the CLBP-STIA dataset. S3. Integrate the pyramid segmentation attention module (PSA) into the residual block of the ResNet to build an angle classification model, and train the model using the CLBP-STIA dataset. S4. Referring to step S2, an enhanced feature representation of the spatiotemporal image to be detected is generated, which is then input into the trained angle classification model to output the corresponding main texture direction angle; S5. Obtain the actual length of the speed measurement line in the video image, and calculate the actual flow velocity of the surface flow feature on the speed measurement line in combination with the main direction angle of the texture.

2. The river video speed measurement method based on the multi-feature fusion PSA-ResNet network according to claim 1 is characterized in that: Step S1 includes the following specific steps: S11. Record river video; S12. Preprocess the recorded river video, including grayscale conversion, video frame extraction, and noise reduction; S13. Set the size parameters of the speed measurement line and select the speed measurement area; wherein the length of the speed measurement line is L pixels and the width is 1 pixel; S14. Based on the image sequence captured from the preprocessed river video, a velocity measurement line is set in each frame along the direction of water flow. The corresponding velocity measurement lines in m consecutive frames are arranged along the time axis to generate an L×m space-time image. In this image, the horizontal direction is the spatial dimension, i.e., the length of the velocity measurement line, and the vertical direction is the time dimension. S15. Manually analyze the generated spatiotemporal images to identify and annotate the main texture direction angle in each spatiotemporal image, thereby forming the STIA dataset.

3. The river video speed measurement method based on the multi-feature fusion PSA-ResNet network according to claim 1 is characterized in that: In step S11 , a plurality of river videos are recorded under different lighting conditions, weather environments, flow rates or river channel shapes.

4. The river video speed measurement method based on multi-feature fusion PSA-ResNet network according to claim 1 is characterized in that: In step S3, the residual network includes an input layer, a convolutional layer, a residual block, an average pooling layer, a fully connected layer, and a softmax layer; the input layer is used to receive the 6-channel CLBP-STIA feature map; the convolutional layer is used to perform initial feature extraction using a 7×7 convolution kernel; there are multiple residual blocks, each of which is composed of two 3×3 convolutional layers, and the gradient is prevented from disappearing by skip connection; the global average pooling layer is used to pool the feature map to reduce computational complexity; the fully connected layer is used to map the pooled features as classifier input; the softmax layer is used to output the final texture main direction angle category probability; Among them, the pyramid segmentation attention module extracts features of different spatial scales through multi-scale convolution kernels, and performs weighted fusion of features of different scales through the Softmax function, thereby enhancing features related to the classification task and suppressing irrelevant features.

5. The river video speed measurement method based on the multi-feature fusion PSA-ResNet network according to claim 4 is characterized in that: In step S3, the training method of the angle classification model includes: The CLBP-STIA dataset is divided into training set, test set and validation set according to the set ratio; Iteratively training the angle classification model on the training set and adjusting hyperparameters to ensure model convergence; the hyperparameters include learning rate and batch size; Evaluate model performance through validation sets and adjust network structure or training strategies to optimize classification accuracy; The test set is input into the trained angle classification model to predict the main texture direction, and the model performance is evaluated by calculating indicators; the indicators include accuracy, correctness and recall.

6. The river video speed measurement method based on multi-feature fusion PSA-ResNet network according to claim 1 is characterized in that: In step S2, the method for generating the three feature maps includes: For each pixel in the spatiotemporal image, calculate the grayscale difference between it and its n neighboring pixels to generate an n-dimensional difference vector; Decomposing the difference vector into a sign component and an amplitude component; wherein the sign component is generated by comparing the grayscale difference between the central pixel and the surrounding pixels to determine the brightness relationship of each neighboring pixel relative to the central pixel, i.e., a binary sign vector; and the amplitude component is formed by calculating the absolute difference between the central pixel and the surrounding pixels; The grayscale value, sign component and amplitude component of the central pixel are encoded to form a central pixel code, a sign component code and an amplitude component code respectively.

7. The river video speed measurement method based on multi-feature fusion PSA-ResNet network according to claim 1 is characterized in that: In step S5, the actual length of the speed measurement line is calculated as follows: Get the mapping function between the phase plane coordinates (x, y) and the actual space rectangular coordinates (X, Y, Z): In the formula, (x p ,y p ) represents the phase plane coordinate of the principal point; (X p ,Y p ,Z p ) represents the actual space rectangular coordinate of the camera; f represents the focal length; (Δx, Δy) represents the lens distortion correction factor; r ij (i, j = 1 to 3) are the conversion coefficients between the phase plane coordinates and the actual space rectangular coordinates, both of which are unknown parameters; At least six ground calibration signs are set up in advance at the river speed measurement site. The actual spatial rectangular coordinates of each calibration sign relative to the measurement station are measured using a total station. The phase plane coordinates corresponding to each calibration sign are obtained from the video image, and the known coordinates are used to solve the unknown parameters in the mapping function. The actual spatial rectangular coordinates of any point on the space-time image are calculated according to the mapping function and combined with the water level value, thereby obtaining the actual length of the speed measurement line.

8. The river video speed measurement method based on multi-feature fusion PSA-ResNet network according to claim 1 is characterized in that: In step S5, the actual flow velocity v of the surface flow characteristic on the velocity measurement line is calculated according to the following formula: Where α represents the main direction angle of the texture; S x Indicates the actual distance represented by each pixel, the unit is: m / pixel; fps indicates the frame rate of the camera, the unit is: frames / second.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the river video speed measurement method based on the multi-feature fusion PSA-ResNet network as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the steps of the river video speed measurement method based on the multi-feature fusion PSA-ResNet network as described in any one of claims 1 to 8 are implemented.

Images