Flow monitoring method and device fusing video spatio-temporal characteristics and physical parameters

Abstract

The invention discloses a flow monitoring method and device fusing video spatio-temporal characteristics and physical parameters, and aims to break through the bottlenecks that a traditional contact type flow measurement method interferes with water flow, equipment is easy to damage, and the prediction reliability is insufficient due to dependence on a single data source in a complex environment. According to the method, video spatio-temporal dynamic features and hydrological physical parameters are deeply fused, spatio-temporal features are extracted from a video through an encoder based on a three-dimensional spatio-temporal convolutional neural network, and meanwhile two key hydrological physical parameters including the water level and the sectional area are collected. Denoising, normalization and dimension reduction processing are performed on multi-source data, a fusion feature set is constructed, then a combined kernel function Gaussian process regression model is adopted for flow prediction, a complex nonlinear relation is accurately captured, and an uncertainty quantization interval of a prediction result is output. The uncertainty quantification result is derived based on a Bayesian statistical framework, the prediction credibility degree is visually presented in a standard deviation form, and risk grading and early warning of hydrological decision can be directly supported.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of hydrological monitoring and artificial intelligence, specifically to a flow monitoring method and device that integrates video spatiotemporal features and physical parameters. Background Technology

[0002] Flow rate is a key parameter in hydrological monitoring, and its accurate measurement is of great significance for water resource assessment, water conservancy project planning, and flood control scheduling. Currently, traditional flow measurement methods, such as the current meter method and the buoy method, still have significant shortcomings in practical applications. The current meter method requires the equipment to be deployed in the water body, which not only interferes with the water flow but is also susceptible to damage from hydraulic erosion, resulting in high maintenance and operating costs. The buoy method is easily affected by environmental factors such as wind fields and river morphology, leading to poor stability of measurement results. Furthermore, neither of these methods can achieve continuous and automated flow monitoring, limiting their application in large-scale, long-term hydrological monitoring.

[0003] In recent years, with the development of computer vision technology, non-contact flow monitoring methods based on video images have been gradually promoted. However, relying solely on video features for flow measurement still has significant limitations. Video images are easily affected by factors such as changes in lighting, water surface reflection, and obstruction by floating objects, making it difficult to robustly extract spatiotemporal features reflecting the laws of water flow. While hydrophysical parameters such as water level and cross-sectional area can characterize the macroscopic state of water bodies, they cannot effectively capture microscopic dynamic information such as flow velocity distribution and vortex structure. Most existing flow prediction models often rely on a single data source, resulting in insufficient reliability in complex and variable river environments, and generally lack the ability to quantify the uncertainty of prediction results, making it difficult to meet the needs of hydrological risk assessment and decision-making.

[0004] Therefore, existing technologies still have room for optimization in practical applications: traditional contact flow measurement methods are easily constrained by the environment; existing models are not sufficiently capable of integrating and utilizing multi-source data, and fail to provide quantitative information on prediction uncertainty. Summary of the Invention

[0005] This invention provides a flow monitoring method and device that integrates video spatiotemporal features and physical parameters, aiming to solve the problems of limitations in traditional contact flow measurement scenarios, insufficient fusion of multi-source data, and lack of quantification of predicted risks, and to provide a more suitable technical solution for flow monitoring in complex rivers.

[0006] In existing technologies, video-based non-contact flow measurement methods are susceptible to environmental interference and struggle to stably characterize complex flow motions; while models relying solely on hydrophysical parameters cannot capture the microscopic dynamic characteristics of flow. Furthermore, existing flow prediction models lack reliability in complex river topography and fail to quantify prediction uncertainties, making them unsuitable for supporting hydrological decision-making risk assessments.

[0007] This invention provides a traffic monitoring method that integrates video spatiotemporal features and physical parameters, comprising the following steps: Step S101: Collect video data, water level and cross-sectional area (two types of hydrological and physical parameters), and measured flow data of the target river section, and preprocess all data. Step S102: Input the preprocessed video frame sequence into an encoder based on a three-dimensional spatiotemporal convolutional neural network to convert the high-dimensional video data into a low-dimensional spatiotemporal feature vector. Step S103: The extracted low-dimensional video spatiotemporal features are integrated with hydrological physical parameters. First, the differences in dimensions and inconsistencies in numerical ranges between features are eliminated. Then, the features are spliced ​​together to form an initial fused feature set. Subsequently, the initial fused features are standardized and dimensionality reduced. Step S104: Construct a Gaussian process regression model. This regression model uses a combination of a constant kernel, a radial basis kernel, and a white noise kernel to capture the complex nonlinear relationship between flow rate and features and to quantify observation noise. The dimensionality-reduced fused features are used as input, and the measured flow rate is used as the label. The kernel function parameters are optimized through maximum likelihood estimation, and the model is trained and evaluated based on a time series cross-validation strategy. The output is the flow rate prediction value and uncertainty.

[0008] Furthermore, the preprocessing includes: The system performs frame extraction, region of interest (ROI) cropping, size standardization, and format conversion on video data to unify the input format. Outlier cleaning was performed on hydrophysical parameters using the 3σ criterion. The mean μ and standard deviation σ were calculated, and data exceeding the range of [μ-3σ, μ+3σ] were marked as outliers and removed. Noise reduction was performed using the moving average method to reduce the interference of instantaneous fluctuations and improve data quality and consistency.

[0009] Furthermore, the encoder described in step S102 includes multiple 3D spatiotemporal convolution, batch normalization, ReLU activation, Dropout3d, and max pooling operations, and the processing flow is as follows: The input video data is initially extracted using 3D spatiotemporal convolution to extract local spatiotemporal features of the water flow. The convolution output feature tensor is then fed into a batch normalization layer for standardization. A nonlinear transformation is introduced by inputting a ReLU activation function layer; subsequently, a Dropout3d layer randomly discards some feature channels. The feature map is downsampled using a max pooling layer to reduce data dimensionality and computational cost while retaining key feature information; By repeating the above processing steps multiple times, the original high-dimensional video data is gradually converted into low-dimensional video spatiotemporal feature vectors.

[0010] Furthermore, the encoder's processing flow is represented by the following formula:

[0011]

[0012]

[0013]

[0014]

[0015] in, Y conv The convolution outputs a feature tensor; X The input video data is time-series data composed of consecutive frame images; * This represents a 3D spatiotemporal convolution operation; W conv These are the weights of the 3D spatiotemporal convolution kernel. b conv For bias terms; Y bn These are the feature vectors after batch normalization. γ , β These are learnable scaling and offset parameters; μ and σ 2 These are the mean and variance of the features within the batch, respectively; ε It is a small constant used to prevent division by zero; Y relu The feature vector is processed by the ReLU activation function; Y dropout This is the output feature vector after passing through the Dropout3d layer; M For a mask tensor that follows a Bernoulli distribution; Y pool The pooling outputs the feature tensor; i The index of the pooled window element; R This is the collection of elements within the pooled window.

[0016] Furthermore, in step S103, the Min-Max normalization method is used to eliminate the problems of dimensional differences and numerical range inconsistencies between features;

[0017] in, x These are the eigenvalues ​​to be normalized; max train (x) and min train (x)These are the maximum and minimum values ​​of this feature in the training set, respectively. The normalized water level data and cross-sectional area data together constitute the feature vector of hydrophysical parameters. The initial fusion feature set is standardized so that the mean of each feature is 0 and the variance is 1, so as to avoid residual dimensions affecting subsequent processing. Principal component analysis (PCA) is used to reduce the dimensionality of the standardized fusion features. By solving the eigenvalues ​​and eigenvectors of the covariance matrix, the eigenvectors corresponding to the k largest eigenvalues ​​are selected to construct a transformation matrix, which maps the high-dimensional features to the low-dimensional space.

[0018]

[0019] in, F std For the standardized fusion features, μ F The mean vector of the fused features. σ F The standard deviation vector of the fused features; λ i The first covariance matrix is ​​the first... i Large eigenvalues m The original feature dimension is used, and the reduced feature vector is used for subsequent training and prediction of the regression model.

[0020] Furthermore, step S104 also includes dividing the training set and the test set: the fused feature set and the corresponding traffic labels are divided into the training set and the test set according to the time series ratio. The training set is used for model parameter optimization, and the test set is used for model generalization ability evaluation. Based on the Bayesian statistical framework, the GPR model prior is defined, assuming a latent function that maps and fuses features and traffic. f (x) It follows a Gaussian process distribution and employs a combined kernel function of a constant kernel, a radial basis kernel, and a white noise kernel, taking into account signal reference, nonlinear relationship capture, and noise processing.

[0021]

[0022] in, X=[x 1 ,x 2 ,...,x n ] T The input sample matrix; μ(X) It is a mean function; k(X,X') The covariance function, also known as the kernel function, is used to measure the similarity between input samples. k const (x,x') It is a constant core that controls the overall amplitude of the signal, ensuring that the signal amplitude is optimized within a reasonable range. k RBF (x,x') It is a radial basis kernel, which can capture the smooth nonlinear relationship between input features and flow. The length scale parameter is used to control the range of feature influence. k white (x,x') It is a white noise kernel used to process observation noise in the data and improve the model's tolerance to noise.

[0023] Furthermore, the time-series K-Fold cross-validation method is used to evaluate the model's stability, i.e., its generalization ability. The dataset containing fused features and corresponding traffic labels is randomly and uniformly divided into K subsets, and each subset is selected sequentially. D i As a validation set, the model is trained using the remaining subset, and then predictions are made and performance metrics are calculated on the validation set; for the first... i In the next iteration, use the division D i Train the model on all subsets other than the one already trained, and use the trained model to... D i Make predictions.

[0024] Furthermore, it also includes step S105, using the coefficient of determination R. 2 Using mean squared error (MSE) as the core evaluation metric, we comprehensively measure the model's fitting accuracy and generalization ability. Combined with the results of prediction uncertainty analysis, we verify the consistency of the model on the training set and the test set, and evaluate its stability and reliability in the actual complex river environment. Using MSE and R 2 As core evaluation metrics, MSE and R0 for the training and test sets are calculated separately. 2 :

[0025]

[0026] in, n The number of samples in the test set; For the first j Predicted flow values ​​for each sample; For the first j The actual traffic value of each sample; This represents the average of the actual traffic volume.

[0027] The present invention also provides a traffic prediction device that integrates video spatiotemporal features and physical parameters, comprising: The data acquisition and preprocessing module is used to acquire video data, water level and cross-sectional area (two types of hydrological and physical parameters), and measured flow data of the target river section, and to preprocess all the data. The feature extraction module is used to input the preprocessed video frame sequence into the encoder based on a three-dimensional spatiotemporal convolutional neural network, and to transform the high-dimensional video data into a low-dimensional spatiotemporal feature vector. The feature fusion module is used to integrate the extracted low-dimensional video spatiotemporal features with hydrological physical parameters. First, it eliminates the differences in dimensions and inconsistencies in numerical ranges between features, and then splices them together to form an initial fused feature set. Subsequently, the initial fused features are standardized and dimensionality reduced. The model prediction module is used to construct a Gaussian process regression model. This regression model uses a combination of kernel functions, including a constant kernel, a radial basis kernel, and a white noise kernel, to capture the complex nonlinear relationship between flow rate and features and to quantify observation noise. The dimensionality-reduced fused features are used as input, and the measured flow rate is used as the label. The kernel function parameters are optimized through maximum likelihood estimation, and the model is trained and evaluated based on a time series cross-validation strategy. The output is the flow rate prediction value and uncertainty.

[0028] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program is executed by a processor to implement a traffic monitoring method that integrates video spatiotemporal features and physical parameters as described in the above technical solution.

[0029] Compared with existing technologies, this invention has the following advantages: 1. It integrates the spatiotemporal features of video with hydrological physical parameters, utilizing the microscopic dynamic information of video data and the macroscopic physical meaning of hydrological parameters to overcome the technical bias of insufficient prediction reliability of single data sources in complex environments; 2. Compared with traditional flow monitoring methods, this scheme achieves high-precision flow prediction by integrating multi-source features and introducing a Gaussian process regression model. The Gaussian regression method can also achieve uncertainty estimation, possessing the ability to quantify the reliability of prediction results, and can intuitively present the confidence level of prediction results, significantly improving the practical value and decision support of flow monitoring; 3. It adopts the time series cross-validation method, fully considering the temporal characteristics of hydrological data, avoiding future data leakage, and improving the generalization ability and reliability of the model.

[0030] The technical solution provided by this invention has strong versatility and scalability, and can adapt to different river topography and hydrological conditions, providing strong technical support for the construction of smart water conservancy.

[0031] According to the above technical solution, the present invention achieves non-contact, high-precision and quantifiable uncertainty monitoring by deeply integrating video spatiotemporal features with hydrological physical parameters and using a Gaussian process regression (GPR) model for flow prediction. Attached Figure Description

[0032] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is an overall flowchart of a traffic prediction method that integrates video spatiotemporal features and physical parameters according to the present invention; Figure 2 This is an overall flowchart of the traffic prediction model that integrates video spatiotemporal features and physical parameters in an embodiment of the present invention; Figure 3 The images show a comparison of the target water flow region before and after cropping in an embodiment of the present invention. (a) is the original video frame, and (b) is the main water flow region frame after ROI cropping. Figure 4 This is a schematic diagram of the 3D CNN feature extraction sub-process of an embodiment of the present invention, showing the layer-by-layer processing logic of 3D spatiotemporal convolution, batch normalization, ReLU activation, Dropout3d and max pooling; Figure 5 This is a schematic diagram of the multi-source feature fusion sub-process of an embodiment of the present invention, showing the normalization, splicing and PCA dimensionality reduction process of video features and hydrological physical parameters; Figure 6 This is a schematic diagram of the GPR model training and prediction sub-process in an embodiment of the present invention, including the construction of combined kernel functions, K-fold cross-validation, and uncertainty output. Figure 7 This is a scatter plot comparing predicted traffic and actual traffic provided in an embodiment of the present invention. Detailed Implementation

[0033] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0034] This invention discloses a method and apparatus for flow prediction that integrates video spatiotemporal features and physical parameters. It aims to address the technical problems of traditional contact-based flow measurement methods, such as interference with water flow and equipment vulnerability, as well as the insufficient reliability of existing models in complex environments due to reliance on a single data source. This method integrates spatiotemporal dynamic information from river videos with hydrological and physical parameters such as water level and cross-sectional area. By combining feature extraction and regression modeling, it achieves non-contact, high-precision flow prediction and uncertainty quantification. It can be widely applied to hydrological monitoring, flood warning, and water resource allocation scenarios at river cross-sections, providing technical support for smart water conservancy construction.

[0035] Specifically, Figure 1The above is an overall flowchart of a traffic prediction method that integrates video spatiotemporal features and physical parameters, provided in an embodiment of the present invention.

[0036] like Figure 1 As shown, the traffic prediction method integrating video spatiotemporal features and physical parameters in this invention follows the technical logic of "data input - feature processing - model modeling - result output," and sequentially includes five core steps: S101 (multi-source data acquisition and preprocessing), S102 (multi-source feature extraction), S103 (multi-source feature fusion), S104 (regression model training and traffic prediction), and S105 (model performance evaluation). Each step forms a complete technical link through data transmission and logical connection, ensuring the validity of the input data, the comprehensiveness of feature extraction, and the reliability of the prediction results. The specific implementation process of each step is described in detail below: In step S101, multi-source data acquisition and preprocessing, as a fundamental step in the entire model process, the aim is to obtain high-quality, interference-free input data through standardized data acquisition and preprocessing operations, laying the data foundation for subsequent feature extraction and model training. The specific implementation steps are as follows: Multi-source data acquisition: Three types of core data are collected simultaneously to ensure data consistency across time. The first type is river video data, which uses high-definition cameras to continuously film the target river monitoring section, extracting fixed-length video clips at fixed time intervals, and recording the collection timestamps using a standardized naming format for subsequent data alignment. The second type is hydrological and physical parameters, which uses high-precision measuring equipment to simultaneously collect water level and cross-sectional area data at corresponding timestamps. The sampling frequency is strictly consistent with the video extraction frequency to ensure temporal matching with the video data. The third type is measured flow data, which is obtained through professional flow measurement equipment and algorithm calculations for the corresponding time periods, serving as a label benchmark for model training.

[0037] Perform standardization processing on the data to eliminate data noise, outliers, and format differences.

[0038] For hydrophysical parameters, the 3σ criterion is used for outlier cleaning. The mean μ and standard deviation σ of the parameters are calculated. Data that exceed the range of [μ-3σ,μ+3σ] are marked as outliers and removed. At the same time, the moving average method is used for noise reduction. For river video data, the video segments are subjected to frame extraction and standardization. If the number of video frames is less than or equal to the preset number of frames, all frames are extracted and the last frame is repeated to make up to the preset number of frames. If the number of video frames is greater than the preset number of frames, the samples are taken at uniform intervals to the preset number of frames. Based on the actual location of the flow measurement section, the region of interest (ROI) is clipped. The frame size is uniformly adjusted to the preset size and converted to the preset image format. After normalization to a fixed numerical range, it is integrated into the standard data format. For measured flow data, it is matched one by one with video data and hydrological physical parameters based on timestamps, and missing and outlier samples are removed to form a complete dataset.

[0039] Optionally, the preset number of frames can be adjusted according to the intensity of the dynamic changes in the water flow. For example, for turbulent flow, the number of frames can be increased to capture more detailed dynamics.

[0040] Optionally, the determination of the ROI region can be based on real-world location or by automatically identifying water areas and generating a mask using image recognition algorithms.

[0041] In step S102, multi-source feature extraction aims to extract representative features from the preprocessed multi-source data, providing effective input for subsequent feature fusion and model training. The specific implementation steps are as follows: Low-dimensional spatiotemporal feature vectors are extracted from preprocessed video data using a 3D spatiotemporal convolutional encoder.

[0042] Specifically, the encoder consists of multiple layers of 3D spatiotemporal convolutional layers, batch normalization layers, ReLU activation function layers, Dropout3d layers, and max pooling layers stacked sequentially. It initially captures the spatiotemporal local features of water flow through 3D spatiotemporal convolution operations. Batch normalization processing accelerates training convergence and enhances stability. Then, ReLU activation introduces nonlinear transformation to fit complex dynamic relationships. Subsequently, Dropout3d layers are used to randomly discard some feature channels to reduce the risk of overfitting. Finally, max pooling layers downsample the feature map, reducing the data dimensionality while retaining key information. The final output is a 128-dimensional video spatiotemporal feature vector, which effectively represents the direction of water flow, velocity distribution, and dynamic change patterns.

[0043] In step S103, multi-source feature fusion aims to integrate video spatiotemporal features with hydrological and physical parameter features to form a unified and balanced fused feature set. The specific implementation steps are as follows: The extracted 128-dimensional video spatiotemporal feature vectors are integrated with 2-dimensional hydrological physical parameters (water level and cross-sectional area) from multiple sources.

[0044] Optionally, to eliminate the problems of dimensional differences and inconsistent numerical ranges among features, the Min-Max normalization method is used to map various features to the [0,1] interval.

[0045] After normalization, video features and hydrological parameters are directly concatenated to form an initial 130-dimensional fused feature set. To further reduce data redundancy and computational complexity, the initial fused features are standardized to have a mean of 0 and a variance of 1. Dimensionality reduction is then performed using PCA. By solving for the eigenvalues ​​and eigenvectors of the covariance matrix, the eigenvectors corresponding to the k largest eigenvalues ​​are selected to construct a transformation matrix, mapping the high-dimensional features to a low-dimensional space. Here, k is determined by the condition of retaining 90% of the variance information.

[0046] The reduced feature set serves as input to the subsequent regression model, preserving core information while improving model efficiency and generalization ability.

[0047] Optionally, the dimensionality reduction process is not limited to PCA, and other linear or nonlinear dimensionality reduction methods can also be used, such as autoencoders, as long as they can achieve the purpose of preserving core information and reducing redundancy.

[0048] Furthermore, the proportion of variance information retained is not limited to 90%, and can be adjusted between 80% and 95% according to the actual data characteristics and model performance requirements.

[0049] In step S104, during regression model training and traffic prediction, a Gaussian process regression (GPR) model is used to construct the prediction model. This model, based on a Bayesian framework, uses a kernel function to flexibly capture the nonlinear relationship between video spatiotemporal features and physical parameters, achieving accurate traffic prediction. The Gaussian regression method used in this patent can perform uncertainty estimation and has the ability to quantify the reliability of prediction results. Its core logic is as follows: assuming that the mapping function between features and traffic follows a Gaussian process distribution, after optimizing the kernel parameters through training data, the prediction stage simultaneously outputs the posterior mean and posterior variance. The square root of the variance is the uncertainty quantification result. By outputting the probability distribution and confidence interval of the predicted values, the credibility of different prediction results is clearly characterized. In the specific training process, the feature vector fused in S103 is divided into a training set and a test set. The kernel function parameters of the GPR model are optimized using maximum likelihood estimation, enabling the model to accurately retain the uncertainty quantification capability while fitting the patterns of the training data. The dimensionality-reduced fused features are input into a Gaussian process regression (GPR) model, and accurate flow prediction and uncertainty quantification are achieved through a Bayesian framework. The GPR model uses a combination kernel function of "constant kernel + radial basis function (RBF) + white noise kernel".

[0050] Maximum likelihood estimation is used to optimize the hyperparameters of the kernel function, and parameter optimization is achieved by maximizing the marginal likelihood function.

[0051] The model training uses the time series K-fold cross-validation method, which divides the dataset into K subsets in time order, selects each subset as the validation set, and uses the rest as the training set. After repeating this process K times, the average performance index is taken to evaluate the model stability.

[0052] Optionally, the initial parameters of each sub-kernel in the combined kernel function can be set based on historical data or experience, and optimized through the maximum likelihood estimation.

[0053] Optionally, the K value in the time series K-Fold cross-validation can be selected according to the dataset size.

[0054] After training, the test set features are input into the model, and the output traffic prediction value and prediction uncertainty are calculated. The prediction value is calculated from the posterior mean, and the uncertainty is obtained from the square root of the posterior variance.

[0055] In step S105, model performance evaluation, the model's fitting effect and generalization ability are assessed through quantitative indicators, providing a basis for model optimization and practical application. The specific implementation steps are as follows: Using MSE (mean squared error) and R 2 The coefficient of determination (CDR) serves as a core evaluation metric, comprehensively measuring the model's fit and generalization ability. If the performance metrics of the training and test sets are similar and R0 is high, then the model is considered to have good performance. 2 A higher MSE (Mean Sequence Equation) indicates a good model fit and no overfitting. Simultaneously, combining uncertainty analysis results identifies the distribution characteristics of high-risk samples, providing a basis for model optimization and practical application. This embodiment, through the above-described streamlined and modular technical solution, achieves non-contact, high-precision flow prediction, providing reliable technical support for hydrological monitoring in complex river environments.

[0056] Combination Figures 2 to 6 As shown, an embodiment is used to illustrate in detail the working principle of the traffic prediction method that fuses video spatiotemporal features and physical parameters provided by the present invention.

[0057] like Figure 2 As shown, the overall process of the traffic prediction model integrating video spatiotemporal features and physical parameters provided in this embodiment follows a technical chain of "multi-source data input—feature extraction and fusion—model training and prediction—result output and evaluation." Each step forms a complete technical solution through data transmission and logical connection, ensuring that the model has high accuracy and strong generalization ability in practical applications. The following describes the specific implementation methods of each step in detail with reference to the illustrated process: Step S201: Data input and preprocessing stage, synchronously collect video data, hydrological and physical parameters and measured flow data of the target river section to ensure that the multi-source data are strictly aligned in the time dimension.

[0058] If the number of video frames is ≤10, extract all frames and repeat the last frame up to 10 frames; if the number of video frames is >10, sample 10 frames at uniform intervals. Based on the actual location of the flow measurement section, the ROI region is cropped, the frame size is uniformly adjusted to 64×64 and converted to RGB format, normalized to the range [0,1] and then integrated into a NumPy array.

[0059] Outlier cleaning is performed on hydrophysical parameters using the 3σ criterion. The mean μ and standard deviation σ are calculated, and data exceeding the range of [μ-3σ, μ+3σ] are marked as outliers and removed. Optionally, the moving average method can be used for noise reduction to weaken the interference of instantaneous fluctuations and improve data quality and consistency.

[0060] like Figure 3 As shown in the figure, the water flow target area before and after clipping is provided in the embodiment of the present invention, wherein... Figure 3 (a) in the image is the original video frame. Figure 3 (b) in the image shows the main water flow region frame after ROI cropping. This comparison image is used to visually demonstrate the implementation effect of the "ROI cropping" operation in the video data preprocessing stage, and to verify the effectiveness of this operation in removing redundant information and focusing on key water flow regions, providing a visual basis for the subsequent accurate extraction of video spatiotemporal features by 3D CNN.

[0061] Figure 3 The original video frame shown in (a) contains complete scene information of the river monitoring section, including not only the core water flow area but also the static background on both banks of the river (such as riverside vegetation, monitoring equipment, and buildings). This redundant information is not directly related to the water flow state. If it is directly used for subsequent feature extraction, it will not only increase the computational complexity of the model but may also cause the model to learn irrelevant features, weakening its ability to capture the dynamic spatiotemporal features of water flow and affecting the accuracy of subsequent flow prediction.

[0062] Figure 3The cropped video frame shown in (b) is the output after determining the ROI range and performing a cropping operation based on the on-site measurement results of the flow cross-section. Optionally, the determination of the ROI range needs to combine the cross-sectional geometric features and water flow movement patterns of the target river channel, prioritizing the mainstream water flow area as the core focus area to ensure that this area can completely cover the main movement trajectory and dynamic changes of the water flow. The cropping operation uses image coordinate positioning to remove all redundant background areas and interference objects in the original video frame except for the ROI, retaining only the image information of the mainstream water flow area. As can be seen from the comparison results, the cropped video frame focuses on the core water flow area, effectively eliminating the interference of non-target information. This allows the subsequent 3D CNN to more efficiently capture key spatiotemporal features such as the local movement direction of the water flow, flow velocity differences, and wave morphology changes during feature extraction, avoiding interference from irrelevant background information in the feature extraction process, and providing more targeted and effective video feature input for the multi-source feature fusion stage.

[0063] Optionally, in practical applications, the coordinate range and size of the ROI can be dynamically adjusted according to the cross-sectional width of the target river channel, the characteristics of water flow distribution, and the installation angle of the monitoring camera. This ensures that the cropped video frames can fully retain the core dynamic information of the water flow while minimizing redundant data. Simultaneously, the rationality of ROI cropping can be verified through multiple sets of comparative experiments. For example, comparing the efficiency of video feature extraction and model prediction accuracy before and after cropping further demonstrates the positive effect of this operation on improving the overall technical solution performance. This verification process can be performed by referring to the relevant logic for evaluating the data preprocessing effect in the referenced documents, ensuring the rigor and repeatability of the technical solution.

[0064] Step S202: Video spatiotemporal feature extraction, such as... Figure 4 As shown, a 3D CNN is used to construct a feature extraction network to extract features from the preprocessed video data.

[0065] The network consists of multiple 3D spatiotemporal convolutional layers, batch normalization layers, ReLU activation function layers, Dropout3d layers, and max pooling layers in sequence.

[0066] The input video data is initially extracted using 3D spatiotemporal convolution to extract local spatiotemporal features of the water flow. The convolution output feature tensor is then fed into a batch normalization layer for standardization. The input ReLU activation function layer introduces a nonlinear transformation; then, the Dropout3d layer randomly discards some feature channels to reduce the model's dependence on local redundant features. The feature map is downsampled using a max pooling layer to reduce data dimensionality and computational cost while retaining key feature information; By repeating the above process multiple times, the original high-dimensional video data is gradually converted into low-dimensional video spatiotemporal feature vectors.

[0067]

[0068]

[0069]

[0070]

[0071]

[0072] in, Y conv The convolution outputs a feature tensor; X The input video data is time-series data composed of consecutive frame images; * This represents a three-dimensional spatiotemporal convolution operation; W conv These are the weights of the three-dimensional spatiotemporal convolution kernel; b conv For bias terms; Y bn These are the feature vectors after batch normalization. γ , β These are learnable scaling and offset parameters; μ and σ 2 These are the mean and variance of the features within the batch, respectively; ε It is a small constant used to prevent division by zero; Y relu The feature vector is processed by the ReLU activation function; Y dropout This is the output feature vector after passing through the Dropout3d layer; M For a mask tensor that follows a Bernoulli distribution; Y pool The pooling outputs the feature tensor; i The index of the pooled window element; R This is the collection of elements within the pooled window.

[0073] Step S203: In the multi-source feature fusion stage, such as Figure 5 As shown, the fusion features are standardized and PCA dimensionality reduction is performed.

[0074] Feature construction was performed on the preprocessed water level and cross-sectional area data, and the Min-Max normalization method was used to map the data to the [0,1] interval to eliminate dimensional differences. The extracted low-dimensional video spatiotemporal feature vectors are directly concatenated with the hydrological physical parameter feature vectors to form an initial fusion feature set. The concatenation logic is "video spatiotemporal feature vector + hydrological physical parameter feature vector" to ensure that the two types of features are fused under the same data dimension. The initial fusion feature set is standardized so that the mean of each feature is 0 and the variance is 1, so as to avoid residual dimensions affecting subsequent processing. PCA is used to reduce the dimensionality of the standardized fused features. By solving the eigenvalues ​​and eigenvectors of the covariance matrix, the eigenvectors corresponding to the top k largest eigenvalues ​​are selected to construct the transformation matrix, which maps the high-dimensional features to the low-dimensional space. Here, k is determined by "retaining 90% information".

[0075]

[0076]

[0077]

[0078] in, x These are the eigenvalues ​​to be normalized; max train (x) and min train (x) These are the maximum and minimum values ​​of this feature in the training set, respectively. The normalized water level data and cross-sectional area data together constitute the feature vector of hydrophysical parameters. F std For the standardized fusion features, μ F The mean vector of the fused features. σ F The standard deviation vector of the fused features; λ i The first covariance matrix is ​​the first... i Large eigenvalues m The original feature dimension is used, and the reduced feature vector is used for subsequent training and prediction of the regression model.

[0079] Step S204: In the GPR model training and prediction stage, such as Figure 6 As shown, the process involves dividing the training / test set, building and optimizing the GPR model and its parameters, and predicting and outputting the flow and uncertainty.

[0080] The fused feature set and corresponding flow labels are divided into training set and test set according to the time series ratio. The training set is used for model parameter optimization, and the test set is used for model generalization ability evaluation. This ensures that the division conforms to the time series characteristics of hydrological data and avoids future data leakage. Based on the Bayesian statistical framework, the prior of the GPR model is defined by assuming that the latent function f(x) that maps and fuses features and flows follows a Gaussian process distribution. A combined kernel function of "constant kernel + radial basis kernel + white noise kernel" is adopted to take into account signal reference, nonlinear relationship capture and noise processing. The time-series K-Fold cross-validation method is used to evaluate model stability. The dataset, containing fused features and corresponding traffic labels, is randomly and uniformly divided into K subsets. Each subset is selected sequentially as the validation set, and the GPR model is trained using the remaining subsets. Predictions are then made on the validation set, and performance metrics are calculated. Taking the i-th iteration as an example, using D... i Train the GPR model on all subsets except D, and use the trained model to analyze D. i Make predictions.

[0081]

[0082]

[0083] in, X=[x 1 ,x 2 ,...,x n ] T The input sample matrix; μ(X) It is a mean function, and usually the zero mean is taken to simplify the calculation; k(X,X') The covariance function, also known as the kernel function, is used to measure the similarity between input samples. k const (x,x') It is a constant core that can control the overall amplitude of the signal, ensuring that the signal amplitude is optimized within a reasonable range. k RBF (x,x') It is a radial basis kernel, which can capture the smooth nonlinear relationship between input features and flow. The length scale parameter is used to control the range of feature influence. k white (x, x’) It is a white noise kernel used to process observation noise in the data and improve the model's tolerance to noise.

[0084] Step S205: Use MSE and R 2 As a core evaluation indicator.

[0085] Calculate the MSE and R for the training and test sets respectively. 2 If the differences between the metrics of the training set and the test set are small, and the MSE and R are small, 2 A value close to 1 indicates that the model has a good fit and stable generalization ability.

[0086]

[0087]

[0088] in, n The number of samples in the test set; For the first j Predicted flow values ​​for each sample; For the first j The actual traffic value of each sample; This represents the average of the actual traffic volume.

[0089] By combining the results of the prediction uncertainty, we analyze the distribution characteristics of samples with high uncertainty, providing a reference for subsequent model optimization and risk warning in practical applications.

[0090] Figure 7 Here is a scatter plot comparing the predicted traffic and actual traffic in an embodiment of this application: 159 valid samples were selected after preprocessing (including 127 training set samples and 32 test set samples). Each sample contains the fusion features of "video spatiotemporal features + hydrological physical parameters" and the corresponding measured flow label.

[0091] The fused features of 127 training sets are input into the trained GPR model to obtain the predicted traffic of the training set; similarly, the fused features of 32 test sets are input into the model to obtain the predicted traffic of the test set. Scatter plots were created using the measured flow rate of each sample as the x-axis and the corresponding predicted flow rate as the y-axis, with two different labels.

[0092] Add a solid black line that reads "Predicted traffic = Actual traffic" as a theoretical reference line. The closer the data points are to this line, the smaller the deviation between the predicted and actual values.

[0093] This graph is used to visually demonstrate the degree of agreement between the model's predicted values ​​and the measured values, and to quantitatively reflect the model's fitting accuracy and generalization ability.

[0094] Secondly, embodiments of the present invention also provide a traffic prediction device that integrates video spatiotemporal features and physical parameters, comprising: The data acquisition and preprocessing module is used to acquire video data, water level and cross-sectional area (two types of hydrological and physical parameters), and measured flow data of the target river section, and to preprocess all the data. The feature extraction module is used to input the preprocessed video frame sequence into the encoder based on a three-dimensional spatiotemporal convolutional neural network, and to transform the high-dimensional video data into a low-dimensional spatiotemporal feature vector. The feature fusion module is used to integrate the extracted low-dimensional video spatiotemporal features with hydrological physical parameters. First, it eliminates the differences in dimensions and inconsistencies in numerical ranges between features, and then splices them together to form an initial fused feature set. Subsequently, the initial fused features are standardized and dimensionality reduced. The model prediction module is used to construct a Gaussian process regression model. This regression model uses a combination of kernel functions, including a constant kernel, a radial basis kernel, and a white noise kernel, to capture the complex nonlinear relationship between flow rate and features and to quantify observation noise. The dimensionality-reduced fused features are used as input, and the measured flow rate is used as the label. The kernel function parameters are optimized through maximum likelihood estimation, and the model is trained and evaluated based on a time series cross-validation strategy. The output is the flow rate prediction value and uncertainty.

[0095] In a third embodiment of the present invention, a computer-readable storage medium is also provided, including a computer program that, when executed by a processor, implements a traffic monitoring method that integrates video spatiotemporal features and physical parameters as described in the above technical solutions.

[0096] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A traffic monitoring method that integrates video spatiotemporal features and physical parameters, characterized in that, Includes the following steps: Step S101: Collect video data, water level and cross-sectional area (two types of hydrological and physical parameters), and measured flow data of the target river section, and preprocess all data. Step S102: Input the preprocessed video frame sequence into an encoder based on a three-dimensional spatiotemporal convolutional neural network to convert the high-dimensional video data into a low-dimensional spatiotemporal feature vector. Step S103: The extracted low-dimensional video spatiotemporal features are integrated with hydrological physical parameters. First, the differences in dimensions and inconsistencies in numerical ranges between features are eliminated. Then, the features are spliced ​​together to form an initial fused feature set. Subsequently, the initial fused features are standardized and dimensionality reduced. Step S104: Construct a Gaussian process regression model. This regression model uses a combination of a constant kernel, a radial basis kernel, and a white noise kernel to capture the complex nonlinear relationship between flow rate and features and to quantify observation noise. The dimensionality-reduced fused features are used as input, and the measured flow rate is used as the label. The kernel function parameters are optimized through maximum likelihood estimation, and the model is trained and evaluated based on a time series cross-validation strategy. The output is the flow rate prediction value and uncertainty.

2. The method according to claim 1, characterized in that: The preprocessing includes: The system performs frame extraction, region of interest (ROI) cropping, size standardization, and format conversion on video data to unify the input format. Outlier cleaning was performed on hydrophysical parameters using the 3σ criterion. The mean μ and standard deviation σ were calculated, and data exceeding the range of [μ-3σ, μ+3σ] were marked as outliers and removed. Noise reduction was performed using the moving average method to reduce the interference of instantaneous fluctuations and improve data quality and consistency.

3. The method according to claim 1, characterized in that: The encoder described in step S102 includes multiple 3D spatiotemporal convolution, batch normalization, ReLU activation, Dropout3d, and max pooling operations. The processing flow is as follows: The input video data is initially extracted using 3D spatiotemporal convolution to extract local spatiotemporal features of the water flow. The output feature tensor of the convolution is then fed into a batch normalization layer for standardization. The input ReLU activation function layer introduces a nonlinear transformation; Then, some feature channels are randomly discarded using the Dropout3d layer; The feature map is downsampled using a max pooling layer to reduce data dimensionality and computational cost while retaining key feature information; By repeating the above processing steps multiple times, the original high-dimensional video data is gradually converted into low-dimensional video spatiotemporal feature vectors.

4. The method according to claim 1, characterized in that: The encoder's processing flow is represented by the following formula: in, Y conv The convolution outputs a feature tensor; X The input video data is time-series data composed of consecutive frame images; * This represents a 3D spatiotemporal convolution operation; W conv These are the weights of the 3D spatiotemporal convolution kernel. b conv For bias terms; Y bn These are the feature vectors after batch normalization. γ , β These are learnable scaling and offset parameters; μ and σ 2 These are the mean and variance of the features within the batch, respectively; ε It is a small constant used to prevent division by zero; Y relu The feature vector is processed by the ReLU activation function; Y dropout This is the output feature vector after passing through the Dropout3d layer; M For a mask tensor that follows a Bernoulli distribution; Y pool The pooling outputs the feature tensor; i The index of the pooled window element; R This is the collection of elements within the pooled window.

5. The method according to claim 1, characterized in that: In step S103, the Min-Max normalization method is used to eliminate the problems of dimensional differences and numerical range inconsistencies between features; in, x These are the eigenvalues ​​to be normalized; max train (x) and min train (x) These are the maximum and minimum values ​​of this feature in the training set, respectively. The normalized water level data and cross-sectional area data together constitute the feature vector of hydrophysical parameters. The initial fusion feature set is standardized so that the mean of each feature is 0 and the variance is 1, so as to avoid residual dimensions affecting subsequent processing. Principal component analysis (PCA) is used to reduce the dimensionality of the standardized fusion features. By solving the eigenvalues ​​and eigenvectors of the covariance matrix, the eigenvectors corresponding to the k largest eigenvalues ​​are selected to construct a transformation matrix, which maps the high-dimensional features to the low-dimensional space. in, F std For the standardized fusion features, μ F The mean vector of the fused features. σ F The standard deviation vector of the fused features; λ i The first covariance matrix is ​​the first... i Large eigenvalues m The original feature dimension is used, and the reduced feature vector is used for subsequent training and prediction of the regression model.

6. The method according to claim 1, characterized in that: Step S104 also includes dividing the training set and the test set: the fused feature set and the corresponding traffic labels are divided into the training set and the test set according to the time series ratio. The training set is used for model parameter optimization, and the test set is used for model generalization ability evaluation. Based on the Bayesian statistical framework, the GPR model prior is defined, assuming a latent function that maps and fuses features and traffic. f(x) It follows a Gaussian process distribution and employs a combined kernel function of a constant kernel, a radial basis kernel, and a white noise kernel, taking into account signal reference, nonlinear relationship capture, and noise processing. in, X=[x 1 ,x 2 ,...,x n ] T The input sample matrix; μ(X) It is a mean function; k(X,X') The covariance function, also known as the kernel function, is used to measure the similarity between input samples. k const (x,x') It is a constant core that controls the overall amplitude of the signal, ensuring that the signal amplitude is optimized within a reasonable range. k RBF (x,x') It is a radial basis kernel, which can capture the smooth nonlinear relationship between input features and flow. The length scale parameter is used to control the range of feature influence. k white (x,x') It is a white noise kernel used to process observation noise in the data and improve the model's tolerance to noise.

7. The method according to claim 1, characterized in that: The time-series K-Fold cross-validation method is used to evaluate the model's stability, i.e., its generalization ability. The dataset containing fused features and corresponding traffic labels is randomly and uniformly divided into K subsets, and each subset is selected sequentially. D i As a validation set, the model is trained using the remaining subset, and then predictions are made and performance metrics are calculated on the validation set; for the first... i In the next iteration, use the division D i Train the model on all subsets other than the one already trained, and use the trained model to... D i Make predictions.

8. The method according to claim 1, characterized in that: It also includes step S105, which uses the coefficient of determination R. 2 Using mean squared error (MSE) as the core evaluation metric, we comprehensively measure the model's fitting accuracy and generalization ability. Combined with the results of prediction uncertainty analysis, we verify the consistency of the model on the training set and the test set, and evaluate its stability and reliability in the actual complex river environment. Using MSE and R 2 As core evaluation metrics, MSE and R0 for the training and test sets are calculated separately. 2 : in, n The number of samples in the test set; For the first j Predicted flow values ​​for each sample; For the first j The actual traffic value of each sample; This represents the average of the actual traffic volume.

9. A traffic prediction device that integrates video spatiotemporal features and physical parameters, characterized in that, include: The data acquisition and preprocessing module is used to acquire video data, water level and cross-sectional area (two types of hydrological and physical parameters), and measured flow data of the target river section, and to preprocess all the data. The feature extraction module is used to input the preprocessed video frame sequence into the encoder based on a three-dimensional spatiotemporal convolutional neural network, and to transform the high-dimensional video data into a low-dimensional spatiotemporal feature vector. The feature fusion module is used to integrate the extracted low-dimensional video spatiotemporal features with hydrological physical parameters. First, it eliminates the differences in dimensions and inconsistencies in numerical ranges between features, splices them together to form an initial fused feature set, and then standardizes and reduces the dimensionality of the initial fused features. The model prediction module is used to construct a Gaussian process regression model. This regression model uses a combination of kernel functions, including a constant kernel, a radial basis kernel, and a white noise kernel, to capture the complex nonlinear relationship between flow rate and features and to quantify observation noise. The dimensionality-reduced fused features are used as input, and the measured flow rate is used as the label. The kernel function parameters are optimized through maximum likelihood estimation, and the model is trained and evaluated based on a time series cross-validation strategy. The output is the flow rate prediction value and uncertainty.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement a traffic monitoring method that integrates video spatiotemporal features and physical parameters as described in any one of claims 1-8.

Images