Intelligent monitoring system and method for water conservancy projects

Abstract

The present application relates to the field of water conservancy project monitoring, and specifically discloses an intelligent monitoring system and method for water conservancy projects, which first obtains a monitoring video of a water conservancy project gate collected by a camera, water level change values ​​on both sides of the gate at multiple predetermined time points collected by a water level sensor installed on the gate, and water flow values ​​at multiple predetermined time points collected by a water flow sensor installed on the gate, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether to issue a safety warning for the water conservancy project gate, thereby reducing the shortcomings of human observation, improving the efficiency and response speed of the system, and timely preventing potential dangers, thereby providing more targeted decision support and improving the management and maintenance strategies of water conservancy projects.

Description

Technical Field

[0001] The present application relates to the field of water conservancy project monitoring, and more specifically, to an intelligent monitoring system and method for water conservancy projects. Background Art

[0002] In the field of water conservancy projects such as hydropower generation and irrigation, gates are indispensable components, mainly used to intercept and store water in rivers, ditches, etc. to ensure the effective management and distribution of water resources. The performance of gates depends largely on their interception and storage effect, which is a key factor in evaluating their technical indicators.

[0003] However, existing technical means usually rely on manual observation to monitor the interception effect of the gate. Especially on small gate equipment, this method has obvious limitations due to the lack of dedicated personnel on duty. Even on large gate equipment, it is difficult for on-duty personnel to achieve 24-hour uninterrupted monitoring, especially at night or in bad weather conditions. This monitoring method is prone to delays in discovering abnormalities in the gate status, such as damage or leakage, which reduces the interception effect and may cause serious water flow safety hazards, such as a sudden increase in upstream water volume and entering the downstream, causing abnormally high downstream water levels, and even causing disasters such as flooding.

[0004] Therefore, an intelligent monitoring system and method for water conservancy projects is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an intelligent monitoring system and method for a water conservancy project, which first obtains a monitoring video of a water conservancy project gate collected by a camera, water level change values ​​on both sides of the gate at multiple predetermined time points collected by a water level sensor installed on the gate, and water flow values ​​at multiple predetermined time points collected by a water flow sensor installed on the gate, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether to issue a safety warning for the water conservancy project gate, thereby reducing the shortcomings of human observation, improving the efficiency and response speed of the system, and timely preventing potential dangers, thereby providing more targeted decision support and improving the management and maintenance strategies of water conservancy projects.

[0006] According to one aspect of the present application, there is provided a water conservancy project intelligent monitoring system, which includes:

[0007] A water conservancy project monitoring data acquisition module is used to acquire water conservancy project gate monitoring videos collected by cameras, water level change values ​​on both sides of the gate at multiple predetermined time points collected by water level sensors installed on the gates, and water flow values ​​at multiple predetermined time points collected by water flow sensors installed on the gates;

[0008] A water conservancy project monitoring data extraction module, used to extract the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector from the water conservancy project gate monitoring video collected by the camera, the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow value at multiple predetermined time points collected by the water flow sensor installed on the gate;

[0009] The water conservancy project gate early warning module is used to determine whether to issue a water conservancy project gate safety early warning based on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector.

[0010] According to another aspect of the present application, a water conservancy project intelligent monitoring method is provided, which includes:

[0011] Obtaining a monitoring video of a water conservancy project gate collected by a camera, water level change values ​​on both sides of the gate at multiple predetermined time points collected by a water level sensor installed on the gate, and water flow values ​​at multiple predetermined time points collected by a water flow sensor installed on the gate;

[0012] Extracting a water conservancy project gate monitoring key frame associated feature vector and a water conservancy project safety multimodal associated feature vector from the water conservancy project gate monitoring video collected by the camera, the water level change values ​​at both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow rate values ​​at multiple predetermined time points collected by the water flow sensor installed on the gate;

[0013] Based on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector, it is determined whether to issue a water conservancy project gate safety warning.

[0014] Compared with the prior art, the present application provides an intelligent monitoring system and method for water conservancy projects, which first obtains the monitoring video of the water conservancy project gate collected by a camera, the water level change values ​​on both sides of the gate at multiple predetermined time points collected by a water level sensor installed on the gate, and the water flow rate values ​​at multiple predetermined time points collected by a water flow sensor installed on the gate, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether to issue a safety warning for the water conservancy project gate, thereby reducing the shortcomings of human observation, improving the efficiency and response speed of the system, and timely preventing potential dangers, thereby providing more targeted decision-making support and improving the management and maintenance strategies of water conservancy projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a block diagram of an intelligent monitoring system for a water conservancy project according to an embodiment of the present application.

[0017] Figure 2 It is a block diagram of a water conservancy project monitoring data extraction module in the water conservancy project intelligent monitoring system according to an embodiment of the present application.

[0018] Figure 3 It is a block diagram of a water flow feature extraction unit in an intelligent monitoring system for a water conservancy project according to an embodiment of the present application.

[0019] Figure 4 It is a block diagram of a water conservancy project gate warning module in a water conservancy project intelligent monitoring system according to an embodiment of the present application.

[0020] Figure 5 It is a flow chart of the intelligent monitoring method of water conservancy projects according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0022] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0023] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0024] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0025] Figure 1 Schematic diagram of the block diagram of the intelligent monitoring system for water conservancy projects in the embodiment of the present application. Figure 1 As shown, the intelligent monitoring system 100 of a water conservancy project according to an embodiment of the present application includes: a water conservancy project monitoring data acquisition module 110, which is used to obtain a water conservancy project gate monitoring video collected by a camera, water level change values ​​on both sides of the gate at multiple predetermined time points collected by a water level sensor installed on the gate, and water flow values ​​at multiple predetermined time points collected by a water flow sensor installed on the gate; a water conservancy project monitoring data extraction module 120, which is used to extract a water conservancy project gate monitoring key frame associated feature vector and a water conservancy project safety multimodal associated feature vector from the water conservancy project gate monitoring video collected by the camera, the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow values ​​at multiple predetermined time points collected by the water flow sensor installed on the gate; a water conservancy project gate early warning module 130, which is used to determine whether to issue a water conservancy project gate safety early warning based on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector.

[0026] In the above-mentioned intelligent monitoring system 100 for water conservancy projects, the water conservancy project monitoring data acquisition module 110 is used to obtain the monitoring video of the water conservancy project gate collected by the camera, the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow value at multiple predetermined time points collected by the water flow sensor installed on the gate. It should be understood that in the hydropower generation and irrigation system, the gate is a key component used to intercept and regulate the water flow in rivers and ditches in order to effectively manage and allocate water resources. The core of evaluating the performance of the gate is its interception and storage effect. At present, monitoring the interception and storage effect of the gate often relies on manual observation, especially on small gate equipment. Due to the lack of full-time personnel, this method has significant limitations. Even for large gate equipment, it is difficult for the on-duty personnel to achieve all-weather monitoring, especially at night or in severe weather conditions. Manual monitoring may lead to delayed discovery of abnormal gate status (such as damage or leakage), thereby affecting the interception and storage effect. This situation may cause water flow safety problems, such as a sharp increase in upstream water volume flowing downstream, resulting in abnormally high downstream water levels, and even flooding disasters. Therefore, in the technical solution of this application, by obtaining the monitoring video of the water conservancy project gate collected by the camera, the water level change values ​​on both sides of the gate collected by the water level sensor installed on the gate at multiple predetermined time points, and the water flow value collected by the water flow sensor installed on the gate at multiple predetermined time points, and combining with deep learning technology, it is determined whether to issue a safety warning for the water conservancy project gate, so as to reduce the shortcomings of manual observation, improve the efficiency and response speed of the system, and thus timely identify and prevent potential risks. This will help provide more accurate decision support and optimize the management and maintenance strategies of water conservancy projects.

[0027] Specifically, the real-time video data provided by the camera can intuitively capture the actual working conditions of the gate, including the switch status, whether the operation is smooth, etc. This visual information is an important basis for judging whether the gate has mechanical failures, water leakage and other problems. Video monitoring can help to detect problems in a timely manner and make corresponding treatments, especially when manual monitoring is difficult to continue. On the other hand, the water level change values ​​recorded by the water level sensor reflect the dynamic changes of the water level on both sides of the gate. These data can provide quantitative information about water flow and its changes, which is crucial for evaluating the interception effect and water flow management of the gate. The water flow value provided by the water flow sensor can quantitatively describe the water flow through the gate. The flow data combined with the water level change can provide a more comprehensive understanding of the overall performance of the gate. Flow information can help optimize the allocation of water resources while preventing potential safety problems caused by excessive flow. The comprehensive analysis of these three types of data can provide a multi-dimensional perspective to help improve the understanding and prediction of the working status of the gate, thereby ensuring the safety and efficiency of water conservancy projects. This intelligent monitoring method is more accurate and timely than traditional manual observation, and is the key to realizing the automation and intelligence of modern water conservancy projects.

[0028] In the above-mentioned intelligent monitoring system 100 for water conservancy projects, the water conservancy project monitoring data extraction module 120 is used to extract the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector from the water conservancy project gate monitoring video collected by the camera, the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow value at multiple predetermined time points collected by the water flow sensor installed on the gate. In this way, the intelligent monitoring system can combine visual information with quantitative data to form a comprehensive safety monitoring model, thereby enabling the system to perform more complex pattern recognition and anomaly detection, so as to timely discover problems and take measures to reduce water safety hazards caused by gate failures.

[0029] Figure 2 FIG. 1 is a block diagram of a water conservancy project monitoring data extraction module in a water conservancy project intelligent monitoring system according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the water conservancy project monitoring data extraction module 120 includes: a gate monitoring feature extraction unit 121, which is used to perform feature extraction on the water conservancy project gate monitoring video collected by the camera to obtain multiple water conservancy project gate monitoring key frame image block embedding vectors; a gate monitoring feature encoding unit 122, which is used to perform feature encoding on the multiple water conservancy project gate monitoring key frame image block embedding vectors to obtain the water conservancy project gate monitoring key frame associated feature vector; a water level change feature extraction unit 123, which is used to perform feature extraction on the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate to obtain the water level change feature vector on both sides of the gate; a water flow feature extraction unit 124, which is used to perform feature extraction on the water flow values ​​at multiple predetermined time points collected by the water flow sensor installed on the gate to obtain the water flow feature vector; a water conservancy project safety multimodal association unit 125, which is used to associate the water level change feature vector on both sides of the gate with the water flow feature vector to obtain the water conservancy project safety multimodal association feature vector.

[0030] It should be understood that video surveillance provides dynamic visual information of the gate. By extracting feature vectors from keyframe image blocks, the specific state of the gate at different time points can be captured. Keyframes are highly representative frames in the video. They can accurately show the current state of the gate and its changes, such as the switch state, whether there is water leakage or other mechanical failures. These keyframes can help identify potential problems, such as whether the gate is opened or closed normally, and whether there are abnormal movements or defects. Embedding keyframe image blocks into vector processing can convert image data into digital features, which can be more conveniently used for further analysis and processing. Image block embedding vectors are high-dimensional numerical representations extracted from image blocks by feature encoders. They contain key visual information of the image, such as edges, textures, color distribution, etc. This conversion enables image data to be combined with other types of data (such as sensor data), thereby achieving multimodal data fusion and more comprehensive analysis.

[0031] Furthermore, feature encoding is performed on the embedded vectors of multiple key frame image blocks for monitoring water conservancy project gates, which can compress the information in the embedded vectors into more representative and recognizable feature vectors. Each key frame image block embedding vector contains the basic visual features of the image, such as texture, shape, color, etc., but these features may be very large and redundant. Through feature encoding, the system can map these high-dimensional embedded vectors to a lower-dimensional space while retaining the most critical information. This efficient data representation can reduce computational complexity and improve the efficiency of data processing.

[0032] Furthermore, considering the huge and continuous amount of data provided by the water level sensor, feature extraction of this data can transform complex time series information into concise feature vectors. These feature vectors can compress and summarize the key features of water level changes, such as the fluctuation amplitude, change rate and trend of the water level. In this way, important information can be extracted from a large amount of data, avoiding the direct processing of lengthy raw data, thereby improving the efficiency of data processing and analysis. The obtained water level change feature vector can reveal the dynamic behavior and potential problems of the water level on both sides of the gate in the water conservancy project.

[0033] In particular, the data provided by water flow sensors usually include a large amount of time series information. Directly processing these raw data may be very cumbersome and inefficient. Through feature extraction, these data can be compressed into a set of concise feature vectors to extract the key features of water flow. For example, the average value, rate of change, peak value and periodicity of the flow. This feature vector can effectively summarize the main trends and behaviors of water flow, avoid processing lengthy and complex raw data, and improve the efficiency of data analysis.

[0034] More specifically, associating the water level change feature vectors on both sides of the gate and the water flow flow feature vectors can make comprehensive use of different types of data to more comprehensively and accurately evaluate the operating status and safety of water conservancy projects. This multimodal data fusion method can significantly improve the understanding and management capabilities of the complex dynamic behavior of water conservancy projects. Among them, the individual water level change feature vectors and water flow flow feature vectors each provide information on different aspects of the water conservancy system. The water level change feature vector mainly reflects the dynamic changes of the water level on both sides of the gate, which may be related to the water flow, gate operation status and other environmental factors. The water flow flow feature vector provides direct information about the intensity and changes of the water flow, which can reveal the fluctuations, trends and anomalies of the flow. Therefore, a single data source may not fully reflect the full picture of the water conservancy project, especially in complex situations. By associating these two types of feature vectors, a multi-angle and all-round analysis of the water conservancy project can be achieved. There is usually a complex relationship between water level and flow, and changes in water level will directly affect the flow, while changes in flow will in turn affect the water level. Combining water level changes and flow characteristics can more accurately describe the overall dynamics of the water flow.

[0035] In a specific embodiment of the present application, the gate monitoring feature extraction unit 121 includes: extracting a water conservancy project gate monitoring key frame sequence from the water conservancy project gate monitoring video collected by the camera; linearly embedding the water conservancy project gate monitoring key frame sequence through a water conservancy project gate monitoring key frame image block sequence into an encoder to obtain the multiple water conservancy project gate monitoring key frame image block embedding vectors.

[0036] It should be understood that video surveillance usually generates a large number of continuous frames, which contain a lot of repeated information and redundant data. Directly processing the entire video stream is not only computationally burdensome, but may also waste a lot of storage and processing resources. By extracting key frame sequences, the information in the video can be compressed into a series of important static images that can represent the main scenes and changes in the video, thereby reducing the amount of data and processing complexity. Key frame extraction helps capture important events and state changes in water conservancy projects. Key frames usually select frames with significant changes or features in the video, such as the moment of gate opening and closing, the occurrence of abnormal conditions, etc. By analyzing these key frames, it is easier to identify and record important operations or abnormal conditions. This method can provide clear visual evidence to help engineers make more accurate status assessments, fault diagnosis, and maintenance decisions.

[0037] Furthermore, key frames in surveillance videos usually contain rich image information, but directly processing these high-resolution images may lead to excessive computational burden and difficulty in extracting useful features. The linear embedding encoder can effectively compress image information into a low-dimensional vector form by decomposing each key frame image into smaller image blocks and converting these image blocks into embedding vectors. These embedding vectors can retain key image features while reducing the dimension of the data, making subsequent data analysis and processing more efficient. Among them, the image block sequence linear embedding encoder can capture and represent the local features of each image block in the key frame. The extraction of such local features helps to identify detailed information in the image, such as the specific status of the gate, the operating position, and the water flow. These features can be accurately represented in the embedding vector, allowing the system to deeply analyze the important information in the image and support more accurate monitoring and diagnosis.

[0038] In a specific embodiment of the present application, the gate monitoring feature encoding unit 122 includes: two-dimensionally arranging the multiple water conservancy project gate monitoring key frame image block embedding vectors to obtain a water conservancy project gate monitoring key frame image block input matrix; passing the water conservancy project gate monitoring key frame image block input matrix through a water conservancy project gate monitoring key frame image block feature encoder to obtain the water conservancy project gate monitoring key frame associated feature vector.

[0039] It should be understood that the two-dimensional arrangement organizes the embedding vectors of the image blocks into a matrix form, which can systematically represent the spatial relationship between the image blocks. The embedding vector of each image block contains the feature information of the block, and the two-dimensional arrangement can preserve the position and layout of the image blocks in the keyframe. This preservation of spatial structure is very important for identifying patterns and structures in images. For example, when analyzing the state of a gate or water flow, preserving the relative position of image blocks helps the system better understand and parse the spatial features and changes in the image.

[0040] Furthermore, the image block feature encoder can extract higher-level and more abstract features from the input matrix. The original image block input matrix usually contains a large amount of pixel information and low-level features, and the feature encoder can refine this information into more representative feature vectors by applying complex transformations and models (such as convolutional neural networks, fully connected layers, etc.). These feature vectors can comprehensively consider the key features in the image block, thereby more accurately representing important information such as the state, structure, and environmental conditions of the gate. The feature encoder helps capture the relationship and global pattern between image blocks by integrating the embedding vectors of the image blocks into a related feature vector. Each image block in the input matrix represents the local information of the image, and the encoder can synthesize this local information to form a global understanding of the entire image. By integrating the features of the image blocks into a related feature vector, the overall structure, changing trends, and potential problems in the image can be better identified and analyzed, which is crucial for monitoring and evaluating the status of water conservancy projects. Specifically, each layer of the water conservancy project gate monitoring key frame image block feature encoder is used to perform convolution processing, mean pooling processing based on the local feature matrix and nonlinear activation processing on the input data in the forward transfer of the layer, so that the last layer of the water conservancy project gate monitoring key frame image block feature encoder outputs the water conservancy project gate monitoring key frame associated feature vector, wherein the input of the water conservancy project gate monitoring key frame image block feature encoder is the water conservancy project gate monitoring key frame image block input matrix.

[0041] In a specific embodiment of the present application, the water level change feature extraction unit 123 includes: arranging the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate as a water level change input vector on both sides of the gate; passing the water level change input vector on both sides of the gate through a water level change timing encoder on both sides of the gate to obtain the water level change feature vector on both sides of the gate.

[0042] It should be understood that arranging the water level change values ​​as input vectors can organize time series data into a unified structure. This structured data form helps to more clearly show the changing trends and patterns of water levels at different time points. For example, by arranging the water level change values ​​in chronological order, the rising, falling or stable trends of the water level can be intuitively observed, which is crucial for analyzing the impact of gate operation on the water level. The arrangement of time series data helps in trend analysis and prediction. By converting the water level change values ​​into input vectors, time series analysis techniques or machine learning models can be used to identify potential patterns, periodic changes, and anomalies. This analysis can help predict future water level changes and provide a basis for the scheduling and management of water conservancy projects, thereby improving the predictability and responsiveness to water level changes.

[0043] Furthermore, the time series encoder can effectively capture the dynamic features and patterns in time series data. The change of water level on both sides of the gate is a time-varying process involving complex time dependencies. Through time series encoders, such as recurrent neural networks (RNN), long short-term memory networks (LSTM), or gated recurrent units (GRU), the temporal correlation in water level change data can be extracted to identify and model temporal trends, periodicities, and anomalies. This is crucial for understanding the long-term trend of water level changes and predicting future water level levels. The time series encoder can compress the high-dimensional time series data in the input vector into a low-dimensional feature vector while retaining key time series information. This feature vector is obtained by learning the temporal dependencies and important features of the input data, which can effectively reduce the complexity of the data and improve the analysis efficiency. The low-dimensional feature vector not only reduces the computational burden, but also improves the efficiency of subsequent data processing and model training, making the real-time analysis and decision-making of water level changes faster and more accurate. Specifically, the fully connected layer of the water level change timing encoder on both sides of the gate is used to perform fully connected encoding on the input vector of the water level change on both sides of the gate to extract high-dimensional implicit features of the eigenvalues ​​at each position in the input vector of the water level change on both sides of the gate; and the one-dimensional convolutional layer of the water level change timing encoder on both sides of the gate is used to perform one-dimensional encoding on the input vector of the water level change on both sides of the gate to extract high-dimensional implicit correlation features of the correlation between the eigenvalues ​​at each position in the input vector of the water level change on both sides of the gate.

[0044] Figure 3 FIG. 1 is a block diagram of a water flow feature extraction unit in a water conservancy project intelligent monitoring system according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the water flow feature extraction unit 124 includes: a water flow arrangement subunit 1241, used to arrange the water flow values ​​at multiple predetermined time points collected by the water flow sensor installed at the gate into a water flow input vector; a water flow convolution encoding subunit 1242, used to pass the water flow input vector through a water flow sequence encoder including a one-dimensional convolution layer and a fully connected layer to obtain the water flow feature vector.

[0045] It should be understood that arranging the flow values ​​as input vectors helps to establish a clear data structure so that the time series data can be normalized. This structured data form makes the patterns and trends of flow changes at different time points intuitive and easy to analyze. For example, by arranging the flow values ​​in chronological order, the changing trend of the flow can be intuitively observed, such as increasing, decreasing, or remaining stable. This is very important for understanding the time series characteristics of flow and its impact on gate operation.

[0046] Furthermore, in the water flow sequence encoder comprising a one-dimensional convolutional layer and a fully connected layer, the one-dimensional convolutional layer can effectively extract local features from time series data. For water flow data, the convolution operation can identify the changing patterns and features of flow values ​​in a short period of time. By performing convolution calculations on the input vector with a sliding convolution kernel (or filter), the one-dimensional convolutional layer can automatically detect important local patterns in the data, such as sudden increases or decreases in flow, periodic fluctuations, etc. The role of the fully connected layer in the sequence encoder is to convert the feature map extracted by the convolutional layer into the final feature vector. This layer is responsible for mapping the high-dimensional feature map generated by the convolutional layer to a low-dimensional feature space through a fully connected operation. The fully connected layer further processes and integrates these features through linear combinations and nonlinear activation functions, so that the final feature vector can contain comprehensive information of the flow data. By passing the water flow input vector to the sequence encoder comprising a one-dimensional convolutional layer and a fully connected layer, the feature extraction and transformation of time series data can be effectively achieved. This method can not only capture local patterns in the data, but also integrate and compress information through the fully connected layer. Specifically, the water flow input vector is fully-connected using the fully-connected layer of a water flow sequence encoder comprising a one-dimensional convolutional layer and a fully-connected layer to extract high-dimensional implicit features of the eigenvalues ​​at each position in the water flow input vector; and, the water flow input vector is one-dimensionally encoded using the one-dimensional convolutional layer of a water flow sequence encoder comprising a one-dimensional convolutional layer and a fully-connected layer to extract high-dimensional implicit correlation features of the correlation between the eigenvalues ​​at each position in the water flow input vector.

[0047] In the above-mentioned intelligent monitoring system 100 for water conservancy projects, the water conservancy project gate warning module 130 is used to determine whether to issue a water conservancy project gate safety warning based on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector. It should be understood that the gate monitoring key frame associated feature vector provides detailed information about the gate status and operation. These feature vectors may contain extracted features from video surveillance, sensor data and other real-time monitoring information, which can reveal the current state, operation mode and potential abnormal conditions of the gate. The water conservancy project safety multimodal associated feature vector combines data from different monitoring sources, such as water flow, water level changes, meteorological conditions, etc. By correlating and analyzing different types of data (such as sensor data, image data, and environmental data), potential risk factors and abnormal patterns can be more accurately identified to improve the accuracy and reliability of the early warning system.

[0048] Figure 4 FIG. 1 is a block diagram of a water conservancy project gate warning module in a water conservancy project intelligent monitoring system according to an embodiment of the present application. Figure 4As shown, in a specific embodiment of the present application, the water conservancy project gate warning module 130 includes: a water conservancy project gate safety feature fusion unit 131, which is used to perform feature fine-grained dimension offset compensation based on local structure on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector to obtain a water conservancy project gate safety warning judgment feature vector; a water conservancy project gate safety warning judgment unit 132, which is used to pass the water conservancy project gate safety warning judgment feature vector through a classifier to obtain a classification result, and the classification result is used to determine whether to issue a water conservancy project gate safety warning.

[0049] It should be understood that fusing feature vectors from different sources can integrate multi-dimensional information, thereby providing a more comprehensive understanding of the status of the gates of a water conservancy project. Keyframe-associated feature vectors are mainly derived from video surveillance or image data, which can capture the actual operating status and visualization characteristics of the gates, such as physical structure changes, surface cracks, and motion anomalies. Safety multimodal-associated feature vectors combine data from sensors, such as water flow, water level, and meteorological conditions. By fusing these two types of feature vectors, the operating status of the gates can be considered together with environmental factors, thereby forming a more comprehensive safety assessment of the entire water conservancy system.

[0050] In particular, in the technical solution of the present application, the key frame of the water conservancy project gate monitoring contains detailed image features, while the water level and flow features reflect the dynamic changes in the time series. The local information of different modal features cannot be effectively aligned or synthesized, which may lead to the loss of the correlation between the image features and the time series features. In addition, the data processed by different feature encoders (such as the key frame image block feature encoder of the water conservancy project gate monitoring, the time series encoder of the water level change on both sides of the gate, and the water flow sequence encoder) have different representation methods and dimensions. These different feature encoders focus on image features, time series features, and flow features, respectively. Although these features are semantically related, they may differ in feature space. For example, image features usually express complex visual information in high-dimensional space, while water level and water flow features exist in time series and numerical forms. When these features are simply fused, dimensional inconsistency may occur, resulting in the loss of fine granularity of information during the integration process. This will lead to inconsistencies in feature representation of different data modalities and difficulties in information alignment during the fusion process. Therefore, in the technical solution of the present application, a water conservancy project gate safety warning judgment feature vector is obtained by performing feature fine-grained dimension offset compensation based on local structure on the key frame associated feature vector of the water conservancy project gate monitoring and the multimodal associated feature vector of the water conservancy project safety.

[0051] Among them, the local structure-based feature fine-grained dimension offset compensation is performed on the key frame associated feature vector of the water conservancy project gate monitoring and the multimodal associated feature vector of the water conservancy project safety to obtain the water conservancy project gate safety early warning judgment feature vector, including: calculating the position difference between the key frame associated feature vector of the water conservancy project gate monitoring and the multimodal associated feature vector of the water conservancy project safety to obtain the offset compensation differential feature vector; calculating the position point multiplication between the key frame associated feature vector of the water conservancy project gate monitoring and the multimodal associated feature vector of the water conservancy project safety to obtain the offset compensation position point multiplication feature vector; calculating the position addition between the key frame associated feature vector of the water conservancy project gate monitoring and the multimodal associated feature vector of the water conservancy project safety to obtain the offset compensation position addition feature vector; adding the offset The compensated differential feature vector, the offset compensation point product feature vector and the offset compensation position addition feature vector are cascaded and input into a one-dimensional convolution layer and maximum pooling is performed to obtain an offset compensation convolutional coding pooling feature vector; the square root of the two norm of the offset compensation convolutional coding pooling feature vector is weighted and added with the one norm of the offset compensation convolutional coding pooling feature vector to obtain an offset compensation factor; the offset compensation point product feature vector and the square root of the length of the feature vector associated with the key frame of the water conservancy project gate monitoring are added by position to obtain an offset compensation intermediate vector; the offset compensation factor and the second weighted hyperparameter are used as weighting coefficients to calculate the position weighted sum of the offset compensation position addition feature vector and the offset compensation intermediate vector to obtain the water conservancy project gate safety warning judgment feature vector.

[0052] Among them, the key frame associated feature vector of the water conservancy project gate monitoring and the multi-modal associated feature vector of the water conservancy project safety are compensated for the feature fine-grained dimension offset based on the local structure to obtain the water conservancy project gate safety early warning judgment feature vector, including:

[0053]

[0054]

[0055]

[0056] in, represents the associated feature vector of the key frame of water conservancy project gate monitoring, represents the multi-modal correlation feature vector of water conservancy project safety, It means subtracting by position. It means point multiplication by position. It means adding by position. represents a one-dimensional convolutional layer, represents maximum pooling, represents the offset-compensated convolutional coding pooling feature vector, represents a norm, represents the two-norm, Indicates cascade, represents the first weighted hyperparameter, represents the offset compensation factor, represents the length of the associated feature vector of the key frame of the water conservancy project gate monitoring, represents the second weighted hyperparameter, Represents the characteristic vector of water conservancy project gate safety early warning judgment.

[0057] In the technical solution of the present application, in the process of fusing the key frame associated feature vector of the water conservancy project gate monitoring and the multimodal associated feature vector of the water conservancy project safety, the feature fine-grained dimension offset between the key frame associated feature vector of the water conservancy project gate monitoring and the multimodal associated feature vector of the water conservancy project safety is taken into account, which may cause the local structure collapse of the fused water conservancy project gate safety early warning judgment feature vector. Based on this, in the technical solution of the present application, the feature fine-grained dimension offset compensation based on the local structure is performed on the key frame associated feature vector of the water conservancy project gate monitoring and the multimodal associated feature vector of the water conservancy project safety.

[0058] Specifically, based on the restricted expression of the foreground-background structure distinction of the high-dimensional manifold of the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector based on the semantic difference vector amplitude between the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector, the key feature fine-grained correspondence between the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector is modeled, and then the active fine-grained correspondence adjustment propagation is performed through the back-propagation displacement compensation information between the corresponding eigenvalues ​​of the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector, so as to avoid the imbalance at the feature vector level of the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector in a focused manner, thereby synchronizing the microscopic dimensions of the two feature vectors, ensuring that in the fusion process, the information between the feature vectors can be effectively aligned and combined, thereby avoiding the abnormal accumulation of local structures.

[0059] Furthermore, in water conservancy projects, feature vectors represent comprehensive information extracted from different sources (such as video surveillance, sensor data), which may contain detailed data about gate status, environmental conditions, and system behavior. The classifier learns the relationship between these features and warning conditions through training, and can automatically determine whether there are safety hazards in the gate based on the input feature vector in practical applications. This automated processing significantly improves the efficiency and response speed of the early warning system. Specifically, by inputting feature vectors into the classifier, the model can automatically identify potential patterns and anomalies in the data, thereby assigning a category to each feature vector, such as "safety" or "warning". This automatic classification capability greatly improves the accuracy and efficiency of data analysis.

[0060] In summary, the embodiment of the present application first obtains the monitoring video of the water conservancy project gate collected by the camera, the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow rate values ​​at multiple predetermined time points collected by the water flow sensor installed on the gate, and then uses deep learning technology to perform feature extraction and correlation analysis on the three. Finally, a classifier is used to determine whether to issue a safety warning for the water conservancy project gate, thereby reducing the shortcomings of human observation, improving the efficiency and response speed of the system, and timely preventing potential dangers, thereby providing more targeted decision-making support and improving the management and maintenance strategies of water conservancy projects.

[0061] As described above, the water conservancy project intelligent monitoring system 100 according to the embodiment of the present application can be implemented in various terminal devices. In one example, the water conservancy project intelligent monitoring system 100 can be integrated into the terminal device as a software module and / or a hardware module. For example, the water conservancy project intelligent monitoring system 100 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the water conservancy project intelligent monitoring system 100 can also be one of the many hardware modules of the terminal device.

[0062] Alternatively, in another example, the water conservancy project intelligent monitoring system 100 and the terminal device may also be separate devices, and the water conservancy project intelligent monitoring system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0063] Figure 5 FIG. 1 is a flow chart of a method for intelligent monitoring of a water conservancy project according to an embodiment of the present application. Figure 5As shown, according to the intelligent monitoring method of a water conservancy project in an embodiment of the present application, it includes: S110, obtaining a water conservancy project gate monitoring video collected by a camera, water level change values ​​on both sides of the gate at multiple predetermined time points collected by a water level sensor installed on the gate, and water flow values ​​at multiple predetermined time points collected by a water flow sensor installed on the gate; S120, extracting a water conservancy project gate monitoring key frame associated feature vector and a water conservancy project safety multimodal associated feature vector from the water conservancy project gate monitoring video collected by the camera, the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow values ​​at multiple predetermined time points collected by the water flow sensor installed on the gate; S130, judging whether to issue a water conservancy project gate safety warning based on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector.

[0064] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned water conservancy project intelligent monitoring method have been referred to above. Figures 1 to 4 The invention has been introduced in detail in the description of the intelligent monitoring system for water conservancy projects, and therefore, its repeated description will be omitted.

[0065] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0066] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0067] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0068] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0069] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0070] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit of the technical solution of the present invention.

Claims

1. An intelligent monitoring system for water conservancy projects, characterized in that: include: A water conservancy project monitoring data acquisition module is used to acquire water conservancy project gate monitoring videos collected by cameras, water level change values ​​on both sides of the gate at multiple predetermined time points collected by water level sensors installed on the gates, and water flow values ​​at multiple predetermined time points collected by water flow sensors installed on the gates; A water conservancy project monitoring data extraction module, used to extract the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector from the water conservancy project gate monitoring video collected by the camera, the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow value at multiple predetermined time points collected by the water flow sensor installed on the gate; A water conservancy project gate early warning module, used to determine whether to issue a water conservancy project gate safety early warning based on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector; Wherein, the water conservancy project gate early warning module includes: A water conservancy project gate safety feature fusion unit is used to perform feature fine-grained dimension offset compensation based on local structure on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multi-modal associated feature vector to obtain a water conservancy project gate safety early warning judgment feature vector; A water conservancy project gate safety warning judgment unit, used for passing the water conservancy project gate safety warning judgment feature vector through a classifier to obtain a classification result, and the classification result is used to determine whether to issue a water conservancy project gate safety warning; Wherein, the water conservancy project gate safety feature fusion unit includes: Calculating the position difference between the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector to obtain an offset compensated differential feature vector; Calculate the position point multiplication between the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector to obtain the offset compensation position point multiplication feature vector; Calculate the positional addition between the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector to obtain an offset compensation positional addition feature vector; The offset-compensated differential feature vector, the offset-compensated positional point-product feature vector, and the offset-compensated positional addition feature vector are concatenated and input into a one-dimensional convolutional layer and subjected to maximum pooling to obtain an offset-compensated convolutional coding pooling feature vector; Weighting the square root value of the two-norm of the offset-compensated convolutional coding pooling feature vector and adding the square root value to the one-norm of the offset-compensated convolutional coding pooling feature vector to obtain an offset compensation factor; The offset compensation point-by-position feature vector and the square root of the length of the associated feature vector of the water conservancy project gate monitoring key frame are added according to the position to obtain an offset compensation intermediate vector; The offset compensation factor and the second weighted hyperparameter are used as weighting coefficients to calculate the position-weighted sum of the offset compensation position-added feature vector and the offset compensation intermediate vector to obtain a water conservancy project gate safety early warning judgment feature vector.

2. The intelligent monitoring system for water conservancy projects according to claim 1 is characterized in that: The water conservancy project monitoring data extraction module includes: A gate monitoring feature extraction unit, used for extracting features from the water conservancy project gate monitoring video collected by the camera to obtain a plurality of water conservancy project gate monitoring key frame image block embedding vectors; A gate monitoring feature encoding unit, used for performing feature encoding on the plurality of water conservancy project gate monitoring key frame image block embedding vectors to obtain the water conservancy project gate monitoring key frame associated feature vectors; A water level change feature extraction unit, used for performing feature extraction on the water level change values ​​on both sides of the gate at a plurality of predetermined time points collected by the water level sensor installed on the gate to obtain a water level change feature vector on both sides of the gate; A water flow characteristic extraction unit, used for performing characteristic extraction on the water flow values ​​at a plurality of predetermined time points collected by the water flow sensor installed at the gate to obtain a water flow characteristic vector; The water conservancy project safety multi-modal association unit is used to associate the water level change feature vector on both sides of the gate with the water flow rate feature vector to obtain the water conservancy project safety multi-modal association feature vector.

3. The intelligent monitoring system for water conservancy projects according to claim 2 is characterized in that: The gate monitoring feature extraction unit comprises: Extracting a water conservancy project gate monitoring key frame sequence from the water conservancy project gate monitoring video collected by the camera; The water conservancy project gate monitoring key frame sequence is linearly embedded into a water conservancy project gate monitoring key frame image block sequence encoder to obtain the plurality of water conservancy project gate monitoring key frame image block embedding vectors.

4. The intelligent monitoring system for water conservancy projects according to claim 3 is characterized in that: The gate monitoring feature encoding unit comprises: Arranging the plurality of water conservancy project gate monitoring key frame image block embedding vectors in two dimensions to obtain a water conservancy project gate monitoring key frame image block input matrix; The water conservancy project gate monitoring key frame image block input matrix is ​​passed through a water conservancy project gate monitoring key frame image block feature encoder to obtain the water conservancy project gate monitoring key frame associated feature vector.

5. The intelligent monitoring system for water conservancy projects according to claim 4 is characterized in that: The water level change feature extraction unit comprises: Arranging the water level change values ​​on both sides of the gate at a plurality of predetermined time points collected by the water level sensor installed on the gate as input vectors of water level change on both sides of the gate; The input vector of the water level change on both sides of the gate is passed through the water level change timing encoder on both sides of the gate to obtain the water level change feature vector on both sides of the gate.

6. The intelligent monitoring system for water conservancy projects according to claim 5 is characterized in that: The water flow characteristic extraction unit comprises: A water flow rate arrangement subunit, used for arranging the water flow rate values ​​at a plurality of predetermined time points collected by the water flow sensor installed at the gate into a water flow rate input vector; The water flow convolutional encoding subunit is used to pass the water flow input vector through a water flow sequence encoder including a one-dimensional convolutional layer and a fully connected layer to obtain the water flow feature vector.

7. A method for intelligent monitoring of water conservancy projects, characterized in that: include: Obtaining a monitoring video of a water conservancy project gate collected by a camera, water level change values ​​on both sides of the gate at multiple predetermined time points collected by a water level sensor installed on the gate, and water flow values ​​at multiple predetermined time points collected by a water flow sensor installed on the gate; Extracting a water conservancy project gate monitoring key frame associated feature vector and a water conservancy project safety multimodal associated feature vector from the water conservancy project gate monitoring video collected by the camera, the water level change values ​​at both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow rate values ​​at multiple predetermined time points collected by the water flow sensor installed on the gate; Based on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector, determining whether to issue a water conservancy project gate safety warning; Wherein, judging whether to issue a safety warning for a water conservancy project gate based on the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector includes: The key frame associated feature vector of the water conservancy project gate monitoring and the multi-modal associated feature vector of the water conservancy project safety are compensated for by fine-grained dimension offset based on local structure to obtain the feature vector of the water conservancy project gate safety early warning judgment; Passing the water conservancy project gate safety warning judgment feature vector through a classifier to obtain a classification result, wherein the classification result is used to determine whether to issue a water conservancy project gate safety warning; Among them, the key frame associated feature vector of the water conservancy project gate monitoring and the multi-modal associated feature vector of the water conservancy project safety are compensated for the feature fine-grained dimension offset based on the local structure to obtain the water conservancy project gate safety early warning judgment feature vector, including: Calculating the position difference between the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector to obtain an offset compensated differential feature vector; Calculate the position point multiplication between the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector to obtain the offset compensation position point multiplication feature vector; Calculate the positional addition between the water conservancy project gate monitoring key frame associated feature vector and the water conservancy project safety multimodal associated feature vector to obtain an offset compensation positional addition feature vector; The offset-compensated differential feature vector, the offset-compensated positional point-product feature vector, and the offset-compensated positional addition feature vector are concatenated and input into a one-dimensional convolutional layer and subjected to maximum pooling to obtain an offset-compensated convolutional coding pooling feature vector; Weighting the square root value of the two-norm of the offset-compensated convolutional coding pooling feature vector and adding the square root value to the one-norm of the offset-compensated convolutional coding pooling feature vector to obtain an offset compensation factor; The offset compensation point-by-position feature vector and the square root of the length of the associated feature vector of the water conservancy project gate monitoring key frame are added according to the position to obtain an offset compensation intermediate vector; The offset compensation factor and the second weighted hyperparameter are used as weighting coefficients to calculate the position-weighted sum of the offset compensation position-added feature vector and the offset compensation intermediate vector to obtain a water conservancy project gate safety early warning judgment feature vector.

8. The intelligent monitoring method for water conservancy projects according to claim 7 is characterized in that: Extracting the key frame associated feature vector of the water conservancy project gate monitoring and the multi-modal associated feature vector of the water conservancy project safety from the water conservancy project gate monitoring video collected by the camera, the water level change values ​​on both sides of the gate at multiple predetermined time points collected by the water level sensor installed on the gate, and the water flow value at multiple predetermined time points collected by the water flow sensor installed on the gate, including: Performing feature extraction on the water conservancy project gate monitoring video collected by the camera to obtain a plurality of water conservancy project gate monitoring key frame image block embedding vectors; Performing feature encoding on the plurality of water conservancy project gate monitoring key frame image block embedding vectors to obtain the water conservancy project gate monitoring key frame associated feature vectors; Performing feature extraction on the water level change values ​​on both sides of the gate at a plurality of predetermined time points collected by the water level sensor installed on the gate to obtain a feature vector of the water level change on both sides of the gate; Performing feature extraction on the water flow values ​​at a plurality of predetermined time points collected by the water flow sensor installed at the gate to obtain a water flow feature vector; The water level change feature vectors on both sides of the gate are associated with the water flow rate feature vector to obtain the multi-modal association feature vector of the water conservancy project safety.

Images