Reservoir real-time monitoring method and system based on multi-mode collaborative awareness

Abstract

The invention discloses a reservoir real-time monitoring method and system based on multi-modal collaborative awareness, and the method comprises the following steps: S1, collecting multi-modal data of a reservoir environment, and carrying out the preprocessing of the multi-modal data; s2, constructing a FusionNet network, performing feature extraction and fusion on the preprocessed multi-modal data, and generating multi-modal feature representation; s3, performing anomaly detection on the multi-modal feature representation through an improved LOF algorithm; s4, executing hierarchical decision optimization according to an anomaly detection result, and generating an early warning signal; s5, adjusting real-time operation parameters of the reservoir monitoring system according to the early warning signal, and dynamically optimizing the feature extraction and fusion process in the FusionNet network; and S6, continuously monitoring the reservoir environment, and optimizing an abnormal event detection and early warning mechanism. According to the invention, through combination of multi-modal perception and an intelligent algorithm, the accuracy and response speed of reservoir monitoring are improved, and occurrence of potential risk events is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal perception, and in particular, to a real-time monitoring method and system for reservoirs based on multimodal collaborative perception. Background Art

[0002] With the increasing importance of reservoir management, the safety monitoring of reservoirs has become a key task for ensuring people's lives, property safety, and environmental protection. The main purpose of reservoir monitoring is to timely detect potential abnormalities in the reservoir structure, environment, and operating status, and issue early warnings in a timely manner, thereby effectively preventing the occurrence of reservoir safety accidents. However, the existing reservoir monitoring methods and systems have certain deficiencies in terms of monitoring accuracy, response speed, and system intelligence.

[0003] Currently, reservoir monitoring technologies mainly rely on traditional single-sensing means, such as video monitoring and sensor data collection. Video monitoring mainly collects real-time images of the external environment of the reservoir through monitoring cameras, but its main limitation is that it can only provide visual information and is difficult to comprehensively reflect the internal and environmental changes of the reservoir. And single-sensor data collection is limited to a specific dimension, such as temperature, humidity, vibration, etc., and cannot comprehensively reflect the operating status and potential risks of the reservoir. This single monitoring means often seems powerless in the face of complex environments and sudden abnormal events, and it is difficult to provide comprehensive and accurate decision-making support for reservoir management.

[0004] In addition, most traditional reservoir monitoring methods rely on manual judgment, and the processing of a large amount of real-time monitoring data relies on traditional data analysis methods, which are easily troubled by the problems of huge data volume and low analysis efficiency. Currently, the data processing in reservoir monitoring systems mostly lacks intelligence, especially in multimodal data fusion, automatic identification, and early warning of abnormal events. In particular, in a complex environment, how to combine multiple sensing data for accurate abnormal detection and risk assessment is still a major problem in reservoir monitoring technology.

[0005] Traditional anomaly detection methods mostly adopt algorithms based on a single data source, such as vision anomaly detection based on images or physical feature analysis based on sensors. The main problems of these methods are that they only rely on a single data source and are difficult to comprehensively and accurately capture the complex changes in the reservoir environment. For example, the vibration of the reservoir dam may not be effectively monitored only through visual information, and the sensor data such as temperature, humidity, and pressure cannot independently reflect the overall operating status of the reservoir. This results in the accuracy and timeliness of existing anomaly detection methods being difficult to meet the actual needs in the face of the changing and complex reservoir environment.

[0006] In addition, existing monitoring systems often lack real-time response capabilities and intelligent decision-making support functions. When an abnormal event occurs, traditional systems usually rely on manual or preset rules for early warning and cannot dynamically adjust monitoring strategies in real time based on actual monitoring data. This not only reduces the efficiency of reservoir management but also increases the problem of being unable to respond in a timely manner in complex situations. Especially when multiple abnormal events occur, it is difficult for traditional methods to quickly judge the priority and severity, often resulting in a lag in response and missing the best opportunity for emergency handling.

[0007] The existing technology also faces the contradiction between monitoring accuracy and data volume. With the gradual increase in monitoring devices, the data acquisition ability of the reservoir environment has been improving year by year. However, how to efficiently process and analyze massive multimodal data has become a new challenge. A single sensor or monitoring device cannot independently provide sufficient information, and the data formats, resolutions, and frequencies collected by different devices vary greatly, resulting in problems such as inconsistent information, excessive redundancy, and high computational complexity when fusing multiple data. Most of the existing data fusion methods rely on traditional fusion algorithms and lack in-depth exploration of the complex correlation relationships between data, leading to unsatisfactory fusion effects and further affecting the accuracy and real-time performance of the monitoring system.

[0008] In summary, the current real-time reservoir monitoring methods have significant defects in terms of monitoring accuracy, intelligent decision-making, anomaly detection, and data processing capabilities. The existing technology cannot comprehensively and effectively utilize multimodal data for precise detection and early warning of abnormal events, nor can it flexibly adjust monitoring strategies according to real-time monitoring data to meet different monitoring requirements. Therefore, how to provide a real-time reservoir monitoring method and system based on multimodal collaborative perception is an urgent problem for those skilled in the art. Summary of the Invention

[0009] An object of the present invention is to propose a real-time reservoir monitoring method and system based on multimodal collaborative perception. The present invention makes full use of multimodal data fusion technology, deep learning algorithms, and anomaly detection methods, and details how to perform real-time monitoring of various abnormal events in the reservoir through the collaborative perception of multiple data sources such as visual, acoustic, vibration, and environmental sensors, and optimize the monitoring strategy on this basis. This method can perform anomaly detection on multimodal data through an improved LOF algorithm and adjust the monitoring strategy in combination with the real-time state of the reservoir, thereby improving the intelligent level of reservoir safety management.

[0010] A real-time reservoir monitoring method based on multimodal collaborative perception according to an embodiment of the present invention includes the following steps:

[0011] S1. Collect multimodal data of the reservoir environment and perform preprocessing;

[0012] S2. Construct the FusionNet network to extract hierarchical features from the preprocessed multi-modal data, and generate multi-modal feature representations through weighted fusion;

[0013] S3. Perform anomaly detection on the multi-modal feature representations through an improved LOF algorithm. The improved LOF algorithm detects potential abnormal events in the reservoir by combining the Minkowski distance function and weighted comprehensive multi-modal features. The abnormal events include water level anomalies, dam vibrations, or potential signs of dam breakage;

[0014] S4. According to the anomaly detection results, perform hierarchical decision optimization and generate warning signals. The warning signals include water level anomaly warnings, dam vibration warnings, and potential dam breakage warnings;

[0015] S5. According to the warning signals, adjust the real-time operation parameters of the reservoir monitoring system, and dynamically optimize the feature extraction and fusion process in the FusionNet network to adapt to different monitoring requirements;

[0016] S6. Continuously monitor the reservoir environment, and update the weight parameters in the FusionNet network in real time to optimize the abnormal event detection and warning mechanism.

[0017] Optionally, the multi-modal data includes image data, acoustic signals, vibration signals, temperature and humidity data. Among them, visual sensors collect image data of the reservoir dam, the surrounding environment, and the water flow state, acoustic sensors collect acoustic signals inside and outside the reservoir, vibration sensors collect vibration signals of the dam and the reservoir structure, and temperature and humidity sensors collect temperature and humidity data in the reservoir area.

[0018] Optionally, the preprocessing includes: denoising, grayscale conversion, and edge detection processing for image data, spectrum analysis, noise filtering, and echo cancellation processing for acoustic signals, waveform analysis, vibration mode recognition, and smoothing processing for vibration signals, and standardization, missing value filling, and unified time series for temperature and humidity data.

[0019] Optionally, the FusionNet network consists of three modules, namely a multi-channel convolution module, a spatio-temporal fusion module, and an adaptive weighting module. The multi-channel convolution module extracts the spatial features of image data, the time-domain features of acoustic signals, the frequency-domain features of vibration signals, and the environmental features of temperature and humidity data respectively. The spatio-temporal fusion module fuses the spatial features, time-domain features, frequency-domain features, and environmental features, and captures the spatio-temporal relationships in the data through the combination of convolutional layers and recurrent layers. The adaptive weighting module adaptively adjusts the weights of different modal features based on the current real-time monitoring state of the reservoir, optimizes the fusion effect, and generates multi-modal feature representations.

[0020] Optionally, the hierarchical decision-making optimization is based on a multi-level decision tree structure, and adjusts the fusion weights of multi-modal data in real time according to the risk level of abnormal events, dynamically optimizing the reservoir monitoring strategy, which includes enhancing the sampling frequency of the monitoring area, adjusting the sensor position, and activating standby sensors.

[0021] Optionally, S2 specifically includes:

[0022] S21. Construct a FusionNet network to perform hierarchical feature extraction on the preprocessed multi-modal data. The FusionNet network consists of a multi-channel convolution module, a spatio-temporal fusion module, and an adaptive weighting module;

[0023] S22. The multi-channel convolution module processes image data, acoustic signals, vibration signals, and temperature and humidity data respectively, where:

[0024] The image data performs spatial feature extraction through multiple convolutional layers, and each convolutional layer uses different convolutional kernels for feature learning;

[0025] The acoustic signal performs time-domain feature extraction through a time-domain convolutional layer, and uses a one-dimensional convolutional kernel to capture the instantaneous changes of the signal;

[0026] The vibration signal performs frequency-domain feature extraction through a frequency-domain convolutional layer. After performing a fast Fourier transform, the data is input into a multi-layer convolutional network for frequency-domain feature learning;

[0027] After the temperature and humidity data is normalized, a convolutional layer is used to extract environmental features, and multi-layer feature mining is performed on the temperature and humidity features; [[ID=X]]

[0028] S24. The spatio-temporal fusion module fuses the spatial features, time-domain features, frequency-domain features, and environmental features obtained from the multi-channel convolution module, and captures the spatio-temporal relationships in the data through a combination of convolutional layers and recurrent layers. The convolutional layer is used to extract local spatial features, and the recurrent layer is used to capture time-series features, and the two are combined to optimize the feature representation;

[0029] S25. The adaptive weighting module automatically adjusts the weighting factors of each modal feature by real-time monitoring of various environmental changes in the reservoir, so that the dependence on different modal data is adapted under different monitoring requirements, enhancing the response ability to reservoir anomalies;

[0030] S26. Generate the final multi-modal feature representation through weighted fusion, which combines multi-level information of image, acoustic, vibration, and environmental data.

[0031] Optionally, S3 specifically includes:

[0032] S31. Input the preprocessed multi-modal feature representation into the improved LOF algorithm. The multi-modal feature representation consists of F mi where F mi is the multi-modal feature data point at the i-th moment, expressed as the fusion result of spatial features, time-domain features, frequency-domain features, and environmental features;

[0033] S32. Calculate the local density of the data point F mi based on the improved LOF algorithm. The local density is calculated by weighting the Minkowski distance, and the Minkowski distance uses the p-order metric:

[0034]

[0035] where d p (F mi , F mj ) is the Minkowski distance, F mik and F mjk are the values of the i-th and j-th data points in the k-th feature dimension respectively, p is the order of the Minkowski distance, d is the number of dimensions of the multi-modal features, ρ(F mi ) is the local density of the data point F mi at the i-th moment, w ij is the weighting factor between the i-th and j-th data points, F mj is the multi-modal feature data point at the j-th moment, K(F mi , F mj ) is the similarity function based on the Minkowski distance d p (F mi , F mj ), and n is the number of neighbors;

[0036] S33. The improved LOF algorithm calculates the LOF value by combining the Minkowski distance function and weighted comprehensive multi-modal features. The LOF value is an index to measure the degree of abnormality of data points:

[0037]

[0038] where LOF(F mi ) is the local outlier factor value of the data point F mi at the i-th moment, ρ(F mj ) is the local density of the data point F mj at the j-th moment, w ijk is the weighting coefficient between the i-th and j-th data points in the k-th feature dimension;

[0039] S34. Perform threshold processing on the LOF value. If LOF(F mi ) > θ(F mi), then determine the data point F mi is an abnormal data point, where θ(F mi ) is a preset abnormal detection threshold, which is dynamically adjusted according to the real-time reservoir monitoring status and the distribution of historical data. The abnormal data points include water level anomalies, dam vibrations, or potential signs of dam breakage.

[0040] Optionally, the similarity metric function calculates the similarity between data points based on the Minkowski distance. The Minkowski distance measures the similarity between two data points in a multi-dimensional feature space by weighted summing the differences between each feature dimension and performing a p-th root operation.

[0041] Optionally, the specific steps of S4 are as follows:

[0042] S41. According to the abnormal detection results, construct a multi-level decision tree structure, assign a risk level to each abnormal event, and perform hierarchical decision-making on the abnormal events according to the risk level;

[0043] S42. Generate warning signals and adjust the reservoir monitoring strategy in real time. The warning signals include water level anomaly warnings, dam vibration warnings, and potential dam breakage warnings. The monitoring strategy includes enhancing the sampling frequency of the monitoring area, adjusting the sensor positions, and activating backup sensors;

[0044] For high-risk abnormal events, give priority to enhancing the sampling frequency and focus on monitoring the reservoir area;

[0045] For relatively low-risk abnormal events, adjust the distribution of sensors and activate the sensors in the standby state to improve the monitoring coverage.

[0046] A real-time reservoir monitoring system based on multi-modal collaborative perception according to an embodiment of the present invention includes:

[0047] A data acquisition and processing module for collecting multi-modal data of the reservoir environment and performing preprocessing;

[0048] A feature extraction and fusion module for performing hierarchical feature extraction on the preprocessed multi-modal data and generating a multi-modal feature representation through weighted fusion;

[0049] An abnormal detection module for performing abnormal detection on the multi-modal feature representation through an improved LOF algorithm. The improved LOF algorithm detects potential abnormal events in the reservoir by combining the Minkowski distance function and weighted comprehensive multi-modal features. The abnormal events include water level anomalies, dam vibrations, or potential signs of dam breakage;

[0050] A decision optimization module, configured to perform hierarchical decision optimization according to the anomaly detection results and generate warning signals, where the warning signals include water level anomaly warning, dam vibration warning, and potential dam break warning;

[0051] An adjustment and adaptation module, configured to adjust the real-time operation parameters of the reservoir monitoring system according to the warning signals and dynamically optimize the feature extraction and fusion process in the FusionNet network to adapt to different monitoring requirements;

[0052] A monitoring and update module, configured to continuously monitor the reservoir environment and update the weight parameters in the FusionNet network in real time to optimize the anomaly event detection and warning mechanism.

[0053] The beneficial effects of the present invention are as follows:

[0054] First, by combining the collaborative perception of multi-modal data, the present invention makes full use of different types of sensor data such as vision, acoustics, vibration, temperature, and humidity, enabling the reservoir monitoring system to comprehensively understand the real-time state of the reservoir from multiple dimensions. Traditional single data sources cannot provide sufficient information, while the present invention through multi-modal data fusion makes the monitoring more accurate and comprehensive, thus improving the detection ability and accuracy of reservoir anomaly events.

[0055] Second, the present invention uses an improved LOF algorithm for anomaly detection, and combines the Minkowski distance function and weighted comprehensive multi-modal features. When fusing data, it not only improves the recognition accuracy of anomaly events, but also can flexibly adjust the weights of different modal data, making the monitoring results more in line with the actual situation. This innovation has strong adaptability in complex environments, can accurately identify potential risks of the reservoir, send out warning signals in advance, and effectively prevent possible catastrophic events.

[0056] Finally, through a hierarchical decision optimization mechanism, the present invention adjusts the reservoir monitoring strategy in real time. According to the anomaly detection results, it optimizes the sampling frequency of the monitoring area, the positions of sensors, and the activation status of backup sensors, enabling the monitoring system to dynamically adjust monitoring parameters according to different anomaly events. In this way, not only the real-time response ability of the monitoring system is improved, but also the intelligent level of the system is enhanced, providing a reliable guarantee for the long-term safe operation of the reservoir. Description of the Drawings

[0057] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0058] Figure 1 is a flowchart of a real-time reservoir monitoring method based on multi-modal collaborative perception proposed by the present invention;

[0059] Figure 2 Schematic diagram of multi-modal data feature extraction and fusion for a real-time reservoir monitoring method based on multi-modal collaborative perception proposed by the present invention;

[0060] Figure 3 Schematic diagram of reservoir abnormal event detection for a real-time reservoir monitoring method based on multi-modal collaborative perception proposed by the present invention;

[0061] Figure 4 Module structure diagram of a real-time reservoir monitoring system based on multi-modal collaborative perception proposed by the present invention. Detailed implementation manners

[0062] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic way, so they only show the components related to the present invention.

[0063] Refer to Figures 1-3 , a real-time reservoir monitoring method based on multi-modal collaborative perception, comprising the following steps:

[0064] S1. Collect multi-modal data of the reservoir environment and perform preprocessing;

[0065] S2. Construct a FusionNet network, perform hierarchical feature extraction on the preprocessed multi-modal data, and generate a multi-modal feature representation through weighted fusion;

[0066] S3. Perform anomaly detection on the multi-modal feature representation through an improved LOF algorithm. The improved LOF algorithm detects potential abnormal events in the reservoir by combining the Minkowski distance function and weighted comprehensive multi-modal features. The abnormal events include water level anomalies, dam vibrations, or potential signs of dam breakage;

[0067] S4. According to the anomaly detection results, perform hierarchical decision optimization and generate warning signals. The warning signals include water level anomaly warnings, dam vibration warnings, and potential dam breakage warnings;

[0068] S5. According to the warning signals, adjust the real-time operation parameters of the reservoir monitoring system and dynamically optimize the feature extraction and fusion process in the FusionNet network to adapt to different monitoring requirements;

[0069] S6. Continuously monitor the reservoir environment and update the weight parameters in the FusionNet network in real time to optimize the abnormal event detection and warning mechanism.

[0070] The present invention provides a real-time monitoring method for reservoirs based on multi-modal collaborative perception. By collecting multi-modal data of the reservoir environment and performing preprocessing, and combining multiple data sources for comprehensive analysis, it can comprehensively monitor the operation status of the reservoir and improve the accuracy and intelligent level of the reservoir monitoring system.

[0071] In this embodiment, the multi-modal data includes image data, acoustic signals, vibration signals, temperature and humidity data. Among them, visual sensors collect image data of the reservoir dam, the surrounding environment, and the water flow state, acoustic sensors collect acoustic signals inside and outside the reservoir, vibration sensors collect vibration signals of the dam and the reservoir structure, and temperature and humidity sensors collect temperature and humidity data of the reservoir area.

[0072] The present invention introduces various types of multi-modal data such as image data, acoustic signals, vibration signals, and temperature and humidity data, enabling the monitoring system to obtain different-dimensional information of the reservoir in all directions. The visual sensors, acoustic sensors, vibration sensors, and temperature and humidity sensors work together to make up for the limitations of traditional single sensors and can monitor the state of the reservoir more comprehensively and accurately.

[0073] In this embodiment, the preprocessing includes: denoising, grayscaling, and edge detection processing for image data, spectral analysis, noise filtering, and echo cancellation processing for acoustic signals, waveform analysis, vibration mode recognition, and smoothing processing for vibration signals, and standardization, missing value filling, and unified time series for temperature and humidity data.

[0074] The preprocessing method of the present invention includes multiple processing steps such as denoising, grayscaling, and edge detection of image data, spectral analysis and noise filtering of acoustic signals, waveform analysis and vibration mode recognition of vibration signals, and standardization of temperature and humidity data, greatly improving the quality of multi-modal data and providing high-quality data input for subsequent feature extraction and anomaly detection.

[0075] In this embodiment, the FusionNet network is composed of three modules, namely a multi-channel convolution module, a spatio-temporal fusion module, and an adaptive weighting module. The multi-channel convolution module extracts the spatial features of image data, the time-domain features of acoustic signals, the frequency-domain features of vibration signals, and the environmental features of temperature and humidity data respectively. The spatio-temporal fusion module fuses the spatial features, time-domain features, frequency-domain features, and environmental features, and captures the spatio-temporal relationship in the data through the combination of convolutional layers and recurrent layers. The adaptive weighting module adaptively adjusts the weights of different modal features based on the current real-time monitoring state of the reservoir, optimizes the fusion effect, and generates a multi-modal feature representation.

[0076] The present invention adopts the FusionNet network, which consists of a multi-channel convolution module, a spatio-temporal fusion module, and an adaptive weighting module. Each module optimizes different types of perceptual data, improves the accuracy of multi-modal feature representation through weighted fusion, and can automatically adjust the weights of different data modalities according to the real-time monitoring status of the reservoir, so as to better adapt to the changing reservoir monitoring requirements.

[0077] In this embodiment, the hierarchical decision optimization is based on a multi-level decision tree structure, and according to the risk level of abnormal events, the fusion weights of multi-modal data are adjusted in real time, and the reservoir monitoring strategy is dynamically optimized. The monitoring strategy includes enhancing the sampling frequency of the monitoring area, adjusting the sensor position, and activating standby sensors.

[0078] The present invention adopts hierarchical decision optimization, based on a multi-level decision tree structure, and makes dynamic decisions according to the risk level of abnormal events. By adjusting the fusion weights of multi-modal data in real time, the monitoring strategy of the reservoir is optimized, ensuring that the most appropriate response measures are taken under different abnormal conditions, and improving the flexibility and adaptability of the monitoring strategy.

[0079] In this embodiment, S2 specifically includes:

[0080] S21. Construct a FusionNet network to perform hierarchical feature extraction on the preprocessed multi-modal data. The FusionNet network consists of a multi-channel convolution module, a spatio-temporal fusion module, and an adaptive weighting module;

[0081] S22. The multi-channel convolution module processes image data, acoustic signals, vibration signals, and temperature and humidity data respectively, where:

[0082] The image data performs spatial feature extraction through multiple convolutional layers, and each convolutional layer uses different convolutional kernels for feature learning;

[0083] The acoustic signal performs time-domain feature extraction through a time-domain convolutional layer, and a one-dimensional convolutional kernel is used to capture the instantaneous changes of the signal;

[0084] The vibration signal performs frequency-domain feature extraction through a frequency-domain convolutional layer. After performing a fast Fourier transform, the data is input into a multi-layer convolutional network for frequency-domain feature learning;

[0085] After the temperature and humidity data is standardized, a convolutional layer is used to extract environmental features, and multi-layer feature mining is performed on the temperature and humidity features;

[0086] S23. The spatio-temporal fusion module fuses the spatial features, temporal features, frequency domain features, and environmental features obtained from the multi-channel convolution module, and captures the spatio-temporal relationships in the data through a combination of convolutional layers and recurrent layers. The convolutional layer is used to extract local spatial features, and the recurrent layer is used to capture time series features. The combination of the two optimizes the feature representation;

[0087] S24. The adaptive weighting module automatically adjusts the weighting factors of each modality feature by monitoring various environmental changes in the reservoir in real time, so that the degree of dependence on different modality data is adapted under different monitoring requirements, enhancing the response ability to abnormal situations in the reservoir;

[0088] S25. Through the method of weighted fusion, a final multi-modal feature representation is generated, which combines multi-level information of images, acoustic waves, vibrations, and environmental data.

[0089] The present invention describes in detail the feature extraction process of the FusionNet network. By using the multi-channel convolution module to process image data, acoustic signals, vibration signals, and temperature and humidity data respectively, the spatial features, temporal features, frequency domain features, and environmental features are extracted. The spatio-temporal fusion module further fuses these features to capture the spatio-temporal relationships in the data, enhancing the response ability to environmental changes in the reservoir.

[0090] In this embodiment, the specific steps of S3 are as follows:

[0091] S31. Input the preprocessed multi-modal feature representation into the improved LOF algorithm. The multi-modal feature representation is composed of F mi , where F mi is the multi-modal feature data point at the i-th moment, expressed as the fusion result of spatial features, temporal features, frequency domain features, and environmental features;

[0092] S32. Calculate the local density of the data point F mi based on the improved LOF algorithm. The local density is calculated by weighting the Minkowski distance, and the Minkowski distance adopts the p-order metric:

[0093]

[0094] Among them, d p (F mi , F mj ) is the Minkowski distance, F mik and F mjk are the values of the i-th and j-th data points in the k-th feature dimension respectively, p is the order of the Minkowski distance, d is the dimension number of the multi-modal feature, and ρ(F mi ) is the local density of the data point F mi at the i-th moment, and wij is the weighting factor between the i-th and j-th data points, F mj is the multi-modal feature data point at the j-th moment, K(F mi , F mj ) is based on the Minkowski distance d p (F mi , F mj ) is the similarity function, and n is the number of neighbors;

[0095] S33. The improved LOF algorithm calculates the LOF value by combining the Minkowski distance function and the weighted comprehensive multi-modal features. The LOF value is an index to measure the degree of abnormality of data points:

[0096]

[0097] where LOF(F mi ) is the local outlier factor value of the data point F at the i-th moment mi , ρ(F mj ) is the local density of the data point F at the j-th moment mj , and w ijk is the weighting coefficient of the i-th and j-th data points in the k-th feature dimension;

[0098] S34. Perform threshold processing on the LOF value. If LOF(F mi ) > θ(F mi ), then determine that the data point F mi is an abnormal data point, where θ(F mi ) is the preset anomaly detection threshold, which is dynamically adjusted according to the real-time reservoir monitoring status and the distribution of historical data. The abnormal data points include water level anomalies, dam vibrations, or potential signs of dam breakage.

[0099] The present invention uses an improved LOF algorithm to perform anomaly detection on multi-modal feature representations. By combining the Minkowski distance function and weighted comprehensive multi-modal features, the recognition accuracy of reservoir anomaly events is improved. By calculating the local density and LOF value of each data point, potential anomaly events such as water level anomalies and dam vibrations can be accurately detected, improving the accuracy and real-time performance of anomaly detection.

[0100] In this embodiment, the similarity metric function calculates the similarity between data points based on the Minkowski distance. The Minkowski distance measures the similarity between two data points in a multi-dimensional feature space by weighted summing the differences between each feature dimension and performing a p-th root operation.

[0101] Through the set similarity measurement function, the present invention combines the Minkowski distance to calculate the similarity between data points, optimizes the interaction between multi-modal features, and improves the effect of anomaly detection. The improved LOF algorithm can more sensitively identify potential risks in the reservoir and capture the occurrence of abnormal events in real time.

[0102] In this embodiment, step S4 specifically includes:

[0103] S41. According to the anomaly detection result, construct a multi-level decision tree structure, assign a risk level to each abnormal event, and perform hierarchical decision-making on the abnormal events according to the risk level;

[0104] S42. Generate warning signals and adjust the monitoring strategy of the reservoir in real time. The warning signals include water level anomaly warning, dam vibration warning and potential dam-break warning. The monitoring strategy includes enhancing the sampling frequency of the monitoring area, adjusting the sensor position, and activating standby sensors;

[0105] For high-risk abnormal events, preferentially enhance the sampling frequency and focus on monitoring the reservoir area;

[0106] For relatively low-risk abnormal events, adjust the distribution of sensors and activate the standby sensors to improve the monitoring coverage.

[0107] Through the hierarchical decision-making mechanism based on risk levels, the present invention generates warning signals according to the anomaly detection results and adjusts the monitoring strategy in real time. For high-risk events, preferentially enhance the sampling frequency and focus on monitoring the reservoir area; for low-risk events, adjust the sensor position and activate the standby sensors, so as to achieve an efficient response to reservoir abnormal events.

[0108] Reference Figure 4 , a real-time monitoring system for reservoirs based on multi-modal collaborative perception, includes:

[0109] A data acquisition and processing module, used to acquire multi-modal data of the reservoir environment and perform preprocessing;

[0110] A feature extraction and fusion module, used to perform hierarchical feature extraction on the preprocessed multi-modal data and generate a multi-modal feature representation through weighted fusion;

[0111] An anomaly detection module, used to perform anomaly detection on the multi-modal feature representation through an improved LOF algorithm. The improved LOF algorithm detects potential abnormal events in the reservoir by combining the Minkowski distance function and weighted comprehensive multi-modal features. The abnormal events include water level anomalies, dam vibrations or potential dam-break signs;

[0112] A decision optimization module, which is used to perform hierarchical decision optimization according to the anomaly detection results and generate warning signals. The warning signals include water level anomaly warning, dam vibration warning and potential dam break warning;

[0113] An adjustment and adaptation module, which is used to adjust the real-time operation parameters of the reservoir monitoring system according to the warning signals, and dynamically optimize the feature extraction and fusion process in the FusionNet network to adapt to different monitoring requirements;

[0114] A monitoring and updating module, which is used to continuously monitor the reservoir environment and real-time update the weight parameters in the FusionNet network to optimize the anomaly event detection and warning mechanism.

[0115] The present invention provides a complete real-time reservoir monitoring system based on multi-modal collaborative perception, covering multiple functional modules such as data acquisition, feature extraction and fusion, anomaly detection, and decision optimization. The system can monitor the reservoir environment in real time, dynamically optimize the monitoring strategy, improve the intelligent level of reservoir monitoring, and effectively prevent potential risks of the reservoir.

[0116] Embodiment 1:

[0117] To verify the feasibility of the present invention in implementation, the present invention is applied to the real-time monitoring system of a certain reservoir. The reservoir is under variable climatic conditions and has a complex surrounding environment, posing a relatively high safety risk. Traditional reservoir monitoring systems rely only on single sensors or visual monitoring, making it difficult to comprehensively and effectively monitor the changes in the reservoir environment. Especially when multiple abnormal events occur, it is often difficult to detect and respond in a timely manner. Therefore, we apply the real-time reservoir monitoring method based on multi-modal collaborative perception of the present invention to this reservoir, using multi-modal data fusion and deep learning algorithms to improve the monitoring accuracy and intelligent level of the reservoir.

[0118] In the monitoring of this reservoir, we use visual sensors, acoustic sensors, vibration sensors and temperature and humidity sensors to collect image data, acoustic signals, dam vibration signals and temperature and humidity data of the reservoir area of the reservoir dam and the surrounding environment. These data are subjected to hierarchical feature extraction and fusion through the FusionNet network of the present invention to ensure that comprehensive information of the reservoir environment can be accurately obtained. The improved LOF algorithm is used to detect anomalies in multi-modal features, identify potential dangers such as water level anomalies, dam vibrations or potential dam break signs, and generate warning signals in the first time to guide the monitoring system to adjust real-time operation parameters.

[0119] During the monitoring process, the system monitors the data in real time and generates the following key performance indicators:

[0120] 1. During a one - month monitoring period, the temperature and humidity data around the reservoir fluctuate greatly. However, through the monitoring of temperature and humidity sensors, abnormal changes can be detected in advance, providing an effective basis for subsequent risk assessment.

[0121] 2. Through acoustic sensors, we detected abnormal acoustic signals near the reservoir dam, reflecting the vibration situation of the reservoir dam. Combining this signal with temperature and humidity changes, it is judged that there are some potential vibration problems with the reservoir dam.

[0122] 3. In image data processing, after analysis by a deep - learning network, the system successfully identified changes in dam cracks and generated corresponding warning signals.

[0123] 4. Continuously monitoring the reservoir state, the system automatically adjusts the sampling frequency to enhance the sampling density of high - risk areas, ensuring the comprehensiveness and real - time nature of monitoring.

[0124] After three months of monitoring and data analysis, the system accurately identified 3 abnormal water levels, 2 dam vibrations, and 1 potential dam - break sign. And within 10 minutes after each anomaly occurred, warning signals were successfully generated and corresponding monitoring strategy optimizations were executed, greatly improving the emergency response speed and processing ability of the reservoir. The specific data is shown in the following table:

[0125] Table 1 Abnormal events in reservoir monitoring and generation of warning signals

[0126]

[0127]

[0128] From the data and monitoring effects of this embodiment, it can be seen that the real - time reservoir monitoring method based on multi - modal collaborative perception of the present invention significantly improves the monitoring accuracy, especially in the detection of abnormal water levels, dam vibrations, and potential dam - break signs. The system can generate warning signals within a short time after an abnormal event occurs and adjust the monitoring strategy in real - time, effectively enhancing the emergency response speed and warning ability of the reservoir. At the same time, the dynamic adjustment function of the system ensures the flexibility and adaptability of monitoring, and can optimize the sampling frequency and sensor position in real - time according to different monitoring requirements, thus improving the comprehensive performance of the monitoring system.

[0129] Through the verification of this embodiment, the real - time reservoir monitoring method based on multi - modal collaborative perception proposed by the present invention has strong application prospects and practical value in complex reservoir environments and can provide strong support for the safety management of reservoirs.

[0130] As described above, it is only the preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent replacements or changes should be covered within the protection scope of the present invention.

Claims

1. A real-time monitoring method for reservoirs based on multi-modal collaborative perception, characterized in that, It includes the following steps: S1. Collect multi-modal data of the reservoir environment and perform preprocessing; S2. Construct a FusionNet network, extract hierarchical features from the preprocessed multi-modal data, and generate a multi-modal feature representation through weighted fusion; S3. Perform anomaly detection on the multi-modal feature representation through an improved LOF algorithm. The improved LOF algorithm detects potential anomaly events in the reservoir by combining the Minkowski distance function and weighted comprehensive multi-modal features. The anomaly events include water level anomalies, dam vibrations, or potential signs of dam failure; S4. According to the anomaly detection results, perform hierarchical decision optimization and generate warning signals. The warning signals include water level anomaly warnings, dam vibration warnings, and potential dam failure warnings; S5. According to the warning signals, adjust the real-time operation parameters of the reservoir monitoring system and dynamically optimize the feature extraction and fusion process in the FusionNet network to adapt to different monitoring requirements; S6. Continuously monitor the reservoir environment and update the weight parameters in the FusionNet network in real time to optimize the anomaly event detection and warning mechanism.

2. The real-time monitoring method for a reservoir based on multi-modal collaborative perception according to claim 1, wherein, The multi-modal data includes image data, acoustic signals, vibration signals, temperature, and humidity data. Among them, visual sensors collect image data of the reservoir dam, surrounding environment, and water flow state, acoustic sensors collect acoustic signals inside and outside the reservoir, vibration sensors collect vibration signals of the dam and reservoir structure, and temperature and humidity sensors collect temperature and humidity data of the reservoir area.

3. A real-time monitoring method for a reservoir based on multi-modal collaborative perception according to claim 1, characterized in that, The preprocessing includes: denoising, grayscale conversion, and edge detection processing for image data, spectrum analysis, noise filtering, and echo cancellation processing for acoustic signals, waveform analysis, vibration mode recognition, and smoothing processing for vibration signals, and standardization, missing value filling, and unified time series for temperature and humidity data.

4. A real-time monitoring method for a reservoir based on multi-modal collaborative perception according to claim 1, characterized in that The FusionNet network consists of three modules, namely a multi-channel convolution module, a spatio-temporal fusion module, and an adaptive weighting module. The multi-channel convolution module extracts the spatial features of image data, the time-domain features of acoustic signals, the frequency-domain features of vibration signals, and the environmental features of temperature and humidity data respectively. The spatio-temporal fusion module fuses the spatial features, time-domain features, frequency-domain features, and environmental features, and captures the spatio-temporal relationships in the data through the combination of convolutional layers and recurrent layers. The adaptive weighting module adaptively adjusts the weights of different modal features based on the current real-time monitoring state of the reservoir, optimizes the fusion effect, and generates a multi-modal feature representation.

5. A real-time monitoring method for a reservoir based on multi-modal collaborative perception according to claim 1, characterized in that, The hierarchical decision optimization is based on a multi-level decision tree structure, and adjusts the fusion weights of multi-modal data in real time according to the risk level of anomaly events, and dynamically optimizes the reservoir monitoring strategy. The monitoring strategy includes increasing the sampling frequency of the monitoring area, adjusting the sensor position, and activating standby sensors.

6. The real-time monitoring method for a reservoir based on multi-modal collaborative perception according to claim 1, wherein The specific content of S2 includes: S21. Construct a FusionNet network and extract hierarchical features from the preprocessed multi-modal data. The FusionNet network consists of a multi-channel convolution module, a spatio-temporal fusion module, and an adaptive weighting module; S22. The multi-channel convolution module processes image data, acoustic signals, vibration signals, and temperature and humidity data respectively, where: The image data undergoes spatial feature extraction through multiple convolutional layers, and each convolutional layer uses different convolutional kernels for feature learning; The acoustic signals undergo time-domain feature extraction through a time-domain convolutional layer, and a one-dimensional convolutional kernel is used to capture the instantaneous changes of the signals; The vibration signals undergo frequency-domain feature extraction through a frequency-domain convolutional layer. After performing a fast Fourier transform, the data is input into a multi-layer convolutional network for frequency-domain feature learning; After the temperature and humidity data is standardized, a convolutional layer is used to extract environmental features, and multi-layer feature mining is performed on the temperature and humidity features; S23. The spatio-temporal fusion module fuses the spatial features, time-domain features, frequency-domain features, and environmental features obtained from the multi-channel convolution module. By combining a convolutional layer and a recurrent layer, the spatio-temporal relationships in the data are captured. The convolutional layer is used to extract local spatial features, and the recurrent layer is used to capture time-series features. The combination of the two optimizes the feature representation; S24. The adaptive weighting module automatically adjusts the weighting factors of each modal feature by real-time monitoring various environmental changes in the reservoir, so that under different monitoring requirements, the dependence on different modal data is adapted, enhancing the response ability to abnormal situations in the reservoir; S25. Through a weighted fusion method, a final multi-modal feature representation is generated, and the multi-modal feature representation combines multi-level information of image, acoustic, vibration, and environmental data.

7. A real-time monitoring method for a reservoir based on multi-modal collaborative perception according to claim 1, characterized in that The specific steps of S3 are as follows: S31. Input the preprocessed multi-modal feature representation into the improved LOF algorithm. The multi-modal feature representation consists of F mi where F mi is the multi-modal feature data point at the i-th moment, expressed as the fusion result of spatial features, time-domain features, frequency-domain features, and environmental features; S32. Calculate the local density of data point F based on the improved LOF algorithm, where the local density is calculated by weighting the Minkowski distance, and the Minkowski distance uses the p-order metric: mi ​ Among them, d p (F mi , F mj ) is the Minkowski distance, F mik and F mjk are the values of the i-th and j-th data points in the k-th feature dimension respectively, p is the order of the Minkowski distance, d is the dimensionality of the multi-modal features, ρ(F mi ) is the local density of the data point F mi at the i-th moment, w ij is the weighting factor between the i-th and j-th data points, F mj is the multi-modal feature data point at the j-th moment, K(F mi , F mj ) is the similarity function based on the Minkowski distance d p (F mi , F mj ), and n is the number of neighbors; S33. The improved LOF algorithm calculates the LOF value by combining the Minkowski distance function and weighted comprehensive multi-modal features. The LOF value is an index to measure the degree of abnormality of data points: Among them, LOF(F mi ) is the local outlier factor value of the data point F at the i-th moment mi , and ρ(F mj ) is the local density of the data point F at the j-th moment mj . w ijk is the weighting coefficient of the i-th and j-th data points in the k-th feature dimension; S34. Perform threshold processing on the LOF value. If LOF(F mi ) > θ(F mi ), then determine that the data point F mi is an abnormal data point, where θ(F mi ) is a preset abnormal detection threshold, which is dynamically adjusted according to the real-time reservoir monitoring status and the distribution of historical data. The abnormal data points include water level anomalies, dam vibrations, or potential signs of dam breakage.

8. A real-time monitoring method for a reservoir based on multi-modal collaborative perception according to claim 7, characterized in that The similarity metric function calculates the similarity between data points based on the Minkowski distance. The Minkowski distance measures the similarity between two data points in a multi-dimensional feature space by performing a weighted sum of the differences between each feature dimension and then taking the p-th root operation.

9. A real-time monitoring method for a reservoir based on multi-modal collaborative perception according to claim 1, characterized in that, The specific steps of S4 are as follows: S41. According to the anomaly detection results, a multi-level decision tree structure is constructed. A risk level is assigned to each abnormal event, and hierarchical decision-making is performed on the abnormal events according to the risk level; S42. An early warning signal is generated, and the monitoring strategy of the reservoir is adjusted in real time. The early warning signal includes water level anomaly warning, dam vibration warning, and potential dam break warning. The monitoring strategy includes enhancing the sampling frequency of the monitoring area, adjusting the sensor position, and activating standby sensors; For high-risk abnormal events, the sampling frequency is preferentially enhanced, and key monitoring is carried out on the reservoir area; For low-risk abnormal events, the distribution of sensors is adjusted, and the sensors in the standby state are activated to improve the monitoring coverage.

10. A real-time monitoring system for a reservoir based on multi-modal collaborative perception, which executes a real-time monitoring method for a reservoir based on multi-modal collaborative perception according to any one of claims 1 to 9, characterized in that, It includes: A data acquisition and processing module, which is used to acquire multi-modal data of the reservoir environment and perform preprocessing; A feature extraction and fusion module, which is used to perform hierarchical feature extraction on the preprocessed multi-modal data and generate a multi-modal feature representation through weighted fusion; Anomaly detection module, which is used to perform anomaly detection on multi-modal feature representations through an improved LOF algorithm. The improved LOF algorithm combines the Minkowski distance function and weighted comprehensive multi-modal features to detect potential anomaly events in the reservoir. The anomaly events include water level anomalies, dam vibrations or potential signs of dam breakage; Decision optimization module, which is used to perform hierarchical decision optimization according to the anomaly detection results and generate warning signals. The warning signals include water level anomaly warnings, dam vibration warnings and potential dam breakage warnings; Adjustment and adaptation module, which is used to adjust the real-time operation parameters of the reservoir monitoring system according to the warning signals and dynamically optimize the feature extraction and fusion process in the FusionNet network to adapt to different monitoring requirements; Monitoring and update module, which is used to continuously monitor the reservoir environment and update the weight parameters in the FusionNet network in real time to optimize the anomaly event detection and warning mechanism.

Images