Dam abnormal operation state monitoring method, device, equipment, medium and product

Abstract

The invention discloses a dam abnormal operation state monitoring method and device, equipment, a medium and a product, and relates to the technical field of dam monitoring, and the dam abnormal operation state monitoring method comprises the steps: obtaining a real-time video and a real-time audio of a dam based on a video collection device and an audio collection device; pixel point processing is carried out on the real-time video to obtain time domain pixel features; performing spectrum analysis on the real-time audio to obtain spectrum features; searching for abnormal points based on the time domain pixel features and the frequency spectrum features, and determining the abnormal time point of the running state of the dam; and performing analysis based on the abnormal video segment and the abnormal audio segment at the abnormal time point of the operation state, and determining the abnormal operation state of the dam. Compared with a remote sensing technology, the scheme of the invention has higher real-time performance and microscopic abnormity monitoring capability, and the timeliness and comprehensiveness of the abnormal operation state of the dam are improved.

Description

Technical Field

[0001] The present application relates to the technical field of dam monitoring, and in particular to methods, devices, equipment, media and products for monitoring abnormal operation status of dams. Background Art

[0002] As an important water conservancy project facility, the safety of dams is directly related to the safety of downstream areas. Through abnormal monitoring of dams, potential problems of dams, such as cracks, deformation, seepage, and high water levels, can be discovered in time, so that corresponding measures can be taken to repair, reinforce or release floodwater, thereby ensuring the safe operation of dams, preventing disasters, and protecting people's lives and property. It is of vital importance.

[0003] Existing dam anomaly monitoring usually uses remote sensing technologies such as satellites and drones to obtain images and data information of the dam and its surrounding environment, and conduct macroscopic monitoring of the dam. However, remote sensing technology mainly relies on the observation and analysis of surface features to identify anomalies, and it is impossible to directly observe abnormal conditions inside the dam. Summary of the invention

[0004] The main purpose of this application is to provide a method, device, equipment, medium and product for monitoring the abnormal operation status of a dam, aiming to solve the technical problem that the existing dam abnormality monitoring method uses remote sensing technology to monitor and cannot directly observe the abnormalities inside the dam.

[0005] To achieve the above objectives, the present application proposes a method for monitoring the abnormal operation status of a dam, the method comprising:

[0006] Based on the video acquisition device and the audio acquisition device, real-time video and real-time audio of the dam are respectively acquired;

[0007] Performing pixel processing on the real-time video to obtain time-domain pixel features;

[0008] Performing spectrum analysis on the real-time audio to obtain spectrum features;

[0009] Searching for abnormal points based on the time domain pixel features and the frequency spectrum features to determine the abnormal time point of the dam's operating status;

[0010] The abnormal operating state of the dam is determined by analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating state.

[0011] In one embodiment, the step of searching for abnormal points based on the time domain pixel features and the frequency spectrum features to determine the abnormal time point of the dam's operating status includes:

[0012] Determining periodic pixel changes based on the time-domain pixel features;

[0013] Determining the abnormal pixel time point during the operation of the dam based on the periodic pixel change situation;

[0014] Determining periodic spectrum changes based on the spectrum characteristics;

[0015] Determine the abnormal spectrum time point when the dam is in operation based on the periodic spectrum change;

[0016] The abnormal time point of the operating state of the dam is determined based on the abnormal pixel time point and the abnormal spectrum time point.

[0017] In one embodiment, the step of determining the abnormal time point of the dam's operating state based on the abnormal pixel time point and the abnormal spectrum time point includes:

[0018] Determine the operation time point based on the timeline of dam operation;

[0019] When the operation time point is both an abnormal pixel time point and an abnormal spectrum time point, the operation time point is used as a dam and an abnormal operation state time point.

[0020] In one embodiment, the step of performing pixel processing on the real-time video to obtain time-domain pixel features includes:

[0021] Determine a video frame sequence for dam monitoring based on the real-time video;

[0022] Performing pixel point processing on the video frames in the video frame sequence to obtain pixel features of the video frames;

[0023] Determining the spatiotemporal features of the video frame sequence based on the pixel features of two adjacent video frames in the video frame sequence;

[0024] The temporal pixel features of the video frame sequence are determined according to the spatiotemporal features.

[0025] In one embodiment, the step of performing spectrum analysis on the real-time audio to obtain spectrum features includes:

[0026] Determine an audio frame sequence for dam monitoring based on the real-time audio;

[0027] Performing Fourier transform on the audio frame sequence to determine frequency spectrum features of the audio frame sequence.

[0028] In one embodiment, the step of analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating state to determine the abnormal operating state of the dam includes:

[0029] Acquire an abnormal video segment and an abnormal audio segment at the abnormal time point of the running state;

[0030] Determining a first abnormal operating state based on pixel features of the abnormal video segment;

[0031] Determining a second abnormal operating state based on the frequency spectrum characteristics of the abnormal audio segment;

[0032] If the first abnormal operation state is the same as the second abnormal operation state, taking the first abnormal operation state or the second abnormal operation state as the main abnormal operation state of the dam;

[0033] If the first abnormal operation state is different from the second abnormal operation state, the abnormal operation state with a higher abnormal occurrence rate is regarded as a primary abnormal operation state, and the abnormal operation state with a lower abnormal occurrence rate is regarded as a secondary abnormal operation state.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a device for monitoring the abnormal operation state of a dam, the device for monitoring the abnormal operation state of a dam comprising:

[0035] A data acquisition module, used for acquiring real-time video and real-time audio of the dam based on a video acquisition device and an audio acquisition device respectively;

[0036] A video processing module, used for performing pixel processing on the real-time video to obtain time-domain pixel features;

[0037] An audio processing module, used for performing spectrum analysis on the real-time audio to obtain spectrum features;

[0038] An abnormality search module, used to search for abnormal points based on the time domain pixel features and the frequency spectrum features, and determine the abnormal time point of the dam operation status;

[0039] The abnormality determination module is used to determine the abnormal operating state of the dam by analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating state.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a device for monitoring the abnormal operation status of a dam, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method for monitoring the abnormal operation status of a dam as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the abnormal operation status monitoring method of the dam as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the abnormal operation status monitoring method of the dam as described above.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] The present application obtains the real-time video and real-time audio of the dam based on the video acquisition device and the audio acquisition device respectively; performs pixel processing on the real-time video to obtain the time domain pixel features; performs spectrum analysis on the real-time audio to obtain the spectrum features; searches for abnormal points based on the time domain pixel features and spectrum features to determine the abnormal time point of the dam's operating status; analyzes the abnormal video segments and abnormal audio segments based on the abnormal time point of the operating status to determine the abnormal operating status of the dam. Since the real-time video and real-time audio of the dam are obtained by the video acquisition device and the audio acquisition device, the operating time point at which the dam may have abnormal conditions is determined by analyzing the real-time video and real-time audio, and then the video segment and audio segment of the operating time point are analyzed to determine whether the dam has abnormalities and abnormal conditions. Compared with the use of remote sensing technology, the solution of the present application has higher real-time and microscopic abnormality monitoring capabilities, which improves the timeliness and comprehensiveness of the abnormal operating status of the dam. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flow chart of the first embodiment of the abnormal operation status monitoring method of the dam in the present application;

[0048] Figure 2 A flow chart of the second embodiment of the abnormal operation state monitoring method of the dam in the present application;

[0049] Figure 3 A flow chart of the abnormal operation state monitoring method of a dam according to the present application is provided in Embodiment 3;

[0050] Figure 4 This is a schematic diagram of the module structure of the abnormal operation status monitoring device of the dam according to the embodiment of the present application;

[0051] Figure 5 It is a schematic diagram of the equipment structure of the hardware operating environment involved in the abnormal operating status monitoring method of the dam in the embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: based on the video acquisition device and the audio acquisition device, respectively obtain the real-time video and real-time audio of the dam; perform pixel point processing on the real-time video to obtain time domain pixel features; perform spectrum analysis on the real-time audio to obtain spectrum features; search for abnormal points based on the time domain pixel features and spectrum features to determine the abnormal time point of the dam's operating status; analyze the abnormal video segments and abnormal audio segments based on the abnormal time point of the operating status to determine the abnormal operating status of the dam.

[0056] Currently, existing technologies are carried out through remote sensing technology, which mainly relies on the observation and analysis of surface features to identify abnormal conditions. However, for these specific types of abnormal conditions, such as cracks and leakage inside the dam, it is impossible to observe them directly. In addition, in the event of a sudden dam situation, the processing and transmission of remote sensing data takes a certain amount of time, and the response to emergency situations is slow.

[0057] This application provides a solution, which obtains real-time video and real-time audio of the dam through a video acquisition device and an audio acquisition device, and determines the operating time point when the dam may have abnormal conditions by analyzing the real-time video and real-time audio, and then analyzes the video segment and audio segment of the operating time point to determine whether the dam has abnormalities and abnormal conditions. Compared with the use of remote sensing technology, the solution of this application has higher real-time and microscopic abnormality monitoring capabilities, and improves the timeliness and comprehensiveness of the abnormal operating status of the dam.

[0058] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as an industrial computer, a server, a monitoring terminal, etc., or an electronic device, a virtual device, etc. that can realize the above functions. The following takes the abnormal operation status monitoring device of the dam (hereinafter referred to as the monitoring device) as an example to illustrate this embodiment and the following embodiments.

[0059] Based on this, the embodiment of the present application provides a method for monitoring the abnormal operation status of a dam, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the abnormal operation status monitoring method of a dam according to the present application.

[0060] In this embodiment, the abnormal operation status monitoring method of the dam includes steps S10 to S50:

[0061] Step S10, obtaining real-time video and real-time audio of the dam based on the video acquisition device and the audio acquisition device respectively.

[0062] It should be noted that in actual applications, due to internal stress in the dam, long-term erosion by flowing water, etc., cracks and water seepage are prone to occur in some specific locations of the dam. In the embodiments of the present application, video acquisition devices can be installed on the dam top, spillway and some locations where cracks or water seepage are prone to occur. Through these video acquisition devices, real-time video acquisition of the dam operation can be performed, which can be used to determine the real-time operating status of the dam.

[0063] It should be understood that during the operation of the dam, the audio features corresponding to the sound of water flow at different water levels are different. And during the long-term operation of the dam, the sound frequency corresponding to the sound of water flow at the same water level tends to float within a range. When cracks or water seepage appear inside the dam, it is often accompanied by turbulent sound, structural vibration sound, etc., which causes abnormal features in the collected sound signal. By determining the current water level of the dam and collecting the real-time audio corresponding to the current water level, the real-time audio is analyzed to determine the real-time operating status of the dam.

[0064] In addition, it should be noted that the video acquisition device and the audio acquisition device in the embodiment of the present application can be arranged in pairs, that is, one video acquisition device can correspond to one audio acquisition device, and the corresponding video acquisition device and audio acquisition device can constitute an acquisition device group. By one-to-one correspondence between the video acquisition device and the audio acquisition device, more accurate monitoring of abnormal conditions occurring during the operation of the dam can be achieved, thereby improving the robustness of the abnormal operation status monitoring method of the dam of the present application.

[0065] It can be understood that since the functions and effects of each group of collection devices in the embodiments of the present application are roughly the same, the embodiments of the present application and the following embodiments take one group as an example to explain in detail the abnormal operation status monitoring method of the dam of the present application.

[0066] In a specific implementation, the monitoring device of the embodiment of the present application can obtain real-time video and real-time audio of the dam based on a group of video acquisition devices and audio acquisition devices. Since data collection is achieved through a group of acquisition devices, compared with the existing remote sensing technology that collects data through satellites and drones, the solution of the present application has higher real-time and flexibility, and can monitor local anomalies during the operation of the dam. At the same time, the accuracy of abnormal monitoring is improved by mutual verification of real-time video and real-time audio.

[0067] Step S20, performing pixel processing on the real-time video to obtain time-domain pixel features.

[0068] It should be noted that pixelization is a technology for processing images, which can convert images into a specific pattern composed of pixels. By pixelizing the video frame sequence corresponding to the real-time video, the image can be decomposed into smaller units, making the analysis and processing of the image more refined, so as to more accurately capture the changes on the dam surface or the surrounding environment, such as cracks, displacement, high water level and other abnormal phenomena.

[0069] It is understandable that real-time video is composed of a continuous sequence of video frames, wherein each video frame can be composed of a large number of pixels. The above pixel processing of real-time video is to perform pixel processing on each video frame in the video frame sequence, thereby determining the time domain pixel features of the real-time video.

[0070] In a specific implementation, the monitoring device of the embodiment of the present application performs pixel point processing on the video frame sequence in the real-time video to obtain the time domain pixel features corresponding to the real-time video, and can monitor the abnormalities in the operation of the dam based on the time domain pixel features.

[0071] Step S30, performing spectrum analysis on the real-time audio to obtain spectrum features.

[0072] It should be noted that spectrum analysis is an audio analysis method that converts real-time audio from the time domain to the frequency domain. By performing spectrum analysis on real-time audio, the spectrum characteristics of the audio signal corresponding to the real-time audio can be determined (such as amplitude, power, intensity or phase characteristics). By analyzing the spectrum characteristics of the audio signal, abnormal changes in the dam's operating status can be identified. For example, when cracks or water seepage occur in the dam, its spectrum characteristics will change accordingly. By monitoring these changes in real time, potential safety issues can be discovered in a timely manner.

[0073] In a specific implementation, the monitoring device of the embodiment of the present application can perform spectral analysis on the audio signal corresponding to the real-time audio, thereby obtaining the spectral characteristics of the real-time audio, and based on the spectral characteristics of the real-time audio, it can monitor abnormalities in the operation of the dam.

[0074] Step S40, searching for abnormal points based on the time domain pixel features and the frequency spectrum features to determine the abnormal time point of the dam's operating status;

[0075] Step S50, analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating status to determine the abnormal operating status of the dam.

[0076] It should be noted that, by performing anomaly analysis based on time domain pixel features, the abnormal video time point in the collected real-time video can be determined; by performing anomaly analysis based on spectrum features, the abnormal audio time point in the collected real-time audio can be determined. Based on the abnormal video time point and the abnormal audio time point, the abnormal time point of the dam operation status can be determined.

[0077] It should be explained that the above-mentioned method for performing abnormal analysis on time-domain pixel features and spectral features can be based on a deep learning model or other methods, and the embodiments of the present application are not limited to this.

[0078] It is understandable that when searching for abnormal points through real-time video or real-time audio alone, some redundant abnormal time points may appear in the analysis results due to the influence of weather, light, obstructions, environmental noise, etc. Therefore, when a time point is both an abnormal video time point and an abnormal audio time point, the time point can be regarded as an abnormal time point of dam operation, thereby improving the accuracy of abnormal monitoring.

[0079] It should be noted that when determining the abnormal time point, the abnormal operating state corresponding to the abnormal time point can be further determined. The abnormal video segment is the real-time video corresponding to the abnormal operating state time point, and the abnormal audio segment is the real-time audio corresponding to the abnormal operating state time point. By sending the abnormal video segment and the abnormal audio segment to relevant management personnel, the abnormal type and degree of the dam can be determined, that is, the abnormal operating state of the dam can be determined.

[0080] In one implementation of an embodiment of the present application, the method for determining the abnormal operating state can be based on a deep learning model, which is not limited by the embodiment of the present application.

[0081] The embodiment of the present application obtains the real-time video and real-time audio of the dam based on the video acquisition device and the audio acquisition device respectively; performs pixel point processing on the real-time video to obtain the time domain pixel features; performs spectrum analysis on the real-time audio to obtain the spectrum features; performs abnormal point search based on the time domain pixel features and spectrum features to determine the abnormal time point of the dam's operating state; analyzes the abnormal video segment and abnormal audio segment based on the abnormal time point of the operating state to determine the abnormal operating state of the dam. Since the real-time video and real-time audio of the dam are obtained by the video acquisition device and the audio acquisition device, the operating time point at which the dam may have abnormal conditions is determined by analyzing the real-time video and real-time audio, and then the video segment and audio segment of the operating time point are analyzed to determine whether the dam has abnormalities and abnormal conditions. Compared with the use of remote sensing technology, the solution of the present application has higher real-time and microscopic abnormality monitoring capabilities, and improves the timeliness and comprehensiveness of the abnormal operating state of the dam.

[0082] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 In the embodiment of the present application, the step of searching for abnormal points based on the time domain pixel features and the frequency spectrum features to determine the abnormal time point of the dam's operating status includes:

[0083] Step S41, determining periodic pixel changes based on the time domain pixel features.

[0084] It should be noted that during the use of the dam, the temporal pixel characteristics of the video and the spectral characteristics of the audio collected by the video acquisition device and the audio acquisition device at the same water level are generally relatively stable. Based on this situation, in the embodiment of the present application, a target sampling period can be set for the real-time video and real-time audio, and the sampling period can be one hour, two hours, one day or other time, which is not limited in the embodiment of the present application.

[0085] Step S42, determining the abnormal pixel time point during the operation of the dam based on the periodic pixel change;

[0086] Step S43, determining periodic spectrum changes based on the spectrum characteristics;

[0087] Step S44, determining the abnormal spectrum time point during the operation of the dam based on the periodic spectrum change;

[0088] It is understandable that, in each sampling period, the corresponding periodic pixel changes and periodic spectrum changes can be determined. Based on the periodic pixel changes and periodic spectrum abnormal changes in each sampling period, it can be determined whether there is a real-time video anomaly or a real-time audio anomaly in each sampling period.

[0089] It should be noted that each sampling period can be regarded as a time point. For a sampling period with real-time video anomalies, its corresponding abnormal pixel time point can be determined; for a sampling period with real-time audio anomalies, its corresponding abnormal spectrum time point can be determined.

[0090] Step S45, determining the abnormal time point of the dam's operating status based on the abnormal pixel time point and the abnormal spectrum time point.

[0091] It should be noted that when a sampling period is both an abnormal pixel time point and an abnormal spectrum time point, the sampling period can be used as an abnormal operation status time point of the dam.

[0092] Specifically, the step of determining the abnormal time point of the dam operation state based on the abnormal pixel time point and the abnormal spectrum time point includes:

[0093] Determine the operation time point based on the timeline of dam operation;

[0094] When the operation time point is both an abnormal pixel time point and an abnormal spectrum time point, the operation time point is used as a dam and an abnormal operation state time point.

[0095] It should be noted that based on the time axis of the dam operation and the duration of the sampling period, it can be divided into several operation time points, each of which corresponds to a sampling period. When the operation time point is both an abnormal pixel time point and an abnormal spectrum time point, the operation time point can be used as the abnormal operation state time point of the dam.

[0096] The embodiment of the present application determines the periodic pixel changes based on the time domain pixel features; determines the abnormal pixel time points during the operation of the dam based on the periodic pixel changes; determines the periodic spectrum changes based on the spectrum features; determines the abnormal spectrum time points during the operation of the dam based on the periodic spectrum changes; determines the abnormal time points of the dam's operating state based on the abnormal pixel time points and the abnormal spectrum time points. Since the abnormal time points of the dam's operating state are determined based on the periodic pixel changes and the periodic spectrum changes, the periodic changes and abnormal change points during the operation of the dam can be accurately identified, thereby improving the accuracy of abnormal monitoring.

[0097] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned first embodiment and / or the second embodiment can refer to the above introduction, and will not be repeated later. Figure 3 In the embodiment of the present application, the step of performing pixel processing on the real-time video to obtain time-domain pixel features includes:

[0098] Step S21, determining a video frame sequence for dam monitoring based on the real-time video;

[0099] Step S22, performing pixel point processing on the video frames in the video frame sequence to obtain pixel features of the video frames;

[0100] It should be noted that when performing pixel processing on the video frames in the video frame sequence, the video frames may be firstly grayed out, and noise therein may be removed by filtering to obtain a standard video image.

[0101] It is understandable that by traversing each pixel in the standard video image, the grayscale value of each pixel is determined. By extracting the gradient, texture, and grayscale value of each pixel, the pixel features of the video frame can be determined. The pixel features can be represented in the form of feature vectors or feature matrices, which is not limited in the embodiments of the present application.

[0102] Step S23, determining the space-time characteristics of the video frame sequence based on the pixel characteristics of two adjacent video frames in the video frame sequence;

[0103] It should be noted that the spatiotemporal features are features that combine the two dimensions of space and time. The spatiotemporal features not only include the static information such as the spatial position, texture, and edge of each pixel in the video frame, but also include the dynamic information of these pixels or features changing over time between consecutive frames. By establishing a motion equation between two adjacent video frames and using a sparse optical flow algorithm to calculate the motion vector of each pixel, the displacement of each pixel can be determined. At the same time, the spatial geometry can be constructed based on the pixel features of the pixels. By fusing the motion vector and the spatial geometry flow field, the spatiotemporal features containing spatial structure information and temporal motion information can be obtained.

[0104] Step S24: determining the temporal pixel features of the video frame sequence according to the spatiotemporal features.

[0105] It is understandable that, when the space-time feature is determined, the space-time feature can be used as the time-domain pixel feature of the video frame sequence, thereby realizing the analysis of the periodic pixel changes of the real-time video.

[0106] In one implementation of the embodiment of the present application, in order to determine the spectrum characteristics of the real-time audio, the step of performing spectrum analysis on the real-time audio to obtain the spectrum characteristics includes:

[0107] Determine an audio frame sequence for dam monitoring based on the real-time audio;

[0108] Performing Fourier transform on the audio frame sequence to determine frequency spectrum features of the audio frame sequence.

[0109] It should be noted that in the embodiment of the present application, the spectrum analysis method for real-time audio may be fast Fourier transform, wavelet transform or other methods, and the embodiment of the present application is not limited to this.

[0110] In an implementation of the embodiment of the present application, the step of analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating state to determine the abnormal operating state of the dam includes:

[0111] Acquire an abnormal video segment and an abnormal audio segment at the abnormal time point of the running state;

[0112] Determining a first abnormal operating state based on pixel features of the abnormal video segment;

[0113] Determining a second abnormal operating state based on the frequency spectrum characteristics of the abnormal audio segment;

[0114] It should be noted that when the abnormal video segment and abnormal audio segment corresponding to the abnormal motion state time period are obtained, the abnormal video segment can be monitored based on the video monitoring model, and the abnormal audio segment can be monitored based on the audio monitoring model. Through the video monitoring model, the abnormal type corresponding to the abnormal video segment and the confidence of the video abnormality can be determined; through the audio monitoring model, the abnormal type corresponding to the abnormal audio segment and the confidence of the audio abnormality can be determined.

[0115] It can be understood that, based on the abnormal type corresponding to the abnormal video segment, the first abnormal operating state can be determined; based on the abnormal type corresponding to the abnormal audio segment, the second abnormal operating state can be determined.

[0116] It should be understood that the above-mentioned video monitoring model or audio monitoring model can be constructed based on an existing neural network model, such as a BP neural network model, a CNN neural network, etc., and the embodiments of the present application are not limited to this.

[0117] If the first abnormal operation state is the same as the second abnormal operation state, taking the first abnormal operation state or the second abnormal operation state as the main abnormal operation state of the dam;

[0118] If the first abnormal operation state is different from the second abnormal operation state, the abnormal operation state with a higher abnormal occurrence rate is regarded as a primary abnormal operation state, and the abnormal operation state with a lower abnormal occurrence rate is regarded as a secondary abnormal operation state.

[0119] It should be noted that, when the first abnormal operating state and the second abnormal operating state are the same, the first abnormal operating state / the second abnormal operating state can be determined as the main abnormal operating state of the dam, and the management user can be reminded based on the main abnormal operating state; when the first abnormal operating state and the second abnormal operating state are different, the video abnormality confidence can be used as the abnormality incidence rate of the first abnormal operating state, and the audio abnormality confidence can be used as the abnormality incidence rate of the second operating state, and the abnormality incidence rate of the first abnormal operating state and the abnormality incidence rate of the second abnormal operating state are compared, and the abnormal operating state with a higher abnormal incidence rate is used as the main abnormal operating state, and the abnormal operating state with a lower abnormal incidence rate is used as the secondary abnormal operating state, thereby improving the accuracy of abnormal operating state monitoring of the present application.

[0120] The embodiment of the present application determines a video frame sequence for dam monitoring based on real-time video; performs pixel point processing on the video frames in the video frame sequence to obtain the pixel features of the video frames; determines the space-time features of the video frame sequence based on the pixel features of two adjacent video frames in the video frame sequence; and determines the time-domain pixel features of the video frame sequence based on the space-time features. Since the space-time features of the video frame sequence are determined, accurate detection of abnormal operation status of the dam is achieved.

[0121] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the abnormal operation status monitoring method of the dam of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0122] This application also provides a dam abnormal operation status monitoring device, please refer to Figure 4 , the abnormal operation status monitoring device of the dam comprises:

[0123] A data acquisition module 10 is used to acquire real-time video and real-time audio of the dam based on a video acquisition device and an audio acquisition device respectively;

[0124] The video processing module 20 is used to perform pixel processing on the real-time video to obtain time-domain pixel features;

[0125] The audio processing module 30 is used to perform spectrum analysis on the real-time audio to obtain spectrum characteristics;

[0126] An abnormality search module 40 is used to search for abnormal points based on the time domain pixel features and the frequency spectrum features to determine the abnormal time point of the dam operation status;

[0127] The abnormality determination module 50 is used to determine the abnormal operating state of the dam by analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating state.

[0128] The abnormal operation state monitoring device of the dam provided by the present application adopts the abnormal operation state monitoring method of the dam in the above-mentioned embodiment, which can solve the technical problem that the existing abnormal operation monitoring method of the dam is monitored by remote sensing technology and the abnormality inside the dam cannot be directly observed. Compared with the prior art, the beneficial effects of the abnormal operation state monitoring device of the dam provided by the present application are the same as the beneficial effects of the abnormal operation state monitoring method of the dam provided by the above-mentioned embodiment, and the other technical features of the abnormal operation state monitoring device of the dam are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0129] The present application provides a device for monitoring the abnormal operating status of a dam, and the device for monitoring the abnormal operating status of a dam includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for monitoring the abnormal operating status of the dam in the above-mentioned embodiment one.

[0130] Reference below Figure 5 , which shows a schematic diagram of the structure of the abnormal operation state monitoring device of the dam suitable for implementing the embodiment of the present application. The abnormal operation state monitoring device of the dam in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The abnormal operation status monitoring device of the dam shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0131] like Figure 5As shown, the abnormal operation state monitoring device of the dam may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the abnormal operation state monitoring device of the dam are also stored. The processing device 1001, the ROM 1002 and the RAM 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the abnormal operation status monitoring device of the dam to communicate wirelessly or wired with other devices to exchange data. Although the abnormal operation status monitoring device of the dam with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0132] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0133] The abnormal operation state monitoring device of the dam provided by the present application adopts the abnormal operation state monitoring method of the dam in the above embodiment, which can solve the technical problem that the existing abnormal dam monitoring method uses remote sensing technology to monitor and cannot directly observe the abnormalities inside the dam. Compared with the prior art, the beneficial effects of the abnormal operation state monitoring device of the dam provided by the present application are the same as the beneficial effects of the abnormal operation state monitoring method of the dam provided by the above embodiment, and the other technical features of the abnormal operation state monitoring device of the dam are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0134] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0135] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0136] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the abnormal operation status monitoring method of the dam in the above-mentioned embodiment.

[0137] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0138] The computer-readable storage medium may be included in the abnormal operation status monitoring device of the dam; or may exist independently without being assembled into the abnormal operation status monitoring device of the dam.

[0139] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the abnormal operation status monitoring device of the dam, the abnormal operation status monitoring device of the dam:

[0140] Based on the video acquisition device and the audio acquisition device, real-time video and real-time audio of the dam are respectively acquired;

[0141] Performing pixel processing on the real-time video to obtain time-domain pixel features;

[0142] Performing spectrum analysis on the real-time audio to obtain spectrum features;

[0143] Searching for abnormal points based on the time domain pixel features and the frequency spectrum features to determine the abnormal time point of the dam's operating status;

[0144] The abnormal operating state of the dam is determined by analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating state.

[0145] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0148] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned abnormal operation status monitoring method of the dam, and can solve the technical problem that the existing abnormal dam monitoring method uses remote sensing technology to monitor, and the abnormalities inside the dam cannot be directly observed. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the abnormal operation status monitoring method of the dam provided by the above-mentioned embodiment, and will not be repeated here.

[0149] The present application also provides a computer program product, including a computer program, which implements the steps of the abnormal operation status monitoring method of the dam as described above when the computer program is executed by a processor.

[0150] The computer program product provided by the present application can solve the technical problem that the existing dam abnormality monitoring method uses remote sensing technology to monitor, but the abnormalities inside the dam cannot be directly observed. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the abnormal operation status monitoring method of the dam provided by the above embodiment, which will not be repeated here.

[0151] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for monitoring abnormal operation status of a dam, characterized in that: The method comprises: Based on the video acquisition device and the audio acquisition device, real-time video and real-time audio of the dam are respectively acquired; Performing pixel processing on the real-time video to obtain time-domain pixel features; Performing spectrum analysis on the real-time audio to obtain spectrum features; Searching for abnormal points based on the time domain pixel features and the frequency spectrum features to determine the abnormal time point of the dam's operating status; The abnormal operating state of the dam is determined by analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating state.

2. The abnormal operation status monitoring method of a dam according to claim 1, characterized in that: The step of searching for abnormal points based on the time domain pixel features and the frequency spectrum features to determine the abnormal time point of the dam's operating status includes: Determining periodic pixel changes based on the time-domain pixel features; Determining the abnormal pixel time point during the operation of the dam based on the periodic pixel change situation; Determining periodic spectrum changes based on the spectrum characteristics; Determine the abnormal spectrum time point when the dam is in operation based on the periodic spectrum change; The abnormal time point of the operating state of the dam is determined based on the abnormal pixel time point and the abnormal spectrum time point.

3. The abnormal operation status monitoring method of a dam according to claim 2, characterized in that: The step of determining the abnormal time point of the dam's operating state based on the abnormal pixel time point and the abnormal spectrum time point includes: Determine the operation time point based on the timeline of dam operation; When the operation time point is both an abnormal pixel time point and an abnormal spectrum time point, the operation time point is used as a dam and an abnormal operation state time point.

4. The abnormal operation status monitoring method of a dam according to claim 1, characterized in that: The step of performing pixel processing on the real-time video to obtain time-domain pixel features includes: Determine a video frame sequence for dam monitoring based on the real-time video; Performing pixel point processing on the video frames in the video frame sequence to obtain pixel features of the video frames; Determining the spatiotemporal features of the video frame sequence based on the pixel features of two adjacent video frames in the video frame sequence; The temporal pixel features of the video frame sequence are determined according to the spatiotemporal features.

5. The abnormal operation status monitoring method of a dam according to claim 1, characterized in that: The step of performing spectrum analysis on the real-time audio to obtain spectrum features comprises: Determine an audio frame sequence for dam monitoring based on the real-time audio; Performing Fourier transform on the audio frame sequence to determine frequency spectrum features of the audio frame sequence.

6. The abnormal operation status monitoring method of a dam according to claim 1, characterized in that: The step of analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating state to determine the abnormal operating state of the dam comprises: Acquire an abnormal video segment and an abnormal audio segment at the abnormal time point of the running state; Determining a first abnormal operating state based on pixel features of the abnormal video segment; Determining a second abnormal operating state based on the frequency spectrum characteristics of the abnormal audio segment; If the first abnormal operation state is the same as the second abnormal operation state, taking the first abnormal operation state or the second abnormal operation state as the main abnormal operation state of the dam; If the first abnormal operation state is different from the second abnormal operation state, the abnormal operation state with a higher abnormal occurrence rate is regarded as a primary abnormal operation state, and the abnormal operation state with a lower abnormal occurrence rate is regarded as a secondary abnormal operation state.

7. A device for monitoring abnormal operation status of a dam, characterized in that: The abnormal operation status monitoring device of the dam comprises: A data acquisition module, used for acquiring real-time video and real-time audio of the dam based on a video acquisition device and an audio acquisition device respectively; A video processing module, used for performing pixel processing on the real-time video to obtain time-domain pixel features; An audio processing module, used for performing spectrum analysis on the real-time audio to obtain spectrum features; An abnormality search module, used to search for abnormal points based on the time domain pixel features and the frequency spectrum features, and determine the abnormal time point of the dam operation status; The abnormality determination module is used to determine the abnormal operating state of the dam by analyzing the abnormal video segment and the abnormal audio segment at the abnormal time point of the operating state.

8. A dam abnormal operation status monitoring device, characterized in that: The device includes: a memory, a processor, and a dam abnormal operation status monitoring program stored in the memory and executable on the processor, wherein the dam abnormal operation status monitoring program is configured to implement the steps of the dam abnormal operation status monitoring method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a program for monitoring the abnormal operation status of a dam, and when the program for monitoring the abnormal operation status of a dam is executed by a processor, the steps of the method for monitoring the abnormal operation status of a dam as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the abnormal operation status monitoring method of a dam according to any one of claims 1 to 6 are implemented.

Images