Intelligent inspection method and device for water conservancy projects based on video monitoring, and electronic equipment

Abstract

The present invention discloses a method and device for intelligent inspection of water conservancy projects based on video monitoring, and electronic equipment, comprising: establishing a standard database of hidden dangers to be inspected; gridding the inspection area of ​​the water conservancy project according to the monitoring range of each camera, wherein each grid has at least one camera and a voice broadcasting device, and at least four feature points are set within the monitoring range of each camera; receiving relevant parameters of the feature points and the corresponding cameras, and binding and geo-locating the two; receiving video images collected by the cameras and the video parameters of the cameras; establishing a sample library required for video recognition; performing video recognition and spatial fusion as well as dynamic evolution monitoring and analysis on the video images according to the standard database of hidden dangers and the sample library, and obtaining hidden danger identification results; and producing early warning information for early warning according to the engineering hidden dangers and irregular behaviors identified in the hidden danger identification results.

Description

Technical Field

[0001] The present application relates to the technical field of water conservancy project inspection and management, and in particular to a water conservancy project intelligent inspection method and device, and electronic equipment based on video surveillance. Background Art

[0002] The safety of water conservancy projects is the bottom line of project management. In order to prevent and resolve major risks and hidden dangers, the safety management of water conservancy projects has always been mainly based on the traditional manual fixed-point inspection method. Affected by terrain, weather factors, and personnel level and experience, there are problems such as incomplete inspection coverage of visual blind spots, low timeliness, untimely discovery of problems, and difficulty in continuous tracking and supervision. During the entire inspection process, human resources are consumed greatly, and hidden danger inspections rely entirely on the personal judgment of inspectors, which cannot guarantee full coverage of hidden danger inspections and true and reliable inspection results. With the reform of the water pipe system, due to the reduction of personnel and the increase in labor costs, the traditional working method of manual inspections can no longer be fully covered by manpower. In addition, manual inspections are time-consuming and labor-intensive, and are prone to omissions and misjudgments. They cannot provide early warning information in real time, which makes the early warning work of project safety management lag and have hidden dangers, and the efficiency of early warning supervision is low. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method and device for intelligent inspection of water conservancy projects based on video surveillance, as well as electronic equipment, so as to realize that intelligent technical means replace part of the manual inspection management, assist in the management scope to form efficient intelligent inspection and management prediction, and closed-loop processing of hidden dangers, so as to solve the technical problems of insufficient personnel and incomplete hidden danger detection existing in related technologies.

[0004] According to a first aspect of an embodiment of the present application, there is provided a method for intelligent inspection of a water conservancy project based on video surveillance, comprising:

[0005] S0: Sort out the inspection list of water conservancy projects according to the supervision and inspection methods of water conservancy projects and the operation and management regulations of water conservancy projects, and establish a standard database of hidden dangers that need to be inspected;

[0006] S1: The water conservancy project inspection area is gridded according to the monitoring range of each camera, wherein each grid has at least one camera and a voice broadcasting device, and at least four feature points are set within the monitoring range of each camera and directly located on the electronic map of the inspection area;

[0007] S2: receiving the relevant parameters of the feature point and the corresponding camera, and binding the two;

[0008] S3: receiving the video image captured by the camera and the video parameters and geographic location of the camera;

[0009] S4: Establish the sample library required for video recognition;

[0010] S5: performing video recognition and spatial fusion as well as dynamic evolution monitoring and analysis calculation on the video image according to the hidden danger standard database and sample library to obtain hidden danger identification results;

[0011] S6: Based on the engineering hazards identified in the hazard identification results, a first warning message is generated, and the first warning message is sent to the client of the person in charge of management; for the irregular behaviors identified in the hazard identification results, a second warning message is generated, and the second warning message is converted into voice and sent to the voice broadcasting device for voice broadcast warning.

[0012] Optionally, also include:

[0013] S7, handle hidden dangers according to the solutions in the early warning information, and transfer the confirmed problems, corresponding pictures, and actual treatment measures to the sample library.

[0014] Optionally, the hidden dangers in the hidden danger standard database are divided into two categories: engineering hidden dangers and irregular behaviors according to whether the objects involved are objects or people. Each type of hidden danger is also divided into two levels: general hidden dangers and serious hidden dangers according to the hazard and difficulty of rectification. The general hidden dangers are those that can be immediately rectified and eliminated after discovery, and the remaining hidden dangers are major hidden dangers. The hidden danger standard database also has corresponding hidden danger handling countermeasures and solutions established for hidden danger categories and levels based on water conservancy project management knowledge and historical experience in safety handling.

[0015] Optionally, receive the video image captured by the camera and the video parameters of the camera, including:

[0016] S31: Image acquisition is performed according to inspection requirements. During the non-flood season, grids are patrolled in accordance with the daily inspection frequency requirements to ensure full coverage within the inspection cycle. During the flood season, grids where key inspection objects are located can be encrypted for identification, and images with higher resolution within the inspection cycle can be selected for identification. During the operation of facilities and equipment, uninterrupted sampling can be performed for real-time or quasi-real-time sampling and identification.

[0017] S32: Acquire camera parameters while capturing video images.

[0018] Optionally, build a sample library required for video recognition, including:

[0019] S41: Select the video images received in the grid according to the inspection object requirements, and cut them into images containing the engineering facilities to be inspected and the set feature points according to certain rules, and mark the hidden danger labels, time labels and camera parameters;

[0020] S42: geometrically correcting and registering the selected series of pictures and geographically positioning them according to the feature points, marking the samples of normal and abnormal categories, and marking the pictures of engineering hazards and irregular behaviors in the abnormal samples.

[0021] Optionally, performing video recognition on the acquired video image according to the hidden danger standard database and sample library to obtain hidden danger recognition results includes:

[0022] S51, selecting images with the same parameters in the sample library according to the collected camera parameters, and performing registration and geolocation according to preset feature points;

[0023] S52, automatically selecting a normal picture of the same period in history or the last normal picture with the same parameters in the same geographic space area from the sample library according to the geographic location of the video on the electronic map as a comparison picture, using a background difference method or a frame difference method to perform inspection and detection of hidden dangers and illegal behaviors, and obtaining a changed area 1 as a preliminary recognition result;

[0024] S53, using an image recognition algorithm based on deep learning, to detect abnormal targets and intelligently identify images on grid images, to build a deep learning model suitable for identifying hidden dangers, to continuously train engineering hidden dangers and irregular behaviors of personnel, to identify different images, and to synchronously recommend suspected problems based on image recognition similarity. During the training process, it is necessary to continuously collect and mark relevant samples; the result with the highest similarity is confirmed as the recognition result, and the other recognition results are sorted from high to low according to the similarity, and are recommended as suspected problems for association, and the change area 2 is obtained;

[0025] S54, performing intersection, adjacency, and inclusion spatial relationship judgment and spatial fusion calculation on the change region 1 and the change region 2. When they are intersection or adjacency, the sum of the two is taken as the change region. When they are inclusion, the larger area is taken as the change region, which is used as the marked area for category intelligent recognition judgment;

[0026] S55, performing morphological and feature comparison between the above-mentioned changed area and abnormal samples in the sample library to obtain the hidden danger category, and at the same time, fuzzy matching the corresponding hidden danger or irregular behavior description in the hidden danger standard database according to the category of image recognition as a suspected problem;

[0027] S56, writing the identified suspected problems into the corresponding suspected problem database according to the two categories of engineering hidden dangers and irregular behaviors of fuzzy matching, and providing the location of the suspected problem, probe number, grid number, picture time, and suspected problem category on the picture;

[0028] S57, according to the classified suspected problems, fuzzy matching the general hidden dangers and serious hidden dangers in the hidden danger standard database to obtain hidden danger classification information;

[0029] S58. Identify general potential hazards through continuous encrypted inspections, conduct dynamic evolution monitoring and analysis. Through the inter-frame difference method, utilize the differences between consecutive frames in the image sequence to monitor and analyze the evolution of potential hazards. For continuous real-time images f t-2 (x, y), f t-1 (x, y), f t (x, y), perform a subtraction operation to obtain the difference image D t-1 (x,y) = f t-1 (x, y) - f t-2 (x, y), D t (x,y) = f t (x, y) - f t-1 (x, y). For the difference image D t-1 (x,y), D t (x,y), divide it by the corresponding time interval to obtain the change rate, V t (x,y) = D t (x,y) / ΔT1, V t-1 (x,y) = D t-1 (x,y) / ΔT2, where ΔT1 is the time interval between the image at time t and the image at time t - 1, and ΔT2 is the time interval between the image at time t - 1 and the image at time t - 2. For the change rate V t 、V t-1 Perform another difference calculation to obtain ΔV(x,y) = V t (x,y) - V t-1 (x,y). When ΔV(x,y) >= T (T is the threshold, simply T can be taken as 0), it is considered that the potential hazard is evolving and developing. When ΔV(x,y) < T, it is considered that the potential hazard is gradually being eliminated. If the potential hazard is continuously evolving, modify the potential hazard level to a serious potential hazard.

[0030] Optionally, it further includes:

[0031] S59. According to the classified and graded potential hazards, fuzzy match the corresponding countermeasures and solutions in the potential hazard standard database, and generate a warning message. The warning message includes the potential hazard category, level, grid (probe) to which it belongs, and solution.

[0032] According to the second aspect of the embodiments of the present application, a water conservancy project intelligent inspection device based on video monitoring is provided. When applied to a server, it includes:

[0033] A first establishment module, configured to sort out the water conservancy project inspection list according to the water conservancy project supervision and inspection measures and the water conservancy project operation management regulations, and establish a potential hazard standard database to be inspected;

[0034] A gridding module is used to grid the inspection area of ​​the water conservancy project according to the monitoring range of each camera, wherein each grid has at least one camera and a voice broadcasting device, and at least four feature points are set within the monitoring range of each camera and directly located on the electronic map of the inspection area;

[0035] A first receiving module is used to receive the relevant parameters of the feature point and the corresponding camera, and bind the two;

[0036] The second receiving module is used to receive the video image collected by the camera and the video parameters and geographic location of the camera;

[0037] The second building module is used to build a sample library required for video recognition;

[0038] An identification module is used to perform video identification and spatial fusion as well as dynamic evolution monitoring and analysis calculation on the video image according to the hidden danger standard database and sample library to obtain hidden danger identification results;

[0039] The early warning module is used to generate a first early warning message based on the engineering hidden dangers identified in the hidden danger identification results, and send the first early warning message to the client of the person in charge of management; generate a second early warning message for the irregular behavior identified in the hidden danger identification results, and convert the second early warning message into voice and send it to the voice broadcasting device for voice broadcast early warning.

[0040] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

[0041] one or more processors;

[0042] A memory for storing one or more programs;

[0043] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0044] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0045] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0046] The present invention avoids the omission of manual inspection of hidden dangers and realizes the list-based supervision of hidden dangers by establishing a standard database of hidden dangers. Feature points are arranged for each image coverage area and directly located on the electronic map of the inspection area, which solves the difficulties of image correction, registration and geographic positioning, and is easy to manage and directly compare through a geographic information system. The accuracy of hidden danger identification is improved by using the video image difference method and deep learning intelligent recognition judgment and spatial fusion calculation of recognition results of different methods, and the influence of insufficient samples on recognition accuracy in deep learning is solved. The development state of hidden dangers is dynamically controlled by dynamic evolution monitoring and analysis calculation, which improves the accuracy of hidden danger classification and risk prevention and control capabilities, changes the traditional management mode that relies heavily on manual inspections and monitoring screens in previous safety management, reduces the investment of human resources, and realizes the remote, visual and real-time safety inspection of water conservancy projects. By automatically warning irregular behaviors through wireless broadcasting in real time, the efficiency of water conservancy project safety management is effectively improved and safe production is guaranteed.

[0047] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] Figure 1 The present invention is a flowchart of a method for intelligent inspection of water conservancy projects based on video monitoring according to an exemplary embodiment.

[0050] Figure 2 is a grid diagram according to an exemplary embodiment.

[0051] Figure 3 1 is a relationship diagram between change region 1 and change region 2 according to an exemplary embodiment, wherein (a) change regions 1 and 2 intersect and the sum is taken; (b) change regions 1 and 2 are adjacent and the sum is taken; (c) change regions 1 and 2 are included and the largest area is taken.

[0052] Figure 4 The present invention is a block diagram of a water conservancy project intelligent inspection device based on video monitoring according to an exemplary embodiment. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0054] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0055] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0056] Figure 1 is a flow chart of a method for intelligent inspection of water conservancy projects based on video monitoring according to an exemplary embodiment. Figure 1 As shown, the method is applied to the server, and may include the following steps:

[0057] S0: Sort out the inspection list of water conservancy projects according to the supervision and inspection methods of water conservancy projects and the operation and management regulations of water conservancy projects, and establish a standard database of hidden dangers that need to be inspected;

[0058] S1: The water conservancy project inspection area is gridded according to the monitoring range of each camera, wherein each grid has at least one camera and a voice broadcasting device, and at least four feature points are set within the monitoring range of each camera and directly located on the electronic map of the inspection area;

[0059] S2: receiving the relevant parameters of the feature point and the corresponding camera, and binding the two;

[0060] S3: receiving the video image captured by the camera and the video parameters and geographic location of the camera;

[0061] S4: Establish a sample library based on the collected video images.

[0062] S5: performing video recognition and spatial fusion as well as dynamic evolution monitoring and analysis calculation on the video image according to the hidden danger standard database and sample library to obtain hidden danger identification results;

[0063] S6: For the engineering hidden dangers identified in the hidden danger identification results, generate first warning information, and send the first warning information to the client of the person in charge of management; for the irregular behavior identified in the hidden danger identification results, generate second warning information, and convert the second warning information into voice and send it to the voice broadcasting device for voice broadcast warning;

[0064] The present invention avoids the omission of manual inspection of hidden dangers and realizes the list-based supervision of hidden dangers by establishing a standard database of hidden dangers; characteristic points are arranged for each image coverage area and directly positioned on the electronic map of the inspection area, which solves the difficulties of image correction, registration and geographic positioning, and is easy to manage and directly compare through a geographic information system. The accuracy of hidden danger identification is improved by the spatial fusion of video image difference method, deep learning intelligent recognition judgment and recognition results of different methods, and the influence of insufficient samples on recognition accuracy in deep learning is solved. The development state of hidden dangers is dynamically controlled through dynamic evolution monitoring and analysis calculation, which improves the accuracy of hidden danger classification and risk prevention and control capabilities, changes the traditional management mode that relies heavily on manual inspections and monitoring screens in previous safety management, reduces the investment of human resources, and realizes the remote, visual and real-time safety inspection of water conservancy projects. By automatically warning irregular behaviors through wireless broadcasting in real time, the efficiency of water conservancy project safety management is effectively improved, and safe production is guaranteed.

[0065] In the specific implementation of S0, the inspection list of water conservancy projects is sorted out according to the supervision and inspection methods of water conservancy projects and the operation and management regulations of water conservancy projects, and a standard database of hidden dangers that need to be inspected is established;

[0066] Specifically, the hidden dangers that need to be checked include engineering hazards such as gate leakage, deformation, jamming, and rusting of metal structures; and irregular behaviors such as dumping of soil, stones, slag, and garbage within the management area, creating land around reservoirs, and blasting fish in the reservoir area.

[0067] The hidden dangers in the hidden danger standard database are divided into two categories: engineering hidden dangers and irregular behaviors according to whether the objects involved are objects or people. Each type of hidden danger is also divided into two levels: general hidden dangers and serious hidden dangers according to the harm and difficulty of rectification. The general hidden dangers are those that can be immediately rectified and eliminated after discovery, and the remaining hidden dangers are major hidden dangers. The hidden danger standard database also has corresponding hidden danger handling countermeasures and solutions established for hidden danger categories and levels based on water conservancy project management knowledge and historical experience in safety handling.

[0068] In the specific implementation of S1, the inspection area of ​​the water conservancy project is gridded according to the monitoring range of each camera, wherein each grid has at least one camera and a voice broadcasting device, and at least four feature points are set within the monitoring range of each camera and directly located on the electronic map of the inspection area;

[0069] Specifically, refer to Figure 2 , deploy camera A in a patrol area V i (i=1,2,...N),establish a monitoring network. Mainly use the existing video surveillance. If the video is insufficient, add more videos. Determine the monitoring range of each camera according to the video parameters to ensure full coverage of the area to be inspected, and encode the probe. Wireless broadcasting is also installed at the camera to form an integrated monitoring and broadcasting network. The monitoring area is gridded according to the camera monitoring range. i (i=1, 2, ...M), obtain the grid probe coding information to ensure that at least one probe in a grid can fully cover the grid to ensure that there are no blind spots or omissions in the inspection.

[0070] At least four feature points are set and marked within each video range. Feature points can be selected from objects with obvious characteristics and not easily damaged, such as observation piers, or feature objects can be re-arranged. Feature points are distributed in a regular polygonal shape as much as possible. At the same time, the geographic coordinates and position serial numbers of the feature points are obtained and bound to the probe and its related parameters, and uploaded to the server to directly locate them on the electronic map of the patrol area for subsequent image stitching, automatic registration, management and comparison.

[0071] In this example, existing non-destructive features (such as measurement and observation piers, etc.) are selected as feature points within the camera imaging area, or markers with obvious features are cast as feature points. The feature points cannot be located in a straight line, and the spatial distribution should be relatively uniform.

[0072] In a specific implementation of S2, relevant parameters of the feature point and the corresponding camera are received and the two are bound;

[0073] Specifically, the video parameters (such as probe geometric position, focal length, inclination, azimuth, etc.) are adjusted to form the patrol range of the probe. At the same time, the corresponding video parameters and the corresponding patrol range are recorded and bound to the feature point serial numbers set within the range. This facilitates the rapid determination of the patrol range and sample selection within the range.

[0074] In a specific implementation of S3, a video image captured by a camera and video parameters and geographic location of the camera are received; this step may include the following sub-steps:

[0075] S31: Collect images according to inspection requirements. During the non-flood season, refer to the daily inspection frequency requirements and patrol the grids to ensure full coverage within the inspection cycle. During the flood season, the grids where key inspection objects are located can be encrypted for identification, and higher-resolution images within the inspection cycle can be selected for identification. During the operation of facilities and equipment, uninterrupted sampling can be carried out for real-time or quasi-real-time sampling and identification.

[0076] Specifically, in the non-flood season, images should be collected at least once every two days for earth dams and at least once every three days for concrete face rockfill dams; in the flood season, images should be collected at least once every day for earth dams and at least once every two days for concrete face rockfill dams.

[0077] This combination of regular and uninterrupted can greatly reduce the computing power requirements of the server and improve the efficiency of intelligent recognition. With the improvement of computing power and recognition ability, data collection can be transformed from regular to real-time, realizing all-weather inspection and supervision.

[0078] Preferably, an intelligent recognition camera is directly used, and the recognition information is directly transmitted to the server through edge computing to initiate an early warning.

[0079] Preferably, image information can also be obtained through drones or the like.

[0080] S32: Acquire camera parameters while capturing video images.

[0081] In the specific implementation of S4, a sample library required for video recognition is established; this step may include the following sub-steps:

[0082] S41: Select the video images received in the grid according to the inspection object requirements, and cut them into images containing the engineering facilities to be inspected and the set feature points according to certain rules, and mark the hidden danger labels, time labels and camera parameters;

[0083] S42: geometrically correcting and registering the selected series of pictures and geographically positioning them according to the feature points, marking the samples of normal and abnormal categories, and marking the pictures of engineering hazards and irregular behaviors in the abnormal samples.

[0084] In the specific implementation of S5, according to the hidden danger standard database and sample library, video recognition and spatial fusion as well as dynamic evolution monitoring analysis calculation are performed on the video image to obtain hidden danger identification results; this step may include the following sub-steps:

[0085] S51, selecting images with the same parameters in the sample library according to the acquired camera parameters, and performing registration according to preset feature points;

[0086] S52, automatically selecting a normal picture of the same period in history or the last normal picture with the same parameters in the same geographic space area from the sample library as a comparison picture according to the video geographic location on the electronic map, and at the same time correcting and registering the collected image according to the feature points on the image; using the background difference method or the frame difference method to inspect and detect hidden dangers and illegal behaviors, and obtaining the changed area 1 as the preliminary recognition result;

[0087] Specifically, according to the video parameters, a normal picture with similar environmental conditions in the same period or a normal picture from the previous time is selected in the sample library as the background 正常 (x, y), the real-time image f t (x, y) and the normal image f 正常 The change between (x, y) can be represented by a binary difference image D(x, y):

[0088]

[0089] T is a given threshold. When the difference image is greater than T, it is considered to have changed. The value of T can be determined based on the image difference f t (x,y)-f 正常 2-3 times the standard deviation of (x, y) is used as the threshold. In practical applications, 2.5 times the standard deviation is used as the threshold. The image difference is obtained by background subtraction, and the abnormality from the normal state is identified to obtain the change area 1;

[0090] S53, using an image recognition algorithm based on deep learning, to detect abnormal targets and intelligently identify images on grid images, to build a deep learning model suitable for identifying hidden dangers, to continuously train engineering hidden dangers and irregular behaviors of personnel, to identify different images, and to synchronously recommend suspected problems based on image recognition similarity. During the training process, it is necessary to continuously collect and mark relevant samples; the result with the highest similarity is confirmed as the recognition result, and the other recognition results are sorted from high to low according to the similarity, and are recommended as suspected problems for association, and the change area 2 is obtained;

[0091] S54, perform intersection, adjacency, and inclusion spatial relationship judgment and spatial fusion calculation on the change area 1 and the change area 2. When they are intersectional or adjacent, the sum of the two is taken as the change area. When they are included, the larger area is taken as the change area, which is used as the marked area for category intelligent recognition judgment. By fusing area 1 and area 2, the recognition rate of the change area can be improved to ensure that the detection of the change area is not missed. Figure 3 .

[0092] S55, the server stores an image similarity algorithm based on deep learning, calculates the morphological and characteristic similarities between the above-mentioned changed area and the abnormal samples in the sample library, sorts them from high to low in terms of similarity, and confirms the result with the highest similarity as the recognition result, obtains the hidden danger category, and at the same time, fuzzy matches the corresponding hidden danger or irregular behavior description in the hidden danger standard database according to the image recognition category as a suspected problem;

[0093] Specifically, for example, to identify gate leakage, the identified change areas are compared with abnormal samples in the sample library one by one to find the most similar samples. At the same time, the hidden danger category features marked on the samples are obtained to identify gate leakage.

[0094] S56, writing the identified suspected problems into the corresponding suspected problem database according to the two categories of engineering hidden dangers and irregular behaviors of fuzzy matching, and marking the location of the suspected problem, probe number, grid number, picture time, and suspected problem category on the picture;

[0095] Preferably, experts can be organized to conduct comprehensive analysis and judgment from an expert perspective, combine monitoring with instrument monitoring factors, such as deformation, seepage, stress-strain, upstream and downstream water levels, rainfall in the reservoir area and other environmental quantities, conduct more professional inspections and identifications, and improve the accuracy of hidden danger identification.

[0096] S57, according to the classified suspected problems, fuzzy matching the hidden danger levels in the hidden danger standard database to obtain hidden danger classification information;

[0097] S58, for general hidden dangers, through uninterrupted and encrypted inspections, dynamic evolution monitoring and analysis are carried out. Through the inter-frame difference method, the difference between continuous multiple frames in the image sequence is used to monitor and analyze the evolution of hidden dangers. The continuous real-time image f t-2 (x, y), f t-1 (x, y), f t (x, y) is subtracted to obtain the difference image D t-1 (x,y)=f t-1 (x, y)-f t-2 (x,y),D t (x,y)=f t (x, y)-f t-1 (x, y), for the difference image D t-1 (x,y),D t (x, y) divided by the corresponding time interval gives the rate of change, V t (x,y)=D t (x,y) / ΔT1,V t-1 (x,y)=D t-1(x,y) / ΔT2, where ΔT1 is the time interval between the image at time t and the image at time t - 1, and ΔT2 is the time interval between the image at time t - 1 and the image at time t - 2. For the change rate V t and V t-1 perform a difference calculation again to obtain ΔV(x,y) = V t (x,y) - V t-1 (x,y). When ΔV(x,y) >= T (T is the threshold, simply T can be taken as 0), it is considered that the hidden danger is evolving and developing. When ΔV(x,y) < T, it is considered that the hidden danger is gradually being eliminated. If the hidden danger is constantly evolving, modify the hidden danger level to a serious hidden danger.

[0098] S59. According to the classified and graded hidden dangers, fuzzy match the corresponding countermeasures and solutions in the hidden danger standard database to generate a warning message. The warning message includes the hidden danger category, level, the grid (probe) to which it belongs, and the solution.

[0099] In the specific implementation of S6, for the engineering hidden dangers identified in the hidden danger recognition result, generate a first warning message and send the first warning message to the client of the management responsible person; for the non-standard behaviors identified in the hidden danger recognition result, generate a second warning message and convert the second warning message into voice and send it to the voice broadcast device for voice broadcast warning.

[0100] Specifically, for the identified engineering hidden dangers, actively push the warning message to the management responsible person in real time through means such as text messages, WeChat, and DingTalk; for the warning message of the identified non-standard behaviors, automatically convert it into voice in real time and conduct on-site real-time automatic warning through wireless broadcasting. In this way, the hidden danger information can be released in a timely manner and the non-standard behaviors can be stopped in a timely manner.

[0101] It may further include S7: Process the hidden danger according to the solution in the warning message, and at the same time transfer the confirmed problems, corresponding pictures, and actual treatment measures into the sample library.

[0102] Specifically, the management responsible person conducts on-site review of the first warning message and processes the hidden danger with reference to the provided solution; for the second warning message, directly notify the parties on-site through the broadcast to stop the non-standard behavior. Eliminate the hidden dangers and non-standard behaviors in the initial stage of the accident in a timely manner.

[0103] Corresponding to the foregoing embodiment of the intelligent inspection method for water conservancy projects based on video monitoring, the present application also provides an embodiment of an intelligent inspection device for water conservancy projects based on video monitoring.

[0104] Figure 4 is a block diagram of an intelligent inspection device for water conservancy projects based on video monitoring shown according to an exemplary embodiment. Refer to Figure 4The device includes a first establishing module 11, a gridding module 12, a first receiving module 13, a second receiving module 14, a first establishing module 15, an identifying module 16, and an early warning module 17.

[0105] The first establishment module 11 is used to sort out the water conservancy project inspection list according to the water conservancy project supervision and inspection method and the water conservancy project operation and management regulations, and establish a standard database of hidden dangers that need to be inspected;

[0106] The gridding module 12 is used to grid the inspection area of ​​the water conservancy project according to the monitoring range of each camera, wherein each grid has at least one camera and a voice broadcasting device, and at least four feature points are set within the monitoring range of each camera and directly located on the electronic map of the inspection area;

[0107] A first receiving module 13 is used to receive the feature point and the relevant parameters of the corresponding camera, and bind the two;

[0108] The second receiving module 14 is used to receive the video image captured by the camera and the video parameters and geographic location of the camera;

[0109] The second establishing module 15 is used to establish a sample library required for video recognition;

[0110] The recognition module 16 is used to perform video recognition on the video image according to the hidden danger standard database and the sample library to obtain a hidden danger recognition result;

[0111] The early warning module 17 is used to generate a first early warning message for the engineering hidden dangers identified in the hidden danger identification results, and send the first early warning message to the client of the person in charge of management; generate a second early warning message for the irregular behavior identified in the hidden danger identification results, and convert the second early warning message into voice and send it to the voice broadcasting device for voice broadcasting early warning.

[0112] It may also include: a hidden danger elimination module, which is used to handle hidden dangers according to the solutions in the early warning information, and at the same time transfer the confirmed problems and corresponding pictures and actual handling measures into the sample library.

[0113] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0114] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0115] Correspondingly, the present application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned intelligent inspection method of water conservancy projects based on video surveillance.

[0116] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, characterized in that when the instructions are executed by a processor, the above-mentioned intelligent inspection method of water conservancy projects based on video monitoring is implemented.

[0117] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0118] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A water conservancy project intelligent inspection method based on video monitoring, characterized in that: include: S0: Sort out the inspection list of water conservancy projects according to the supervision and inspection methods of water conservancy projects and the operation and management regulations of water conservancy projects, and establish a standard database of hidden dangers that need to be inspected; S1: The water conservancy project inspection area is gridded according to the monitoring range of each camera, wherein each grid has at least one camera and a voice broadcasting device, and at least four feature points are set within the monitoring range of each camera and directly located on the electronic map of the inspection area; S2: receiving the relevant parameters of the feature point and the corresponding camera, and binding the two; S3: receiving the video image and the video parameters of the camera, and performing geolocation on the image; S4: Establish the sample library required for video recognition; S5: performing video recognition and spatial fusion as well as dynamic evolution monitoring and analysis calculation on the video image according to the hidden danger standard database and sample library to obtain hidden danger identification results; S6: generating first warning information according to the engineering hidden dangers identified in the hidden danger identification result, and sending the first warning information to the client of the management responsible person; For the irregular behavior identified in the hidden danger identification result, a second warning information is generated, and the second warning information is converted into voice and sent to the voice broadcasting device for voice broadcast warning; According to the hidden danger standard database and sample library, the acquired video images are subjected to video recognition, and the recognition results of different methods are subjected to spatial fusion and dynamic evolution monitoring, analysis and calculation to obtain hidden danger recognition results, including: S51, selecting images with the same parameters in the sample library according to the collected camera parameters, and performing registration and geolocation according to preset feature points; S52, automatically selecting a normal picture of the same period in history or the last normal picture with the same parameters in the same geographic space area from the sample library according to the geographic location of the video on the electronic map as a comparison picture, using a background difference method or a frame difference method to perform inspection and detection of hidden dangers and illegal behaviors, and obtaining a changed area 1 as a preliminary recognition result; S53, using an image recognition algorithm based on deep learning, to detect abnormal targets and intelligently identify images on grid images, to build a deep learning model suitable for identifying hidden dangers, to continuously train engineering hidden dangers and irregular behaviors of personnel, to identify different images, and to synchronously recommend suspected problems based on image recognition similarity. During the training process, it is necessary to continuously collect and mark relevant samples; the result with the highest similarity is confirmed as the recognition result, and the other recognition results are sorted from high to low according to the similarity, and are recommended as suspected problems for association, and the change area 2 is obtained; S54, performing intersection, adjacency, and inclusion spatial relationship judgment and spatial fusion analysis calculation on the change region 1 and the change region 2. When they are intersection or adjacency, the sum of the two is taken as the change region. When they are inclusion, the larger area is taken as the change region, which is used as the marked area for category intelligent recognition judgment; S55, performing morphological and feature comparison between the above-mentioned changed area and abnormal samples in the sample library to obtain the hidden danger category, and at the same time, fuzzy matching the corresponding hidden danger or irregular behavior description in the hidden danger standard database according to the category of image recognition as a suspected problem; S56, writing the identified suspected problems into the corresponding suspected problem database according to the two categories of engineering hidden dangers and irregular behaviors of fuzzy matching, and providing the location of the suspected problem, probe number, grid number, picture time, and suspected problem category on the picture; S57, according to the classified suspected problems, fuzzy matching the general hidden dangers and serious hidden dangers in the hidden danger standard database to obtain hidden danger classification information; S58. General hidden dangers are identified through continuous encrypted inspections, and dynamic evolution monitoring and analysis are carried out. Through the inter-frame difference method, the differences in multiple consecutive frames of images in the image sequence are used to monitor and analyze the evolution of hidden dangers. The continuous real-time images f t-2 (x, y), f t-1 (x, y), f t (x, y) are subtracted to obtain the difference image D t-1 (x,y) = f t-1 (x, y) - f t-2 (x, y), D t (x,y) = f t (x, y) - f t-1 (x, y). For the difference image D t-1 (x,y), D t (x,y) is divided by the corresponding time interval to obtain the change rate, V t (x,y) = D t (x,y) / ΔT1, V t-1 (x,y) = D t-1 (x,y) / ΔT2. ΔT1 is the time interval between the image at time t and the image at time t - 1, and ΔT2 is the time interval between the image at time t - 1 and the image at time t - 2. For the change rate V t and V t-1 are subtracted again to obtain ΔV(x,y) = V t (x,y) - V t-1 (x,y). When ΔV(x,y) >= T, where T is the threshold, it is considered that the hidden danger is evolving and developing. When ΔV(x,y) < T, it is considered that the hidden danger is gradually being eliminated. If the hidden danger is continuously evolving, the hidden danger level is modified to a serious hidden danger.

2. The method according to claim 1, characterized in that Also includes: S7, handle hidden dangers according to the solutions in the early warning information, and transfer the confirmed problems, corresponding pictures, and actual treatment measures to the sample library.

3. The method according to claim 1, characterized in that The hidden dangers in the hidden danger standard database are divided into two categories: engineering hidden dangers and irregular behaviors according to whether the objects involved are objects or people. Each type of hidden danger is also divided into two levels: general hidden dangers and serious hidden dangers according to the harm and difficulty of rectification. The general hidden dangers are those that can be immediately rectified and eliminated after discovery, and the remaining hidden dangers are major hidden dangers. The hidden danger standard database also has corresponding hidden danger handling countermeasures and solutions established for hidden danger categories and levels based on water conservancy project management knowledge and historical experience in safety handling.

4. The method according to claim 1, characterized in that: Receive the video images and video parameters collected by the camera, including: S31: Image acquisition is performed according to inspection requirements. During the non-flood season, grids are patrolled in accordance with the daily inspection frequency requirements to ensure full coverage within the inspection cycle. During the flood season, grids where key inspection objects are located can be encrypted for identification, and images with higher resolution within the inspection cycle can be selected for identification. During the operation of facilities and equipment, uninterrupted sampling can be performed for real-time or quasi-real-time sampling and identification. S32: Acquire camera parameters while capturing video images.

5. The method according to claim 1, characterized in that Build the sample library required for video recognition, including: S41: Select the video images received in the grid according to the inspection object requirements, and cut them into images containing the engineering facilities to be inspected and the set feature points according to certain rules, and mark the hidden danger labels, time labels and camera parameters; S42: geometrically correct and align the selected series of pictures according to the set feature points, directly geolocate the video, mark the samples of normal and abnormal categories, and mark the pictures of engineering hazards and irregular behaviors in the abnormal samples.

6. The method according to claim 1, characterized in that Also includes: S59, generating warning information according to the corresponding treatment countermeasures and solutions in the hidden danger standard database according to the fuzzy matching of the classified and graded hidden dangers, wherein the warning information includes the hidden danger category, level, the grid to which the probe belongs, and the solution.

7. An intelligent inspection device for water conservancy projects based on video monitoring, characterized in that: Applied to the server, including: The first module is used to sort out the inspection list of water conservancy projects according to the water conservancy project supervision and inspection methods and water conservancy project operation and management regulations, and establish a standard database of hidden dangers that need to be inspected; A gridding module is used to grid the inspection area of ​​the water conservancy project according to the monitoring range of each camera, wherein each grid has at least one camera and a voice broadcasting device, and at least four feature points are set within the monitoring range of each camera and directly located on the electronic map of the inspection area; A first receiving module is used to receive the relevant parameters of the feature point and the corresponding camera, and bind the two; The second receiving module is used to receive the video image collected by the camera and the video parameters and geographic location of the camera; The second building module is used to build a sample library required for video recognition; An identification module is used to perform video identification and spatial fusion as well as dynamic evolution monitoring and analysis calculation on the video image according to the hidden danger standard database and sample library to obtain hidden danger identification results; An early warning module is used to generate a first early warning message according to the engineering hidden dangers identified in the hidden danger identification results, and send the first early warning message to the client of the person in charge of management; generate a second early warning message for the irregular behavior identified in the hidden danger identification results, and convert the second early warning message into voice and send it to the voice broadcasting device for voice broadcasting early warning; According to the hidden danger standard database and sample library, the acquired video images are subjected to video recognition, and the recognition results of different methods are subjected to spatial fusion and dynamic evolution monitoring, analysis and calculation to obtain hidden danger recognition results, including: S51, selecting images with the same parameters in the sample library according to the collected camera parameters, and performing registration and geolocation according to preset feature points; S52, automatically selecting a normal picture of the same period in history or the last normal picture with the same parameters in the same geographic space area from the sample library according to the geographic location of the video on the electronic map as a comparison picture, using a background difference method or a frame difference method to perform inspection and detection of hidden dangers and illegal behaviors, and obtaining a changed area 1 as a preliminary recognition result; S53, using an image recognition algorithm based on deep learning, to detect abnormal targets and intelligently identify images on grid images, to build a deep learning model suitable for identifying hidden dangers, to continuously train engineering hidden dangers and irregular behaviors of personnel, to identify different images, and to synchronously recommend suspected problems based on image recognition similarity. During the training process, it is necessary to continuously collect and mark relevant samples; the result with the highest similarity is confirmed as the recognition result, and the other recognition results are sorted from high to low according to the similarity, and are recommended as suspected problems for association, and the change area 2 is obtained; S54, performing intersection, adjacency, and inclusion spatial relationship judgment and spatial fusion analysis calculation on the change region 1 and the change region 2. When they are intersection or adjacency, the sum of the two is taken as the change region. When they are inclusion, the larger area is taken as the change region, which is used as the marked area for category intelligent recognition judgment; S55, performing morphological and feature comparison between the above-mentioned changed area and abnormal samples in the sample library to obtain the hidden danger category, and at the same time, fuzzy matching the corresponding hidden danger or irregular behavior description in the hidden danger standard database according to the category of image recognition as a suspected problem; S56, writing the identified suspected problems into the corresponding suspected problem database according to the two categories of engineering hidden dangers and irregular behaviors of fuzzy matching, and providing the location of the suspected problem, probe number, grid number, picture time, and suspected problem category on the picture; S57, according to the classified suspected problems, fuzzy matching the general hidden dangers and serious hidden dangers in the hidden danger standard database to obtain hidden danger classification information; S58. General hidden dangers are identified through continuous encrypted inspections, and dynamic evolution monitoring and analysis are carried out. Through the inter-frame difference method, the differences in multiple consecutive frames of images in the image sequence are used to monitor and analyze the evolution of hidden dangers. The continuous real-time images f t-2 (x, y), f t-1 (x, y), f t (x, y) are subtracted to obtain the difference image D t-1 (x,y) = f t-1 (x, y) - f t-2 (x, y), D t (x,y) = f t (x, y) - f t-1 (x, y). For the difference image D t-1 (x,y), D t (x,y) is divided by the corresponding time interval to obtain the change rate, V t (x,y) = D t (x,y) / ΔT1, V t-1 (x,y) = D t-1 (x,y) / ΔT2. ΔT1 is the time interval between the image at time t and the image at time t - 1, and ΔT2 is the time interval between the image at time t - 1 and the image at time t - 2. For the change rate V t and V t-1 are subtracted again to obtain ΔV(x,y) = V t (x,y) - V t-1 (x,y). When ΔV(x,y) >= T, where T is the threshold, it is considered that the hidden danger is evolving and developing. When ΔV(x,y) < T, it is considered that the hidden danger is gradually eliminated. If the hidden danger is continuously evolving, the hidden danger level is modified to a serious hidden danger.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images