Intelligent supervision method and system for standardized operation of water gate and related equipment

Abstract

The invention relates to the technical field of water conservancy project standardization management, in particular to a water gate standardization operation intelligent supervision method and system and related equipment, and the method comprises the steps: obtaining a water gate operation video of a water gate operator; performing content identification on the sluice operation video to obtain operation behavior information; analyzing the difference between the operation behavior information and standard behavior information to obtain an operation abnormal value; and under the condition that the operation abnormal value is greater than a preset abnormal threshold value, generating alarm information for indicating abnormal operation of the water gate. According to the invention, the difference between the actual operation behavior and the standardized operation can be analyzed by identifying the operation picture of the water gate operator, and an early warning is given out in time when the difference degree between the actual operation behavior and the standardized operation is identified to be too large, so that the supervision effect on the water gate operation is improved, and the operation safety of the water gate is ensured.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of standardization management of water conservancy projects, in particular to a water gate standardization operation intelligent supervision method and system and related equipment. BACKGROUND

[0002] As an important part of water conservancy projects, water gates bear the key functions of regulating water level, controlling water flow, flood control and drainage, etc. The safe operation of water gates is directly related to the realization of important functions such as flood control and water supply, and the safe operation of water gates is directly related to the safety of people's lives and property and social stability. In order to ensure the safety of water gates, the water conservancy industry has developed strict operation specifications and management standards, requiring water gate management units to establish and improve management systems, strictly implement safety operation procedures, strengthen standardization management, and ensure project safety.

[0003] In actual operation, as the main body of engineering operation and management activities, people have high subjective initiative and uncertainty, can flexibly respond to various random disturbances in complex environments, have characteristics of wide activity range and many autonomous behaviors. Due to negligence, non-standard operation, etc., irregular operation often occurs, which threatens the safe operation of water gates. The behavior supervision of people in the operation and management process is mainly on paper and mainly through management systems and operation manuals for specification and constraint. In a sense, it depends on the self-consciousness of people in implementing the system, and the daily supervision and inspection has great randomness and uncertainty. For example, in the gate operation process, whether two people are on duty, whether they have a certificate, whether they operate according to the operation manual, and whether they complete the water release warning, etc. There are certain difficulties in supervision, and there are problems of untimely and unintelligent water gate operation supervision. However, the traditional water gate operation supervision mainly relies on manual methods, including on-site inspection, manual recording, and experience judgment, etc. There are low efficiency, poor real-time performance, and other shortcomings.

[0004] Firstly, manual inspection often has limited coverage and lag, and it is difficult to monitor all operation links in real time all day long, and there are problems such as low efficiency, strong subjectivity, and easy to be affected by human factors, etc. It is difficult to find operation deviation in real time, especially in complex operation processes (such as multi-stage opening and closing, linkage operation), the error rate is high, it is difficult to cover all operation details comprehensively, once irregular operation or abnormal situation occurs, it is difficult to find and correct in time, and potential safety hazards cannot be found in time. At the same time, through manual inspection, it is inevitable to cause a certain degree of interference to the behavior of the observed person, and there may be a situation of non-standard daily operation and inspection.

[0005] Secondly, the self-determination behavior characteristics of human beings correspondingly lead to some behaviors that cannot be supervised. In some management scenarios with high standardization requirements, spontaneous rule violation behaviors of personnel may cause production quality problems, or even cause irreparable catastrophic accidents. For example, the operation process of the sluice gate requires high compliance of personnel behavior, and needs to be operated according to the operation manual specification, otherwise safety accidents may occur. Whether the operation is performed according to the procedure in the daily operation process of the sluice gate depends on the confirmation of the guardian by checking the operation ticket, and the authenticity and reliability depend on the quality of the personnel. The operation record depends on manual filling, and the phenomena of formalism, fraud and the like are often found in the standardized supervision process.

[0006] Thirdly, the existing video monitoring always depends on special visual monitoring, and is mostly for post playback, and lacks intelligent analysis capability of the video content, and cannot judge whether the operator operates according to the specification in real time.

[0007] Fourthly, the sluice gate automatic monitoring system only monitors a single parameter (such as current, water level, torque), and cannot evaluate the compliance of the operation behavior, such as incorrect operation sequence and excessive action amplitude. The system lacks intelligent identification of the whole operation process, and cannot effectively warn the rule violation operation and potential risks, such as sluice gate jamming and foreign matter interference.

[0008] That is, the existing technology has poor monitoring effect on the sluice gate operation. SUMMARY

[0009] The purpose of the present disclosure is to provide a sluice gate standardized operation intelligent monitoring method, system and related equipment, which solves the technical problem of poor monitoring effect of the existing technology on the sluice gate operation.

[0010] In a first aspect, the present disclosure provides a sluice gate standardized operation intelligent monitoring method, which comprises: obtaining a sluice gate operation video of a sluice gate operator; performing content recognition on the sluice gate operation video to obtain operation behavior information, the operation behavior information being used to represent actual operation behavior of the sluice gate operator in the sluice gate operation video; analyzing the difference between the operation behavior information and standard behavior information to obtain an operation abnormal value, wherein the operation abnormal value is used to represent the degree of mismatch between the actual operation behavior of the sluice gate operator and the standardized operation; generating alarm information indicating sluice gate operation abnormality in the case that the operation abnormal value is greater than a preset abnormal threshold.

[0011] In one embodiment, the content recognition of the sluice gate operation video to obtain the operation behavior information comprises: frame the water gate operation video to obtain a plurality of operation video frames; In the plurality of operation video frames, a feature is extracted from each operation video frame to obtain a plurality of feature data, wherein the plurality of feature data correspond one-to-one to the plurality of operation video frames; Based on the collection time of each operation video frame, the plurality of feature data are combined into a feature sequence in the order of collection time from early to late; The feature sequence is subjected to timing analysis to obtain the operation behavior information.

[0012] In one embodiment, the timing analysis of the feature sequence to obtain the operation behavior information comprises: The feature sequence is subjected to timing analysis to obtain an analysis result, wherein the analysis result includes a plurality of behavior similarity values, the plurality of behavior similarity values correspond one-to-one to a plurality of preset behaviors, and the behavior similarity value is used to represent the similarity between the actual operation behavior and the corresponding preset behavior; In the plurality of behavior similarity values included in the analysis result, the maximum behavior similarity value is determined as a target behavior similarity value; The operation behavior information is formed according to the preset behavior corresponding to the target behavior similarity value.

[0013] In one embodiment, the standard behavior information is used to indicate the operation behavior to be completed and the operation sequence of the water gate operation process is the smallest; The difference between the operation behavior information and the standard behavior information is analyzed to obtain an operation abnormal value, comprising: In the case where the preset behavior indicated by the operation behavior information is different from the operation behavior indicated by the standard behavior information, a preset first value is determined as the operation abnormal value; In the case where the preset behavior indicated by the operation behavior information is the same as the operation behavior indicated by the standard behavior information, a preset second value is determined as the operation abnormal value; Wherein, the first value is greater than the abnormal threshold value, and the second value is less than the abnormal threshold value.

[0014] In one embodiment, after obtaining the water gate operation video of the water gate operator, the method further comprises: Target detection is performed on the water gate operation video to obtain a video person number, wherein the video person number is used to represent the number of water gate operators included in the water gate operation video; In the case where the video person number is less than the water gate operation standard number, an alarm information indicating water gate operation abnormality is generated.

[0015] In an embodiment, after the target detection on the water gate operation video to obtain the video person number, the method further comprises: In a case where the video person number is greater than or equal to the water gate operation standard person number, distinguishing, in the water gate operation personnel included in the water gate operation video, an actual operation personnel from a guardian personnel, wherein the actual operation personnel is a water gate operation personnel closest to a corresponding to-be-operated target among the water gate operation personnel, and the guardian personnel is a water gate operation personnel other than the actual operation personnel among the water gate operation personnel; calculating a guardian time length ratio between a guardian time period of the guardian personnel and a video collection time period of the water gate operation video, to obtain a guardian time length ratio, wherein the guardian time period is used to indicate a time period during which a line-of-sight area of the corresponding guardian personnel includes the actual operation personnel; in a case where the guardian time length ratio is less than a preset time length ratio threshold, generating alarm information indicating a water gate operation abnormality.

[0016] In an embodiment, after the generation of the alarm information indicating the water gate operation abnormality, the method further comprises: operation locking on each to-be-operated object in a water gate operation scene, and visual display of a water conservancy accident associated with the actual operation behavior.

[0017] In a second aspect, an embodiment of the present application further provides a water gate standardized operation intelligent supervision system, the system comprising: a video acquisition module configured to acquire a water gate operation video of a water gate operation personnel; a video recognition module configured to perform content recognition on the water gate operation video to obtain operation behavior information, the operation behavior information being used to represent actual operation behavior presented by the water gate operation personnel in the water gate operation video; a behavior analysis module configured to analyze a difference between the operation behavior information and standard behavior information to obtain an operation abnormality value, wherein the operation abnormality value is used to represent a degree to which the actual operation behavior of the water gate operation personnel does not match the standardized operation; an alarm module configured to, in a case where the operation abnormality value is greater than a preset abnormality threshold, generate alarm information indicating a water gate operation abnormality.

[0018] In a third aspect, an embodiment of the present application further provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the water gate standardized operation intelligent supervision method described above.

[0019] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the water gate standardized operation intelligent supervision method.

[0020] In a fifth aspect, the present disclosure provides a computer program product comprising computer instructions, which, when executed by a processor, implement the steps of the water gate standardized operation intelligent supervision method.

[0021] In the embodiments of the present application, the content of the water gate operation video of the water gate operator is identified to identify the actual operation behavior of the water gate operator in the water gate operation video, and the difference between the actual operation behavior and the standardized operation indicated by the standard behavior information is analyzed to determine the degree of mismatch between the actual operation behavior of the water gate operator and the standardized operation, and a warning is issued in time when the difference between the actual operation behavior and the standardized operation is too large, thereby improving the supervision effect of the water gate operation and ensuring the safety of the water gate operation. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a water gate standardized operation intelligent supervision method provided by the embodiments of the present application; Figure 2 is a structural diagram of a water gate standardized operation intelligent supervision system provided by the embodiments of the present application; Figure 3 is a schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0024] The embodiments of the present application provide a water gate standardized operation intelligent supervision method, referring to Figure 1 , Figure 1 is a flowchart of a water gate standardized operation intelligent supervision method provided by the embodiments of the present application, as shown in Figure 1 , comprising the following steps: Step 101, obtaining a water gate operation video of a water gate operator.

[0025] In the present application, the water gate operator should be understood as: the relevant personnel going to the water gate to perform water gate operation (such as reading the data of water gate related instruments and recording, controlling the opening and closing of the water gate, monitoring the opening and closing control process of the water gate, etc.).

[0026] The water gate operation video is a video detected with a face periodically collected based on a plurality of high-definition cameras arranged on the water gate site. The plurality of high-definition cameras cover key areas such as the water gate, the water gate opening and closing mechanism, and the water gate operation platform. The duration of the water gate operation video can be 3 seconds, 30 seconds, 1 minute, etc.

[0027] In the application, in order to accurately capture the water gate operation behavior of the water gate operator at night or in bad weather conditions, an infrared thermal imager or a black light camera can be arranged on the water gate site.

[0028] Among them, the detection of the face in the video can be completed based on the improved YOLOv5 target detection model.

[0029] Step 102, content recognition is performed on the water gate operation video to obtain operation behavior information.

[0030] Among them, the operation behavior information is used to represent the actual operation behavior of the water gate operator in the water gate operation video.

[0031] Step 103, analyze the difference between the operation behavior information and the standard behavior information to obtain an operation abnormal value.

[0032] Among them, the operation abnormal value is used to represent the degree of mismatch between the actual operation behavior of the water gate operator and the standardized operation.

[0033] Step 104, in the case where the operation abnormal value is greater than a preset abnormal threshold, an alarm information indicating water gate operation abnormality is generated.

[0034] In the embodiments of the present application, the content of the water gate operation video of the water gate operator is recognized to identify the actual operation behavior of the water gate operator in the water gate operation video, and the difference between the actual operation behavior and the standardized operation indicated by the standard behavior information is analyzed to determine the degree of mismatch between the actual operation behavior of the water gate operator and the standardized operation, and a warning is issued in time when the difference between the actual operation behavior and the standardized operation is too large, thereby improving the supervision effect of the water gate operation and ensuring the safety of the water gate operation.

[0035] In some embodiments, the content recognition of the water gate operation video to obtain operation behavior information comprises: Frame extraction processing is performed on the water gate operation video to obtain a plurality of operation video frames; In the plurality of operation video frames, feature extraction is performed on each operation video frame to obtain a plurality of feature data, wherein the plurality of feature data corresponds one-to-one with the plurality of operation video frames; Based on the acquisition time of each video frame, multiple feature data are combined into a feature sequence in order of acquisition time from first to last. The feature sequence is subjected to time series analysis to obtain the operation behavior information.

[0036] Further, the step of performing time-series analysis on the feature sequence to obtain the operational behavior information includes: A time-series analysis is performed on the feature sequence to obtain analysis results, wherein the analysis results include multiple behavioral similarity values, each of which corresponds one-to-one with multiple preset behaviors, and the behavioral similarity values ​​are used to represent the degree of similarity between the actual operation behavior and the corresponding preset behavior; Among the multiple behavioral similarity values ​​included in the analysis results, the largest behavioral similarity value is determined as the target behavioral similarity value; The operation behavior information is generated based on the preset behavior corresponding to the similarity value of the target behavior.

[0037] The frame-sampling frequency used in the above frame-sampling process can be three frames per second or five frames per second.

[0038] In this invention, a hybrid model combining convolutional neural networks and long short-term memory networks (CNN-LSTM) is used to extract features from multiple consecutive operation video frames in order to accurately identify the complex operation behaviors of sluice gate operators.

[0039] The feature extraction operation on the operation video frames is performed using a CNN model (such as a ResNet model) within the hybrid model, while the temporal analysis operation is performed using an LSTM model within the same hybrid model. To improve recognition accuracy, an attention mechanism can be added after the LSTM model, allowing the hybrid model to focus more on key action frames. The hybrid model is trained using a large amount of labeled sluice gate operation video data, including training samples of normal operations and common violations, to ensure that the trained hybrid model has good classification performance for various sluice gate operation behaviors.

[0040] The above analysis results are specifically probability distribution vectors. ,in This indicates that the video clip belongs to the first... The probability (i.e., the similarity value of the aforementioned behaviors) of a class of preset behaviors (such as "inspecting equipment", "pressing the start button", "opening the gate", "observing the instruments", etc.), and satisfying the following conditions: .

[0041] The operation behavior information at least includes a preset behavior corresponding to a target behavior similarity value.

[0042] In some embodiments, the standard behavior information is used to indicate an operation behavior to be completed and having a minimum execution order in the sluice operation process. The analysis of the difference between the operation behavior information and the standard behavior information obtains an operation abnormal value, including: In a case where the preset behavior indicated by the operation behavior information is different from the operation behavior indicated by the standard behavior information, a preset first value is determined as the operation abnormal value. In a case where the preset behavior indicated by the operation behavior information is the same as the operation behavior indicated by the standard behavior information, a preset second value is determined as the operation abnormal value. The first value is greater than the abnormal threshold value, and the second value is less than the abnormal threshold value.

[0043] Exemplarily, the operation behaviors included in the sluice operation process are in sequence: [arriving at the scene ], [wearing protection ], [checking equipment ], [starting equipment ], [operating the gate ], [monitoring readings ], [turning off equipment ], and [operation feedback ].

[0044] In the above setting, by analyzing whether the actual operation behavior of the sluice operator is the same as the operation behavior to be completed and having a minimum execution order, it is determined whether the sluice operator performs the related sluice operation in accordance with the sluice operation process, which can timely find out the case that the sluice operator skips or omits part of the operation behaviors in the sluice operation process.

[0045] For example, if the operation behavior to be completed is and , the operation behavior to be completed and having a minimum execution order in the sluice operation process indicated by the standard behavior information is , i.e., checking equipment, and when it is found through video analysis that the preset behavior indicated by the operation behavior information is , i.e., starting equipment, it can be considered that the sluice operator skips or omits part of the operation behaviors in the sluice operation process, i.e., there is a violation operation, at this time, by issuing an alarm information indicating that the sluice operation is abnormal, the corresponding management personnel of the sluice operator can be reminded to timely discourage the sluice operator from violating the operation, thereby reducing the risk of water conservancy accidents caused by the violation operation.

[0046] Exemplarily, the first value, the abnormal threshold value and the second value can be 3, 2 and 1 in sequence.

[0047] In one embodiment, after the water gate operation video of the water gate operator is acquired, the method further comprises: performing target detection on the water gate operation video to obtain a video person number, wherein the video person number is used to represent the number of water gate operators included in the water gate operation video; in a case where the video person number is less than a water gate operation standard person number, generating alarm information indicating a water gate operation abnormality.

[0048] In the above setting, by detecting the number of persons and comparing the value with the standard number of persons, the case that at least two operators (one operator and another monitor) need to be configured in the water gate operation site is adapted, and whether the actual water gate operation meets the requirements of the standard water gate operation specification is assisted to be judged by the number of operators.

[0049] In the present application, the water gate operation standard person number can be set to 2.

[0050] Further, after the target detection on the water gate operation video to obtain the video person number, the method further comprises: in a case where the video person number is greater than or equal to the water gate operation standard person number, distinguishing an actual operator from a monitor among the plurality of water gate operators included in the water gate operation video, wherein the actual operator is a water gate operator closest to a corresponding target to be operated among the plurality of water gate operators, and the monitor is a water gate operator other than the actual operator among the plurality of water gate operators; calculating a monitoring time length ratio between a monitoring time period of the monitor and a video collection time period of the water gate operation video to obtain a monitoring time length ratio, wherein the monitoring time period is used to indicate a time period during which a line-of-sight area of the corresponding monitor includes the actual operator; in a case where the monitoring time length ratio is less than a preset time length ratio threshold value (which can be set to 0.8 based on experience), generating alarm information indicating a water gate operation abnormality.

[0051] The target to be operated can be understood as a target object corresponding to an operation behavior to be completed and having a smallest execution order in a water gate operation process. For example, in a case where the operation behavior to be completed and having the smallest execution order in the water gate operation process is to wear protective clothing , the target to be operated is a device rack on which the protective clothing is placed; and in a case where the operation behavior to be completed and having the smallest execution order in the water gate operation process is to start a device , the target to be operated is a console or an operation table of the water gate device.

[0052] In the application, after multiple sluice operators are recognized in the sluice operation video based on target detection, the pixel distance between the center of the video pixel area corresponding to each sluice operator and the center of the video pixel area corresponding to the target to be operated is compared to serve as the interval distance between the sluice operator and the corresponding target to be operated.

[0053] It is to be explained that in the case where there are multiple guardians, the corresponding guardianship duration of each guardian is obtained, and the longest guardianship duration is used to calculate the guardianship duration.

[0054] In the present application, the determination process of the line-of-sight area of the guardian is as follows: Based on the design drawings of the sluice and the field surveying data, a three-dimensional digital twin model of the sluice is established on a virtual simulation platform, including the gate, the hoist, the control room equipment and the surrounding environment, etc. The intrinsic and extrinsic information of the camera is embedded in the digital twin model to determine the position and angle of view of each camera in the virtual scene, realizing the alignment of the virtual camera and the real camera. Through video stream driving, the real-time video frame is mapped to the corresponding angle of view of the digital twin model to form a video twin environment that runs synchronously with the physical sluice.

[0055] According to the CAD drawings of the sluice and the field measurement data, a three-dimensional virtual model of the sluice is established in a digital twin platform (such as Unity 3D or GIS simulation environment). The model includes the operating table of the control room, the gate and its hoist mechanical structure, the upstream and downstream river channel topography, etc. The corresponding position and orientation of each camera in the three-dimensional model are determined through camera calibration, and a virtual camera is added to correspond to a real camera. The system projects the real-time video frame mapping module into the viewport of the virtual camera, so that the corresponding angle of view of the digital twin model presents a picture consistent with the real scene.

[0056] Then the video frame is subjected to face detection to obtain the 2D key points of the guardian's face in the corresponding video frame, and the key points on the face 3D model of the standard face are matched with the 2D key points of the guardian's face to estimate the rotation vector (corresponding to Euler angle) and translation vector of the guardian's head, from which the orientation of the guardian's head (represented by a vector) can be obtained.

[0057] The video frame is subjected to portrait recognition and human body posture recognition to determine the portrait position and skeletal point sequence (indicating the posture of the guardian) of the guardian in the corresponding video frame in the digital twin model, and the orientation of the guardian's head determined in the foregoing is combined to generate a digital person corresponding to the guardian in the digital twin model. The line-of-sight area of the guardian corresponding to the digital person is the line-of-sight area of the guardian.

[0058] In some embodiments, the synchronization between the change of the line of sight of the guardian and the change of the position of the actual operator can be analyzed, and / or the amplitude of the change of the expression of the guardian can be analyzed, to determine whether the guardian is performing effective supervision from other dimensions, which can further improve the supervision effect on the operation of the sluice.

[0059] Specifically, the position of the actual operator in the video of the operation of the sluice can be tracked, and the actual operator can be represented by the center of mass of the pixel area corresponding to the actual operator in each video frame of the video of the operation of the sluice. The data sequence formed by a plurality of centers of mass is used to represent the change of the position of the actual operator. The guardian can be represented by any point on the center line of the line of sight area of the digital person corresponding to the guardian. The data formed by different point positions corresponding to different video frames is used to represent the change of the line of sight of the guardian. The dynamic time warping (DTW) distance between the two data sequences (the data sequence corresponding to the center of mass and the data sequence corresponding to the point on the center line) is calculated to represent the synchronization between the change of the line of sight of the guardian and the change of the position of the actual operator. The smaller the DTW distance between the two data sequences, the higher the synchronization between the change of the line of sight of the guardian and the change of the position of the actual operator, and the higher the probability that the guardian is performing effective supervision. Conversely, the higher the probability that the guardian is not performing effective supervision.

[0060] In addition, the facial features of the guardian in each video frame are extracted, and the feature similarity (such as using the cosine similarity method) between different facial features of adjacent video frames is calculated to obtain a plurality of feature similarity values (used to quantify the feature similarity between different facial features of adjacent video frames). The absolute difference between the maximum feature similarity value and the minimum feature similarity value in the plurality of feature similarity values is calculated to obtain a feature similarity extreme coefficient, and the standard deviation of the plurality of feature similarity values is calculated to obtain a feature similarity dispersion coefficient. The product of the feature similarity extreme coefficient and the feature similarity dispersion coefficient represents the amplitude of the change of the expression of the guardian. The larger the product, the greater the amplitude of the change of the expression of the guardian, the lower the concentration of the guardian, and the lower the probability that the guardian is performing effective supervision. Conversely, the higher the probability that the guardian is not performing effective supervision.

[0061] In some embodiments, after generating the alarm information indicating the abnormality of the operation of the sluice, the method further comprises: locking the operation of each object to be operated in the operation scene of the sluice, and visually displaying the water conservancy accident associated with the actual operation behavior.

[0062] It should be understood that the actual operator cannot operate the object to be operated in the operation-locked state (such as the control panel of the sluice).

[0063] Based on the above settings, in the case of identifying the abnormal operation of the water gate operator, the operation of each object to be operated in the water gate operation scene is locked to prevent the water gate operator from performing more abnormal water gate operations, thereby reducing the risk of water conservancy accidents caused by continuous abnormal operations.

[0064] It should be noted that after the operation of each object to be operated in the water gate operation scene is locked, the management personnel can determine the accident risk is removed, and the operation lock of the object to be operated can be released by issuing a lock release instruction, so that the water gate operator can continue to operate the object to be operated.

[0065] As for the visual display of the water conservancy accident associated with the actual operation behavior, the corresponding management personnel of the water gate operator can be accurately prompted about the specific accident that may be caused by the actual operation behavior of the water gate operator, so as to help the management personnel quickly and clearly identify the safety hazards caused by the actual operation behavior of the water gate operator, thereby better handling the abnormal water gate operation of the water gate operator.

[0066] For example, the above-mentioned abnormal water gate operation can be: not performing a patrol inspection action around the opening and closing mechanism of the water gate before opening the water gate, such as bending over to check the equipment, flipping through the record table, etc.; not wearing protective equipment correctly, and attempting to operate in the opposite direction when the gate is not completely closed.

[0067] The alarm information can be completed by on-site sound and light, as well as push message or mobile application notification. The alarm information at least includes the violation type, occurrence time and on-site video screenshot corresponding to the abnormal water gate operation of the water gate operator, so that the management personnel can take timely disposal measures.

[0068] In practical application, the whole process of each water gate operation can be recorded, including video recording and analysis result log. For the detected violation event, a detailed violation report is generated to record the violation time, type and on-site picture. For subsequent safety management analysis and personnel training.

[0069] For example: save the time, type and corresponding video segment of all violation events to the database, automatically generate weekly, monthly and annual operation reports, and provide frequently occurring violation types and high-occurrence time periods.

[0070] In addition, the digital twin model, real-time video picture and analysis result are comprehensively displayed. The management personnel can view the real-time running state of the water gate digital twin through the interface, including the gate opening degree, equipment state, personnel position, etc., as well as the execution of the current operation step. The standard operation process and the actual operation are displayed in the form of a flowchart on the interface, and normal steps and violation steps are marked with different colors. The digital twin simulation of historical operation can also be played back to analyze the violation reason.

[0071] When it is necessary to adjust the standard process or model parameters, authorized users can modify the step sequence, threshold settings, etc. of the standard process library to adapt to the specific requirements of different sluices and continuously improve the operation specifications.

[0072] In summary, the sluice standardization intelligent supervision method provided by the present application can accurately identify whether the behavior of the sluice operator conforms to the predetermined safe operation process and timely alarm when a violation is found, solving the problem of untimely and unintelligent sluice operation supervision in the prior art, effectively preventing water conservancy accidents caused by human error, and ensuring the safe operation of sluice projects.

[0073] The beneficial effects mainly include: The sluice three-dimensional digital twin model is deeply integrated with real-time video stream, the virtual and real scene view alignment is realized through camera calibration, a "video-driven dynamic twin environment" is constructed, the limitations of traditional digital twin relying only on sensor data are broken through, and the visualization simulation and real-time mapping of operation behavior are realized. Through real-time video stream acquisition by a high-definition camera, target detection and behavior recognition are performed by a light-weight neural network (such as YOLOv5s) of an edge computing node, the digital twin model is updated synchronously, the whole process of operation is dynamically monitored, illegal behavior (such as single-person operation, step omission) can be timely alarmed, the response speed is improved to "seconds", and the problem of insufficient pre-alarm in traditional supervision is solved, and the safety management is changed from post-tracing to pre-prevention, from "people defense" to "technology defense + intelligence defense".

[0074] Computer vision (YOLOv5 target detection), time series modeling (CNN-LSTM + attention mechanism) and dynamic time warping algorithm are integrated to form a whole-chain intelligent supervision system of "video acquisition-behavior recognition-process comparison-risk warning". The operation behavior requirements of the sluice operator are taken as the physical behavior constraints, and the macro-behavior video AI of the sluice operator is taken as the driving real-time mapping of the field activities, to realize the automatic quantitative evaluation of the sluice operation compliance.

[0075] Not only real-time alarm, but also digital twin pre-play of illegal consequences and triggering of interlock protection (i.e. operation locking), upgrade of "passive monitoring" to "active intervention", combined with manual confirmation of unlocking, balance of intelligence and safety. Digital twin model and real-time video mapping support illegal operation consequence pre-play and trigger interlock protection, which can alarm at the first time of illegal operation, ensure that the management personnel and on-site personnel can know the abnormality at the first time, take timely measures, correct errors in time, and avoid small mistakes from evolving into big accidents. The operation process is presented in a three-dimensional visual way to assist the management personnel in intuitively understanding the risk, the pre-play function can avoid catastrophic accidents in advance, and the emergency decision-making efficiency is improved.

[0076] It automatically records operation logs and generates violation reports, automatically completing a large amount of tedious supervision work, reducing the burden on managers, reducing errors and formalism in manual recording, providing real-world cases for training, and accumulating standardized management data over the long term.

[0077] See Figure 2 , Figure 2 This is a structural schematic diagram of a standardized operation intelligent monitoring system 200 for sluice gates provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the sluice gate standardized operation intelligent monitoring system 200 includes: Video acquisition module 201 is used to acquire video of sluice gate operation by sluice gate operators; The video recognition module 202 is used to perform content recognition on the sluice gate operation video to obtain operation behavior information, which is used to represent the actual operation behavior of the sluice gate operator in the sluice gate operation video. The behavior analysis module 203 is used to analyze the difference between the operation behavior information and the standard behavior information to obtain operation anomaly values, wherein the operation anomaly values ​​are used to characterize the degree to which the actual operation behavior of the sluice gate operator does not match the standardized operation. The alarm module 204 is used to generate alarm information indicating an abnormal operation of the sluice gate when the abnormal operation value is greater than a preset abnormal threshold.

[0078] In one embodiment, the step of performing content recognition on the sluice gate operation video to obtain operation behavior information includes: The video of the sluice gate operation is processed by frame extraction to obtain multiple operation video frames; In the plurality of operation video frames, feature extraction is performed on each operation video frame to obtain a plurality of feature data, wherein the plurality of feature data corresponds one-to-one with the plurality of operation video frames; Based on the acquisition time of each video frame, multiple feature data are combined into a feature sequence in order of acquisition time from first to last. The feature sequence is subjected to time series analysis to obtain the operation behavior information.

[0079] In one embodiment, the step of performing time-series analysis on the feature sequence to obtain the operational behavior information includes: A time-series analysis is performed on the feature sequence to obtain analysis results, wherein the analysis results include multiple behavioral similarity values, each of which corresponds one-to-one with multiple preset behaviors, and the behavioral similarity values ​​are used to represent the degree of similarity between the actual operation behavior and the corresponding preset behavior; determining a maximum behavior similarity value as a target behavior similarity value from the multiple behavior similarity values included in the analysis result; forming the operation behavior information according to a preset behavior corresponding to the target behavior similarity value.

[0080] In an embodiment, the standard operation information is used to indicate an operation behavior to be completed and having a minimum execution order in a sluice operation process; The step of analyzing the difference between the operation behavior information and the standard operation information to obtain an operation abnormal value includes: In a case where the preset behavior indicated by the operation behavior information is different from the operation behavior indicated by the standard operation information, a preset first value is determined as the operation abnormal value; In a case where the preset behavior indicated by the operation behavior information is the same as the operation behavior indicated by the standard operation information, a preset second value is determined as the operation abnormal value; The first value is greater than the abnormal threshold value, and the second value is less than the abnormal threshold value.

[0081] In an embodiment, the sluice standardized operation intelligent supervision system 200 further includes: A number of people monitoring module is configured to perform target detection on the sluice operation video to obtain a video number of people, where the video number of people is used to represent a number of sluice operation personnel included in the sluice operation video. The alarm module 204 is further configured to generate alarm information indicating sluice operation abnormality in a case where the video number of people is less than a sluice operation standard number of people.

[0082] In an embodiment, the alarm module 204 is further configured to: In a case where the video number of people is greater than or equal to the sluice operation standard number of people, distinguish an actual operation personnel from a guardian personnel in the multiple sluice operation personnel included in the sluice operation video, where the actual operation personnel is a sluice operation personnel closest to a corresponding target to be operated in the multiple sluice operation personnel, and the guardian personnel is a sluice operation personnel other than the actual operation personnel in the multiple sluice operation personnel; Calculate a guardian time length ratio between a guardian time period of the guardian personnel and a video collection time period of the sluice operation video to obtain a guardian time length ratio, where the guardian time period is used to indicate a time period during which a line-of-sight area of the corresponding guardian personnel includes the actual operation personnel; Generate alarm information indicating sluice operation abnormality in a case where the guardian time length ratio is less than a preset time length ratio threshold value.

[0083] In an embodiment, the sluice standardized operation intelligent supervision system 200 further includes: The linkage module is used for operation locking of each object to be operated in a water gate operation scene, and visual display of a water conservancy accident associated with the actual operation behavior.

[0084] The water gate standardized operation intelligent supervision system 200 can realize the method embodiment in the application Figure 1 The various processes of the method embodiment and the same beneficial effects are not repeated here.

[0085] The application also provides an electronic device. Figure 3 The electronic device can include a processor 301, a memory 302, and a program 3021 stored in the memory 302 and executable on the processor 301.

[0086] The program 3021, when executed by the processor 301, can realize Figure 1 Any step in the corresponding method embodiment and the same beneficial effects can be achieved, and are not repeated here.

[0087] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by program instructions related to hardware, and the program can be stored in a readable medium.

[0088] The application also provides a readable storage medium, and the readable storage medium stores a computer program. Figure 1 Any step in the corresponding method embodiment and the same beneficial effects can be achieved, and are not repeated here.

[0089] The computer readable storage medium of the application embodiment can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0090] A computer readable signal medium can include a propagated data signal with computer executable instructions. A propagated signal can be an electromagnetic signal, an optical signal, and / or any other suitable type of signal. A computer readable storage medium can include any non-transitory medium that can store the computer executable instructions, such as a hard disk, a floppy disk, a CD-ROM, a DVD, a Blu-ray disk, a RAM, a ROM, a flash memory, and / or any other suitable type of non-transitory computer readable medium.

[0091] The program code embodied on the computer readable storage medium can be transmitted by any programmed medium including but not limited to wireless, wired, optical fiber cable, RF, and / or other suitable type of medium.

[0092] The computer program code can also be implemented in one or more computer programs or one or more modules that execute on or in conjunction with the various hardware components, such as a processing unit an input device, and / or an output device.

[0093] The above description is the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the principles of the present application, can make some improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A water gate standardized operation intelligent monitoring method, characterized in that, The method comprises: acquiring a water gate operation video of a water gate operator; performing content recognition on the water gate operation video to obtain operation behavior information, the operation behavior information being used to indicate actual operation behavior exhibited by the water gate operator in the water gate operation video; analyzing differences between the operation behavior information and standard behavior information to obtain an operation abnormal value, wherein the operation abnormal value is used to represent a degree to which the actual operation behavior of the water gate operator does not match standardized operation; in a case where the operation abnormal value is greater than a preset abnormal threshold, generating alarm information indicating water gate operation abnormality.

2. The intelligent monitoring method for standardized operation of a water gate according to claim 1, characterized in that, The content recognition on the water gate operation video to obtain operation behavior information comprises: performing frame extraction processing on the water gate operation video to obtain a plurality of operation video frames; performing feature extraction on each operation video frame in the plurality of operation video frames to obtain a plurality of feature data, wherein the plurality of feature data correspond one-to-one to the plurality of operation video frames; combining the plurality of feature data into a feature sequence in an order from earlier to later based on collection times of each operation video frame; performing time series analysis on the feature sequence to obtain the operation behavior information.

3. The method of claim 2, wherein, The time series analysis on the feature sequence to obtain the operation behavior information comprises: performing time series analysis on the feature sequence to obtain an analysis result, wherein the analysis result comprises a plurality of behavior similarity values, the plurality of behavior similarity values corresponding one-to-one to a plurality of preset behaviors, and the behavior similarity value is used to represent a similarity degree between the actual operation behavior and the corresponding preset behavior; determining a maximum behavior similarity value in the plurality of behavior similarity values included in the analysis result as a target behavior similarity value; forming the operation behavior information according to the preset behavior corresponding to the target behavior similarity value.

4. The method of claim 3, wherein, The standard behavior information is used to indicate an operation behavior to be completed and having a minimum execution order in a water gate operation process; The analysis of differences between the operation behavior information and the standard behavior information to obtain an operation abnormal value comprises: in a case where a preset behavior indicated by the operation behavior information is different from an operation behavior indicated by the standard behavior information, determining a preset first value as the operation abnormal value; in a case where the preset behavior indicated by the operation behavior information is the same as the operation behavior indicated by the standard behavior information, determining a preset second value as the operation abnormal value; wherein the first value is greater than the abnormal threshold, and the second value is less than the abnormal threshold.

5. The water gate standardized operation intelligent monitoring method according to claim 1, characterized in that, After the acquisition of the water gate operation video of the water gate operator, the method further comprises: performing target detection on the water gate operation video to obtain a video person number, wherein the video person number is used to represent a quantity of water gate operators included in the water gate operation video; in a case where the video person number is less than a water gate operation standard person number, generating alarm information indicating water gate operation abnormality.

6. The intelligent monitoring method for standardized operation of a water gate according to claim 5, characterized in that, After the target detection on the water gate operation video to obtain the video person number, the method further comprises: In a case where the number of persons in the video is greater than or equal to the number of persons in the standard operation of the sluice, real operators and guardians are distinguished among the plurality of sluice operators included in the sluice operation video, wherein the real operator is a sluice operator closest to the corresponding target to be operated among the plurality of sluice operators, and the guardian is a sluice operator other than the real operator among the plurality of sluice operators; A guardian time length ratio is calculated between a guardian time period and a video acquisition time period of the sluice operation video, to obtain a guardian time length ratio, wherein the guardian time period is used to indicate a time period during which a line-of-sight area of the corresponding guardian includes the real operator; In a case where the guardian time length ratio is less than a preset time length ratio threshold, alarm information indicating sluice operation abnormality is generated.

7. The water gate standardized operation intelligent monitoring method according to claim 1, characterized in that, After the alarm information indicating sluice operation abnormality is generated, the method further comprises: Operation locking is performed on each object to be operated in a sluice operation scene, and a water conservancy accident associated with the actual operation behavior is visually displayed.

8. An intelligent monitoring system for standardizing operation of a water gate, characterized by, The system comprises: A video acquisition module is configured to acquire a sluice operation video of a sluice operator; A video recognition module is configured to perform content recognition on the sluice operation video to obtain operation behavior information, wherein the operation behavior information is used to represent actual operation behavior of the sluice operator in the sluice operation video; An operation analysis module is configured to analyze differences between the operation behavior information and standard behavior information to obtain an operation abnormality value, wherein the operation abnormality value is used to represent a degree to which the actual operation behavior of the sluice operator does not match standardized operation; An alarm module is configured to generate alarm information indicating sluice operation abnormality in a case where the operation abnormality value is greater than a preset abnormality threshold.

9. An electronic device, comprising: A computer program is stored on a readable storage medium and executable on a processor, and the computer program is executed by the processor to implement steps of the sluice standardized operation intelligent monitoring method according to any one of claims 1 to 7.

10. A readable storage medium, characterized by, A computer program is stored on a readable storage medium and executable on a processor, and the computer program is executed by the processor to implement steps of the sluice standardized operation intelligent monitoring method according to any one of claims 1 to 7.

Images