A safety monitoring system for power engineering construction based on deep learning

Abstract

The present invention discloses a power engineering construction safety monitoring system based on deep learning, which belongs to the field of power engineering construction safety technology, and includes: a data acquisition module for real-time acquisition of RGB images, laser radar point clouds and depth maps of the construction site; an intelligent analysis module for parallel prediction of the risk levels of material dumping, personnel falling, mechanical collision and electrical hazards based on a multi-task deep learning model; a dynamic decision-making module for dispatching personnel, AGVs and drones based on reinforcement learning according to risk prediction results and risk priorities, sending mechanical emergency stop instructions, and guiding personnel to avoid risks in real time through alarm information; a feedback optimization module for optimizing the model through incremental learning. The present invention realizes real-time prediction and automatic response of multiple types of accidents through multimodal data fusion and intelligent decision-making.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power engineering construction safety, and particularly relates to a safety monitoring system for electric power engineering construction based on deep learning. Background Art

[0002] Electric power engineering is related to the production, transmission, and distribution of electric energy. Electric power engineering construction is a production site with multi-type work, multi-level three-dimensional cross-operation, many temporary facilities, and large changes in the operation surface. There are many unsafe factors and it belongs to a high-risk industry. Moreover, the on-site operation points are numerous and wide-ranging, the participating personnel have a large mobility, and the statistical caliber of project safety information is chaotic. Safety control mainly relies on traditional means. During the construction process, when the staff makes non-compliant operations, the safety officer cannot quickly discover them, resulting in potential safety hazards.

[0003] Traditional electric power construction safety monitoring systems mostly rely on single sensors (such as weight sensors, inclination sensors) or manual inspections, and have the following defects: limited monitoring range: it can only cover the risk of material stability and cannot identify personnel, mechanical, and electrical hazards; response lag: relying on manual decision-making, it is difficult to respond to dynamic risks in a timely manner; hardware redundancy: independent deployment of multiple sensors leads to high costs and complex maintenance. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. For this purpose, the present invention proposes a safety monitoring system for electric power engineering construction based on deep learning, which realizes real-time prediction and automated response to multiple types of accidents through multi-modal data fusion and intelligent decision-making.

[0005] On the one hand, an embodiment of the present invention provides a power engineering construction safety monitoring system based on deep learning, including: a data acquisition module, which is used to collect RGB images, laser radar point clouds and depth maps of the construction site in real time; an intelligent analysis module, which is used to predict the risk levels of material dumping, personnel falling, mechanical collision and electrical hazards in parallel based on a multi-task deep learning model, and when the risk of material dumping exceeds a threshold, detect the personnel status and material type; the multi-task deep learning model includes: a shared feature extraction layer, which is used to extract RGB image features based on YOLOv8, process the laser radar point cloud based on PointNet++, extract spatial features, and extract deep features based on a convolutional network; a multimodal fusion layer, which is used to extract spatial features based on a Transformer codec. The encoder aligns RGB, point cloud and depth features to obtain fused features; the task-specific head network, including GRU, LSTM, Kalman filter, HRNet and ResNet-50 classification models, is used to output four types of risk prediction results based on the fused features, and dynamically trigger the detection of personnel status and material type; the dynamic decision-making module is used to dispatch personnel, AGVs and drones based on reinforcement learning according to risk prediction results and risk priorities, send mechanical emergency stop instructions, and guide personnel to avoid risks in real time through alarm information; when dispatching personnel or AGVs to perform handling tasks, the handling decision is adjusted according to the personnel status, material type and AGV status, and the handling decision includes AGV handling and mixed handling; the feedback optimization module is used to optimize the model through incremental learning.

[0006] According to some embodiments of the present invention, the data acquisition module includes: a camera unit for acquiring RGB images and video streams of the construction site; a lidar unit for acquiring three-dimensional point cloud data of the construction site; and a depth camera unit for acquiring a depth map of the construction site.

[0007] According to some embodiments of the present invention, the shared feature extraction layer includes: a visual branch unit, which is used to extract RGB image features based on the YOLOv8 backbone network to capture the position and category of the target, wherein the target includes materials, personnel and equipment; a point cloud branch unit, which is used to process lidar data based on PointNet++ to extract spatial features, wherein the spatial features include obstacle density and motion trajectory; and a depth map unit, which is used to extract depth features through a convolutional network to calculate the inclination angle of the material and the position of the personnel.

[0008] According to some embodiments of the present invention, the task-specific head network includes: a material stability analysis unit, configured to obtain the material stacking area recognized by YOLOv8, calculate the inclination angle, height, and central position of the material stack through the depth map corresponding to the material stacking area and the point cloud data; and obtain the wind speed and ground inclination through lidar data; input the inclination angle, height, central position, wind speed, and ground inclination of the material stack into the GRU network, and output the tipping risk level.

[0009] According to some embodiments of the present invention, the task-specific head network includes: a personnel status detection unit, configured to activate the HRNet branch when the tipping risk level is high, extract skeletal key points from the RGB image, count the number of times of bending over per unit time through the change in the angle of the skeletal key points in consecutive frames, and obtain the personnel status score; a material type recognition unit, configured to obtain the material area image detected by YOLOv8 when the tipping risk level is high, input it into the ResNet-50 classification model, and output whether the material type is a fragile material or a non-fragile material.

[0010] According to some embodiments of the present invention, the task-specific head network includes: a personnel fall warning unit, configured to extract multiple key points of the human body from the RGB image through HRNet, calculate the angle of the connection line of the key points, determine whether it is in a dangerous posture according to the angle of the connection line of the key points, and locate the distance between the personnel and the dangerous area through the depth map and the point cloud data, and analyze the action trend in consecutive frames based on LSTM, and output the fall probability.

[0011] According to some embodiments of the present invention, the task-specific head network includes: a mechanical collision prediction unit, configured to obtain the trajectory features extracted by PointNet++ and the mechanical position detected by YOLOv8, obtain the mechanical position and movement direction, predict the mechanical movement path in the next period of time based on the Kalman filter combined with the historical trajectory point cloud data, and calculate the Euclidean distance between the mechanical movement path and the obstacle, and trigger an alarm when the Euclidean distance is less than the safety threshold.

[0012] According to some embodiments of the present invention, the task-specific head network includes: an electrical hazard detection unit, configured to detect the visual features of cable damage and identify the cable damage area based on the RGB image using the ResNet-50 classification model.

[0013] According to some embodiments of the present invention, the dynamic decision-making module includes: a priority sorting unit for generating a risk queue in real time, with the sorting order being electrical hazards, mechanical collisions, personnel falls, and material spills; a reinforcement learning decision-making unit for constructing a state space based on various risk levels, the positions of AGVs / drones, the mechanical operating states, the path feasibility coefficients, the battery levels of AGVs, the task loads of drones, and the mechanical emergency stop response delays, and collaboratively optimizing the policy network and the value network based on the PPO algorithm. The policy network outputs action parameters, and the action parameters include the AGV path direction, the drone target probability, the mechanical emergency stop binary decision, and the manual intervention binary decision. The value network predicts the cumulative reward of the state.

[0014] According to some embodiments of the present invention, the dynamic decision-making module includes: a handling decision-making unit for calculating a handling strategy score based on the material fragility coefficient, the personnel status score, and the cost-benefit coefficient. When the handling strategy score is greater than a threshold, the handling method is determined to be AGV handling; otherwise, the handling method is hybrid handling. The material fragility coefficient is obtained based on whether the material is a fragile material or a non-fragile material; the personnel status score is calculated based on the number of bending times per unit time; and the cost-benefit coefficient is calculated through the battery level of the AGV device and the electricity cost corresponding to the electricity consumption period.

[0015] The power engineering construction safety monitoring system based on the Internet of Things according to the embodiments of the present invention has at least the following beneficial effects: Through multi-modal data fusion and intelligent decision-making, the embodiments of the present invention achieve real-time prediction and automated response to various types of accidents.

[0016] The additional aspects and advantages of the present invention will be partly given in the following description, partly will become obvious from the following description, or will be understood through the practice of the present invention. Description of the Drawings

[0017] The above and / or additional aspects and advantages of the present invention will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0018] Figure 1 It is a schematic block diagram of the modules of the system according to the embodiments of the present invention.

[0019] Reference Signs:

[0020] Data acquisition module 100, intelligent analysis module 200, dynamic decision-making module 300, feedback optimization module 400. Detailed Embodiments

[0021] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where like or similar reference numerals denote like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0022] In the description of the present invention, the meaning of "a number of" is one or more, the meaning of "a plurality of" is two or more, "greater than", "less than", "exceeding", etc. are understood not to include the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" and "second", it is only for the purpose of distinguishing technical features and should not be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features.

[0023] Kalman filter: An algorithm for estimating the state of a system, applicable to noisy linear systems. By fusing historical trajectory data with real-time observations, the Kalman filter can effectively predict the future movement path of machinery (such as engineering vehicles, drones, etc.), especially suitable for linear or weakly non-linear motion models (such as uniform motion, uniform acceleration).

[0024] YOLOv8: The latest iteration of the YOLO series, supporting a full range of vision tasks, including object detection, instance segmentation, image classification, pose estimation, and tracking, etc.

[0025] HRNet: A deep neural network designed specifically for high-precision vision tasks (such as human pose estimation, image segmentation). Its core idea is to maintain a high-resolution feature map throughout the process and achieve the interactive fusion of features with different resolutions through a multi-branch parallel architecture, solving the problem of spatial information loss caused by downsampling in traditional networks.

[0026] ResNet-50: A deep residual network that solves the problem of deep network degradation by introducing residual learning, supports the construction of deep models with more than 50 layers, and achieves high-precision classification through residual structures and depth stacking.

[0027] PointNet++: An improved version of PointNet, mainly solving the problem of insufficient local feature extraction ability of the original model, and improving the processing ability for complex point cloud data through hierarchical feature learning and multi-scale feature fusion.

[0028] The system of the embodiments of the present invention collects environmental data in real time through devices such as cameras, lidars, and depth cameras, and transmits it to the edge nodes. Then, a multi-task deep learning model is used to parallelly process four types of risk predictions. Next, resources (such as AGV reinforcement materials, drone spraying of insulating materials) are dynamically scheduled according to the risk level, and personnel are guided to avoid risks in real time through AR / voice. Finally, the model is continuously optimized through incremental learning to adapt to new scenarios and new risks.

[0029] Referring to Figure 1 , the embodiments of the present invention propose a power engineering construction safety monitoring system based on deep learning, including:

[0030] A data acquisition module 100, configured to collect RGB images, lidar point clouds, and depth maps of the construction site in real time;

[0031] An intelligent analysis module 200, configured to parallelly predict the risk levels of material dumping, personnel falling, mechanical collision, and electrical hazards based on a multi-task deep learning model, and when the material dumping risk exceeds the threshold, detect the personnel status and material type;

[0032] A dynamic decision-making module 300, configured to schedule personnel, AGVs, and drones based on reinforcement learning according to the risk prediction results and risk priorities, send mechanical emergency stop instructions, and guide personnel to avoid risks in real time through warning information; when scheduling personnel or AGVs for handling tasks, adjust the handling decision according to the personnel status, material type, and AGV status, and the handling decision includes AGV handling and mixed handling;

[0033] A feedback optimization module 400, configured to optimize the model through incremental learning.

[0034] In some embodiments, the data acquisition module 100 of the embodiments of the present invention includes: a camera unit, configured to obtain RGB images and video streams of the construction site; a lidar unit, configured to obtain three-dimensional point cloud data of the construction site; and a depth camera unit, configured to obtain depth maps of the construction site.

[0035] In some embodiments, the multi-task deep learning model of the present embodiment includes:

[0036] A shared feature extraction layer, configured to extract RGB image features based on YOLOv8, process lidar point clouds based on PointNet++, extract spatial features, and extract depth features based on a convolutional network;

[0037] A multi-modal fusion layer, configured to align RGB, point cloud, and depth features based on a Transformer encoder to obtain fused features;

[0038] The task-specific head network, including GRU, LSTM, Kalman filter, HRNet, and ResNet-50 classification models, is used to respectively output four types of risk prediction results based on the fusion features, and dynamically trigger the detection of personnel status and material type.

[0039] In some embodiments, the shared feature extraction layer includes:

[0040] The visual branch unit is used to extract RGB image features based on the backbone network of YOLOv8, and capture the position and category of the targets, where the targets include materials, personnel, and equipment.

[0041] The point cloud branch unit is used to process lidar data based on PointNet++ and extract spatial features, where the spatial features include obstacle density and motion trajectory.

[0042] The depth map unit is used to extract depth features through a convolutional network to calculate the tilt angle of the material and the position of the personnel.

[0043] In some embodiments, the task-specific head network includes: a material stability analysis unit, which is used to obtain the material stacking area recognized by YOLOv8, calculate the tilt angle, height, and central position of the material stack through the depth map and point cloud data corresponding to the material stacking area; and obtain the wind speed and ground inclination through lidar data; input the tilt angle, height, central position, wind speed, and ground inclination of the material stack into the GRU network to output the tipping risk level.

[0044] In some embodiments, the task-specific head network includes: a personnel status detection unit, which is used to activate the HRNet branch when the tipping risk level is high, extract skeletal key points from the RGB image, and obtain the personnel status score by statistically counting the number of bending times per unit time based on the angle change of the skeletal key points in consecutive frames.

[0045] The material type recognition unit is used to obtain the material area image detected by YOLOv8 when the tipping risk level is high, input it into the ResNet-50 classification model, and output whether the material type is a fragile material or a non-fragile material.

[0046] In some embodiments, the task-specific head network includes: a personnel fall warning unit, which is used to extract multiple key points of the human body from the RGB image through HRNet, calculate the angle of the connection line of the key points, determine whether it is in a dangerous posture according to the angle of the connection line of the key points, and locate the distance between the personnel and the dangerous area through the depth map and point cloud data, and analyze the action trend in consecutive frames based on LSTM to output the fall probability.

[0047] In some embodiments, the task-specific head network includes: a mechanical collision prediction unit, which is used to obtain the trajectory features extracted by PointNet++ and the mechanical positions detected by YOLOv8, obtain the mechanical positions and movement directions, predict the mechanical movement path within a certain period in the future based on the Kalman filter combined with historical trajectory point cloud data, and calculate the Euclidean distance between the mechanical movement path and the obstacles. When the Euclidean distance is less than the safety threshold, a warning is triggered.

[0048] In some embodiments, the task-specific head network includes: an electrical hazard detection unit, which is used to detect the visual features of cable breakage using a ResNet-50 classification model based on RGB images and identify the cable breakage area.

[0049] In some embodiments, the dynamic decision-making module 300 of the embodiments of the present invention includes:

[0050] A priority ranking unit, which is used to generate a risk queue in real time, and the ranking is electrical hazard, mechanical collision, personnel fall, and material dumping in sequence.

[0051] A reinforcement learning decision-making unit, which is used to construct a state space according to each risk level, the position of the AGV / drone, the mechanical operating state, the path feasibility coefficient, the AGV battery power, the drone task load, and the mechanical emergency stop response delay, and co-optimize the policy network and the value network based on the PPO algorithm. The policy network outputs action parameters, and the action parameters include the AGV path direction, the drone target probability, the mechanical emergency stop binary decision, and the human intervention binary decision. The value network predicts the state cumulative reward.

[0052] In some embodiments, the dynamic decision-making module of the embodiments of the present invention includes: a handling decision-making unit, which is used to calculate the handling strategy score according to the material fragility coefficient, the personnel status score, and the cost-benefit coefficient. When the handling strategy score is greater than the threshold, it is determined that the handling method is AGV handling, otherwise the handling method is hybrid handling. The material fragility coefficient is obtained according to whether the material is a fragile material or a non-fragile material; the personnel status score is calculated according to the number of bending times per unit time; the cost-benefit coefficient is calculated through the AGV device battery power and the electricity cost corresponding to the electricity consumption period.

[0053] Although specific implementation schemes are described herein, those of ordinary skill in the art will recognize that many other modifications or alternative implementation schemes are also within the scope of the present disclosure. For example, any one of the functions and / or processing capabilities described in connection with a specific device or component can be performed by any other device or component. Additionally, although various illustrative specific implementations and architectures have been described according to the embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications to the illustrative specific implementations and architectures described herein are also within the scope of the present disclosure.

[0054] Those of ordinary skill in the art will appreciate that all or some of the steps and systems disclosed above in the methods can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or can be implemented as hardware, or can be implemented as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically contains computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery media.

[0055] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present application, and these equivalent deformations or substitutions are all included within the scope defined by the claims of the present application.

Claims

1. A power engineering construction safety monitoring system based on deep learning, characterized in that, Including: A data acquisition module for real-time acquisition of RGB images, lidar point clouds, and depth maps at the construction site; An intelligent analysis module for parallel prediction of risk levels of material dumping, personnel falling, mechanical collision, and electrical hazards based on a multi-task deep learning model, and for detecting personnel status and material types when the material dumping risk exceeds a threshold; The multi-task deep learning model includes: A shared feature extraction layer for extracting RGB image features based on YOLOv8, processing lidar point clouds based on PointNet++, extracting spatial features, and extracting depth features based on a convolutional network; A multi-modal fusion layer for aligning RGB, point cloud, and depth features based on a Transformer encoder to obtain fused features; Task-specific head networks, including GRU, LSTM, Kalman filter, HRNet, and ResNet-50 classification models, for respectively outputting four types of risk prediction results based on the fused features, and dynamically triggering the detection of personnel status and material types; The task-specific head networks include: A material stability analysis unit for obtaining the material stacking area identified by YOLOv8, calculating the tilt angle, height, and central position of the material stack through the depth map and point cloud data corresponding to the material stacking area; and obtaining the wind speed and ground tilt through lidar data; inputting the tilt angle, height, central position, wind speed, and ground tilt of the material stack into a GRU network to output the dumping risk level; A personnel status detection unit for activating the HRNet branch when the dumping risk level is high, extracting skeletal key points from the RGB image, and obtaining a personnel status score by statistically counting the number of times of bending over per unit time through the angle change of consecutive frame skeletal key points; A material type recognition unit for obtaining the material area image detected by YOLOv8 when the dumping risk level is high, inputting it into a ResNet-50 classification model, and outputting whether the material type is a fragile material or a non-fragile material; A dynamic decision-making module for scheduling personnel, AGVs, and drones based on reinforcement learning according to the risk prediction results and risk priorities, sending mechanical emergency stop instructions, and guiding personnel to avoid danger in real time through warning information; when scheduling personnel or AGVs for handling tasks, adjusting the handling decision according to the personnel status, material type, and AGV status, and the handling decision includes AGV handling and mixed handling; The dynamic decision-making module includes: A handling decision unit for calculating a handling strategy score according to the material fragility coefficient, personnel status score, and cost-benefit coefficient, determining the handling method as AGV handling when the handling strategy score is greater than a threshold, otherwise the handling method is mixed handling, and the material fragility coefficient is obtained according to whether the material is a fragile material or a non-fragile material; the personnel status score is calculated according to the number of times of bending over per unit time; the cost-benefit coefficient is calculated through the battery power of the AGV device and the electricity cost corresponding to the electricity consumption period; A feedback optimization module for optimizing the model through incremental learning.

2. The safety monitoring system for power engineering construction based on deep learning according to claim 1, wherein The data acquisition module includes: A camera unit for obtaining RGB images and video streams at the construction site; A laser radar unit, used to obtain three-dimensional point cloud data of the construction site; Depth camera unit, used to obtain depth map of the construction site.

3. The safety monitoring system for power engineering construction based on deep learning according to claim 1, characterized in that, The shared feature extraction layer comprises: A visual branch unit, which is used to extract RGB image features based on the YOLOv8 backbone network and capture the location and category of targets, including materials, personnel, and equipment; A point cloud branch unit, which is used to process lidar data based on PointNet++ and extract spatial features, including obstacle density and motion trajectory; The depth map unit is used to extract depth features through a convolutional network to calculate the material tilt angle and the position of the person.

4. The safety monitoring system for power engineering construction based on deep learning according to claim 1, characterized in that, The task-specific head network includes: The personnel fall warning unit is used to extract multiple key points of the human body from the RGB image through HRNet, calculate the angle of the key point connection line, determine whether it is in a dangerous posture based on the key point connection angle, and locate the distance between the person and the dangerous area through the depth map and point cloud data. It analyzes the action trend in continuous frames based on LSTM and outputs the fall probability.

5. The safety monitoring system for power engineering construction based on deep learning according to claim 1, characterized in that, The task-specific head network includes: The mechanical collision prediction unit is used to obtain the trajectory features extracted by PointNet++ and the mechanical position detected by YOLOv8, obtain the mechanical position and movement direction, predict the mechanical movement path in the future period based on Kalman filtering combined with historical trajectory point cloud data, and calculate the Euclidean distance between the mechanical movement path and the obstacle. When the Euclidean distance is less than the safety threshold, an early warning is triggered.

6. The safety monitoring system for power engineering construction based on deep learning according to claim 1, characterized in that, The task-specific head network includes: The electrical hazard detection unit is used to detect the visual features of cable damage and identify the cable damage area based on RGB images using the ResNet-50 classification model.

7. The safety monitoring system for power engineering construction based on deep learning according to claim 1, characterized in that, The dynamic decision module includes: Prioritization unit, used to generate risk queues in real time, ranked in order of electrical hazards, mechanical collisions, personnel falls, and material dumping; The reinforcement learning decision unit is used to construct the state space according to each risk level, AGV / UAV position, mechanical operation status, path feasibility coefficient, AGV power, UAV mission load and mechanical emergency stop response delay. The policy network and value network are collaboratively optimized based on the PPO algorithm. The policy network outputs action parameters, which include AGV path direction, UAV target probability, mechanical emergency stop binary decision and manual intervention binary decision. The value network predicts the state cumulative reward.

Images