Electric power engineering construction safety monitoring system based on deep learning

Abstract

The invention discloses an electric power engineering construction safety monitoring system based on deep learning, and belongs to the technical field of electric power engineering construction safety, and the system comprises a data collection module which is used for collecting an RGB image, a laser radar point cloud and a depth map of a construction site in real time; the intelligent analysis module is used for predicting the risk levels of material dumping, personnel falling, mechanical collision and electrical danger in parallel based on a multi-task deep learning model; the dynamic decision-making module is used for dispatching personnel, the AGV and the unmanned aerial vehicle based on reinforcement learning according to the risk prediction result and the risk priority, sending a mechanical sudden stop instruction and guiding the personnel to avoid risks in real time through alarm information; and the feedback optimization module is used for optimizing the model through incremental learning. According to the invention, real-time prediction and automatic response of various accidents are realized through multi-modal data fusion and intelligent decision.

Description

Technical Field

[0001] The present invention relates to the field of power engineering construction safety technology, and in particular to a power engineering construction safety monitoring system based on deep learning. Background Art

[0002] Power engineering refers to engineering related to the production, transmission and distribution of electric energy. Power engineering construction is a production site with multiple types of work, multiple levels of three-dimensional cross-operation, many temporary facilities, and large changes in the working surface. There are many unsafe factors and it is a high-risk industry. In addition, there are many on-site working points, the mobility of construction personnel is large, and the statistical caliber of engineering safety information is chaotic. Safety management and control mostly rely on traditional means. During the construction process, workers may perform non-compliant operations, which safety officers cannot quickly discover, resulting in safety hazards.

[0003] Traditional power construction safety monitoring systems mostly rely on single sensors (such as weight sensors, inclination sensors) or manual inspections, and have the following defects: limited monitoring scope: only covering material stability risks, unable to identify personnel, mechanical and electrical hazards; delayed response: relying on manual decision-making, it is difficult to respond to dynamic risks in a timely manner; hardware redundancy: independent deployment of multiple sensors leads to high costs and complex maintenance. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a power engineering construction safety monitoring system based on deep learning, which realizes real-time prediction and automatic response of multiple types of accidents through multi-modal data fusion and intelligent decision-making.

[0005] On the one hand, an embodiment of the present invention provides a power engineering construction safety monitoring system based on deep learning, including: a data acquisition module, which is used to collect RGB images, laser radar point clouds and depth maps of the construction site in real time; an intelligent analysis module, which is used to predict the risk levels of material dumping, personnel falling, mechanical collision and electrical hazards in parallel based on a multi-task deep learning model, and when the risk of material dumping exceeds a threshold, detect the personnel status and material type; the multi-task deep learning model includes: a shared feature extraction layer, which is used to extract RGB image features based on YOLOv8, process the laser radar point cloud based on PointNet++, extract spatial features, and extract deep features based on a convolutional network; a multimodal fusion layer, which is used to extract spatial features based on a Transformer codec. The encoder aligns RGB, point cloud and depth features to obtain fused features; the task-specific head network, including GRU, LSTM, Kalman filter, HRNet and ResNet-50 classification models, is used to output four types of risk prediction results based on the fused features, and dynamically trigger the detection of personnel status and material type; the dynamic decision-making module is used to dispatch personnel, AGVs and drones based on reinforcement learning according to risk prediction results and risk priorities, send mechanical emergency stop instructions, and guide personnel to avoid risks in real time through alarm information; when dispatching personnel or AGVs to perform handling tasks, the handling decision is adjusted according to the personnel status, material type and AGV status, and the handling decision includes AGV handling and mixed handling; the feedback optimization module is used to optimize the model through incremental learning.

[0006] According to some embodiments of the present invention, the data acquisition module includes: a camera unit for acquiring RGB images and video streams of the construction site; a lidar unit for acquiring three-dimensional point cloud data of the construction site; and a depth camera unit for acquiring a depth map of the construction site.

[0007] According to some embodiments of the present invention, the shared feature extraction layer includes: a visual branch unit, which is used to extract RGB image features based on the YOLOv8 backbone network to capture the position and category of the target, wherein the target includes materials, personnel and equipment; a point cloud branch unit, which is used to process lidar data based on PointNet++ to extract spatial features, wherein the spatial features include obstacle density and motion trajectory; and a depth map unit, which is used to extract depth features through a convolutional network to calculate the inclination angle of the material and the position of the personnel.

[0008] According to some embodiments of the present invention, the task-specific head network includes: a material stability analysis unit, which is used to obtain the material stacking area identified by YOLOv8, calculate the inclination angle, height and center position of the material pile through the depth map and point cloud data corresponding to the material stacking area; and obtain the wind speed and ground inclination through lidar data; input the inclination angle, height, center position, wind speed and ground inclination of the material pile into the GRU network, and output the dumping risk level.

[0009] According to some embodiments of the present invention, the task-specific head network includes: a personnel status detection unit, which is used to activate the HRNet branch when the dumping risk level is high, extract bone key points from the RGB image, and count the number of bends per unit time through the angle changes of the bone key points in consecutive frames to obtain a personnel status score; a material type identification unit, which is used to obtain the material area image detected by YOLOv8 when the dumping risk level is high, input the ResNet-50 classification model, and output the material type as fragile material or non-fragile material.

[0010] According to some embodiments of the present invention, the task-specific head network includes: a personnel fall warning unit, which is used to extract multiple key points of the human body from the RGB image through HRNet, calculate the angle of the key point connection line, determine whether it is in a dangerous posture according to the angle of the key point connection line, and locate the distance between the person and the dangerous area through the depth map and point cloud data, analyze the action trend in continuous frames based on LSTM, and output the fall probability.

[0011] According to some embodiments of the present invention, the task-specific head network includes: a mechanical collision prediction unit, which is used to obtain the trajectory features extracted by PointNet++ and the mechanical position detected by YOLOv8, obtain the mechanical position and movement direction, predict the mechanical movement path in the future period based on Kalman filtering combined with historical trajectory point cloud data, and calculate the Euclidean distance between the mechanical movement path and the obstacle. When the Euclidean distance is less than the safety threshold, an early warning is triggered.

[0012] According to some embodiments of the present invention, the task-specific head network includes: an electrical hazard detection unit for detecting cable damage visual features based on RGB images using a ResNet-50 classification model to identify cable damage areas.

[0013] According to some embodiments of the present invention, the dynamic decision module includes: a priority sorting unit, which is used to generate a risk queue in real time, and the sorting order is electrical hazard, mechanical collision, personnel falling and material dumping; a reinforcement learning decision unit, which is used to construct a state space according to each risk level, AGV / UAV position, mechanical operation status, path feasibility coefficient, AGV power, UAV mission load and mechanical emergency stop response delay, and collaboratively optimize the strategy network and value network based on the PPO algorithm. The strategy network outputs action parameters, and the action parameters include AGV path direction, UAV target probability, mechanical emergency stop binary decision and manual intervention binary decision, and the value network predicts the state cumulative reward.

[0014] According to some embodiments of the present invention, the dynamic decision module includes: a handling decision unit, which is used to calculate a handling strategy score based on the material fragility coefficient, the personnel status score and the cost-effectiveness coefficient. When the handling strategy score is greater than a threshold, the handling method is determined to be AGV handling, otherwise the handling method is mixed handling. The material fragility coefficient is obtained based on whether the material is fragile or non-fragile; the personnel status score is calculated based on the number of bends per unit time; and the cost-effectiveness coefficient is calculated based on the power of the AGV equipment and the electricity cost corresponding to the electricity consumption period.

[0015] An Internet of Things-based power engineering construction safety monitoring system according to an embodiment of the present invention includes at least the following beneficial effects: The embodiment of the present invention realizes real-time prediction and automatic response of multiple types of accidents through multimodal data fusion and intelligent decision-making.

[0016] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0018] Figure 1 The module schematic block diagram of the system according to the embodiment of the present invention.

[0019] Reference numerals:

[0020] Data collection module 100, intelligent analysis module 200, dynamic decision module 300, feedback optimization module 400. DETAILED DESCRIPTION

[0021] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0022] In the description of the present invention, "several" means one or more, "more" means two or more, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0023] Kalman filter: An algorithm for estimating system states, suitable for noisy linear systems. Kalman filter can effectively predict the future motion path of machinery (such as engineering vehicles, drones, etc.) by fusing historical trajectory data with real-time observations, especially for linear or weakly nonlinear motion models (such as uniform speed and uniform acceleration).

[0024] YOLOv8: The latest iteration of the YOLO series, supporting a full range of visual tasks, including object detection, instance segmentation, image classification, pose estimation and tracking, etc.

[0025] HRNet: A deep neural network designed for high-precision visual tasks (such as human pose estimation and image segmentation). Its core idea is to maintain high-resolution feature maps throughout the process and achieve interactive fusion of features of different resolutions through a multi-branch parallel architecture, thus solving the problem of spatial information loss caused by downsampling in traditional networks.

[0026] ResNet-50: Deep residual network, which solves the degradation problem of deep networks by introducing residual learning, supports the construction of deep models with more than 50 layers, and achieves high-precision classification through residual structure and deep stacking.

[0027] PointNet++: An improved version of PointNet, which mainly solves the problem of insufficient local feature extraction capability of the original model and improves the ability to process complex point cloud data through hierarchical feature learning and multi-scale feature fusion.

[0028] The system of the embodiment of the present invention collects environmental data in real time through devices such as cameras, lidars, and depth cameras, and transmits it to edge nodes. Then, a multi-task deep learning model is used to process four types of risk predictions in parallel. Then, resources are dynamically scheduled according to the risk level (such as AGV reinforcement materials, drone spraying insulation materials), and personnel are guided to avoid risks in real time through AR / voice. Finally, the model is continuously optimized through incremental learning to adapt to new scenarios and new risks.

[0029] Reference Figure 1 The embodiment of the present invention proposes a power engineering construction safety monitoring system based on deep learning, including:

[0030] The data acquisition module 100 is used to collect RGB images, laser radar point clouds and depth maps of the construction site in real time;

[0031] Intelligent analysis module 200, for predicting the risk level of material dumping, personnel falling, mechanical collision and electrical hazards in parallel based on a multi-task deep learning model, and detecting personnel status and material type when the risk of material dumping exceeds a threshold;

[0032] The dynamic decision module 300 is used to dispatch personnel, AGVs and drones based on reinforcement learning according to risk prediction results and risk priorities, send mechanical emergency stop instructions, and guide personnel to avoid risks in real time through alarm information; when dispatching personnel or AGVs to perform handling tasks, the handling decision is adjusted according to the personnel status, material type and AGV status, and the handling decision includes AGV handling and mixed handling;

[0033] Feedback optimization module 400 is used to optimize the model through incremental learning.

[0034] In some embodiments, the data acquisition module 100 of the embodiment of the present invention includes: a camera unit for acquiring RGB images and video streams of the construction site; a lidar unit for acquiring three-dimensional point cloud data of the construction site; and a depth camera unit for acquiring a depth map of the construction site.

[0035] In some embodiments, the multi-task deep learning model of this embodiment includes:

[0036] Shared feature extraction layer, used to extract RGB image features based on YOLOv8, process lidar point clouds based on PointNet++, extract spatial features, and extract deep features based on convolutional networks;

[0037] The multimodal fusion layer is used to align RGB, point cloud and depth features based on the Transformer encoder to obtain fused features;

[0038] The task-specific head network, including GRU, LSTM, Kalman filter, HRNet and ResNet-50 classification models, is used to output four types of risk prediction results based on the fused features, as well as dynamically trigger the detection of personnel status and material type.

[0039] In some embodiments, the shared feature extraction layer includes:

[0040] The visual branch unit is used to extract RGB image features based on the YOLOv8 backbone network and capture the location and category of the target, which includes materials, personnel, and equipment.

[0041] The point cloud branch unit is used to process the lidar data based on PointNet++ and extract spatial features, wherein the spatial features include obstacle density and motion trajectory.

[0042] The depth map unit is used to extract depth features through a convolutional network to calculate the material tilt angle and the position of the person.

[0043] In some embodiments, the task-specific head network includes: a material stability analysis unit, which is used to obtain the material stacking area identified by YOLOv8, calculate the inclination angle, height and center position of the material pile through the depth map and point cloud data corresponding to the material stacking area; and obtain the wind speed and ground inclination through lidar data; the inclination angle, height, center position, wind speed and ground inclination of the material pile are input into the GRU network, and the dumping risk level is output.

[0044] In some embodiments, the task-specific head network includes: a personnel status detection unit, which is used to activate the HRNet branch when the dumping risk level is high, extract skeleton key points from the RGB image, and count the number of bends per unit time through the changes in the angles of the skeleton key points in consecutive frames to obtain a personnel status score.

[0045] The material type recognition unit is used to obtain the material area image detected by YOLOv8 when the dumping risk level is high, input it into the ResNet-50 classification model, and output the material type as fragile material or non-fragile material.

[0046] In some embodiments, the task-specific head network includes: a personnel fall warning unit, which is used to extract multiple key points of the human body from the RGB image through HRNet, calculate the angle of the key point connection line, determine whether the person is in a dangerous posture according to the angle of the key point connection line, and locate the distance between the person and the dangerous area through the depth map and point cloud data, analyze the action trend in continuous frames based on LSTM, and output the fall probability.

[0047] In some embodiments, the task-specific head network includes: a mechanical collision prediction unit, which is used to obtain the trajectory features extracted by PointNet++ and the mechanical position detected by YOLOv8, obtain the mechanical position and movement direction, and predict the mechanical movement path in the future based on Kalman filtering combined with historical trajectory point cloud data, and calculate the Euclidean distance between the mechanical movement path and the obstacle. When the Euclidean distance is less than the safety threshold, an early warning is triggered.

[0048] In some embodiments, the task-specific head network includes: an electrical hazard detection unit for detecting cable damage visual features based on RGB images using a ResNet-50 classification model to identify cable damage areas.

[0049] In some embodiments, the dynamic decision module 300 of the embodiment of the present invention includes:

[0050] The priority sorting unit is used to generate risk queues in real time, with the order of electrical hazards, mechanical collisions, personnel falls and material dumping.

[0051] The reinforcement learning decision unit is used to construct the state space according to each risk level, AGV / UAV position, mechanical operation status, path feasibility coefficient, AGV power, UAV mission load and mechanical emergency stop response delay. The policy network and value network are collaboratively optimized based on the PPO algorithm. The policy network outputs action parameters, which include AGV path direction, UAV target probability, mechanical emergency stop binary decision and manual intervention binary decision. The value network predicts the state cumulative reward.

[0052] In some embodiments, the dynamic decision module of the embodiment of the present invention includes: a handling decision unit, which is used to calculate a handling strategy score based on the material fragility coefficient, the personnel status score and the cost-effectiveness coefficient. When the handling strategy score is greater than a threshold, the handling method is determined to be AGV handling, otherwise the handling method is mixed handling. The material fragility coefficient is obtained based on whether the material is fragile or non-fragile; the personnel status score is calculated based on the number of bends per unit time; the cost-effectiveness coefficient is calculated based on the power of the AGV equipment and the electricity cost corresponding to the electricity consumption period.

[0053] Although specific embodiments are described herein, those of ordinary skill in the art will recognize that many other modifications or alternative embodiments are also within the scope of the present disclosure. For example, any of the functions and / or processing capabilities described in conjunction with a particular device or component may be performed by any other device or component. In addition, although various exemplary implementations and architectures have been described according to embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications to the exemplary implementations and architectures described herein are also within the scope of the present disclosure.

[0054] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0055] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present application. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A power engineering construction safety monitoring system based on deep learning, characterized in that: include: Data acquisition module, used to collect RGB images, LiDAR point clouds and depth maps of the construction site in real time; Intelligent analysis module, which is used to predict the risk level of material dumping, personnel falling, mechanical collision and electrical hazards in parallel based on multi-task deep learning models, and detect personnel status and material type when the risk of material dumping exceeds the threshold; The multi-task deep learning model includes: Shared feature extraction layer, used to extract RGB image features based on YOLOv8, process lidar point clouds based on PointNet++, extract spatial features, and extract deep features based on convolutional networks; The multimodal fusion layer is used to align RGB, point cloud and depth features based on the Transformer encoder to obtain fused features; Task-specific head network, including GRU, LSTM, Kalman filter, HRNet and ResNet-50 classification models, is used to output four types of risk prediction results based on fusion features, as well as dynamically trigger detection of personnel status and material type; Dynamic decision-making module, which is used to dispatch personnel, AGVs and drones based on reinforcement learning according to risk prediction results and risk priorities, send mechanical emergency stop instructions, and guide personnel to avoid risks in real time through alarm information; when dispatching personnel or AGVs to perform handling tasks, the handling decision is adjusted according to the personnel status, material type and AGV status, and the handling decision includes AGV handling and mixed handling; Feedback optimization module, used to optimize the model through incremental learning.

2. The power engineering construction safety monitoring system based on deep learning according to claim 1 is characterized in that: The data acquisition module comprises: A camera unit for acquiring RGB images and video streams of the construction site; A laser radar unit, used to obtain three-dimensional point cloud data of the construction site; Depth camera unit, used to obtain depth map of the construction site.

3. The power engineering construction safety monitoring system based on deep learning according to claim 1 is characterized in that: The shared feature extraction layer comprises: A visual branch unit, which is used to extract RGB image features based on the YOLOv8 backbone network and capture the location and category of targets, including materials, personnel, and equipment; A point cloud branch unit, which is used to process lidar data based on PointNet++ and extract spatial features, including obstacle density and motion trajectory; The depth map unit is used to extract depth features through a convolutional network to calculate the material tilt angle and the position of the person.

4. The power engineering construction safety monitoring system based on deep learning according to claim 1 is characterized in that: The task-specific head network includes: The material stability analysis unit is used to obtain the material stacking area identified by YOLOv8, calculate the inclination angle, height and center position of the material pile through the depth map and point cloud data corresponding to the material stacking area; and obtain the wind speed and ground inclination through lidar data; input the inclination angle, height, center position, wind speed and ground inclination of the material pile into the GRU network, and output the dumping risk level.

5. The power engineering construction safety monitoring system based on deep learning according to claim 4 is characterized in that: The task-specific head network includes: The personnel status detection unit is used to activate the HRNet branch when the dumping risk level is high, extract the skeleton key points from the RGB image, and count the number of bending times per unit time through the angle changes of the skeleton key points in consecutive frames to obtain the personnel status score; The material type recognition unit is used to obtain the material area image detected by YOLOv8 when the dumping risk level is high, input it into the ResNet-50 classification model, and output the material type as fragile material or non-fragile material.

6. The power engineering construction safety monitoring system based on deep learning according to claim 1 is characterized in that: The task-specific head network includes: The personnel fall warning unit is used to extract multiple key points of the human body from the RGB image through HRNet, calculate the angle of the key point connection line, determine whether it is in a dangerous posture based on the key point connection angle, and locate the distance between the person and the dangerous area through the depth map and point cloud data. It analyzes the action trend in continuous frames based on LSTM and outputs the fall probability.

7. The power engineering construction safety monitoring system based on deep learning according to claim 1 is characterized in that: The task-specific head network includes: The mechanical collision prediction unit is used to obtain the trajectory features extracted by PointNet++ and the mechanical position detected by YOLOv8, obtain the mechanical position and movement direction, predict the mechanical movement path in the future period based on Kalman filtering combined with historical trajectory point cloud data, and calculate the Euclidean distance between the mechanical movement path and the obstacle. When the Euclidean distance is less than the safety threshold, an early warning is triggered.

8. The power engineering construction safety monitoring system based on deep learning according to claim 1 is characterized in that: The task-specific head network includes: The electrical hazard detection unit is used to detect the visual features of cable damage and identify the cable damage area based on RGB images using the ResNet-50 classification model.

9. The power engineering construction safety monitoring system based on deep learning according to claim 1 is characterized in that: The dynamic decision module includes: Prioritization unit, used to generate risk queues in real time, ranked in order of electrical hazards, mechanical collisions, personnel falls, and material dumping; The reinforcement learning decision unit is used to construct the state space according to each risk level, AGV / UAV position, mechanical operation status, path feasibility coefficient, AGV power, UAV mission load and mechanical emergency stop response delay. The policy network and value network are collaboratively optimized based on the PPO algorithm. The policy network outputs action parameters, which include AGV path direction, UAV target probability, mechanical emergency stop binary decision and manual intervention binary decision. The value network predicts the state cumulative reward.

10. The power engineering construction safety monitoring system based on deep learning according to claim 5, characterized in that: The dynamic decision module includes: The handling decision unit is used to calculate the handling strategy score based on the material fragility coefficient, the personnel status score and the cost-effectiveness coefficient. When the handling strategy score is greater than the threshold, the handling method is determined to be AGV handling, otherwise the handling method is mixed handling. The material fragility coefficient is obtained based on whether the material is fragile or non-fragile; the personnel status score is calculated based on the number of bends per unit time; the cost-effectiveness coefficient is calculated based on the power consumption of the AGV equipment and the electricity cost corresponding to the electricity consumption period.

Images