Power operation site safety monitoring method based on multi-target tracking and related equipment

Abstract

The invention discloses a power equipment fault prediction method based on multi-modal data and related equipment, and belongs to the technical field of power operation safety monitoring, and the method comprises the steps: collecting the multi-modal data of a power operation site; inputting the data into a target detection model, determining a target category and position information in combination with a target identity re-association algorithm, and performing position tracking; performing preprocessing according to the channel state information of the real-time position of the target, performing attitude reasoning in combination with a deep neural network to obtain a target attitude, and determining an abnormal state of a target behavior; determining a position area of the target according to the real-time position of the target and an electronic fence technology; and when the target is in the dangerous operation area, determining the abnormal wearing state of the protection equipment of the target personnel based on the image of the target and by adopting a personal protection equipment detection model, and outputting early warning information through edge cloud cooperative early warning. High-precision safety monitoring of an electric power operation site is realized, the personnel behavior identification capability is improved, and the operation risk is comprehensively reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of power operation safety monitoring, and particularly relates to a power operation site safety monitoring method and related equipment based on multi-target tracking. Background Art

[0002] Safety monitoring of power operation sites is an important link in the power industry to ensure the safety of operators and the standardization of operations. Power operations usually involve high-voltage environments, high-altitude operations, and complex power equipment operations. Operators need to strictly abide by safety regulations when performing tasks to reduce risks such as electric shock, falls, and equipment damage. With the development of intelligent monitoring technologies, methods based on computer vision, sensor fusion, and big data analysis have been widely applied in the field of power operation safety monitoring to improve the intelligent management level of the operation process.

[0003] Existing power operation safety monitoring technologies mainly rely on traditional video surveillance and manual inspections to supervise the violation behaviors of operators and the safety status of operations. However, the traditional monitoring methods have the following limitations: on the one hand, relying solely on video surveillance is easily affected by environmental factors such as light changes and occlusions, resulting in a decrease in monitoring accuracy; on the other hand, the manual inspection method has hysteresis and subjective judgment errors, making it difficult to detect abnormal situations in a timely manner. In addition, although some intelligent monitoring methods have introduced computer vision technologies, there are still technical bottlenecks in multi-target tracking, human pose recognition, and environmental risk analysis, and they cannot meet the high-precision monitoring requirements of complex power operation sites.

[0004] Current power operation safety monitoring technologies have problems such as insufficient accuracy, low real-time performance, and late warning in target detection and tracking, human operation pose analysis, and environmental risk perception. Especially when the tracking of operators is lost due to factors such as pose changes and occlusions, existing methods are difficult to maintain a stable monitoring effect. Therefore, there is an urgent need for a power operation site safety monitoring method that can combine multi-modal data fusion, target identity re-association, multi-target tracking, and an intelligent warning mechanism to improve the accuracy and real-time performance of operation monitoring and reduce safety hazards. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a power operation site safety monitoring method and related equipment based on multi-target tracking for solving the technical problems of insufficient accuracy, low real-time performance, and late warning of existing power operation safety monitoring technologies.

[0006] The present invention adopts the following technical solutions: In a first aspect, the present invention provides a power operation site safety monitoring method based on multi-target tracking, including the following steps: Collect multi-modal data of the power operation site; Input the multi-modal data into the target detection model, and combine the target identity re-association algorithm to determine the target category and target location information, track the target location, and obtain the real-time target location; Preprocess according to the channel state information of the real-time target location, and combine with the deep neural network for pose inference to obtain the target pose, and determine the abnormal target behavior state according to the target pose; Determine the location area of the target according to the real-time target location and combine with the electronic fence technology; when the location area of the target is a dangerous operation area, based on the image of the target and using the personal protective equipment detection model, determine the abnormal state of the target's protective equipment wearing; Based on the abnormal target behavior state and the abnormal state of the target's protective equipment wearing, perform edge-cloud collaborative warning and output warning information.

[0007] Preferably, it also includes equipment anomaly monitoring and operation safety training and simulation; Equipment anomaly monitoring, analyze the temperature state of the equipment based on infrared thermal imaging technology, calculate the temperature gradient to judge whether there is a risk of overheating or poor contact of the equipment, and analyze the abnormal current situation of the equipment based on the current fluctuation index; when the temperature gradient exceeds the set threshold, the system triggers a warning to indicate that the equipment may have a risk of overheating or poor contact; when the current fluctuation index exceeds the set threshold, the system triggers an alarm to indicate that the equipment may have problems such as short circuit or unstable current; Edge-cloud collaborative warning, use the edge computing device to perform real-time calculation on the data and upload it to the cloud server for in-depth analysis. When detecting the abnormal target behavior state and the abnormal state of the target's protective equipment wearing, trigger an audible and visual alarm, and push the alarm and intelligent voice broadcast to the management personnel through the mobile terminal; Operation safety training and simulation, based on virtual reality technology, construct an operation accident scenario according to historical monitoring data for operation safety training.

[0008] Preferably, the multi-modal data includes: RGB images of the operation site, equipment temperature state information, human body pose information, 3D point cloud data, and operation environment information; The steps of collecting multi-modal data of the power operation site specifically include: Use a visible light camera to collect RGB images of the operation site; Use an infrared thermal imager to collect equipment temperature state information and monitor whether there is overheating; Use a WiFi sensing device to collect the disturbance of the wireless signal by the operation personnel using the channel state information, and map out the human body pose information through a deep neural network model, fuse it with the visual data, and perform non-contact pose detection of the operation personnel; Lidar is used to obtain the 3D point cloud data of the site and construct an operation scenario model; Wind speed sensors, temperature sensors and humidity sensors are used to monitor the operation environment information.

[0009] Preferably, the target detection model is the YOLOv8 model, and the target category and target position information are calculated by the following formula:

[0010] In the formula, is the output vector of the detection network, including the target category and the target position information; represents the neural network mapping of YOLOv8; is the input image.

[0011] Preferably, the relationship between the channel state information and the target attitude is expressed as follows:

[0012] In the formula, is the channel state information; is the channel response in a barrier-free environment; is the influence coefficient of the human body on the channel; is the signal propagation delay; f is the frequency of the signal; t is the time of the signal; j is the imaginary unit.

[0013] Preferably, the steps of determining the position area of the target according to the real-time position of the target and combining with the electronic fence technology specifically include: based on the deep learning semantic segmentation technology, automatically identifying the energized equipment and dangerous operation areas in the operation site according to the real-time position of the target, establishing an intelligent electronic fence, real-time monitoring and warning the behavior of unauthorized personnel entering the high-risk operation area, and judging whether the operation personnel correctly wear safety helmets, insulating gloves and other protective equipment based on the personal protective equipment detection model; The personal protective equipment detection model adopts the following behavior recognition loss function:

[0014] In the formula, is the true label, indicating whether there is a violation; is the model prediction probability; L is the behavior recognition loss function.

[0015] In a second aspect, the present invention provides a power operation site safety monitoring system based on multi-target tracking, including: A multi-modal data acquisition module for collecting multi-modal data of the power operation site; The target detection and position tracking module is used to input multi-modal data into the target detection model, and combine the target identity re-association algorithm to determine the target category and target position information, track the position of the target, and obtain the real-time position of the target; The target pose inference and abnormal behavior detection module is used to preprocess according to the channel state information of the target real-time position, combine with a deep neural network for pose inference to obtain the target pose, and determine the abnormal state of the target behavior according to the target pose; The target violation behavior detection module is used to determine the position area of the target according to the target real-time position and combine with the electronic fence technology; when the position area of the target is a dangerous operation area, based on the image of the target and using a personal protective equipment detection model, determine the abnormal state of the target's protective equipment wearing; The warning module is used to perform edge-cloud collaborative warning according to the target behavior abnormal state and the target protective equipment wearing abnormal state, and output warning information.

[0016] In a third aspect, the present invention provides a computer-readable storage medium, including a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned power operation site safety monitoring method based on multi-target tracking are implemented.

[0017] In a fourth aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned power operation site safety monitoring method based on multi-target tracking are implemented.

[0018] In a fifth aspect, the present invention provides an electronic device, including a computer program, and when the computer program is executed by the electronic device, the steps of the above-mentioned power operation site safety monitoring method based on multi-target tracking are implemented.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects: The present invention discloses a safety monitoring method for power operation sites based on multi-object tracking. By combining multi-object tracking technology with WiFi sensing, deep learning object detection, infrared thermal imaging, and edge-cloud collaborative computing, high-precision safety monitoring of power operation sites is achieved, and the real-time perception ability of the states of operators and equipment is improved. The target identity re-association technology is adopted to improve the continuous tracking accuracy of operators and tools, and overcome the problems of light change and occlusion interference. The posture estimation based on WiFi channel state information effectively enhances the human behavior recognition ability, and accurately detects unsafe operations such as falling and not wearing a safety rope during high-altitude operations. Combined with the electronic fence technology, the situation of unauthorized personnel entering the dangerous area is monitored in real time, and multi-modal early warnings are provided to ensure that safety risks can be identified and handled in a timely manner. The equipment anomaly detection is based on infrared thermal imaging and current fluctuation index analysis, realizing early warning of abnormal temperature and current fluctuation of high-voltage equipment, and reducing the safety hazards caused by equipment failures. In addition, virtual reality technology is used to reproduce operation accident scenes and train standard operation procedures, improving the safety awareness and emergency handling ability of operators, and thus comprehensively improving the safety management level of power operation sites.

[0020] Furthermore, the present invention combines the deep learning object detection algorithm with the target identity re-association technology to achieve accurate identification and continuous tracking of operators, tools, and equipment, overcome the problems of light change and occlusion interference, and significantly improve the target tracking accuracy in complex scenarios. Through the multi-object tracking technology, multiple targets can be monitored simultaneously, including the states and trajectories of personnel, tools, and equipment, ensuring comprehensive coverage of the operation site.

[0021] Furthermore, the present invention realizes non-contact human posture estimation based on WiFi channel state information and deep neural network, can accurately detect unsafe behaviors such as falling and not wearing a safety rope during suspended operations, and solves the limitations of traditional vision methods under occlusion or low-light conditions. Combining WiFi sensing and computer vision technology enhances the accuracy and robustness of behavior recognition.

[0022] Furthermore, the present invention uses deep learning semantic segmentation technology to automatically identify live equipment and dangerous areas, establish an intelligent electronic fence, and monitor the behavior of unauthorized personnel entering high-risk areas in real time. Based on the object detection model, it automatically identifies whether operators are correctly wearing protective equipment such as safety helmets and insulating gloves, reducing safety accidents caused by improper protection.

[0023] Furthermore, the present invention analyzes the equipment temperature gradient to early warn of the risks of equipment overheating or poor contact, reducing the safety hazards caused by equipment failures. Based on the current fluctuation index, it monitors the abnormal current situation of equipment in real time, warns of short-circuit or current instability problems, and improves the reliability of equipment operation.

[0024] Furthermore, the present invention uses edge computing devices to process data in real time, improve system response speed, and ensure that abnormal behavior or equipment failure can be discovered in time. The cloud server combines historical data for in-depth analysis and provides multi-modal warnings, including sound and light alarms, mobile terminal push, and voice broadcasts, to ensure that security risks can be communicated and handled in a timely manner.

[0025] Furthermore, the present invention constructs operation accident scenarios based on historical monitoring data, and combines VR technology for immersive safety training to improve the safety awareness and emergency response capabilities of operators. VR simulation of standard operating procedures helps operators master standardized operations and reduce human errors.

[0026] Furthermore, the present invention monitors the behavior of operators and the wearing of protective equipment in real time, significantly reducing safety accidents caused by illegal operations. Through infrared thermal imaging and current fluctuation analysis, equipment abnormalities can be discovered in advance, reducing unplanned downtime and safety accidents. Through multi-modal data collection, intelligent analysis and early warning mechanisms, the safety management level of power operation sites can be comprehensively improved.

[0027] The present invention realizes intelligent safety monitoring of power operation sites through the deep integration of technologies such as multi-target tracking, WiFi perception, infrared thermal imaging, edge-cloud collaborative computing and virtual reality. High-precision target detection and tracking solves the problem of target recognition in complex scenarios; enhanced behavior recognition capabilities accurately detect unsafe operations; intelligent electronic fences and violation detection, real-time monitoring of dangerous areas and compliance of protective equipment; early warning of equipment abnormalities to reduce safety hazards caused by equipment failures; edge-cloud collaborative early warning mechanism to ensure timely identification and handling of safety risks; VR simulation training to enhance the safety awareness and emergency response capabilities of operators. It significantly reduces the safety risks at power operation sites, improves operating efficiency and safety management level, and has broad application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the architecture of a power operation site safety monitoring system based on multi-target tracking according to an embodiment of the present invention; Figure 2 The present invention is a flowchart of a method for on-site safety monitoring of electric power operations based on multi-target tracking according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In the description of the present invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0031] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0032] It should be further understood that the term "and / or" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the present invention, the character " / " generally represents an "or" relationship between the preceding and following associated objects.

[0033] It should be understood that although terms such as first, second, and third may be used in the embodiments of the present invention to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0034] Depending on the context, the word "if" as used herein can be interpreted as "when", "while", "in response to determining", or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" can be interpreted as "when determined", "in response to determining", "when detected (stated condition or event)", or "in response to detecting (stated condition or event)".

[0035] The present invention provides a method for safety monitoring of a power operation site based on multi-target tracking, including the following steps: Data acquisition: Deploy multiple types of sensors at the power operation site, including visible light cameras, infrared thermal imagers, WiFi sensing devices, lidar, and environmental sensors, to collect information on operating personnel, tools, equipment, and the environment; Object Detection and Tracking: Use deep learning object detection algorithms to process the collected data, detect workers, tools, equipment, and dangerous areas in the operation site, and track the trajectories of the objects by combining object identity re-association technology; WiFi Sensing and Pose Estimation: Analyze the poses of workers based on WiFi channel state information, and combine computer vision technology to detect whether workers have unsafe operation behaviors such as falling, working at height without wearing a safety rope, etc.; Electronic Fence and Violation Detection: Use deep learning semantic segmentation technology to identify live equipment and dangerous areas, establish an intelligent electronic fence, and judge whether workers are wearing safety helmets, insulating gloves, and other protective equipment correctly based on the object detection model; Equipment Abnormality Monitoring: Analyze the temperature state of equipment based on infrared thermal imaging technology, calculate the temperature gradient to judge whether the equipment has risks of overheating or poor contact, and analyze the abnormal current situation of the equipment based on the current fluctuation index; Edge-Cloud Collaborative Warning: Use edge computing devices to perform real-time calculations on data and upload them to the cloud server for in-depth analysis. When abnormal operation behaviors, equipment failures, or environmental abnormalities are detected, trigger audible and visual alarms and push alarm information to managers; Operation Safety Training and Simulation: Build operation accident scenarios based on historical monitoring data, and combine virtual reality technology to conduct operation safety training to improve the safety awareness and emergency response capabilities of workers.

[0036] Among them, object detection and tracking adopt a Transformer-based object detection model and combine object identity re-association technology to improve the multi-object tracking accuracy in complex scenarios.

[0037] The WiFi sensing device uses channel state information to collect the influence of workers on wireless signals, and maps out human pose information through a deep neural network model, which is fused with visual data to improve the accuracy of pose recognition.

[0038] The electronic fence is based on deep learning semantic segmentation technology, automatically identifies different areas of the operation site, and real-time monitors and warns of the behavior of unauthorized personnel entering high-risk areas.

[0039] Equipment abnormality monitoring calculates the temperature gradient of the equipment based on infrared thermal imaging data. When the temperature gradient exceeds the set threshold, the system triggers an alarm, indicating that the equipment may have risks of overheating or poor contact.

[0040] Equipment abnormality monitoring obtains the current data of the equipment based on current sensors and calculates the current fluctuation index. If the current fluctuation index exceeds the set threshold, it is judged that the equipment may have problems such as short circuit or unstable current, and an alarm is triggered.

[0041] For the detection of violations, a deep learning object detection model is adopted to automatically identify the wearing situation of safety protection equipment of operators, and combined with a behavior recognition algorithm to analyze whether the operations of operators comply with the standard operation procedures.

[0042] In the edge-cloud collaborative computing architecture, edge computing devices use lightweight deep learning models for object detection and preliminary analysis, and the cloud server performs deep reasoning based on large-scale data sets to improve the response speed and analysis accuracy of the monitoring system.

[0043] When detecting abnormal operation behaviors, equipment failures or environmental abnormalities, a multimodal early warning mechanism is adopted, including on-site audible and visual alarms, mobile terminal push alarms and intelligent voice broadcasts, to ensure that operators can obtain safety information in a timely manner.

[0044] It also includes constructing a power operation accident simulation scenario based on virtual reality technology, enabling operators to experience the accident occurrence process and emergency response measures through immersive simulation, so as to improve the effect of operation safety training.

[0045] In another embodiment of the present invention, a terminal device is provided. The terminal device includes a processor and a memory. The memory is used to store a computer program. The computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the power operation site safety monitoring method based on multi-object tracking.

[0046] In another embodiment of the present invention, a storage medium is further provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a terminal device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, and the operating system of the terminal is stored in this storage space. And, one or more instructions suitable for being loaded and executed by the processor are also stored in this storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk memory.

[0047] One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the method for safety monitoring of the power operation site based on multi-object tracking in the above embodiments.

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0049] The present invention provides a method for safety monitoring of the power operation site based on multi-object tracking. Through technologies such as multi-modal data collection, deep learning object detection, WiFi perception of human body posture recognition, electronic fence monitoring, environmental perception, illegal behavior recognition, and edge-cloud collaborative computing, real-time monitoring, analysis, and early warning of the power operation site are realized. The present invention improves the safety of the operation site through intelligent monitoring technology, reduces personnel's illegal operations, and reduces the risk of accidents.

[0050] Figure 1 This is a schematic diagram of the system architecture of a safety monitoring system for a power operation site based on multi-object tracking according to an embodiment of the present invention; as can be seen from the figure, this system includes: On-site data collection: including visible light cameras, infrared thermal imagers, WiFi sensing devices, lidar, and environmental sensors to obtain multi-modal data of the operation site; Edge computing unit: deployed on-site for preliminary data analysis to improve data processing efficiency; Cloud intelligent analysis platform: based on deep learning algorithms for high-precision target detection, anomaly analysis, and safety risk prediction; Safety management system: continuously monitor the behavior of operators and the operating status of equipment, and issue safety warnings; Visualization terminal: supports mobile and PC terminals for operators and managers to view monitoring data and warning information.

[0051] Figure 2 The following is a flowchart of a safety monitoring method for a power operation site based on multi-object tracking according to an embodiment of the present invention. As can be seen from the figure, the method includes the following steps: Step 1: Data collection Deploy a variety of sensors at the power operation site to collect information on operators, tools, equipment, and the environment.

[0052] The visible light camera is used to collect RGB images of the operation site, and the resolution is set to 1920×1080; The infrared thermal imager collects the temperature status of the equipment to monitor whether there is overheating; The WiFi sensing device uses channel state information (CSI) for non-contact posture detection of operators; The lidar obtains 3D point cloud data of the site to construct an operation scene model; Wind speed, temperature, and humidity sensors monitor the operation environmental conditions to evaluate the impact of weather on the operation.

[0053] Step 2: Target detection and tracking Use deep learning algorithms to perform target detection and multi-object tracking on operators, tools, and equipment at the operation site.

[0054] YOLOv8 model is used for target detection, and the output includes the category, location, and confidence of the target. For target tracking, the Transformer is combined with the target identity re-association (Re-ID) algorithm to ensure continuous tracking of the target in the scene.

[0055] The output of the target detection network is the target category and the bounding box , where:

[0056] In the formula, is the output vector of the detection network, including the target category and location information; The neural network mapping representing YOLOv8; is the input image.

[0057] Step 3: WiFi Sensing and Pose Estimation Based on WiFi CSI (Channel State Information), human pose estimation is achieved to avoid the influence of light changes and occlusion problems. After preprocessing the WiFi CSI signal, it is input into a deep neural network for pose inference.

[0058] The relationship between the change of WiFi signal and human pose is expressed as follows:

[0059] In the formula, is the CSI channel state information; is the channel response in a barrier-free environment; is the influence coefficient of the human body on the channel; is the signal propagation delay; f is the frequency of the signal; t is the time of the signal; j is the imaginary unit.

[0060] Step 4: Electronic Fence and Violation Detection Based on the deep learning semantic segmentation technology (DeepLabV3+), the live electrical equipment and dangerous areas at the operation site are automatically identified to establish an intelligent electronic fence; When unauthorized personnel or objects enter the high-risk area, a warning signal is triggered; Adopt a deep learning PPE detection model to judge whether the operators are wearing protective equipment such as safety helmets and insulating gloves correctly.

[0061] The following behavior recognition loss function is adopted for violation detection:

[0062] In the formula: is the true label (whether there is a violation); is the model prediction probability; L is the behavior recognition loss function.

[0063] Step 5: Edge-Cloud Collaborative Computing and Safety Warning The edge computing device uses a lightweight deep learning model for real-time calculation and uploads the data to the cloud; The cloud server conducts historical data analysis and in-depth reasoning to predict the operation safety risks; Multi-modal warning mechanism: Adopt audible and visual alarms, mobile terminal push, and intelligent voice broadcast to ensure that the operators can obtain safety information in a timely manner.

[0064] Step 6: Operation Safety Training and Simulation Based on historical monitoring data, construct a VR simulation scenario for job training; Accident scene reconstruction: Based on actual accident data, simulate the accident occurrence process to improve personnel's emergency response capabilities; Demonstration of standard operation procedures: Provide drills for standard operation procedures to improve job compliance.

[0065] The present invention realizes intelligent safety monitoring of the power operation site through multi-target tracking, deep learning, WiFi sensing, thermal imaging, and edge-cloud collaborative computing, improving the accident prevention ability. It improves the target recognition accuracy and reduces the influence of light and occlusion; it monitors violations in real time and reduces the risk of operation accidents; it enhances the effect of job safety training and improves personnel's safety awareness. The present invention adopts a variety of advanced technologies to realize the safety monitoring of the power operation site, providing an efficient and accurate intelligent management solution for the power industry.

[0066] To illustrate the implementation effect of the present invention in more detail, the following introduces an application example of the present invention in substation maintenance operations in combination with actual application scenarios.

[0067] Application background A 220 kV substation conducts annual maintenance operations, and the operation contents include primary equipment inspection, secondary equipment testing, high-voltage equipment replacement, etc. The operation site has relatively high risks and the following potential safety hazards exist: 1. When operating personnel work near high-voltage equipment, an electric shock accident may occur if the operation is improper; 2. Some personnel do not wear protective equipment such as insulating gloves and safety helmets as required; 3. During high-altitude operations, there is a situation where safety ropes are not used correctly; 4. The on-site environment is complex, and weather changes may affect operation safety.

[0068] To ensure the safety of maintenance operations, the substation introduces the safety monitoring method for power operation sites based on multi-target tracking proposed by the present invention to realize real-time monitoring and safety early warning of the operation process.

[0069] Implementation steps Step 1: Multi-modal data collection at the operation site Deploy RGB cameras, WiFi sensing devices, infrared thermal imagers, lidar, and environmental sensors at the substation site to collect information on operating personnel, tools, equipment, and the environment; RGB camera: Real-time collect the on-site video stream (resolution 1920×1080, frame rate 30 FPS) to obtain information on personnel and equipment; WiFi sensing device: Use channel state information (CSI) to detect the posture and movement trajectory of personnel; Infrared thermal imager: Monitor the temperature distribution of high-voltage equipment to determine whether there is a risk of overheating. LiDAR: Establish a three-dimensional point cloud model to assist in detecting the positions of operators and equipment. Wind speed, temperature and humidity sensors: Monitor weather conditions and evaluate the safety of operations in combination with historical data.

[0070] Step 2: Object detection and operation behavior analysis Operator detection and tracking: Use the YOLOv8+Transformer object tracking algorithm to detect the position of the operator and assign a unique identification ID. Combine Re-ID technology to ensure the identity consistency of personnel under multiple camera views.

[0071] Tool and equipment monitoring: Detect the correct use of tools such as insulating rods, wrenches, and grounding wires; detect high-voltage equipment to ensure its normal state.

[0072] Operator posture analysis: Combine WiFi CSI and deep neural network to detect whether the operator's posture meets the standard operation requirements. For example, when an abnormal operator posture is detected (such as standing unsteadily for a long time or falling), the system automatically triggers an alarm.

[0073] Step 3: Intelligent detection of illegal operations Identification of safety protection equipment Use a personal protective equipment (PPE) detection model to determine whether operators are wearing safety helmets, insulating gloves, etc. If it is found that a certain person is not wearing a safety helmet, the system automatically identifies and pushes an alarm message to the management personnel.

[0074] Detection of illegal high-altitude operations Use posture recognition and electronic fence technology to detect whether high-altitude operators are correctly wearing safety ropes; the electronic fence delimits a safety area. If it is detected that a person is working at height without wearing a safety rope, the system immediately triggers an alarm.

[0075] Step 4: Equipment anomaly detection Equipment status monitoring based on infrared thermal imaging, collect infrared thermal imaging data of high-voltage equipment, and calculate the temperature gradient: If it exceeds the set threshold, it indicates that the equipment may have a risk of overheating or poor contact, and the system automatically sends an alarm to the maintenance personnel.

[0076] Analysis of abnormal current fluctuations The current signal of the high-voltage equipment is collected through the sensor, and the current fluctuation index is calculated; if it exceeds the set threshold, it indicates that the equipment may have a short circuit or current instability problem, and the system automatically alarms.

[0077] Step 5: Security warning and intelligent decision-making Edge computing unit: performs real-time analysis on local devices at the work site to quickly respond to safety incidents; Cloud-based intelligent analysis: Upload operation data to the cloud, combine historical data to make trend predictions, and provide optimized safety management suggestions; Multimodal warning: On-site sound and light alarm: remind operators to correct violations; Mobile push: Send operational risk information to managers; Voice broadcast: remind operators of the current risk status.

[0078] Step 6: VR Simulation Safety Training Combined with historical monitoring data from the work site, VR simulation scenes are constructed for safety training of operators; Through VR simulation of accident scenarios, operators can understand the cause of the accident and improve their emergency response capabilities.

[0079] Monitoring results During the maintenance operation of the substation, the monitoring effect of the method of the present invention is as follows: 1. Detected 2 violations of not wearing helmets in real time and pushed warning information; 2. If it is detected that an operator is not wearing a safety rope when working at height, the system will automatically trigger an alarm; 3. Based on infrared thermal imaging analysis, it was found that the temperature of a transformer was abnormal, and equipment failure was warned in advance; 4. Based on the current fluctuation analysis, it was detected that a certain high-voltage switchgear had unstable current and maintenance was arranged in advance; 5. Improve the safety awareness and operational standardization of operators through VR simulation training.

[0080] This application example shows that the power operation site safety monitoring method based on multi-target tracking provided by the present invention can accurately detect the behavior of operators, the use of tools and the operating status of equipment, effectively improve the safety of power maintenance operations, reduce safety accidents caused by human errors, and has high practical value.

[0081] The example of the present invention reduces safety accidents: the intelligent monitoring system can detect illegal operations in advance and reduce the accident incidence rate; improves operation compliance: it supervises the safety behavior of operators in real time to ensure that the operation process meets the standards; optimizes equipment management: based on infrared detection and current fluctuation analysis, it predicts equipment failures in advance and reduces unplanned outages; enhances the effect of safety training: through VR simulation training, it improves the safety awareness and emergency handling ability of operators.

[0082] In summary, the present invention relates to a safety monitoring method for power operation sites based on multi-object tracking. The method includes steps such as data collection, target detection and tracking, WiFi perception and pose estimation, electronic fence and illegal behavior detection, equipment anomaly monitoring, edge-cloud collaborative warning, and operation safety training. By using deep learning object detection combined with target identity re-association technology, it realizes the accurate tracking of operators and tools; based on the pose estimation of WiFi channel state information, it improves the accuracy of fall detection and illegal operation recognition; it analyzes the equipment state through infrared thermal imaging and current fluctuation index to warn of equipment anomalies in advance; combined with a multi-modal warning mechanism, it ensures the timely identification and handling of safety risks. In addition, it constructs an operation accident scenario based on virtual reality technology to enhance the effect of safety training for operators. The present invention improves the monitoring accuracy and intelligent level of power operation sites and reduces safety risks.

[0083] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0084] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0085] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present invention can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0086] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal and method can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0087] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0088] In addition, the functional units in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0089] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random-access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0090] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0091] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions for implementing the steps of the process Figure 1 in one process or a plurality of processes and / or blocks Figure 1 or steps for implementing the functions specified in one block or a plurality of blocks.

[0093] The above is only to illustrate the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any modification made on the basis of the technical solution according to the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for safety monitoring of power operation sites based on multi-target tracking, characterized in that: The following steps are involved: Collect multimodal data at power operation sites; Input the multimodal data into the target detection model, and combine it with the target identity re-association algorithm to determine the target category and target location information, track the target position, and obtain the target's real-time location; Based on the channel state information of the target's real-time position and preprocessing, the deep neural network is combined to perform posture reasoning to obtain the target posture, and the abnormal state of the target behavior is determined based on the target posture; Determine the location area of ​​the target based on the real-time location of the target and in combination with the electronic fence technology; when the location area of ​​the target is a dangerous operation area, determine the abnormal state of the target's protective equipment based on the image of the target and using the personal protective equipment detection model; Based on the abnormal state of target behavior and the abnormal state of target protective equipment, edge-cloud collaborative warning is carried out and warning information is output.

2. The method for safety monitoring of power operation sites based on multi-target tracking according to claim 1 is characterized in that: It also includes equipment anomaly monitoring and operational safety training and simulation; The device abnormality monitoring analyzes the device temperature status based on infrared thermal imaging technology, calculates the temperature gradient to determine whether the device has the risk of overheating or poor contact, and analyzes the device current abnormality based on the current fluctuation index; when the temperature gradient exceeds the set threshold, the system triggers an early warning, indicating that the device may have the risk of overheating or poor contact; when the current fluctuation index exceeds the set threshold, the system triggers an alarm, indicating that the device may have a short circuit or current instability problem; The edge-cloud collaborative warning uses edge computing devices to calculate data in real time and upload it to the cloud server for in-depth analysis. When abnormal behavior of the target and abnormal wearing of the target protective equipment are detected, an audible and visual alarm is triggered, and an alarm and intelligent voice broadcast are pushed to the management personnel through the mobile terminal; The operation safety training and simulation is based on virtual reality technology and constructs operation accident scenarios according to historical monitoring data to conduct operation safety training.

3. The method for safety monitoring of electric power operation site based on multi-target tracking according to claim 1 is characterized in that: The multimodal data includes: RGB images of the work site, equipment temperature status information, human posture information, 3D point cloud data and work environment information; The step of collecting multimodal data at the power operation site specifically includes: Use visible light cameras to collect RGB images of the work site; Use infrared thermal imagers to collect equipment temperature status information and monitor whether there is overheating; WiFi sensing equipment is used to collect the disturbance of wireless signals by operators using channel status information, and human posture information is mapped out through a deep neural network model, which is then integrated with visual data to perform non-contact posture detection of operators. Use laser radar to obtain 3D point cloud data on site and build a work scene model; Wind speed sensors, temperature sensors and humidity sensors are used to monitor working environment information.

4. The method for safety monitoring of power operation sites based on multi-target tracking according to claim 1 is characterized in that: The target detection model is a YOLOv8 model, and the target category and target location information are calculated using the following formula: In the formula, is the output vector of the detection network, containing the target category and target location information; Represents the neural network map of YOLOv8; is the input image.

5. The method for safety monitoring of power operation site based on multi-target tracking according to claim 1 is characterized in that: The relationship between channel state information and target posture is expressed as follows: In the formula, is the channel state information; Channel response in an obstacle-free environment; is the influence coefficient of human body on the channel; is the signal propagation delay; f is the signal frequency; t is the signal time; j is the imaginary unit.

6. The method for safety monitoring of electric power operation site based on multi-target tracking according to claim 1 is characterized in that: The step of determining the location area of ​​the target based on the real-time location of the target and in combination with the electronic fence technology specifically includes: based on the deep learning semantic segmentation technology, automatically identifying the live equipment and dangerous working areas at the work site according to the real-time location of the target, establishing an intelligent electronic fence, real-time monitoring and early warning of the behavior of unauthorized personnel entering the high-risk working area, and judging whether the operating personnel are correctly wearing safety helmets, insulating gloves and other protective equipment based on the personal protective equipment detection model; The personal protective equipment detection model uses the following behavior recognition loss function: In the formula, is the real label, indicating whether it is a violation; is the model prediction probability; L is the behavior recognition loss function.

7. A power operation site safety monitoring system based on multi-target tracking, characterized in that: include: Multimodal data acquisition module, used to collect multimodal data at the power operation site; The target detection and location tracking module is used to input multimodal data into the target detection model, and combine it with the target identity re-association algorithm to determine the target category and target location information, track the target, and obtain the target's real-time location; The target posture reasoning and abnormal behavior detection module is used to perform preprocessing based on the channel state information of the target's real-time position, perform posture reasoning in combination with a deep neural network, obtain the target posture, and determine the abnormal state of the target behavior based on the target posture; The target violation detection module is used to determine the target's location area based on the target's real-time location and combined with electronic fence technology; When the location area of ​​the target is a dangerous operation area, determining the abnormal state of the target's protective equipment based on the image of the target and using a personal protective equipment detection model; The early warning module is used to conduct edge-cloud collaborative early warning and output early warning information based on abnormal target behavior and abnormal wearing status of target protective equipment.

8. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1-6.

9. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable in the processor, wherein the processor implements the steps of the method for safety monitoring of power operation sites based on multi-target tracking as described in any one of claims 1 to 6 when executing the computer program.

Images