Radar and infrared fusion-based human body recognition system for power plant mistaken running detection

Abstract

The invention discloses a radar and infrared fusion-based human body recognition system for power plant mistaken intrusion detection, which can realize all-weather and accurate recognition and positioning alarm of personnel intrusion. Comprising a sensing layer, a processing layer and an output layer. The sensing layer comprises an infrared camera and a laser radar which are fixedly installed on the same installation support, the infrared camera and the laser radar keep a fixed spatial relative position relation, the optical axes of the infrared camera and the laser radar point to the same monitoring area, thermal imaging images are collected through the infrared camera, three-dimensional point cloud data are obtained through the laser radar, and the three-dimensional point cloud data are sent to the monitoring layer. Performing time synchronization; the processing layer and a processor are internally provided with a personnel detection deep learning model, and the processing layer comprises an image and point cloud preprocessing module, a multi-modal fusion detection module and a human body detection and recognition module; the output layer comprises an alarm unit and a control center display terminal interface; and the alarm unit triggers the sound-light alarm unit according to a preset strategy, so that monitoring personnel can perform field management conveniently.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence safety monitoring, and in particular to a human body recognition system based on radar and infrared fusion for detecting accidental entry into power plants. It can identify personnel who have accidentally entered dangerous areas around the clock and accurately locate and alarm them. Background Technology

[0002] High-risk work areas such as power plants have extremely high requirements for personnel access control; any unauthorized or accidental entry could lead to serious accidents. However, traditional monitoring methods have limited reliability in complex environments. For example, conventional video surveillance relies on visible light, but its use is prohibited in power plant settings due to security requirements. While infrared thermal imaging cameras can detect human heat sources in the dark, infrared images suffer from low contrast and high noise levels against high-temperature equipment backgrounds, and lack distance and depth information, making false detections likely. On the other hand, while lidar sensors can acquire 3D point cloud data for object detection and ranging within a certain range, they struggle to distinguish object attributes, easily confusing workers with other debris. Furthermore, lidar reliability decreases under interference from high-temperature steam and smoke. Existing single-sensor solutions cannot meet the 24 / 7 personnel intrusion detection needs of complex environments like power plants. Therefore, to ensure personal safety and prevent accidents, there is an urgent need for an intelligent monitoring system that integrates information from multiple sensors. Summary of the Invention

[0003] To address the shortcomings of the existing technologies, this invention provides a human body recognition system based on radar and infrared fusion for power plant intrusion detection. By deeply fusing infrared thermal imaging and lidar data, and combining the aforementioned deep learning model for personnel detection, it achieves all-weather, accurate identification and location alarm for personnel intrusion.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a human body recognition system based on radar and infrared fusion for detecting accidental entry into power plants, comprising a three-layer structure of a perception layer, a processing layer, and an output layer; The perception layer includes an infrared camera and a lidar fixedly mounted on the same mounting bracket. The infrared camera and lidar maintain a fixed spatial relative position, with their optical axes pointing to the same monitoring area, so that the infrared imaging field of view and the lidar scanning field of view overlap and cover the target area. The infrared camera acquires thermal imaging images, and the lidar acquires three-dimensional point cloud data, and the data is synchronized in time. The processing layer, with a built-in deep learning model for person detection, includes: The image and point cloud preprocessing modules preprocess the thermal imaging images acquired by the infrared camera and the 3D point cloud data acquired by the lidar, respectively. The multimodal fusion detection module is used to execute a deep learning-based cross-modal feature fusion algorithm, which jointly analyzes the thermal target region in the infrared image and the spatial structure features in the point cloud to complete the detection, 3D localization and trajectory estimation of personnel targets. When the human body detection and recognition module detects that a person has illegally entered the monitored area, it generates the location coordinates of the intruder and alarm information, and sends them to the output layer through the communication interface for alarm linkage. The output layer includes an alarm unit and a control center display terminal interface. The alarm unit is used to receive alarm information and target location coordinates sent by the processing layer, and triggers the audible and visual alarm unit according to a preset strategy. The control center display terminal, which is connected to the control center display terminal interface, is used to display the personnel location, alarm status and related information pushed by the processing layer in real time, so that monitoring personnel can perform on-site management.

[0005] Furthermore, the perception layer uses an external parameter calibration and time synchronization mechanism to ensure that the infrared image and point cloud data correspond in time and space.

[0006] Furthermore, when the image and point cloud preprocessing module performs preprocessing on the infrared image, it performs the following operations: Median filtering is used to denoise the original infrared image. The median value of each pixel is replaced within its neighborhood window to suppress random noise. Super-resolution reconstruction is performed using bicubic interpolation, which generates interpolated pixel values ​​by weighted combination of neighboring pixels, thereby improving the clarity of distant and small targets. Homomorphic filtering is used to normalize the illumination of the image, thereby enhancing details by suppressing low-frequency illumination components and enhancing high-frequency reflection components.

[0007] Furthermore, the image and point cloud preprocessing module preprocesses the point cloud data, including: (1) Based on the external parameter calibration, transform the lidar point cloud from the radar coordinate system to the infrared camera coordinate system; (2) Based on the camera intrinsic parameters, the pixel coordinates are obtained by projecting the 3D point cloud points onto the image plane; (3) Depth map generation and pixel-level alignment.

[0008] Furthermore, the process of depth map generation and pixel-level alignment includes: (1) Hole filling and interpolation: For pixels hit by the point cloud, their corresponding depth value or Euclidean distance is used as the gray value; for pixels not hit by any point, they are marked as blank; then, the blank area is filled by bilinear interpolation according to the distance weight of the surrounding effective depth points, thereby completing the sparse area. (2) Depth map generation and normalization: Obtain a depth grayscale map with the same size as the infrared image and spatially aligned, and normalize the depth values ​​according to the maximum detection distance. Normalize to 0–255 grayscale: (3) Output a depth map that is the same size as the infrared image and spatially aligned, for use as cross-modal fusion input.

[0009] Furthermore, the cross-modal fusion detection module includes: The GhostHGNetV2 backbone network is used to extract multi-scale features from infrared images and depth maps. It uses the HGStem module and Ghost convolution to generate efficient feature representations. The ASF-P2 multi-scale feature fusion module performs concatenation and fusion of features at different scales, and enhances small target features through P2-level low-level features and CBAM attention. The DPDH module employs dynamic convolution in both the classification and regression branches, adaptively adjusting the convolution kernel weights to achieve more accurate target classification and bounding box localization, outputting human category confidence, two-dimensional bounding boxes, and three-dimensional spatial coordinates recovered based on depth information.

[0010] Furthermore, the multimodal fusion detection module jointly analyzes the thermal target region in the infrared image with the spatial structure features in the point cloud, specifically including: 1) Training data construction: The infrared-lidar synchronous fusion data collected in the actual power plant scenario is used as the training set, and the bounding boxes and categories of personnel targets are marked on the infrared images. The categories are human / non-human, which are used for supervised training. 2) Loss Function Design: To simultaneously optimize classification and localization performance, a comprehensive loss function is adopted, including: Localization Loss: Using CIoU loss, which comprehensively considers factors such as bounding box overlap (IoU), center point distance, and aspect ratio consistency, the localization accuracy of the bounding box is improved.

[0011] in, The intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box. The distance between the center points of the two frames. The length of the diagonal of the smallest closed rectangle that encloses the predicted bounding box and the ground truth bounding box; Used to measure the inconsistency between the aspect ratio of the predicted bounding box and the ground truth bounding box. This is the balance coefficient; Classification loss: Use binary cross-entropy or Focal Loss to alleviate class imbalance and enhance the learning ability for hard-to-detect samples; 3) Phased training strategy: including: The first stage involves training the backbone network on infrared single-modal data to obtain initial model parameters that can effectively extract infrared features. The second stage uses both deep grayscale images and infrared images to continue training the initial model. The model parameters from the first stage are used as the starting weights. By further updating the model parameters, the model learns the joint features of cross-modal inputs, thereby improving the accuracy and stability of personnel detection. 4) Performance evaluation strategy: The model performance is evaluated using commonly used evaluation metrics for target detection, including: the average accuracy mAP at an IoU of 0.5, which measures whether the target can be reliably detected; and the average accuracy mAP calculated across multiple thresholds from IoU of 0.5 to 0.95, which reflects the overall accuracy performance of the detection results under different levels of stringency. Calculate recall, which reflects the proportion of true targets detected by the model; 5) Dangerous Area Determination and Intrusion Event Generation: The three-dimensional coordinates of personnel are compared with a preset danger distance threshold. The danger distance threshold is used to limit the minimum safe distance around critical equipment or controlled areas. When a person's position falls within the threshold range, it is considered to have entered a dangerous area. At this time, the system generates an intrusion event, encapsulates the event information, and sends it to the output layer to trigger an alarm.

[0012] Furthermore, the mounting brackets are installed above the power plant fence, passageway, or around key equipment. The infrared camera and lidar are installed at a height of 2 to 4 meters with a downward angle of 10° to 30° to ensure full coverage of personnel targets and reduce obstruction.

[0013] Furthermore, the human body category confidence score refers to the probability value that the person target belongs to the "human body" category by performing semantic analysis on the fused infrared and depth features through the classification branch of the DPDH module. This confidence score is used to measure the credibility of the detection results. The two-dimensional bounding box refers to the range of two-dimensional position of a person in an infrared image predicted by the regression branch of the detection head, including the center position and width and height of the target. This bounding box is used to determine the accurate position of the person in the image. The three-dimensional spatial coordinates refer to the three-dimensional coordinates of the personnel target in real space, which are recovered from the depth information corresponding to the bounding box based on the pixel-level alignment of the depth map and the infrared image. This three-dimensional positioning information is used to determine the actual distance between the personnel and the danger zone.

[0014] The present invention has the following beneficial effects: (1) Achieve stable monitoring around the clock By employing a co-mounted design with infrared cameras and lidar at the perception layer, aligning their optical axes and overlapping their fields of view, the system can simultaneously acquire thermal imaging and three-dimensional spatial structure information. The infrared camera maintains clear imaging even under conditions such as nighttime, strong backlighting, and smoke obstruction, while the lidar provides geometric measurements independent of lighting conditions. This enables the system to operate stably under typical power plant operating conditions, including low light and high background interference, significantly enhancing its all-weather, all-time monitoring capabilities.

[0015] (2) Improve the stability and robustness of human body detection The processing layer employs a high-performance processor to perform tasks such as preprocessing of infrared images and point cloud data, cross-modal fusion detection, 3D localization, and trajectory estimation, ensuring that the system maintains high throughput and low latency processing capabilities even under real-time constraints. By fusing thermal imaging features and 3D structural features within a unified framework, the detection process no longer relies on a single modality, effectively improving the system's robustness in human detection in environments with multiple backgrounds and interferences.

[0016] (3) Realize real-time alarm linkage to improve response efficiency After determining that a person has illegally entered the monitored area, the human detection and recognition module can automatically generate the intruder's three-dimensional location coordinates and alarm information, and send them to the output layer through the communication interface. The alarm unit can immediately trigger an audible and visual alarm, and the control center display terminal displays information such as the person's location and alarm status in real time, enabling on-site management personnel to receive intrusion alerts at the first moment, thereby improving the response speed and security management level of dangerous areas.

[0017] (4) The system architecture is clear and the deployment and maintenance are convenient. The system's perception, processing, and output layers are functionally decoupled through a hierarchical structure, resulting in a simple data link and clear processing flow, facilitating flexible installation, expansion, and maintenance within power plants. Coupled with a unified time synchronization mechanism and stable data flow, the system maintains consistency and reliability during long-term operation, providing continuous and accurate intrusion detection capabilities for hazardous areas. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the system framework of the present invention. Figure 2 This is a schematic diagram of converting a lidar point cloud macro-infrared image into a grayscale image with depth information. Figure 3 This is a schematic diagram of the network structure of the multimodal target detection model used in this invention.

[0019] Figure 4 This is a schematic diagram of the target detection results of the present invention. Detailed Implementation

[0020] The system composition, workflow, and specific working principle of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0021] As shown in Figure 1, this system consists of a perception layer, a processing layer, and an output layer. The layers work together through a unified time synchronization mechanism and a spatial calibration mechanism to achieve real-time intrusion identification and location in the power plant monitoring area.

[0022] The sensing layer includes an infrared thermal imaging camera and a lidar, both fixedly mounted on the same bracket and maintaining a stable spatial relative position. The bracket is positioned above the power plant fence, passageway, or around key equipment, at a height of 2–4 m and a downward viewing angle of 10°–30° to ensure effective coverage of the monitoring area. The installation orientation of the infrared camera and lidar is adjusted so that their optical axes point to the same monitoring area, resulting in overlapping coverage between the infrared imaging field of view and the lidar scanning field of view. The infrared camera continuously acquires raw thermal imaging images and records the thermal radiation characteristics of the human body. The lidar synchronously scans the monitoring area to acquire corresponding three-dimensional point cloud data. The spatial coordinate relationship between the two types of sensors is established through extrinsic parameter calibration (rigid body transformation matrix), and time synchronization is achieved through PTP / NTP or hardware triggering, ensuring that the infrared image and point cloud data correspond in time and space.

[0023] The processing layer uses a high-performance processor (such as an NVIDIA Jetson Orin NX or an industrial computer) to connect with the perception layer, such as... Figure 2 As shown, infrared images and raw point cloud data from the lidar are received via the network, and the following operations are performed: S2-1: Infrared image preprocessing, including: S2-1-1: Median filtering for noise reduction: Traverse the pixels of the infrared image, sort the gray values ​​of all pixels in the N×N neighborhood window (N is an odd number, such as 3, 5 or 7) centered on each pixel, and replace the original pixel value with the median value to suppress random noise.

[0024] S2-1-2: Bicubic Interpolation Super-Resolution Reconstruction; This method uses bicubic interpolation to enlarge median-filtered and denoised images, calculating the pixel value of the interpolation point using the grayscale values ​​of neighboring pixels. Assuming... The neighboring pixels participate in the calculation, where Represents the pixels in the neighborhood grayscale value, This represents the corresponding interpolation weight coefficients, which then determine the new pixels after super-resolution. It can be calculated using the following formula:

[0025] In the formula, the weights It can be decomposed into the product of row and column weights:

[0026] S2-1-3: Homomorphic filtering normalizes the illumination of the image and enhances contrast: Infrared image preprocessed after step S2-1-2 This can be represented as illumination components. With reflection component The product of. For Performing a two-dimensional discrete Fourier transform (DFT) yields the frequency domain representation:

[0027] remember The system includes low-frequency illumination components and high-frequency reflection components, and then a homomorphic high-pass filter function is designed. , for the spectrum Perform filtering as follows:

[0028] in, The amplification range of the filter for low-frequency and high-frequency components is controlled separately to suppress the overall lighting intensity and improve the contrast of details; This represents the frequency distance from the origin on the spectrum. These are the filter cutoff parameters. After processing, the result undergoes an inverse Fourier transform and exponential restoration, i.e., the calculations are performed sequentially. Then, the exponent is taken to reconstruct the enhanced image.

[0029] Step S2-2: Point Cloud Preprocessing Basic processing of the lidar point cloud, including: S2-2-1: Coordinate Transformation: Using extrinsic parameter calibration, the 3D point cloud is transformed from the radar coordinate system to the infrared camera coordinate system. For any point in the lidar point cloud... (LiDAR coordinate system), transform it to the infrared camera coordinate system through rigid body transformation:

[0030] in, , Let be the rotation and translation matrices of the radar coordinate system relative to the camera coordinate system.

[0031] S2-2-2: Pixel Projection: Based on the camera's intrinsic parameters, project the point onto the image plane to obtain the pixel coordinates. :

[0032] in , This is the camera's focal length in pixels. , Offset to the main point. After projection, use the depth value of that point. (or Euclidean distance obtained by lidar ranging) The pixel grayscale value is used to generate the initial depth map. If some pixels do not match any point, they are assigned a null value.

[0033] Step S2-2-3: Depth map generation and pixel-level alignment S2-2-3-1: Hole Filling and Interpolation: For pixels hit by the point cloud, their corresponding depth value or Euclidean distance is used as the gray value; for pixels not hit by any points, they are marked as blank. Subsequently, bilinear interpolation is used to fill the blank areas according to the distance weights of surrounding effective depth points, thereby completing the sparse areas.

[0034] S2-2-3-2: Depth Map Generation and Normalization: Obtain a depth grayscale map with the same size as the infrared image and spatially aligned, and normalize the depth values ​​according to the maximum detection distance. Normalize to 0–255 grayscale:

[0035] S2-2-3-3: Outputs a depth map of the same size as the infrared image and spatially aligned, used as input for cross-modal fusion.

[0036] Step S2-4: Construction of Multimodal Target Detection Model and Target Detection like Figure 3 As shown, an improved YOLO (You Only Look Once) target detection model is used as the core algorithm, with the processed infrared image and depth point cloud map as network inputs. The multimodal target detection model includes GhostHGNetV2, ASF-P2, and DPDH modules (dynamic detection head). Figure 3 This demonstrates the overall structure of the network and the positional relationships of its modules. Specifically, it includes: S2-4-1: Use the improved GhostHGNetV2 to replace the original YOLO backbone: (1) Use the HGStem module and learnable downsampling layer (LDS) to achieve adaptive downsampling by using depthwise separable convolution, so as to retain more effective features while maintaining resolution; (2) The main path uses standard convolution to extract key semantic features to ensure effective modeling of human contours and thermal features; (3) The secondary path generates a large number of inexpensive feature maps through Ghost convolution and reuses the features of the main path, which significantly reduces the amount of computation and parameters while enhancing the diversity of feature expression; (4) Through the four-stage multi-level feature structure and Ghost_HGBlock stacking, stronger multi-scale representation ability is achieved, and higher feature extraction robustness is achieved in complex backgrounds and small target scenes.

[0037] S2-4-2: ASF-P2 multi-scale feature fusion mechanism enhances small targets; (1) Scalseq multi-scale cascade fusion is adopted to arrange feature maps of different scales in order from fine to coarse, and scale sequence features are extracted by Gaussian pyramid convolution and 3D convolution; at the same time, P2 level low-level features are introduced to enhance the fine representation of tiny targets; (2) CBAM attention module is cascaded on the Scalseq output: first, channel attention (global average pooling, global max pooling + MLP) is used to reweight the features of each channel, and then spatial attention is used to generate position attention map to highlight the target area and suppress background interference; (3) Through the combined effect of scale sequence fusion and channel-spatial attention, adaptive enhancement of key features of multiple scales is achieved to improve the detection capability of low-resolution and tiny human targets.

[0038] S2-4-3: The DPDH dynamic detection head is used for target classification and regression, and the DPDH dual-path dynamic detection head is used for adaptive modeling of classification and localization. (1) Based on the YOLO decoupled detection head, a dynamic convolution mechanism is introduced to adaptively generate convolution kernel weights according to the input features, so that the classification branch and the regression branch can dynamically adjust the receptive field and feature response to achieve information complementarity; (2) The classification branch enhances the semantic feature expression through dynamic convolution, thereby improving the ability to distinguish the categories of small human targets; (3) The regression branch uses dynamic convolution to adaptively adjust the position and scale prediction of the bounding box, thereby improving the localization accuracy under different target sizes and imaging conditions; S2-4-4: Output of personnel target detection results: After completing feature extraction of the GhostHGNetV2 backbone network, ASF-P2 multi-scale fusion, and classification and regression modeling of the DPDH dynamic detection head, the multimodal target detection model performs forward inference on the fused features of the infrared image and depth map. Finally, the DPDH dynamic detection head simultaneously generates the following output results: Figure 4 As shown: (1) Human body category confidence: The fused infrared and depth features are semantically analyzed through the classification branch of the DPDH dynamic detection head, and the probability value of the human target belonging to the "human body" category is output. This confidence is used to measure the credibility of the detection results.

[0039] (2) Two-dimensional bounding box: The regression branch of the detector head is used to predict the two-dimensional position range of the person target in the infrared image, including the center position and width and height dimensions of the target. This bounding box is used to determine the accurate position of the person in the image.

[0040] (3) Three-dimensional spatial coordinates: Based on pixel-level alignment of the depth map and the infrared image, the three-dimensional coordinates of the personnel target in real space are recovered from the depth information corresponding to the bounding box. This three-dimensional positioning information is used to determine the actual distance between the personnel and the danger zone.

[0041] Step S2-5: Model Training and Performance Metrics S2-5-1: Training Data Construction: The infrared-LiDAR synchronous fusion data collected in the actual power plant scenario is used as the training set, and the bounding boxes and categories (human / non-human) of personnel targets are marked on the infrared images for supervised training.

[0042] S2-5-2: Loss Function Design: To simultaneously optimize classification and localization performance, a comprehensive loss function is adopted, including: (1) Positioning loss: Using CIoU loss, considering factors such as box overlap (IoU), center point distance, and aspect ratio consistency, the positioning accuracy of the bounding box is improved.

[0043] in, The intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box. The distance between the center points of the two frames. The length of the diagonal of the minimum closed rectangle that encloses the predicted bounding box and the ground truth bounding box. Used to measure the inconsistency between the aspect ratio of the predicted bounding box and the ground truth bounding box. This is the balance coefficient. By minimizing... It can simultaneously optimize the positional overlap and shape matching of the detection box, thereby improving positioning accuracy.

[0044] (2) Classification loss: Use binary cross-entropy or Focal Loss to alleviate class imbalance and enhance the learning ability for hard-to-detect samples.

[0045] S2-5-3: Phased Training Strategy: Model training is divided into two phases: (1) First stage: Pre-train the backbone network on infrared single-modal data to accelerate feature learning and obtain initial model parameters that can effectively extract infrared features; (2) Second stage: Simultaneously use depth grayscale images and infrared images to continue training the above initial model. Use the model parameters from the first stage as the starting weights, and further update the model parameters to enable it to learn the joint features of cross-modal inputs, thereby improving the accuracy and stability of personnel detection.

[0046] S2-5-4: Performance Evaluation Strategy: The model performance is evaluated using commonly used object detection evaluation metrics, including: (1) mAP (mean Average Precision): The detection performance of the system is mainly evaluated by two types of indicators: One is the average accuracy AP when IoU is 0.5, which is used to measure whether the target can be reliably detected. The specific method is that when IoU>0.5, the current prediction is recorded as True, otherwise it is recorded as False. Then, when sampling S times, the average accuracy is:

[0047] Another type is the average accuracy (mAP) calculated across multiple thresholds with IoU values ​​ranging from 0.5 to 0.95, reflecting the overall accuracy performance of the detection results at different levels of stringency. Define the IoU threshold set:

[0049] For each threshold t, calculate AP separately, and then take the average:

[0050] (2) Recall: Reflects the proportion of true targets detected by the model.

[0051] Step S2-5-5: Hazardous Area Determination and Intrusion Event Generation The system compares the personnel's 3D coordinates with a preset danger distance threshold. When a personnel's location falls within this threshold range, it is considered to have entered a danger zone, an intrusion event is generated, and the event information is encapsulated and sent to the output layer to trigger an alarm. (12) in, c The center of the danger zone is designated by humans; x,y,z To reconstruct the three-dimensional spatial coordinates of the personnel using the depth map, W(x,y,z) To accurately indicate that an intrusion has occurred and to issue an alert, dth This is the danger distance value; The output layer includes an audible and visual alarm unit and a control center interface. It is connected to the processing layer via I / O, relays, RS-485 / Modbus, or Ethernet. When an intrusion trigger signal is received from the processing layer, an audible and visual alarm (buzzer, flashing light, alarm light) is activated to create a clear warning effect on site and prevent personnel from continuing to enter by mistake. The output layer sends information such as intrusion events and human body three-dimensional coordinates to the monitoring center interface in real time for display, recording, and subsequent management.

[0052] Combination Figure 4 The complete inference results of the multimodal human detection model of this invention are shown in the left figure: the input infrared thermal image is used for feature extraction by the GhostHGNetV2 backbone network, ASF-P2 multi-scale fusion, and classification and regression inference by the DPDH dynamic detection head. The model outputs the human detection results in the right figure, including the two-dimensional bounding box of the target (red box), the human category confidence (e.g., Conf=0.88), and the three-dimensional spatial coordinates (x, y, z) recovered using the depth map. By aligning the infrared image and the depth map at the pixel level, the model can pinpoint the location of the person in the infrared image and infer its three-dimensional position in real space from the corresponding depth information, thus achieving visual detection and accurate positioning of intruders.

[0053] The multimodal human body recognition system provided by this invention significantly outperforms existing single-sensor or traditional monitoring solutions in both functionality and performance. Compared to traditional methods that rely solely on visible light cameras for monitoring, this system can still operate normally at night and under obstructed conditions, achieving all-weather monitoring. Compared to methods that only use infrared imaging, this invention significantly reduces the false alarm rate by fusing depth information from lidar, for example, avoiding misidentifying isolated heat sources as people, and extending positioning accuracy from planar position to three-dimensional coordinates. Compared to methods that rely solely on lidar detection, this system effectively distinguishes human bodies from other objects by combining infrared thermal features, solving the problem of lidar's weak target attribute identification. This invention significantly improves upon existing technologies in terms of detection reliability, recognition accuracy, positioning precision, and environmental adaptability, and can more effectively ensure the safety of personnel in high-risk areas such as power plants.

Claims

1. A human recognition system based on radar and infrared fusion for detecting accidental entry into power plants, comprising a perception layer, a processing layer, and an output layer; The perception layer includes an infrared camera and a lidar fixedly mounted on the same mounting bracket. The two maintain a fixed spatial relative position relationship, and their optical axes point to the same monitoring area, so that the infrared imaging field of view and the lidar scanning field of view overlap and cover the target area. The infrared camera acquires thermal imaging images, and the lidar acquires three-dimensional point cloud data, and the time is synchronized. The processing layer, with a built-in deep learning model for person detection, includes: The image and point cloud preprocessing modules preprocess the thermal imaging images acquired by the infrared camera and the 3D point cloud data acquired by the lidar, respectively. The multimodal fusion detection module is used to execute a deep learning-based cross-modal feature fusion algorithm, which jointly analyzes the thermal target region in the infrared image and the spatial structure features in the point cloud to complete the detection, 3D localization and trajectory estimation of personnel targets. When the human body detection and recognition module detects that a person has illegally entered the monitored area, it generates the location coordinates of the intruder and alarm information, and sends them to the output layer through the communication interface for alarm linkage. The output layer includes an alarm unit and a control center display terminal interface. The alarm unit is used to receive alarm information and target location coordinates sent by the processing layer, and triggers an audible and visual alarm according to a preset strategy. The control center display terminal, which is connected to the control center display terminal interface, is used to display the personnel location, alarm status and related information pushed by the processing layer in real time, so that monitoring personnel can perform on-site management.

2. The human body recognition system based on radar and infrared fusion for power plant trespass detection according to claim 1, characterized in that: The perception layer uses an external parameter calibration and time synchronization mechanism to ensure that the infrared image and point cloud data correspond in time and space.

3. The human body recognition system based on radar and infrared fusion for power plant trespass detection according to claim 1, characterized in that, When the image and point cloud preprocessing module performs preprocessing on the infrared image, it performs the following operations: Median filtering is used to denoise the original infrared image. The median value of each pixel is replaced within its neighborhood window to suppress random noise. Super-resolution reconstruction is performed using bicubic interpolation, which generates interpolated pixel values ​​by weighted combination of neighboring pixels, thereby improving the clarity of distant and small targets. Homomorphic filtering is used to normalize the illumination of the image, thereby enhancing details by suppressing low-frequency illumination components and enhancing high-frequency reflection components.

4. The human body recognition system based on radar and infrared fusion for power plant trespass detection according to claim 1, characterized in that, The image and point cloud preprocessing module preprocesses the point cloud data, including: (1) Based on the external parameter calibration, transform the lidar point cloud from the radar coordinate system to the infrared camera coordinate system; (2) Based on the camera intrinsic parameters, the pixel coordinates are obtained by projecting the 3D point cloud points onto the image plane; (3) Depth map generation and pixel-level alignment.

5. The human body recognition system based on radar and infrared fusion for power plant trespass detection according to claim 4, characterized in that, The process of depth map generation and pixel-level alignment includes: (1) Hole filling and interpolation: For pixels hit by the point cloud, their corresponding depth value or Euclidean distance is used as the gray value; for pixels not hit by any point, they are marked as blank; then, the blank area is filled by bilinear interpolation according to the distance weight of the surrounding effective depth points, thereby completing the sparse area. (2) Depth map generation and normalization: Obtain a depth grayscale map with the same size as the infrared image and spatially aligned, and normalize the depth values ​​according to the maximum detection distance. Normalize to 0–255 grayscale: (3) Output a depth map that is the same size as the infrared image and spatially aligned, for use as cross-modal fusion input.

6. The human body recognition system based on radar and infrared fusion for power plant trespass detection according to claim 1, characterized in that, The cross-modal fusion detection module includes: The GhostHGNetV2 backbone network is used to extract multi-scale features from infrared images and depth maps. It uses the HGStem module and Ghost convolution to generate efficient feature representations. The ASF-P2 multi-scale feature fusion module performs concatenation and fusion of features at different scales, and enhances small target features through P2-level low-level features and CBAM attention. The DPDH module employs dynamic convolution in both the classification and regression branches, adaptively adjusting the convolution kernel weights to achieve more accurate target classification and bounding box localization, outputting human category confidence, two-dimensional bounding boxes, and three-dimensional spatial coordinates recovered based on depth information.

7. The human body recognition system based on radar and infrared fusion for power plant trespass detection according to claim 1, characterized in that, The multimodal fusion detection module jointly analyzes the thermal target region in the infrared image and the spatial structure features in the point cloud, specifically including: 1) Training data construction: The infrared-lidar synchronous fusion data collected in the actual power plant scenario is used as the training set, and the bounding boxes and categories of personnel targets are marked on the infrared images. The categories are human / non-human, which are used for supervised training. 2) Loss Function Design: To simultaneously optimize classification and localization performance, a comprehensive loss function is adopted, including: Localization Loss: Using CIoU loss, which comprehensively considers factors such as bounding box overlap IoU, center point distance, and aspect ratio consistency, the localization accuracy of the bounding box is improved. in, The intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box. The distance between the center points of the two frames. The length of the diagonal of the minimum closed rectangle that encloses the predicted bounding box and the ground truth bounding box. Used to measure the inconsistency between the aspect ratio of the predicted bounding box and the ground truth bounding box. This is the balance coefficient; Classification loss: Use binary cross-entropy or Focal Loss to alleviate class imbalance and enhance the learning ability for hard-to-detect samples; 3) Phased training strategy: including: The first stage involves training the backbone network on infrared single-modal data to obtain initial model parameters that can effectively extract infrared features. The second stage uses both deep grayscale images and infrared images to continue training the initial model. The model parameters from the first stage are used as the starting weights. By further updating the model parameters, the model learns the joint features of cross-modal inputs, thereby improving the accuracy and stability of personnel detection. 4) Performance evaluation strategy: The model performance is evaluated using commonly used evaluation metrics for target detection, including: the average accuracy mAP at an IoU of 0.5, which measures whether the target can be reliably detected; and the average accuracy mAP calculated across multiple thresholds from IoU of 0.5 to 0.95, which reflects the overall accuracy performance of the detection results under different levels of stringency. Calculate recall, which reflects the proportion of true targets detected by the model; 5) Dangerous Area Determination and Intrusion Event Generation: The three-dimensional coordinates of personnel are compared with a preset danger distance threshold. The danger distance threshold is used to limit the minimum safe distance around critical equipment or controlled areas. When a person's position falls within the threshold range, it is considered to have entered a dangerous area. At this time, the system generates an intrusion event, encapsulates the event information, and sends it to the output layer to trigger an alarm.

8. The human body recognition system based on radar and infrared fusion for power plant trespass detection according to claim 1, characterized in that: The mounting brackets are set above the power plant fence, passageway, or around key equipment. The infrared camera and lidar are installed at a height of 2 to 4 meters and have a downward angle of 10° to 30° to ensure full coverage of personnel targets and reduce obstruction.

9. The human body recognition system based on radar and infrared fusion for power plant trespass detection according to claim 6, characterized in that: The human body category confidence score refers to the probability value that a person target belongs to the "human body" category by performing semantic analysis on the fused infrared and depth features through the classification branch of the DPDH module. This confidence score is used to measure the credibility of the detection results. The two-dimensional bounding box refers to the range of two-dimensional position of a person in an infrared image predicted by the regression branch of the detection head, including the center position and width and height of the target. This bounding box is used to determine the accurate position of the person in the image. The three-dimensional spatial coordinates refer to the three-dimensional coordinates of the personnel target in real space, which are recovered from the depth information corresponding to the bounding box based on the pixel-level alignment of the depth map and the infrared image. This three-dimensional positioning information is used to determine the actual distance between the personnel and the danger zone.

Images