Personnel and work target detection method based on YOLOv11 empty top region

Abstract

The application discloses a kind of based on YOLOv11's empty top area personnel and job target detection method, first, constructs empty top area image dataset, and each group of empty top area image in it includes infrared image and visible light image;Then, an empty top area target detection model with YOLOv11 model as benchmark network is constructed, including two main networks, three MAAF-Net modules, neck network and head network, then the empty top area image dataset is used to iteratively train the empty top area target detection model, finally, the trained empty top area target detection model is used to detect the empty top area personnel and job behavior, and the detection result is obtained.The application has multi-modal feature alignment and fusion capability, can stably detect personnel and job target under conditions such as occlusion, weak light, complex background, and combines empty top area semantics to judge spatial relationship, so as to realize automatic identification of personnel entering dangerous area.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically a method for detecting people and work targets in an empty top area based on YOLOv11. Background Technology

[0002] The underground working environment in coal mines is complex, often accompanied by unfavorable factors such as insufficient lighting, confined spaces, and smoke and dust obstruction. When workers are operating in areas with open roofs (such as the top of unreinforced roadways or above goaf areas), they are highly susceptible to serious safety accidents caused by roof collapse or rockfall. Traditional methods for monitoring and early warning of open roof operations mainly rely on manual inspections, video surveillance, or underground positioning systems. However, due to limitations such as the lag in human judgment, the ambiguity of visual information, and the insufficient precision of systems, it is difficult to achieve efficient, accurate, and real-time identification of worker behavior, and thus cannot effectively prevent major safety hazards caused by accidentally entering dangerous areas.

[0003] With the development of artificial intelligence and computer vision technologies, deep learning-based image target detection algorithms have been widely used in the field of industrial safety monitoring. Among them, the YOLO series models, with their advantages of fast detection speed, high accuracy, and flexible deployment, have shown broad application prospects in personnel identification and work behavior analysis in scenarios such as coal mines. However, due to the special characteristics of the underground environment in coal mines, traditional single-modal target detection models are still prone to missed detections and false detections under conditions such as low light, occlusion, and dust interference, and it is difficult to accurately determine whether personnel are inside the open ceiling area.

[0004] Infrared thermal imaging technology provides strong support for solving such problems. Infrared images can stably acquire thermal source feature information of personnel in low-light or even completely dark environments, without relying on external lighting conditions, and have good anti-interference capabilities. In contrast, visible light images provide more detail and texture information. The two are significantly complementary, and fusing infrared and visible light dual-modal images for personnel detection helps improve recognition accuracy and robustness in complex environments, becoming an important direction in multimodal perception research in recent years. However, dual-modal images in underground environments face challenges such as spatial alignment difficulties, complex fusion strategy design, and high model computational overhead, resulting in shortcomings in practicality and real-time performance of existing methods. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a method for detecting personnel and work targets in the headroom area based on YOLOv11. It has the ability to align and fuse multimodal features, and can stably detect personnel and work targets under conditions such as occlusion, low light, and complex background. It also combines the semantics of the headroom area to make spatial relationship judgments, thereby realizing the automatic identification of personnel entering dangerous areas and providing technical support for the safety of downhole operations.

[0006] The technical solution of this invention is as follows:

[0007] The YOLOv11-based method for detecting personnel and work targets in the overhead area includes the following steps:

[0008] (1) Collect multiple sets of top-of-the-sky region images and perform image preprocessing to construct a top-of-the-sky region image dataset. Each set of top-of-the-sky region images includes infrared images and visible light images.

[0009] (2) Construct a top-of-the-air target detection model. The top-of-the-air target detection model uses the YOLOv11 model as the base network. The top-of-the-air target detection model includes two backbone networks, three MAAF-Net modules, a neck network, and a head network. The two backbone networks have the same network architecture and extract features from infrared and visible light images respectively. The three feature maps output by the two backbone networks are fused using the MAAF-Net module to obtain three fused feature maps at different scales. The three multi-scale fused feature maps are input into the neck network and fused based on the BiFPN module. Then, the three feature maps output by the neck network are input into the head network for detection and the final detection result is output.

[0010] (3) Construct a composite loss function and use the top-of-the-sky area image dataset to iteratively train the top-of-the-sky area target detection model to obtain a trained top-of-the-sky area target detection model. Use the trained top-of-the-sky area target detection model to detect personnel and work behavior in the top-of-the-sky area and obtain the detection results.

[0011] The processing procedure of the MAAF-Net module specifically includes the following steps:

[0012] S21. First, global average pooling is performed on the input infrared image features and the input visible light image features to extract the global semantic level response intensity. The infrared image features and visible light image features after global average pooling are dimensionality reduced by 1×1 convolution. The dimensionality-reduced infrared image features and visible light image features are added element by element and activated by the Sigmoid activation function to obtain the modal attention weight coefficients.

[0013] S22. The infrared image features and visible light image features after global average pooling are concatenated. The concatenated modal features are then input into the channel attention subnetwork. First, global pooling is performed to obtain channel-level response statistics. Then, channel dependencies are extracted through convolution. After activation by the Sigmoid activation function, channel attention weights are obtained. Finally, the channel attention weights are used to perform a dot product operation on the concatenated modal features to obtain the channel-weighted feature map.

[0014] S23. The channel-weighted feature map is compressed and integrated through a 1×1 convolution, and then multiplied by the modality attention weight coefficient. The fusion result obtained after the multiplication operation is activated by the Sigmoid activation function, and finally the fused feature map is output.

[0015] The backbone network includes five Conv modules, four C3K2 modules, one SPPF module, and one C2PSA module. The processing procedure of the backbone network is shown in the following formula (1):

[0016] (1);

[0017] In equation (1), The input to the backbone network is the acquired infrared or visible light image. , , These are feature maps at three different scales extracted from the backbone network;

[0018] The feature map output after the infrared image is processed by the backbone network Feature maps output after processing visible light images by the backbone network The input is fed into the first MAAF-Net module for feature fusion, and the output is a fused feature map. Feature maps output after infrared images are processed by the backbone network Feature maps output after processing visible light images by the backbone network The input is fed into the second MAAF-Net module for feature fusion, and the output is a fused feature map. Feature maps output after infrared images are processed by the backbone network Feature maps output after processing visible light images by the backbone network The input is fed into the third MAAF-Net module for feature fusion, and the output is a fused feature map. .

[0019] The neck network includes five Conv modules, two Upsample modules, five C3K2 modules, five BiFPN modules, and one SCDown module. The processing procedure of the neck network is shown in the following formula (2):

[0020] (2);

[0021] In equation (2), , , , , and This is the intermediate feature map of the neck network. , and These are three feature maps output by the neck network.

[0022] The composite loss function is composed of the Wise-IoU loss function and the Shape-IoU loss function, as shown in the following equation (3):

[0023] (3);

[0024] In equation (3), Represents the composite loss function. Represents the Wise-IoU loss function. This represents the Shape-IoU loss function.

[0025] The Wise-IoU loss function is calculated by the following equation (4):

[0026] (4);

[0027] In equation (4), Represents IoU loss, Represents the IoU loss variant with a decay factor. Represents outlier degree. Represents gradient gain. , All of these are hyperparameters.

[0028] The Shape-IoU loss function is calculated by the following equation (5):

[0029] (5);

[0030] In equation (5), The intersection-over-union ratio (IoU) measures the degree of overlap between the predicted bounding box and the ground truth bounding box. Represents distance loss, Represents shape loss;

[0031] The aforementioned The specific calculation is shown in the following formula (6):

[0032] (6);

[0033] In equation (6), Represents the prediction box. Represents the actual bounding box;

[0034] The aforementioned The specific calculation is shown in the following formula (7):

[0035] (7);

[0036] In equation (7), A normalized representation of the actual bounding box width; A normalized representation of the actual bounding box height; Represents the weighting coefficient in the width direction; Represents the weighting coefficient in the height direction; The x-coordinate representing the center of the prediction box; The ordinate representing the center of the prediction box; The x-coordinate representing the center of the true bounding box; The ordinate representing the center of the true bounding box; This represents the normalization constant, specifically the diagonal length of the image;

[0037] The aforementioned The specific calculation is shown in the following formula (8):

[0038] (8);

[0039] In equation (8), Represents the width of the prediction box. Represents the height of the prediction box. Represents the width of the actual bounding box. Represents the height of the actual bounding box. This represents taking the maximum of the two. This represents the normalized error in the width direction. This represents the normalized error in the height direction. This is a hyperparameter with a value of 4.

[0040] Advantages of this invention:

[0041] (1) The MAAF-Net module of the present invention realizes dynamic weighted fusion of infrared and visible light modal features, enabling the target detection model in the top-of-the-line area to adaptively adjust the modal contribution according to the scene content, thereby effectively improving the discrimination ability of the fused features. Combined with the guidance of channel attention and spatial attention, it shows good results in enhancing the expression of salient regional features and suppressing background interference, significantly improving the consistency of multimodal feature expression and target detection accuracy in complex environments. It shows good robustness in scenarios such as weak light, strong interference, and complex backgrounds, and is suitable for multimodal target detection tasks in high-safety scenarios such as monitoring personnel operations in the top-of-the-line area.

[0042] (2) The present invention uses BiFPN (Bidirectional Feature Pyramid) module in the neck network for feature fusion, which realizes efficient cross-scale feature transfer and fusion. Through the bidirectional connection mechanism, it can simultaneously retain low-level details and high-level semantic information. The weighted fusion method further optimizes the feature selection process, which significantly improves the detection accuracy of multi-scale targets, especially small targets and occluded targets. While ensuring detection performance, it also takes into account computational efficiency. It is suitable for real-time target detection tasks under multimodal input and has strong practicality and promotion value. Attached Figure Description

[0043] Figure 1 This is a flowchart of the present invention.

[0044] Figure 2 This is a network framework diagram of two backbone networks and three MAAF-Net modules in the top-of-field target detection model of this invention.

[0045] Figure 3 This is a network framework diagram of the MAAF-Net module of the present invention.

[0046] Figure 4 This is a network framework diagram of the neck network and head network in the target detection model of the empty top region of this invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] See Figure 1 The method for detecting personnel and work targets in the headspace based on YOLOv11 includes the following steps:

[0049] (1) Collect multiple sets of top-of-the-line region images and perform image preprocessing (size uniformity, noise suppression and brightness normalization) to construct a top-of-the-line region image dataset. Each set of top-of-the-line region images includes infrared images and visible light images.

[0050] (2) Construct a top-of-the-air target detection model. The top-of-the-air target detection model uses the YOLOv11 model as the baseline network. The top-of-the-air target detection model includes two backbone networks, three MAAF-Net modules, a neck network, and a head network.

[0051] S1, see Figure 2The backbone network consists of five Conv modules, four C3K2 modules, one SPPF module, and one C2PSA module. The processing procedure of the backbone network is shown in the following formula (1):

[0052] (1);

[0053] In equation (1), The input to the backbone network is the acquired infrared or visible light image. , , These are feature maps at three different scales extracted from the backbone network; This represents the Conv (convolutional) module, whose function is to extract local features and adjust the number of channels or spatial resolution; Representing the C3K2 module, its function is to enhance gradient propagation capabilities and enrich feature representation; The SPPF module is used to obtain a larger receptive field through multi-scale pooling, thereby enhancing the ability to express spatial context. The C2PSA module is used to weight channels and spatial dimensions, highlighting significant target areas and suppressing redundant features.

[0054] S2, see Figure 2 The feature map output after the infrared image is processed by the backbone network Feature maps output after processing visible light images by the backbone network The input is fed into the first MAAF-Net module for feature fusion, and the output is a fused feature map. Feature maps output after infrared images are processed by the backbone network Feature maps output after processing visible light images by the backbone network The input is fed into the second MAAF-Net module for feature fusion, and the output is a fused feature map. Feature maps output after infrared images are processed by the backbone network Feature maps output after processing visible light images by the backbone network The input is fed into the third MAAF-Net module for feature fusion, and the output is a fused feature map. ;

[0055] See Figure 3 The processing procedure of the MAAF-Net module specifically includes the following steps:

[0056] S21. First, global average pooling (GAP) is performed on the input infrared image features IR and the input visible light image features RGB to extract the global semantic level response intensity. The infrared image features and visible light image features after global average pooling are dimensionality reduced by 1×1 convolution (1×1 Conv). The dimensionality-reduced infrared image features and visible light image features are added element-wise and activated by the Sigmoid activation function to obtain the modality attention weight coefficients. The modality attention weight coefficients are used to measure the importance ratio of the two modalities in the current task scenario. The original features of the two modalities are adaptively fused according to the modality attention weight coefficients to achieve dynamic modality selection.

[0057] S22. The infrared image features and visible light image features after global average pooling are concatenated. The concatenated modal features are then input into the channel attention subnetwork. First, global pooling is performed to obtain channel-level response statistics. Then, convolution is performed to extract channel dependencies. After activation by the sigmoid activation function, channel attention weights are obtained. These weights are used to modulate the channel dimension of the fused feature map, thereby highlighting significant semantic channels, suppressing redundant interference channels, and enhancing feature representation capabilities. Finally, the channel attention weights are used to perform a dot product operation on the concatenated modal features to obtain the channel-weighted feature map.

[0058] S23. The channel-weighted feature map is compressed and integrated through 1×1 convolution (1×1 Conv), and then multiplied with the modality attention weight coefficient. The fusion result obtained after the multiplication operation is activated by the Sigmoid activation function, and finally outputs a high-quality fusion feature with modality adaptation capability, channel selectivity and spatial attention.

[0059] S3, see Figure 4 The neck network consists of five Conv modules, two Upsample modules, five C3K2 modules, five BiFPN modules, and one SCDown module. The processing procedure of the neck network is shown in the following formula (2):

[0060] (2);

[0061] In equation (2), , , , , and This is the intermediate feature map of the neck network. , and Three feature maps output by the neck network; feature maps Feature maps representing the detailed features required for small object detection Feature maps are used to provide target features that balance spatial and semantic information. Integrating deep global semantics to optimize large object detection, three feature maps , and By focusing on the feature representation of small, medium and large targets respectively, we can ensure the deep fusion of multimodal information and avoid redundant calculations, thus achieving a balance between detection accuracy and efficiency.

[0062] S4, see Figure 4 The three feature maps output by the neck network , and The input is fed into three detector heads (Detect) of the head network for detection. Target location regression, category discrimination, and confidence scoring are performed at different scales, and the final detection result is output.

[0063] (3) Construct a composite loss function and use the top-of-the-sky area image dataset to iteratively train the top-of-the-sky area target detection model to obtain a trained top-of-the-sky area target detection model. Use the trained top-of-the-sky area target detection model to detect personnel and work behavior in the top-of-the-sky area and obtain the detection results.

[0064] Subsequently, based on construction drawings or manual calibration, the spatial boundary of the open ceiling area is pre-delineated in the image and represented by polygon coordinate annotation or 3D projection. The spatial relationship between the detected personnel target box (predicted box) and the boundary of the open ceiling area is calculated to determine whether the personnel have entered the danger zone, and output a binary classification result of "entered / not entered". When a worker is detected to have entered the open ceiling area, further analysis can be performed based on information such as duration and location to determine whether it is a violation of work regulations, and trigger an audible and visual alarm or remote notification mechanism to achieve real-time early warning and safety closed-loop control.

[0065] The composite loss function consists of the Wise-IoU loss function and the Shape-IoU loss function, as shown in equation (3) below:

[0066] (3);

[0067] In equation (3), Represents the composite loss function. Represents the Wise-IoU loss function. Represents the Shape-IoU loss function;

[0068] The Wise-IoU loss function is calculated by the following equation (4):

[0069] (4);

[0070] In equation (4), Represents IoU loss, Represents the IoU loss variant with a decay factor. Represents outlier degree. Represents gradient gain. , All are hyperparameters;

[0071] The Shape-IoU loss function is calculated by the following equation (5):

[0072] (5);

[0073] In equation (5), The intersection-over-union ratio (IoU) measures the degree of overlap between the predicted bounding box and the ground truth bounding box. Represents distance loss, Represents shape loss;

[0074] The specific calculation is shown in the following formula (6):

[0075] (6);

[0076] In equation (6), Represents the prediction box. Represents the actual bounding box;

[0077] The specific calculation is shown in the following formula (7):

[0078] (7);

[0079] In equation (7), A normalized representation of the actual bounding box width; A normalized representation of the actual bounding box height; Represents the weighting coefficient in the width direction; Represents the weighting coefficient in the height direction; The x-coordinate representing the center of the prediction box; The ordinate representing the center of the prediction box; The x-coordinate representing the center of the true bounding box; The ordinate representing the center of the true bounding box; This represents the normalization constant, specifically the diagonal length of the image;

[0080] The specific calculation is shown in the following formula (8):

[0081] (8);

[0082] In equation (8), Represents the width of the prediction box. Represents the height of the prediction box. Represents the width of the actual bounding box. Represents the height of the actual bounding box. This represents taking the maximum of the two. This represents the normalized error in the width direction. The normalized error in the height direction, the normalized error in the width direction, and the normalized error in the height direction are cross-multiplied by the weight coefficients of the height and width directions, so that the target detection model in the top-of-the-air region can adaptively distinguish the core features of different shapes, and ultimately improve the positioning accuracy—especially for targets with extreme aspect ratios. This is a hyperparameter with a value of 4.

[0083] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting personnel and work targets in the headspace based on YOLOv11, characterized in that: Specifically, it includes the following steps: (1) Collect multiple sets of top-of-the-sky region images and perform image preprocessing to construct a top-of-the-sky region image dataset. Each set of top-of-the-sky region images includes infrared images and visible light images. (2) Construct a top-of-the-air target detection model. The top-of-the-air target detection model uses the YOLOv11 model as the base network. The top-of-the-air target detection model includes two backbone networks, three MAAF-Net modules, a neck network, and a head network. The two backbone networks have the same network architecture and extract features from infrared and visible light images respectively. The three feature maps output by the two backbone networks are fused using the MAAF-Net module to obtain three fused feature maps at different scales. The three multi-scale fused feature maps are input into the neck network and fused based on the BiFPN module. Then, the three feature maps output by the neck network are input into the head network for detection and the final detection result is output. The processing procedure of the MAAF-Net module specifically includes the following steps: S21. First, global average pooling is performed on the input infrared image features and the input visible light image features to extract the global semantic level response intensity. The infrared image features and visible light image features after global average pooling are dimensionality reduced by 1×1 convolution. The dimensionality-reduced infrared image features and visible light image features are added element by element and activated by the Sigmoid activation function to obtain the modal attention weight coefficients. S22. The infrared image features and visible light image features after global average pooling are concatenated. The concatenated modal features are then input into the channel attention subnetwork. First, global pooling is performed to obtain channel-level response statistics. Then, channel dependencies are extracted through convolution. After activation by the Sigmoid activation function, channel attention weights are obtained. Finally, the channel attention weights are used to perform a dot product operation on the concatenated modal features to obtain the channel-weighted feature map. S23. The channel-weighted feature map is compressed and integrated through 1×1 convolution, and then multiplied with the modality attention weight coefficient. The fusion result obtained after the multiplication operation is activated by the Sigmoid activation function, and finally the fused feature map is output. (3) Construct a composite loss function and use the top-of-the-sky area image dataset to iteratively train the top-of-the-sky area target detection model to obtain a trained top-of-the-sky area target detection model. Use the trained top-of-the-sky area target detection model to detect personnel and work behavior in the top-of-the-sky area and obtain the detection results.

2. The method for detecting personnel and work targets in the headspace based on YOLOv11 according to claim 1, characterized in that: The backbone network includes five Conv modules, four C3K2 modules, one SPPF module, and one C2PSA module. The processing procedure of the backbone network is shown in the following formula (1): （1）； In equation (1), The input to the backbone network is the acquired infrared or visible light image. , , These are feature maps at three different scales extracted from the backbone network; The feature map output after the infrared image is processed by the backbone network Feature maps output after processing visible light images by the backbone network The input is fed into the first MAAF-Net module for feature fusion, and the output is a fused feature map. Feature maps output after infrared images are processed by the backbone network Feature maps output after processing visible light images by the backbone network The input is fed into the second MAAF-Net module for feature fusion, and the output is a fused feature map. Feature maps output after infrared images are processed by the backbone network Feature maps output after processing visible light images by the backbone network The input is fed into the third MAAF-Net module for feature fusion, and the output is a fused feature map. .

3. The method for detecting personnel and work targets in the headspace based on YOLOv11 according to claim 2, characterized in that: The neck network includes five Conv modules, two Upsample modules, five C3K2 modules, five BiFPN modules, and one SCDown module. The processing procedure of the neck network is shown in the following formula (2): （2）； In equation (2), , , , , and This is the intermediate feature map of the neck network. , and These are three feature maps output by the neck network.

4. The method for detecting personnel and work targets in the headspace based on YOLOv11 according to claim 1, characterized in that: The composite loss function is composed of the Wise-IoU loss function and the Shape-IoU loss function, as shown in the following equation (3): （3）； In equation (3), Represents the composite loss function. Represents the Wise-IoU loss function. This represents the Shape-IoU loss function.

5. The method for detecting personnel and work targets in the headspace based on YOLOv11 according to claim 4, characterized in that: The Wise-IoU loss function is calculated by the following equation (4): （4）； In equation (4), Represents IoU loss, Represents the IoU loss variant with a decay factor. Represents outlier degree. Represents gradient gain. , All of these are hyperparameters.

6. The method for detecting personnel and work targets in the headspace based on YOLOv11 according to claim 4, characterized in that: The Shape-IoU loss function is calculated by the following equation (5): （5）； In equation (5), The intersection-over-union ratio (IoU) measures the degree of overlap between the predicted bounding box and the ground truth bounding box. Represents distance loss, Represents shape loss; The aforementioned The specific calculation is shown in the following formula (6): （6）； In equation (6), Represents the prediction box. Represents the actual bounding box; The aforementioned The specific calculation is shown in the following formula (7): （7）； In equation (7), A normalized representation of the actual bounding box width; A normalized representation of the actual bounding box height; Represents the weighting coefficient in the width direction; Represents the weighting coefficient in the height direction; The x-coordinate representing the center of the prediction box; The ordinate representing the center of the prediction box; The x-coordinate representing the center of the true bounding box; The ordinate representing the center of the true bounding box; This represents the normalization constant, specifically the diagonal length of the image; The aforementioned The specific calculation is shown in the following formula (8): （8）； In equation (8), Represents the width of the prediction box. Represents the height of the prediction box. Represents the width of the actual bounding box. Represents the height of the actual bounding box. This represents taking the maximum of the two. This represents the normalized error in the width direction. This represents the normalized error in the height direction. This is a hyperparameter with a value of 4.

Images