Underground coal mine personnel detection method and device based on deep learning

Abstract

The invention relates to the technical field of image detection, and provides a coal mine underground personnel detection method and device based on deep learning. The method comprises the following steps: acquiring a multi-modal environment image from an underground coal mine through a preset intelligent sensor, preprocessing the environment image, and generating preprocessing features of time-space alignment; on the basis of visible light features and red network light features in the preprocessed features, adaptive fusion based on modal weight is carried out to generate fusion features, and spatio-temporal context relationships of underground personnel movement are extracted from the fusion features to generate enhanced features; updating the detection model based on the enhanced features, and generating a target set containing personnel detection results; determining a detection score corresponding to each target based on a neighborhood target corresponding to the target set; and performing dynamic threshold adaptive evaluation based on the detection score, and determining personnel information. High-precision and low-delay personnel detection is realized in an underground changeable environment, the real-time and robustness requirements of intelligent monitoring are met, and the omission ratio is remarkably reduced compared with that of a traditional method.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and more specifically, to a method and apparatus for detecting personnel underground in coal mines based on deep learning. Background Technology

[0002] The underground working environment in coal mines is complex, posing multiple safety hazards such as gas explosions, roof collapses, and equipment failures. As a core production element, real-time monitoring of personnel's location, behavior, and safety status is crucial for accident prevention and ensuring life safety. Traditional manual inspections or fixed sensor monitoring methods suffer from blind spots and delayed response times, making them unsuitable for meeting the needs of dynamic safety management underground.

[0003] In practical applications, insufficient lighting, dust, and confined space in underground mines lead to a decline in the imaging quality of visible light cameras, and infrared sensors are easily interfered with by heat sources. Traditional detection methods based on thresholds or template matching have a high false detection rate. Summary of the Invention

[0004] This application provides a method and device for detecting personnel underground in coal mines based on deep learning, which can at least partially solve the problem of high false detection rate in coal mine personnel detection methods.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of this application, a deep learning-based method for detecting personnel underground in coal mines is provided, comprising: acquiring multimodal environmental images from underground coal mines using a pre-set intelligent sensor; preprocessing the environmental images to generate spatiotemporally aligned preprocessed features; adaptively fusing visible light features and rednet light features in the preprocessed features based on modality weights to generate fused features; extracting the spatiotemporal context of personnel movement underground from the fused features to generate enhanced features; updating a pre-set detection model based on the enhanced features to generate a target set containing personnel detection results; determining the detection score corresponding to each target in the target set based on the neighborhood targets corresponding to the target set; and adaptively evaluating the detection scores based on a dynamic threshold to determine personnel information.

[0007] In this application, based on the aforementioned scheme, the step of adaptively fusing the visible light features and infrared light features in the preprocessed features based on modal weights to generate fused features includes: compressing and nonlinearly transforming the preprocessed features in the spatial dimension to generate modal weights for visible light in coal mines; extracting a first feature corresponding to visible light and a second feature corresponding to infrared light from the preprocessed features; and fusing the first feature and the second feature based on the modal weights for visible light in coal mines to generate fused features.

[0008] In this application, based on the aforementioned scheme, the step of compressing and nonlinearly transforming the preprocessed features in the spatial dimension to generate the modal weights of visible light in coal mines includes: compressing the preprocessed features in the spatial dimension to obtain channel-level descriptors; mapping the descriptors to the hidden layer space through a fully connected layer and an activation function; and then generating the modal weights a(t) of visible light in coal mines through a fully connected layer and an activation function as follows:

[0009]

[0010] Where W1 and W2 represent the weight matrices of the reduced-dimensional fully connected layer and the increased-dimensional fully connected layer, respectively, σ and ReLU represent the activation functions, and F pre (t) represents the preprocessing feature, H and W represent the total height and total width of the preprocessing feature, h and w represent the height and width identifiers of the preprocessing feature, and t represents time.

[0011] In this application, based on the aforementioned scheme, the step of fusing the first feature and the second feature based on the mode weights of visible light in the coal mine to generate a fused feature includes: fusing the first feature and the second feature based on the mode weights of visible light in the coal mine to generate a fused feature F. fuse (t) is:

[0012] F fuse (t)=a(t)·F vis (t)+(1-a(t))·F ir (t)

[0013] Among them, F vis (t), F ir (t) represents the first feature and the second feature, respectively.

[0014] In this application, based on the aforementioned scheme, the step of extracting the spatiotemporal context of personnel movement in the fused features and generating enhanced features includes: extracting mapping features from the fused features through a preset convolutional network; and extracting the spatiotemporal context of personnel movement in the fused features based on the mapping features and generating enhanced features.

[0015] In this application, based on the aforementioned scheme, the step of updating the preset detection model based on the enhanced features to generate a target set containing personnel detection results includes: inputting the enhanced features into the detection model to obtain prediction results, generating a total loss based on the prediction results and preset real labels, updating the detection model through the total loss, and generating a target set.

[0016] In this application, based on the aforementioned scheme, determining the detection score corresponding to each target in the target set based on the neighboring targets corresponding to the target set includes: detecting the movement speed of the candidate targets in the target set; determining the neighboring targets of the candidate targets based on the spatial overlap of the candidate targets, and constructing a neighboring set; and determining the detection score corresponding to each target in the target combination based on the movement speed and the neighboring targets in the neighboring set.

[0017] In this application, based on the aforementioned scheme, the step of performing adaptive evaluation of the detection score based on a dynamic threshold to determine personnel information includes: determining an environmental adaptability index based on the environmental complexity of the coal mine; if the environmental adaptability index is less than a preset performance threshold, triggering a parameter adjustment mechanism to generate an updated threshold; and determining personnel information based on the updated threshold.

[0018] According to one aspect of this application, a deep learning-based underground personnel detection device for coal mines is provided, comprising:

[0019] The acquisition unit is used to acquire multimodal environmental images from underground coal mines through preset smart sensors, preprocess the environmental images, and generate spatiotemporally aligned preprocessed features.

[0020] The feature unit is used to perform adaptive fusion based on modal weights on the visible light features and red network light features in the preprocessed features to generate fused features, and extract the spatiotemporal context relationship of the underground personnel movement from the fused features to generate enhanced features;

[0021] The set unit is used to update the preset detection model based on the enhanced features and generate a target set containing the personnel detection results;

[0022] The scoring unit is used to determine the detection score of each target in the target set based on the neighborhood targets corresponding to the target set.

[0023] An evaluation unit is used to perform an adaptive evaluation of the detection score based on a dynamic threshold to determine personnel information.

[0024] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the deep learning-based method for detecting personnel underground in coal mines as described in the above embodiments.

[0025] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the deep learning-based method for detecting personnel underground in coal mines as described in the above embodiments.

[0026] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the deep learning-based method for detecting personnel underground in coal mines provided in the various optional implementations described above.

[0027] In the technical solution of this application, multimodal environmental images are acquired from underground coal mines using pre-set intelligent sensors. These environmental images are pre-processed to generate spatiotemporally aligned pre-processed features. Visible light features and infrared light features in the pre-processed features are adaptively fused based on modal weights to generate fused features. The spatiotemporal context of personnel movement underground is extracted from these fused features to generate enhanced features. A pre-set detection model is updated based on these enhanced features to generate a target set containing personnel detection results. Based on the neighborhood targets corresponding to the target set, the detection score corresponding to each target in the target set is determined. The detection score is adaptively evaluated based on a dynamic threshold to determine personnel information. Spatiotemporally aligned pre-processing solves the synchronization problem between visible light and infrared images, ensuring feature consistency. Modal weight fusion based on environmental perception automatically adjusts the contribution of the two modalities, improving feature reliability in complex scenarios such as dust and low light. Spatiotemporal context modeling captures personnel movement trajectories and group interactions, suppressing false detections. Combining neighborhood target information optimizes the detection score, and dynamic threshold evaluation further ensures real-time performance and accuracy. The overall solution achieves high-precision, low-latency personnel detection in the variable underground environment, meeting the real-time and robust requirements of intelligent monitoring, and significantly reducing the false detection rate compared to traditional methods.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0030] Figure 1 The flowchart illustrating a deep learning-based method for detecting personnel underground in a coal mine is shown in one embodiment of this application.

[0031] Figure 2 The flowchart illustrating the generation of fusion features in one embodiment of this application is shown schematically.

[0032] Figure 3 The illustration shows a schematic diagram of a deep learning-based underground personnel detection device in a coal mine according to one embodiment of this application.

[0033] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0035] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0038] The implementation details of the technical solution of this application are described below:

[0039] Figure 1 A flowchart illustrating a deep learning-based method for detecting personnel underground in coal mines according to an embodiment of this application is shown. (Refer to...) Figure 1 As shown, this deep learning-based method for detecting personnel underground in coal mines includes at least steps S110 to S150, which are detailed below:

[0040] S110: Using a preset intelligent sensor, multimodal environmental images are acquired from underground coal mines. The environmental images are preprocessed to generate spatiotemporally aligned preprocessed features.

[0041] In one embodiment of this application, the pre-installed intelligent sensor array in the coal mine typically includes visible light cameras, infrared thermal imagers, lidar, and multispectral sensors. These sensors are deployed along the roadway walls at pre-defined intervals, forming a monitoring network covering key areas. During data acquisition, the system ensures that the timestamps of each sensor are aligned through a hardware synchronization mechanism, avoiding time misalignment caused by equipment delays. Simultaneously, to address motion blur caused by vibrations of underground equipment and personnel movement, a short exposure time (e.g., 1ms) and high-frequency sampling (≥30Hz) strategy is employed to capture clear instantaneous frames. For example, the visible light sensor prioritizes acquiring high-resolution images to preserve texture details, while the infrared sensor focuses on thermal radiation distribution, generating point cloud data for 3D spatial modeling.

[0042] In one embodiment of this application, the visible light image employs adaptive wavelet thresholding to suppress random noise caused by dust particles; the infrared image uses top-hat transform to eliminate thermal radiation inhomogeneity; and statistical filtering is used to remove outliers. Sensor installation deviations are corrected based on the initial parameters of the calibration board, and then dynamically adjusted using a feature matching algorithm. For example, visible light and infrared images are aligned at the pixel level by calculating the homography matrix through SIFT feature point matching; and point clouds and visible light images are registered from three dimensions to two dimensions through projection mapping.

[0043] Optionally, to address differences in sensor sampling frequencies, such as a radar point cloud frequency of 10Hz and a camera frequency of 30Hz, a temporal interpolation method is used to generate an aligned frame sequence. For example, the radar point cloud is interpolated according to the camera timestamp to ensure that each frame of visible light image corresponds to accurate radar spatial data. The final output is a spatiotemporally aligned feature tensor with dimensions C×H×W×T (channels×height×width×time), providing a unified input for subsequent dynamic weight fusion.

[0044] The above process acquires environmental images using multimodal sensors and utilizes spatiotemporal alignment technology to eliminate viewing angle bias and time asynchrony issues between different sensors. The preprocessing stage significantly improves image quality in low-light, high-dust environments through noise reduction, contrast enhancement, and dynamic range compression, providing robust input for subsequent feature fusion. Spatiotemporal alignment ensures precise matching of visible light and infrared features of the same target in both spatial and temporal dimensions, avoiding detection bias caused by modal differences.

[0045] S120, the visible light features and red network light features in the preprocessed features are adaptively fused based on modal weights to generate fused features. The spatiotemporal context of personnel movement in the mine is extracted from the fused features to generate enhanced features.

[0046] In this embodiment, the adaptive fusion and spatiotemporal context extraction steps based on modality weights first perform weighted fusion of visible light and infrared features through a dynamic weight allocation network: the environmental perception weight is calculated using a spatiotemporal attention gating mechanism, and this weight is dynamically adjusted according to the spatiotemporal statistical information of the preprocessed features (such as dust density and light intensity) to ensure that the high signal-to-noise ratio modality dominates the fusion process; after the fused feature map is channel-aligned through 1×1 convolution, it is input into the spatiotemporal nonlocal attention module. This module captures the continuous trajectory of human movement (such as walking and standing) and spatial interaction relationships (such as group gathering) by calculating the dot product similarity between features of different time frames, and finally generates enhanced features containing motion patterns and contextual information, effectively suppressing local noise and strengthening the semantic representation of the target.

[0047] like Figure 2 As shown, in one embodiment of this application, the visible light features and red network light features in the preprocessed features are adaptively fused based on modal weights to generate fused features, including:

[0048] S210, the preprocessed features are compressed and nonlinearly transformed in the spatial dimension to generate the modal weights of visible light in coal mines;

[0049] S220, extract the first feature corresponding to visible light and the second feature corresponding to infrared light from the preprocessed features;

[0050] S230, based on the modal weights of visible light in the coal mine, the first feature and the second feature are fused to generate a fused feature.

[0051] In one embodiment of this application, the preprocessed features are compressed and nonlinearly transformed in the spatial dimension. Specifically, the preprocessed features are compressed in the spatial dimension to obtain channel-level descriptors; then, the descriptors are mapped to the hidden layer space through a fully connected layer and an activation function; finally, the modal weights a(t) of visible light in coal mines are generated through a fully connected layer and an activation function as follows:

[0052]

[0053] Where W1 and W2 represent the weight matrices of the reduced-dimensional fully connected layer and the increased-dimensional fully connected layer, respectively, σ and ReLU represent the activation functions, and F pre (t) represents the preprocessed feature, H and W represent the total height and width of the preprocessed feature, h and w represent the height and width identifiers of the preprocessed feature, and t represents time. In this embodiment, the calculated weights dynamically reflect the effectiveness of the visible light mode in the current environment; for example, the weights are reduced when the dust concentration is high. This suppresses interference from low signal-to-noise ratio modes during feature fusion. For instance, when the dust concentration exceeds a threshold, the infrared feature weights are automatically increased to avoid missed detections caused by blurred visible light images.

[0054] In one embodiment of this application, a first feature corresponding to visible light and a second feature corresponding to infrared light are extracted from the preprocessed features. Optionally, detailed features, such as texture and color, are extracted through a convolutional layer in the visible light branch as the first feature corresponding to visible light. Thermal radiation features, such as temperature changes, are extracted through a convolutional layer in the infrared branch as the second feature corresponding to infrared light.

[0055] In one embodiment of this application, the first feature and the second feature are fused based on the mode weights of visible light in the coal mine to generate a fused feature F. fuse (t) is:

[0056] F fuse (t)=a(t)·F vis (t)+(1-a(t))·F ir (t)

[0057] Among them, F vis (t), F ira(t) represents the first feature and the second feature, respectively. The fused features obtained through the above process contain dynamically weighted modal information, providing richer context for subsequent spatiotemporal nonlocal attention feature extraction. When the visible light modality is interfered with by dust or low light, a(t) automatically decreases, while the infrared modality weight is increased, thereby improving the signal-to-noise ratio of the fused features.

[0058] In one embodiment of this application, the spatiotemporal context of personnel movement in the mine is extracted from the fused features to generate enhanced features, including:

[0059] Mapping features are extracted from the fused features using a pre-defined convolutional network;

[0060] Based on the mapping features, the spatiotemporal context of personnel movement in the mine is extracted from the fused features to generate enhanced features.

[0061] In one embodiment of this application, a pre-defined convolutional network is used to extract mapping features from the fused features. Based on the mapping features, the spatiotemporal context of personnel movement in the mine is extracted from the fused features to generate enhanced features F. enh (t) is:

[0062]

[0063] Among them, W z denoted as the linear projection matrix, θ, φ, and g represent 1×1 convolutions, used for dimensionality reduction or dimensionality increase features, T represents the transpose of the feature, and Δt represents the time interval.

[0064] The above process, by capturing local spatiotemporal patterns within the spatiotemporal context and modeling global dependencies using non-local attention, enhances the feature responses of low-resolution targets. This allows the enhanced features to retain the dynamic trajectory and spatial distribution information of moving people, effectively suppressing false detections of stationary objects (such as devices). Real-world testing shows a 15% improvement in recall in densely populated scenes. For example, in detecting people at a distance of 50 meters, the recall rate is improved. Experiments also demonstrate that the non-local attention mechanism improves tracking accuracy for fast-moving targets (such as runners) by 12%. This step, through deep mining of the spatiotemporal context, provides highly semantic and robust feature inputs for subsequent detection processes.

[0065] S130, based on the enhanced features, update the preset detection model to generate a target set containing personnel detection results.

[0066] In one embodiment of this application, when updating the detection model based on enhanced features, the enhanced features are first input into the detection head. The probability of the personnel category is predicted through the classification branch, and the bounding box coordinate offset is generated through the regression branch. Then, the total loss is calculated by combining the environment-aware loss function. The total loss may include classification loss, regression loss, and dynamic environment penalty term. The model weights are updated by backpropagation using the optimizer to ensure that the loss function optimizes both detection accuracy and environmental adaptability. Finally, redundant boxes are filtered by context-aware nonmaximum suppression (combined with motion consistency constraints) to generate a target set containing personnel location, category, and confidence. This set is output after dynamic threshold adjustment to achieve accurate positioning and stable tracking of personnel in complex underground scenarios.

[0067] In one embodiment of this application, a preset detection model is updated based on the enhanced features to generate a target set containing personnel detection results, including:

[0068] The enhanced features are input into the detection model to obtain the prediction result. Based on the prediction result and the preset real label, a total loss is generated. The detection model is updated by the total loss to generate the target set.

[0069] Specifically, in this embodiment, the target set includes target coordinates and classification confidence scores.

[0070] This embodiment designs an environment-aware loss function that dynamically combines classification and regression tasks with environmental complexity. Its core lies in balancing the weights of different tasks and adjusting the loss function's sensitivity to noise or complex scenes in real time using an environmental complexity index calculated from image gradients. The classification loss employs an adaptive weight strategy to suppress interference from easily classified samples, allowing the model to focus more on difficult samples. The regression loss combines smoothness and robustness to stably optimize bounding box localization accuracy. The environment penalty term automatically adjusts the loss weights based on scene complexity, preventing the model from overfitting to noise in environments such as dust or low light.

[0071] Using the predicted results from the detection head and the ground truth labels as input, the total loss, which includes classification, regression, and environment penalties, is calculated. Dynamic weight allocation ensures the optimization priority of key tasks (such as bounding box regression) while enhancing feature learning in complex scenarios. The final output total loss not only guides model parameter updates but also improves the model's generalization ability in the variable downhole environment through an environment-aware mechanism, providing crucial guidance for subsequent detection result optimization.

[0072] In the above process, when updating the detection model based on enhanced features, a joint loss function composed of loss functions such as environmental awareness loss can be used. The environmental awareness loss term dynamically adjusts the weights of classification and regression tasks by calculating the difference in environmental complexity between the current frame and historical frames, such as gradient variance. The online update mechanism allows the model to adapt to dynamic changes in the underground environment, such as equipment layout adjustments. In 72 hours of continuous testing, the model's performance degradation rate was reduced to 1 / 3 of that of traditional methods. The target set is generated through context-aware nonmaximum suppression, combined with motion speed consistency constraints, to reduce duplicate detections caused by occlusion.

[0073] S140, Based on the neighborhood targets corresponding to the target set, determine the detection score corresponding to each target in the target set.

[0074] In one embodiment of this application, when determining the detection score based on the neighborhood targets of the target set, a spatial neighborhood graph is first constructed, treating each detected target as a node, and the neighborhood relationship is defined by IoU or Euclidean distance; then, a graph attention network is used to aggregate neighborhood features, calculate the attention weights between the current target and the neighborhood targets, and dynamically adjust the initial detection score by combining contextual information such as motion trajectory consistency and category similarity; finally, a final score that integrates the neighborhood context is generated by weighted summation, which effectively suppresses isolated false detections and enhances the detection confidence of group targets. For example, in a dense crowd, the consistency of the motion direction of the neighborhood targets can improve the score of the current target by 15%.

[0075] In one embodiment of this application, determining the detection score corresponding to each target in the target set based on the neighborhood targets corresponding to the target set includes:

[0076] The movement speed of the candidate targets in the target set is detected, the neighborhood targets of the candidate targets are determined according to the spatial overlap of the candidate targets, and a neighborhood set is constructed.

[0077] Based on the movement speed and the neighboring targets in the neighborhood set, the detection score corresponding to each target in the target combination is determined.

[0078] In one embodiment of this application, the motion velocity of candidate targets in the target set is detected by optical flow. Based on the spatial overlap or Euclidean distance of the candidate targets, neighboring targets are determined, and a neighborhood set is constructed.

[0079] In one embodiment of this application, the detection score corresponding to each target in the target combination is determined based on the movement speed and the neighboring targets in the neighborhood set. for:

[0080]

[0081] in, v represents the identifier and classification confidence of the candidate target, respectively. i v j Let σ represent the velocity of targets i and j, σ represent the control parameter used to control the exponentially decaying velocity, and i and j represent the identifiers of the targets.

[0082] The above process incorporates the consistency of optical flow velocity with the motion of neighboring detection boxes into the detection process, and improves detection stability in complex environments by controlling parameters to adapt to motion differences in different scenes. In conjunction with the environment-aware loss function, it forms a dynamic adaptation mechanism throughout the entire process from training to inference. Through deep mining of motion context, it provides a guarantee of high accuracy and low false positives for the final detection results.

[0083] S150, perform adaptive evaluation based on dynamic threshold on the detection score to determine personnel information.

[0084] In this embodiment, the dynamic threshold adaptive evaluation based on detection scores dynamically adjusts the classification threshold by calculating the joint distribution of the Environment Adaptation Index (EAI) and the detection scores in real time. First, a sliding window is used to statistically analyze the detection score distribution of the current frame and N historical frames, and the EAI value is calculated in conjunction with environmental complexity indicators (such as dust density and light fluctuations). Then, a preset EAI-threshold mapping function (such as a piecewise linear function or a lightweight regression model) is used to determine the current optimal threshold. This function learns the score-threshold correspondence under different environmental conditions through offline training. Finally, the adjusted threshold is applied to filter detection scores, retaining targets above the threshold as personnel information output. Simultaneously, a feedback loop updates the mapping function with false positives / false negatives, achieving continuous adaptive optimization of the threshold and improving the confidence of detection in complex scenarios such as dust and low light.

[0085] In one embodiment of this application, the detection score is adaptively evaluated based on a dynamic threshold to determine personnel information, including:

[0086] The environmental adaptability index is determined based on the complexity of the underground coal mine environment.

[0087] If the environmental adaptability index is less than the preset performance threshold, the parameter adjustment mechanism is triggered to generate an updated threshold.

[0088] Personnel information is determined based on the updated threshold.

[0089] In one embodiment of this application, in order to measure the volatility of environmental complexity, environmental complexity is calculated using image gradients. Then, based on the environmental complexity in a coal mine, the Environmental Adaptability Index (EAI) is determined as follows:

[0090]

[0091] Where AP(t) represents the average precision at time t, and ENV(t) represents the environmental complexity. max This represents the preset maximum environmental complexity threshold, Var represents the variance of environmental complexity, and T represents the total time.

[0092] In one embodiment of this application, if the environmental adaptability index is less than a preset performance threshold, a parameter adjustment mechanism is triggered to generate an update threshold. Specifically, by adjusting key parameters in the above steps, such as filter banks, model parameters, or nonlocal attention weights, the model's adaptability to complex environments is improved. Finally, personnel information is determined based on the update threshold.

[0093] The above process comprehensively evaluates model performance and environmental stability through the Environmental Adaptability Index (EAI). When the EAI falls below a performance threshold, a parameter adjustment mechanism is triggered, updating the detection threshold using a gradient ascent method. This mechanism reduces detection latency in sudden scenarios such as a surge in dust concentration, ensuring the real-time and reliable output of personnel information.

[0094] This application's technical solution acquires multimodal environmental images from underground coal mines using pre-set intelligent sensors. The environmental images are pre-processed to generate spatiotemporally aligned pre-processed features. Visible light and infrared light features from the pre-processed features are adaptively fused based on modal weights to generate fused features. The spatiotemporal context of personnel movement underground is extracted from these fused features to generate enhanced features. A pre-set detection model is updated based on these enhanced features to generate a target set containing personnel detection results. Based on the neighborhood targets corresponding to the target set, a detection score is determined for each target in the target set. The detection scores are adaptively evaluated based on dynamic thresholds to determine personnel information. Spatiotemporally aligned pre-processing solves the synchronization problem between visible light and infrared images, ensuring feature consistency. Modal weight fusion based on environmental perception automatically adjusts the contribution of the two modalities, improving feature reliability in complex scenarios such as dust and low light. Spatiotemporal context modeling captures personnel movement trajectories and group interactions, suppressing false detections. Combining neighborhood target information to optimize detection scores and dynamic threshold evaluation further ensures real-time performance and accuracy. The overall solution achieves high-precision, low-latency personnel detection in the variable underground environment, meeting the real-time and robust requirements of intelligent monitoring, and significantly reducing the false detection rate compared to traditional methods.

[0095] The following describes embodiments of the deep learning-based underground personnel detection device for coal mines according to this application, which can be used to execute the deep learning-based underground personnel detection method for coal mines described in the above embodiments of this application. It is understood that the deep learning-based underground personnel detection device for coal mines can be a computer program (including program code) running on a computer device; for example, the deep learning-based underground personnel detection device for coal mines is an application software. This deep learning-based underground personnel detection device for coal mines can be used to execute the corresponding steps in the method provided in the embodiments of this application. For details not disclosed in the embodiments of the deep learning-based underground personnel detection device for coal mines according to this application, please refer to the embodiments of the deep learning-based underground personnel detection method for coal mines described above.

[0096] Figure 3 A block diagram of a deep learning-based underground personnel detection device according to an embodiment of this application is shown.

[0097] Reference Figure 3 As shown, a deep learning-based underground personnel detection device for coal mines according to an embodiment of this application includes:

[0098] The acquisition unit 310 is used to acquire multimodal environmental images from underground coal mines through preset smart sensors, preprocess the environmental images, and generate spatiotemporally aligned preprocessed features.

[0099] Feature unit 320 is used to perform adaptive fusion based on modal weights on the visible light features and red network light features in the preprocessed features to generate fused features, and extract the spatiotemporal context relationship of the underground personnel movement from the fused features to generate enhanced features;

[0100] The set unit 330 is used to update the preset detection model based on the enhanced features and generate a target set containing personnel detection results;

[0101] The scoring unit 340 is used to determine the detection score of each target in the target set based on the neighborhood targets corresponding to the target set.

[0102] Evaluation unit 350 is used to perform adaptive evaluation of the detection score based on dynamic threshold to determine personnel information.

[0103] In this application, based on the aforementioned scheme, the step of adaptively fusing the visible light features and infrared light features in the preprocessed features based on modal weights to generate fused features includes: compressing and nonlinearly transforming the preprocessed features in the spatial dimension to generate modal weights for visible light in coal mines; extracting a first feature corresponding to visible light and a second feature corresponding to infrared light from the preprocessed features; and fusing the first feature and the second feature based on the modal weights for visible light in coal mines to generate fused features.

[0104] In this application, based on the aforementioned scheme, the step of compressing and nonlinearly transforming the preprocessed features in the spatial dimension to generate the modal weights of visible light in coal mines includes: compressing the preprocessed features in the spatial dimension to obtain channel-level descriptors; mapping the descriptors to the hidden layer space through a fully connected layer and an activation function; and then generating the modal weights a(t) of visible light in coal mines through a fully connected layer and an activation function as follows:

[0105]

[0106] Where W1 and W2 represent the weight matrices of the reduced-dimensional fully connected layer and the increased-dimensional fully connected layer, respectively, σ and ReLU represent the activation functions, and F pre (t) represents the preprocessing feature, H and W represent the total height and total width of the preprocessing feature, h and w represent the height and width identifiers of the preprocessing feature, and t represents time.

[0107] In this application, based on the aforementioned scheme, the step of fusing the first feature and the second feature based on the mode weights of visible light in the coal mine to generate a fused feature includes: fusing the first feature and the second feature based on the mode weights of visible light in the coal mine to generate a fused feature F. fuse (t) is:

[0108] F fuse (t)=a(t)·F vis (t)+(1-a(t))·F ir (t)

[0109] Among them, F vis (t), F ir (t) represents the first feature and the second feature, respectively.

[0110] In this application, based on the aforementioned scheme, the step of extracting the spatiotemporal context of personnel movement in the fused features and generating enhanced features includes: extracting mapping features from the fused features through a preset convolutional network; and extracting the spatiotemporal context of personnel movement in the fused features based on the mapping features and generating enhanced features.

[0111] In this application, based on the aforementioned scheme, the step of updating the preset detection model based on the enhanced features to generate a target set containing personnel detection results includes: inputting the enhanced features into the detection model to obtain prediction results, generating a total loss based on the prediction results and preset real labels, updating the detection model through the total loss, and generating a target set.

[0112] In this application, based on the aforementioned scheme, determining the detection score corresponding to each target in the target set based on the neighboring targets corresponding to the target set includes: detecting the movement speed of the candidate targets in the target set; determining the neighboring targets of the candidate targets based on the spatial overlap of the candidate targets, and constructing a neighboring set; and determining the detection score corresponding to each target in the target combination based on the movement speed and the neighboring targets in the neighboring set.

[0113] In this application, based on the aforementioned scheme, the step of performing adaptive evaluation of the detection score based on a dynamic threshold to determine personnel information includes: determining an environmental adaptability index based on the environmental complexity of the coal mine; if the environmental adaptability index is less than a preset performance threshold, triggering a parameter adjustment mechanism to generate an updated threshold; and determining personnel information based on the updated threshold.

[0114] This application's technical solution acquires multimodal environmental images from underground coal mines using pre-set intelligent sensors. The environmental images are pre-processed to generate spatiotemporally aligned pre-processed features. Visible light and infrared light features from the pre-processed features are adaptively fused based on modal weights to generate fused features. The spatiotemporal context of personnel movement underground is extracted from these fused features to generate enhanced features. A pre-set detection model is updated based on these enhanced features to generate a target set containing personnel detection results. Based on the neighborhood targets corresponding to the target set, a detection score is determined for each target in the target set. The detection scores are adaptively evaluated based on dynamic thresholds to determine personnel information. Spatiotemporally aligned pre-processing solves the synchronization problem between visible light and infrared images, ensuring feature consistency. Modal weight fusion based on environmental perception automatically adjusts the contribution of the two modalities, improving feature reliability in complex scenarios such as dust and low light. Spatiotemporal context modeling captures personnel movement trajectories and group interactions, suppressing false detections. Combining neighborhood target information to optimize detection scores and dynamic threshold evaluation further ensures real-time performance and accuracy. The overall solution achieves high-precision, low-latency personnel detection in the variable underground environment, meeting the real-time and robust requirements of intelligent monitoring, and significantly reducing the false detection rate compared to traditional methods.

[0115] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0116] It should be noted that the computer system of the electronic device in this embodiment is only an example and should not impose any limitations on the function and scope of use of the embodiments of this application.

[0117] In this embodiment, the computer system includes a central processing unit 401, which can perform various appropriate actions and processes based on a program stored in the read-only memory 402 or a program loaded from the storage section 408 into the random access memory 403, such as executing the deep learning-based coal mine personnel detection method described in the above embodiment. The random access memory 403 also stores various programs and data required for system operation. The central processing unit 401, the read-only memory 402, and the random access memory 403 are interconnected via a bus 404. An input / output interface 405 is also connected to the bus 404.

[0118] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0119] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit 401, it performs various functions defined in the system of this application.

[0120] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0123] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0124] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the deep learning-based method for detecting personnel underground in coal mines as described in the above embodiments.

[0125] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0126] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0127] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0128] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A deep learning-based method for detecting personnel underground in coal mines, characterized in that, include: Multimodal environmental images are acquired from underground coal mines using pre-set intelligent sensors. The environmental images are then pre-processed to generate spatiotemporally aligned pre-processed features. The visible light features and red network light features in the preprocessed features are subjected to adaptive fusion based on modal weights to generate fused features. The spatiotemporal context of personnel movement in the mine is extracted from the fused features to generate enhanced features. The preset detection model is updated based on the enhanced features to generate a target set containing personnel detection results; Based on the neighborhood targets corresponding to the target set, determine the detection score corresponding to each target in the target set; The detection scores are adaptively evaluated based on dynamic thresholds to determine personnel information.

2. The deep learning-based method for detecting personnel underground in coal mines according to claim 1, characterized in that, The visible light features and red network features in the preprocessed features are adaptively fused based on modal weights to generate fused features, including: The preprocessed features are compressed and nonlinearly transformed in the spatial dimension to generate the modal weights of visible light in underground coal mines. Extract the first feature corresponding to visible light and the second feature corresponding to infrared light from the preprocessed features; Based on the modal weights of visible light in the coal mine, the first feature and the second feature are fused to generate a fused feature.

3. The deep learning-based method for detecting personnel underground in coal mines according to claim 2, characterized in that, The preprocessed features are compressed and nonlinearly transformed in the spatial dimension to generate mode weights for visible light in coal mines, including: Spatial dimensionality compression is performed on the preprocessed features to obtain channel-level descriptors; The descriptor is mapped to the hidden layer space through a fully connected layer and an activation function. Then, through a fully connected layer and an activation function, the modal weights a(t) of visible light in coal mines are generated as follows: Where W1 and W2 represent the weight matrices of the reduced-dimensional fully connected layer and the increased-dimensional fully connected layer, respectively, σ and ReLU represent the activation functions, and F pre (t) represents the preprocessing feature, H and W represent the total height and total width of the preprocessing feature, h and w represent the height and width identifiers of the preprocessing feature, and t represents time.

4. The deep learning-based method for detecting personnel underground in coal mines according to claim 3, characterized in that, Based on the mode weights of visible light in the coal mine, the first feature and the second feature are fused to generate a fused feature, including: Based on the mode weights of visible light in the coal mine, the first feature and the second feature are fused to generate a fused feature F. fuse (t) is: F fuse (t)=a(t)·F vis (t)+(1-a(t))·F ir (t) Among them, F vis (t), F ir (t) represents the first feature and the second feature, respectively.

5. The deep learning-based method for detecting personnel underground in coal mines according to claim 1, characterized in that, Extracting the spatiotemporal context of personnel movement underground from the fused features to generate enhanced features, including: Mapping features are extracted from the fused features using a pre-defined convolutional network; Based on the mapping features, the spatiotemporal context of personnel movement in the mine is extracted from the fused features to generate enhanced features.

6. The deep learning-based method for detecting personnel underground in coal mines according to claim 1, characterized in that, Based on the enhanced features, the preset detection model is updated to generate a target set containing personnel detection results, including: The enhanced features are input into the detection model to obtain the prediction result. Based on the prediction result and the preset real label, a total loss is generated. The detection model is updated by the total loss to generate the target set.

7. The deep learning-based method for detecting personnel underground in coal mines according to claim 1, characterized in that, Based on the neighborhood targets corresponding to the target set, determine the detection score corresponding to each target in the target set, including: The movement speed of the candidate targets in the target set is detected, the neighborhood targets of the candidate targets are determined according to the spatial overlap of the candidate targets, and a neighborhood set is constructed. Based on the movement speed and the neighboring targets in the neighborhood set, the detection score corresponding to each target in the target combination is determined.

8. The deep learning-based method for detecting personnel underground in coal mines according to claim 1, characterized in that, The detection score is adaptively evaluated based on a dynamic threshold to determine personnel information, including: The environmental adaptability index is determined based on the complexity of the underground coal mine environment. If the environmental adaptability index is less than the preset performance threshold, the parameter adjustment mechanism is triggered to generate an updated threshold. Personnel information is determined based on the updated threshold.

9. A deep learning-based underground personnel detection device for coal mines, characterized in that, include: The acquisition unit is used to acquire multimodal environmental images from underground coal mines through preset smart sensors, preprocess the environmental images, and generate spatiotemporally aligned preprocessed features. The feature unit is used to perform adaptive fusion based on modal weights on the visible light features and red network light features in the preprocessed features to generate fused features, and extract the spatiotemporal context relationship of the underground personnel movement from the fused features to generate enhanced features; The set unit is used to update the preset detection model based on the enhanced features and generate a target set containing the personnel detection results; The scoring unit is used to determine the detection score of each target in the target set based on the neighborhood targets corresponding to the target set. An evaluation unit is used to perform an adaptive evaluation of the detection score based on a dynamic threshold to determine personnel information.

10. The deep learning-based underground personnel detection device for coal mines according to claim 9, characterized in that, The visible light features and red network features in the preprocessed features are adaptively fused based on modal weights to generate fused features, including: The preprocessed features are compressed and nonlinearly transformed in the spatial dimension to generate the modal weights of visible light in underground coal mines. Extract the first feature corresponding to visible light and the second feature corresponding to infrared light from the preprocessed features; Based on the modal weights of visible light in the coal mine, the first feature and the second feature are fused to generate a fused feature.

Images