Coal mine behavior detection method and device, electronic equipment and storage medium

Abstract

The invention provides a coal mine behavior detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining multi-modal original data of a coal mine operation area, carrying out the feature extraction of the multi-modal original data, obtaining a multi-modal feature, and carrying out the detection of the multi-modal feature. The multi-modal features at least comprise image features corresponding to the image data, radar features corresponding to the radar point cloud and sound features corresponding to the sound data, carrying out feature fusion on the multi-modal features based on an attention mechanism to obtain a fusion feature vector, and inputting the fusion feature vector into a preset behavior recognition model, the behavior classification result of the fusion feature vector is obtained, risk level judgment is carried out based on the behavior classification result, behavior detection and alarm are carried out on the coal mine operation area according to the risk level, and the technical problem that the safety detection accuracy is low in related technologies is solved through the method.

Description

A method, apparatus, electronic device, and storage medium for detecting coal mine behavior. Technical Field

[0001] This invention relates to the field of coal mine safety detection technology, specifically to a coal mine behavior detection method, device, electronic equipment, and storage medium. Background Technology

[0002] With the continuous increase in coal mining depth and the extension of mining operations, the underground working environment is becoming increasingly complex, characterized by low light, high dust, high noise, limited space, and numerous obstacles. Under these conditions, traditional safety monitoring systems that rely on a single modality (especially visible light video surveillance) face severe challenges: low light and dust interference lead to serious degradation of image quality, resulting in a significant decrease in the accuracy of vision-based personnel behavior recognition and frequent false alarms and missed alarms.

[0003] Most existing coal mine safety monitoring systems rely on passive surveillance using fixed-point cameras or single-source data analysis based on preset rules (such as UWB (Ultra Wide Band) location-based boundary crossing alarms). These monitoring methods are rigid and lack sufficient intelligence, making it difficult to achieve accurate and real-time perception and early warning of complex and dynamically unsafe behaviors by personnel. They are no longer able to meet the new safety management demands of the increasingly intelligent and less-staffed development of coal mines. When facing dynamic and temporary underground work scenarios, existing systems suffer from poor environmental adaptability, superficial behavioral understanding, delayed early warning response, and blind spots in monitoring coverage. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for detecting behavior in coal mines, in order to solve the technical problem of low accuracy in safety detection using related technologies.

[0005] This invention provides a coal mine behavior detection method, the method comprising: acquiring multimodal raw data of a coal mine operating area, the multimodal raw data including at least image data, radar point clouds, and sound data; extracting features from the multimodal raw data to obtain multimodal features, the multimodal features including at least image features corresponding to the image data, radar features corresponding to the radar point clouds, and sound features corresponding to the sound data; fusing the multimodal features based on an attention mechanism to obtain a fused feature vector, inputting the fused feature vector into a preset behavior recognition model to obtain a behavior classification result of the fused feature vector, and determining a risk level based on the behavior classification result, so as to perform behavior detection and alarm for the coal mine operating area according to the risk level.

[0006] In one embodiment of the present invention, inputting the fused feature vector into a preset behavior recognition model to obtain the behavior classification result of the fused feature vector, and determining the risk level based on the behavior classification result includes: inputting the fused feature vector into the preset behavior recognition model based on a self-attention mechanism and a three-dimensional convolutional neural network to obtain the behavior classification result; if the behavior classification result is a preset dangerous behavior, then calculating a risk value based on the duration of the dangerous behavior and the historical records of dangerous behaviors in the coal mine operation area; if the risk value is greater than or equal to a preset first risk threshold, then determining the risk level as high risk; if the risk value is less than the preset first risk threshold and greater than or equal to a preset second risk threshold, then determining the risk level as medium risk; if the risk value is less than the preset second risk threshold, then determining the risk level as low risk.

[0007] In one embodiment of the present invention, the behavior detection and alarm of the coal mine operation area according to the risk level includes: if the risk level is low risk, a terminal prompt alarm is issued; if the risk level is medium risk, an audible and visual alarm is triggered and information is pushed to the area safety officer; if the risk level is high risk, an audible and visual alarm is triggered and information is pushed to the area safety officer, and if no response is received within a preset waiting time, the area power is cut off.

[0008] In one embodiment of the present invention, feature fusion based on an attention mechanism to obtain a fused feature vector includes: extracting the cross-modal feature interaction relationship between the image features, the radar features, and the sound features through the attention mechanism; and performing feature fusion based on a preset contribution weight and the interaction relationship to obtain the fused feature vector.

[0009] In one embodiment of the present invention, acquiring multimodal raw data of a coal mine operation area includes: acquiring image data based on a dual-spectrum camera, acquiring radar point cloud based on a dual-frequency millimeter-wave radar, and acquiring sound data based on a multi-channel microphone array; the dual-spectrum camera, the dual-frequency millimeter-wave radar, and the multi-channel microphone array are synchronized with bus time synchronization via hardware triggering to synchronously acquire the multimodal raw data at a preset frequency.

[0010] In one embodiment of the present invention, before feature extraction of the multimodal raw data, the method further includes: synchronizing the image data, the radar point cloud, and the sound data according to a preset time reference, and dynamically adjusting and correcting the asynchronous data; and mapping the pixel coordinates of the image data, the spatial coordinates of the radar point cloud, and the azimuth information of the sound data to a unified spatial coordinate system under the three-dimensional dynamic map established based on synchronous positioning and map building technology.

[0011] In one embodiment of the present invention, inputting the fused feature vector into a preset behavior recognition model includes: performing image recognition on the fused feature vector to obtain the worker's helmet protection status; calculating the worker's abnormal posture status based on posture key points; and determining the worker's spatial violation status based on the percentage of pixels where the worker's center of mass falls into a preset danger zone; performing radar motion trajectory analysis on the fused feature vector to obtain the worker's abnormal movement status; obtaining the worker's equipment contact violation status based on the distance between the worker and a preset dangerous equipment point cloud; and obtaining the worker's micro-movement abnormal status based on frequency analysis; performing voiceprint and spectrum analysis on the fused feature vector to obtain the worker's voice distress call status; and obtaining the worker's equipment abnormal operation status based on Mel spectrum entropy values; and determining the behavior classification result based on the worker's helmet protection status, abnormal posture status, spatial violation status, abnormal movement status, equipment contact violation status, micro-movement abnormal status, voice distress call status, and equipment abnormal operation status.

[0012] The present invention also provides a coal mine behavior detection device, the device comprising: a data input module for acquiring multimodal raw data of a coal mine operating area, the multimodal raw data including at least image data, radar point clouds, and sound data; a feature extraction module for extracting features from the multimodal raw data to obtain multimodal features, the multimodal features including at least image features corresponding to the image data, radar features corresponding to the radar point clouds, and sound features corresponding to the sound data; and a behavior detection module for performing feature fusion on the multimodal features based on an attention mechanism to obtain a fused feature vector, inputting the fused feature vector into a preset behavior recognition model to obtain a behavior classification result of the fused feature vector, and determining a risk level based on the behavior classification result, so as to perform behavior detection and alarm on the coal mine operating area according to the risk level.

[0013] The present invention also provides an electronic device, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the coal mine behavior detection method as described in any of the above embodiments.

[0014] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a computer processor, causes the computer to perform any of the coal mine behavior detection methods described in the above embodiments.

[0015] The beneficial effects of this invention are as follows: This invention proposes a coal mine behavior detection method, device, electronic device, and storage medium. It acquires multimodal raw data from a coal mine operating area, including at least image data, radar point clouds, and sound data. Feature extraction is performed on the multimodal raw data to obtain multimodal features, which include at least image features corresponding to the image data, radar features corresponding to the radar point clouds, and sound features corresponding to the sound data. Feature fusion is then performed on the multimodal features based on an attention mechanism to obtain a fused feature vector. This fused feature vector is then input into a preset behavior recognition model to obtain the fused feature vector. The system classifies behaviors and determines risk levels based on these classifications. This allows for behavior detection and alarms in coal mine operating areas according to the risk level. By simultaneously collecting and fusing perception data from three modalities—images, radar point clouds, and sound—the system constructs a redundant and complementary perception system. Even when a single modality fails, other modalities can still provide effective information. This overcomes the inherent defects of traditional single-vision monitoring systems, such as a sharp drop in recognition rate and high false alarm rates in complex underground scenarios. It ensures the stable operation of the monitoring system around the clock and under all operating conditions. Furthermore, it establishes an intelligent decision-making chain from behavior recognition to risk assessment, achieving precise and differentiated early warnings.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0018] In the attached diagram:

[0019] Figure 1 is a schematic diagram of the implementation environment of a coal mine behavior detection method according to an embodiment of the present invention; Figure 2 is a flowchart of a coal mine behavior detection method according to an embodiment of the present invention; Figure 3 is a block diagram of a coal mine behavior detection device according to an embodiment of the present invention; Figure 4 is a structural schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0023] Please refer to Figure 1, which is a schematic diagram of the implementation environment of a coal mine behavior detection method according to an embodiment of the present invention. As shown in Figure 1, the implementation environment includes a multimodal sensor 110 and a computer device 120. The multimodal sensor 110 may include a dual-spectrum (infrared + visible light) camera that can operate under low illumination and high dust concentration, a dual-frequency millimeter-wave radar with dust penetration capability, and a microphone array with high dynamic range and adaptive noise reduction function. The computer device 120 may be at least one of a microcomputer, an embedded computer, a neural network computer, etc.

[0024] For example, multimodal raw data of the coal mine operation area is acquired by a multimodal sensor 110. The multimodal raw data includes at least image data, radar point cloud, and sound data. The multimodal raw data is then sent to a computer device 120. The computer device 120 performs time synchronization and spatial alignment processing on the multimodal raw data and extracts multimodal features. The multimodal features include at least image features, radar features, and sound features. Based on an attention mechanism, the multimodal features are fused to obtain a fused feature vector. The fused feature vector is then input into a preset behavior recognition model to obtain the behavior classification result of the fused feature vector and to determine the risk level, so as to detect and alarm the safety behavior in the coal mine operation area.

[0025] In one embodiment of the present invention, the implementation environment can also be a coal mine behavior detection system. This system includes a perception layer, an edge layer, a cloud layer, a terminal layer, and an information and data management layer. The perception layer is used to collect multimodal raw data, including a dual-spectrum camera with automatic dust removal, a dual-band millimeter-wave radar, and an array microphone. The perception layer is deployed in coal mine operating areas such as underground tunneling faces and transport containers. The edge layer includes an industrial-grade edge computing gateway for preprocessing, feature extraction, and feature fusion calculation of the multimodal raw data. The cloud layer includes a cloud platform, a CPU (Central Processing Unit) server, and a distributed storage system for training and updating the behavior recognition model, global data aggregation, and historical backtracking analysis. The terminal layer includes miners' positioning terminals, safety officer handheld terminals, and a dispatch center monitoring platform for receiving alarm information and performing hierarchical linkage control. The communication and data management layer is used to realize cloud-edge-device collaborative operation.

[0026] Please refer to Figure 2, which is a flowchart of a coal mine behavior detection method provided in one embodiment of the present invention. This method can be applied to the implementation environment shown in Figure 1, and can also be applied to other exemplary implementation environments and specifically executed by devices in those environments. This embodiment does not limit the implementation environment to which this method is applicable.

[0027] As shown in Figure 2, in an exemplary embodiment, the coal mine behavior detection method includes at least steps S210 to S240, which are described in detail below: Step S210: Obtain multimodal raw data of the coal mine operation area. The multimodal raw data includes at least image data, radar point cloud and sound data.

[0028] In one embodiment of the present invention, acquiring multimodal raw data of a coal mine operation area includes: acquiring image data based on a dual-spectrum camera, acquiring radar point clouds based on a dual-frequency millimeter-wave radar, and acquiring sound data based on a multi-channel microphone array; the dual-spectrum camera, dual-frequency millimeter-wave radar, and multi-channel microphone array are synchronized with the bus time synchronization through hardware triggering to synchronously acquire multimodal raw data at a preset frequency.

[0029] For example, the image data acquisition can be performed using a high-definition dual-spectrum camera (infrared + visible light) with a resolution of 2560×1440, a frame rate of 25fps, a wide dynamic range of ≥120dB, and a lens with a dustproof and anti-fog coating. It can stably image under conditions of 0.01Lux illuminance and dust concentration ≤100mg / m³, with a coverage radius of 15–20m.

[0030] For example, radar point cloud acquisition can be achieved using a 77GHz / 24GHz dual-frequency millimeter-wave radar. The 77GHz channel is used for motion trajectory detection (positioning error ≤0.3m, sampling rate 10Hz), and the 24GHz channel is used for penetrating dust to detect static obstacles (detection distance ≥30m).

[0031] For example, the acquisition of sound data can be achieved by deploying an 8-channel microphone array with a sampling rate of 48kHz, a dynamic range of 96dB, and integrated adaptive noise suppression (≥30dB noise reduction), which can accurately capture shouts, collision sounds, etc. in an environment with 85dB background noise.

[0032] Step S220: Extract features from the multimodal raw data to obtain multimodal features. The multimodal features include at least image features corresponding to the image data, radar features corresponding to the radar point cloud, and sound features corresponding to the sound data.

[0033] In one embodiment of the present invention, before feature extraction of the multimodal raw data, the method further includes: synchronizing the image data, radar point cloud and sound data according to a preset time reference, and dynamically adjusting and correcting the asynchronous data; and mapping the pixel coordinates of the image data, the spatial coordinates of the radar point cloud and the azimuth information of the sound data to a unified spatial coordinate system under the three-dimensional dynamic map based on synchronous positioning and map building technology.

[0034] For example, hardware triggering and CAN bus time synchronization are used for time synchronization, with a synchronization error ≤1ms; and the Dynamic Time Warping (DTW) algorithm is used to correct asynchronous frames. A three-dimensional dynamic map of the well is constructed based on SLAM (Simultaneous Localization and Mapping), mapping the pixel coordinates (u,v) of image data, the spatial coordinates (x,y,z) of radar point clouds, and the sound azimuth angles (θ,φ) of sound data to a unified coordinate system; and a trajectory prediction model is used to correct spatial deviations caused by personnel movement (correction accuracy ≤0.5m).

[0035] For example, image feature extraction includes extracting texture and pose features of key human body parts (helmet, hands, torso) using a MobileViT network and outputting a 512-dimensional feature vector: ,in For image features, This is image data.

[0036] For example, extracting radar features includes extracting trajectory velocity, acceleration, and micro-body movements (such as tremor frequency) based on point cloud clustering and motion vector decomposition, outputting a 256-dimensional vector: ,in As a radar characteristic, Radar point cloud.

[0037] For example, extracting sound features includes: extracting speech and device noise using Mel spectrum and a voiceprint recognition model, outputting a 128-dimensional vector: ,in For sound characteristics, This is audio data.

[0038] Step S230: Based on the attention mechanism, feature fusion is performed on multimodal features to obtain a fused feature vector. The fused feature vector is then input into a preset behavior recognition model to obtain the behavior classification result of the fused feature vector. Based on the behavior classification result, the risk level is determined so as to detect and alarm the behavior in the coal mine operation area according to the risk level.

[0039] In one embodiment of the present invention, feature fusion based on attention mechanism to obtain fused feature vector includes: extracting cross-modal feature interaction relationship between image features, radar features and sound features through attention mechanism; and performing feature fusion based on preset contribution weight and interaction relationship to obtain fused feature vector.

[0040] For example, image features, radar features, and sound features are mapped to a common feature space through their respective projection matrices. A cross-modal multi-head self-attention mechanism is used to calculate the degree of correlation (i.e., interaction relationship) between each pair of image features, radar features, and sound features in the common feature space. Based on the degree of correlation, the features of each modality are weighted and fused with contextual information so that the features of each modality can be fused with supplementary information from other modalities, thereby extracting enhanced features that represent cross-modal interaction relationships.

[0041] For example, deep feature fusion is achieved using a cross-modal multi-head self-attention mechanism, including: In equation (1), For image features, As a radar characteristic, For sound characteristics, The image features, radar features, and sound features are each assigned a pre-defined projection matrix (including pre-defined contribution weights), and the final output is a 1024-dimensional fused feature. .

[0042] The preset contribution weights in the preset projection matrix can be adjusted according to the actual situation. For example, if the illumination is too low, the contribution weights corresponding to radar features will be increased and the contribution weights corresponding to image features will be decreased.

[0043] In one embodiment of the present invention, inputting the fused feature vector into a preset behavior recognition model includes: performing image recognition on the fused feature vector to obtain the worker's helmet protection status; calculating the worker's abnormal posture status based on posture key points; and determining the worker's spatial violation status based on the percentage of pixels in the human body's center of mass falling into a preset dangerous area; performing radar motion trajectory analysis on the fused feature vector to obtain the worker's abnormal movement status; determining the worker's equipment contact violation status based on the distance between the worker and the preset dangerous equipment point cloud; and determining the worker's micro-movement abnormal status based on frequency analysis; performing voiceprint and spectrum analysis on the fused feature vector to obtain the worker's voice distress call status; and determining the worker's equipment abnormal operation status based on Mel spectrum entropy values; and determining the behavior classification result based on the worker's helmet protection status, abnormal posture status, spatial violation status, abnormal movement status, equipment contact violation status, micro-movement abnormal status, voice distress call status, and equipment abnormal operation status.

[0044] For example, an improved YOLOv8 algorithm is used to analyze image features. Analysis was performed to identify key risk labels, including: through semantic segmentation of safety helmets / work clothes (IoU ≥ 0.92), if there are no safety helmet features in the head area, the worker's safety helmet protection status is marked as "not wearing a safety helmet" (confidence level). Based on MediaPipe posture key points, limb angles are calculated. If a worker's bending angle is detected to be >120° and lasts for >5 seconds, the worker's abnormal posture state is marked as "fatigue work" (confidence level). ); Dangerous areas are pre-defined. Through pixel mapping of electronic fences, if the percentage of pixels in a worker's center of gravity falling within the danger zone is greater than 30%, the worker's spatial violation status is marked as "Entering a restricted area" (confidence level). ).

[0045] For example, by analyzing the trajectory and distance characteristics of radar features, if the deviation between the Kalman filter predicted trajectory and the actual trajectory of the operator is >0.5m (for 3 consecutive frames), then the operator's abnormal movement state is marked as "abnormal running" (confidence level). If the distance between the worker and the pre-defined hazardous equipment point cloud cluster center is less than 1m, then the worker's equipment contact violation status will be marked as "equipment contact violation" (confidence level). ); Analyze the frequency of radar characteristics of workers. If the detected tremor frequency of a worker is >3Hz (deviating from the normal working threshold), then the abnormal state of the worker's micro-movements is marked as "abnormal physical state" (confidence level). ).

[0046] For example, based on the spectral and voiceprint analysis of sound features, if keywords such as "call for help" and "danger" are detected using the Wav2Vec2 model (cosine similarity ≥ 0.88), then the worker's voice distress call status is marked as "emergency distress call" (confidence level ≥ 0.88). If a Mel spectrum entropy value > 5.0 (collision sound) or < 2.8 (metal friction sound) is detected, the operator's abnormal equipment operation status is marked as "abnormal equipment operation" (confidence level). ).

[0047] In one embodiment of the present invention, inputting the fused feature vector into a preset behavior recognition model to obtain the behavior classification result of the fused feature vector, and determining the risk level based on the behavior classification result includes: inputting the fused feature vector into a preset behavior recognition model based on a self-attention mechanism and a three-dimensional convolutional neural network to obtain the behavior classification result; if the behavior classification result is a preset dangerous behavior, then calculating a risk value based on the duration of the dangerous behavior and the historical records of dangerous behaviors in the coal mine operation area; if the risk value is greater than or equal to a preset first risk threshold, then determining the risk level as high risk; if the risk value is less than the preset first risk threshold but greater than or equal to a preset second risk threshold, then determining the risk level as medium risk; if the risk value is less than the preset second risk threshold, then determining the risk level as low risk.

[0048] For example, the pre-defined dangerous behaviors include at least not wearing a safety helmet, working while fatigued, entering a restricted area, running abnormally, violating equipment contact regulations, abnormal physical condition, calling for help, and abnormal equipment operation.

[0049] In one embodiment of the present invention, the behavior detection and alarm of the coal mine operation area according to the risk level includes: if the risk level is low, a terminal prompt alarm is issued; if the risk level is medium, an audible and visual alarm is triggered and information is pushed to the area safety officer; if the risk level is high, an audible and visual alarm is triggered and information is pushed to the area safety officer, and if no response is received within a preset waiting time, the area power is cut off.

[0050] For example, the calculation method for risk level determination is as follows: In equation (2), The value represents the risk level, where L represents the distance between the worker and the hazard source (pre-defined hazardous area or pre-defined hazardous equipment), T represents the duration of the hazardous behavior, and H represents the historical number of hazardous behaviors in the coal mine work area. Indicates the preset weight, satisfying

[0051] For example, when R ≥ 0.8, the risk level is determined to be high risk; when 0.5 ≤ R < 0.8, the risk level is determined to be medium risk; and when R < 0.5, the risk level is determined to be low risk. In this embodiment, the first risk threshold is preset to 0.8, and the second risk threshold is preset to 0.5.

[0052] In one embodiment of the present invention, if the coal mine operation area is a hydraulic support operation area, and multiple workers are detected entering the equipment operation area simultaneously, according to safety regulations, a single-person operation system should be implemented in the hydraulic support operation area, and other workers must maintain a safe distance of more than 2 meters. However, during an actual operation, two workers were detected entering the hydraulic support operation area simultaneously, one as the main operator and the other approaching the preset operating equipment within 0.8 meters, operating the preset operating equipment simultaneously. This constitutes a clear violation of parallel operation regulations, i.e., the distance between the other worker and the preset operating equipment is less than 2 meters. Therefore, this worker is considered to be "violating regulations by operating in parallel" and is marked.

[0053] In an embodiment of the invention, a dual-spectrum camera captured two workers simultaneously appearing within 1 meter in front of the hydraulic support in visible light mode. The three-dimensional coordinates of the two workers were determined using a dynamic 3D map of the well, resulting in a calculated spatial distance of only 0.8 meters between them, lower than the system's preset safety threshold of 2 meters. Further verification of the relative positions of the two workers and the preset operating equipment was conducted using radar point cloud data. The point cloud trajectory showed a high degree of overlap in the activity ranges of the two workers, with a spatial overlap coefficient reaching 0.42, significantly exceeding the risk threshold of 0.3. Therefore, the workers were deemed to be "operating in parallel without authorization" and were marked accordingly.

[0054] For example, "unauthorized parallel operation" is a dangerous behavior. A risk level is determined for this behavior, and based on the risk level, a 110dB high-decibel alarm and a red flashing warning are triggered. An edge computing node directly issues a control command, causing the hydraulic support equipment to stop within one second, preventing an accident. Simultaneously, the dispatch center receives the alarm information and the three-dimensional coordinates of the personnel involved (X=352.7, Y=210.5, Z=-63.8), and verifies this information using real-time video footage. Safety personnel quickly intervene and guide the unauthorized personnel to evacuate the danger zone. After evacuation, the system confirms through image data recognition that only one person remains in the work area, and radar trajectory detection results also indicate no overlapping activity. The system ultimately determines that the risk has been eliminated and restores normal equipment operation.

[0055] In one embodiment of the present invention, if the image feature detection can only vaguely identify the human outline due to excessive dust interference in the image data, the confidence level for marking it as "not wearing a safety helmet" is only 0.6 (low confidence); based on radar features, the relative position (distance < 0.3m) between the person's head and the preset hazardous equipment is clearly captured, and the trajectory shows continuous contact with the equipment button, marking it as "equipment contact violation" with a confidence level of 0.92 (high confidence); based on sound feature analysis, the abnormal start-up sound of the equipment (spectral entropy value of 5.3) and the head collision sound (short pulse sound) when the person is not wearing a safety helmet are obtained, marking it as "abnormal equipment operation" with a confidence level of 0.88. The projection matrices corresponding to each modal feature are set according to the actual situation. Because the image quality is low due to excessive dust interference in this embodiment, the projection matrices corresponding to the image features, radar features, and sound features can be set to... Cross-validation revealed that both radar and acoustic features pointed to "abnormal equipment operation." The final fusion identification result was "not wearing a safety helmet + abnormal equipment operation," and the corresponding risk level was determined to be R=0.85 (high risk), triggering a power outage and alarm.

[0056] Please refer to Figure 3, which is a block diagram of a coal mine behavior detection device provided in one embodiment of the present invention. This device can be applied to the implementation environment shown in Figure 1, and it can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0057] As shown in Figure 3, this exemplary coal mine behavior detection device includes: a data input module 310, used to acquire multimodal raw data of the coal mine operation area, the multimodal raw data including at least image data, radar point clouds, and sound data; a feature extraction module 320, used to extract features from the multimodal raw data to obtain multimodal features, the multimodal features including at least image features corresponding to image data, radar features corresponding to radar point clouds, and sound features corresponding to sound data; and a behavior detection module 330, used to perform feature fusion on the multimodal features based on an attention mechanism to obtain a fused feature vector, and input the fused feature vector into a preset behavior recognition model to obtain the behavior classification result of the fused feature vector, and to determine the risk level based on the behavior classification result, so as to perform behavior detection and alarm on the coal mine operation area according to the risk level.

[0058] The data input module 310 acquires multimodal raw data of the coal mine operation area, including: image data acquired based on a dual-spectrum camera, radar point cloud acquired based on a dual-frequency millimeter-wave radar, and sound data acquired based on a multi-channel microphone array; the dual-spectrum camera, dual-frequency millimeter-wave radar, and multi-channel microphone array are synchronized with the bus time synchronization through hardware triggering to acquire multimodal raw data synchronously at a preset frequency.

[0059] Before the feature extraction module 320 extracts features from the multimodal raw data, it also includes: synchronizing the image data, radar point cloud and sound data according to a preset time base, and dynamically adjusting and correcting the asynchronous data in time; and mapping the pixel coordinates of the image data, the spatial coordinates of the radar point cloud and the azimuth information of the sound data to a unified spatial coordinate system under the three-dimensional dynamic map based on synchronous positioning and map building technology.

[0060] The behavior detection module 330 performs feature fusion on multimodal features based on the attention mechanism to obtain a fused feature vector, which includes: extracting the interaction relationship between cross-modal features of image features, radar features and sound features through the attention mechanism; and performing feature fusion based on preset contribution weights and interaction relationships to obtain a fused feature vector.

[0061] The behavior detection module 330 inputs the fused feature vector into a preset behavior recognition model, including: performing image recognition on the fused feature vector to obtain the worker's helmet protection status; calculating the worker's abnormal posture status based on posture key points; and determining the worker's spatial violation status based on the percentage of pixels where the human center of mass falls into a preset danger zone; performing radar motion trajectory analysis on the fused feature vector to obtain the worker's abnormal movement status; determining the worker's equipment contact violation status based on the distance between the worker and the preset dangerous equipment point cloud; and determining the worker's micro-movement abnormal status based on frequency analysis; performing voiceprint and spectrum analysis on the fused feature vector to obtain the worker's voice distress call status; and determining the worker's equipment abnormal operation status based on Mel spectrum entropy values; and determining the behavior classification result based on the worker's helmet protection status, abnormal posture status, spatial violation status, abnormal movement status, equipment contact violation status, micro-movement abnormal status, voice distress call status, and equipment abnormal operation status.

[0062] The behavior detection module 330 inputs the fused feature vector into a preset behavior recognition model to obtain the behavior classification result of the fused feature vector, and determines the risk level based on the behavior classification result, including: inputting the fused feature vector into a preset behavior recognition model based on a self-attention mechanism and a three-dimensional convolutional neural network to obtain the behavior classification result; if the behavior classification result is a preset dangerous behavior, then calculating the risk value based on the duration of the dangerous behavior and the historical records of dangerous behaviors in the coal mine operation area; if the risk value is greater than or equal to a preset first risk threshold, then the risk level is determined to be high risk; if the risk value is less than the preset first risk threshold but greater than or equal to a preset second risk threshold, then the risk level is determined to be medium risk; if the risk value is less than the preset second risk threshold, then the risk level is determined to be low risk.

[0063] The behavior detection module 330 detects and alarms safety behaviors in the coal mine operation area, including: detecting and alarming behaviors in the coal mine operation area according to the risk level, including: if the risk level is low, a terminal prompt alarm is issued; if the risk level is medium, an audible and visual alarm is triggered and information is pushed to the area safety officer; if the risk level is high, an audible and visual alarm is triggered and information is pushed to the area safety officer, and if there is no response within a preset waiting time, the area power is cut off.

[0064] By using the aforementioned device, and by simultaneously collecting and fusing perception data from three modalities—images, radar point clouds, and sound—this solution constructs a redundant and complementary perception system. When a single modality fails, other modalities can still provide effective information. This overcomes the inherent defects of traditional single-vision monitoring systems, such as a sharp drop in recognition rate and high false alarm rates in complex downhole scenarios. It ensures the stable operation of the monitoring system around the clock and under all working conditions. At the same time, it establishes an intelligent decision-making chain from behavior recognition to risk assessment, achieving more accurate and differentiated early warnings.

[0065] It is understood that the coal mine behavior detection device and the coal mine behavior detection method provided in the above embodiments belong to the same concept. The specific way in which the coal mine behavior detection method performs its operation has been described in detail in the above embodiments and will not be repeated here. In practical applications, the coal mine behavior detection device provided in the above embodiments can be assigned to different functional modules as needed. That is, the internal structure of the coal mine behavior detection device can be divided into different functional modules, and then all or part of the functions of the corresponding functional modules can be implemented by the coal mine behavior detection method described in the above embodiments. No specific limitations are imposed here. For example, the data input module 310 includes steps for executing step S210 and related steps, the feature extraction module 320 includes steps for executing step S220 and related steps, and the behavior detection module 330 includes steps for executing step S230 and related steps.

[0066] Figure 4 is a schematic diagram of an electronic device provided in one embodiment of the present invention. It should be noted that the computer system 400 of the electronic device shown in Figure 4 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0067] As shown in Figure 4, the computer system 400 includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 402 or programs loaded from storage portion 408 into Random Access Memory (RAM) 403, such as performing the methods described in the above embodiments. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An Input / Output (I / O) interface 405 is also connected to the bus 404.

[0068] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0069] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs various functions defined in the system of the present invention.

[0070] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0071] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0072] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0073] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the coal mine behavior detection method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0074] Another aspect of the present invention provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the coal mine behavior detection method provided in the various embodiments described above.

[0075] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for detecting behavior in coal mines, characterized in that, The method includes: acquiring multimodal raw data of a coal mine operating area, wherein the multimodal raw data includes at least image data, radar point cloud, and sound data; extracting features from the multimodal raw data to obtain multimodal features, wherein the multimodal features include at least image features corresponding to the image data, radar features corresponding to the radar point cloud, and sound features corresponding to the sound data; fusing the multimodal features based on an attention mechanism to obtain a fused feature vector, and inputting the fused feature vector into a preset behavior recognition model to obtain a behavior classification result of the fused feature vector, and determining a risk level based on the behavior classification result, so as to perform behavior detection and alarm in the coal mine operating area according to the risk level.

2. The coal mine behavior detection method according to claim 1, characterized in that, The process of inputting the fused feature vector into a preset behavior recognition model to obtain the behavior classification result of the fused feature vector, and determining the risk level based on the behavior classification result includes: inputting the fused feature vector into the preset behavior recognition model based on a self-attention mechanism and a three-dimensional convolutional neural network to obtain the behavior classification result; if the behavior classification result is a preset dangerous behavior, then calculating a risk value based on the duration of the dangerous behavior and the historical records of dangerous behaviors in the coal mine operation area; if the risk value is greater than or equal to a preset first risk threshold, then determining the risk level as high risk; if the risk value is less than the preset first risk threshold and greater than or equal to a preset second risk threshold, then determining the risk level as medium risk; if the risk value is less than the preset second risk threshold, then determining the risk level as low risk.

3. The coal mine behavior detection method according to claim 1, characterized in that, The behavior detection and alarm for the coal mine operation area based on the risk level includes: if the risk level is low, a terminal prompt alarm is issued; if the risk level is medium, an audible and visual alarm is triggered and information is pushed to the area safety officer; if the risk level is high, an audible and visual alarm is triggered and information is pushed to the area safety officer, and if no response is received within a preset waiting time, the area power is cut off.

4. The coal mine behavior detection method according to any one of claims 1-3, characterized in that, The multimodal features are fused based on an attention mechanism to obtain a fused feature vector, which includes: extracting the cross-modal feature interaction relationship between the image features, the radar features, and the sound features through the attention mechanism; and performing feature fusion based on a preset contribution weight and the interaction relationship to obtain the fused feature vector.

5. The coal mine behavior detection method according to any one of claims 1-3, characterized in that, Acquiring multimodal raw data of the coal mine operation area includes: acquiring image data based on a dual-spectrum camera, acquiring radar point cloud based on a dual-frequency millimeter-wave radar, and acquiring sound data based on a multi-channel microphone array; the dual-spectrum camera, the dual-frequency millimeter-wave radar, and the multi-channel microphone array are synchronized with the bus time synchronization through hardware triggering to synchronously acquire the multimodal raw data at a preset frequency.

6. The coal mine behavior detection method according to any one of claims 1-3, characterized in that, Before performing feature extraction on the multimodal raw data, the method further includes: synchronizing the image data, the radar point cloud, and the sound data according to a preset time reference, and dynamically adjusting and correcting the asynchronous data in time; and mapping the pixel coordinates of the image data, the spatial coordinates of the radar point cloud, and the azimuth information of the sound data to a unified spatial coordinate system under the three-dimensional dynamic map established based on synchronous positioning and map building technology.

7. The coal mine behavior detection method according to any one of claims 1-3, characterized in that, The process of inputting the fused feature vector into a preset behavior recognition model includes: performing image recognition on the fused feature vector to obtain the worker's helmet protection status; calculating the worker's abnormal posture status based on posture key points; and determining the worker's spatial violation status based on the percentage of pixels where the worker's center of mass falls into a preset danger zone; performing radar motion trajectory analysis on the fused feature vector to obtain the worker's abnormal movement status; determining the worker's equipment contact violation status based on the distance between the worker and the preset dangerous equipment point cloud; and determining the worker's micro-movement abnormal status based on frequency analysis; performing voiceprint and spectrum analysis on the fused feature vector to obtain the worker's voice distress call status; and determining the worker's equipment abnormal operation status based on Mel spectrum entropy values; and determining the behavior classification result based on the worker's helmet protection status, abnormal posture status, spatial violation status, abnormal movement status, equipment contact violation status, micro-movement abnormal status, voice distress call status, and equipment abnormal operation status.

8. A coal mine behavior detection device, characterized in that, The device includes: a data input module for acquiring multimodal raw data of a coal mine operating area, the multimodal raw data including at least image data, radar point clouds, and sound data; a feature extraction module for extracting features from the multimodal raw data to obtain multimodal features, the multimodal features including at least image features corresponding to the image data, radar features corresponding to the radar point clouds, and sound features corresponding to the sound data; and a behavior detection module for fusing the multimodal features based on an attention mechanism to obtain a fused feature vector, inputting the fused feature vector into a preset behavior recognition model to obtain a behavior classification result of the fused feature vector, and determining a risk level based on the behavior classification result, so as to perform behavior detection and alarm for the coal mine operating area according to the risk level.

9. An electronic device, characterized in that, The electronic device includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the coal mine behavior detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by the computer's processor, causes the computer to perform the coal mine behavior detection method as described in any one of claims 1-7.

Images