Underground personnel collision warning method and system based on environmental degradation adaptive fusion

Abstract

The application discloses a kind of based on environmental degradation adaptive fusion underground personnel collision warning method and system, environmental degradation assessment is carried out to coal mine underground image data and point cloud data, and first degradation factor and second degradation factor are obtained;In the case where first degradation factor and second degradation factor are less than respective safety threshold, according to the speed vector of the last time coal mine personnel, short-time trajectory deduction is carried out, and the three-dimensional space state of personnel is obtained;Otherwise, according to first degradation factor, second degradation factor, image data and point cloud data, determine the objective function, solve the objective function, obtain the three-dimensional space state of personnel, to construct personnel dynamic direction bounding box and the dynamic direction bounding box of each component of heading machine;To construct space-time prediction swept volume bounding box and carry out collision detection, and obtain collision detection result;In the case where collision detection result is collision risk, according to collision detection result and preset warning threshold, generate hierarchical warning information.

Description

Technical Field

[0001] This application relates to the field of intelligent safety production monitoring technology in coal mines, specifically to a method and system for early warning of underground personnel collisions based on environmental degradation adaptive fusion. Background Technology

[0002] The mining environment in my country's coal mines is extremely complex, with narrow underground tunnels, poor lighting conditions, and high dust concentrations. Especially at the tunneling face, large tunneling equipment and workers are confined to the same space, making them highly susceptible to human-machine conflict leading to mechanical injuries and other safety accidents. To prevent such accidents, the industry has developed various underground personnel monitoring and early warning technologies, but significant limitations remain in practical application.

[0003] In terms of personnel sensing and positioning, existing underground personnel monitoring methods mainly rely on Radio Frequency Identification (RFID) or Ultra-Wideband (UWB) positioning cards to obtain the approximate location of personnel. However, the positioning accuracy of these technologies is easily affected by metal equipment and electromagnetic interference, and they cannot perceive the specific limb postures of personnel at all. In recent years, some systems have introduced single sensors for personnel identification and positioning, but the high dust concentration, uneven lighting, and low illuminance underground make single sensors highly susceptible to environmental interference, resulting in significant personnel positioning deviations. Even some existing technologies that attempt to overlay visual and radar data often use simple splicing with fixed weights. Due to the lack of adaptive assessment and dynamic adjustment mechanisms for environmental degradation, when extreme conditions such as sudden fog explosions or dust surges occur underground, causing severe degradation of data from a single sensor, its high-noise data directly contaminates the entire fusion calculation process, leading to severe drift or even complete failure of the system positioning.

[0004] In terms of hazard zone identification and collision detection, existing early warning systems mostly adopt traditional numerical comparison methods or use axis-aligned bounding box (AABB) technology to establish electronic fences. AABB bounding boxes have a fixed shape and cannot rotate with the equipment. When facing complex rotating machinery such as tunnel boring machine cutting arms or irregular human movement postures, they are difficult to closely fit the target, resulting in a large amount of redundant space in the defined hazard zone. In practical applications, this leads to a high false alarm rate and a high missed alarm rate. Summary of the Invention

[0005] This application aims to address the current lack of adaptive adjustment mechanisms in underground coal mines, which leads to severe drift or even complete failure in personnel positioning. At the same time, the determination of dangerous areas relies on fixed numerical thresholds or AABB bounding boxes, which are difficult to closely match rotating machinery targets, resulting in high false alarm and false negative rates.

[0006] To achieve the above objectives, this application provides the following technical solution: In a first aspect, embodiments of this application provide a method for early warning of personnel collisions in underground mines based on adaptive fusion of environmental degradation, including: Acquire image data and point cloud data from underground coal mines at the same time; Environmental degradation assessment was performed on image data and point cloud data to obtain a first degradation factor and a second degradation factor; If both the first degradation factor and the second degradation factor are less than their respective safety thresholds, the three-dimensional spatial state of the personnel is obtained by short-time trajectory extrapolation based on the velocity vector of the personnel in the coal mine at the previous moment; otherwise, the objective function is determined based on the first degradation factor, the second degradation factor, image data, and point cloud data, and the objective function is solved to obtain the three-dimensional spatial state of the personnel. Construct a dynamic orientation bounding box for personnel based on their three-dimensional spatial state; The dynamic orientation bounding boxes of each component of the tunneling machine are constructed based on the kinematic model of the tunneling machine; Based on the three-dimensional spatial state of personnel, the dynamic orientation bounding box of personnel, the dynamic orientation bounding box of each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine, a spatiotemporal prediction sweep body bounding box is constructed. Collision detection is performed on the bounding box of the spatiotemporal prediction sweep volume to obtain the collision detection results; If the collision detection result indicates a collision risk, a tiered warning message is generated based on the collision detection result and the preset warning threshold.

[0007] In some embodiments of this application, environmental degradation assessment is performed on image data and point cloud data to obtain a first degradation factor and a second degradation factor, including: Determine the information entropy of the image data; whereby information entropy is used to quantify the information richness of the image data. The first degradation factor is determined based on information entropy; Determine the proportion of discrete point cloud noise within the local space of people in the point cloud data; The second degradation factor is determined based on the noise ratio of discrete point clouds.

[0008] In embodiments of this application, the objective function is determined based on a first degradation factor, a second degradation factor, image data, and point cloud data, including: Determine the visual observation coordinates of personnel based on image data; Determine personnel radar observation coordinates based on point cloud data; The first cost term is determined based on the first degradation factor, the personnel's visual observation coordinates, and the visual measurement noise covariance matrix; The second cost term is determined based on the second degradation factor, personnel radar observation coordinates, and radar measurement noise covariance matrix; The objective function is determined based on the first cost term and the second cost term.

[0009] In some embodiments of this application, solving the objective function to obtain the three-dimensional spatial state of the person includes: The objective function is minimized by using a numerical optimization framework, and the personnel state vector corresponding to the minimum value is determined as the three-dimensional spatial state of the personnel. Among them, the numerical optimization framework represents a mathematical framework for solving the minimum value of the objective function in real time in an edge computing environment.

[0010] In some embodiments of this application, constructing a dynamic orientation bounding box for a person based on their three-dimensional spatial state includes: Determine the position coordinates and limb posture angles of the personnel based on their three-dimensional spatial state; A dynamic orientation bounding box for a person is generated based on their position coordinates and limb posture angles. The dynamic orientation bounding box represents a tightly bounding spatial boundary that is dynamically updated based on the person's position and posture and rotates as the person's posture changes.

[0011] In some embodiments of this application, the dynamic orientation bounding boxes of each component of the tunneling machine are constructed based on the kinematic model of the tunneling machine, including: The real-time kinematic parameters of each component of the tunneling machine are obtained; among which, the real-time kinematic parameters include real-time joint angles and real-time speeds. Based on the kinematic model and real-time kinematic parameters of the tunneling machine, determine the position and attitude of each component of the tunneling machine at the current moment; Based on position and orientation, a directional bounding box is generated that tightly wraps around each component and rotates in real time with the movement of each component, serving as the dynamic directional bounding box for each component of the tunneling machine.

[0012] In some embodiments of this application, a spatiotemporal prediction sweep body bounding box is constructed based on the three-dimensional spatial state of personnel, the dynamic orientation bounding box of personnel, the dynamic orientation bounding boxes of various components of the tunneling machine, and the real-time kinematic parameters of the tunneling machine, including: The movement trajectory of personnel within a preset time window is determined based on their three-dimensional spatial state. Based on the real-time kinematic parameters and kinematic model of the tunneling machine, the motion trajectory of each component of the tunneling machine within a future preset time window is determined; Based on the bounding boxes of the personnel's dynamic orientation, the bounding boxes of the various components of the tunneling machine, the movement trajectories of the personnel within a future preset time window, and the movement trajectories of the various components of the tunneling machine within a future preset time window, a spatiotemporal prediction sweep body bounding box is generated.

[0013] In some embodiments of this application, collision detection is performed on the spatiotemporal prediction sweep bounding box to obtain collision detection results, including: Intersection tests of the separation axes were performed on the bounding box of the spatiotemporal prediction sweep volume, and the test results were obtained. If the test results show that the projection intervals of the spatiotemporally predicted swept bounding box on any separation axis do not overlap, the collision detection result is determined to be without collision risk; otherwise, the minimum separation distance is determined based on the separation axis where the projection intervals overlap, and the collision detection result is determined based on the minimum separation distance.

[0014] In some embodiments of this application, the preset warning threshold includes a first threshold and a second threshold; the first threshold is less than the second threshold; Based on collision detection results and preset warning thresholds, tiered warning information is generated, including: If the minimum separation distance is less than or equal to the first threshold, the graded early warning information is determined to be a dangerous intrusion early warning information; wherein, the dangerous intrusion early warning information is used to trigger equipment shutdown; If the minimum separation distance is greater than the first threshold and less than or equal to the second threshold, the graded early warning information is determined to be an approach warning information; wherein, the approach warning information is used to trigger audible and visual warnings and terminal synchronous warnings.

[0015] Secondly, embodiments of this application provide an underground personnel collision early warning system based on environmental degradation adaptive fusion, including an edge computing device and a host computer; An edge computing device is used to acquire image data and point cloud data from underground coal mines at the same time. Environmental degradation assessment is performed on the image data and point cloud data to obtain a first degradation factor and a second degradation factor. If both the first and second degradation factors are less than their respective safety thresholds, a short-time trajectory is extrapolated based on the velocity vector of the personnel underground at the previous moment to obtain the personnel's three-dimensional spatial state. Otherwise, an objective function is determined based on the first degradation factor, the second degradation factor, the image data, and the point cloud data, and the objective function is solved to obtain the personnel's three-dimensional spatial state. A dynamic orientation bounding box for the personnel is constructed based on their three-dimensional spatial state. Dynamic orientation bounding boxes for each component of the tunneling machine are constructed based on the kinematic model of the tunneling machine. A spatiotemporal predicted sweep bounding box is constructed based on the personnel's three-dimensional spatial state, the personnel's dynamic orientation bounding box, the dynamic orientation bounding boxes for each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine. Collision detection is performed on the spatiotemporal predicted sweep bounding box to obtain the collision detection results. The host computer is used to generate graded early warning information based on the collision detection results and preset early warning thresholds when the collision detection results indicate that there is a collision risk. Attached Figure Description

[0016] To more intuitively illustrate the prior art and this application, several exemplary figures are provided below. It should be understood that the specific shapes and structures shown in the figures should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary figures, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0017] Figure 1 A schematic diagram of the implementation process of the downhole personnel collision early warning method based on environmental degradation adaptive fusion provided in this application embodiment. Figure 1 ; Figure 2 A schematic diagram of the system architecture for downhole personnel positioning and collision warning based on environmental degradation adaptive fusion provided in an embodiment of this application; Figure 3 A schematic diagram of the implementation process of the downhole personnel collision early warning method based on environmental degradation adaptive fusion provided in this application embodiment. Figure 2 ; Figure 4 This is a schematic diagram of the main interface of the monitoring platform provided in this embodiment of the application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Any combination of different embodiments is possible.

[0019] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0020] This application provides a method for early warning of personnel collisions in underground mines based on adaptive fusion of environmental degradation, such as... Figure 1 As shown, the downhole personnel collision early warning method based on environmental degradation adaptive fusion may include the following steps: Step 101: Obtain image data and point cloud data of the same moment in the coal mine.

[0021] In the embodiments of this application, the edge computing device can first acquire image data and point cloud data of the coal mine at the same time.

[0022] In embodiments of this application, the image data may be a two-dimensional RGB image captured by a vision camera, used to identify the human area and extract skeletal key points.

[0023] In the embodiments of this application, the point cloud data can be a set of three-dimensional spatial points collected by lidar, each point containing three-dimensional coordinates (x, y, z) and reflection intensity information, used to obtain distance information of people and the environment.

[0024] Step 102: Perform environmental degradation assessment on the image data and point cloud data to obtain the first degradation factor and the second degradation factor.

[0025] In the embodiments of this application, the edge computing device can perform environmental degradation assessment on the image data and point cloud data at the same time in the coal mine to obtain a first degradation factor and a second degradation factor after acquiring the image data and point cloud data at the same time.

[0026] In the embodiments of this application, environmental degradation assessment refers to the process of quantitatively assessing the interference factors affecting the sensor sensing quality in the downhole environment. The main degradation factors in the downhole environment include image blurring and increased point cloud noise caused by increased dust concentration, and image information loss caused by low light.

[0027] In some embodiments of this application, when the edge computing device performs environmental degradation assessment on image data and point cloud data to obtain a first degradation factor and a second degradation factor, it can determine the information entropy of the image data; wherein, the information entropy is used to quantify the information richness of the image data; the first degradation factor is determined based on the information entropy; wherein, the first degradation factor is positively correlated with the information entropy; the proportion of discrete point cloud noise in the local space of the person in the point cloud data is determined; the second degradation factor is determined based on the proportion of discrete point cloud noise; wherein, the second degradation factor is negatively correlated with the proportion of discrete point cloud noise.

[0028] In the embodiments of this application, information entropy refers to an indicator used to quantify the information richness of image data. When dust concentration increases, leading to decreased image contrast and fogging, information entropy decreases significantly. The lower the information entropy, the more severe the loss of image information and the worse the perception quality of the visual sensor.

[0029] In the embodiments of this application, the discrete point cloud noise ratio is used as an indicator to quantify the proportion of invalid noise points in the lidar point cloud caused by dust scattering. In high-dust environments, the interaction between the laser beam and dust particles generates a large number of suspended noise points. The higher the noise ratio, the worse the sensing quality of the lidar sensor.

[0030] In the embodiments of this application, the first degradation factor is a quantified value of the degree of visual sensor degradation determined based on the information entropy of image data, denoted as... It is positively correlated with information entropy. The lower the information entropy, the smaller the first degradation factor, indicating a lower confidence level of the visual sensor.

[0031] In the embodiments of this application, the second degradation factor is a quantified value of the radar sensor degradation degree determined based on the proportion of discrete point cloud noise in the local space of the personnel in the point cloud data, denoted as... It is negatively correlated with the noise ratio. The higher the noise ratio, the smaller the second degradation factor, indicating a lower confidence level of the radar sensor.

[0032] For example, the first degradation factor can be determined by the following formula: (1); in, The visual attenuation adjustment coefficient is calibrated based on the actual working conditions of the mine. Information entropy; It is an abbreviation for the natural exponential function.

[0033] For example, information entropy can be calculated using the following formula: (2); in, Let be the probability of a pixel with gray value i appearing in the image; when the dust concentration increases, causing the image contrast to decrease and fogging to occur, the gray distribution of the image tends to be uniform and the information entropy decreases significantly.

[0034] For example, the second degradation factor can be determined by the following formula: (3); in, This is the dust scattering penalty coefficient; This represents the noise ratio of discrete point clouds.

[0035] For example, the calculation method for the noise ratio of discrete point clouds can be expressed as the following formula: (4); in, This represents the number of discrete point cloud noises within a localized area of ​​the personnel. This represents the total number of point clouds within this local space.

[0036] Step 103: If both the first degradation factor and the second degradation factor are less than their respective safety thresholds, perform short-time trajectory extrapolation based on the velocity vector of the personnel underground in the coal mine at the previous moment to obtain the three-dimensional spatial state of the personnel; otherwise, determine the objective function based on the first degradation factor, the second degradation factor, image data, and point cloud data, solve the objective function, and obtain the three-dimensional spatial state of the personnel.

[0037] In the embodiments of this application, after the edge computing device performs environmental degradation assessment on image data and point cloud data to obtain a first degradation factor and a second degradation factor, it can perform short-time trajectory extrapolation based on the velocity vector of the personnel in the coal mine at the previous moment to obtain the three-dimensional spatial state of the personnel if both the first degradation factor and the second degradation factor are less than their respective safety thresholds; otherwise, it determines an objective function based on the first degradation factor, the second degradation factor, the image data, and the point cloud data, solves the objective function, and obtains the three-dimensional spatial state of the personnel.

[0038] In the embodiments of this application, when the degradation factors of both the vision and radar sensors are below their respective safety thresholds, it indicates that both sensors have severely failed and cannot provide reliable perception data. At this time, the system automatically switches to a pure kinematics extrapolation mode, using the personnel velocity vector from the previous moment to perform short-term trajectory extrapolation, maintaining the continuity of positioning.

[0039] In some embodiments of this application, the velocity vector is initialized to zero upon initial system startup. The velocity vector is zero when the person is stationary, and short-term trajectory simulation is performed to maintain position stability.

[0040] In the embodiments of this application, the objective function is a multi-source nonlinear least squares optimization objective function with dynamic weights, which is used to fuse visual observation and radar observation data to solve for the optimal three-dimensional spatial state of the personnel.

[0041] In some embodiments of this application, when determining the objective function based on a first degradation factor, a second degradation factor, image data, and point cloud data, the edge computing device may determine the visual observation coordinates of a person based on the image data; determine the radar observation coordinates of a person based on the point cloud data; determine a first cost term based on the first degradation factor, the visual observation coordinates of a person, and the visual measurement noise covariance matrix; determine a second cost term based on the second degradation factor, the radar observation coordinates of a person, and the radar measurement noise covariance matrix; and determine the objective function based on the first cost term and the second cost term.

[0042] For example, the objective function can be expressed as the following formula: (5); in, This represents the actual state vector of the person to be solved (i.e., the three-dimensional spatial state), including the person's three-dimensional spatial coordinates and posture angles; This represents the prior value of the current state, predicted based on the previous state of the person and the motion model. Indicates the first degradation factor; Indicates the second degradation factor; Indicates the coordinates of the person's visual observation; Indicates the radar observation coordinates of personnel; The observation model function of the vision sensor represents the state vector. Mapped to the observation space of the visual sensor; The observation model function of the radar sensor represents the state vector. Mapped to the radar sensor observation space; Represents the visual measurement noise covariance matrix; Let represent the radar measurement noise covariance matrix. The first cost term is... The second cost term is .

[0043] In some embodiments of this application, when the edge computing device solves the objective function to obtain the three-dimensional spatial state of a person, it can solve the minimum value of the objective function through a numerical optimization framework, and determine the person's state vector corresponding to the minimum value as the three-dimensional spatial state of the person; wherein, the numerical optimization framework represents a mathematical framework for solving the minimum value of the objective function in real time in an edge computing environment.

[0044] In the embodiments of this application, the numerical optimization framework is a mathematical framework for solving the minimum value of the objective function in real time in an edge computing environment; this application adopts the CasaADi piecewise optimization mathematical framework, which outputs the optimal three-dimensional spatial state of the personnel in milliseconds through iterative calculation.

[0045] Furthermore, the minimum value of the objective function can be solved by iterative numerical optimization method, and the personnel state vector corresponding to the minimum value can be determined as the three-dimensional spatial state of the personnel. This state vector includes the personnel's three-dimensional spatial coordinates and posture angles.

[0046] In the embodiments of this application, posture angles include limb posture angles / joint angles.

[0047] Step 104: Construct a dynamic orientation bounding box for personnel based on their three-dimensional spatial state.

[0048] In the embodiments of this application, after obtaining the three-dimensional spatial state of a person, the edge computing device can construct a dynamic orientation bounding box of the person based on the three-dimensional spatial state of the person.

[0049] In the embodiments of this application, the dynamic orientation bounding box adopts an Oriented Bounding Box (OBB), which is a tight bounding box that can rotate and translate in real time with the target's movement. Unlike the traditional axis-aligned bounding box (AABB), the OBB can dynamically adjust its orientation according to the target's attitude changes, thereby fitting the target boundary more closely.

[0050] In the embodiments of this application, when the edge computing device constructs a dynamic orientation bounding box for a person based on the three-dimensional spatial state of the person, it can determine the position coordinates and limb posture angles of the person based on the three-dimensional spatial state of the person; and generate a dynamic orientation bounding box for the person based on the position coordinates and limb posture angles; wherein, the dynamic orientation bounding box for the person is used to represent a tightly bounding spatial boundary that is dynamically updated according to the position and posture of the person and rotates with the change of the person's posture.

[0051] In the embodiments of this application, the person dynamic orientation bounding box is dynamically updated according to the person's position and posture, and rotates in real time as the person's posture changes. The person dynamic orientation bounding box can be defined by the center point position, the lengths of the three semi-axis, and the rotation matrix. The rotation matrix of the OBB is determined by the limb posture angles to ensure that the orientation of the OBB is consistent with the human posture; the center point position of the OBB is determined by the position coordinates; and the semi-axis lengths are determined by the human body dimensions.

[0052] Step 105: Construct the dynamic orientation bounding box of each component of the tunneling machine based on the kinematic model of the tunneling machine.

[0053] In the embodiments of this application, the edge computing device can construct dynamic orientation bounding boxes for each component of the tunneling machine based on the kinematic model of the tunneling machine.

[0054] In the embodiments of this application, when the edge computing device constructs the dynamic orientation bounding box of each component of the tunneling machine based on the kinematic model of the tunneling machine, it can obtain the real-time kinematic parameters of each component of the tunneling machine; wherein, the real-time kinematic parameters include real-time joint angles and real-time speeds; based on the kinematic model of the tunneling machine and the real-time kinematic parameters, the position and attitude of each component of the tunneling machine at the current moment are determined; and an orientation bounding box that tightly wraps around each component and rotates in real time with the movement of each component is generated based on the position and attitude, which serves as the dynamic orientation bounding box of each component of the tunneling machine.

[0055] For example, for the cutting arm of a tunneling machine, the spatial position and orientation of each segment of the cutting arm can be calculated based on the joint angle of the cutting arm to generate the corresponding OBB; for other components such as the shovel, the corresponding OBB is also generated based on their kinematic parameters.

[0056] Step 106: Construct a spatiotemporal prediction sweep body bounding box based on the three-dimensional spatial state of personnel, the dynamic orientation bounding box of personnel, the dynamic orientation bounding boxes of each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine.

[0057] In the embodiments of this application, after obtaining the dynamic orientation bounding box of personnel and the dynamic orientation bounding boxes of each component of the tunneling machine, the edge computing device can construct a spatiotemporal prediction sweep body bounding box based on the three-dimensional spatial state of personnel, the dynamic orientation bounding box of personnel, the dynamic orientation bounding boxes of each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine.

[0058] In the embodiments of this application, the spatiotemporal predicted swept body bounding box is a swept body bounding box generated based on the dynamic direction bounding box, combining the future movement trajectory of personnel and the future movement trajectory of various components of the tunneling machine, and covering the movement range within a preset future time window. This bounding box is used for forward-looking collision detection and can predict the risk of human-machine collision within the future time window.

[0059] In some embodiments of this application, when the edge computing device constructs a spatiotemporal prediction sweep bounding box based on the three-dimensional spatial state of the personnel, the dynamic orientation bounding box of the personnel, the dynamic orientation bounding boxes of each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine, it can determine the movement trajectory of the personnel within a future preset time window based on the three-dimensional spatial state of the personnel; determine the movement trajectory of each component of the tunneling machine within a future preset time window based on the real-time kinematic parameters and kinematic model of the tunneling machine; and generate a spatiotemporal prediction sweep bounding box based on the dynamic orientation bounding box of the personnel, the dynamic orientation bounding box of each component of the tunneling machine, the movement trajectory of the personnel within a future preset time window, and the movement trajectory of each component of the tunneling machine within a future preset time window.

[0060] In some embodiments of this application, based on the construction of the static OBB at the current moment, a short-term extrapolation of the future spatiotemporal trajectory of the human and machine is further performed by combining particle swarm optimization algorithm and Dubins curves. Specifically, the movement trajectory of the personnel within a preset future time window is determined based on the three-dimensional spatial state of the personnel, and the movement trajectory of each component of the tunneling machine within the preset future time window is determined based on the real-time kinematic parameters and kinematic model of the tunneling machine. Then, a spatiotemporal prediction sweep body bounding box is generated based on the dynamic orientation bounding box of the personnel, the dynamic orientation bounding boxes of each component of the tunneling machine, and the predicted movement trajectories of both parties. This sweep body covers the movement range of the personnel and the tunneling machine within the preset future time window and is used to proactively predict the risk of human-machine collision within the future time window.

[0061] In some embodiments of this application, the construction of the spatiotemporal predicted sweep volume bounding box is a core step in collision detection. When determining the movement trajectory of a person within a future preset time window based on their three-dimensional spatial state, the position sequence of the person within the future time window can be predicted based on their current position coordinates, velocity vector, and movement direction, combined with the motion constraints of the underground environment. When determining the movement trajectory of each component of the tunneling machine within a future preset time window based on its real-time kinematic parameters and kinematic model, the position and attitude sequence of each component within the future time window can be predicted based on its real-time joint angles, velocities, and kinematic equations. Then, based on the volume generated by the sweeping of the person's dynamic orientation bounding box along their movement trajectory and the volume generated by the sweeping of the dynamic orientation bounding boxes of each component of the tunneling machine along their respective movement trajectories, a spatiotemporal predicted sweep volume bounding box is generated. This sweep volume covers all spatial positions that the person and the tunneling machine may occupy within the future preset time window.

[0062] Step 107: Perform collision detection on the spatiotemporal prediction sweep bounding box and obtain the collision detection results.

[0063] In the embodiments of this application, after the edge computing device constructs a spatiotemporal predicted sweep body bounding box based on the three-dimensional spatial state of the personnel, the dynamic orientation bounding box of the personnel, the dynamic orientation bounding box of each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine, it can perform collision detection on the spatiotemporal predicted sweep body bounding box and obtain the collision detection result.

[0064] In some embodiments of this application, the Separating Axis Theorem (SAT) can be used to perform high-frequency real-time intersection tests on the spatiotemporal prediction sweep bounding box to determine whether there is a collision risk.

[0065] In some embodiments of this application, when the edge computing device performs collision detection on the bounding box of the spatiotemporal prediction sweep volume and obtains the collision detection result, it can perform an intersection test of the separation axis on the bounding box of the spatiotemporal prediction sweep volume and obtain the test result. If the test result shows that the projection intervals of the bounding box of the spatiotemporal prediction sweep volume do not overlap on any separation axis, the collision detection result is determined to be without collision risk; otherwise, the minimum separation distance is determined according to the separation axis where the projection intervals overlap, and the collision detection result is determined according to the minimum separation distance.

[0066] In embodiments of this application, the minimum separation distance can be used to characterize the severity of collisions between humans and machines. The minimum separation distance can be the magnitude of the minimum separation vector between the overlapping separation axes of the projection intervals.

[0067] In some embodiments of this application, 15 potential separation axes of the spatiotemporal predicted sweep bounding box (for collision detection between 3D OBBs, the potential separation axes include the cross products of the three face normal vectors and the three edge direction vectors of each bounding box, for a total of 15 axes) are traversed to perform projection interval overlap tests. If the projection intervals on any separation axis do not overlap, the collision detection result is determined to be without collision risk.

[0068] Step 108: If the collision detection result indicates a collision risk, generate tiered warning information based on the collision detection result and the preset warning threshold.

[0069] In the embodiments of this application, after the edge computing device performs collision detection on the spatiotemporal prediction sweep bounding box and obtains the collision detection result, it can send the collision detection result, multi-dimensional spatial state related data of personnel and tunneling machine to the host computer. If the collision detection result indicates that there is a collision risk, the host computer can generate graded early warning information based on the collision detection result and the preset early warning threshold.

[0070] In the embodiments of this application, a warning is triggered only when a collision risk is detected. A tiered warning system is implemented based on the relationship between the minimum separation distance in the collision detection results and a preset warning threshold.

[0071] In some embodiments of this application, the preset warning threshold includes a first threshold and a second threshold; the first threshold is less than the second threshold; when the edge computing device generates graded warning information based on the collision detection result and the preset warning threshold, it can determine the graded warning information as dangerous intrusion warning information if the minimum separation distance is less than or equal to the first threshold; wherein, the dangerous intrusion warning information is used to trigger device shutdown; if the minimum separation distance is greater than the first threshold and less than or equal to the second threshold, the graded warning information is determined as proximity warning information; wherein, the proximity warning information is used to trigger audible and visual warnings and terminal synchronous warnings.

[0072] In some embodiments of this application, the danger intrusion warning information may be sent to the underlying control system of the tunneling machine as a flexible shutdown or forced emergency stop command, while the corresponding personnel model and pop-up window are turned red in the three-dimensional twin scene.

[0073] In some embodiments of this application, the proximity warning information may be pushed synchronously to the management personnel through the background terminal, and the model in the 3D twin scene may be turned yellow.

[0074] This application provides a method for underground personnel collision early warning based on environmental degradation adaptive fusion, breaking through the limitations of simple splicing with fixed weights in existing technologies and introducing an environmental degradation adaptive dynamic weighting mechanism. Specifically, the system calculates the information entropy of the visual image and the discrete noise ratio of the local space of personnel in the lidar point cloud in real time, dynamically calculates the visual degradation factor and the radar degradation factor, and constructs a nonlinear least squares optimization objective function with dynamic weight coefficients, which is then solved in real time through a numerical optimization framework in an edge computing environment. This mechanism enables the system to automatically reduce the weights of degraded sensors under extreme conditions such as sudden fog explosions and dust surges, preventing high-noise data from contaminating the fusion results, and fundamentally solving the underlying algorithm pain point of multi-source sensing data experiencing positioning drift or complete failure due to environmental interference. It abandons the traditional AABB bounding box with a fixed shape and unable to rotate, and constructs a dynamic directional bounding box that tightly wraps the equipment based on the kinematic equations and attitude parameters of complex moving parts such as the tunneling machine cutting arm. Building upon this foundation, the particle swarm optimization algorithm and Dubins curves are further combined to perform short-term extrapolation of the future spatiotemporal trajectory of the human-machine interface, generating a spatiotemporal predicted sweep bounding box. High-frequency intersection tests are then conducted using the separating axis theorem to calculate the minimum separation distance between the human and machine. This mechanism upgrades collision detection from a simple static comparison in geometric space to a deep coupling of mechanical kinematics and spatiotemporal prediction, eliminating a significant amount of redundant space inherent in traditional electronic fences. It fundamentally overcomes the technical challenges of high false alarm and high false negative rates in underground human-machine collision early warning systems, achieving millimeter-level high-precision three-dimensional spatial human-machine collision risk prediction.

[0075] Based on the above embodiments, in another embodiment of this application, a method and system for underground personnel positioning and collision warning based on environmental degradation adaptive fusion is provided. This application introduces an environmental degradation assessment model at the algorithm's underlying layer to dynamically quantify the interference caused by changes in underground dust and lighting on different heterogeneous sensors in real time. Based on this, the confidence weights of each sensing source in the fusion calculation are dynamically adjusted, constructing a nonlinear optimization objective function with dynamic weights. This ensures that even under extreme and harsh conditions such as sudden fog explosions in mines, the system can still accurately calculate the high-precision three-dimensional spatial coordinates and three-dimensional limb postures of the workers, fundamentally solving the underlying algorithmic pain point of positioning drift or failure due to environmental interference in multi-source sensing data.

[0076] Building upon this foundation, this application binds the calculated high-precision personnel spatial state to the tunneling machine components. Utilizing dynamic directional bounding boxes or virtual collision mesh technology, it constructs a dynamic three-dimensional safety boundary that rotates in real-time with the movement of mechanical components, tightly enclosing the human body. This eliminates redundant gaps generated by traditional spatial bounding boxes, thereby significantly reducing the system's false alarm and false negative rates while ensuring comprehensive coverage of hazardous areas. Finally, this application aims to use the calculated multi-source high-precision spatial data to drive a digital twin platform in real-time, synchronously mapping the movement of underground physical entities to a three-dimensional virtual scene with millimeter-level precision. Through intuitive three-dimensional visualization judgment methods and a precise hierarchical linkage early warning mechanism, it achieves rapid and intuitive human-machine collision avoidance response, comprehensively improving the inherent safety level of human-machine collaborative operations in coal mine tunneling faces.

[0077] like Figure 2 As shown, this application provides an underground personnel positioning and collision warning system based on environmental degradation adaptive fusion. Its hardware architecture is mainly divided into a perception layer and a computing layer. The perception layer achieves full three-dimensional spatial coverage of the core working area around the equipment by deploying intrinsically safe mining lidar and industrial-grade high-definition depth cameras around the tunneling machine and at key locations. The computing layer uses an edge computing device or industrial control computer equipped with a high-performance GPU to run multi-source heterogeneous sensor adaptive fusion positioning and collision detection algorithms.

[0078] like Figure 3 As shown, the core implementation steps of this application are as follows: The environmental degradation assessment and dynamic weight calculation system, after acquiring synchronous sensing data from heterogeneous sensors, does not employ traditional fixed-weight concatenation but instead quantifies environmental degradation interference in real time. Visual processing branch: After the visual sensor acquires image frames, the system uses a deep learning network to identify personnel areas and extract skeletal key points. To address underground dust and low-light interference, the system calculates the information entropy of the current frame image I in real time. When dust concentration increases, leading to decreased image contrast and fogging, information entropy decreases significantly. Therefore, the visual confidence weight (first degradation factor) is defined as shown in the aforementioned formula (1).

[0079] Radar processing branch: Point cloud data is simultaneously acquired by lidar. For suspended noise generated in high-dust environments, the system calculates the proportion of discrete point cloud noise within a person's local space. To assess the degree of interference. Therefore, the radar confidence weight (second degradation factor) is defined as shown in the aforementioned formula (3).

[0080] Adaptive weighted fusion and high-precision solution: Let the true state vector of the personnel downhole be... , For personnel visual observation coordinates, To this end, a multi-source nonlinear least squares optimization objective function with dynamic weights is constructed for the system. As shown in the aforementioned formula (5).

[0081] To ensure the real-time performance of the system on edge computing devices, the system's underlying fusion processing unit uses a piecewise optimization mathematical framework based on CasADi to solve for the minimum value of the above objective function, thereby accurately outputting the optimal high-precision three-dimensional coordinates and attitude angles of the personnel, which are not affected by the degradation of a single sensor.

[0082] At the same time, the system has a degradation operation mechanism under extreme conditions: when it is determined that... and When all values ​​are below the set safety threshold, the system automatically switches to pure kinematics extrapolation and prediction mode, using the velocity vector from the previous moment to perform short-term trajectory extrapolation.

[0083] The system inputs the calculated high-precision personnel status data into the collision detection module.

[0084] Unlike traditional fixed virtual collision meshes or AABB bounding boxes, this embodiment constructs a dynamically rotating OBB. Based on the tunneling machine's kinematic model, the system updates the OBBs of its components such as the cutting arm and shovel in real time. Building upon this, the system combines the kinematic constraints of the equipment chassis and actuators, utilizing particle swarm optimization and Dubins curve smoothing to predict the trajectory trends of the tunneling machine and personnel within a very short future time window, thus constructing a spatiotemporally predictive dynamic swept body bounding box.

[0085] Subsequently, using the separation axis theorem, the system traverses 15 potential separation axes of the device's OBB and the human's dynamic OBB, performing high-frequency real-time intersection tests. If the projection intervals overlap, the magnitude of the minimum translation vector (MTV) is precisely calculated as the minimum human-machine separation distance. .

[0086] like Figure 4 The image shows the main interface of the digital twin monitoring platform. The system transmits all multidimensional data to the host computer with low latency via the Message Queuing Telemetry Transport (MQTT) protocol.

[0087] The host computer recreates a 1:1 three-dimensional scene model of the underground tunneling face in a coal mine based on CAD drawings, achieving millimeter-level synchronous mapping of personnel positions, limb postures, and mechanical component operations.

[0088] Based on the minimum separation distance calculated above The early warning module executes an automated, closed-loop linkage mechanism: when < When a dangerous intrusion is detected, the corresponding personnel model and pop-up window in the 3D scene turn red, and a flexible shutdown or forced emergency stop command is automatically sent to the tunneling machine's underlying control system.

[0089] when When the time is right, it is determined to be a warning state, the model turns yellow and triggers audible and visual warnings on site as well as multi-level synchronous warnings on the back-end terminal.

[0090] This application presents an underground personnel positioning and collision warning system based on environmental degradation adaptive fusion. The core working principle is to dynamically adjust the algorithm fusion weights by real-time assessment of the impact of the harsh mine environment on the sensing quality of heterogeneous sensors, and combine this with spatiotemporal kinematic prediction to achieve precise obstacle avoidance. The specific working process is as follows: After the system is activated for dynamic perception and online monitoring of environmental degradation, the sensor perception module simultaneously activates visual and laser detection. In the initial stages of data processing, the system does not perform weighted fusion of the raw data. Instead, the preprocessing unit separately assesses the environmental degradation health of the two signals. The visual unit calculates the information entropy of the current image frame. When a large area of ​​dust in the tunnel causes blurring of the image, the information entropy decreases, and the system automatically triggers a visual confidence downgrading mechanism. The laser detection unit statistically analyzes the point cloud scattering density within the local scanning range in real time. If abnormal noise occurs due to dust scattering, the system automatically calculates the radar confidence loss. The purpose of this stage is to perceive the current level of environmental interference in real time, providing a quantitative confidence indicator for subsequent fusion calculations.

[0091] Based on the degradation factor obtained in the first stage, the core fusion solution and nonlinear optimization stage model the fusion process of the sensed data as a nonlinear dynamic optimization problem. The system's underlying fusion processing unit, using the CasADi numerical computation framework, performs real-time extreme value optimization in an edge computing environment, based on a constructed objective function with dynamic weights. Through iterative calculation, the system calculates a set of state vectors that best represent the current real spatial location of the personnel every millisecond. When environmental interference exceeds safety limits, the system triggers fail-safe logic, seamlessly switching from high-precision sensor fusion mode to pure kinematic inference mode, ensuring the continuity and logical stability of the personnel's position coordinates even in sudden dense dust environments.

[0092] After obtaining high-precision coordinates during the spatiotemporal dynamic collision detection phase, the system maps them to the digital twin 3D environment. The core of its working principle lies in upgrading static spatial early warning to spatiotemporal dynamic prediction. The system extracts kinematic parameters such as joint angles and velocities of the tunneling machine's robotic arm and inputs them, along with personnel coordinates, into a dynamic bounding box calculator. The bounding box rotates and translates in real time along with the tunneling machine's movements, ensuring that the bounding box always remains close to the robotic arm's boundary. High-frequency collision detection is performed using the separation axis theorem. When the calculated minimum separation distance d between the human and machine approaches or triggers the warning boundary, the system immediately extracts the dynamic collision vector and determines the warning level.

[0093] In the coordinated early warning and visual response phase, after receiving the early warning command from the underlying calculation, the host computer management platform performs intuitive visual mapping based on the hierarchical early warning mechanism. The system not only synchronously updates the motion posture of the tunneling machine and personnel in the 3D twin interface, but also synchronizes the calculated early warning level signal to the on-site execution terminal in real time. Through the closed-loop logic of the control system, automatic response from audible and visual warnings to flexible deceleration and even forced shutdown is achieved, completing a full-process collision avoidance closed loop from environmental perception, feature fusion, high-precision positioning to intelligent decision-making and execution. The principle of this solution lies in reconstructing the originally discrete monitoring system based on simple logical judgment into an intelligent collaborative system with environmental fault tolerance, mechanical kinematic constraints, and real-time numerical optimization capabilities, thereby completely solving the problems of high false alarm rate and insufficient accuracy of traditional technologies under complex working conditions.

[0094] In summary, this application addresses the core shortcomings of existing intelligent human-machine collision avoidance technology, such as the drift and failure of sensor fusion localization caused by the harsh environment of underground coal mines, and the spatial redundancy and high false alarm rate of traditional collision detection models. The core idea is to not only achieve the physical superposition of multi-source sensing data, but also to introduce adaptive environmental health perception and spatiotemporal dynamic bounding box modeling combined with mechanical kinematics. Based on the environmental degradation adaptive dynamic weighted fusion localization algorithm, this application constructs an environmental degradation assessment model at the underlying layer of the multi-source heterogeneous sensor fusion algorithm. The system does not rely on fixed fusion weights, but instead calculates the information entropy of the visual image in real time. and the discrete noise ratio of local laser point clouds , dynamically solve for two degradation factors, so as to construct a nonlinear least squares optimization objective function with dynamic weight coefficients, and use a numerical optimization framework to solve for the minimum value to output a high-precision three-dimensional state. This breaks through the limitation of simple splicing with fixed weights of multiple sensors in the prior art, endowing the system with underlying anti-interference and adaptive fault-tolerant capabilities for harsh working conditions. In extreme cases such as sudden dense fog, it can automatically reduce the weight of degraded sensors to prevent data pollution, and ensure the continuity and high precision of personnel positioning. Dynamic OBB construction and intersection testing technology based on equipment kinematic constraints for spatio-temporal prediction: it abandons the traditional AABB bounding box with a fixed shape that cannot rotate following movement, or simple distance threshold judgment. Based on the kinematic equations and attitude parameters of complex moving parts such as the cutting arm of a roadheader, the system constructs a dynamic oriented bounding box that tightly encloses the equipment, and combines the particle swarm optimization algorithm to perform short-term deduction of the future spatio-temporal trajectories of personnel and the machine, so as to generate a swept volume bounding box for spatio-temporal prediction. The separating axis theorem is used for high-frequency intersection testing to calculate the minimum separation distance between personnel and the machine. This upgrades collision detection from a simple static comparison in geometric space to the coupling of mechanical kinematics and spatio-temporal prediction. It solves the pain point that large heavy-duty rotating machinery and irregular human postures cannot be accurately fitted in unstructured spaces, eliminates a large amount of redundant space, and fundamentally overcomes the technical problems of high false alarm rate and high missed alarm rate in underground human-machine collision warning. This is a full-chain digital closed-loop system for perception, calculation, and digital twin warning, as well as the software and hardware system architecture of the above adaptive fusion algorithm and dynamic OBB collision grid calculation. It covers data collection by the front-end explosion-proof LiDAR and depth camera, high-precision calculation by the edge computing unit, low-latency transmission based on the MQTT protocol, and the millimeter-level motion mapping and multi-level linkage warning response mechanism from underground physical entities to the three-dimensional virtual scene realized through the digital twin platform.

[0095] Based on the above embodiments, another embodiment of the present application provides an underground personnel collision warning system based on environment degradation adaptive fusion, which includes an edge computing device and an upper computer; An edge computing device is used to acquire image data and point cloud data of personnel underground at the same time. Environmental degradation assessment is performed on the image data and point cloud data to obtain a first degradation factor and a second degradation factor. If both the first degradation factor and the second degradation factor are less than their respective safety thresholds, a short-time trajectory is extrapolated based on the velocity vector of the personnel underground at the previous moment to obtain the personnel's three-dimensional spatial state. Otherwise, an objective function is determined based on the first degradation factor, the second degradation factor, the image data, and the point cloud data, and the objective function is solved to obtain the personnel's three-dimensional spatial state. A dynamic orientation bounding box for the personnel is constructed based on their three-dimensional spatial state. Dynamic orientation bounding boxes for each component of the tunneling machine are constructed based on the kinematic model of the tunneling machine. A spatiotemporal predicted swept body bounding box is constructed based on the personnel's three-dimensional spatial state, the personnel's dynamic orientation bounding box, the dynamic orientation bounding boxes for each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine. Collision detection is performed on the spatiotemporal predicted swept body bounding box to obtain the collision detection results.

[0096] The host computer is used to generate graded early warning information based on the collision detection results and preset early warning thresholds when the collision detection results indicate that there is a collision risk.

[0097] The edge computing device is also used to determine the information entropy of image data; wherein the information entropy is used to quantify the information richness of image data; determine a first degradation factor based on the information entropy; wherein the first degradation factor is positively correlated with the information entropy; determine the proportion of discrete point cloud noise in the local space of a person in point cloud data; determine a second degradation factor based on the proportion of discrete point cloud noise; wherein the second degradation factor is negatively correlated with the proportion of discrete point cloud noise.

[0098] The edge computing device is also used to determine the visual observation coordinates of personnel based on image data; determine the radar observation coordinates of personnel based on point cloud data; determine a first cost term based on a first degradation factor, the visual observation coordinates of personnel, and the visual measurement noise covariance matrix; determine a second cost term based on a second degradation factor, the radar observation coordinates of personnel, and the radar measurement noise covariance matrix; and determine an objective function based on the first and second cost terms.

[0099] The edge computing device is also used to solve for the minimum value of the objective function through a numerical optimization framework, and to determine the personnel state vector corresponding to the minimum value as the three-dimensional spatial state of the personnel; wherein, the numerical optimization framework represents the mathematical framework used to solve for the minimum value of the objective function in real time in the edge computing environment.

[0100] The edge computing device is also used to determine the position coordinates and limb posture angles of the person based on the three-dimensional spatial state of the person; and to generate a dynamic orientation bounding box of the person based on the position coordinates and limb posture angles; wherein, the dynamic orientation bounding box of the person is used to represent a tightly wrapped spatial boundary that is dynamically updated according to the position and posture of the person and rotates with the change of the person's posture.

[0101] The edge computing device is also used to acquire real-time kinematic parameters of each component of the tunneling machine; the real-time kinematic parameters include real-time joint angles and real-time speeds; based on the kinematic model of the tunneling machine and the real-time kinematic parameters, the position and attitude of each component of the tunneling machine at the current moment are determined; based on the position and attitude, a directional bounding box that tightly wraps around each component and rotates in real time with the movement of each component is generated as the dynamic directional bounding box of each component of the tunneling machine.

[0102] The edge computing device is also used to determine the movement trajectory of personnel within a future preset time window based on their three-dimensional spatial state; to determine the movement trajectory of each component of the tunneling machine within a future preset time window based on the real-time kinematic parameters and kinematic model of the tunneling machine; and to generate a spatiotemporal prediction sweep bounding box based on the dynamic orientation bounding box of personnel, the dynamic orientation bounding box of each component of the tunneling machine, the movement trajectory of personnel within a future preset time window, and the movement trajectory of each component of the tunneling machine within a future preset time window.

[0103] The edge computing device is also used to perform an intersection test of the separation axes of the spatiotemporal prediction sweep bounding box and obtain the test results. If the test results show that the projection intervals of the spatiotemporal prediction sweep bounding box on any separation axis do not overlap, the collision detection result is determined to be without collision risk. Otherwise, the minimum separation distance is determined based on the separation axis where the projection intervals overlap, and the collision detection result is determined based on the minimum separation distance.

[0104] The host computer is also used to determine the graded warning information as dangerous intrusion warning information when the minimum separation distance is less than or equal to the first threshold; wherein, the dangerous intrusion warning information is used to trigger equipment shutdown; and to determine the graded warning information as proximity warning information when the minimum separation distance is greater than the first threshold and less than or equal to the second threshold; wherein, the proximity warning information is used to trigger audible and visual warnings and terminal synchronous warnings.

[0105] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0106] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment.

[0107] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0108] This application is described with reference to schematic diagrams and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each step and / or block in the schematic diagrams and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more steps of the schematic diagrams and / or one or more blocks in the block diagrams.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. A method for early warning of personnel collisions in underground mines based on adaptive fusion of environmental degradation, characterized in that, The method includes: Acquire image data and point cloud data from underground coal mines at the same time; Environmental degradation assessment is performed on the image data and the point cloud data to obtain a first degradation factor and a second degradation factor; If both the first degradation factor and the second degradation factor are less than their respective safety thresholds, a short-time trajectory is extrapolated based on the velocity vector of the personnel in the coal mine at the previous moment to obtain the three-dimensional spatial state of the personnel; otherwise, an objective function is determined based on the first degradation factor, the second degradation factor, the image data, and the point cloud data, and the objective function is solved to obtain the three-dimensional spatial state of the personnel. Construct a dynamic orientation bounding box for the personnel based on their three-dimensional spatial state; The dynamic orientation bounding boxes of each component of the tunneling machine are constructed based on the kinematic model of the tunneling machine; Based on the three-dimensional spatial state of the personnel, the dynamic orientation bounding box of the personnel, the dynamic orientation bounding box of each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine, a spatiotemporal prediction sweep body bounding box is constructed. Collision detection is performed on the bounding box of the spatiotemporal prediction sweep to obtain the collision detection results; If the collision detection result indicates a collision risk, a graded warning message is generated based on the collision detection result and a preset warning threshold.

2. The downhole personnel collision early warning method based on environmental degradation adaptive fusion according to claim 1, characterized in that, The environmental degradation assessment of the image data and the point cloud data to obtain a first degradation factor and a second degradation factor includes: Determine the information entropy of the image data; wherein the information entropy is used to quantify the information richness of the image data; The first degradation factor is determined based on the information entropy; Determine the proportion of discrete point cloud noise in the local space of the personnel in the point cloud data; The second degradation factor is determined based on the noise ratio of the discrete point cloud.

3. The method according to claim 1, characterized in that, The step of determining the objective function based on the first degradation factor, the second degradation factor, the image data, and the point cloud data includes: Determine the visual observation coordinates of the personnel based on the image data; The personnel radar observation coordinates are determined based on the point cloud data; The first cost term is determined based on the first degradation factor, the personnel's visual observation coordinates, and the visual measurement noise covariance matrix. The second cost term is determined based on the second degradation factor, the personnel radar observation coordinates, and the radar measurement noise covariance matrix. The objective function is determined based on the first cost term and the second cost term.

4. The method according to claim 1, characterized in that, Solving the objective function to obtain the three-dimensional spatial state of the personnel includes: The objective function is minimized using a numerical optimization framework, and the personnel state vector corresponding to the minimum value is determined as the three-dimensional spatial state of the personnel. The numerical optimization framework represents a mathematical framework for solving the minimum value of the objective function in real time in an edge computing environment.

5. The method according to claim 1, characterized in that, The step of constructing a dynamic orientation bounding box for the personnel based on their three-dimensional spatial state includes: The position coordinates and limb posture angles of the person are determined based on the person's three-dimensional spatial state; The dynamic orientation bounding box of the person is generated based on the position coordinates and the limb posture angle; wherein, the dynamic orientation bounding box of the person is used to represent a tightly wrapped spatial boundary that is dynamically updated according to the position and posture of the person and rotates with the changes in the posture of the person.

6. The method according to claim 1, characterized in that, The construction of dynamic orientation bounding boxes for each component of the tunneling machine based on its kinematic model includes: The real-time kinematic parameters of each component of the tunneling machine are obtained; wherein, the real-time kinematic parameters include real-time joint angles and real-time speeds; Based on the kinematic model of the tunneling machine and the real-time kinematic parameters, determine the position and attitude of each component of the tunneling machine at the current moment; Based on the position and the attitude, a directional bounding box is generated that tightly wraps around each component and rotates in real time with the movement of each component, serving as the dynamic directional bounding box for each component of the tunneling machine.

7. The method according to claim 1, characterized in that, The step of constructing a spatiotemporal prediction sweep body bounding box based on the three-dimensional spatial state of the personnel, the dynamic orientation bounding box of the personnel, the dynamic orientation bounding boxes of each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine includes: The movement trajectory of the person within a future preset time window is determined based on the person's three-dimensional spatial state; Based on the real-time kinematic parameters and kinematic model of the tunneling machine, the motion trajectory of each component of the tunneling machine within a future preset time window is determined; The spatiotemporal prediction sweep body bounding box is generated based on the bounding box of the personnel's dynamic orientation, the bounding box of the dynamic orientation of each component of the tunneling machine, the movement trajectory of the personnel within a future preset time window, and the movement trajectory of each component of the tunneling machine within a future preset time window.

8. The method according to claim 1, characterized in that, The collision detection of the spatiotemporal prediction sweep bounding box to obtain the collision detection result includes: An intersection test of the separation axis was performed on the bounding box of the spatiotemporal prediction sweep, and the test results were obtained. If the test result shows that the projection intervals of the spatiotemporally predicted sweeping bounding box on any separation axis do not overlap, the collision detection result is determined to be without collision risk; otherwise, the minimum separation distance is determined based on the separation axis where the projection intervals overlap, and the collision detection result is determined based on the minimum separation distance.

9. The method according to claim 8, characterized in that, The preset warning threshold includes a first threshold and a second threshold; the first threshold is less than the second threshold. The step of generating graded warning information based on the collision detection results and preset warning thresholds includes: If the minimum separation distance is less than or equal to the first threshold, the graded early warning information is determined to be a dangerous intrusion early warning information; wherein, the dangerous intrusion early warning information is used to trigger equipment shutdown; If the minimum separation distance is greater than the first threshold and less than or equal to the second threshold, the graded warning information is determined to be an approach warning information; wherein, the approach warning information is used to trigger audible and visual warnings and terminal synchronous warnings.

10. A downhole personnel collision early warning system based on environmental degradation adaptive fusion, characterized in that, Including edge computing devices and host computers; The edge computing device is used to acquire image data and point cloud data of the coal mine at the same moment; to perform environmental degradation assessment on the image data and the point cloud data to obtain a first degradation factor and a second degradation factor; when both the first degradation factor and the second degradation factor are less than their respective safety thresholds, to perform short-time trajectory extrapolation based on the velocity vector of the personnel in the coal mine at the previous moment to obtain the three-dimensional spatial state of the personnel. Otherwise, a target function is determined based on the first degradation factor, the second degradation factor, the image data, and the point cloud data; the target function is solved to obtain the three-dimensional spatial state of the personnel; a dynamic orientation bounding box for the personnel is constructed based on the three-dimensional spatial state of the personnel; dynamic orientation bounding boxes for each component of the tunneling machine are constructed based on the kinematic model of the tunneling machine; and a spatiotemporal prediction sweep body bounding box is constructed based on the three-dimensional spatial state of the personnel, the dynamic orientation bounding box of the personnel, the dynamic orientation bounding boxes for each component of the tunneling machine, and the real-time kinematic parameters of the tunneling machine. Collision detection is performed on the bounding box of the spatiotemporal prediction sweep to obtain the collision detection results; The host computer is used to generate graded early warning information based on the collision detection result and a preset early warning threshold when the collision detection result indicates that there is a collision risk.

Images