Mine safety situation assessment method and system based on monitoring image

Abstract

The invention provides a mine safety situation assessment method and system based on a monitoring image, and relates to the technical field of image processing, and the method comprises the steps: obtaining a monitoring image sequence covering a mine operation area, dynamically updating the projection range of a monitoring blind area, recognizing a slope wriggling behavior of a step edge area through combining with a texture offset trend, and obtaining a slope wriggling behavior of the step edge area; meanwhile, personnel position and personnel attitude parameters are extracted, whether personnel enter a monitoring blind area projection range or not is judged according to center-of-gravity shift, multi-time scale comparison is performed on support texture under the condition that the personnel enter a blind area so as to identify abnormal deformation, and coupling characteristics are constructed with slope wriggling and personnel disturbance information; recognizing the equipment collision risk according to the relative movement trend between the mine card and the personnel, constructing a multi-source coupling risk vector, performing continuous evaluation on the multi-source coupling risk vector, and outputting a security situation evaluation result according to the dynamic security situation score. According to the method, the multi-source risk coupling situation identification capability in a dynamic blind area scene can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method and system for assessing mine safety status based on surveillance images. Background Technology

[0002] Mine safety situation assessment typically relies on a combination of fixed surveillance cameras and manual inspections. By monitoring personnel activities, vehicle operations, and the condition of infrastructure such as steps and equipment within the mining area, it identifies and warns of risks that may lead to accidents such as slips, collapses, and collisions. Existing technologies are mostly based on single-risk-source identification methods such as video target detection, personnel behavior recognition, and equipment operation monitoring. In typical open-pit mine scenarios, they mainly focus on explicit safety issues such as unauthorized entry into dangerous areas by personnel, speeding or deviation from the route by large vehicles such as mining trucks, and significant signs of damage to slope structures, in order to achieve basic perception and management capabilities of the mine's safety status.

[0003] However, in complex environments such as open-pit metal mines, the dynamic occlusion effect created by large mining trucks in areas with extremely large turning radii causes monitoring blind spots to constantly change in time and space. When personnel enter deep into the monitoring blind spot and slope creep at the edge of the step occurs slowly, the deformation of the support structure and early signs of slope failure typically exhibit low contrast, micro-displacement, and gradual evolution characteristics. Conventional algorithms based on target action recognition or static deformation detection struggle to capture this type of coupled risk. Furthermore, there is a significant amplification effect between personnel instability risk, equipment collision risk, and structural instability risk. If the system cannot jointly determine the triple coupled hazards of equipment, personnel, and steep steps, it may miss detections in the early stages of the hazard, causing potential accidents to rapidly evolve into serious disasters before safety mechanisms are triggered. Therefore, existing technologies still lack sufficient ability to identify multi-source risk coupling situations in dynamic blind spot scenarios. Summary of the Invention

[0004] This invention provides a method and system for assessing mine safety situation based on monitoring images, which can improve the ability to identify multi-source risk coupling situation in dynamic blind spot scenarios.

[0005] In a first aspect, the present invention provides a method for assessing mine safety status based on surveillance images, the method comprising: Acquire monitoring image sequence information covering the mining operation area, and dynamically update the projection range of the monitoring blind spot based on the location, outline and turning status of the mining truck; Background displacement extraction is performed on the monitoring image sequence, and slope creep behavior in the step edge region is identified based on the continuous offset trend of texture slip direction and texture displacement amplitude. Extract personnel position and posture parameters, and identify unstable disturbance information in personnel posture based on personnel center of gravity offset signal, thereby determining whether personnel have entered the projection range of the monitoring blind zone; When it is determined that personnel have entered the projection range of the monitoring blind zone, multi-time-scale comparative analysis is performed on the texture of the support material in the edge area of ​​the step to identify abnormal support material deformation and establish coupling features with the slope creep behavior and the disturbance information. Based on the relative motion trend between the outline of the mining truck and the position of the personnel, the risk of equipment collision is identified, and a multi-source coupling risk vector is constructed based on the coupling features. A continuous assessment is performed on the multi-source coupled risk vector, and the safety situation assessment result is output based on the spatial relationship between the dynamic safety situation score and the mine card avoidance path.

[0006] Based on the above technical solutions, preferably, the step of performing background displacement extraction on the monitored image sequence and identifying slope creep behavior in the step edge region based on the continuous offset trend of texture slip direction and texture displacement amplitude specifically includes: The static surface texture outside the mining card occlusion area is used as the background texture reference, and the texture displacement vector between adjacent frames is calculated based on the optical flow field estimation algorithm. Time accumulation processing is performed on the texture displacement vector within multiple consecutive sampling periods, and potential deformation information is extracted based on the offset trend of the texture slip direction and the texture displacement amplitude on the time axis. The texture slip direction and texture displacement amplitude at different spatial locations are constructed into a time series and trend offset analysis is performed to determine the activity level of the slope creep behavior; Spatial aggregation analysis is performed on the texture slip direction located in the edge region of the step, and the creep behavior is determined to be directed to the free surface based on the angle relationship between the slip direction and the geometric boundary of the step, thereby forming a slope creep behavior characterization for constructing coupled features.

[0007] Based on the above technical solutions, preferably, the step of extracting personnel position and posture parameters, and identifying unstable disturbance information in the personnel posture based on the personnel center of gravity offset signal, thereby determining whether the personnel have entered the projection range of the monitoring blind zone, specifically includes: Perform pedestrian detection in the surveillance image sequence to obtain the location areas of people; Perform key point identification within the personnel location area to obtain the coordinates of multiple joint points of the personnel skeleton; The center of gravity position is constructed based on the coordinates of the joint points, and the center of gravity displacement vector is calculated to extract the center of gravity offset signal; The disturbance information is identified based on the periodic fluctuation amplitude and direction change of the center of gravity offset signal on the time axis. The spatial overlay relationship between the personnel location area and the projection range of the monitoring blind spot is determined to determine whether the personnel have entered a dangerous area that the mine card and safety management system cannot observe.

[0008] Based on the above technical solutions, preferably, when it is determined that a person has entered the projection range of the monitoring blind zone, multi-time-scale comparative analysis is performed on the texture of the support structure in the edge area of ​​the step to identify abnormal support deformation, and a coupling feature is established with the slope creep behavior and the disturbance information, specifically including: The texture direction and texture energy of the support structure are extracted in the edge region of the step to construct texture features; The texture features are constructed into a scale texture sequence, and cumulative deformation trends and displacement events are identified through differential analysis and local drift analysis. The deflection relationship between the texture drift direction of the support structure and the geometric boundary of the step is used as the basis for determining the deformation direction of the support structure, and the spatial concentration of the deformation of the support structure is used to characterize the severity of the deformation. The deformation trend of the support structure is correlated with the execution time of the slope creep behavior, and the obtained disturbance information, support structure deformation characteristics and slope creep behavior are constructed into a coupled feature vector.

[0009] Based on the above technical solutions, preferably, the step of identifying equipment collision risks based on the relative motion trend between the outline of the mining truck and the position of personnel, and constructing a multi-source coupling risk vector based on the coupling features, specifically includes: Based on the polygon occupied by the outline of the mining truck in the mining area coordinate system and the position point of the personnel in the mining area coordinate system, a unified coordinate alignment process is performed, and the movement of the outline of the mining truck and the personnel position are respectively constructed as time series. In a local coordinate system with the mining truck as a reference, the relative displacement vector, relative velocity vector, and relative acceleration estimate of the personnel's position relative to the centroid of the mining truck's outline are calculated based on the time series. Potential approach behaviors are identified based on the angle between the relative displacement vector and the mining truck's direction of travel vector; Based on the comparison between the minimum time to contact value within the time window and the preset safety threshold, the equipment collision risk is identified, and the spatial and temporal margins of the equipment collision risk are characterized by combining the turning state and deceleration capability of the mining truck. The device collision risk and the coupling characteristics are integrated to construct a multi-source coupling risk vector.

[0010] Based on the above technical solutions, preferably, the step of performing continuous evaluation on the multi-source coupling risk vector and outputting the safety situation evaluation result according to the spatial relationship between the dynamic safety situation score and the mining truck avoidance path specifically includes: Time-cumulative analysis is performed on the multi-source coupled risk vector to construct a dynamic risk trajectory; By applying time weights to the continuously rising risk components in the dynamic risk trajectory to amplify the gradual risk manifestation, a processed risk trajectory is obtained. Multi-dimensional normalization is performed on the processed risk trajectory to form a risk offset metric; The risk offset metric is converted into a dynamic security posture score; Spatial sensitivity analysis is performed based on the dynamic safety situation score and the minimum safe distance between the mine truck avoidance path. When the dynamic safety situation score exceeds a preset threshold and the minimum safe distance is lower than the avoidance boundary, the danger situation assessment result is output.

[0011] Based on the above technical solutions, preferably, the step of acquiring monitoring image sequence information covering the mining operation area and dynamically updating the monitoring blind spot projection range according to the location, outline, and turning status of the mining truck specifically includes: Multiple fixed monitoring cameras are deployed in the mining truck operating area, step area and personnel passage area, and internal and external parameters are calibrated to enable the monitoring image sequence to be uniformly mapped to the three-dimensional coordinate reference system, thereby acquiring the monitoring image sequence information; The mining truck area is identified in the monitoring image sequence, and a contour description is extracted based on the mining truck's external outline and main driving direction. Based on the displacement vector of the mining truck's outline and the main driving direction vector of the mining truck in consecutive frames, the position and turning state of the mining truck are calculated, and the polygonal area occupied by the mining truck on the ground plane is obtained. The obstruction area is constructed based on the polygonal area occupied by the mining card and the turning status of the mining card, and the leading edge distance and lateral expansion width of the monitoring blind spot area are dynamically corrected. The blind spot area is projected onto the monitoring image plane to form the blind spot projection range, and cross-interaction analysis is performed under multiple monitoring camera perspectives to obtain a continuous blind spot projection range.

[0012] In a second aspect, the present invention provides a mine safety situation assessment system based on surveillance images. The system is used to execute a mine safety situation assessment method based on surveillance images as described in any of the above embodiments. The system includes an acquisition module, a processing module, and an output module, wherein: The acquisition module is used to acquire monitoring image sequence information covering the mining operation area, and dynamically update the monitoring blind spot projection range based on the location, outline and turning status of the mining truck. The processing module is used to extract background displacement from the monitoring image sequence and identify slope creep behavior in the step edge region based on the continuous offset trend of texture slip direction and texture displacement amplitude. The processing module is used to extract personnel position and posture parameters, and identify unstable disturbance information in personnel posture based on personnel center of gravity offset signal, thereby determining whether personnel have entered the projection range of the monitoring blind zone. The processing module is used to perform multi-timescale comparative analysis on the texture of the support material in the edge area of ​​the step when it is determined that a person has entered the projection range of the monitoring blind zone, identify abnormal support material deformation, and establish coupling features with the slope creep behavior and the disturbance information. The processing module is used to identify equipment collision risks based on the relative motion trend between the outline of the mining truck and the position of the personnel, and to construct a multi-source coupling risk vector based on the coupling features. The output module is used to perform continuous evaluation on the multi-source coupled risk vector and output the safety situation evaluation result based on the spatial relationship between the dynamic safety situation score and the mine card avoidance path.

[0013] In a third aspect of the invention, an electronic device is provided, including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the preceding embodiments.

[0014] In a fourth aspect of the invention, a non-transitory computer-readable storage medium is provided, the computer-readable storage medium storing instructions that, when executed, perform the method as described in any of the preceding claims.

[0015] In summary, one or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: 1. This invention ensures that personnel can still be indirectly perceived based on images even when they are in obscured areas by dynamically updating the projection range of the monitoring blind zone. It constructs coupled features of slope creep behavior, abnormal deformation of support structures, and personnel posture disturbances within a unified spatiotemporal framework. Furthermore, it predicts equipment collision risks by combining the relative motion trends between mining trucks and personnel. It also enhances the ability to identify progressive hazards in advance through continuous evolution analysis of multi-source coupled risk vectors. It can simultaneously focus on the synergistic amplification effect between personal safety, structural stability, and equipment operation risks in dynamic blind zone scenarios, thereby significantly improving the accuracy and real-time performance of multi-source risk coupling situation identification and early warning.

[0016] 2. By extracting minute displacement changes in surface texture from the visible area even under dynamic shading conditions of mining cards, the slow-progressing slope creep process can be accurately identified in the early stages, thereby predicting the risk of structural instability in advance.

[0017] 3. By deeply capturing subtle perturbations in human posture and combining them with dynamic occlusion areas for judgment, the monitoring system can effectively perceive the risk of instability of personnel in areas not visible to the naked eye, avoiding missed risk assessments due to blind spots.

[0018] 4. After personnel enter the projection range of the monitoring blind zone, perform multi-timescale comparative analysis on the texture of the support structure, and establish coupling features with slope creep behavior and personnel disturbance information. This can simultaneously detect local cumulative deformation and sudden micro-displacement events, and associate structural deformation risk with personnel behavior risk, so that signs of local support failure are amplified in the early stage of development, and improve the sensitivity to the precursors of structural instability.

[0019] 5. Based on the relative motion trend between the outline of the mining truck and the position of personnel, identify equipment collision risks and construct a multi-source coupled risk vector. By predicting the contact time and controllability boundary through kinematics, potential collision risks can be identified in advance during the normal operation of the equipment. The risk is then quantified in a unified manner with structural and personnel risks to form a coupled risk component, so that the synergistic amplification effect of equipment risks and other risk sources can be reflected in real time.

[0020] 6. By combining the temporal evolution trend with operational controllability in situational decision-making, a dynamic scoring mechanism is used to identify stages of sudden risk escalation. Combined with triggering uncontrollable dangerous outputs at the minimum safe distance, the system is equipped with early warning and proactive intervention capabilities, thereby significantly improving the overall safety assurance level in complex mining scenarios. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a mine safety situation assessment method based on monitoring images disclosed in an embodiment of the present invention. Figure 2This is a schematic diagram of a mine safety situation assessment system based on monitoring images disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention.

[0022] Explanation of reference numerals in the attached drawings: 201, acquisition module; 202, processing module; 203, output module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0024] In the description of the embodiments of the present invention, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0025] In the description of the embodiments of the present invention, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0026] Current mine safety situation assessments mainly rely on fixed monitoring and manual inspections, focusing on identifying obvious anomalies of single risk sources. This makes it difficult to handle monitoring blind spots in open-pit metal mines caused by the dynamic obstruction of large mining trucks. Especially when personnel enter the blind spot and the edge of the bench slowly creeps, signs of support deformation and structural damage appear in the form of low contrast and micro-displacement. Traditional methods are unable to perceive the synergistic amplification effect between multiple sources of risks such as personnel instability, equipment collisions, and slope instability, thus making it impossible to identify and warn of triple-coupled hazards in a timely manner in the early stages.

[0027] This embodiment discloses a method for assessing mine safety status based on monitoring images, referring to... Figure 1 This includes the following steps S110-S160: S110 acquires monitoring image sequence information covering the mining operation area, and dynamically updates the monitoring blind zone projection range based on the location, outline, and turning status of the mining truck.

[0028] This invention discloses a mine safety situation assessment method based on surveillance images, which is applied to a server. The server includes, but is not limited to, electronic devices such as mobile phones, tablets, wearable devices, and PCs (Personal Computers), and can also be a backend server running a mine safety situation assessment method based on surveillance images. The server can be implemented using a standalone server or a server cluster composed of multiple servers.

[0029] In one possible implementation, the process involves acquiring a sequence of monitoring images covering the mining operation area and dynamically updating the projection range of the monitoring blind spot based on the location, outline, and turning status of the mining truck. Specifically, this includes: deploying multiple fixed monitoring cameras in the mining truck operating area, step area, and personnel passage area, and performing intrinsic and extrinsic parameter calibration to ensure the monitoring image sequence is uniformly mapped to a three-dimensional coordinate reference system, thereby acquiring the monitoring image sequence information; identifying the mining truck area in the monitoring image sequence and extracting its outline description based on its outline and main driving direction; calculating the mining truck position and turning status based on the displacement vector of the mining truck's outline and its main driving direction vector in consecutive frames, and obtaining the polygonal area occupied by the mining truck on the ground plane; constructing an occupies an obstruction area based on the polygonal area occupied by the mining truck and its turning status, and dynamically correcting the leading edge distance and lateral expansion width of the monitoring blind spot area; projecting the monitoring blind spot area onto the monitoring image plane to form the monitoring blind spot projection range, and performing intersection and union analysis from multiple monitoring camera perspectives to obtain a continuous monitoring blind spot projection range.

[0030] Specifically, when deploying multiple fixed monitoring cameras in the mining truck operating area, step area, and personnel passage area, it is preferable to deploy monitoring cameras in locations with large turning radii of the mining truck, frequent changes in the edge of the steps, and dense personnel passage. The imaging resolution, frame rate, and field of view of each monitoring camera should be configured with parameters suitable for long-distance target recognition. Intrinsic and extrinsic parameter calibrations should be performed on each monitoring camera. Intrinsic parameter calibration is used to determine the imaging focal length, principal point position, and distortion coefficient of the monitoring camera. Multiple sets of calibration images are acquired using a checkerboard calibration board or a calibration board of known size to estimate the camera's intrinsic parameter matrix. Extrinsic parameter calibration is used to determine the attitude and position of the monitoring camera in the mining area's three-dimensional coordinate reference system. By recording the monitoring camera's installation height, pitch angle, azimuth angle, and translation position in the mining area coordinate system, a mapping relationship is established from the mining area's three-dimensional coordinate reference system to the monitoring camera's coordinate system and then to the image coordinate system, which can be expressed as:

[0031] Where X represents the spatial coordinates of a point in the three-dimensional coordinate reference system of the mining area, R represents the rotation matrix of the monitoring camera's attitude, t represents the translation vector of the monitoring camera in the mining area coordinate system, K represents the intrinsic parameter matrix composed of parameters such as the imaging focal length and principal point position, and u represents the pixel coordinates on the image plane. The above mapping relationship is used as a scale factor. The monitoring image sequences collected by each monitoring camera are uniformly mapped to the same three-dimensional coordinate reference system of the mining area. This gives the location of the mine car, the step, and the personnel from different monitoring camera perspectives a unified spatial measurement basis, and on this basis, a monitoring image sequence information covering the mining operation area is formed.

[0032] In the monitoring image sequence, the mining truck area is identified. Foreground detection is performed on each frame of the monitoring image based on a deep learning object detection network, and candidate detection boxes containing the mining truck are output. Within the candidate detection boxes, semantic segmentation or edge detection algorithms are combined to extract the outline of the mining truck. The outline of the truck body, the front outline, and the truck bed outline are combined to form a polygonal outline describing the outline structure of the mining truck. To obtain the main driving direction of the mining truck, the geometric distribution difference between the front and rear areas on the outline is used. Based on the feature points of the headlight position, the front edge, and the rear edge, the side where the front of the mining truck is located is identified. The direction of the line connecting the front and rear directions is defined as the main driving direction vector of the mining truck. By calculating the vector direction between the centroid of the mining truck outline polygon and the feature points of the front, a standard expression of the main driving direction vector of the mining truck is constructed. This ensures that the outline of the mining truck and the main driving direction of the mining truck are consistent and traceable in the continuous monitoring image sequence, and serves as the basic outline description for subsequent calculation of the mining truck's position and turning state.

[0033] Based on the displacement vector of the mining truck's outline and the main driving direction vector in consecutive frames, the centroid position of the mining truck's outline polygon in each frame is first calculated. The difference in centroid position between adjacent frames constitutes a sequence of displacement vectors for the mining truck's outline. By smoothing and differentiating this displacement vector sequence, the trajectory of the mining truck's position change in the three-dimensional coordinate reference system of the mining area is estimated. Simultaneously, the angle between the main driving direction vectors of the mining truck in consecutive frames is calculated to obtain a sequence of angle changes in the main driving direction of the mining truck over time. The rate of angle change is then used as a quantitative indicator of the mining truck's turning state. By back-projecting the coordinates of the vertices of the mining truck's outline polygon in the image coordinate system onto the ground plane, the homography transformation matrix H of the ground plane is used to represent the mapping relationship between the image plane and the ground plane.

[0034] in, Represents the homogeneous coordinates of the vertices of the mining truck outline in the image plane. H represents the homogeneous coordinates of the vertices of the mining truck outline in the ground plane, which are obtained from the known correspondence of feature points in the ground plane. By performing the above mapping on all outline vertices, the polygonal region occupied by the mining truck in the ground plane is obtained, so that the mining truck position, the mining truck turning state and the polygonal region occupied by the mining truck are uniformly represented in the ground plane, providing an accurate planar geometric boundary for constructing dynamic occlusion areas.

[0035] When constructing the obstruction area based on the polygonal area occupied by the mining truck and its turning state, the centroid of the polygonal area occupied by the mining truck is used as the reference point, and the main driving direction vector of the mining truck is used as the directional reference. A fan-shaped obstruction area is constructed in front of the mining truck, and rectangular or trapezoidal obstruction areas are extended on both sides of the mining truck in a direction perpendicular to the main driving direction. The forward fan-shaped obstruction area and the lateral obstruction area are merged to form the basic geometry of the monitoring blind spot area. The distance of the leading edge of the fan-shaped obstruction area is determined by comprehensively considering the length of the mining truck body, the braking distance of the mining truck, and the blind spot range of the monitoring camera. The lateral expansion width is determined by the width of the mining truck body and the lateral swing margin of the mining truck in the turning state. According to the change of turning angle reflected in the turning state of the mining truck, the center direction of the fan-shaped obstruction area is rotated and corrected in real time. According to the spatial relationship between the current position of the mining truck and the position of the monitoring camera, the distance of the leading edge of the obstruction area and the lateral expansion width are dynamically adjusted so that the monitoring blind spot area can change synchronously with the movement and turning state of the mining truck, thereby truly reflecting the obstruction range of the surface area caused by the mining truck in different postures.

[0036] When projecting the blind spot area onto the monitoring image plane to form the blind spot projection range, a projection mapping from the ground plane to the image plane is established using the extrinsic and intrinsic parameters of the monitoring camera. The polygon vertices of the blind spot area on the ground plane are converted into blind spot polygons in the image plane through the aforementioned mapping relationship, thus obtaining the blind spot projection range corresponding to the viewpoint of each monitoring camera. For multiple monitoring cameras deployed in different locations, the blind spot projection ranges under their respective viewpoints are synchronized in time. At the same timestamp, intersection and union analysis are performed on multiple blind spot projection ranges. The union operation is used to obtain the globally continuous blind spot projection range at that timestamp, and the intersection operation is used to identify the deep blind spot area jointly occluded by multiple viewpoints, thereby forming the blind spot projection range that evolves continuously over time within the mining operation area.

[0037] S120 performs background displacement extraction on the monitoring image sequence and identifies the slope creep behavior in the step edge area based on the continuous offset trend of texture slip direction and texture displacement amplitude.

[0038] In one possible implementation, background displacement extraction is performed on the monitoring image sequence, and slope creep behavior in the step edge region is identified based on the continuous offset trend of texture slip direction and texture displacement amplitude. Specifically, this includes: using the static surface texture outside the mining card shading area as a background texture reference, and calculating the texture displacement vector between adjacent frames based on the optical flow field estimation algorithm; performing time accumulation processing on the texture displacement vectors in multiple consecutive sampling periods, and extracting potential deformation information based on the offset trend of texture slip direction and texture displacement amplitude on the time axis; constructing a time series of texture slip direction and texture displacement amplitude at different spatial locations and performing trend offset analysis to determine the activity level of slope creep behavior; performing spatial aggregation analysis on the texture slip direction located in the step edge region, and determining whether the creep behavior points to the free surface based on the angle relationship between the slip direction and the geometric boundary of the step, thereby forming a slope creep behavior characterization used to construct coupled features.

[0039] Specifically, the surface area outside the area obscured by the mine card outline in the monitoring image sequence is selected as the analysis object. The rock texture, road texture, and support structure surface texture that are morphologically stable and structurally clear in this area over a long time scale are defined as static surface texture. A mask of the mine card obscured area is generated by foreground and background segmentation and mine card detection results. Pixels within the mask range are removed from the displacement estimation, and the optical flow field estimation algorithm is run only in the static surface texture area. In two adjacent frames, the pixel motion vector that satisfies the brightness consistency constraint and spatial smoothness constraint is solved for each surface texture pixel block, and the texture displacement vector field indexed by the pixel position is obtained. The texture displacement vector is used to describe the displacement direction and displacement length of the texture pattern on the image plane within a unit sampling period, thereby providing a basic motion description for the subsequent slope creep behavior characterization on the time axis.

[0040] For the texture displacement vector field obtained from multiple consecutive sampling periods, texture displacement vectors spanning time windows are collected at each spatial location. These vectors are then accumulated and smoothed in chronological order to construct a decomposition representation of long-term displacement accumulation effects and short-term disturbance components. By decomposing the texture displacement vector into two components—texture slip direction and texture displacement amplitude—the slow rotational change trend of the slip direction and the increasing or decreasing trend of the displacement amplitude over time can be tracked, respectively. When the texture slip direction remains highly consistent between adjacent sampling periods and the texture displacement amplitude exhibits a monotonically increasing or periodically amplifying characteristic within a certain time window, the motion pattern at that location is judged as potential deformation information, corresponding to a slow and continuous displacement evolution process occurring in the surface or step edge region. To highlight the long-term accumulation effect, a weighted time-cumulative displacement vector can be constructed.

[0041] in, Indicates spatial location The cumulative displacement vector at that point in time. This represents the texture displacement vector at sampling time tk. The weighting coefficient represents the weight that decays or increases over time, and N represents the number of sampling periods contained within the time window. By adjusting the way the weighting coefficient changes, recent displacement changes or long-term cumulative changes can be emphasized, thereby distinguishing between short-term noise and true deformation evolution in the time accumulation processing of texture displacement vectors.

[0042] When performing time-domain analysis on potential deformation information, the texture slip direction and texture displacement amplitude at each spatial location are constructed as time series. The texture slip direction time series reflects the angular change trajectory of the texture slip direction at that location in different sampling periods, while the texture displacement amplitude time series reflects the increase or fluctuation of the texture displacement amplitude at that location over time. By performing trend offset analysis on the above two time series, such as linear fitting or piecewise linear fitting on the texture displacement amplitude time series, a slope parameter describing the rate of change of amplitude over time is obtained. The variance or directional concentration index is calculated for the texture slip direction time series. When the slope parameter of the displacement amplitude time series remains positive and exceeds a preset threshold within a certain time window, and the directional variance of the texture slip direction time series remains within a small range, it can be determined that the slope creep behavior at that location is active. At this point, a type of creep activity index can be defined:

[0043] in, Indicates spatial location The peristaltic activity index at the site. Trend parameters representing the time series of texture displacement amplitude, such as the fitting slope, An index representing the directional concentration of a time series of texture slip direction. and The weighting coefficient is used to balance the influence of the two components; the higher the creep activity index, the more significant the creep behavior of the slope at that location, which is beneficial for screening out key creep areas in the early stage.

[0044] When identifying the creep direction at the edge of a step, a pre-constructed geometric model of the step is first used in the three-dimensional coordinate reference system of the mining area or the plane coordinate system of the surface to obtain the geometric boundary line of the step edge and the outward normal direction of the step at each point. This outward normal direction is regarded as the direction of the free surface of the step. In the step edge region, the texture slip direction at all spatial locations within the region is collected. The texture slip direction is then clustered or statistically analyzed using spatial aggregation analysis methods to obtain the distribution of the dominant slip direction. By calculating the angle between the dominant slip direction and the direction of the free surface of the step at the corresponding location, it is determined whether the creep behavior develops along the direction pointing to the free surface. This can be calculated at each location:

[0045] in, This indicates the angle between the texture slip direction and the normal direction of the step free surface. Indicates spatial location The unit vector corresponding to the texture slip direction at that location. This represents the unit vector normal to the free surface of the step at that location; when When the value is close to 1, it indicates that the direction of texture slippage is approximately the same as the normal direction of the free surface of the step, and the creep behavior tends to point towards the free surface, which poses a significant threat to slope stability. By jointly statistically analyzing the above-mentioned angle index and creep activity index in the edge area of ​​the step, a slope creep behavior characterization including creep spatial range, creep intensity and creep direction can be formed. This characterization can be directly used as the input for subsequent construction of coupled features, and the structural deformation risk, personnel disturbance information and support deformation information can be integrated in a unified spatial and temporal framework.

[0046] S130 extracts personnel position and posture parameters, and identifies unstable disturbance information in personnel posture based on personnel center of gravity offset signal, thereby determining whether personnel have entered the monitoring blind zone projection range.

[0047] In one possible implementation, personnel position and posture parameters are extracted, and unstable disturbance information in the personnel posture is identified based on the personnel center of gravity offset signal to determine whether the personnel have entered the projection range of the monitoring blind zone. Specifically, this includes: performing pedestrian detection in the monitoring image sequence to obtain the personnel position area; performing key point identification within the personnel position area to obtain the coordinates of multiple joint points of the personnel skeleton; constructing the center of gravity position based on the joint point coordinates and calculating the center of gravity displacement vector to extract the center of gravity offset signal; identifying disturbance information based on the periodic fluctuation amplitude and direction change phenomenon of the center of gravity offset signal on the time axis; and performing spatial superposition relationship determination between the personnel position area and the projection range of the monitoring blind zone to determine whether the personnel have entered a dangerous area that the mine truck and safety management system cannot observe.

[0048] Specifically, when performing pedestrian detection to obtain the location region of people in a sequence of surveillance images, a pre-trained pedestrian detection model performs target detection on each frame of the surveillance image to separate suspected human targets from complex backgrounds. Preferably, a detection framework based on convolutional neural networks is used to generate human detection boxes that cover the human body area. The location region of the person is described by the coordinates of the upper left corner, lower right corner, or center point of the detection box, along with width and height parameters. By associating detection results and tracking trajectories between consecutive frames, the same person can form a continuous trajectory of their location on the time axis. Pedestrian detection is used to quickly filter out local areas where people may be active within the global surveillance image range, thereby limiting the spatial range of subsequent pose analysis and center of gravity analysis, and reducing the probability of misanalyzing non-person background areas, making the location region of the person traceable in both time and space.

[0049] When performing keypoint recognition within the personnel location area to obtain the coordinates of multiple joints of the personnel skeleton, a human pose estimation algorithm is run on the local image obtained by cropping the aforementioned personnel detection box. Key anatomical locations such as the head, neck, shoulders, elbows, wrists, hips, knees, and ankles are identified as skeletal joints, and image coordinates and confidence scores are assigned to each skeletal joint. By analyzing the joint heatmap and association vector output by the keypoint recognition network, the coordinates of each joint in the image plane are obtained, forming a personnel skeleton data structure containing the coordinates of multiple joints. The skeletal joint coordinates are used to finely describe the personnel posture configuration, enabling the personnel's torso posture, limb posture, and overall balance state to be quantified in the form of point sets in the two-dimensional image space, providing basic data for subsequent construction of the center of gravity position and calculation of the center of gravity displacement vector.

[0050] When constructing the center of gravity position and calculating the center of gravity displacement vector based on joint coordinates, the torso joints in the human skeleton that are highly related to body stability are used as the main reference points for center of gravity estimation. For example, the left and right shoulder joints, left and right hip joints, and neck joints are selected. In each frame, the approximate center of gravity position of the human body is calculated based on the coordinates of these joints, and the center of gravity coordinates are obtained in the form of a weighted average.

[0051] in, This represents the coordinates of the centroid at sampling time t. This represents the image coordinates of the i-th trunk joint at sampling time t. The value represents the weight coefficient of the corresponding joint in the center of gravity estimation, and M represents the number of trunk joints involved in the calculation; the center of gravity position is differencing between adjacent sampling times to obtain the center of gravity displacement vector.

[0052] in, It represents the displacement vector of the center of gravity position from sampling time t-1 to sampling time t. By superimposing the center of gravity displacement vectors of consecutive frames on the time axis, a sequence of center of gravity offset signals reflecting the swaying, forward leaning, and backward leaning of the person's body is formed, providing a quantitative measure for identifying unstable disturbance information in the person's posture.

[0053] When identifying disturbance information based on the periodic fluctuation amplitude and direction changes of the center of gravity offset signal on the time axis, the center of gravity displacement vector sequence is decomposed into amplitude and direction components, and the center of gravity displacement amplitude is calculated at each sampling time:

[0054] And the direction angle of the center of gravity displacement:

[0055] in, This represents the magnitude of the centroid displacement at sampling time t. This represents the direction angle of the centroid displacement vector in the plane at sampling time t. and These represent the components of the center of gravity displacement vector in the horizontal and vertical directions, respectively. By statistically analyzing the maximum, minimum, and average values ​​of the center of gravity displacement amplitude within a time window, the periodic fluctuation characteristics of the center of gravity displacement amplitude are analyzed. Combined with the abrupt changes in the center of gravity displacement direction angle on the time axis, it is possible to distinguish between minor center of gravity swaying and unstable posture changes such as imbalance, slipping, and sharp forward leaning when standing normally and stably. When the center of gravity displacement amplitude increases significantly in a short period of time or the direction angle changes rapidly, and this phenomenon persists in multiple consecutive sampling periods, the behavior corresponding to that time period is marked as disturbance information, which is used to characterize the state of instability risk and fall risk in the posture of personnel.

[0056] When determining the spatial overlay relationship between the personnel location area and the projection range of the monitoring blind zone, the geometric description of the aforementioned projection range of the monitoring blind zone on the image plane is first used to represent the monitoring blind zone area as a polygon or irregular mask area. Then, geometric relationship detection is performed on the personnel location area at each sampling time. The center point, the center of gravity of the skeleton, or any representative feature point of the personnel location area is compared with the projection range of the monitoring blind zone to determine the point within the polygon or to perform region intersection and union operations. When the representative feature point falls within the projection range of the monitoring blind zone or there is a significant intersection between the personnel location area and the projection range of the monitoring blind zone, it is determined that the personnel are currently in a dangerous area that cannot be directly observed by the mine card and safety management system. Based on this determination, the status mark of whether the personnel are within the projection range of the monitoring blind zone is jointly analyzed with the aforementioned disturbance information. When the personnel are both within the projection range of the monitoring blind zone and there is a significant center of gravity shift disturbance, the personnel status is marked as a high-risk personnel status. This allows the instability risk in the invisible area to be given priority consideration in the subsequent safety situation assessment stage and to form a key component of multi-source coupled risk together with slope creep behavior and support deformation information.

[0057] S140: When it is determined that personnel have entered the projection range of the monitoring blind zone, perform multi-time-scale comparative analysis on the texture of the support in the edge area of ​​the step, identify abnormal support deformation, and establish coupling features with slope creep behavior and disturbance information.

[0058] In one possible implementation, when it is determined that personnel have entered the projection range of the monitoring blind zone, a multi-timescale comparative analysis is performed on the texture of the support structure in the edge area of ​​the step to identify abnormal support deformation and establish coupling features with slope creep behavior and disturbance information. Specifically, this includes: extracting the texture direction and texture energy of the support structure in the edge area of ​​the step to construct texture features; constructing the texture features into a scale texture sequence, and identifying the cumulative deformation trend and displacement events through differential analysis and local drift analysis; using the deflection relationship between the drift direction of the support texture and the geometric boundary of the step as the basis for determining the deformation direction of the support structure, and characterizing the severity of the deformation by the spatial concentration of the support deformation; performing a time correlation analysis on the deformation trend of the support structure and the slope creep behavior, and constructing the obtained disturbance information, support deformation features, and obtained slope creep behavior into a coupled feature vector.

[0059] Specifically, when extracting the texture direction and texture energy of the support structure in the step edge region, firstly, based on the pre-determined geometric boundary of the step in the three-dimensional coordinate reference system of the mining area, the image region within a certain buffer zone near the step edge is mapped back to the monitoring image plane. The step edge region covering structures such as support beams, anchor plates, and shotcrete surfaces is then extracted from the monitoring image. Subsequently, grayscale conversion and noise suppression filtering are applied to this region to reduce the interference of dust and light spots on texture description. Local gradient calculations are then performed on each pixel or small block in the grayscale image to calculate the gradient in the horizontal and vertical directions, denoted as... and And construct the local texture direction:

[0060] in, Indicates spatial location Texture direction angle at that location, This represents the gradient value of grayscale at that location in the horizontal direction. This represents the gradient value of grayscale in the vertical direction at that location. The function outputs a signed two-dimensional vector direction angle. Simultaneously, by statistically analyzing the texture directions of all pixels within the step edge region, a support texture direction field can be formed, describing the overall arrangement direction of cracks, bracing marks, and sprayed layer textures on the support surface. To measure the energy distribution of the support surface texture, a gray-level co-occurrence matrix can be constructed and texture energy features calculated, such as obtaining the probability distribution of the gray-level co-occurrence matrix given a direction and distance. Then, the texture energy is calculated:

[0061] Where E represents the texture energy within the selected region, This represents the probability of a pixel pair with gray level i and gray level j occurring under a given spatial relationship. The summation operation is performed across all gray level combinations. The higher the texture energy, the more uniform and concentrated the texture tends to be; the lower the texture energy, the more discrete and rough the texture tends to be. By continuously calculating the texture direction and texture energy of the support structure within different time sampling periods, basic texture features can be provided for subsequent identification of the expansion of small cracks and local peeling on the surface of the support structure.

[0062] When constructing the texture features of the support structure into a scale-based texture sequence, in order to take into account both fine-scale crack evolution and large-scale structural block displacement, the step edge region is divided into multiple spatial scales. For example, texture direction and texture energy are calculated at the original resolution, downsampled resolution by one time, and downsampled resolution by two times, respectively, thus forming a corresponding texture feature sequence at each scale. At each scale s, a time series can be constructed on the time axis for the texture energy features at the same spatial location. And use differential analysis to measure the change between adjacent sampling periods:

[0063] in, This represents the difference between sampling time t and sampling time at scale s. The change in texture energy between them Represents the texture energy at scale s and sampling time t; when When the values ​​show a large accumulation in the same direction at multiple consecutive moments, it indicates a cumulative deformation trend at that scale. On the other hand, by combining optical flow field or block matching methods, the local texture drift vector can be obtained at scale s. The drift change between adjacent time points is calculated through local drift analysis:

[0064] in, Represents spatial location at scale s At sampling time t relative to sampling time The local drift change This represents the texture drift vector at that location at scale s; when some locations... When a sudden increase occurs in a short period of time but does not continue in the long-term trend, it can be regarded as a local displacement event or a small-scale cracking event. Remain stable over a longer period of time and When the direction of change is consistent, it indicates that there is a continuous deformation process of the support structure in the area. By combining differential analysis and local drift analysis at multiple scales, it is possible to distinguish between transient disturbances and true cumulative deformation.

[0065] When using the deflection relationship between the texture drift direction of the support structure and the geometric boundary of the step as the basis for determining the deformation direction of the support structure, the geometric boundary line of the step edge in the surface plane is first obtained based on the three-dimensional geometric model of the mining area, and the outward normal direction of the step is calculated at each boundary point, denoted as... This direction is used to indicate the orientation of the free surface of the step; in texture drift analysis, it can be used to determine the texture drift vector of a suitable scale. Normalize the direction to obtain the unit drift direction vector, and calculate the cosine of the angle between the unit drift direction and the normal direction of the step free surface:

[0066] in, Indicates spatial location And the angle between the texture drift direction and the normal direction of the free surface of the step at sampling time t. This represents the vector indicating the direction of texture drift of the support structure. Represents the unit vector of the normal direction of the free surface of the step; when When the value is close to 1, it indicates that the texture drift direction is approximately in the same direction as the normal direction of the free surface of the step, and the deformation of the support points towards the free surface outside the slope, which has a significant impact on the stability of the step. To measure the spatial concentration of the deformation of the support, an active deformation region A can be selected in the edge area of ​​the step, and the proportion of pixels with deformation exceeding a preset threshold can be counted within this region to construct a spatial concentration index.

[0067] in, Indices representing the spatial concentration at sampling time t This represents the total number of pixels within region A. Indicates spatial location The combined texture drift amplitude or cumulative deformation at sampling time t, This represents the threshold used to determine the significance of deformation. This is an indicator function that outputs 1 when the condition is met and 0 otherwise. A higher spatial concentration index indicates that the deformation area is more concentrated and more likely to evolve into local block instability, thus it can be used as a quantitative measure of the severity of deformation.

[0068] When performing correlation analysis between the deformation trend of the support structure and the execution time of slope creep behavior, the creep activity time series is first extracted from the slope creep behavior. Deformation analysis of the support structure yielded a time series of deformation intensity. The time series of attitude disturbance intensity corresponding to the disturbance information obtained from personnel attitude analysis. On a unified time axis, align the three elements and calculate the correlation or linkage between the deformation intensity of the support structure and the creep activity of the slope using a sliding time window. For example, calculate the correlation index within the time window W.

[0069] in, This represents the correlation index between the deformation intensity of the support structure and the slope creep activity within a time window W centered at sampling time t. This indicates an operation that calculates the correlation between two time series within a window W. This represents the time index within the window; similarly, a correlation index between the deformation intensity of the support and the attitude disturbance intensity can also be constructed. Based on this, the aforementioned creep activity, support deformation intensity, attitude disturbance intensity, and corresponding correlation indicators are combined into a coupled feature vector:

[0070] in, Indicates at the sampling time The coupling feature vector, This indicates the level of activity of slope creep behavior at that moment. This indicates the strength of the deformation of the support structure at that moment. This indicates the intensity of the personnel posture disturbance at that moment. This indicates the correlation between the deformation of the support structure and the slope creep at that moment. This indicates the correlation between the deformation of the support structure and the posture disturbance of personnel at that moment; this coupled feature vector can comprehensively characterize the co-evolution relationship between the slope structure state, the local deformation state of the support structure, and the instability state of personnel at the same time point.

[0071] S150 identifies equipment collision risks based on the relative motion trend between the outline of the mining truck and the position of personnel, and constructs a multi-source coupling risk vector based on coupling features.

[0072] In one possible implementation, equipment collision risks are identified based on the relative motion trend between the mining truck's outline and the personnel's position. A multi-source coupling risk vector is then constructed based on coupling characteristics. Specifically, this includes: performing unified coordinate alignment based on the polygon occupied by the mining truck's outline in the mining area coordinate system and the personnel's position points in the same system; constructing time series for the movements of both the mining truck's outline and the personnel's position; calculating the relative displacement vector, relative velocity vector, and estimated relative acceleration of the personnel's position relative to the centroid of the mining truck's outline in a local coordinate system with the mining truck as a reference; identifying potential approach behaviors based on the angle between the relative displacement vector and the mining truck's travel direction vector; identifying equipment collision risks by comparing the minimum time to contact value within a time window with a preset safety threshold; and characterizing the spatial and temporal margins of the equipment collision risks by combining the mining truck's turning state and deceleration capability; and finally, integrating the equipment collision risks and coupling characteristics to construct a multi-source coupling risk vector.

[0073] Specifically, the three-dimensional coordinate reference system of the mining area obtained by multi-camera calibration is used as the global spatial reference. The vertex coordinates of the mine truck outline obtained by back projection in each frame of the monitoring image are represented as the occupied polygon on the ground plane. Similarly, the centroid of the personnel skeleton or the center point of the personnel position is mapped to the position point on the ground plane, so that the mine truck outline and the personnel position are described by a unified two-dimensional or three-dimensional coordinate system. At each sampling time... Below, each record the set of polygon vertices occupied by the mining card. Personnel location These positions are then sorted on the timeline to construct a time series of the movement of the mining truck's outline. Time series of personnel location movement ,in Used to describe the spatial occupancy pattern of the mining card at different sampling times. It is used to describe the spatial location trajectory of personnel at different sampling times. Through unified coordinate alignment and time series construction, the motion state of mining trucks and personnel can be analyzed for subsequent relative motion within the same spatiotemporal reference framework.

[0074] When calculating the relative displacement vector, relative velocity vector, and relative acceleration estimates of the personnel position relative to the centroid of the mine truck's outline in a local coordinate system with the mine truck as the reference, the polygon occupied by the mine truck's outline is represented at each sampling time. The position of the center of mass is denoted as The main driving direction vector of the mining truck at that moment is denoted as By constructing a local coordinate system in the mining area coordinate system with the centroid of the mining truck as the origin and the main direction of travel of the mining truck as the forward axis, and using a rotation matrix... The direction transformation from the global coordinate system to the local coordinate system is represented by: The relative position of a person in the local coordinate system of the mining truck can then be expressed as:

[0075] in, Indicates at the sampling time The relative position vector of the personnel with respect to the centroid of the mining truck. This represents the position vector of the personnel in the mining area coordinate system at that moment. This represents the position vector of the centroid of the mining truck in the mining area coordinate system at that moment. This represents the rotation matrix from the mining area coordinate system to the local coordinate system of the mining card at that moment; on the time axis... The relative velocity vector can be obtained by performing a difference operation:

[0076] And the relative acceleration estimate:

[0077] in, Indicates at the sampling time The relative velocity vector of the personnel with respect to the center of mass of the mining truck. Indicates at the sampling time Estimates of the relative acceleration of personnel with respect to the center of mass of the mining truck. This represents the time interval between adjacent sampling moments. Through the calculation of the relative displacement vector, relative velocity vector, and relative acceleration estimate, the relative motion state between the mining truck and personnel can be accurately quantified, providing basic kinematic quantities for judging the approach trend and collision risk.

[0078] When identifying potential approach behaviors based on the angle between the relative displacement vector and the mining truck's travel direction vector, the unit direction vector of the mining truck's main travel direction at the current moment is denoted as follows in the mining truck's local coordinate system: The relative displacement vector at that moment is denoted as The cosine of the angle between the two is calculated using the inner product operation:

[0079] in, Indicates at the sampling time The angle between the direction of the personnel's relative displacement to the center of mass of the mining truck and the direction of the mining truck's travel. This represents the relative displacement vector at that moment. This represents the unit vector indicating the direction of travel of the mining truck at that moment. Represents the magnitude of a vector; by... Analyze the sign and size, when Greater than a certain proximity threshold and relative distance As time decreases, it can be determined that the personnel are approaching the mining truck in a forward direction. At the same time, by combining the sign and magnitude of the component of the relative velocity vector in front of the mining truck, it is possible to further rule out situations where personnel are moving away backward or crossing laterally, which have a smaller impact on the collision risk. Thus, approach behaviors with potential collision threats can be identified in space and time.

[0080] When identifying equipment collision risks based on the comparison between the minimum time to contact value within the time window and the preset safety threshold, the projection of the relative position vector onto the direction of travel of the mining truck in the local coordinate system of the mining truck is recorded as the forward distance component. The projection of the relative velocity vector onto the direction of travel of the mining truck is denoted as the forward relative velocity component. Through projection calculations, we can obtain:

[0081]

[0082] in, This indicates the relative distance between personnel on the forward axis of the mining truck. This indicates the relative velocity of personnel on the forward axis of the mining truck. Greater than zero and A value less than zero indicates that the personnel are in front of the mine truck and approaching along the truck's direction of travel; when this approach condition is met and the relative velocity component is non-zero, the time to contact value can be estimated:

[0083] in, Indicates at the sampling time The estimated time required for spatial contact to occur between personnel and mining trucks under the current relative motion state is determined by taking the minimum time to contact value within the sliding time window:

[0084] in, Indicates the sampling time Centered time window Minimum time to contact value within, Indicates the time index within the window; when When the speed is below the preset safety threshold, a potential equipment collision risk can be identified. Simultaneously, considering the mining truck's steering status and deceleration capability, the theoretical shortest braking time is calculated by combining the truck's maximum available deceleration with the current speed, and then compared with... By comparison, a time margin index for equipment collision risk can be further constructed. The relative position of personnel and the avoidance space that the mining truck can achieve under the current lane geometry constraints and surrounding obstacle constraints can be estimated to construct a space margin index. The smaller the time margin and space margin, the higher the equipment collision risk.

[0085] When constructing a multi-source coupling risk vector by integrating equipment collision risk and coupling characteristics, the equipment collision risk quantification results obtained from the aforementioned calculations based on time-to-contact value, time margin, and spatial margin are denoted as equipment risk components. The coupled feature vector composed of slope creep behavior, support deformation characteristics, and personnel posture disturbance information is denoted as... A multi-source coupled risk vector is constructed through feature concatenation and normalization operations:

[0086] in, Indicates at the sampling time The multi-source coupling risk vector This indicates the component of the equipment collision risk at that moment. This represents the coupled feature vector at that moment, consisting of slope creep, support deformation, and human disturbance. This represents the weighting coefficient used to balance the relative importance of equipment risk components and coupling feature components in the overall risk representation; in practical implementation, it can be adjusted... Each component is normalized to ensure comparability of different risk sources on a numerical scale. Continuous evaluation of evolution over time can simultaneously reflect the combined changes in equipment collision risk, personnel instability risk, and structural instability risk within the same high-dimensional risk space.

[0087] S160 performs continuous evaluation on the multi-source coupled risk vector and outputs the safety situation evaluation results based on the spatial relationship between the dynamic safety situation score and the mine card avoidance path.

[0088] In one possible implementation, continuous evaluation is performed on the multi-source coupled risk vector, and the safety situation evaluation result is output based on the spatial relationship between the dynamic safety situation score and the mine truck avoidance path. Specifically, this includes: performing time cumulative analysis on the multi-source coupled risk vector to construct a dynamic risk trajectory; applying time weights to amplify the progressive risk manifestation based on the continuous upward state of the risk components in the dynamic risk trajectory to obtain the processed risk trajectory; performing multi-dimensional normalization on the processed risk trajectory to form a risk offset metric; converting the risk offset metric into a dynamic safety situation score; performing spatial sensitivity analysis based on the minimum safe distance between the dynamic safety situation score and the mine truck avoidance path, and outputting a dangerous situation evaluation result when the dynamic safety situation score exceeds a preset threshold and the minimum safe distance is lower than the avoidance boundary.

[0089] Specifically, when performing time-cumulative analysis on multi-source coupled risk vectors, the multi-source coupled risk vectors obtained at each sampling time are arranged in chronological order to form a discrete time series containing equipment collision risk components, personnel posture disturbance risk components, slope creep risk components, and support deformation risk components. Each risk component is accumulated, superimposed, and smoothed across sampling periods to construct a dynamic risk trajectory reflecting the evolution trend of risk over time. In implementation, the sampling time can be denoted as... The corresponding multi-source coupling risk vector is denoted as By performing a cumulative operation on the vector through a sliding time window, the dynamic risk trajectory representation within the time window is obtained:

[0090] in, Indicates the sampling time This is the dynamic risk trajectory vector at the current moment with a window length of N. This represents the multi-source coupled risk vector from the current moment to the m-th sampling period, where N represents the number of sampling periods contained within the time window. Through this time accumulation method, the instantaneous fluctuations in a single sampling are smoothed out, while the continuously rising or consistently high risk state is highlighted in the dynamic risk trajectory, thus reflecting the evolution of multi-source risks in the time dimension.

[0091] After constructing the dynamic risk trajectory, to highlight the continuous cumulative effect of progressive risk, time weights are applied to each risk component in the dynamic risk trajectory vector according to its continuously rising state. Risk components whose time is closer to the current moment are given higher weights. Simultaneously, an amplification factor can be applied to risk components that monotonically increase within multiple consecutive windows, thus obtaining a processed risk trajectory that has been emphasized in the time dimension. In implementation, the time weights can be written as... The weights are determined based on their relative positions between sampling times, using an exponential decay method.

[0092]

[0093] in, This represents the processed risk trajectory vector after introducing time weights. This represents the time weight relative to the m-th sampling period prior to the current time. The time weight coefficient represents the rate at which the weight decays, and N represents the length of the time window. A larger time weight indicates a stronger influence on the risk state at more recent times. Applying weights to form This can amplify the continuously rising risks in the risk management process, thus more sensitively reflecting the gradual accumulation of risks.

[0094] To unify the different risk components in the risk processing trajectory to comparable dimensions and magnitudes, multi-dimensional normalization is performed on the risk processing trajectory vector to form a risk offset metric. The statistical characteristics of each risk component within historical data or a sliding time window are used as a benchmark to calculate the degree of deviation of the current risk processing trajectory from the benchmark state. In implementation, this can be done by targeting the risk processing trajectory vector... Each risk component Based on its average in historical data with standard deviation Forming a normalized offset:

[0095] in, This indicates that the i-th risk component is at the sampling time. The corresponding risk offset metric, This represents the i-th component of the risk trajectory vector at that moment. This represents the mean of the component in historical statistics. This represents the standard deviation of the component in historical statistics. This prevents the introduction of small positive numbers when the denominator is zero; combining the normalized offsets of all risk components yields the risk offset metric vector.

[0096] in, Indicates at the sampling time The risk offset metric vector is M, which represents the number of risk components contained in the multi-source coupled risk vector. Through this normalization method, risk components from different sources such as equipment collision, personnel posture instability, slope creep and support deformation can be superimposed and compared on a unified scale, thereby forming a multi-dimensional offset description that can be used for further scoring.

[0097] When transforming risk offset measurement into dynamic security situation scoring, the components of the risk offset measurement vector are weighted and aggregated according to their importance to the overall security situation. Simultaneously, a nonlinear mapping can be introduced so that high offset states exhibit a more significant risk level in the scoring, while low offset states remain within a lower score range. In one embodiment, the dynamic security situation scoring can be defined using a weighted summation combined with a saturation function:

[0098] in, Indicates the sampling time Dynamic security situation score, This represents the risk offset metric for the i-th risk component. Let M represent the weight coefficient corresponding to the i-th risk component, and M represent the number of risk components. This represents a function that performs a nonlinear mapping on the result of linear risk aggregation; for example, a sigmoid, piecewise linear, or exponentially amplified function can be used. Weighting coefficients... This nonlinear function is used to represent the relative impact of equipment collision risk, personnel posture disturbance risk, slope creep risk, and support deformation risk on the overall safety situation. This is used to increase sensitivity when the score is close to the high-risk range, so that the dynamic security situation score rises rapidly when it approaches the danger threshold, thus facilitating the subsequent triggering of early warning strategies.

[0099] Based on the dynamic safety situation score, when performing spatial sensitivity analysis according to the score and the minimum safe distance between the mine truck and the avoidance path, the avoidance path of the mine truck in a short period of time is first predicted in the mining area coordinate system using the current speed, current turning state, and vehicle dynamics model of the mine truck. The avoidance path is represented as a trajectory curve on the ground plane, and an avoidance channel area is formed on both sides of the trajectory curve according to the vehicle width and safety buffer zone. At the same time, the personnel positions, dangerous areas of the slope, and high deformation areas of the support structure at the current moment and the short-term prediction moment are mapped to the same mining area coordinate system, and the minimum safe distance between the avoidance channel area and the above-mentioned dangerous targets is calculated.

[0100] in, Indicates at the sampling time The minimum safe distance between the dynamically predicted avoidance zone and the risk target area. This represents a set of risk targets that includes personnel locations, areas of high slope creep activity, and areas of abnormal deformation of the support structure. This indicates the predicted mining truck avoidance zone area at that moment. This represents the distance measurement calculation from point to area or from area to area; it combines the minimum safe distance with the dynamic safety situation score. When the dynamic safety situation score exceeds the preset risk threshold and the minimum safe distance is lower than the preset avoidance boundary, it indicates that the current risk is not only at a high level, but also difficult to avoid effectively through the conventional avoidance path of the mining truck. At this time, the system outputs the danger situation assessment result and uses the result to drive the emergency braking of the mining truck, the replanning of the avoidance path, and the generation of personnel evacuation instructions. Thus, under the premise of comprehensively considering the time evolution risk and spatial controllability, it completes the dynamic assessment and active intervention of the mine safety situation.

[0101] This embodiment also discloses a mine safety situation assessment system based on monitoring images, referring to... Figure 2 The system includes an acquisition module 201, a processing module 202, and an output module 203. It is used to execute any of the above-described mine safety situation assessment methods based on monitoring images, wherein: The acquisition module 201 is used to acquire monitoring image sequence information covering the mining operation area, and dynamically update the monitoring blind zone projection range based on the location, outline and turning status of the mining truck. Processing module 202 is used to extract background displacement from the monitoring image sequence and identify slope creep behavior in the step edge region based on the continuous offset trend of texture slip direction and texture displacement amplitude. The processing module 202 is used to extract personnel position and posture parameters, and to identify unstable disturbance information in personnel posture based on personnel center of gravity offset signal, thereby determining whether personnel have entered the monitoring blind zone projection range. Processing module 202 is used to perform multi-timescale comparative analysis on the texture of the support structure in the edge area of ​​the step when it is determined that a person has entered the projection range of the monitoring blind zone, identify abnormal support deformation, and establish coupling features with slope creep behavior and disturbance information. The processing module 202 is used to identify equipment collision risks based on the relative motion trend between the outline of the mining truck and the position of personnel, and to construct a multi-source coupling risk vector based on coupling features; Output module 203 is used to perform continuous evaluation on multi-source coupled risk vectors and output safety situation evaluation results based on the spatial relationship between dynamic safety situation score and mining truck avoidance path.

[0102] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0103] This embodiment also discloses an electronic device, referring to... Figure 3 The electronic device may include: at least one processor 301, at least one communication bus 302, user interface 303, network interface 304, and at least one memory 305.

[0104] The communication bus 302 is used to enable communication between these components.

[0105] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0106] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0107] The processor 301 may include one or more processing cores. The processor 301 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 305, and by calling data stored in memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0108] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory may include a non-transitory computer-readable storage medium. The memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. As a computer storage medium, the memory 305 may include an operating system, a network communication module, a user interface 303 module, and an application program for a mine safety situation assessment method based on monitoring images.

[0109] exist Figure 3In the electronic device shown, the user interface 303 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 301 can be used to call an application program stored in the memory 305 that is a mine safety situation assessment method based on monitoring images. When executed by one or more processors 301, the electronic device executes one or more methods as described in the above embodiments.

[0110] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0111] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0112] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 305 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned memory 305 includes various media capable of storing program code, such as a USB flash drive, external hard drive, magnetic disk, or optical disk.

[0116] The present invention also discloses a non-transitory computer-readable storage medium storing instructions. When executed by one or more processors 301, these instructions cause an electronic device to perform one or more methods as described in the above embodiments.

[0117] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the specification and the disclosure of practical truths. This invention is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A mine safety situation assessment method based on monitoring images, characterized in that, The method comprises: acquiring monitoring image sequence information covering the mine operation area, and dynamically updating the monitoring blind area projection range according to the mine card position, the mine card contour, and the mine card turning state; performing background displacement extraction on the monitoring image sequence, and identifying the slope creep behavior of the step edge region according to the continuous offset trend of the texture slip direction and the texture displacement amplitude; extracting personnel position and personnel posture parameters, and identifying unstable disturbance information in the personnel posture according to the center of gravity offset signal to determine whether the personnel enter the monitoring blind area projection range; when it is determined that the personnel enter the monitoring blind area projection range, performing multi-time scale comparative analysis on the support texture of the step edge region, identifying abnormal support deformation, and establishing coupling features with the slope creep behavior and the disturbance information; identifying the equipment collision risk according to the relative motion trend between the mine card contour and the personnel position, and constructing a multi-source coupling risk vector based on the coupling features; performing continuous evaluation on the multi-source coupling risk vector, and outputting the safety situation evaluation result according to the spatial relationship between the dynamic safety situation score and the mine card avoidance path. 2.The mine safety situation assessment method based on monitoring images of claim 1, characterized in that, The background displacement extraction on the monitoring image sequence and the identification of the slope creep behavior of the step edge region according to the continuous offset trend of the texture slip direction and the texture displacement amplitude specifically comprise: taking the static ground texture outside the mine card blocking area as a background texture reference, and calculating the texture displacement vector between adjacent frames based on an optical flow field estimation algorithm; performing time accumulation processing on the texture displacement vector in multiple continuous sampling periods, and extracting potential deformation information according to the offset trend of the texture slip direction and the texture displacement amplitude on the time axis; constructing the texture slip direction and the texture displacement amplitude at different spatial positions into a time sequence and performing trend offset analysis to determine the activity level of the slope creep behavior; performing spatial aggregation analysis on the texture slip direction located in the step edge region, and judging whether the creep behavior is directed to the free surface according to the included angle relationship between the slip direction and the step geometric boundary, so as to form the slope creep behavior representation for constructing the coupling features. 3.The mine safety situation assessment method based on monitoring images of claim 1, characterized in that, The extraction of personnel position and personnel posture parameters and the identification of unstable disturbance information in the personnel posture according to the center of gravity offset signal to determine whether the personnel enter the monitoring blind area projection range specifically comprise: performing pedestrian detection in the monitoring image sequence to obtain a personnel position area; performing key point identification in the personnel position area to obtain a plurality of joint coordinates of the personnel skeleton; constructing the center of gravity position based on the joint coordinates and calculating the center of gravity displacement vector to extract the center of gravity offset signal; identifying the disturbance information according to the periodic fluctuation amplitude and direction change phenomenon of the center of gravity offset signal on the time axis; performing spatial superposition relationship judgment between the personnel position area and the monitoring blind area projection range to determine whether the personnel enter the dangerous area that cannot be observed by the mine card and the safety management system.

4. The mine safety situation assessment method based on monitoring images according to claim 1, characterized in that, When it is determined that a person enters the monitoring blind area projection range, multi-time scale contrast analysis is performed on the support texture of the step edge region, abnormal support deformation is identified, and coupling features are established with the slope creep behavior and the disturbance information, specifically including: The texture direction and texture energy of the support in the step edge region are extracted to construct texture features; The texture features are constructed as scale texture sequences, and the cumulative deformation trend and displacement events are identified through difference analysis and local drift analysis; The deflection relationship between the support texture drift direction and the step geometric boundary is taken as the basis for judging the deformation direction of the support, and the spatial concentration of the support deformation is taken as the severity of the deformation; The support deformation trend and the slope creep behavior are analyzed in time, and the disturbance information, support deformation features, and slope creep behavior are constructed as a coupling feature vector.

5. The mine safety situation assessment method based on monitoring images according to claim 1, characterized in that, The relative motion trend between the mine card shape contour and the personnel position is used to identify the equipment collision risk, and a multi-source coupling risk vector is constructed based on the coupling features, specifically including: Based on the occupancy polygon of the mine card shape contour in the mine area coordinate system and the position point of the personnel position in the mine area coordinate system, unified coordinate alignment processing is performed, and the motion of the mine card shape contour and the personnel position is constructed as a time series; In the local coordinate system with the mine card as the reference, the relative displacement vector, relative velocity vector, and relative acceleration estimate of the personnel position relative to the centroid of the mine card shape contour are calculated according to the time series; The potential approach behavior is identified according to the included angle relationship between the relative displacement vector and the mine card driving direction vector; According to the comparison relationship between the minimum time-to-contact value in the time window and the preset safety threshold, the equipment collision risk is identified, and the spatial margin and time margin of the equipment collision risk are represented by combining the mine card turning state and the deceleration ability; The equipment collision risk and the coupling features are unified and fused to construct a multi-source coupling risk vector.

6. The mine safety situation assessment method based on monitoring images according to claim 1, characterized in that, Continuous evaluation is performed on the multi-source coupling risk vector, and the safety situation evaluation result is output according to the spatial relationship between the dynamic safety situation score and the mine card avoidance path, specifically including: Time accumulation analysis is performed on the multi-source coupling risk vector to construct a dynamic risk trajectory; According to the continuous rising state of the risk components in the dynamic risk trajectory, time weight is applied to amplify the progressive risk, and a processed risk trajectory is obtained; Multi-dimensional normalization is performed on the processed risk trajectory to form a risk offset measure; The risk offset measure is converted into a dynamic safety situation score; According to the minimum safety distance between the dynamic safety situation score and the mine card avoidance path, spatial sensitivity analysis is performed, and when the dynamic safety situation score exceeds the preset threshold and the minimum safety distance is lower than the avoidance boundary, a dangerous situation evaluation result is output.

7. The mine safety situation assessment method based on monitoring images according to claim 1, characterized in that, The monitoring image sequence information covering the mine operation area is obtained, and the monitoring blind area projection range is dynamically updated according to the mine card position, mine card shape contour, and mine card turning state, specifically including: In the mine card operation area, the step area and the personnel access area, a plurality of fixed monitoring cameras are arranged and the internal and external parameter calibration is performed to enable the monitoring image sequence to be uniformly mapped to a three-dimensional coordinate reference system, thereby collecting the monitoring image sequence signals; In the monitoring image sequence, the mine card area is identified, and the contour description is extracted based on the mine card contour and the main driving direction of the mine card; Based on the displacement vector of the mine card contour in the continuous frame and the main driving direction vector of the mine card, the mine card position and the mine card turning state are calculated, and the occupied polygon area of the mine card on the ground plane is obtained; According to the mine card occupied polygon area and the mine card turning state, the shielding area is constructed, and the front edge distance and the lateral expansion width of the monitoring blind area are dynamically corrected; The monitoring blind area is projected to the monitoring image plane to form the monitoring blind area projection range, and the intersection and union analysis is performed under the multi-monitoring camera visual angle to obtain the continuous monitoring blind area projection range.

8. A mine safety situation assessment system based on monitoring images, characterized by, The system is used to perform a mine safety situation evaluation method based on monitoring images as claimed in any one of claims 1-7, and the system comprises an acquisition module, a processing module and an output module, wherein: The acquisition module is used to acquire monitoring image sequence information covering the mine operation area, and dynamically update the monitoring blind area projection range according to the mine card position, the mine card contour and the mine card turning state; The processing module is used to perform background displacement extraction on the monitoring image sequence, and identify the slope creep behavior of the step edge area according to the continuous offset trend of the texture slip direction and the texture displacement amplitude; The processing module is used to extract the personnel position and the personnel posture parameters, and identify the disturbance information in the unstable personnel posture according to the personnel gravity center offset signal, so as to determine whether the personnel enter the monitoring blind area projection range; The processing module is used to perform multi-time scale comparative analysis on the support texture of the step edge area when it is determined that the personnel enter the monitoring blind area projection range, identify the abnormal support deformation, and establish the coupling features with the slope creep behavior and the disturbance information; The processing module is used to identify the equipment collision risk according to the relative motion trend between the mine card contour and the personnel position, and construct a multi-source coupling risk vector based on the coupling features; The output module is used to perform continuous evaluation on the multi-source coupling risk vector, and output the safety situation evaluation result according to the spatial relationship between the dynamic safety situation score and the mine card avoidance path.

9. An electronic device, comprising: The electronic device comprises a processor, a communication bus, a user interface, a network interface and a memory, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, the communication bus is used to realize the connection and communication between the components in the electronic device, and the processor is used to execute the instructions stored in the memory, so that the electronic device performs the method as claimed in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, comprising: The computer readable storage medium stores instructions, when the instructions are executed, the method as claimed in any one of claims 1-7 is performed.

Images