System and method for safety monitoring and early warning of human-machine collaborative operation at working face

Abstract

The present invention belongs to the field of multi-equipment safety management and control for personnel and equipment clusters at an underground coal mine driving working face, and solves the problem of safety hazards caused by human-machine collaborative operations as the mechanization level of underground equipment increases. Provided are a system and method for safety monitoring and early warning of human-machine collaborative operation at a working face. The system comprises a safety management and control system for mobile equipment at a working face, which is used for performing in real time position, orientation and environmental sensing of individual equipment and behavior analysis of drivers; working-face collaborative intelligent wearable devices, which are used for environmental sensing of operating personnel and individual equipment and acquiring in real time the motion postures of the operating personnel; and a working face output control subsystem, which is used for identification and management of operating personnel entry and exit, position monitoring of personnel and individual equipment, operating personnel behavior identification and monitoring and working environment monitoring, and generating corresponding alarm instruction information. The present invention can realize human-equipment and equipment-equipment safety management and control during multi-equipment collaborative operation at a driving working face.

Description

Safety Monitoring and Early Warning System and Method for Human-Machine Collaborative Operations at the Working Face Technical Field

[0001] This invention belongs to the field of multi-equipment safety management of personnel and equipment clusters in underground coal mine tunneling faces, specifically involving a safety monitoring and early warning system and method for human-machine collaborative operation in working faces. Background Technology

[0002] With the continuous improvement of mechanization in coal mining equipment, the number of machines at tunneling faces is increasing. During production operations, the constant movement and alternation of equipment poses significant safety hazards to personnel at the tunneling face. This is especially true during production periods when dust concentrations are high, visibility is poor, and noise levels are high. Large blind spots exist during equipment movement, making collisions and crushing accidents between equipment and personnel highly likely. Due to the large number of workers and the lack of self-discipline among some employees, accidents often occur due to inadequate supervision.

[0003] At present, personnel safety protection at the tunneling face mainly relies on the protection of individual equipment, and the technologies used are relatively simple, usually one of ultrasonic radar, infrared pyroelectric sensors, infrared ranging sensors, millimeter-wave radar, etc. The safety protection effect is not ideal, and it is impossible to achieve collaborative operation between equipment.

[0004] The collaborative operation of personnel and equipment at the tunneling face first requires knowing their relative positions. In recent years, with the rise of UWB technology, some mines have installed UWB-based underground precision positioning systems. However, these systems cannot achieve two-dimensional precision positioning underground and cannot calculate the relative positions between personnel and equipment, or between different pieces of equipment. Therefore, these systems cannot meet the technical requirements for collaborative operation between personnel and equipment at the underground working face. Furthermore, underground precision positioning systems are generally only used in established roadways. As the tunneling face continues to extend forward, positioning base stations cannot be deployed in a timely manner, resulting in untimely updates to the positioning data at the tunneling face, and even significant deviations in the positioning data. This makes it impossible to obtain the precise and relative positions of personnel and equipment, thus compromising the safety of both personnel and equipment during operations at the working face.

[0005] This invention establishes a safety early warning system and its control logic between equipment and personnel at the tunneling face. The aim is to enable equipment to actively avoid danger and reduce harm to personnel when danger occurs. At the same time, based on the relative positional relationship between personnel and equipment, and between equipment, safe collaborative operation between equipment and personnel is achieved. Thus, while ensuring personnel safety, orderly operation at the working face is achieved, and underground work efficiency is improved. Summary of the Invention

[0006] In order to solve at least one of the above-mentioned technical problems in the prior art, the present invention provides a safety monitoring and early warning system and method for human-machine collaborative operation at a work surface.

[0007] This invention employs the following technical solution: a safety monitoring and early warning system for human-machine collaborative operations at a workface, comprising a workface mobile equipment safety management and control system, a workface collaborative intelligent wearable device, an information sharing platform, and a workface output control subsystem; the workface mobile equipment safety management and control system is used for real-time position and direction perception of each individual machine, perception of the operating status of each individual machine and the environment within a set range around it, and collection and analysis of the behavior of each individual machine operator; the workface collaborative intelligent wearable device includes a portable positioning terminal and a motion monitoring device installed at the worker's location; the portable positioning terminal is used for perception of the relative position and direction of the worker and each individual machine, as well as the position perception of the workface, while the motion monitoring device is used for real-time acquisition of the worker's movement posture; the information sharing platform is used to collect information from each individual machine and each workface collaborative intelligent wearable device, and to achieve information sharing; the workface output control subsystem is used to receive shared information from the information sharing platform, perform worker entry and exit identification management, personnel and individual machine position identification monitoring, worker behavior identification monitoring and prediction, and work environment monitoring, and generate corresponding alarm command information; the work environment includes the gas concentrations of methane, carbon monoxide, and oxygen, as well as roof displacement, wind speed, and wind pressure.

[0008] Preferably, the safety management and control system for mobile equipment on the work surface includes a wireless positioning terminal, a camera device, a driving behavior analysis device, and multi-source sensors installed on each individual machine; the wireless positioning terminal is used to perceive the position and orientation of each individual machine in real time; the camera device is used to acquire environmental video images within a set range around each individual machine in real time and to perform environmental perception; the driving behavior analysis device is used to acquire video images of the driver's behavior in real time and to analyze the driver's behavior; and the multi-source sensors are used to acquire the operating status of each individual machine in real time.

[0009] Preferably, the work surface output control subsystem includes a wireless positioning base station, a wireless network transmission device, an equipment control device, a voice alarm device, and a centralized control platform. The wireless positioning base station is used to coordinate with the wireless positioning terminals of each individual device and the portable positioning terminals of each worker to perceive the position and direction of each individual device and worker. The wireless network transmission device is used to enable communication between the collaborative individual devices, wireless positioning terminals, and collaborative smart wearable devices of each work surface and the information sharing platform. The equipment control device is used to analyze and process data to determine the dangerous status of individual devices and workers, perform worker entry and exit identification management, personnel and individual device position identification monitoring, worker behavior identification monitoring and prediction, and work environment monitoring. It also outputs corresponding control signals according to the control strategy to control each device to perform corresponding risk avoidance actions and transmits corresponding alarm command information to the centralized control platform. The voice alarm device receives alarm command information and, based on different alarm command information categories, issues corresponding alarms. The centralized control platform, based on the shared information of the information sharing platform, performs worker entry and exit identification management, personnel and individual device position identification monitoring, worker behavior identification monitoring and prediction, and work environment monitoring, and generates corresponding alarm command information.

[0010] Preferably, the modes for staff to enter and exit the work area include: normal operation mode, shift handover mode, linkage control mode, and temporary authorization mode.

[0011] Preferably, the normal operating mode is as follows: The on-duty manager enters the identity of the on-duty staff into the control system of the workface output control subsystem, and sets the upper limit of the number of staff on the workface to N. a The access control device in the workface mobile equipment safety management system performs the current nth operation. a Identification of each worker and recording of the time T when each worker enters the work area. in and the time T to leave the work surface out If n a ≤N a If n a >N a If this condition is met, the current worker is prohibited from entering the work area, and an over-capacity alarm message is generated; where n a The time T is a positive integer; if the current worker has been in the work area for a certain period of time. w >T y If this occurs, an alarm message for staff exceeding the timeout period will be generated; where T y For the set working hours, T w =T out -T in .

[0012] Preferably, the shift handover mode is as follows: the on-duty manager and the next-duty manager confirm the handover through the workface collaborative smart wearable device and the workface output control subsystem; the next-duty manager enters the identity of the next-duty staff into the control system of the workface output control subsystem and identifies the workface staff. If the current staff does not belong to the next-duty staff, an identity mismatch alarm instruction is generated.

[0013] Preferably, the linkage control mode is as follows: When the camera device of the workface mobile equipment safety management system detects personnel from outside the workface attempting to illegally enter, climb over, or bypass the access control device, a continuous illegal entry alarm command is generated; When a worker passes through the access control device, the identity is cross-compared based on the worker's workface collaborative smart wearable device and the visual recognition information of the workface mobile equipment safety management system. If the identity cross-comparison fails, an alarm command is generated indicating that the worker's identity does not match the identification; When a manager attempts to access the system, the identity is cross-compared based on the manager's workface collaborative smart wearable device and the visual recognition information of the workface mobile equipment safety management system. If the identity cross-comparison fails, an alarm command is generated indicating that the manager's identity does not match the identification; When any worker from the workface passes through the access control device, all equipment on the workface is powered off; When any worker from the workface enters the workface, the workface output control subsystem monitors the working environment: When environmental parameters cannot meet the safe production operation requirements of the workers, the access control device opens the escape route.

[0014] Preferably, the temporary authorization mode is as follows: When a temporary worker not currently working on this work surface needs to enter, the manager issues temporary authorization through the work surface output control subsystem, and limits the duration of the temporary entry to t. n If the duration t′ of the current temporary worker's entry into the work area is... n >t n If t′ is exceeded, a temporary worker timeout alarm instruction message will be generated; where t′ n This represents the duration of time a temporary worker is currently on the work surface; the total number of temporary workers and current workers on the work surface does not exceed N. a ; where N a This is the maximum number of staff allowed on a given work surface during a shift.

[0015] Preferably, the safety control strategies of the human-machine collaborative operation safety monitoring and early warning system at the working face include normal operation control strategies, human-machine mutual avoidance control strategies, wireless shutdown control strategies, job monitoring control strategies, and special personnel management strategies.

[0016] Preferably, the normal operation control strategy is as follows: An electronic fence is defined, including a work area, an alarm area, a shutdown area, and a safety area; the status of personnel within these areas is defined as working state, dangerous state, off-duty state, and safe state; when other personnel approach the work area of ​​this unit, or when personnel of this unit approach the work area of ​​another unit, an approaching danger alarm is generated; when other personnel enter the work area of ​​this unit, or when personnel of this unit enter the work area of ​​another unit, an entering danger zone alarm is generated; the work surface output control subsystem identifies the behavior of personnel in the work area; if a personnel extends their head and / or limbs out of the cab protection area, or exhibits fatigue or yawning, an abnormal personnel behavior alarm is generated.

[0017] Preferably, the human-machine mutual avoidance control strategy is as follows: Personnel not working on this machine are defined as pedestrians, personnel controlling the movement of this machine are defined as drivers, and other personnel operating the equipment on this machine are defined as operators; When a pedestrian needs to enter the protected area of ​​this machine, an entry request is sent to the driver's work surface mobile device safety management system via the pedestrian's work surface collaborative smart wearable device; Upon receiving the request, the driver suspends the personnel approach protection function of the equipment; If the actual protection function suspension time t... r Greater than the set protection function pause time t R If the work surface output control subsystem detects that the pedestrian has safely passed, this unit will automatically restore the protection function.

[0018] Preferably, the wireless shutdown control strategy is as follows: When the work surface output control subsystem or staff detects an emergency, an emergency shutdown signal is sent via a wireless terminal, and all wireless positioning terminals within the signal coverage area immediately control each individual machine to shut down; if the actual shutdown duration is t... wq Greater than the set downtime t ws Alternatively, after management personnel check the operating conditions and confirm the safety of each individual machine, they can restore the working status of each individual machine.

[0019] Preferably, the monitoring and control strategy for a work site is as follows: If the number of operators of a single machine exceeds the set number, all equipment on the work surface is prohibited from starting; If this single machine enters the protection range of the set collaborative single machine, neither single machine will generate alarm command information; If one of the operators of this single machine leaves this single machine, an alarm command information will be generated and the alarm device will be controlled to sound the alarm when the operator leaves the equipment range or approaches this single machine again.

[0020] Preferably, the special personnel management strategy is as follows: When a single unit needs maintenance, the management staff inputs the maintenance task into the system through a smart wearable device on the work surface, and the unit to be maintained stops outputting alarm and control signals; the system sets the maintenance duration, and when maintenance personnel enter the protection range of the unit to be maintained, the system outputs relevant prompt voice to remind the maintenance personnel to pay attention; when the maintenance duration exceeds the set maintenance duration, or the management staff issues a maintenance completion instruction to the system, the unit to be maintained resumes outputting alarm and control signals.

[0021] This invention also provides an image recognition method for a safety monitoring and early warning system for human-machine collaborative operations at a work site, comprising the following steps: acquiring video images collected by a camera device in a work site mobile equipment safety management system as training data, and using images of workers labeled with personnel information as training data labels; performing unsupervised training and deep learning on an improved YOLOv5 system using the training data and corresponding training data labels to obtain an image recognition model; and performing video image recognition using the trained image recognition model to complete worker identification, target tracking and behavior prediction, and work environment identification.

[0022] Preferably, the training data labels include posture images of workers in different poses, different camera angles, and shooting categories, as well as facial images of workers in different states; poses include: standing, squatting, bending over, standing to the left, and standing to the right; shooting angles include: overhead long-range shot, overhead close-up shot, upward long-range shot, upward close-up shot, frontal long-range shot, and frontal close-up shot; shooting categories include: visible light high-definition shooting, low-light supplementary lighting shooting, and thermal imager shooting; different worker states include: unobstructed state, wearing a protective mask state, wearing a safety helmet state, wearing earplugs state, wearing earmuffs state, wearing a protective mask and safety helmet state, wearing a safety helmet and earplugs state, wearing a safety helmet and earmuffs state, wearing a protective mask and earplugs state, wearing a protective mask and earmuffs state, wearing a protective mask, safety helmet, and earplugs state, and wearing a protective mask, safety helmet, and earmuffs state.

[0023] Preferably, the image recognition model adjusts the anchor boxes using K-means clustering, and the optimal cluster centers of K-means clustering are obtained by dividing the training samples and calculating their fitness using a swarm intelligence optimization method.

[0024] Preferably, the swarm intelligence optimization method is as follows: Initialize the particle swarm, randomly generate N particles, where 1 < N < n, and n is the number of anchor boxes to be classified. Divide the n anchor boxes using the N particles as cluster centers. Calculate and normalize the distance from each sample data to the cluster center, calculate and normalize the distance between any two cluster centers, and calculate the fitness of the sample data. Optimize based on the fitness of the sample data using the PSO method to obtain the optimal cluster center.

[0025] Preferably, the distance from each sample data point to the cluster center is calculated and normalized, and the method is as follows: Among them, D in ω represents the j-th cluster in the sample dataset. j The u-th sample data A u Distance from cluster center C j The distance, K, is the number of cluster centers; Among them, D′ in Represents the normalized D in ,max(D in ),min(D in ) represent all D in The maximum and minimum values ​​in the range.

[0026] Preferably, the distance between any two cluster centers is calculated and normalized, and the method is as follows: d(ω i ,ω j )=|C i -C j | Among them, D out Let C be the distance between any two cluster centers. i and C j ω represents the i-th cluster i and the j-th cluster ω j The cluster center, N is the number of particles in the initial particle swarm, d(ω) i ,ω j ) is the cluster center C i and C j The absolute value of the difference, D out ′ is the normalized D out ,max(D out ),min(D out ) represent all D out The maximum and minimum values ​​in the range.

[0027] Preferably, the fitness of the sample data is: Where F represents the fitness of the sample data.

[0028] Preferably, after the image recognition model performs image recognition on each frame of the acquired video, it further includes: generating a video trajectory for each worker in the image; and tracking and predicting the worker's trajectory based on the video trajectory to achieve hazard assessment under conditions of partial or complete occlusion of the worker.

[0029] This invention also provides a wireless positioning method for a human-machine collaborative operation safety monitoring and early warning system, comprising the following steps: acquiring measurement data between the wireless positioning terminal and the wireless positioning base station at time t, and defining the position of the wireless positioning terminal within the wireless coverage area of ​​the wireless positioning base station as (x... t ,y t The system obtains the distance data s between the wireless positioning terminal and the wireless positioning base station; it performs a set number of differential operations on the distance data s to obtain the velocity v and acceleration a of the wireless positioning terminal; and it calculates the velocity v at time t. t acceleration a t The velocity v at time t-1 t-1 acceleration a t-1 Compare and judge: When the velocity v t acceleration a t When the change is within the set threshold range, the current location (x) of the wireless positioning terminal t ,y t ) Valid; otherwise, based on the velocity v at time t-1 t-1 acceleration a t-1 The velocity v at time t-2 t-2 acceleration a t-2 Predict the predicted velocity v′ at time t t and predicted acceleration a′ t And based on the predicted velocity v′ t and predicted acceleration a′ t Calculate the predicted location (x′) of the wireless positioning terminal at time t. t ,y′ t ).

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: The safety monitoring and early warning system for human-machine collaborative operation in the working face of the present invention is a safety control logic and strategy proposed to ensure the safety of personnel and equipment during human-machine collaborative operation in the working face. Compared with the traditional underground personnel safety protection system, the present invention is not limited to the personnel safety protection of a single piece of equipment and solves the safety protection of the work. Instead, it starts from the mining process of the working face equipment group and combines the logical control strategy of human-machine collaboration and equipment linkage to establish a safety early warning device and its control logic that fits the underground working face production process, so as to achieve dynamic unity of underground personnel, machines and processes.

[0031] The image recognition method of the human-machine collaborative operation safety monitoring and early warning system of the present invention improves the K-means in YOLOv5 by using a swarm intelligence optimization method to automatically find the optimal K value and achieve the best clustering effect, thereby improving the accuracy and precision of image recognition and providing an effective direction for improving the human-machine collaborative safety management of the system.

[0032] The wireless positioning method of the working face human-machine collaborative operation safety monitoring and early warning system of the present invention realizes single base station positioning of underground mobile equipment, avoiding the situation where the positioning data of the tunneling working face cannot be updated in real time due to the failure of the positioning base station to update the location information in a timely manner, resulting in large positioning data deviation. It can obtain the precise location and relative position of personnel and equipment in real time, thereby realizing real-time monitoring of personnel around the equipment, and can realize fine control of equipment based on the location of personnel, so as to ensure efficient production while achieving personnel safety protection.

[0033] This invention relates to a safety monitoring and early warning system for human-machine collaborative operations at work sites. It utilizes various technologies such as wireless positioning, video surveillance, and intelligent vision processing to provide safety warnings and risk control during human-machine collaborative operations, improving the false alarm rate of safety warning devices and the accuracy of personnel identification. Furthermore, it leverages intelligent wearable devices, environmental monitoring devices, and access control devices to further monitor the health of workers and the working environment in real time, thereby enhancing safety management of personnel. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 is a schematic diagram of the composition of the human-machine collaborative operation safety monitoring and early warning system of the present invention; Figure 2 is a schematic diagram of the composition of a single unit of the human-machine collaborative operation safety monitoring and early warning system of the present invention; Figure 3 is a schematic diagram of the composition of the intelligent wearable device of the human-machine collaborative operation safety monitoring and early warning system of the present invention; Figure 4 is a schematic diagram of the control strategy of the access control device of the human-machine collaborative operation safety monitoring and early warning system of the present invention; Figure 5 is a schematic diagram of the temporary human-machine mutual avoidance control strategy of the human-machine collaborative operation safety monitoring and early warning system of the present invention; Figure 6 is a schematic diagram of the wireless emergency stop control strategy of the human-machine collaborative operation safety monitoring and early warning system of the present invention; Figure 7 is a schematic diagram of the normal collaborative operation control between devices of the human-machine collaborative operation safety monitoring and early warning system of the present invention; Figure 8 is a schematic diagram of the dangerous operation control between devices of the human-machine collaborative operation safety monitoring and early warning system of the present invention; Figure 9 is a schematic diagram of the electronic fence size changing dynamically when the tail of the human-machine collaborative operation safety monitoring and early warning system of the present invention swings.

[0036] In the diagram: 11-Workface Environmental Monitoring System; 12-Workface Video Monitoring System; 13-Workface Collaborative Intelligent Wearable Device; 14-Turning Gate; 15-Centralized Control Platform; 16-Workface Entrance Video Monitoring System; 21-Camera; 22-Intelligent Visual Analysis System; 23-Data Transmission Device; 24-Multi-Source Sensor; 25-Driving Behavior Analysis Device; 26-Wireless Positioning Terminal; 27-Thermal Imager; 31-Portable Positioning Terminal; 32-Instant Messaging System; 33-Intelligent Camera; 34-Intelligent Terminal Device; 35-Intelligent Bracelet; 36-Motion Monitoring Device. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0038] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] This invention proposes a safety monitoring and early warning system for human-machine collaborative operations at a work site, comprising a work site mobile equipment safety management and control system, a work site collaborative intelligent wearable device, an information sharing platform, and a work site output control subsystem. The work site mobile equipment safety management and control system is used for real-time position and direction perception of each individual machine, perception of the operating status of each individual machine and the surrounding environment within a set range, and collection and analysis of the behavior of the operators of each individual machine. The work site collaborative intelligent wearable device 13 includes a portable positioning terminal 31 and a motion monitoring device 36 installed at the operator's location. The portable positioning terminal 31 is used for perception of the relative position and direction of the operator and each individual machine, as well as the position of the work site. The motion monitoring device 36 is used to acquire the operator's movement posture in real time. The information sharing platform is used to collect information from each individual machine and each work site collaborative intelligent wearable device 13, and to achieve information sharing. The work site output control subsystem is used to receive shared information from the information sharing platform, perform operator entry and exit identification management, personnel and individual machine position identification monitoring, operator behavior identification monitoring and prediction, and work environment monitoring, and generate corresponding alarm command information. The work environment includes the gas concentrations of methane, carbon monoxide, and oxygen, as well as roof displacement, wind speed, and wind pressure.

[0040] To more clearly illustrate the safety monitoring and early warning system for human-machine collaborative operations at working faces according to the present invention, the following detailed description of the embodiments of the present invention is provided in conjunction with Figure 1. The working face in this invention specifically refers to a tunneling working face. When conducting safety pre-screening and control of human-machine collaborative operations at other working faces, adaptive adjustments can be made based on specific application scenarios; the present invention does not limit this.

[0041] The safety monitoring and early warning system for human-machine collaborative operation at the work surface of the present invention is described in detail below: As shown in Figure 2, the safety management and control system for mobile equipment at the work surface includes a wireless positioning terminal 26, a camera device, a driving behavior analysis device 25, a multi-source sensor 24, an intelligent vision analysis system 22, and a data transmission device 23, all installed on each individual machine. The wireless positioning terminal 26 is used to perceive the position and orientation of each individual machine in real time. The camera device includes a camera 21 and a thermal imager 27, used to collect environmental video images within a set range around each individual machine in real time and to perform environmental perception. The driving behavior analysis device 25 is used to collect video images of the driver's behavior in real time and to analyze the driver's behavior. The multi-source sensor 24 is used to acquire the operating status of each individual machine in real time.

[0042] As shown in Figure 3, the work-face collaborative smart wearable device 13 includes a portable positioning terminal 31, an instant messaging system 32, a smart camera 33, a smart terminal device 34, a smart bracelet 35, and a motion monitoring device 36, all of which are installed at the work site.

[0043] The work area output control subsystem includes a wireless positioning base station, a wireless network transmission device, an equipment control device, a voice alarm device, and a centralized control platform 15. The wireless positioning base station is used to coordinate with the wireless positioning terminals 26 of each individual device and the portable positioning terminals 31 of each worker to perceive the position and direction of each individual device and worker. The wireless network transmission device is used to enable communication between the collaborative individual devices, wireless positioning terminals 26, and collaborative smart wearable devices 13 of each work area and the information sharing platform. The equipment control device is used to analyze and process data to determine the dangerous status of individual devices and workers, and to manage worker entry and exit, and personnel... The system monitors the location of each individual device, identifies and predicts the behavior of personnel, and monitors the working environment. Based on the control strategy, it outputs corresponding control signals to control each device to perform corresponding risk avoidance actions and transmits corresponding alarm command information to the centralized control platform 15. The voice alarm device receives alarm command information and, based on different alarm command information categories, issues corresponding alarms. The centralized control platform 15 is used to manage personnel entry and exit, monitor the location of personnel and individual devices, identify and predict the behavior of personnel, and monitor the working environment based on shared information from the information sharing platform, and generate corresponding alarm command information.

[0044] The management process and control strategy of the human-machine collaborative operation safety monitoring and early warning system of the present invention will be explained through several examples below.

[0045] Example 1: Work Area Access Management Process: A turnstile 14 or other personnel access management device (hereinafter collectively referred to as "access management device") equipped with an identity recognition device (iris recognition, facial recognition, or other identity recognition devices) is used to manage the access of personnel arriving at the work area entrance, preventing unauthorized personnel from entering the work area. Simultaneously, the working time of personnel entering the work area is recorded and included in attendance management. A video surveillance system 16 at the work area entrance monitors and manages the access management device, preventing personnel from illegally climbing over barriers and entering the work area. When an unauthorized person enters the security alarm range of the device, the system will issue an alarm and send a notification to the administrator's smart terminal device 34.

[0046] There are different modes for staff to enter and exit the work area: normal operation mode, shift handover mode, linkage control mode, and temporary authorization mode.

[0047] The normal operating mode is as follows: The administrator (usually the shift leader) pre-enters the identities of the on-duty workers who are allowed to pass through the system, and sets the upper limit N of the number of workers on the work surface during the shift. a When workers enter the work area, they must first pass through the access control device. The device verifies their identity; if their identity is verified, entry is permitted, and the current time T is recorded. in Unauthorized personnel are prohibited from passing through the access control device. When personnel repeatedly break in illegally, the system outputs an alarm signal.

[0048] When the number of people n passes a ≤N a When N is present, personnel are allowed to continue entering; when N is present a >N a Entry is prohibited at this time.

[0049] When staff need to leave the work area, after passing through identity verification, gate 14 opens the passage, allowing the staff to leave. The time T is recorded after the staff leaves the work area. out .

[0050] The working time T of the staff entering the work area w =T out -T in ;T in Based on the earliest time of passing through the access control device, T out The time of final passage through the access control device shall prevail. w The time cannot exceed the set working time T y The working time T set in this inventiony =24h. When T w >T y The system outputs an alarm signal to notify the super administrator (usually the system administrator).

[0051] The system records the time and number of times each staff member enters and exits. Once in normal operating mode, staff must be authorized by management personnel to enter or exit the work area. Information can be entered via portable smart terminal device 34 or other information input devices. Management personnel information is entered by the super-manager; only one manager is allowed at the start of each shift; handover is only permitted when both managers are on duty simultaneously.

[0052] The shift handover process is as follows: During shift handover, the on-duty manager and the manager of the next shift conduct a handover to confirm whether the conditions for a two-shift handover are met. When the handover conditions are met, the managers of both shifts confirm and activate the handover mode via smart terminal device 34. After the mode is activated, the manager of the next shift enters the information of the shift's staff into the device. At this time, staff with valid identification are allowed to enter the work area through the access control device.

[0053] After the personnel at the work site complete the handover work and the system is authorized and confirmed by the management personnel, it enters normal operation mode.

[0054] After the shift handover is completed, the system compiles statistics on the personnel at the work site. The system issues an alarm reminder to staff who do not belong to the current shift until they leave. Relevant alarm command information is simultaneously sent to the intelligent terminal device 34 of the management personnel and the centralized control platform 15.

[0055] The linkage control mode is as follows: The video surveillance system at the work area entrance 16 monitors the work area entrance in real time. When personnel not working on this work area illegally enter without permission or attempt to climb over or bypass the access control device, the system continuously issues alarm signals.

[0056] When the number of personnel exceeds the maximum personnel limit N specified for the work area. a Afterwards, the system will issue an alarm message such as "Work area is full" or prohibit personnel from entering the work area.

[0057] When all personnel pass through the access control device, all equipment on the workface will be powered off and stop operating, thus ensuring the safety of personnel entering and exiting the workface during shift changes. In particular, after personnel enter the workface, the workface environmental monitoring system 11 will periodically acquire the environmental parameters of the workface. When the environmental parameters cannot meet the safe production operation conditions for the personnel and an emergency evacuation from the workface is required, the access control device can promptly open the escape route to ensure that personnel on the workface can evacuate in an orderly and rapid manner.

[0058] When staff enter the work area, the video surveillance system 16 at the entrance intelligently identifies the staff member and obtains information from the staff member's smart terminal device 34. This information is then cross-referenced to prevent situations where the person's identification does not match their ID, or someone enters with a different card. This also prevents unauthorized personnel from operating or configuring the system using management terminals. All alarm signals can only be deactivated after confirmation by management personnel, and all alarm-related information is simultaneously sent to the management personnel's smart terminal device 34 and the centralized control platform 15.

[0059] The temporary authorization mode is as follows: If temporary personnel not assigned to this work area (such as gas inspectors, patrol inspectors, etc.) need to enter the work area, temporary access can be granted after approval from management, for a limited temporary entry duration t. n Entry is permitted.

[0060] No personnel (including temporary workers) may enter the work area exceeding the permitted maximum number of personnel (N) allowed per shift. a The time t′ for temporary workers to enter the work area. n >t n The system issued an alarm, reminding staff to evacuate.

[0061] The system is linked to the self-test function of the wireless positioning terminal 26 on the workface equipment. During shift handover, each worker on each piece of equipment on the workface must perform a self-test on the wireless positioning terminal 26. Only after the management personnel receive a completion signal from all equipment self-tests can the system enter the operational mode. When auxiliary transport vehicles need to enter the work area, the driver is permitted to drive the vehicle into the workface after authorization from the management personnel.

[0062] Example 2: Safety Control Strategies: The safety control strategies of the human-machine collaborative operation safety monitoring and early warning system at the working face include normal operation control strategies, human-machine mutual avoidance control strategies, wireless shutdown control strategies, job monitoring control strategies, and special personnel management strategies.

[0063] After entering the working face, the workers communicate wirelessly with the portable positioning terminal 31 (identification card, information mining lamp or other positioning terminal equipment) worn by the workers through the wireless positioning terminal 26 deployed on each working face mobile unit, so as to obtain the relative position and direction between the workers and the equipment, and between the equipment in real time.

[0064] The positioning technology can be UWB high-precision wireless positioning technology or other high-precision wireless positioning technologies; the portable positioning terminal 31 has a sleep function. When there is no effective communication with the wireless positioning terminal 26 for a long time, the portable positioning terminal 31 will go into sleep mode to extend its usage time.

[0065] When personnel enter the alarm and shutdown range of the operating equipment, the equipment's alarm will emit voice and light signals to alert the personnel, and output control signals to control the equipment to actively avoid hazards and prevent dangerous incidents. The portable positioning terminal 31 will issue alarms, vibrations, or other alerts to remind personnel. The control signals can precisely control the mechanical structure of different functional parts of the equipment, such as the cutting and traveling sections.

[0066] The intelligent vision analysis system 22 deployed on the equipment dynamically tracks pedestrians and workers at all times, and performs real-time behavior trajectory analysis and prediction, thereby realizing real-time control of personnel safety at the work site. The system stores trajectory information, alarm events and other related information on the local device. When the centralized control platform 15 establishes communication with the equipment at the work site, the system uploads the equipment data to the centralized control platform 15 through the data transmission device 23 and stores it for later traceability.

[0067] As shown in Figure 4, the normal operation control strategy is as follows: Wireless communication technology is used to measure distances between the wireless positioning terminal 26 on the device, the portable positioning terminal 31, and the wireless positioning terminals 26 on other devices. The wireless positioning terminal 26 can obtain the distance, orientation, and relative position to the portable positioning terminals 31 around the device. The wireless positioning terminal 26 can be configured with an electronic fence, i.e., the protected area of ​​the device. When the portable positioning terminal 31 enters different ranges of the wireless positioning terminal 26, it will output different control signals.

[0068] The system acquires data on various mechanical structures of the equipment, including the tail of the transport aircraft, and dynamically changes these data to enable the protection range to change accordingly. For example, by collecting information on the angle changes of the transport aircraft's tail, the size of the electronic fence changes with the swing of the tail, as shown in Figure 9. The inner frame represents the outline of the equipment, and the area between the inner and outer frames represents the electronic fence outside the equipment outline. It can be seen that the range of the electronic fence changes with the angle of the tail.

[0069] The wireless positioning terminal 26 uses parameters to classify personnel on the work surface into workers (including all personnel on the equipment such as equipment drivers and operators), pedestrians (i.e. all personnel not on this equipment), and special personnel (including maintenance personnel, cable draggers, and other special personnel). Different management and control strategies are used for different categories of personnel.

[0070] The wireless positioning terminal 26 can be configured to define the work area for staff, dividing it into different zones such as work area, alarm area, shutdown area, and safety area. Staff are in different states within each work area: on-duty in the work area; off-duty in the shutdown area; and in a safe state in the safety area.

[0071] The wireless positioning terminal 26 shares data with other wireless positioning terminals 26 and wireless positioning base stations in the vicinity via data transmission device 23, enabling real-time acquisition of the personnel and working status of each wireless positioning terminal 26. The wireless positioning terminal 26 is associated with an alarm system and employs a tiered alarm system, outputting different sound and light alarm signals based on different system working states to alert personnel. The control signals of the wireless positioning terminal 26 can control different functional mechanical structures of the equipment, such as the cutting section, scraper chain section, conveyor tail section, traveling section, and shovel, achieving precise control of the equipment.

[0072] The portable positioning terminal 31 of the same staff member can be set to staff members with different wireless positioning terminals 26.

[0073] The intelligent vision analysis system 22 on the equipment monitors the surrounding conditions in real time, enabling it to measure distances, analyze behavior, and predict the trajectories of personnel in the vicinity. When personnel violate regulations or engage in unsafe behavior, the system outputs corresponding alarm and shutdown signals.

[0074] The driving behavior analysis device 25 uses a behavior analysis camera to monitor abnormal driving behavior of the driver. When the system detects the driver's head or other limbs extending out of the driver's cab or within a specified range, it can output alarm and control signals; when the driver exhibits abnormal driving behavior such as fatigue or yawning, the system can output alarm and control signals.

[0075] As shown in Figure 5, when personnel not working on the equipment (hereinafter referred to as pedestrians) temporarily pass through the equipment's protection zone, the personnel protection function of the equipment is deactivated. The human-machine mutual avoidance control strategy is as follows: When a pedestrian needs to enter the equipment's safety protection zone, they send a request signal to the wireless positioning terminal 26 via a button on an identification card, miner's lamp, or other positioning terminal device to alert the personnel (or operators, hereinafter referred to as drivers). Upon receiving the request signal, the driver presses the confirmation button, the personnel protection function is deactivated, and all moving mechanisms of the system-controlled equipment that could potentially cause injury stop moving, allowing the pedestrian to pass. After the confirmation button is pressed, a timer begins; when t... r ≤t R At that time, the human-machine mutual avoidance function is activated; when t r >t R When the pedestrian leaves the protected area, the human-machine mutual avoidance function is deactivated, and the personnel protection function is restored, outputting corresponding alarm and control signals according to the area where the pedestrian is located. When the system detects that a pedestrian has left the protected area of ​​the equipment, the human-machine mutual avoidance function is automatically deactivated. When the pedestrian needs to enter the safe protected area of ​​the equipment again, the above steps are repeated.

[0076] To improve the production efficiency of the equipment, the system constantly monitors the relative position of the equipment and pedestrians, and then determines the specific location of the pedestrians around the equipment based on the relative position. The system then controls some moving mechanisms of the equipment to stop operating based on the location of the pedestrians.

[0077] For example, after a pedestrian passes through the protection range of the equipment and activates the human-machine mutual avoidance mode, the wireless positioning terminal 26 obtains the relative position between the pedestrian and the equipment. Then, the system controls the relevant mechanical mechanisms of the equipment to stop working based on the pedestrian's displacement trajectory. If the pedestrian passes one side of the equipment, the system controls the mechanical structure on the pedestrian side to stop operating, while the mechanical structures on other sides of the equipment can operate normally.

[0078] After receiving the signal requesting passage, the driver determines whether there are conditions for pedestrians to pass based on the equipment's operating status. If there are no conditions for passage, the driver does not need to take any action, and the human-machine mutual avoidance function cannot be activated.

[0079] Function cancellation is enabled under one of the following conditions: 1. After the pedestrian leaves the protected area of ​​the equipment; 2. After more than t r Afterwards, the system function will be automatically canceled. Specifically, this function has a time limit; after a certain period of time (t) following its activation, the function will be automatically canceled. r This function will be automatically canceled afterward. If a pedestrian remains within the protected area for an extended period, the system will cancel the pedestrian's human-machine avoidance function and output corresponding alarm and control signals based on their location. The time t is calculated for each pedestrian based on the time the human-machine avoidance function was triggered. r .

[0080] The wireless shutdown control strategy is as follows: In case of an emergency, personnel outside the equipment's protected area can activate the wireless shutdown function by pressing a button on the portable positioning terminal 31, transmitting a shutdown control signal to the wireless positioning terminal 26. Upon receiving the signal, the wireless positioning terminal 26 outputs a control signal to shut down the equipment, thereby preventing accidents.

[0081] For example, when workers are working on the work surface, and the distance between the workers and the surrounding moving or operating equipment is less than the safe distance; or when workers are working in a confined work space and are under safety threat while the equipment operator is unaware of the dangerous workers, other nearby workers can use the portable positioning terminal 31 to continuously press the emergency stop button and send a stop signal to control the equipment to stop.

[0082] The wireless shutdown function has the highest priority control. After the portable positioning terminal 31 sends a wireless emergency stop signal, all wireless positioning terminals 26 within the signal coverage area immediately output control signals to control the equipment to stop. Once the wireless shutdown function is triggered, it can only be deactivated by specific management personnel (such as: work site management personnel, shift leader, or authorized drivers or equipment operators).

[0083] The time interval between the activation and deactivation of this function must be greater than the set downtime t. ws After the equipment is shut down, the management personnel (or authorized operators) must check the equipment's operating condition and the surrounding environment to ensure the equipment is safe before restarting it. This is to prevent drivers from starting the equipment without checking its condition, which could lead to dangerous situations. The management personnel's (or authorized operators') portable positioning terminal 31 will circle around the equipment; if the time spent in this position exceeds a specified period, the shutdown function will be deactivated.

[0084] As shown in Figure 6, the specific control process for wireless shutdown is described as follows: When a pedestrian discovers a dangerous situation, they press the combination button on the portable positioning terminal 31, and the portable positioning terminal 31 issues a wireless shutdown command; after receiving the wireless shutdown command, the wireless positioning terminal 26 of the device within the signal coverage area of ​​the portable positioning terminal 31 immediately outputs a control signal to control the device to shut down; simultaneously, the system starts timing t. wq After the equipment stops working, the operator checks the surrounding operating conditions. When t wq ≤t ws The wireless positioning terminal 26 continuously outputs a shutdown signal to control the equipment to stop; when t wq >t ws When the wireless positioning terminal 26 allows the driver to deactivate the wireless emergency stop and stop outputting the stop signal, the driver can deactivate the wireless emergency stop function and start the equipment when the surrounding conditions meet normal operating requirements.

[0085] The on-duty monitoring and control strategy is as follows: When the driver operates the equipment to carry out production operations, the system monitors the location of the personnel, thereby enabling the monitoring of leaving the post, overstaffing of the post, and driver driving behavior, so as to ensure the safety of the driver and the production operation of the equipment.

[0086] When the driver operates the equipment, the wireless positioning terminal 26 on the equipment will output corresponding control signals according to the driver's location. When the driver is within the working range or safe area, the system will not output an alarm signal; when the driver enters the alarm range set by the wireless positioning terminal 26, the system will output an alarm signal; when the driver enters a dangerous area, the system will output a shutdown signal.

[0087] When the number of operators on duty exceeds the equipment's permitted operating capacity, the system will output corresponding alarm and control signals. For example, only one operator is allowed to operate the equipment within the shuttle car's working range. If more than two operators are detected within the working range, the system will fail to start normally. When an operator enters the protection range of other equipment to work collaboratively, the equipment will operate normally and will not trigger the personnel positioning system's alarm and shutdown functions. When an operator leaves the equipment and enters the protection range of other equipment, the personnel safety protection function will be activated to ensure personnel safety.

[0088] When special personnel (such as anchor maintenance personnel of the tunneling and anchoring machine) leave the equipment, and other personnel are still operating the equipment and the equipment is in a non-moving state, the system outputs an alarm signal but does not output a control signal to stop the equipment.

[0089] When personnel performing anchoring work leave their work positions and move away from the equipment, the equipment outputs a control signal to stop its movement and outputs an alarm signal to alert other personnel. When personnel approach the equipment, the equipment outputs a corresponding alarm signal. Simultaneously, based on the location of the personnel entering the equipment, the system controls the corresponding movement mechanism of the equipment to stop.

[0090] The special personnel management strategy is as follows: Special personnel refers to workers who perform special operations at the work site, such as maintenance personnel and auxiliary workers (e.g., cable dragging).

[0091] Maintenance personnel: The system will not trigger alarms for maintenance personnel during the maintenance period. Due to the special nature of their work, maintenance personnel need to continuously adjust and operate equipment during maintenance. Therefore, the system will disable alarms for maintenance personnel during this period. After the system enters maintenance mode, maintenance personnel can perform maintenance work. However, if the system detects that the maintenance personnel's working space is less than the safe space or distance, the system will send a control signal to stop the movement of the equipment's moving mechanism to ensure the safety of the maintenance personnel.

[0092] The control process for maintenance personnel is described as follows: Upon commencement of maintenance work, management personnel input the maintenance task via intelligent terminal device 34 or other means. After receiving the relevant information, the system activates the equipment maintenance mode. When maintenance personnel enter the equipment's protected area, the system blocks corresponding control signals and simultaneously outputs a voice prompt stating "Equipment is under maintenance" or other relevant messages. The system sets the maintenance duration; after the set duration is exceeded, the system will resume normal alarm and control signal output. When other special personnel (including drivers and other equipment operators) enter or intrude into the equipment's alarm range, the wireless positioning terminal 26 outputs alarm and control signals.

[0093] During maintenance, the visual analysis system (including various intelligent visual analysis systems 22 deployed on the work surface and equipment) monitors maintenance personnel and performs behavioral analysis and prediction. When maintenance personnel violate regulations or engage in improper operations, the system outputs an alarm.

[0094] The relative positions among the equipment, the coal wall and the maintenance personnel are monitored in real time through a positioning system and a video monitoring system to evaluate the safety of the maintenance personnel; when the working space of the maintenance personnel is less than the safe distance between the equipment, the coal wall and the maintenance personnel, the equipment is controlled to stop. The positioning system includes a wireless positioning terminal 26, a portable positioning terminal 31 and a wireless positioning base station, and the video monitoring system includes a working face video monitoring system 12 and camera devices arranged at the equipment and the staff.

[0095] Auxiliary operating personnel: Auxiliary operating personnel refer to the temporary operating personnel who cooperate in the operation around the equipment when the equipment is performing operations such as machine change and machine withdrawal. For example: Since the continuous miner does not have a relevant cable coiling device, an auxiliary operating personnel is required to cooperate in dragging the cable during the machine withdrawal operation to prevent the cable from being crushed.

[0096] When the equipment on the working face needs to perform a machine withdrawal operation, other auxiliary operating personnel are required to cooperate in dragging the cable. At the same time, the positioning system and the video monitoring system continuously monitor the personnel dragging the cable, and protect the safety of the personnel dragging the cable through machine vision recognition technology and the positioning system.

[0097] When dragging the cable personnel, the visual analysis system analyzes the behaviors of the surrounding staff. When it is judged that the person is performing the cable dragging operation, the visual analysis system identifies and judges the identity information of the operating personnel, and integrates it with the information of the positioning system, triggers the auxiliary operation control strategy, and monitors the operating status of the equipment. When the equipment is in the reverse machine withdrawal state, the safety protection range of the operating personnel will be changed, and the positioning system will temporarily exempt the operating personnel from stopping. That is, when the personnel enter the protection area of the equipment, the equipment will issue an alarm prompt but will not output a control signal. However, when the personnel enter the shutdown area of the equipment or are in a dangerous state, the system issues a control signal to control the equipment to stop.

[0098] The system judges the operating environment of the auxiliary operating personnel dragging the cable based on the distance and positioning direction between the equipment and the personnel for safety evaluation, and corresponding alarm and control signals can be issued when the auxiliary operating personnel are in different protection ranges of the equipment. When the auxiliary operating personnel are within the alarm range, the wireless positioning terminal 26 will control the tail of the conveyor of the equipment to stop swinging, and the equipment can walk normally; when the auxiliary operating personnel enter the shutdown range, the wireless positioning terminal 26 controls the oil pump to stop working, thereby ensuring the safety of the auxiliary operating personnel.

[0099] Example 3: Cooperative operation between equipment on the working face: Through data sharing, the relative position relationship between equipment and the working status of surrounding equipment can be obtained in real time, so as to achieve cooperative control operations among equipment clusters on the working face. It includes no alarm when equipment gets close to each other, interlocking control between equipment, etc.

[0100] The anti-alarm when devices are close to each other means that the devices on the working face need to perform collaborative operations, and it is necessary to ensure that when the devices perform collaborative operations, the driver cannot trigger the device shutdown due to approaching each other and entering the protection range of non-this device. Through the data sharing between devices, the combination of the positioning system and the video monitoring system, the interlocking control between devices is realized.

[0101] When the devices perform collaborative operations, the relative positions of the devices are relatively fixed. For example, there is a collaborative operation relationship between the discharge hopper of the shuttle car and the receiving hopper of the crusher, while there is no possible collaborative operation relationship between the receiving hopper of the shuttle car and the crusher.

[0102] 1. When the devices are close to each other, analyze and predict the relative positions between the devices through the shared data of the devices, and then judge whether the conditions for collaborative operations are met between the devices.

[0103] 2. If the devices meet the conditions for collaborative operations and can perform collaborative operations, the devices are close to each other without alarm, and at the same time, the relevant moving parts of the collaborative operations are started for interlocking operations; if the positions of the devices do not meet the conditions for collaborative operations, when the distance is too close and there is a risk of collision, and the device enters the protection range, corresponding alarm and control signals will be output.

[0104] 3. The collaborative control strategy between devices is shown in Figure 7, and Figure 8 is a schematic diagram of dangerous operations between devices; taking the collaborative operation of the shuttle car and the roadheader-anchoring machine as an example, the control logic between the devices during collaborative operations is specifically described: when the shuttle car and the roadheader-anchoring machine are close to each other, the system judges whether the devices meet the conditions for collaborative operations according to the relative positions and orientations between the devices.

[0105] When the conditions for collaborative operations are met, that is, when the coal receiving part of the shuttle car is lapped with the coal discharging part of the roadheader-anchoring machine, the system controls the relevant moving parts, the devices are interlocked, and the relevant parts are automatically started and stopped for operations; the system analyzes the coal stacking amount through the intelligent vision analysis system 22 and the weighing system; and then controls the movement of the scraper chains of the shuttle car and the roadheader-anchoring machine to ensure the normal progress of the coal unloading operation and prevent the occurrence of coal stacking; when the coal receiving amount of the shuttle car reaches the maximum amount, the scraper chain stops running. The system issues a reminder signal to prompt the driver to operate the device to leave.

[0106] When the relative positions and orientations between the devices do not meet the conditions for collaborative operations, and the driving direction and attitude of the shuttle car are close to the continuous miner, the device issues corresponding alarm and shutdown signals and performs hazard avoidance actions to ensure the safety of the device.

[0107] Example 4: Control Strategy for the Tunneling and Anchoring Face: Taking the tunneling and anchoring face of the tunneling and anchoring team as an example, the control strategy is explained in detail: The tunneling and anchoring team is equipped with a tunneling and anchoring machine, a shuttle car, and a haulage head car. The tunneling and anchoring machine is equipped with 5 workers, including one driver and two anchoring workers on each side; the shuttle car is equipped with one driver; the haulage head car is unmanned when performing crushing and transportation operations, but is equipped with one driver when moving.

[0108] The working face equipment is equipped with several wireless positioning terminals 26, camera devices, driving behavior analysis devices 25, multi-source sensors 24, an intelligent vision analysis system 22, and a data transmission device 23. The system is connected to the equipment's electrical control box to control the moving mechanisms of the equipment, such as the oil pump motor, conveyor tail, cutting motor, and anchoring drill arm, precisely controlling the equipment's operation. Cameras 21 and thermal imagers 27 monitor the operating environment around the equipment. The intelligent vision analysis system 22 analyzes the equipment's operating environment and, in conjunction with the positioning system, outputs alarm and control signals. Multi-source sensors 24 monitor the operating status of each piece of equipment and share relevant data between the devices through the data transmission device 23.

[0109] Control logic for each piece of equipment: 1. Roadheader / Anchor: The roadheader's positioning base station monitors the driver and anchor maintenance workers. Only one driver is allowed to work at a time. If the number of drivers exceeds the limit, the system will output an alarm and control the equipment to stop. The working range of the anchor maintenance workers is different from the driver's operating range. Within the alarm range, the system will control the relevant moving structures to stop working. When the equipment enters the stop range, the system will control the oil pump to stop working.

[0110] When the tunneling and anchoring machine is cutting, the driver and anchoring personnel must not leave their posts. If they leave the work area, the equipment will issue an alarm and a shutdown signal to stop the machine and ensure personnel safety.

[0111] When the tunneling and anchoring machine is performing anchoring operations, the system will only alarm without stopping the machine. The tunneling and anchoring machine can perform anchoring operations, and the system will control other moving mechanisms (including tail swing, cutting head, platform extension, etc.) to stop moving. When the equipment anchoring personnel leave the operating range of the equipment to perform tasks such as material preparation, the system will only output an alarm signal, and the tunneling and anchoring machine will continue to operate normally.

[0112] 2. Shuttle Car: The shuttle car's positioning base station monitors the driver. Only one driver is allowed to operate the equipment within the shuttle car's working range. If more than two workers are detected within the working range, the system cannot start normally; when the driver leaves the cab, the system controls the equipment to stop operating. The driving behavior analysis device 25 analyzes the driver's behavior. When the driver's head or body protrudes from the shuttle car's cab area, the system outputs an alarm signal and controls the equipment to stop.

[0113] 3. Continuous Transport Head Car: When the continuous transport head car is performing crushing and transport operations, the system will only alarm without stopping the machine. When the video monitoring system detects that someone has entered the receiving hopper or the conveyor belt, the system will output a control signal to stop the equipment. When the continuous transport head car is moving, the driver will operate the equipment. Only one person is allowed to operate the equipment at the driver's position. If the number of people in the position exceeds the limit, the system will output an alarm signal and a control signal, and the equipment will not be able to start. When the driver leaves the cab, the system will stop the equipment.

[0114] Pedestrian control strategy: When a pedestrian enters the protected area of ​​the wireless positioning terminal 26, the system outputs an alarm or shutdown signal to control the equipment to stop operating.

[0115] When a pedestrian temporarily passes through the protected area of ​​the device, the portable positioning terminal 31 sends a request signal when it presses the temporary human-machine mutual avoidance button on the portable positioning terminal 31. After the driver receives the signal and presses the confirmation button, the system activates the temporary mutual avoidance function, and the pedestrian passes through the protected area of ​​the device without triggering the device to stop.

[0116] When a pedestrian activates the temporary human-machine mutual avoidance function, the system monitors the direction and position of the pedestrian as they pass through the equipment in real time. When a person enters the protected area of ​​the equipment, the system controls the corresponding movement mechanism of the equipment to stop operating based on the pedestrian's trajectory, thereby ensuring the pedestrian's personal safety.

[0117] Wireless shutdown function: When a pedestrian discovers that someone is in the danger zone of the equipment, the pedestrian can press the wireless emergency stop button on the portable positioning terminal 31, and the system will output a control signal to shut down the equipment.

[0118] Coordinated control strategy between devices: 1. No alarm when devices are close together: When two devices are close to each other, the system does not output alarm control signals. For example, when the shuttle car operator drives the shuttle car into the protection range of the roadheader, neither the shuttle car operator nor the roadheader operator will trigger the alarm or shutdown of their respective devices; however, when the shuttle car operator gets off the shuttle car and enters the protection range of the roadheader alone, the alarm and shutdown signals of the device will be triggered, and the device will be shut down.

[0119] 2. Roadheader and Shuttle Car: When the roadheader is cutting, the scraper chain of the roadheader contains a large amount of coal. When the monitoring system of the roadheader detects that the shuttle car receiving hopper is approaching the tail of the roadheader's conveyor, the equipment does not alarm. At the same time, when the tail of the roadheader's conveyor and the shuttle car receiving hopper overlap, the scraper chain starts to start unloading coal.

[0120] The intelligent vision analysis system 22, installed on the shuttle car, monitors and analyzes the amount of coal piled in the shuttle car's receiving hopper in real time, controls the movement of the shuttle car's scraper chain, and completes the automatic coal unloading operation. When the shuttle car's coal load reaches its maximum value, the system outputs a control alarm signal to remind the driver.

[0121] 3. Shuttle Car and Connecting Car: When the tail of the shuttle car's conveyor approaches the receiving hopper of the connecting car, and the shuttle car's direction of travel meets the conditions for coordinated operation with the connecting car, the system will not output an alarm signal, but will output a control signal to start the connecting car's conveyor. When the system detects that the tail of the shuttle car's conveyor is connected to the receiving hopper of the connecting car, the shuttle car's scraper chain will automatically unload coal.

[0122] Maintenance Personnel Management: When equipment maintenance is underway, the shift leader inputs the maintenance task and duration via the smart terminal device 34. Upon receiving this information, the equipment system activates the maintenance mode. During this time, the system will not trigger an alarm when maintenance personnel enter the equipment's protected area. However, if the driver or staff enters the equipment's alarm range, the wireless positioning terminal 26 will output corresponding alarm and control signals.

[0123] Once the maintenance task is completed, the administrator cancels the maintenance mode, and the system resumes normal alarm and control signal output. However, if the maintenance operation exceeds the set time limit, the system will automatically resume normal alarm and control signal output. In this case, it is necessary to contact the administrator to re-enter the maintenance task and restart the maintenance mode.

[0124] Shift handover mode: When the system is in shift handover mode, all equipment cannot be started. Personnel use the intelligent terminal device 34 to perform self-checks on all system functions. After the self-check is complete, the intelligent terminal device 34 transmits the relevant information to the management platform. Once the management personnel receive the completion signal for all equipment self-checks, they enter the operating mode, and the equipment can begin operation.

[0125] Example 5: Worker Status Monitoring: Gas concentration sensors (such as those for methane and oxygen) and roof displacement monitors are used to monitor the working environment in real time. When the working environment becomes unfavorable to workers, such as when methane levels exceed limits, oxygen concentration decreases, or roof displacement exceeds specified values, the system will issue an alarm. The alarm information is shared with the workers' smart terminal devices 34 via the communication network. Upon receiving the information, the smart terminal devices 34 issue an alarm, notifying workers to evacuate the working face immediately. The system is linked to the working face turnstiles 14. When the system issues an alarm, the working face turnstiles 14 will open all channels; when an alarm is issued, the system will output a shutdown signal to control all equipment on the working face to stop.

[0126] Example 6: Health Status Monitoring: The smart bracelet 35 monitors the staff's vital signs in real time, such as heart rate, blood pressure, and blood oxygen saturation, and analyzes the staff's health index and fatigue level. When the staff's vital signs exceed the normal range, their health index declines, and they show signs of fatigue. The smart terminal device 34 will issue an alarm to remind the staff to pay attention to their physical condition.

[0127] When the device issues an alarm, staff must stop working. Staff must rest and adjust in a safe area, ensuring their own safety, for at least 20 minutes before resuming work. The rest time is recorded by the smart bracelet 35. If staff continue working without resting, the smart terminal device 34 will continuously issue alarm signals. If the alarm duration exceeds 10 minutes, the system will push alarm instructions to the manager's smart terminal device 34. When vital signs parameters abnormally exceed the normal range, indicating that the staff's vital signs have reached a dangerous level, the system will continuously push the staff's relevant information to the manager's smart terminal device 34.

[0128] The motion monitoring device 36 uses motion sensors (MEMS sensors or other sensors) to measure the movement posture of the staff to detect information such as gait and abnormal movement postures. When the staff exhibits abnormal movement postures such as falling or falling from a height, the system issues relevant alarm commands and pushes the alarm commands to the smart terminal device 34 of the management personnel.

[0129] After receiving continuous alarm notifications from the system, the manager's smart terminal device 34 checks the staff's physical condition. When the manager and staff are at a distance... The system can only stop alarming when the time is right. L1 represents the distance between the administrator and the staff. This refers to the duration during which management personnel and staff are less than 0.5 meters apart. Management personnel will take appropriate measures based on the staff's physical condition and the system's processing recommendations.

[0130] The system's alarm commands are shared wirelessly with the underground control platform 15 and the ground dispatch platform. Dispatchers then implement relevant emergency plans based on the alarm commands from the control platform 15 (e.g., dispatching additional medical personnel for on-site rescue, contacting local hospitals), thereby ensuring the safety of personnel.

[0131] The image recognition method of the human-machine collaborative operation safety monitoring and early warning system of the present invention uses deep learning target recognition technology to segment, process, and intelligently analyze images captured by cameras, thereby obtaining the movement posture, identity information, and behavioral trajectory of workers, and then linking them with the positioning system, centralized control platform 15, access management device, etc. for coordinated control. However, due to the high dust levels, limited background texture information, and the large amount of protective equipment worn by personnel at the tunneling face, coupled with the fact that workers carry tools while working, facial and body features are not obvious, resulting in poor image recognition performance. To address this, an improved YOLOv5 deep learning algorithm is used to identify personnel at the tunneling face, thereby achieving identity authentication, behavioral posture recognition, target tracking, and behavioral trajectory prediction, and realizing the supervision of personnel at the working face. By improving the traditional YOLOv5 algorithm, the generalization ability of the algorithm is enhanced, thereby improving the accuracy of the algorithm in identifying human targets.

[0132] The image recognition method of the workface human-machine collaborative operation safety monitoring and early warning system of the present invention includes acquiring video images collected by the camera device in the workface mobile equipment safety management system as training data, and using images of workers labeled with personnel information as training data labels; the training data labels include posture images of workers in different postures, different camera angles and shooting categories, as well as facial images of workers in different states; postures include: standing, squatting, bending down, standing on the left and standing on the right; shooting angles include: downward far shot, downward close shot, upward far shot, upward close shot, frontal far shot and frontal close shot; shooting categories include: visible light high-definition shooting, low-light supplementary lighting shooting and thermal imager shooting; different worker states include: unobstructed state, wearing a protective mask state, wearing a safety helmet state, wearing earplugs state, wearing earmuffs state, wearing a protective mask and safety helmet state, wearing a safety helmet and earplugs state, wearing a safety helmet and earmuffs state, wearing a protective mask and earplugs state, wearing a protective mask and earmuffs state, wearing a protective mask, safety helmet and earplugs state, and wearing a protective mask, safety helmet and earplugs state.

[0133] The collected training set was manually labeled and divided, with bounding boxes used to mark people, their identities, and their postures in the images, establishing a training set of personnel images against the complex background of the tunneling face. Using the training data and corresponding labels, the improved YOLOv5 was subjected to unsupervised training and deep learning to obtain an image recognition model. This trained model was then used for video image recognition, enabling the identification of workers, target tracking and behavior prediction, and work environment identification.

[0134] First, the images in the training set are preprocessed: Mosaic data augmentation is used to enlarge, reduce, and rotate the input training set, and this is used as the input to the algorithm.

[0135] YOLOv5 uses the K-means clustering algorithm to adjust anchor boxes. During application, the K value needs to be manually calibrated. This over-reliance on the initial K value makes the results prone to getting trapped in local optima, resulting in weak generalization ability. Therefore, this invention optimizes the K-means algorithm using the PSO algorithm to improve clustering accuracy and stability. The improved PSO-K-means algorithm adaptively selects the optimal anchor box value for different training sets, ensuring the system's clustering accuracy.

[0136] The K-means algorithm constructs a set {A1, A2, ..., A} of all n anchor boxes to be classified. n Cluster analysis is performed, where n is the number of anchor boxes to be classified. First, K points are selected as the initial cluster centers. The Euclidean distance from each sample to the K cluster centers is calculated, where K is the number of cluster centers (similar to the K value mentioned earlier). The cluster center closest to each sample is found and assigned to its corresponding cluster. The centroid of each cluster is calculated to obtain the new cluster centers. The cluster with the minimum sum of squared distances from each sample to its cluster center is the cluster center. As can be seen, the classification result is highly dependent on the choice of initial cluster center values; an inappropriate K value can easily lead to local optima.

[0137] Therefore, by employing a swarm intelligence algorithm to select an appropriate K value for clustering, this invention uses the PSO (Particle Swarm Optimization) algorithm to classify the sample set and selects the optimal K value as the K-means clustering partition. The specific steps are as follows: Initialize the particle swarm, randomly generate N particles, where 1 < N < n, and n is the number of anchor boxes to be classified. Use the N particles as cluster centers to partition the n anchor boxes.

[0138] Calculate and normalize the distance from each sample data point to the cluster center. Calculate and normalize the distance between any two cluster centers. Calculate the fitness of the sample data. The method for calculating and normalizing the distance from each sample data point to the cluster center is as follows: Among them, D in ω represents the j-th cluster in the sample dataset. j The u-th sample data A u Distance from cluster center C j The distance, K, is the number of cluster centers; Among them, D′ in Represents the normalized D in ,max(D in ),min(D in ) represent all D in The maximum and minimum values ​​in the cluster; calculate the distance between any two cluster centers and normalize it, the method is as follows: d(ωi ,ω j )=|C i -C j | Among them, D out Let C be the distance between any two cluster centers. i and C j ω represents the i-th cluster i and the j-th cluster ω j The cluster center, N is the number of particles in the initial particle swarm, d(ω) i ,ω j ) is the cluster center C i and C j The absolute value of the difference, D out ′ is the normalized D out ,max(D out ), min(D out ) represent all D out The maximum and minimum values ​​in the sample data; the fitness of the sample data is: Where F represents the fitness of the sample data.

[0139] The PSO method is used to optimize based on the fitness of sample data to obtain the optimal cluster centers. The feature map is input into the Neck network, employing an FPN+PAN structure to further improve feature extraction, and then fused to output a detection map. The detection map is then input into the Head network to obtain information such as the size and location of the workers. This information is compared with the size and location of the bounding boxes, and the CIoU_Loss function is used to measure the accuracy of the target boxes and detection boxes. A s B s These represent the detection box and the target box, respectively. area(A s area(B) represents the area of ​​the detection box. s The area of ​​the target bounding box, F IoU_Loss F represents the ratio of the overlapping area between the detection bounding box and the target bounding box. CIoU_Loss This is the expression for the CIoU_Loss function, where d0 represents the Euclidean distance between the center points of the target box and the predicted box, and d... c This is the distance from the diagonal of the target bounding box.

[0140] v is a parameter that measures the aspect ratio, and it is defined as: in, and h gt These correspond to the width and height of the actual target bounding box, respectively. and h p These correspond to the width and height of the prediction box, respectively.

[0141] Using the CIoU_Loss function as the output evaluation metric not only considers the overlap area between the detection box and the target box, but also adds influencing factors such as overlap area, center point distance, and aspect ratio, which can effectively detect overlapping target objects.

[0142] The traditional YOLOv5 algorithm uses the standard NMS (Non-Maximum Suppression) method. During object detection, many bounding boxes are generated. Boxes containing the same object are then filtered to obtain non-overlapping bounding boxes. This is done by comparing the Intersection over Union (IoU) of the highest-scoring predicted bounding box with the remaining boxes to a set threshold. Boxes with IoU values ​​exceeding the threshold are considered highly overlapping and discarded. Finally, a single set of non-overlapping bounding boxes is obtained. During execution, the threshold needs to be manually set, and its value significantly impacts the accuracy of object detection.

[0143] Because downhole equipment is large and the working space for personnel is relatively narrow, personnel positions are relatively dense. Using IoU-NMS can easily lead to target overlap and the rejection of detection boxes. Therefore, this invention uses an adaptive threshold DIoU-NMS algorithm to filter redundant boxes.

[0144] All predicted bounding boxes for the same target are arranged in descending order of confidence. The DIoU-NMS algorithm is then used to filter out redundant predicted bounding boxes, retaining only the best predicted bounding box. b 1 and b 2 Let B and C represent the center points of candidate boxes 1 and 2, respectively; ε represents the NMS threshold; M represents the highest-scoring predicted box; IoU is the ratio of the intersection to the union of candidate boxes 1 and 2; and B represents the center point of candidate box 2. p S represents the p-th detection box. p R represents the confidence score of the p-th detection box. DIoU It is the penalty factor of DIoU-NMS, where ρ represents the Euclidean distance between the two bounding boxes, and c 12 This represents the diagonal distance between the minimum closure regions of candidate box 1 and candidate box 2.

[0145] ε represents the NMS threshold. An adaptive algorithm is used to automatically adjust the size of ε. When the target is far away and the crowd is dense, ε is smaller; when the target is close and the crowd is small, ε is larger, thereby improving the accuracy of long-distance person identification.

[0146] The high-definition camera at the work site captures personnel around it. The improved YOLOv5 algorithm is used to perform target detection on continuous single frames of real-time video streams, thereby enabling the detection of personnel and safety helmets at the work site and achieving safety monitoring of personnel at the work site.

[0147] The camera feed is divided into electronic fence zones. Based on the hazard level of the monitored area, different safety zones are defined, such as safe zones, alarm zones, and shutdown zones. Continuous single-frame images decoded from the camera's real-time stream are used as input to the algorithm. A target detection algorithm detects miners and their safety helmets in the images, checks whether personnel are wearing helmets, and retrieves the identity of alarm-triggered personnel using a portable positioning terminal 31.

[0148] A visual analysis system monitors personnel at the work site. When personnel enter a hazardous area or are not wearing safety helmets correctly, the system identifies and triggers an alarm. Upon intrusion, the system issues an alarm and, if necessary, outputs a control signal to shut down the equipment. The system's identification results and alarm signals are transmitted to the centralized control platform 15, where the video of the current alarm event is recorded and saved, and then distributed to the intelligent terminal devices 34 of management personnel.

[0149] After performing image recognition on each frame of the acquired video, the image recognition model also includes: frame-by-frame processing of the input video, and based on the complete trajectory formed by each worker in the camera video at the working face, realizing pedestrian detection and tracking. This solves the problem of temporary loss of personnel (workers, drivers, pedestrians, etc.) due to workers being obscured by tunneling equipment, making effective monitoring and safety protection impossible. The system processes the input video using the DeepSort algorithm and the Osnet pedestrian re-identification network model to track workers, form tracking trajectories, and predict trajectories. When a worker's body is partially or completely obscured by equipment or other pedestrians, the system can predict dangerous behaviors based on the worker's trajectory information and output control signals according to the degree of danger.

[0150] Due to the high dust concentration and low light intensity at the tunneling face, ordinary cameras produce unclear images and cannot acquire effective personnel image information. This invention uses a dual-light fusion-based camera 21 and thermal imager 27 to acquire video images as input to the system's image algorithm. The specific implementation process includes: Step S10, collecting the dust concentration under mine conditions using a dust monitor, determining and executing: if the dust concentration is below a set threshold, then the camera 21 acquires a video image of the mine conditions, and the process jumps to step S20; otherwise, the thermal imager 27 and camera 21 simultaneously acquire video images of the mine conditions, and the process jumps to step S30.

[0151] Step S20: A deep convolutional neural network is used to extract high-level multi-scale feature maps and low-level multi-scale feature maps of the image acquired by the visible light imager. Iterative feedback fusion of the feature maps is performed, and the complementary relationship between the features in the fused feature maps is combined to obtain the position of the target detection bounding box in the image and identify the target detection result.

[0152] In step S20, iterative feedback fusion of feature map groups is performed. The method is as follows: Gaussian pyramids and Laplacian pyramids corresponding to each feature map in the high-level multi-scale feature map group are established; the Gaussian pyramids and Laplacian pyramids are fused with the corresponding low-level multi-scale feature maps in the feature channel dimension to obtain a multi-scale fused feature map group.

[0153] Step S30: Extract the thermal imaging contour and the position of the thermal imaging contour in the image acquired by the thermal imager 27, and obtain the candidate target detection results: Extract the thermal imaging contour of the image acquired by the thermal imager 27 using an adaptive edge detection method, and calculate the position of the thermal imaging contour in the image; calculate the similarity between the extracted contour and each target image pre-stored in the infrared thermal imaging library, and use the target image category with a similarity greater than a set value and the corresponding thermal imaging contour position as the obtained candidate target detection results.

[0154] Step S40: Construct a multi-scale pyramid pooling residual network, extract features from the image acquired by the thermal imager 27 corresponding to the image acquired by the camera 21, and filter candidate target detection results through a fully connected network to obtain the final target detection result.

[0155] In step S40, a multi-scale pyramid pooling residual network is constructed by replacing the convolutional layers of the deep convolutional neural network with dilated convolutional layers of a set scale, and replacing the last average pooling layer with a multi-scale pyramid pooling layer.

[0156] In one embodiment of the present invention, the multi-scale pyramid pooling layer has three scales, which divide the features transmitted from the previous layer into three scales. The first scale is the same feature map as the original feature map. The second scale divides the original feature map into 2×2=4 feature maps of the same size. The third scale divides the original feature map into 4×4=16 feature maps of the same size. In this way, the features transmitted from the previous layer become 21 feature maps.

[0157] A multi-layer residual block is added between the specified layers of the network after updating the convolutional and pooling layers to construct a multi-scale pyramid pooling residual network. The multi-layer residual block can be a two-layer or a three-layer residual structure: The two-layer residual structure consists of a first convolutional layer with a 3×3 kernel and a second convolutional layer with a 3×3 kernel, connected sequentially, with the input of the first convolutional layer and the output of the second convolutional layer concatenated together; The three-layer residual structure consists of a third convolutional layer with a 1×1 kernel, a fourth convolutional layer with a 3×3 kernel, and a fifth convolutional layer with a 1×1 kernel, connected sequentially, with the input of the third convolutional layer and the output of the fifth convolutional layer concatenated together.

[0158] The loss function used in the training of the multi-scale pyramid pooling residual network is: Where m is the number of training samples in network training, q is the number of object detection categories in the training samples, and x k Let y be the feature corresponding to the k-th training sample. k For training sample x k The labels are: W is the weight matrix of the network, b is the bias matrix of the network, T represents the matrix transpose, e is the natural logarithm; g represents the number of object detection categories of the g-th training sample, and L is the loss function.

[0159] At the tunneling face, there are objects with human-like outlines that are easily captured by the thermal imager 27, such as heated cables, heated motors, and heated water pumps. The outlines of these objects captured by the thermal imager 27 are similar to human outlines. As a result, during system operation, the personnel recognition algorithm may identify the infrared images of these human-like outlines as human bodies, leading to false alarms and affecting the production efficiency of the equipment at the working face.

[0160] In this invention, the temperature data of each pixel in the image acquired by the thermal imager 27 is used to reconstruct the temperature data and form a temperature matrix. The temperature image generated by the thermal imager 27 is input into the improved YOLOv5 algorithm to obtain the human body. Since the temperature of each object is different, the temperature matrix can be used to determine whether the outline of the heat-generating object collected by the thermal imager 27 is a human body, thereby improving the human target recognition rate of the system. Specifically: The temperature data collected by the thermal imager 27 is used to form a temperature matrix according to the position of each pixel in the infrared image; the value of each element in the temperature matrix represents the actual temperature value of the corresponding pixel in the thermal imaging infrared image; Thermal imaging data of the tunneling face is collected by the thermal imager 27, and the temperature matrix data corresponding to human targets and heat-generating objects such as motors, cables, and water pumps in the collected infrared images are labeled. The SVM algorithm is used to classify them to obtain datasets such as the temperature range of human targets in the underground tunneling face and the heat-generating temperature range of motors during operation based on the thermal imager 27; The continuous single-frame infrared images obtained from the thermal imaging video collected by the thermal imager 27 are used as input to the improved YOLOv5 algorithm to finally obtain the coordinates of the detection boxes labeled as human bodies; The temperature data marked within the detection box is classified using SVM. If the temperature data matches the temperature range of a human target, it is identified as a human body; if the temperature data matches the temperature range of objects such as motors and cables, it is identified as a non-human body.

[0161] The wireless positioning method of the human-machine collaborative operation safety monitoring and early warning system of the present invention addresses the challenges of large equipment and complex working environment at the tunneling face, where numerous metal obstructions such as anchor bolts and wire mesh exist. The propagation path of the UWB signal is affected by reflections and diffractions caused by obstacles such as tunnel walls and mining equipment, leading to signal attenuation, non-line-of-sight errors, and low positioning accuracy. Since the real-time accuracy requirements of the UWB positioning system are high, and the control unit of the UWB positioning base station has low processor speed and limited performance while also handling data upload and download functions, an acceleration-based filtering algorithm is employed to reduce errors caused by the wireless positioning algorithm. This algorithm is simple and reliable, and can complete the filtering task without significantly increasing the processor's workload.

[0162] The wireless positioning method based on the above-mentioned human-machine collaborative operation safety monitoring and early warning system for the work surface includes the following steps: acquiring the measurement data between the wireless positioning terminal 26 and the wireless positioning base station at time t, and defining the position of the wireless positioning terminal 26 within the wireless coverage area of ​​the base station with the wireless positioning base station as the origin (x... t ,y t The distance data s between the wireless positioning terminal 26 and the wireless positioning base station is obtained; the distance data s is differentiated a set number of times to obtain the velocity v and acceleration a of the wireless positioning terminal 26; the velocity v at time t is... t acceleration a t The velocity v at time t-1 t-1 acceleration a t-1 Compare and judge: When the velocity v t acceleration a t When the change is within the set threshold range, the current position (x) of the wireless positioning terminal 26 is... t ,y t ) Valid; otherwise, based on the velocity v at time t-1 t-1 acceleration a t-1 The velocity v at time t-2 t-2 acceleration a t-2 Predict the predicted velocity v′ at time t t and predicted acceleration a′ t And based on the predicted velocity v′ t and predicted acceleration a′ t Calculate the predicted position (x′) of wireless positioning terminal 26 at time t. t y′ t The specific prediction method is a conventional technique, which will not be elaborated upon again.

[0163] The multi-module single-base station positioning method of the present invention uses multi-module single-base station positioning technology to achieve positioning of the portable positioning terminal 31 by the wireless positioning terminal 26. A single base station refers to a ranging unit composed of multiple modules and an antenna array within a single base station; multiple modules refer to multiple identical UWB modules. Positioning is achieved by utilizing the time and angle of arrival of wireless signals at each module. The present invention is described using three modules: the wireless positioning terminal 26 is a single base station, which internally consists of ranging module A, ranging module B, ranging module C, and a calculation module, etc. Three antennas form an equilateral triangle array, using a base station antenna array composed of three 120° sector antennas to cover 360° space; each antenna corresponds to a UWB module, and one antenna and one UWB module constitute a ranging module; the portable positioning terminal 31 contains a UWB signal generation module; the ranging module can receive UWB signals, and the signal analysis and parsing are completed by the calculation module inside the base station. Specifically: t b The portable positioning terminal 31 emits UWB wireless signals. b The ranging module inside the wireless positioning terminal 26 receives the UWB signal. b Then, signal processing is performed.

[0164] Based on the signal arrival time T of the module f The distance L between the portable positioning terminal 31 and the base station is obtained. f L f =T f ·c; where c = 3 * 10 8 m / s.

[0165] After calculation, the UWB signal of the portable positioning terminal 31 is obtained. b The distances to each ranging module are as follows: the distance to ranging module A is la; the distance to ranging module B is lb; and the distance to ranging module C is lc.

[0166] The specific angle calculation process is as follows: The arrival angle of the UWB signal is calculated using the AOA algorithm to obtain the UWB signal signal of the portable positioning terminal 31. b The angles to each ranging module are as follows: the angle to ranging module A is α; the angle to ranging module B is β; and the angle to ranging module C is γ.

[0167] UWB signal sighalt b The difference in path length is related to the distance d between ranging module A and ranging module B, the angle α for reaching ranging module A, and the angle β for reaching ranging module B. The angles for reaching each module can be calculated.

[0168] For example: Calculate the angle α reaching the ranging module A: Since d is known, and la and lb are calculated, according to the law of cosines: The angle α can then be calculated using relevant geometric knowledge.

[0169] Similarly, the angle reaching ranging module B is β; the angle reaching ranging module C is γ.

[0170] According to the UWB signal of portable positioning terminal 31 b The angles and distances to each ranging module are calculated: angle α and distance la to ranging module A; angle β and distance lb to ranging module B; and angle γ and distance lc to ranging module C. This yields the position of the portable positioning terminal 31. Since the relative positions of each module are fixed, the position of the portable positioning terminal 31 can be calculated from the distances and angles. Furthermore, the specific position, orientation, and distance of the portable positioning terminal 31 can be obtained through calculation.

[0171] The present invention also proposes an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to implement the above-described method for safety monitoring and early warning of human-machine collaborative operation on the work surface.

[0172] This invention also proposes a computer-readable storage medium storing computer instructions, which are executed by a computer to implement the aforementioned method for monitoring and warning safety of human-machine collaborative operations at the work surface. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes and related descriptions of the storage device and processing device described above can be found in the corresponding processes of the foregoing method embodiments, and will not be repeated here.

[0173] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

[0174] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0175] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A safety monitoring and early warning system for human-machine collaborative operations at a work site, characterized in that: This includes a workface mobile equipment safety management and control system, workface collaborative smart wearable devices, an information sharing platform, and a workface output control subsystem; The workface mobile equipment safety management and control system is used to perceive the position and orientation of each unit in real time, perceive the operating status of each unit and the environment within the surrounding set range, and collect and analyze the behavior of the drivers of each unit. The work surface collaborative smart wearable device (13) includes a portable positioning terminal (31) and a motion monitoring device (36) installed at the worker's location; the portable positioning terminal (31) is used to sense the relative position and direction of the worker and each single machine, as well as the position of the work surface, and the motion monitoring device (36) is used to obtain the worker's motion posture in real time. The information sharing platform is used to collect information from each stand-alone device and each work surface collaborative smart wearable device (13), and to realize information sharing; The working face output control subsystem is used to receive shared information from the information sharing platform, perform personnel entry and exit identification management, personnel and individual equipment position identification monitoring, personnel behavior identification monitoring and prediction, and working environment monitoring, and generate corresponding alarm command information; the working environment includes the gas concentrations of methane, carbon monoxide and oxygen, as well as roof displacement, wind speed and wind pressure.

2. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 1, characterized in that: The safety management system for mobile equipment on the work surface includes a wireless positioning terminal (26), a camera device, a driving behavior analysis device (25), and a multi-source sensor (24) installed on each individual machine; The wireless positioning terminal (26) is used to perceive the position and orientation of each unit in real time; The camera device is used to acquire environmental video images within a set range around each individual unit in real time, and to perform environmental perception. The driving behavior analysis device (25) is used to collect video images of the driver's behavior in real time and analyze the driver's behavior; Multi-source sensors (24) are used to acquire the operating status of each unit in real time.

3. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 2, characterized in that: The working face output control subsystem includes a wireless positioning base station, a wireless network transmission device, an equipment control device, a voice alarm device, and a centralized control platform (15); The wireless positioning base station is used to combine the wireless positioning terminals (26) of each individual device and the portable positioning terminals (31) of each staff member to perceive the position and direction of each individual device and each staff member. The wireless network transmission device is used to realize communication between the collaborative stand-alone equipment, wireless positioning terminal (26), and collaborative smart wearable device (13) of each work surface and the information sharing platform; The equipment control device is used to analyze and process data to determine the dangerous status of individual machines and personnel, to identify and manage personnel entry and exit, to identify and monitor the positions of personnel and individual machines, to identify, monitor and predict the behavior of personnel, and to monitor the working environment. It also outputs corresponding control signals according to the control strategy to control each device to perform corresponding risk avoidance actions, and transmits the generated alarm instruction information to the centralized control platform (15). The voice alarm device is used to receive alarm command information and then trigger the corresponding type of alarm based on the different alarm command information categories. The centralized control platform (15) is used to manage the entry and exit of staff, monitor the location of personnel and individual equipment, monitor and predict the behavior of staff, and monitor the working environment based on the shared information of the information sharing platform, and generate corresponding alarm instruction information.

4. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 3, characterized in that: The modes for staff to enter and exit the work area include: normal operation mode, shift handover mode, linkage control mode, and temporary authorization mode.

5. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 4, characterized in that: The normal operating mode is: The on-duty manager enters the identities of the on-duty staff into the control system of the workface output control subsystem, and sets the maximum number of staff on the workface for the current shift to N. a ; The access control device in the work surface mobile equipment safety management system performs the current nth operation. a Identification of each worker and recording of the time T when each worker enters the work area. in and the time T to leave the work surface out ; If n a ≤N a If n a >N a If this condition is met, the current worker is prohibited from entering the work area, and an over-capacity alarm message is generated; where n a It is an integer greater than 0; If the current worker has been in the work area for a duration T... w >T y If this occurs, an alarm message for staff exceeding the timeout period will be generated; where T y For the set working hours, T w =T out -T in .

6. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 4, characterized in that: The shift handover procedure is as follows: The on-duty manager and the next shift manager confirm the handover through the workface collaborative smart wearable device (13) and the workface output control subsystem; The next shift manager will enter the identity of the next shift's staff into the control system of the work surface output control subsystem and identify the staff on the work surface. If the current staff does not belong to the next shift, an identity mismatch alarm instruction will be generated.

7. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 4, characterized in that: The linkage control mode is: When the camera device of the mobile equipment safety management system detects that personnel not working on the site intend to illegally enter, climb over, or bypass the access control device, it generates a continuous illegal entry alarm instruction message. When staff members pass through the access control device, their identity is cross-compared with the visual recognition information of the staff member's work surface collaborative smart wearable device (13) and the work surface mobile device safety control system. If the identity cross-comparison fails, an alarm instruction message is generated indicating that the staff member's identity does not match the identification. When a manager attempts to access the system, the system performs a cross-identity comparison based on the visual recognition information of the manager's work surface collaborative smart wearable device (13) and the work surface mobile device safety control system. If the cross-identity comparison fails, an alarm instruction message is generated indicating that the manager's identity does not match the identification. When any worker passes through the access control device at any work site, power is cut off to all equipment at that work site. Once any worker enters the work area, the work area output control subsystem monitors the working environment: When environmental parameters fail to meet the safety requirements for workers' production operations, the access control device will open the escape route.

8. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 4, characterized in that: The temporary authorization mode is: When temporary workers not currently working on this work area need to enter, the management personnel issue temporary authorization through the work area output control subsystem and limit the duration of the temporary entry to t. n ; If the duration t′ of the current temporary worker's entry into the work area is... n >t n If t′ is exceeded, a temporary worker timeout alarm instruction message will be generated; where t′ n The duration of time that temporary workers are currently on the work surface; The total number of temporary workers and current workers on the work site shall not exceed N. a ; where N a This is the maximum number of staff allowed on a given work surface during a shift.

9. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 3, characterized in that: The safety control strategies of the human-machine collaborative operation safety monitoring and early warning system at the working face include normal operation control strategy, human-machine mutual avoidance control strategy, wireless shutdown control strategy, job monitoring control strategy, and special personnel management strategy.

10. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 9, characterized in that: The normal operation control strategy is as follows: Set the scope of the electronic fence, including the work area, alarm area, shutdown area and safety area, and set the status of the staff in different areas as working status, dangerous status, off-duty status and safe status; When other workers approach the work area of ​​this unit, or when workers of this unit approach the work area of ​​other units, an approaching danger alarm message is generated. When other staff enter the work area of ​​this unit, or when staff of this unit enter the work area of ​​other units, an alarm command message for entering a dangerous area is generated. The work area output control subsystem identifies the behavior of workers in the work area. If a worker extends their head and / or limbs out of the cab protection area, or if they appear fatigued or yawning, an alarm command message for abnormal worker behavior is generated.

11. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 9, characterized in that: The human-machine mutual avoidance control strategy is as follows: Personnel not working on this machine are defined as pedestrians; personnel controlling the movement of this machine are defined as drivers; and other personnel operating the equipment on this machine are defined as operators. When a pedestrian needs to enter the protection range of this stand-alone machine, the pedestrian's work surface collaborative smart wearable device (13) sends an entry request to the driver's work surface mobile device safety management system; Upon receiving the request, the driver suspended the equipment's personnel approach protection function; If the actual protection function is suspended for time t r Greater than the set protection function pause time t R If the work surface output control subsystem detects that the pedestrian has safely passed, this unit will automatically restore the protection function.

12. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 9, characterized in that: The wireless shutdown control strategy is as follows: When the work surface output control subsystem or staff monitors an emergency, an emergency stop signal is sent through the wireless terminal, and all wireless positioning terminals (26) within the signal coverage area immediately control each unit to stop. If the actual downtime is t wq Greater than the set downtime t ws Alternatively, after management personnel check the operating conditions and confirm the safety of each individual machine, they can restore the working status of each individual machine.

13. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 9, characterized in that: The job monitoring and control strategy is as follows: If the number of operators for a single machine exceeds the set number, all equipment on the work surface shall be prohibited from starting; If this single unit enters the protection range of the set collaborative single unit, neither single unit will generate alarm command information; If one of the operators of this unit leaves the unit, and upon leaving the equipment area or approaching the unit again, an alarm command message is generated and the alarm device is activated accordingly.

14. The working face human-machine collaborative operation safety monitoring and early warning system according to claim 9, characterized in that: Special personnel management strategies are as follows: When a single machine needs to be repaired, the management staff inputs the repair task into the system through the work surface collaborative smart wearable device (13), and the single machine to be repaired stops alarming and outputs control signals. The system is set to maintain a maintenance time. When maintenance personnel enter the protection range of the unit to be maintained, the system will output relevant voice prompts to remind the maintenance personnel to pay attention. When the maintenance time exceeds the set maintenance time, or when the management personnel issue a maintenance completion instruction to the system, the unit awaiting maintenance will resume outputting alarm and control signals.

15. An image recognition method for a working face human-machine collaborative operation safety monitoring and early warning system, based on the working face human-machine collaborative operation safety monitoring and early warning system as described in any one of claims 3 to 14, characterized in that, Includes the following steps: The video images captured by the camera device in the workface mobile equipment safety management system are used as training data, and the images of workers labeled with personnel information are used as training data labels. The improved YOLOv5 was trained and deep learned using training data and corresponding training data labels to obtain an image recognition model. By using a trained image recognition model, video images can be recognized to identify workers, track targets and predict their behavior, and identify the work environment.

16. The image recognition method of the working face human-machine collaborative operation safety monitoring and early warning system according to claim 15, characterized in that: The training data labels include images of staff in different poses, camera angles, and shooting categories, as well as facial images of staff in different states; Postures include: standing, squatting, bending over, standing on the left, and standing on the right; Shooting angles include: overhead long shot, overhead close shot, upward long shot, upward close shot, front long shot, and front close shot; The shooting categories include: visible light high-definition shooting, low-light supplementary lighting shooting and thermal imaging (27) shooting; The different states of the staff include: unprotected state, wearing a protective mask state, wearing a safety helmet state, wearing earplugs state, wearing earmuffs state, wearing a protective mask and safety helmet state, wearing a safety helmet and earplugs state, wearing a safety helmet and earmuffs state, wearing a protective mask and earplugs state, wearing a protective mask and earmuffs state, wearing a protective mask, safety helmet and earplugs state, and wearing a protective mask, safety helmet and earmuffs state.

17. The image recognition method of the working face human-machine collaborative operation safety monitoring and early warning system according to claim 15, characterized in that: The image recognition model adjusts the anchor boxes using the K-means clustering method. The optimal cluster centers of the K-means clustering method are obtained by dividing the training samples and calculating their fitness using a swarm intelligence optimization method.

18. The image recognition method of the working face human-machine collaborative operation safety monitoring and early warning system according to claim 17, characterized in that: The swarm intelligence optimization method is as follows: Initialize the particle swarm by randomly generating N particles, where 1 < N < n, and n is the number of anchor boxes to be classified. Use the N particles as cluster centers to divide the n anchor boxes. Calculate and normalize the distance from each sample data point to the cluster center, calculate and normalize the distance between any two cluster centers, and calculate the fitness of the sample data. The PSO method is used to optimize based on the fitness of the sample data to obtain the optimal cluster centers.

19. The image recognition method of the working face human-machine collaborative operation safety monitoring and early warning system according to claim 18, characterized in that: The method for calculating and normalizing the distance from each sample data point to the cluster center is as follows: Among them, D in ω represents the j-th cluster in the sample dataset. j The u-th sample data A u Distance from cluster center C j The distance, K is the number of cluster centers; Among them, D′ in Represents the normalized D in ,max(D in ), min(D in ) represent all D in The maximum and minimum values ​​in the range.

20. The image recognition method of the working face human-machine collaborative operation safety monitoring and early warning system according to claim 19, characterized in that: The method for calculating and normalizing the distance between any two cluster centers is as follows: d(ω i ,ω j )=|C i -C j | Among them, D out Let C be the distance between any two cluster centers. i and C j ω represents the i-th cluster i and the j-th cluster ω j The cluster center, N is the number of particles in the initial particle swarm, d(ω) i ,ω j ) is the cluster center C i and C j The absolute value of the difference, D out ′ is the normalized D out ,max(D out ),min(D out ) represent all D out The maximum and minimum values ​​in the range.

21. The image recognition method of the working face human-machine collaborative operation safety monitoring and early warning system according to claim 20, characterized in that: The fitness of the sample data is: Where F represents the fitness of the sample data.

22. The image recognition method of the working face human-machine collaborative operation safety monitoring and early warning system according to claim 15, characterized in that: After performing image recognition on each frame of the acquired video, the image recognition model also includes: Generate a video trajectory for each worker in the image; Based on video trajectory tracking and prediction, the system can assess the danger of workers when their positions are partially or completely obscured.

23. A wireless positioning method for a human-machine collaborative operation safety monitoring and early warning system at a work site, based on the human-machine collaborative operation safety monitoring and early warning system as described in any one of claims 3 to 14, characterized in that, Includes the following steps: Obtain the measurement data between the wireless positioning terminal (26) and the wireless positioning base station at time t. The position of the wireless positioning terminal (26) within the wireless coverage area of ​​the wireless positioning base station, with the wireless positioning base station as the origin, is (x...). t y t ), and obtain the distance data s between the wireless positioning terminal (26) and the wireless positioning base station; Perform a set number of differential operations on the distance data s to obtain the velocity v and acceleration a of the wireless positioning terminal (26); The velocity v at time t t acceleration a t The velocity v at time t-1 t-1 acceleration a t-1 Compare and judge: When the velocity v t acceleration a t When the change is within the set threshold range, the current position (x) of the wireless positioning terminal (26) is... t ,y t )efficient; Otherwise, based on the velocity v at time t-1 t-1 acceleration a t-1 The velocity v at time t-2 t-2 acceleration a t-2 Predict the predicted velocity v′ at time t t and predicted acceleration a′ t And based on the predicted velocity v′ t and predicted acceleration a′ t Calculate the predicted position (x′) of the wireless positioning terminal (26) at time t. t y′ t ).

Images