Coal mine equipment control system and control method based on gesture analysis

Abstract

The invention discloses a coal mine equipment control system and control method based on gesture analysis, and the system is characterized in that an RGB camera and a thermal infrared instrument are spliced left and right as a camera device; one camera device is installed on a fully mechanized coal mining face every six hydraulic supports, one camera device is installed on an equipment train every 10 meters, and one camera device is installed at the position, where fully mechanized coal mining equipment is arranged, in a mine laneway; a GPU computing power server and a control server are deployed in a control center; a UWB positioning tag is attached to each self-rescuer device, and a wireless button of a signal sending module is fixed to each UWB positioning tag; one UWB positioning base station is arranged every 200 meters in the mine; the UWB positioning base station communicates with the UWB positioning labels in the communication range of the UWB positioning base station, and the UWB positioning base station and the camera device are connected with the GPU computing power server and the control server through the switch. According to the invention, the manpower and labor force of equipment operating personnel can be reduced, the safety of the personnel is ensured, and intelligent control is realized.

Description

Technical Field

[0001] The invention relates to a coal mine equipment control system based on gesture analysis and a method for realizing equipment control by using the system. Background Art

[0002] At present, with the development of the economy, the country has put forward high-quality development as the primary task of our country, and has also introduced a series of cost-cutting and efficiency-enhancing policies to benefit enterprises. In the field of coal mining, with the continuous deepening of the national coal mine intelligent construction, there are more and more intelligent software and automation equipment in the ground transportation, underground tunneling, comprehensive mining, transportation, ventilation, security and other links of coal mines. In the process of coal mining, the operation between equipment is now generally done by workers through voice shouting equipment to let workers operate. Due to the long distance between the various coal mining equipment, the startup noise is loud, and the underground tunnels are winding. The workers can only operate the coal mining equipment through voice shouting to the workers at the corresponding equipment position. During the shouting process, due to the influence of environmental noise and the equipment's own noise, the workers hear the sound vaguely, and need to confirm it many times. After the operation, they need to shout that the equipment has been operated. This voice shouting method makes it difficult for workers to operate the equipment, and equipment operation errors lead to coal mining safety accidents, especially when the equipment is turned on incorrectly, and personnel are in dangerous areas such as scraper conveyors, transfers, etc., which is extremely easy to cause personnel dangerous accidents. This phenomenon is common underground.

[0003] The patent application with publication number CN104750397A proposes a natural interaction method for virtual mines based on somatosensory perception. The method uses Kinect sensors to obtain gesture signals, depth information, and skeleton information, and static gesture recognition through traditional algorithms such as histograms and contour tracking. Through continuous multi-frame images, the motion trajectory of the hand, wrist, elbow, and shoulder joints is dynamically identified according to the skeleton point information. However, the application has the following disadvantages:

[0004] 1. Kinect relies on infrared light and depth sensors to capture data and cannot capture information in strong or weak light underground.

[0005] 2. Kinect has a fixed viewing angle and effective detection range. When the user leaves the central field of view, it cannot obtain complete gesture data, and the accuracy of gesture recognition is reduced.

[0006] 3. Kinect is more effective in recognizing whole-body movements, but lacks the ability to recognize fine single-hand or single-finger gestures. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art, provide a coal mine equipment control system based on gesture analysis, and provide a method for controlling equipment using the above system. The control system and control method of the present invention can reduce the manpower and labor of equipment operators, ensure the safety of personnel, and realize intelligent control.

[0008] The object of the present invention is achieved through the following technical solutions: a coal mine equipment control system based on gesture analysis, including a wireless button, a UWB positioning tag, an RGB camera, a thermal infrared instrument and a GPU computing server; the RGB camera and the thermal infrared instrument are spliced ​​left and right to serve as a camera device;

[0009] A camera device is installed every 6 hydraulic supports on the fully-mechanized mining face, a camera device is installed every 10 meters on the equipment train, and a camera device is installed at the location where fully-mechanized mining equipment is installed in the mine tunnel; a GPU computing server and a control server are deployed in the control center; a UWB positioning tag is attached to each self-rescuer device, and a wireless button of a signal sending module is fixed to each UWB positioning tag; a UWB positioning base station is arranged every 200 meters in the mine;

[0010] The UWB positioning base station communicates with the UWB positioning tags within its communication range. The UWB positioning base station and the camera device are connected to the GPU computing server and the control server through a switch, and are also connected to the control center through a switch.

[0011] Another object of the present invention is to provide a coal mine equipment control method based on gesture analysis, which is implemented using the above control system and includes the following steps:

[0012] Step 1: After the personnel wear the self-rescuer equipment and go down the well, the personnel's position is located through the UWB positioning tag, and the video stream data and thermal infrared image of the personnel are collected through the camera device; after the personnel presses the wireless button, the signal sent by the signal sending module is transmitted to the GPU computing power server through the wireless network, and the GPU computing power server starts to perform feature fusion and gesture recognition on the RGB video stream and thermal infrared video stream, and analyzes the meaning of the gesture and the number of the control device;

[0013] Step 2: Record RGB camera C i IP address, location (X i , Y i , Z i ), optical parameters, and the width w and height h of the image sensor in the RGB camera; the real-time position of the person (x i ,y i , z i ) as input, the camera position as output, and establish a mapping relationship between the personnel position and the camera position;

[0014] Step 3: Use the yolov8 model to identify people in the RGB camera video stream, and use deepsort to continuously track people; predict the movement path of people as they move, calculate the camera that needs to be tracked next time in advance according to the mapping relationship in step 2, and design a multi-camera intelligent collaborative algorithm to achieve smooth switching between cameras;

[0015] Step 4: After the camera successfully identifies and tracks a person, the GPU computing server activates the gesture analysis module, accurately identifies gestures through the visual analysis model, and converts them into specific instructions;

[0016] Step 5: The instructions obtained by the GPU computing server are transmitted to the control server. The control server issues control instructions through coal mining technology and logic algorithms to realize intelligent control of coal mining equipment.

[0017] The beneficial effects of the present invention are:

[0018] 1. The present invention combines the current dangerous situation where personnel manually shout to control coal mining equipment, takes into account the existing infrastructure of coal mines, and innovatively proposes a method for controlling coal mining equipment based on gesture analysis without adding additional equipment.

[0019] 2. The present invention utilizes the combination of color cameras and thermal infrared instruments to monitor personnel, extracts hand features through feature-level fusion, monitors and analyzes personnel gestures in real time, perceives the safety and status of controlled equipment in real time, automatically controls the status of coal mine equipment in combination with the coal mine process scene, and gives voice feedback to operators. The present invention reduces the manpower and labor of equipment operators, ensures personnel safety, realizes intelligent control, and improves coal mining efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a structural schematic diagram of a coal mine equipment control system based on gesture analysis of the present invention;

[0021] Figure 2 is a flow chart of the control method of the present invention;

[0022] Figure 3 Schematic diagram of gesture types of the present invention. DETAILED DESCRIPTION

[0023] Based on the existing equipment in coal mines, the present invention integrates the control of each independent device or system into a fully intelligent control system for coal mines through the existing comprehensive mining and excavation control systems, video monitoring systems, light source systems and servers, etc., through personnel positioning, gesture analysis and other technologies combined with existing control system technologies, through logic control algorithms and fuzzy control algorithms, and reduces the problems of difficult control, difficult communication, and extreme danger of the operating system for staff. The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0024] like Figure 1 As shown, a coal mine equipment control system based on gesture analysis of the present invention includes a wireless button, a UWB (ultra-wideband) positioning tag, an RGB (color) camera, a thermal infrared instrument and a GPU computing server, a coal mining machine, a conveyor and other equipment to form a comprehensive mining equipment; the RGB camera and the thermal infrared instrument are spliced ​​on the left and right to serve as a camera device;

[0025] A camera device is installed every 6 hydraulic supports on the fully-mechanized mining face, a camera device is installed every 10 meters on the equipment train, and a camera device is installed at the location where fully-mechanized mining equipment is installed in the mine tunnel; a GPU computing server and a control server are deployed in the control center; a UWB positioning tag is attached to each self-rescuer device, and a wireless button of a signal sending module is fixed to each UWB positioning tag; a UWB positioning base station is arranged every 200 meters in the mine;

[0026] The UWB positioning base station communicates with the UWB positioning tags within its communication range. The UWB positioning base station and the camera device are connected to the GPU computing server and the control server through a switch, and are also connected to the control center through a switch.

[0027] Every time a person wears a self-rescuer device and goes down the mine, the person's position is located through the UWB positioning system, and the person is tracked in real time through the camera device. After the person presses the wireless button, the GPU computing power server performs feature fusion and gesture recognition on the RGB video stream and thermal infrared video stream, analyzes the meaning of the gesture and the number of the controlled device, perceives the surrounding environment of the controlled equipment, and feedbacks the safety status around the controlled equipment, thereby realizing closed-loop safety control of the coal mining equipment.

[0028] like Figure 2 As shown, the coal mine equipment control method based on gesture analysis provided by the present invention is implemented by using the above control system, and includes the following steps:

[0029] Step 1: After the personnel wear the self-rescuer equipment and go down the well, the UWB positioning tag is used to locate the personnel's position, and the camera device is used to collect the personnel's video stream data and thermal infrared images; after the personnel presses the wireless button, the signal sent by the signal sending module is transmitted to the GPU computing server through the wireless network, awakening the GPU computing server. The GPU computing server begins to perform feature fusion and gesture recognition on the RGB video stream and thermal infrared video stream, analyze the meaning of the gesture and the number of the controlled device, perceive the surrounding environment of the controlled device, and feedback the safety status around the controlled device.

[0030] Step 2: Record RGB camera C i IP address, location (X i , Y i , Z i ), optical parameters (field of view FOV, focal length f), and the width w and height h of the image sensor in the RGB camera; the real-time position of the person (x i ,y i , z i ) is input, and the camera position is output, and a mapping relationship between the personnel position and the camera position is established; the mapping relationship between the personnel position and the camera position is calculated as follows:

[0031] Step 2-1: First, select K cameras (C 1 …C K ) as candidate cameras;

[0032] Step 2-2: Calculate each candidate camera C i The horizontal field of view angle θ and the vertical field of view angle φ, where the horizontal field of view angle Vertical field of view f is the focal length of the RGB camera;

[0033] Step 2-3: Calculate the real-time position of the person and each candidate camera C i The horizontal field of view angle difference Δθ i and the vertical field of view angle difference Δφ i , where Δθ i =|arctan(y i -Y i , x i -X i )|,Δφ i =|arctan(z i -Z i ,sqrt((x i -X i ) 2 +yi-Yi2));

[0034] Step 2-4: Determine whether the person is within the camera's field of view: Determine whether the camera satisfies both If yes, it is determined that the person is within the field of view of the camera, and all candidate camera sets that meet the conditions are recorded (C 1 …C m );

[0035] Step 2-5: Select the camera for tracking specific personnel: Due to the harsh environment in coal mines, such as strong or dark light and large amounts of coal dust, the camera selection needs to be combined with the camera image quality Q i Therefore, the present invention decides to select the final camera by weighted summation of field of view angle difference and image quality, and the image quality Q i Determined by the entropy of the grayscale image; the overall camera score S i =w 1 (Δθ i +Δφ i )+w 2 Q i , select the camera with the largest total score as the camera for tracking people;

[0036] Step 2-6: Establish a functional relationship between personnel position and camera position according to the personnel position and the camera position.

[0037] Step 3: Identify and track people in real time: Use the yolov8 model to identify people in the RGB camera video stream, and use deepsort to continuously track people; predict the movement path of people as they move, calculate the camera that needs to be tracked next time in advance based on the mapping relationship in step 2, and design a multi-camera intelligent collaborative algorithm to achieve smooth switching between cameras, thereby ensuring that people are tracked at all times.

[0038] The multi-camera intelligent collaboration algorithm is implemented by predicting the path of people and switching the camera video streams in advance; the details are as follows:

[0039] Step 3-1: Receive the real-time location coordinates (x i ,y i , z i ), and input it as an observation into the Kalman filter to estimate the current state And predict the location of people in the next few seconds;

[0040] Step 3-2, input the personnel position predicted in step 3-1 into the functional relationship between personnel position and camera position, calculate the position of the next camera, and activate the personnel recognition and tracking algorithm of the camera in advance;

[0041] Step 3-3, when the person moves to the transition screen, that is, the person appears in the screen captured by the two cameras; the images of the two cameras are weighted averaged to obtain a weighted image, and the weighted image is subjected to a nonlinear interpolation algorithm to generate a transition video frame. When the position of the person changes, the camera screen is switched (as the position of the person changes, the camera also switches to the next camera) to ensure a smooth transition between cameras.

[0042] Step 4. After the camera successfully identifies and tracks the person, the GPU computing server activates the gesture analysis module. Once the person is locked, the camera will not only continue to monitor the person's movement trajectory, but also capture his hand movements in real time. The GPU computing server accurately identifies gestures through a visual analysis model and converts them into specific instructions; gesture analysis is implemented based on the yolov8 algorithm and gesture recognition algorithm. Currently published patent applications CN114898464B and CN118522036A are both based on RGB visual technology to achieve gesture recognition. The underground environment of coal mines is complex, and there are many situations such as excessive light, too dark light, large amounts of coal dust in the working environment, and coal dust blocking of camera lenses. At the same time, the underground environments of various coal mines are very complex and vary greatly, and the environmental differences of different coal mining faces in each coal mine are also large. In order to improve the accuracy and omissions of gesture recognition, the present invention is based on a combination of unsupervised algorithms and supervised algorithms, and adopts a method combining RGB images and thermal infrared images to achieve gesture recognition. The specific method is as follows:

[0043] Step 4-1: Use contrast-limited adaptive histogram equalization (CLAHE) to enhance local features of the thermal infrared image to make the human figure and hand features in the image more prominent. Then use the OpenPose tool to extract gesture key points in the thermal infrared image and RGB image respectively;

[0044] Step 4-2: CycleGAN is used to convert the heat map into an RGB image, and the RGB image converted from the heat map and the RGB image collected by the RGB camera are respectively subjected to feature extraction using a convolutional neural network (CNN) to obtain their respective feature maps;

[0045] Step 4-3: Weighted concatenation of the two feature maps in the channel dimension (i.e., the last dimension) of the feature map, and the gesture key points extracted in step 4-1 and the concatenated features Figure 1 The input convolution layer maps it to a high-dimensional space to achieve the splicing of key points and feature maps, and the high-dimensional data features are converted into one-dimensional data features through the fully connected layer;

[0046] Step 4-4: Input the one-dimensional data features obtained in step 4-3 into the Transformer to obtain gesture instructions.

[0047] The present invention designs 0-99 digital gestures to number coal mining equipment. Figure 3 As shown. At the same time, the functions of opening, pausing, accelerating, decelerating, closing, three-machine linkage, spray linkage, and gas linkage are designed. The opening is a fist gesture, the acceleration is an upward movement of the palm, the deceleration is an upward movement of the palm, the closing is an OK gesture, and the three-machine linkage is a digital gesture. At the same time, in the gesture input state, the selection of coal mining equipment is to select the corresponding equipment number by gesturing with both hands. When the left hand and the right hand gesture numbers at the same time, the current gesture represents a two-digit number, the left hand represents the tens digit, and the right hand represents the units digit; when the left hand gestures a number and the right hand gesture is invalid, the current gesture is invalid; when the right hand gestures a number and the left hand is invalid, the current gesture represents the units digit number. Specific gesture actions and control instructions can be adjusted as needed.

[0048] Step 5: The instructions obtained by the GPU computing server are transmitted to the control server. The control server issues control instructions through coal mining technology and logic algorithms to realize intelligent control of coal mining equipment.

[0049] Gesture control is implemented based on the control server and feedback mechanism. The steps are as follows:

[0050] Step 5-1, the GPU computing server transmits the gesture command to the control server via HTTP;

[0051] Step 5-2: After receiving the gesture command, the control server determines whether the gesture action comes from one camera device or multiple camera devices; when gesture commands appear simultaneously in multiple camera devices, the control server makes a decision based on the importance of each coal mining area and processes the gesture command in the following priority order: gesture command controlled device > fully mechanized mining face > control server > others; and notifies other personnel by voice that the device is working;

[0052] Step 5-3: While analyzing the source of the gesture, call the camera device at the controlled device to determine whether there is a person on or near the controlled device. If so, the gesture command is invalid and the operator is notified that there is someone on the device;

[0053] Step 5-4: After confirming that the area around the controlled device is safe, the camera device at the controlled device begins to automatically identify whether the machine is working normally; the control server begins to send control instructions, and the controlled device starts to work after receiving the control instructions; when the controlled device does not receive the control instruction or receives the control instruction but does not work normally, the control server will notify the operator that the device is not working normally.

[0054] The present invention combines the existing equipment control center, video monitoring system, lighting system and mine server equipment in the coal mine. No additional hardware equipment is needed. The implementation is simple and effective. It only needs to call the camera of the local operating computer, face recognition staff, identify staff authority, and control the local computer software through gesture recognition technology to realize the query and control of coal mine business. It provides a prerequisite for the subsequent intelligent control of important links such as comprehensive excavation, comprehensive mining, and transportation. It enables underground workers to operate equipment in real time and accurately, achieves the purpose of reducing costs and increasing efficiency, and improves the safety and convenience of personnel and equipment. It improves the sensitivity, safety, and accuracy of workers in operating equipment, and can ensure the safety between personnel and equipment.

[0055] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A coal mine equipment control system based on gesture analysis, characterized in that: It includes a wireless button, a UWB positioning tag, an RGB camera, a thermal infrared instrument, and a GPU computing server; the RGB camera and the thermal infrared instrument are spliced ​​left and right to form a camera device; A camera device is installed every 6 hydraulic supports on the fully-mechanized mining face, a camera device is installed every 10 meters on the equipment train, and a camera device is installed at the location where fully-mechanized mining equipment is installed in the mine tunnel; a GPU computing server and a control server are deployed in the control center; a UWB positioning tag is attached to each self-rescuer device, and a wireless button of a signal sending module is fixed to each UWB positioning tag; a UWB positioning base station is arranged every 200 meters in the mine; The UWB positioning base station communicates with the UWB positioning tags within its communication range. The UWB positioning base station and the camera device are connected to the GPU computing server and the control server through a switch.

2. A coal mine equipment control method based on gesture analysis, implemented by the control system according to claim 1, characterized in that: The following steps are involved: Step 1: After the personnel wear the self-rescuer equipment and go down the well, the personnel's position is located through the UWB positioning tag, and the video stream data and thermal infrared image of the personnel are collected through the camera device; after the personnel presses the wireless button, the signal sent by the signal sending module is transmitted to the GPU computing power server through the wireless network, and the GPU computing power server starts to perform feature fusion and gesture recognition on the RGB video stream and thermal infrared video stream; Step 2: Record RGB camera C i IP address, location (X i ,Y i ,Z i ), optical parameters, and the width w and height h of the image sensor in the RGB camera; the real-time position of the person (x i ,y i ,z i ) as input, the camera position as output, and establish a mapping relationship between the personnel position and the camera position; Step 3: Use the yolov8 model to identify people in the RGB camera video stream, and use deepsort to continuously track people; As the personnel move, their movement paths are predicted, and the next camera to be tracked is calculated in advance based on the mapping relationship in step 2. A multi-camera intelligent collaborative algorithm is designed to achieve smooth switching between cameras. Step 4: After the camera successfully identifies and tracks a person, the GPU computing server activates the gesture analysis module, accurately identifies gestures through the visual analysis model, and converts them into specific instructions; Step 5: The instructions obtained by the GPU computing server are transmitted to the control server. The control server issues control instructions through coal mining technology and logic algorithms to realize intelligent control of coal mining equipment.

3. A coal mine equipment control method based on gesture analysis according to claim 2, characterized in that: In step 2, the mapping relationship between the personnel position and the camera position is calculated as follows: Step 2-1: First, select K cameras (C1…C K ) as candidate cameras; Step 2-2: Calculate each candidate camera C i The horizontal field of view angle θ and the vertical field of view angle φ, where the horizontal field of view angle Vertical field of view f is the focal length of the RGB camera; Step 2-3: Calculate the real-time position of the person and each candidate camera C i The horizontal field of view angle difference Δθ i and the vertical field of view angle difference Δφ i , where Δθ i =|arctan(y i -Y i ,x i -X i )|,Δφ i =|arctan(z i -Z i ,sqrt((x i -X i ) 2 +(y i -Y i ) 2 ))|; Step 2-4: Determine whether the person is within the camera's field of view: Determine whether the camera satisfies both If yes, it is determined that the person is within the field of view of the camera, and all candidate camera sets that meet the conditions are recorded (C1…C m ); Step 2-5: Select the camera for tracking specific people: The final camera is selected by weighted summation of field of view angle difference and image quality, and the image quality Q i Determined by the entropy of the grayscale image; the overall camera score S i =w1(Δθ i +Δφ i )+w2Q i , select the camera with the largest total score as the camera for tracking people; Step 2-6: Establish a functional relationship between personnel position and camera position according to the personnel position and the camera position.

4. A coal mine equipment control method based on gesture analysis according to claim 2, characterized in that: In step 3, the multi-camera intelligent collaboration algorithm is implemented by predicting the path of the person and switching the camera video stream in advance; the details are as follows: Step 3-1: Receive the real-time location coordinates (x i ,y i ,z i ), and input it as an observation into the Kalman filter to estimate the current state And predict the location of people in the next few seconds; Step 3-2, input the personnel position predicted in step 3-1 into the functional relationship between personnel position and camera position, calculate the position of the next camera, and activate the personnel recognition and tracking algorithm of the camera in advance; Step 3-3, when the person moves to the transition screen, that is, the person appears in the screen captured by the two cameras; the images of the two cameras are weighted averaged to obtain a weighted image, and a nonlinear interpolation algorithm is used to generate a transition video frame for the weighted image. When the position of the person changes, the camera screen is switched.

5. The coal mine equipment control method based on gesture analysis according to claim 2 is characterized in that: The specific method in step 4 is as follows: Step 4-1: Use contrast-limited adaptive histogram equalization to enhance local features of the thermal infrared image, and then use the OpenPose tool to extract gesture key points in the thermal infrared image and RGB image respectively; Step 4-2: Use CycleGAN to convert the heat map into an RGB image, and use a convolutional neural network to extract features from the RGB image converted from the heat map and the RGB image collected by the RGB camera to obtain their respective feature maps; Step 4-3: weighted concatenation of the two feature maps, input the gesture key points extracted in step 4-1 and the concatenated feature map into the convolution layer, map them to a high-dimensional space to concatenate the key points and feature map, and convert the high-dimensional data features into one-dimensional data features through a fully connected layer; Step 4-4: Input the one-dimensional data features obtained in step 4-3 into the Transformer to obtain gesture instructions.

6. A coal mine equipment control method based on gesture analysis according to claim 2, characterized in that: In step 5, gesture control is implemented based on the control server and feedback mechanism, the steps are as follows: Step 5-1, the GPU computing server transmits the gesture command to the control server via HTTP; Step 5-2: After receiving the gesture command, the control server determines whether the gesture action comes from one camera device or multiple camera devices; when gesture commands appear simultaneously in multiple camera devices, the control server makes a decision based on the importance of each coal mining area and processes the gesture command in the following priority order: gesture command controlled device > fully mechanized mining face > control server > others; and notifies other personnel by voice that the device is working; Step 5-3: While analyzing the source of the gesture, call the camera device at the controlled device to determine whether there is a person on or near the controlled device. If so, the gesture command is invalid and the operator is notified that there is someone on the device; Step 5-4: After confirming that the area around the controlled device is safe, the camera device at the controlled device automatically identifies whether the device is working properly; the control server starts sending control instructions, and the controlled device starts working after receiving the control instructions; When the controlled device does not receive the control instruction or receives the control instruction but does not work normally, the control server will notify the operator that the device is not working normally.

Images