Mine mechanical arm safety management method and mechanical arm safety management system

Abstract

The invention provides a mine mechanical arm safety management method and a mechanical arm safety management system. A plurality of infrared cameras are used for collecting environment videos around a mechanical arm in real time; inputting the environment video into a target detection model, and outputting a bounding box of the target person; calculating the actual distance between the target person and the infrared camera according to the size of the bounding box of the target person and the actual size of the person; determining the azimuth angle of the target person according to the position coordinates of the target person and the position coordinates of the infrared camera; in combination with the actual distance between the target person and the infrared camera and the azimuth angle of the target person, the actual position of the target person is determined, the current area where the target person is located is determined, different processing schemes are adopted for different areas, the actual positions of surrounding persons are automatically detected and tracked by the mechanical arm under the application scene of insufficient light, and the working efficiency is improved. When a dangerous condition occurs, different processing schemes are adopted for different areas so as to achieve the purpose of active safety.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical arms, and in particular to a mine mechanical arm safety management method and a mechanical arm safety management system. Background Art

[0002] A robotic arm is an intelligent and automated device that mimics the structure and movements of a human arm. It is widely used in all walks of life because of its flexibility, high precision, high efficiency, programmable control, and ease of automation. It has very broad application prospects and development trends in the future. However, safety accidents often occur due to the lack of safety awareness of operators in the working area of ​​the robotic arm. Due to the fast movement speed and strong impact force of the robotic arm, people entering its range of motion may suffer from collisions, pinching and other injuries. It is difficult for humans to determine whether there are people within the working range of the robotic arm, especially for application scenarios with insufficient light such as mine tunnels, which are more likely to cause personal injury. Summary of the invention

[0003] The present invention provides a mine mechanical arm safety management method and a mechanical arm safety management system, which are used to solve the defects that it is difficult to judge the safety of the mechanical arm working area manually in the traditional way, which is easy to cause personal injury.

[0004] The present invention provides a mine mechanical arm safety management method, comprising: Use multiple infrared cameras to collect real-time video of the environment around the robotic arm; Input the environment video into the target detection model and output the bounding box of the target person; Calculating the actual distance between the target person and the infrared camera according to the size of the bounding box of the target person and the actual size of the person; Determine the azimuth of the target person according to the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera; determine the actual position of the target person in combination with the actual distance between the target person and the infrared camera and the azimuth of the target person; The area where the current target person is located is confirmed according to the actual location of the target person, and different processing solutions are adopted for different areas.

[0005] According to the mine mechanical arm safety management method provided by the present invention, after using multiple infrared cameras to collect environmental videos around the mechanical arm in real time, it also includes: The environmental video is preprocessed, and the preprocessing includes at least one of denoising, image enhancement, frame selection and synchronization, key frame extraction, data standardization, and filtering processing.

[0006] According to the mine robot arm safety management method provided by the present invention, the target detection model is trained based on video data collected from multiple specific application scenarios of the robot arm.

[0007] According to the mine mechanical arm safety management method provided by the present invention, the determining the azimuth of the target person according to the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera includes: The center of mass coordinates of the target person in the environmental video are converted from the image coordinate system to the world coordinate system using the internal and external parameters of the infrared camera. In the world coordinate system, the azimuth of the target object is determined according to the position coordinates of the target object and the position coordinates of the infrared camera.

[0008] According to the mine manipulator safety management method provided by the present invention, the area includes a safe area, a low-risk area, a high-risk area and a safety island. The area where the current target person is located is confirmed according to the actual position of the target person, and different processing solutions are adopted for different areas, including: If the actual location of the target person does not send an alarm message within the safety area; If the actual location of the target person is within the low-risk area, a first alarm sound is sent and a yellow light is controlled to light up; If the actual location of the target person is within the high-risk area, a second alarm sound is sent, a red light is controlled to light up, and a robot arm stop command is issued to control the robot arm to stop moving; If the actual position of the target person is within the safety island, the action range of the robotic arm is controlled so that the joints of the robotic arm will not enter the safety island.

[0009] According to the mine mechanical arm safety management method provided by the present invention, the safety area includes a safety island, and the safety island range acquisition method includes: Obtaining the predicted position and movement range of the target person according to the mine operation scene; The safety island range is confirmed according to the movement range of the target person and the predicted position, and the safety island range includes a ground range and a height range.

[0010] The present invention provides a robotic arm safety management system, comprising: active safety controller and multiple infrared cameras; The active safety controller includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the mine mechanical arm safety management method as described in any one of the above when executing the program; The multiple infrared cameras use multiple infrared cameras to collect environmental videos around the robotic arm in real time; The active safety controller is used to input the environmental video into a target detection model and output a bounding box of a target person; calculate the actual distance between the target person and the infrared camera based on the size of the bounding box of the target person and the actual size of the person; determine the azimuth of the target person based on the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera; determine the actual position of the target person in combination with the actual distance between the target person and the infrared camera and the azimuth of the target person; confirm the area where the target person is currently located based on the actual position of the target person, and adopt different processing solutions for different areas.

[0011] According to the robotic arm safety management system provided by the present invention, the number of the multiple infrared cameras is 3, and the angle between two adjacent infrared cameras is 120°.

[0012] According to the present invention, a robot arm safety management system is provided, which also includes: A near-infrared fill light is used to emit near-infrared light to supplement the light around the robotic arm.

[0013] The robotic arm safety management system provided by the present invention also includes: An alarm device is connected to the active safety controller and is used to issue different sound alarms and light alarm messages when the target person is in low-risk and high-risk areas.

[0014] The mine manipulator safety management method and manipulator safety management system provided by the present invention use multiple infrared cameras to collect environmental videos around the manipulator in real time; input the environmental videos into a target detection model and output a boundary box of a target person; calculate the actual distance between the target person and the infrared camera according to the size of the boundary box of the target person and the actual size of the person; determine the azimuth of the target person according to the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera; determine the actual position of the target person in combination with the actual distance between the target person and the infrared camera and the azimuth of the target person; confirm the area where the current target person is located according to the actual position of the target person, and adopt different processing solutions for different areas, so that the manipulator can automatically detect and track the actual positions of surrounding personnel in application scenarios with insufficient light, and determine the positions. When a dangerous situation occurs, different processing solutions are adopted for different areas to achieve the purpose of active safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 This is one of the flow charts of the mine mechanical arm safety management method provided by the embodiment of the present invention; Figure 2 This is the second flow chart of the mine mechanical arm safety management method provided by the embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a robotic arm safety management system provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the distribution of infrared cameras provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Figure 1 A flowchart of a mine mechanical arm safety management method provided by an embodiment of the present invention, such as Figure 1 As shown, the mine mechanical arm safety management method provided by the embodiment of the present invention includes: Step 101: Use multiple infrared cameras to collect real-time environmental video around the robotic arm; Step 102: Input the environment video into the target detection model and output the bounding box of the target person; Step 103, calculating the actual distance between the target person and the infrared camera according to the size of the boundary box of the target person and the actual size of the person; Step 104: determining the azimuth of the target person according to the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera; determining the actual position of the target person according to the actual distance between the target person and the infrared camera and the azimuth of the target person; Step 105: confirm the area where the current target person is located according to the actual location of the target person, and adopt different processing solutions for different areas.

[0019] The traditional safety management method for mine robotic arms is to determine whether there are people within the working range of the robotic arm through manpower. This determination method is likely to cause personal injury in application scenarios with insufficient light, such as mine tunnels.

[0020] The mine manipulator safety management method provided by the embodiment of the present invention uses multiple infrared cameras to collect environmental videos around the manipulator in real time; inputs the environmental video into a target detection model to output a boundary box of a target person; calculates the actual distance between the target person and the infrared camera according to the size of the boundary box of the target person and the actual size of the person; determines the azimuth of the target person according to the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera; determines the actual position of the target person in combination with the actual distance between the target person and the infrared camera and the azimuth of the target person; confirms the area where the current target person is located according to the actual position of the target person, and adopts different processing solutions for different areas, so that the manipulator can automatically detect and track the actual positions of surrounding personnel in application scenarios with insufficient light, and judge the positions. When a dangerous situation occurs, different processing solutions are adopted for different areas to achieve the purpose of active safety.

[0021] Based on any of the above embodiments, the mine mechanical arm safety management method provided by the embodiment of the present invention includes: Step 201: Use multiple infrared cameras to collect real-time environmental video around the robotic arm; Step 202: preprocess the environment video, wherein the preprocessing includes at least one of denoising, image enhancement, frame selection and synchronization, key frame extraction, data standardization, and filtering processing.

[0022] In the embodiment, after the source of the infrared camera is transmitted to the controller, data preprocessing is first performed. The functions of preprocessing include: noise removal, image enhancement, frame selection and synchronization, key frame extraction, data standardization, filtering processing, etc. Preprocessing can improve the quality of video data and the accuracy of subsequent target detection algorithms.

[0023] Step 203: input the preprocessed environment video into the target detection model, and output the bounding box of the target person; In an embodiment of the present invention, a pre-processed image is received, and a deep learning-based target detection algorithm is used to identify, classify, and track key targets that appear in the video. According to the specific application scenario of the robotic arm, a specific video data set can be collected, and then specific training can be performed to adapt to the application scenario and have a higher accuracy. The target detection outputs the bounding box of the target object as the basis for subsequent processing.

[0024] Step 204, calculating the actual distance between the target person and the infrared camera according to the size of the boundary box of the target person and the actual size of the person; In an embodiment of the present invention, the bounding box or feature points of the object output by the target detection model are used to calculate the distance using triangulation or the like according to the actual size of the object and the size in the image in combination with the internal and external parameters of the camera.

[0025] Step 205: determining the azimuth of the target person according to the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera; determining the actual position of the target person according to the actual distance between the target person and the infrared camera and the azimuth of the target person; Step 206: confirm the area where the current target person is located according to the actual location of the target person, and adopt different processing solutions for different areas.

[0026] In an embodiment of the present invention, the target detection model is trained based on video data collected from multiple specific application scenarios of the robotic arm.

[0027] In the embodiment of the present invention, determining the azimuth of the target person according to the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera includes: The center of mass coordinates of the target person in the environmental video are converted from the image coordinate system to the world coordinate system using the internal and external parameters of the infrared camera. In the world coordinate system, the azimuth of the target object is determined based on the position coordinates of the target object and the position coordinates of a reference point (such as the camera position).

[0028] In the embodiment of the present invention, the centroid coordinates of the target object are converted from the image coordinate system to the world coordinate system using the internal and external parameters of the infrared camera. In the world coordinate system, the azimuth of the target object is determined based on the position coordinates of the target object and the position coordinates of the reference point (such as the camera position); and then the position of the target is determined by combining the distance information of the distance detection module.

[0029] In the embodiment of the present invention, the area includes a safe area, a low-risk area, a high-risk area and a safe island. The area where the current target person is located is confirmed according to the actual location of the target person, and different processing solutions are adopted for different areas, including: If the actual location of the target person does not send an alarm message within the safety area; If the actual location of the target person is within the low-risk area, a first alarm sound is sent and a yellow light is controlled to light up; If the actual location of the target person is within the high-risk area, a second alarm sound is sent, a red light is controlled to light up, and a robot arm stop command is issued to control the robot arm to stop moving; If the actual position of the target person is within the safety island, the action range of the robotic arm is controlled so that the joints of the robotic arm will not enter the safety island.

[0030] Different controls and warnings are adopted for different risks to provide security.

[0031] The embodiment of the present invention can determine the possibility of the target object being harmed based on its position information, thereby making a risk judgment. The safe area, low-risk area, and high-risk area of ​​the robotic arm are determined in advance based on the range of motion of the robotic arm, special needs, and other information. When the target object is in different risk areas, the system makes a risk level assessment, and sends out different sound and light information for low-risk and high-risk areas respectively. When a high-risk situation occurs, a robotic arm shutdown command is issued to control the robotic arm to stop moving to avoid the occurrence of harm.

[0032] In an embodiment of the present invention, the safety area includes a safety island, and the method for obtaining the range of the safety island includes: Obtaining the predicted position and movement range of the target person according to the mine operation scene; The safety island range is confirmed according to the movement range of the target person and the predicted position, and the safety island range includes a ground range and a height range.

[0033] For the underground operation environment of coal mines, manual operation of the robot arm is often required to complete semi-automatic action work, and due to the limitation of the operation space, the robot arm operator is usually required to operate within the range of the robot arm's movement. In view of this situation, by setting a safety island, the safety island is a safe interval, the various joints of the robot arm will not enter the safe interval, and the alarm system will not be triggered when the operator operates in the safety island. In some embodiments of the present invention, the method for obtaining the safety island range also includes: a pre-editing method: that is, one or more safety islands are divided around the robot arm by means of coordinate definition, the position of the operator is determined by the positioning device on the operator, and a certain range is expanded around the position as a safety island, and the safety island can be automatically updated according to the position of the operator.

[0034] The mine robotic arm safety management method provided by the embodiment of the present invention can actively avoid accidents or injuries in low-light scenarios, cover a 360° detection range, take different control and alarm processing for different risks, and provide a safety island mechanism for special underground operation scenarios. Operators can complete close-range auxiliary operations on the safety island without being accidentally injured by the robotic arm. At the same time, the alarm system will not be triggered, thereby further improving mine operation safety.

[0035] The robot arm safety management system provided by the present invention is described below. The robot arm safety management system described below and the mine robot arm safety management method described above can be referenced to each other.

[0036] Figure 3 A schematic diagram of the structure of a robotic arm safety management system provided by an embodiment of the present invention, such as Figure 3 As shown, the robotic arm safety management system provided by the embodiment of the present invention includes: Active safety controller and multiple infrared cameras (not shown); The active safety controller includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the mine mechanical arm safety management method as described in any one of the above when executing the program; The multiple infrared cameras use multiple infrared cameras to collect environmental videos around the robotic arm in real time; The active safety controller is used to input the environmental video into a target detection model and output a bounding box of a target person; calculate the actual distance between the target person and the infrared camera based on the size of the bounding box of the target person and the actual size of the person; determine the azimuth of the target person based on the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera; determine the actual position of the target person in combination with the actual distance between the target person and the infrared camera and the azimuth of the target person; confirm the area where the target person is currently located based on the actual position of the target person, and adopt different processing solutions for different areas.

[0037] In an embodiment of the present invention, the robot arm safety management system further includes: a near-infrared fill light, which is used to emit near-infrared light to supplement the light around the robot arm.

[0038] In an embodiment of the present invention, the robotic arm safety management system further includes: an alarm device, which is connected to the active safety controller and is used to issue different sound alarms and light alarm messages when the target personnel are in low-risk and high-risk areas.

[0039] The near-infrared camera collects real-time video around the robot arm for personnel detection and tracking; the near-infrared fill light emits near-infrared light to supplement the surrounding light to improve the camera's imaging effect and environmental adaptability. The video signal of the infrared camera is input into the active safety controller, which completes tasks such as video preprocessing, target detection, position determination, safety judgment, alarm, and control. The robot arm control is the original controller of the robot arm, which maintains communication with the active safety controller to receive safety instructions from the active safety controller in real time. After receiving the alarm command, the sound and light alarm system issues a warning message through sound, light, etc.

[0040] In the embodiment of the present invention, the number of the plurality of infrared cameras is 3, and the angle between two adjacent infrared cameras is 120°.

[0041] A near-infrared camera is a camera device that can shoot in total darkness or in low-light conditions, and is suitable for use in mines. It uses near-infrared technology for imaging, and can obtain image information that is invisible to the human eye, and is usually used in night vision, monitoring, and security. The spectral range that a near-infrared camera can capture is usually 700-1000 nanometers. The characteristic of this spectral range is that the strong thermal radiation in the far-infrared band is filtered out, thereby reducing background noise and various interferences. This enables near-infrared cameras to provide clearer images and reduce the impact of interference factors on image quality. In addition, near-infrared cameras can not only reflect the color and shape of objects, but also display a variety of optical properties of objects, including transparency, thickness, surface roughness, and surface spectral reflectance characteristics.

[0042] The near-infrared camera and near-infrared fill light are arranged on the base of the robot arm. To cover the 360° environment around the robot arm, the system includes three sets of near-infrared cameras and near-infrared fill lights: near-infrared camera 1 + fill light 1, near-infrared camera 2 + fill light 2, and near-infrared camera 3 + fill light 3. Figure 4 As shown, the three groups of cameras and fill lights are distributed at equal angles, and the azimuth angle between each group is 120°.

[0043] The dynamic generation device of litigation documents provided by the embodiment of the present invention is an active safety system for mine manipulators based on near-infrared technology, which enables the manipulator to automatically detect and track the actions of surrounding personnel in application scenarios with insufficient light. When a dangerous situation occurs, an audible and visual alarm is issued, and the manipulator is controlled to stop moving, so as to achieve the purpose of active safety. The solution includes a near-infrared camera, a near-infrared fill light, an active safety controller, an audible and visual alarm system, etc.

[0044] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0045] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mine mechanical arm safety management method, characterized in that: include: Use multiple infrared cameras to collect real-time video of the environment around the robotic arm; Input the environment video into the target detection model and output the bounding box of the target person; Calculating the actual distance between the target person and the infrared camera according to the size of the bounding box of the target person and the actual size of the person; Determine the azimuth of the target person according to the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera; determine the actual position of the target person in combination with the actual distance between the target person and the infrared camera and the azimuth of the target person; The area where the current target person is located is confirmed according to the actual location of the target person, and different processing solutions are adopted for different areas.

2. The mine mechanical arm safety management method according to claim 1, characterized in that: After using multiple infrared cameras to collect the environment video around the robotic arm in real time, it also includes: The environmental video is preprocessed, and the preprocessing includes at least one of denoising, image enhancement, frame selection and synchronization, key frame extraction, data standardization, and filtering processing.

3. The mine mechanical arm safety management method according to claim 1, characterized in that: The target detection model is trained based on video data collected from multiple specific application scenarios of the robotic arm.

4. The mine mechanical arm safety management method according to claim 1, characterized in that: Determining the azimuth of the target person according to the position coordinates of the target person in the environment video and the position coordinates of the infrared camera includes: The center of mass coordinates of the target person in the environmental video are converted from the image coordinate system to the world coordinate system using the internal and external parameters of the infrared camera. In the world coordinate system, the azimuth of the target object is determined according to the position coordinates of the target object and the position coordinates of the infrared camera.

5. The mine mechanical arm safety management method according to claim 1, characterized in that: The areas include safe areas, low-risk areas, high-risk areas and safe islands. The area where the current target person is located is determined according to the actual location of the target person, and different treatment plans are adopted for different areas, including: If the actual location of the target person does not send an alarm message within the safety area; If the actual location of the target person is within the low-risk area, a first alarm sound is sent and a yellow light is controlled to light up; If the actual location of the target person is within the high-risk area, a second alarm sound is sent, a red light is controlled to light up, and a robot arm stop command is issued to control the robot arm to stop moving; If the actual position of the target person is within the safety island, the action range of the robotic arm is controlled so that the joints of the robotic arm will not enter the safety island.

6. The mine mechanical arm safety management method according to claim 5, characterized in that: The safety area includes a safety island, and the method for obtaining the range of the safety island includes: Obtaining the predicted position and movement range of the target person according to the mine operation scene; The safety island range is confirmed according to the movement range of the target person and the predicted position, and the safety island range includes a ground range and a height range.

7. A robotic arm safety management system, characterized in that: include: active safety controller and multiple infrared cameras; The active safety controller includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the mine mechanical arm safety management method according to any one of claims 1 to 6 when executing the program; The multiple infrared cameras use multiple infrared cameras to collect environmental videos around the robotic arm in real time; The active safety controller is used to input the environmental video into a target detection model and output a bounding box of a target person; The actual distance between the target person and the infrared camera is calculated according to the size of the target person's boundary box and the actual size of the person; the azimuth of the target person is determined according to the position coordinates of the target person in the environmental video and the position coordinates of the infrared camera; the actual position of the target person is determined by combining the actual distance between the target person and the infrared camera and the azimuth of the target person; the area where the target person is currently located is confirmed according to the actual position of the target person, and different processing solutions are adopted for different areas.

8. The robot arm safety management system according to claim 7, characterized in that: The number of the multiple infrared cameras is 3, and the angle between two adjacent infrared cameras is 120°.

9. The robot arm safety management system according to claim 7, characterized in that: Also includes: A near-infrared fill light is used to emit near-infrared light to supplement the light around the robotic arm.

10. The robot arm safety management system according to claim 7, characterized in that: Also includes: An alarm device is connected to the active safety controller and is used to issue different sound alarms and light alarm messages when the target person is in low-risk and high-risk areas.

Images