A human-machine safety collaboration online monitoring method and system based on RGB-D camera

Through the online monitoring method of human-machine safety collaboration based on RGB-D cameras, using augmented reality tags and deep image data to detect operator 3D skeletons and point clouds, the problems of safety and operational normativeness in human-machine coordinated operation are solved, and the problem of safety and operational normativeness is realized. Effective monitoring of the human-computer collaboration process has been achieved, reducing the product defect rate.

CN113971800BActive Publication Date: 2024-12-06SHENYANG INST OF AUTOMATION - CHINESE ACAD OF SCI
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202010708966.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-22
Publication Date
2024-12-06
Estimated Expiration
2040-07-22

AI Technical Summary

Technical Problem

Existing automated operation equipment cannot effectively ensure the safety of workers in the coordinated operation of human-machine, and cannot identify and correct irregular operation behaviors, resulting in a decline in product quality.

Method used

Using the human-machine safety collaborative online monitoring method based on RGB-D camera, position calibration is performed through augmented reality tags, depth image data is captured, operator 3D skeleton data and body point clouds are extracted, joint position relationship is detected, work status is judged, and the Alerts in dangerous situations.

Benefits of technology

It realizes low-cost dual monitoring, ensures operator safety, reduces product defect rate, is suitable for most human-machine collaboration scenarios, and is highly versatile.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113971800B_ABST
    Figure CN113971800B_ABST
Patent Text Reader

Abstract

The present invention discloses an online monitoring method and system for safe human-machine collaboration based on an RGB-D camera. By dividing and establishing monitoring areas in the camera space, the position of the operator in the workspace is tracked based on 3D skeleton tracking technology and RGB-D images, and the safety and operational standardization are judged and monitored online according to the operating procedures and the dynamic position relationship between the operator and each monitoring area. The positions of each area of ​​the workspace are calibrated using ArUco augmented reality tags, and the coordinate range in the camera space is calculated by coordinate transformation to establish a virtual monitoring area. The joint data is processed using outlier removal and Holt's two-parameter exponential smoothing algorithm, and the 3D point cloud of the operator is extracted using background removal, point cloud conversion and filtering algorithms. The online position tracking of the operator in the workspace is achieved by combining the joint and point cloud data. The present invention has low cost and strong applicability, and can be widely used in online monitoring scenarios of human-machine collaboration.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of human-machine collaboration, and in particular relates to an online monitoring method and system for human-machine safe collaboration based on an RGB-D camera. Background Art

[0002] With the development of automation technology, the manufacturing process of products in various industries has gradually transitioned from pure manual state to semi-automatic or fully automated state. However, at the current stage, the dexterity and intelligence of most automated operation equipment are still far behind that of humans, and it is difficult for many industries to achieve full automation. Therefore, it has become a trend to adopt a semi-automatic manufacturing mode of human-machine collaborative operation in various industries. However, automated machinery is usually highly dangerous. In the case of irregular operation of operators and no safety measures, the safety of workers in the process of human-machine collaborative operation cannot be guaranteed; secondly, irregular operation behavior usually leads to a decline in product quality, such as defective products. In recent years, with the growth of demand for human-machine collaboration, some vision-based safety monitoring equipment has begun to appear (such as Pilz SafetyEYE). However, these devices are usually expensive and inflexible in configuration. Secondly, they have a single function and cannot identify irregular operation behaviors of operators, making it difficult to meet the multiple needs of the industry. Summary of the invention

[0003] The purpose of the present invention is to address the above problems and provide an online monitoring method and system for human-machine safe collaboration based on an RGB-D camera (such as Kinect), so as to achieve dual monitoring of operator safety and operational standardization in the process of human-machine collaboration at a low cost.

[0004] In order to achieve the above-mentioned object, the present invention provides a method and system for online monitoring of human-machine safety collaboration based on an RGB-D camera.

[0005] A human-machine safety collaborative online monitoring method based on RGB-D camera, comprising:

[0006] Divide the workspace of the human-machine collaborative task into different operation areas, and use augmented reality tags to calibrate the positions of the divided areas.

[0007] The position coordinates of each tag are captured by an RGB-D camera to determine the position of each operation area in the camera space. The coordinate range in the camera space is calculated based on the space occupied by each area to establish a virtual operation area. A virtual monitoring area is established outside the dangerous operation area based on the minimum safety distance between humans and machines.

[0008] An RGB-D camera is used to collect depth image data to obtain a background depth image of the workspace excluding the operator;

[0009] Collect the on-site depth image, extract the operator's 3D skeleton data, remove the workspace background, and obtain the operator's body point cloud of position information in the camera space;

[0010] The operator's 3D skeleton data and body point cloud data are extracted and further processed; the execution status of the processing procedure is determined by detecting the dynamic position relationship between each joint and each virtual operation area in the 3D skeleton data; and whether the operator approaches the machine while the machine is operating is determined by detecting whether the operator's body point cloud intrudes into the danger monitoring area, and an alarm is given.

[0011] The operation areas are divided into a raw material acquisition area, a raw material processing area, an operator operation area, and a machine operation area that can be entered by an operator.

[0012] ArUco augmented reality tags are used for position calibration in different operating areas.

[0013] The background removal is to perform pixel-by-pixel subtraction between the collected scene depth image and the pre-stored workspace background depth image to obtain a depth map containing only the operator.

[0014] The PCL point cloud library is used to convert the operator's depth map into a 3D point cloud, which is then further filtered and clustered into body point cloud data.

[0015] The extracted joint data are subjected to outlier removal and smoothing processing; the outlier removal processing is that if the position change of the same joint point within the time interval of two consecutive frames of image data is not within a preset range, then this joint point is an outlier; the smoothing processing is the Holt dual-parameter exponential algorithm.

[0016] The determination of the execution status of the processing step is used to identify whether the operator is working in the area specified by the operation specification, whether the operator misses or takes too much raw material, and whether the raw materials are processed in the specified order.

[0017] The execution status of the processing procedure is determined by detecting the position relationship between the joint data in the operator's 3D skeleton data and each area of ​​the virtual operation area, including:

[0018] When the head joint, neck joint or shoulder-spine joint is always within the operator's working area during the collaboration phase, it is judged that the operator is working in the designated area;

[0019] When the operator's hand joint enters the raw material acquisition area, it is judged that the operator picks up the raw material in the current raw material acquisition area. If the operator enters the area repeatedly, it is judged as repeated picking.

[0020] When the hand joint enters the raw material processing area, it is judged that the operator is performing processing operations in the raw material processing area.

[0021] A human-machine safety collaboration online monitoring system based on an RGB-D camera comprises: an industrial control computer and an RGB-D camera communicating therewith, a warning module, a reset button, and a control system of an automated machine that collaborates with humans; the RGB-D depth camera is used to collect depth images of an actual workspace of a human-machine collaboration task, the warning module is used to alarm when a processing error occurs or involves personal danger, and the reset button is used to press the reset button to continue detection after the operator leaves the danger zone; the industrial control computer stores a program, and when the program is loaded, the steps of a human-machine safety collaboration online monitoring method based on an RGB-D camera as described in claims 1 to 8 are executed to determine the execution status of the processing procedure and to give an alarm when the operator enters a danger monitoring zone.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0023] 1. The present invention can use low-cost RGB-D cameras to complete online monitoring of the human-machine collaboration process, ensuring the safety of human operators;

[0024] 2. The present invention can monitor the standardization of the operator's work in the process of human-machine collaboration online, and reduce the defective rate of products;

[0025] 3. The present invention does not require other three-dimensional measuring instruments to divide and mark the workspace, and can complete the position calibration work using only RGB-D camera and ArUco augmented reality tags;

[0026] 4. The present invention can be applied to most human-machine collaboration scenarios and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a judgment logic diagram of the operator's work standardization and safety in a specific embodiment of the present invention;

[0028] Figure 2 3D skeleton layout diagram of RGB-D camera;

[0029] Figure 3 The operator point cloud extraction and processing flow chart of the present invention;

[0030] Figure 4 A general system architecture diagram for implementing the method provided in the present invention;

[0031] Among them, the joints numbered 20, 7, and 11 are respectively: shoulder spine joint, left hand joint, and right hand joint. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0033] The present invention provides a method and system for online monitoring of human-machine safe collaboration based on an RGB-D camera. To make the implementation method clearer and more understandable, in this embodiment, it is assumed that operation area 1 is the operator's operation area, operation areas 2 and 3 are raw material acquisition areas, operation area 4 is a raw material processing area, and operation area 5 is a machine operation area; the operation process is: the operator works in area 1 throughout the whole process and shall not leave without authorization during the collaboration stage, obtains raw materials in areas 2 and 3 and processes them in area 4, and finally puts the processed raw materials into area 5 when the machine is not in operation for further processing by the machine and transports them to the next link, wherein the demand for raw materials 1 and 2 in each task cycle is 1.

[0034] The method includes two parts: offline phase and online phase, and is implemented according to the following steps:

[0035] 1) Offline stage:

[0036] Step 1. Workspace division and labeling:

[0037] First, the workspace of the human-machine collaborative task to be monitored is divided into the raw material acquisition area, the operator operation area, the machine operation area, etc. (collectively referred to as the operation area). Finally, different ArUco augmented reality tags are used to calibrate the positions of the divided areas.

[0038] Step 2. Establishment of the operating area and monitoring area in the camera space:

[0039] Based on the capture of the ArUco tag pose by the RGB-D camera, the pose data of each tag in step 1 in the camera space is obtained, and a virtual operation area is established in the camera space according to the pose data and the geometric dimensions defined by each operation area. A monitoring area is established outside the dangerous area (such as the machine operation area) according to the minimum human-machine safety distance specified in ISO / 13855. Finally, the coordinate range of the established operation area and monitoring area is saved offline.

[0040] Step 3. Saving the background depth image of the workspace:

[0041] Based on the acquisition of the RGB-D camera depth image data, a workspace background depth map that does not include the operator is saved to provide workspace background data for the operator 3D point cloud extraction in step 6;

[0042] Step 4. Writing of operator safety and work standardization judgment logic:

[0043] Based on the specific human-machine collaborative tasks and the divided operation areas in step 1, design and write the logic and program for judging the operator's safety and work standardization status according to the collaborative task's operation procedures and the coordinate range detection method. The coordinate range detection method refers to tracking the operator's position based on the operator's 3D skeleton and point cloud data, and online detection of whether its position is within the coordinate range of a certain area;

[0044] The basis for judging the safety of the operator is whether a part of his body enters the monitoring area established in step 2 when approaching the danger zone (machine operation area) is prohibited. If there are data points in the operator's 3D point cloud that invade the coordinate range of the monitoring area when the machine is in operation, the operator is judged to be in a dangerous state;

[0045] The main basis for judging the standardization of the operator's operation is whether he is working in the area specified in the operation specification, whether he misses or takes too much raw materials, whether he processes the raw materials in the specified order, etc. The above status is mainly judged based on whether the operator's hand joints and shoulder spine joints are within the coordinate range of the above operation area. If his shoulder spine joints are already within the coordinate range of the operator's operation area, it is judged that the operator has been working in the specified area. Similarly, if his hand joints have entered the raw material acquisition area, it is judged that the raw material has been picked up. If it enters again, it is judged as repeated acquisition.

[0046] 2) Online stage:

[0047] Step 5. Acquisition of operator joint data:

[0048] Based on the 3D skeleton tracking technology of RGB-D camera, the 3D coordinates of the operator's hand joints and shoulder spine joints in the camera space are captured online. The joint layout is as follows: Figure 2 As shown;

[0049] Step 6. Removal of abnormal points and smoothing of joint data:

[0050] Based on the original joint data captured online in step 4, outliers are removed and smoothed. After outliers with large errors are removed, Holt's two-parameter exponential smoothing is performed on the original data with small errors to output more stable and reliable joint data.

[0051] The principle of the outlier removal algorithm is as follows: if the RGB-D camera acquires depth image data and joint data at a speed of 30 frames per second, the movement distance d and the change in movement direction (movement angle) θ of the operator within the time interval of two consecutive frames of data (1 / 30s) should be within a reasonable range due to the limited movement ability of human operators. Based on this, a joint reliability measurement algorithm is established to remove some unreliable joint data with large-scale jitter caused by occlusion or sensor noise. Finally, the joint data points that meet the reliability requirements are smoothed using the Holt dual-parameter exponential algorithm to remove the influence of small-scale jitter, thereby obtaining more stable and reliable joint data points.

[0052] Step 7. Extraction of operator 3D point cloud:

[0053] Based on the workspace depth image data stream obtained online by the RGB-D camera and the workspace background depth map obtained in step 3, the background removal algorithm is used to separate the operator's depth image, and it is converted into a 3D point cloud using the PCL point cloud library. The point cloud filtering algorithm is used to filter out excess noise, and finally a pure operator 3D point cloud is obtained. The extraction process of the operator's 3D point cloud is as follows: Figure 3 As shown;

[0054] Step 8. Position monitoring of the operator’s 3D point cloud:

[0055] Based on the operator point cloud data obtained in step 6, it is detected point by point whether there are data points entering the coordinate range of the monitoring area, providing input for the operator safety judgment logic in step 3.

[0056] In order to implement the method provided by the present invention, the present invention provides a general system model of a human-machine safety collaborative online monitoring system based on an RGB-D camera, including an RGB-D camera, a three-primary-color LED warning light with a buzzer, a reset button, and an industrial control computer; wherein the ArUco augmented reality tag is only used for the calibration of the operating area position in the offline stage, so it is not reflected in the system architecture; the industrial control computer uses the Linux operating system and integrates the ROS robot operating system platform, the PCL point cloud library, etc., to provide the operating environment of the present method; the I / O interface provided by the industrial computer is connected to the RGB-D camera, the three-primary-color LED warning light, the reset button, and the control system of the automated machine that collaborates with humans; wherein the depth image data stream of the RGB-D camera and the 3D The skeleton data stream is transmitted to the industrial computer through the I / O interface and processed. The operator status can be judged according to the method provided by the present invention. If the operator's status is judged to be normal, the industrial computer sets the LED warning light to a green light that is always on. If the operator's status is judged to be in violation of the standard operation, the LED warning light is set to flash yellow with a buzzer. At this time, depending on the type of machine that cooperates with people, a deceleration or shutdown instruction can be sent through the industrial computer I / O. After the operator returns to normal logic (such as returning to the specified working area), the system automatically returns to normal mode; if the operator's status is judged to be a dangerous operation, the LED warning light is set to flash red with a buzzer, and a shutdown instruction is sent. After the operator leaves the dangerous area and presses the reset button, the system returns to normal operation mode.

[0057] The human-machine safety collaboration online monitoring system of this embodiment is constructed according to the general system architecture provided by the present invention. Figure 4 shown.

[0058] According to step 1 above, the operation areas of this embodiment are divided and marked.

[0059] According to step 2, the coordinate range of each operation area is obtained, a monitoring area is established outside the operation area 5, and the coordinate ranges of the operation area and the monitoring area and the obtained background depth map of the workspace of this embodiment are saved in the industrial control computer.

[0060] according to Figure 1 The judgment logic of the operator's operation standardization and safety judgment program of this embodiment is written and loaded into the industrial computer.

[0061] According to steps 5-6, the joint data is obtained online, and outliers are removed and smoothed to obtain stable and reliable joint data. Based on the principle of the joint outlier removal algorithm provided in step 6, as a preferred method, the maximum distance that the operator moves within two consecutive frame time intervals can be set to d max =0.01m, the movement angle is θ max =85°.

[0062] Follow step 7 to extract the clean operator 3D point cloud without the workspace background online.

[0063] Based on the joint data and point cloud data acquired and processed online, the online monitoring of the human-machine collaboration task of this embodiment is completed according to the judgment procedure.

[0064] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A human-machine safe collaboration online monitoring method based on RGB-D camera, It is characterized in that include: Step 1: Divide the workspace of the human-machine collaborative task into different operation areas, and use ArUco augmented reality tags to calibrate the positions of the divided areas in different operation areas; Step 2: Use the RGB-D camera to capture the position coordinates of each tag, determine the position of each operation area in the camera space, and calculate the coordinate range in the camera space based on the space occupied by each area to establish a virtual operation area; establish a virtual monitoring area outside the dangerous operation area based on the minimum safety distance between humans and machines; Step 3: Use an RGB-D camera to collect depth image data to obtain a background depth image of the workspace excluding the operator; Step 4: Collect the scene depth image, extract the operator's 3D skeleton data, and perform pixel-by-pixel subtraction between the collected scene depth image and the pre-stored workspace background depth image to obtain a depth map containing only the operator, and then convert it into a body point cloud with position information in the camera space; Step 5: Combine the joint and point cloud data to determine the execution status of the processing procedure, realize the online position tracking of the operator in the workspace, and give an alarm when danger occurs; Step 51, extracting the operator's 3D skeleton data and body point cloud data and further processing them; Step 52: determining the execution status of the processing procedure by detecting the dynamic position relationship between each joint and each virtual operation area in the 3D skeleton data. The determination of the execution status of the processing procedure is used to identify whether the operator is operating in the area specified by the operation specification, whether the operator misses or takes too much raw material, and whether the raw material is processed in the specified order. Each operating area is divided into a raw material acquisition area, a raw material processing area, an operator operation area, and a machine operation area accessible to operators; including: When the head joint, neck joint or shoulder-spine joint is always within the operator's working area during the collaboration phase, it is judged that the operator is working in the designated area; When the operator's hand joint enters the raw material acquisition area, it is judged that the operator picks up the raw material in the current raw material acquisition area. If the operator enters the area repeatedly, it is judged as repeated picking. When the hand joint enters the raw material processing area, it is judged that the operator is performing processing operations in the raw material processing area; Step 53: By detecting whether the operator's body point cloud intrudes into the danger monitoring area, it is determined whether the operator approaches the machine while the machine is in operation, and an alarm is given.

2. According to the method of online monitoring of human-machine safety collaboration based on RGB-D camera in claim 1, It is characterized in that The PCL point cloud library is used to convert the operator's depth map into a 3D point cloud, which is then further filtered and clustered into body point cloud data.

3. According to the method of online monitoring of human-machine safety collaboration based on RGB-D camera in claim 2, It is characterized in that The extracted joint data are subjected to outlier removal and smoothing processing; the outlier removal processing is that if the position change of the same joint point within the time interval of two consecutive frames of image data is not within a preset range, then this joint point is an outlier; the smoothing processing is the Holt dual-parameter exponential algorithm.

4. A human-machine safety collaboration online monitoring system based on RGB-D camera, It is characterized in that include: An industrial control computer and an RGB-D camera, a warning module, a reset button, and a control system for an automated machine that collaborates with humans that communicate with the industrial control computer; the RGB-D depth camera is used to collect depth images of an actual workspace of a human-machine collaborative task; the warning module is used to alarm when a processing error occurs or involves personal danger; the reset button is used to press the reset button to continue detection after the operator leaves the danger zone; the industrial control computer stores a program, and when the program is loaded, the steps of a human-machine safe collaboration online monitoring method based on an RGB-D camera as described in claims 1-3 are executed to determine the execution status of the processing procedure and to give an alarm when the operator enters a danger monitoring zone.

Citation Information

Patent Citations

  • Kinect based field operation simulation method and Kinect based field operation simulation system

    CN103455657A

  • Visual-based man-machine safety system of industrial mechanical arm

    CN110253570A

  • Industrial mechanical arm man-machine safety distance detection method based on Kinect camera

    CN110378937A

  • Safety cooperation method and device based on man-machine integration

    CN110561432A

  • Power grid infrastructure site virtual fence system based on RGB-D camera and control method thereof

    CN111009089A