A method for multi-industrial robot 2D and 3D key point estimation based on RGB-D camera

Abstract

The application provides a multi-industrial robot 2D and 3D key point estimation method based on an RGB-D camera, and specifically comprises the following steps: establishing a measurement system hardware configuration of an industrial robot, wherein the measurement system hardware configuration comprises an RGB-D camera and a plurality of industrial robots; establishing a simulation dataset generation system, selecting key points of the industrial robot, and generating a simulation dataset of key point coordinates based on a virtual environment; building a coordinate detection network based on the RGB-D camera, training the coordinate detection network through the simulation dataset; and using the trained coordinate detection network to realize real-time observation of 2D and 3D coordinates of the key points of the plurality of industrial robots. The method provided by the application can realize accurate and real-time estimation of 2D and 3D key points of the plurality of industrial robots by using only a low-cost RGB-D camera, and provides an effective solution for measurement of key points of the plurality of industrial robots.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of industrial robot measurement, and particularly relates to a multi-industrial robot 2D and 3D key point estimation method based on an RGB-D camera. BACKGROUND

[0002] In the high-end equipment industry field such as aerospace and energy, in order to improve the precision and efficiency of machining and assembly, the machining and assembly of large and complex components such as large wind turbine blades and space station spacecraft cabins gradually develop from traditional machine tool operation to industrial robot and human collaborative operation mode. In the human-robot collaborative process, real-time pose estimation of humans and robots can provide a data basis for human-robot safety, and is an important part of ensuring the smooth progress of overall collaborative operation.

[0003] The current industrial robot joint pose estimation applied in human-robot safety monitoring can be divided into active estimation and passive estimation. Passive estimation refers to establishing a communication connection between a computer and a robot, and the industrial robot controller cyclically sends its joint angles to the computer, so as to complete the perception of the actual state of the robot. However, the base coordinate system of the robot still needs to be calibrated, which not only needs the participation of high-cost large-field measurement equipment, but also the measurement field of view of the calibration will be affected by multiple factors such as on-site equipment in complex on-site environments. In addition, in the large component machining scene, there are usually multiple mobile industrial robots, and AGVs are installed at the bottom of the industrial robots to process different parts of the large components. Due to the different positions of the AGVs each time and the existence of AGV positioning errors, the base coordinate system of the industrial robot carried by the AGV will also change, making it extremely difficult to calibrate the base coordinate system online. Therefore, passive estimation cannot be used for online measurement of the pose of the robot. Finally, this method completely depends on the support degree of the joint angle information transmission function of the industrial robot controller, and different degrees of development and adaptation are required for different brands of industrial robots.

[0004] Active estimation refers to actively estimating the pose of the industrial robot through cameras, lidars and other sensors. This method completely relies on the perception information of the sensor itself, and there is no physical connection between the sensor and the industrial robot, which reduces the degree of system coupling, improves the scene adaptability, and can monitor the safety of the robot from an objective perspective, improving the stability of the system. In addition, the active pose estimation of the robot can be combined with human pose estimation, human-robot interaction and other technologies to simultaneously estimate the state and interaction information of humans and robots, which is an important technical approach to realize human-robot safety monitoring and improve human-robot collaboration efficiency. SUMMARY

[0005] In view of the problems in the prior art, the present application provides a multi-industrial robot 2D and 3D key point estimation method based on an RGB-D camera, so as to realize accurate and real-time estimation of the 2D and 3D key points of the multi-industrial robot, and objectively and stably monitor the pose of the robot.

[0006] To achieve the above technical purpose, the present application provides the following technical scheme:

[0007] A multi-industrial robot 2D and 3D key point estimation method based on an RGB-D camera, which specifically comprises:

[0008] S1, establishing a measurement system hardware configuration of the industrial robot, the measurement system hardware configuration comprising an RGB-D camera and a plurality of industrial robots;

[0009] S2, establishing a simulation dataset generation system, selecting key points of the industrial robot, and generating a simulation dataset of the key point coordinates based on a virtual environment;

[0010] S3, building a coordinate detection network based on the RGB-D camera, and training the coordinate detection network through the simulation dataset;

[0011] S4, using the trained coordinate detection network to realize real-time observation of the 2D and 3D coordinates of the key points of the plurality of industrial robots.

[0012] Further, in step S1, the plurality of industrial robots are arranged within the measurement field of view of the RGB-D camera, and the flanges of the plurality of industrial robots can be connected to a plurality of actuator mechanisms that do not block the measurement field of view of the camera.

[0013] Further, step S2 specifically comprises:

[0014] S21, selecting 9 key points for each industrial robot;

[0015] S22, setting the generation target of the simulation dataset generation system to include: 2D coordinates of the plurality of key points of the industrial robot in an image, 3D coordinates of the plurality of key points of the industrial robot in a camera coordinate system, a color image captured by the RGB-D camera, a depth image captured by the RGB-D camera, and a bounding box of the industrial robot in the image;

[0016] S23, building the simulation dataset generation system based on an Unreal game engine, the system comprising a simulation environment, a plurality of virtual RGB-D cameras, and a plurality of virtual industrial robots, and the virtual cameras capturing the virtual industrial robots in real time from different angles;

[0017] S24, the running logic of the design data set generation system is: first, the virtual robot performs random motion by generating random numbers in the feasible joint space, detects self-collision and external collision, and forms the reachable pose of the robot in the real environment; then a plurality of virtual RGB-D cameras sequentially shoot and respectively calculate the 3D point coordinates of the 9 key points of the pre-selected industrial robot relative to the camera, and project the 3D points to 2D space to obtain image coordinates, finally combine the data into the same label file, wherein the depth map is encoded in exr format, the color map is in png format, and the label file is in txt format.

[0018] Further, step S3 specifically comprises:

[0019] S31, the coordinate detection network is composed of YOLOv8 network, HRNet network and industrial robot key point depth estimation network IRDepthNet;

[0020] S32, YOLOv8 network, HRNet network and IRDepthNet network are respectively trained using simulation data set, and data enhancement method is used to add Gaussian noise, Gaussian blur to the picture, and randomly adjust the contrast, saturation and brightness of the picture to simulate the picture effect in different real environments;

[0021] S33, a top-down detection mechanism is established, specifically: the coordinate detection network inputs the color image collected by the RGB-D camera, first passes through the YOLOv8 network to obtain the 2D bounding box of the industrial robot in the color image and the cropped image in the 2D bounding box; the cropped image in the 2D bounding box is input into the HRNet network to detect the key points of the industrial robot, and the heat map of the 9 key points is obtained; on the one hand, the heat map is processed by the maximum value response to obtain the 2D coordinates of the key points of the industrial robot, and on the other hand, the depth image and the color image are input into the IRDepthNet network to obtain the 3D coordinates of the key points of the industrial robot. Figure One Figure One

[0022] Further, in the training process, the HRNet network adjusts the number of output channels to 18 channels, and each channel outputs a 2D coordinate value of a key point.

[0023] Further, the IRDepthNet network is:

[0024] ​​The IRDepthNet network consists of a stitching layer, a ResNet50 network, and several fully connected layers. The input depth map and color map are first cropped and filled, and the length, width, and number of channels of the image are adjusted. Then, they are stitched together with the key point heatmap in the stitching layer. After that, the images pass through the ResNet50 network and several fully connected layers of different scales in sequence. Through continuous convolution operations, the xyz coordinate values ​​of the 3D coordinates of the 9 key points are finally obtained.

[0025] Furthermore, the loss function of the IRDepthNet network The design is the sum of the absolute MSE error and the relative MSE error, specifically:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] in, For absolute MSE error, This is the relative MSE error. The first i The x-axis, y-axis, and z-axis coordinates of each predicted key point; The first i The x-axis, y-axis, and z-axis coordinates of each theoretical key point; For the first i The first key prediction point and the first The distance between each predicted key point For the first i The first key theoretical point and the first The distance between the key theoretical points.

[0032] Based on the above technical solution, the present invention has at least the following beneficial effects:

[0033] The method proposed in this invention can complete the real-time key point measurement of multiple industrial robots using only a low-cost RGB-D camera, reducing equipment costs while enabling real-time measurement and output of the 2D and 3D coordinates of key points of multiple industrial robots, providing an effective solution for the measurement of key points of multiple industrial robots. Attached Figure Description

[0034] Figure 1This is a layout diagram of the RGB-D camera and industrial robot in this application; 1 in the diagram is the RGB-D camera, and 2 is the industrial robot;

[0035] Figure 2 A schematic diagram showing the key points of the industrial robot in this application;

[0036] Figure 3 Generate a system operation flowchart for the simulation dataset in this application;

[0037] Figure 4 This is an overall flowchart of a method for estimating 2D and 3D key points of multiple industrial robots based on an RGB-D camera, as proposed in this application.

[0038] Figure 5 This is a diagram of the IRDepthNet network structure designed for this invention;

[0039] Figure 6 This is a screenshot showing the robot bounding box detection results on a simulation dataset.

[0040] Figure 7 The image shows the 2D coordinate detection results of robot key points on a simulation dataset;

[0041] Figure 8 An estimated 3D coordinate diagram of the robot's key points. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figures 1-8 The present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0043] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0044] like Figure 4 As shown, this invention proposes a method for estimating 2D and 3D key points of multiple industrial robots based on an RGB-D camera, which specifically includes the following steps:

[0045] S1. Establish the hardware configuration for the measurement system of the industrial robot, such as... Figure 1As shown, the measurement system hardware configuration includes an RGB-D camera and multiple industrial robots; in this embodiment, the multiple industrial robots are arranged within the measurement field of view of the RGB-D camera, and the flanges of the multiple industrial robots can be connected to various actuator mechanisms that do not block the camera measurement field of view.

[0046] S2, establish a simulation dataset generation system, select key points of the industrial robot, and generate a simulation dataset of the key point coordinates based on a virtual environment;

[0047] As a preferred embodiment, step S2 specifically includes:

[0048] S21, as shown, 9 key points are selected for each industrial robot; Figure 2

[0049] S22, the generation target of the simulation dataset generation system includes: two-dimensional coordinates of multiple key points of the industrial robot in the image, three-dimensional coordinates of multiple key points of the industrial robot in the camera coordinate system, a color image captured by the RGB-D camera, a depth image captured by the RGB-D camera, and a bounding box of the industrial robot in the image;

[0050] S23, based on the Unreal game engine, a simulation dataset generation system is built, which is composed of a simulation environment, multiple virtual RGB-D cameras, and multiple virtual industrial robots. The virtual cameras capture the virtual industrial robots in real time from different angles;

[0051] S24, the running logic of the dataset generation system is designed as follows: first, the virtual robot performs random motion by generating random numbers in the feasible joint space, and detects its own collision and external collision to form the reachable pose of the robot in the real environment; then, multiple virtual RGB-D cameras sequentially capture and calculate the 3D point coordinates of the 9 key points of the pre-selected industrial robot relative to the camera, and project the 3D points to the 2D space to obtain the image coordinates; finally, the data is combined into the same label file, wherein the depth image is encoded in exr format, the color image is in png format, and the label file is in txt format.

[0052] S3, build a coordinate detection network based on the RGB-D camera, and train the coordinate detection network through the simulation dataset;

[0053] As a preferred embodiment, step S3 specifically includes:

[0054] S31, the coordinate detection network is composed of a YOLOv8 network, an HRNet network, and an industrial robot key point depth estimation network IRDepthNet;

[0055] ​S32. Use a simulation dataset to train the YOLOv8 network, HRNet network, and IRDepthNet network respectively, and use data augmentation methods to add Gaussian noise to blur the image, and randomly adjust the contrast, saturation, and brightness of the image to simulate the image effect under different real-world environments.

[0056] S33. Establish a top-down detection mechanism, specifically: The coordinate detection network takes the color image captured by the RGB-D camera as input, first passes it through the YOLOv8 network to obtain the 2D bounding box of the industrial robot in the color image and the cropped image within the 2D bounding box; the cropped image within the 2D bounding box is then passed through the HRNet network to perform key point detection on the industrial robot, resulting in a heat map of 9 key points; the heat map... Figure One On the one hand, the 2D coordinates of the key points of the industrial robot are obtained through maximum response processing; on the other hand, they are compared with depth maps and color maps. Figure One The data is fed into the IRDepthNet network to obtain the 3D coordinates of the key points of the industrial robot.

[0057] In addition, in this embodiment, during the training process, the HRNet network adjusts the number of output channels to 18 channels, with each channel outputting a 2D coordinate value of a key point.

[0058] The specific structure of the IRDepthNet network is as follows: Figure 5 As shown, the algorithm includes a stitching layer, a ResNet50 network, and several fully connected layers. The input depth map and color map are first cropped and filled to adjust the length, width, and number of channels of the image. Then, they are stitched together with the key point heatmap in the stitching layer. After that, the images pass through the ResNet50 network and several fully connected layers of different scales in sequence. Through continuous convolution operations, the xyz coordinate values ​​of the 3D coordinates of the 9 key points are finally obtained.

[0059] Furthermore, this application also designs a loss function for training the IRDepthNet network. Specifically, it is designed as the sum of absolute MSE error and relative MSE error, expressed by the formula:

[0060] ;

[0061] ;

[0062] ;

[0063] ;

[0064] ;

[0065] in, is an absolute MSE error, is a relative MSE error, is the x-axis, y-axis, z-axis coordinate of the first i predicted key point, respectively; is the x-axis, y-axis, z-axis coordinate of the first i theoretical key point, respectively; is the distance between the first i predicted key point and the second predicted key point, is the distance between the first i theoretical key point and the second theoretical key point.

[0066] S4, using the trained coordinate detection network to realize real-time observation of the 2D and 3D coordinates of the key points of the multiple industrial robots.

[0067] The entire process of the method proposed in the application has been introduced.

[0068] The specific detection effect of the method proposed in the application is also given in this embodiment, as shown in Figures 6-8 ;

[0069] Among them, Figure 6 shows the 2D bounding box detection effect of the industrial robot in the color image in the simulation environment, and the red rectangular frame in the figure is the industrial robot, from which it can be seen that the industrial robot can be accurately framed in any scene;

[0070] Figure 7 shows the effect of estimating the 2D coordinates of the key points of the industrial robot in the color image in the simulation environment; straight lines are used to connect adjacent key points in the figure, and it can be seen that the connecting lines of the detected key points accurately outline the skeleton of the industrial robot, reflecting the accurate estimation of the 2D key points;

[0071] Figure 8 shows the estimation effect of the 3D coordinates of the key points, in which blue is the predicted value and red is the theoretical value; it can be seen that the predicted value of the 3D coordinates of the key points is also highly consistent with the theoretical value.

[0072] In summary, the method proposed in the application can complete the real-time key point measurement of multiple industrial robots using only a low-cost RGB-D camera, which reduces the equipment cost and can realize the real-time measurement and output of the 2D and 3D coordinates of the key points of multiple industrial robots, and provides an effective solution for the measurement of the key points of multiple industrial robots.

[0073] It will be obvious to a person skilled in the art that the application is not limited to the details of the foregoing exemplary embodiments and can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. The embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims to the identity of the reference signs therein.

[0074] Furthermore, it should be understood that although the description is made on the basis of the embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. A method for estimating 2D and 3D key points of multiple industrial robots based on an RGB-D camera, characterized in that, Specifically, the following steps are included: S1. Establish the hardware configuration of the measurement system for industrial robots. The hardware configuration of the measurement system includes RGB-D cameras and multiple industrial robots. S2. Establish a simulation dataset generation system, select the key points of the industrial robot, and generate a simulation dataset of key point coordinates based on the virtual environment; Specifically, it includes: S21. Nine key points are selected for each industrial robot; S22. The generation objectives of the simulation dataset generation system include: the two-dimensional coordinates of multiple key points of the industrial robot in the image, the three-dimensional coordinates of multiple key points of the industrial robot in the camera coordinate system, the color image captured by the RGB-D camera, the depth map captured by the RGB-D camera, and the bounding box of the industrial robot in the image. S23. A simulation dataset generation system is built based on the Unreal Engine. The system consists of a simulation environment, multiple virtual RGB-D cameras, and multiple virtual industrial robots. The virtual cameras capture images of the virtual industrial robots in real time from different angles. S24. The operating logic of the dataset generation system is as follows: First, the virtual robot performs random motion by generating random numbers in the feasible joint space and detects its own collisions and external collisions to form the robot's reachable pose in the real environment; then, multiple virtual RGB-D cameras sequentially capture and calculate the 3D coordinates of 9 key points of the pre-selected industrial robot relative to the camera, and project the 3D points onto 2D space to obtain image coordinates. Finally, the data is combined into a single label file, where the depth map is encoded in exr format, the color image is in png format, and the label file is in txt format. S3. Construct a coordinate detection network based on an RGB-D camera, and train the network using a simulation dataset; specifically including: S31. The coordinate detection network consists of the YOLOv8 network, the HRNet network, and the IRDepthNet network for estimating the depth of key points of industrial robots. S32. Use a simulation dataset to train the YOLOv8 network, HRNet network, and IRDepthNet network respectively, and use data augmentation methods to add Gaussian noise to blur the image, and randomly adjust the contrast, saturation, and brightness of the image to simulate the image effect under different real-world environments. S33. Establish a top-down detection mechanism, specifically: The coordinate detection network inputs the color image captured by the RGB-D camera, which first passes through the YOLOv8 network to obtain the 2D bounding box of the industrial robot in the color image and the cropped image within the 2D bounding box; the cropped image within the 2D bounding box is then passed through the HRNet network to perform key point detection on the industrial robot, resulting in a heatmap of 9 key points; the heatmap is processed by maximum response to obtain the 2D coordinates of the key points of the industrial robot, and then fed into the IRDepthNet network along with the depth map and the color image to obtain the 3D coordinates of the key points of the industrial robot; S4. Real-time observation of the 2D and 3D coordinates of key points of multiple industrial robots is achieved using the trained coordinate detection network.

2. The method for estimating 2D and 3D key points of multiple industrial robots based on an RGB-D camera according to claim 1, characterized in that, In step S1, multiple industrial robots are positioned within the measurement field of view of the RGB-D camera, and the flanges of the multiple industrial robots can be connected to various actuator mechanisms that do not obstruct the camera's measurement field of view.

3. The method for estimating 2D and 3D key points of multiple industrial robots based on an RGB-D camera according to claim 1, characterized in that, During training, the HRNet network adjusts the number of output channels to 18, with each channel outputting a 2D coordinate value of a key point.

4. The method for estimating 2D and 3D key points of multiple industrial robots based on an RGB-D camera according to claim 1, characterized in that, The IRDepthNet network is: The IRDepthNet network consists of a stitching layer, a ResNet50 network, and several fully connected layers. The input depth map and color map are first cropped and filled to adjust the length, width, and number of channels of the image. Then, they are stitched together with the key point heatmap in the stitching layer. After that, the images pass through the ResNet50 network and several fully connected layers of different scales in sequence. Through continuous convolution operations, the xyz coordinate values ​​of the 3D coordinates of the 9 key points are finally obtained.

5. The method for estimating 2D and 3D key points of multiple industrial robots based on an RGB-D camera according to claim 4, characterized in that, Loss function of IRDepthNet network The design is the sum of the absolute MSE error and the relative MSE error, specifically: ； ； ； ； ； in, For absolute MSE error, This is the relative MSE error. The first i The x-axis, y-axis, and z-axis coordinates of each predicted key point; The first i The x-axis, y-axis, and z-axis coordinates of each theoretical key point; For the first i The first key prediction point and the first The distance between each predicted key point For the first i The first key theoretical point and the first The distance between the key theoretical points.

Images