Texture object 6D pose data set construction method and device based on depth camera

Abstract

The invention discloses a texture object 6D pose data set construction method based on a depth camera, and the method comprises the steps: S1, carrying out the modeling of a target object through Solidworks, and outputting a target object point cloud with texture; s2, obtaining ordered initial ball feature points by using an Aruco code; and S3, the initial pose of the target object is calculated through the TCP coordinates of the tail end of the mechanical arm. And S4, the rotating platform is started, the tail end pose of the mechanical arm is changed during the period, and image data are collected. And S5, solving the relative pose of the interval frame according to the pose calibration plate sphere center feature point set so as to obtain the real pose of the target object. And S6, re-projecting the point cloud into the image to obtain the content of the pose data set. According to the method, initial pose calculation of the target object is completed in a simpler, more convenient and quicker manner, introduction of two-dimensional code corner detection errors is avoided by using spherical features, the mechanical arm and the rotary table are combined in pose collection, and pose sampling of the target object is fully considered.

Description

Technical Field

[0001] The present disclosure relates to the fields of image technology, machine vision and deep learning technology, and mainly provides training data for intelligent perception of textured objects, and specifically relates to a method and device for constructing a 6D pose data set of textured objects. Background Art

[0002] It is foreseeable that the future will present problems such as an aging population, so service robots will also become the best choice for tedious tasks such as elderly care. At present, such robots are only relatively mature in the field of plane grasping, such as data-driven target detection, two-dimensional grasping detection and other deep learning applications. However, due to the inherent lack of two degrees of freedom, it is impossible to ensure that the visual sensor is perpendicular to the grasping plane in real time in reality, and the application is very limited. Therefore, it is very necessary to consider 6D grasping. This type of grasping requires a clear understanding of the rigid transformation of the target object relative to the visual sensor, which involves 6D pose estimation, which is also a major problem in the field of robotic arm grasping today.

[0003] Plane grabbing simplifies the pose estimation process and can complete target positioning only from the image. This type of image data is easy to obtain and has a low processing cost. Therefore, even if traditional algorithms are used for plane grabbing, their efficiency is very high. However, 6D pose estimation often also needs to consider three-dimensional data such as depth, because the added depth information can better enhance environmental perception capabilities. However, if the point cloud obtained by depth processing is to be processed in the algorithm, it will be very time-consuming and resource-intensive. This is also the main reason why traditional point cloud registration methods are inefficient for pose estimation. Therefore, the use of deep learning technology is the main trend of 6D pose estimation today, and the 6D pose dataset is an important part of it.

[0004] There is a big difference between 6D pose datasets and target detection datasets. The latter can be completed only by manually labeling the target object information, but the former requires not only the label of the target object on the image, but also the rigid transformation data of the target object in the camera. Manual operation of such data is very cumbersome and inefficient. Therefore, a method for automatic labeling of 6D pose datasets is very necessary; and the process of making a pose estimation dataset is similar to the process of continuous sampling. If the trained pose estimation network is to have good robustness, it is necessary to ensure that the target object has enough pose transformations under the visual sensor. Therefore, in the process of automatic labeling, the diversity of sampling also needs to be considered.

[0005] In the prior art, the publication (announcement) number is CN 113920191A, and the publication (announcement) date is 2020.02.28: It involves a 6D data set construction method based on a depth camera, which first presets manual feature points of the target object, uses the manual feature point data to obtain the initial pose of this type of original point cloud to the actual point cloud, and then performs ICP registration on the initial pose to obtain the precise pose of the target object; however, it is very time-consuming to manually mark the feature points of each frame of the result point cloud, and the ICP point set registration requires a sufficient number of feature points to ensure accuracy, and during the sampling process, this method cannot handle the problem of missing feature points caused by related occlusion of the target object when the pose is placed. Publication (Announcement) Number: CN 115761407A, Publication (Announcement) Date: 2023.01.17: It relates to a method for constructing a 6D pose dataset for general industrial parts, which uses a pose estimation board such as AprilTag that can provide feature pose data. Each AprilTag QR code can provide feature points, which fully considers the occlusion problem. However, in the pose estimation calculation of the target object, a large amount of AprilTag QR code corner point information is required for PnP calculation, but the error of pixel corner point detection will be passed to the pose calculation, and the process of calculating the initial pose of the target object by this method is very complicated, and it needs to be rendered first, and then manually placed to the rendered point cloud map, which is very time-consuming. In addition, these two methods do not fully transform the pose of the target object in the camera during a collection process, and the sampling diversity of the pose is not sufficient. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method and device for constructing a 6D pose dataset of texture objects based on a depth camera, which fully realizes multi-pose sampling of the target object through the joint cooperation of a rotating platform and a robotic arm; simplifies the initial pose acquisition process of the target object; and adopts a combination of QR code and spherical feature point cloud to solve the problem of missing feature points caused by occlusion.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A device for constructing a 6D pose data set of texture objects based on a depth camera includes a robotic arm vision device and a rotating platform.

[0009] The robot arm vision device includes a robot arm, a camera mounting frame, a depth camera of model RealsenseD455 and a TCP tool end. The robot arm base is fixed on the object-carrying table, the camera mounting frame is fixed on the end joint of the robot arm, the depth camera is fixed on the camera mounting frame, and the TCP tool end is fixed on the end flange of the robot arm.

[0010] The rotating platform includes a target object, an Aruco code, a posture calibration plate, a rotating turntable and an adapter plate. The adapter plate is fixedly connected to the object table, the rotating turntable is fixed on the adapter plate, the posture calibration plate is fixed on the rotatable surface of the rotating turntable, the Aruco code is fixed on the posture calibration plate, and the target object is fixed on the blank surface of the posture calibration plate.

[0011] Preferably, the robotic arm is connected to the object-carrying table by bolts, the camera mounting bracket is fixed at the end joint of the robotic arm by the pre-tightening force of the bolts, the depth camera is connected to the camera mounting bracket by bolts and nuts, the TCP tool end is connected to the end of the robotic arm by bolts, and the depth camera changes with the posture of the end of the robotic arm; the adapter plate is fixed to the object-carrying table by bolts, the rotating turntable is also fixed to the adapter plate by bolts, the posture calibration plate is adhered to the rotatable surface of the rotating turntable, and the target object is also adhered to the blank placement surface of the posture calibration plate.

[0012] The present invention also provides a method for constructing a 6D pose dataset of a texture object based on a depth camera, which is implemented by applying the device for constructing a 6D pose dataset of a texture object based on a depth camera, including:

[0013] Step S1: Use Solidworks to model the target object, perform UV texture mapping, load it into MeshLab to convert the model into point cloud, perform point cloud processing operations in sequence, and finally output the target object model point cloud with texture, and preset the farthest point of the model in the target object coordinate system as the target object feature point;

[0014] Step S2: put the rotating platform in a static state, use the depth camera to capture the RGB and Depth images of the pose acquisition board and the target object, record the initial pose of the end of the robot arm, and use the Aruco code to obtain the ordered initial ball feature points;

[0015] Step S3: Control the robot arm to reach the visible feature points of the target object in the captured image one by one, record the TCP coordinates reached, and calculate the initial position and posture of the target object according to the coordinate system transformation relationship.

[0016] Step S4: Start the rotating platform to rotate at a fixed rate, and set the robot arm program to control the end pose transformation. During operation, use the depth camera to collect RGB-D images at a fixed frequency, record the end pose of each transformation, and calculate the relative matrix relationship between the camera coordinate systems at the interval end pose.

[0017] Step S5: After completing the data acquisition, the image is output, and the set of spherical center feature points on the pose calibration plate of each frame pose is solved, and the relative pose of the spherical feature points between the interval frames is solved, and then the initial pose of the target object is used to calculate the true pose of the target object in each frame of the image.

[0018] Step S6: According to the point cloud model of the target object and its true pose, reproject it into each frame image corresponding to the true pose to obtain the pose data set content such as the minimum bounding box and mask image.

[0019] Preferably, in step S1, the target object is modeled in Solidworks, UV texture mapping is performed until a textured target object model point cloud is output, and the method includes:

[0020] Step S11: Use Solidworks to model the target object at its original size and export it to obj format, set the captured texture map of the target object as the mtl material content, and complete the UV mapping.

[0021] Step S12: Import the textured obj model of the target object into the MeshLab software, perform sampling and other point cloud processing, output the target object ply point cloud file, reset the point cloud coordinate system, calculate the farthest point as the preset feature point in the target object coordinate system and save it as .

[0022] Preferably, in step S2, the rotating platform is placed in a static state, a depth camera is used to capture RGB and Depth images of the pose acquisition board and the target object, and the initial pose of the end of the robotic arm is recorded, and the ordered initial ball feature points are obtained using the Aruco code, specifically including:

[0023] Step S21: Place the target object on the position calibration plate holder of the rotating platform, keep the rotating platform and the robotic arm stationary, record the initial robotic arm end position, and use the depth camera to capture the static RGB and Depth images of the position calibration plate and the target object.

[0024] Step S22: Reconstruct the spherical marker point cloud, obtain the pose feature points through point cloud algorithm processing, and sort the pose feature points according to the Aruco marker code in the static RGB image to obtain an ordered initial spherical marker point set.

[0025] Preferably, in step S22, the method for obtaining posture feature point data based on the collected spherical marker point cloud includes:

[0026] The equations of these spherical marker point clouds that satisfy N spatial fitting spherical planes SP are assumed to be:

[0027]

[0028] Obtain the center set of N spherical landmarks through the PCL point cloud sphere fitting algorithm , the three-dimensional sphere center point cloud obtained in this way is disordered.

[0029] Preferably, in step S22, the method of sorting the pose feature points using the Aruco marker code to obtain an ordered initial spherical marker point set includes:

[0030] First, the Aruco codes of ID1-12 have been sorted on the pose calibration board, and are arranged in a clockwise order starting from the upper left corner.

[0031] Using the aruco module in the open source library OpenCV, the interface function detectMarkers is used to detect the ID number and corner coordinates of each Aruco annotation map on the RGB image, and the estimatePoseSingleMarkers interface function is used for position detection to finally obtain the Aruco code coordinate system position set of M IDs. ;

[0032] An Aruco sign with a different ID is preset between each two spherical markers. Centroid set with spherical markers The Euclidean distance is:

[0033]

[0034] According to the Euclidean distance , we can determine the Aruco code ID that is closer to the center of the ball, determine the serial number of the center of the ball marker, and push the corresponding serial number of the center into the stack in sequence to complete the sorting of the center set of the spherical marker, that is, to obtain the initial spherical marker point set under the initial posture of the posture calibration plate. .

[0035] Preferably, in step S3, the method of controlling the robot arm to reach the visible feature points of the target object in the captured image one by one, recording the TCP coordinates reached, and solving the initial posture of the target object according to the coordinate system conversion relationship includes:

[0036] Preferably, in step S31, the method of calculating the initial pose of the target object according to the visualized content of the target object in the static image and the data of controlling the TCP end of the robot arm to reach the feature point of the target object includes:

[0037] First, complete the TCP calibration of the tool end of the robot arm, and then use the real feature point set of the target object preset in step S12 , observe the visible area of ​​the target object in the RGB image captured in step S21.

[0038] Control the tool end of the robot arm to reach the preset real feature points of the target object placed on the pose calibration board, and record the TCP coordinate position of the stamping point in real time, which is recorded point by point as the real stamping point set. .

[0039] Using coordinate system transfer relations:

[0040]

[0041] Combined with S12 preset real feature point set , the initial pose of the target object on the pose calibration plate can be obtained by the least squares algorithm, that is, the coordinate system of the target object can be obtained. Relative to the camera coordinate system The first rigid transformation matrix under .

[0042] Preferably, in step S4, after the initial posture of the posture calibration plate and the target object relative to the camera is completed, the interface function of the Realsense depth camera for continuously acquiring image frames is used to collect RGB-D images at a fixed frequency.

[0043] When the acquisition begins, start the rotating platform to make the posture calibration plate and the target object rotate together at a fixed speed, and simultaneously start the running program of the robotic arm with a depth camera at the end to ensure uninterrupted posture change in a certain position space and range angle.

[0044] The posture transformation in step S4 also requires real-time recording of the end posture of the robot arm and calculation of the relative matrix relationship between the camera coordinate systems at the interval end posture. The method for calculating the relative posture includes:

[0045] By using the installation method of fixing the depth camera to the end of the robotic arm, when the end of the robotic arm changes its posture, the camera coordinate system has changed along with the end, so the following matrix transformation relationship is satisfied:

[0046]

[0047] in is the matrix relationship between the end and the base of the robot arm, which can be obtained by reading the teach pendant and is a known quantity, and is the result of hand-eye calibration, and , is a known quantity. By using the matrix transformation relationship between the depth camera and the end of the robotic arm, the relative matrix relationship between the camera coordinate systems at the interval end pose can be obtained. .

[0048] Preferably, in step S5, after completing the data acquisition and outputting the image, the spherical center feature point set on the pose calibration plate of each frame pose is solved, and the relative pose of the spherical feature points between the interval frames is solved, and then the initial pose of the target object is used to calculate the real pose of the target object in each frame of the image. The overall method includes:

[0049] Step S51: After completing the image acquisition of the pose calibration plate and the target object within the preset time, perform Aruco code pose estimation on each frame of the acquired image, and record the ID and coordinate values ​​obtained by the algorithm.

[0050] Step S52: Convert the depth map to point cloud for the spherical feature set next to the Aruco code, use the point cloud sphere fitting algorithm to obtain the center of the spherical feature of each visible part, and record the sorted center point set of the spherical feature according to the Aruco code ID matching principle.

[0051] Step S53 calculates the registration between the point sets according to the ball feature center point set with the same ID between the interval frames, obtains the relative pose transformation of the pose calibration plate in the interval frame image, and multiplies the initial pose of the target object by the relative pose multiple times to obtain the real pose of the target object in each frame.

[0052] Preferably, in step S51, after the set 2-minute pose data collection time is over, the turntable and the robotic arm are stopped from swinging, and each frame of RGB and Depth image during the pose data collection period is processed in a series of ways to obtain the Aruco code coordinate system position set of 12 IDs. .

[0053] After processing all the S collected RGB images, S different matrices containing 12 columns of Aruco ID and position are obtained. ,Right now:

[0054]

[0055] Preferably, in step S52, the method for solving and sorting the spherical feature center point set includes:

[0056] The depth map of the 12 hemispherical features preset on the pose calibration board is converted into point cloud information. In order to deal with the occlusion problem, the spherical platform features with too few point clouds are eliminated. Such point clouds meet the spatial spherical characteristics.

[0057] The point cloud sphere fitting algorithm in PCL is used to fit the spherical feature center of the visible part of each frame collected to obtain robust point cloud feature points.

[0058] According to the Aruco information of different IDs and positions extracted synchronously in RGB, the Euclidean distance is calculated, and finally the order of the spherical feature centers is sorted, which can be obtained. The corresponding sphere center matrix for:

[0059]

[0060] Preferably, in step S53, the method for calculating the true pose of the target object coordinate system relative to the camera coordinate system in each frame includes:

[0061] The local coordinate system of the ball feature on the preset pose acquisition board is , set the coordinate origin at the center of the ball feature. Since the ball feature size and other parameters are known, a set of relative local coordinates can be obtained. Point set , so the ball feature center data of each frame satisfies the following coordinate system transformation relationship:

[0062]

[0063] Since the camera coordinate system is constantly changing when collecting poses, the ball feature center dataset obtained in the collected images other than the initial frame The point clouds are all based on the camera coordinate system after the movement, so it is necessary to transform the coordinate system of the point cloud of the next frame of the interval frame into the camera coordinate system of the previous frame for discussion, and perform the following matrix transformation to the same camera coordinate system:

[0064]

[0065] The interval between two sets of sphere center sets satisfies the following matrix transformation relationship:

[0066]

[0067] This becomes a rigid transformation problem between two sets of points, which can be solved by applying the point cloud registration method to obtain the homogeneous transformation matrix of the sphere center point cloud in the next frame image outside the initial frame relative to the sphere center point cloud in the previous frame. ;

[0068] At the same time, since the pose calibration plate and the target object do not undergo relative transformation during the pose collection process, refer to the following coordinate system transformation relationship:

[0069]

[0070] The initial pose of the target object is multiplied by the relative pose between interval frames to obtain the true pose of the target object coordinate system relative to the camera coordinate system in each frame. ;

[0071] Preferably, in step S6, according to the point cloud model of the target object and its true posture, the method of reprojecting it to each frame image corresponding to the true posture to obtain the posture data set content such as the minimum bounding box and the mask image includes:

[0072] According to the preset target coordinate system point set , and the obtained rigid body transformation between the target object coordinate system and the camera coordinate system , and then combine the intrinsic parameter K information obtained by camera calibration to finally map the real target back to the pixel coordinate system and obtain the mask map of the target. This process mainly involves the transfer between multiple coordinate systems, namely:

[0073]

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] Compared with the prior art, the present invention uses professional model software for three-dimensional modeling and fully considers texture patterns, and can construct a textured high-quality point cloud, which ensures accuracy and precision when mapping back to the mask image after the complete relative pose is subsequently acquired; the initial pose calculation process of the target object is optimized, and the target pose can be placed arbitrarily; the present invention uses feature points obtained by spherical fitting in the process of calculating the relative pose, and this type of feature point has high robustness and high precision. Through this type of high-precision feature point data, the relative pose obtained by point cloud registration can be guaranteed to have good accuracy; at the same time, the present invention jointly applies a turntable and a robotic arm, making full use of the spatial accessibility of the two, and realizing automatic multi-position and multi-angle target pose data acquisition, avoiding the cumbersome process of single acquisition and single transposition. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0077] Figure 1 It is a schematic diagram of an exemplary embodiment of a device for constructing a 6D pose data set of a texture object based on a depth camera according to an embodiment of the present invention;

[0078] Figure 2 A schematic diagram of an exemplary implementation of a method for constructing a 6D pose dataset of a texture object based on a depth camera to obtain an initial pose of a target object according to an embodiment of the present invention;

[0079] Figure 3 A flowchart of obtaining ball feature points in a method for constructing a 6D pose data set of a texture object based on a depth camera according to an embodiment of the present invention;

[0080] 4 is an actual pose calibration plate and a detection diagram of the initial pose of the target object according to the method for constructing a 6D pose data set of a texture object based on a depth camera according to an embodiment of the present invention;

[0081] FIG5 is a flowchart of actual working application steps of an exemplary embodiment of a method for constructing a 6D pose data set of a texture object based on a depth camera according to an embodiment of the present invention;

[0082] FIG6 is a flowchart of specific steps of an exemplary embodiment of a method for constructing a 6D pose dataset of a texture object based on a depth camera according to an embodiment of the present invention in actual working application.

[0083] Explanation of the numbers in the figure: 1-Robotic arm, 2-Camera mounting bracket, 3-Depth camera, 4-TCP tool end, 5-Target object, 6-Aruco code, 7-Position calibration plate, 8-Turntable, 9-Connection plate, 10-Object table DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0086] Embodiment 1

[0087] like Figure 1 As shown, this embodiment provides a device for constructing a 6D pose data set of a texture object based on a depth camera, including: a mechanical arm vision device and a rotating platform;

[0088] The robot vision device includes a robot 1, a camera mounting frame 2, a depth camera 3 of model RealsenseD455 and a TCP tool end 4. The base of the robot 1 is fixed on the object-carrying table 10, the camera mounting frame 2 is fixed on the end joint of the robot, the depth camera 3 is fixed on the camera mounting frame 2, and the TCP tool end 4 is fixed on the end flange of the robot.

[0089] The rotating platform includes a target object 5, an Aruco code 6, a posture calibration plate 7, a rotating turntable 8 and an adapter plate 9. The adapter plate 9 is fixedly connected to the object-carrying table 10, the rotating turntable 8 is fixed on the adapter plate 9, the posture calibration plate 7 is fixed on the rotatable surface of the rotating turntable 8, the Aruco code is fixed on the posture calibration plate, and the target object 5 is fixed on the blank surface of the posture calibration plate;

[0090] Specifically, the robot arm 1 is bolted to the object holding table 10, the camera mounting bracket 2 is fixed at the end joint of the robot arm 1 by the pre-tightening force of the bolts, the depth camera 3 is connected to the camera mounting bracket 2 with bolts and nuts, the end of the TCP tool is connected to the end of the robot arm with bolts, the adapter plate 9 is fixed to the object holding table 10 with bolts, the rotating turntable 8 is also fixed to the adapter plate 9 by bolts, the posture calibration plate 7 is adhered to the rotatable surface of the rotating turntable 8, and the target object 5 is also adhered to the blank surface of the posture calibration plate for placing objects.

[0091] Embodiment 2

[0092] like Figure 5 , Figure 6 As shown, this embodiment provides a method for constructing a 6D pose dataset of a texture object based on a depth camera, such as Figure 5 As shown, the following steps are included:

[0093] Step S1: Use Solidworks to model the target object, perform UV texture mapping, load it into MeshLab to convert the model into point cloud, perform point cloud processing operations in sequence, and finally output the target object model point cloud with texture, and preset the farthest point of the model in the target object coordinate system as the target object feature point;

[0094] Step S2: put the rotating platform in a static state, use the depth camera to capture the RGB and Depth images of the pose acquisition board and the target object, record the initial pose of the end of the robot arm, and use the Aruco code to obtain the ordered initial ball feature points;

[0095] Step S3: Control the robot arm to reach the visible feature points of the target object in the captured image one by one, record the TCP coordinates reached, and calculate the initial position and posture of the target object according to the coordinate system transformation relationship.

[0096] Step S4: Start the rotating platform to rotate at a fixed rate, and set the robot arm program to control the end pose transformation. During operation, use the depth camera to collect RGB-D images at a fixed frequency, record the end pose of each transformation, and calculate the relative matrix relationship between the camera coordinate systems at the interval end pose.

[0097] Step S5: After completing the data acquisition, the image is output, and the set of spherical center feature points on the pose calibration plate of each frame pose is solved, and the relative pose of the spherical feature points between the interval frames is solved, and then the initial pose of the target object is used to calculate the true pose of the target object in each frame of the image.

[0098] Step S6: According to the point cloud model of the target object and its true pose, reproject it into each frame image corresponding to the true pose to obtain the pose data set content such as the minimum bounding box and mask image.

[0099] like Figure 6 As shown, a method for constructing a 6D pose dataset of a texture object based on a depth camera includes the following steps:

[0100] Step S11: Perform three-dimensional modeling in Solidworks software according to the original size of the target object. At this time, the target object coordinate system is the coordinate system of the initial modeling, and the model is exported in obj format. However, this type of modeling file does not have texture color information, and additional texture mapping operations are required. The method of adding mtl material files is adopted, wherein the texture information in the mtl file can be obtained by manually shooting the model image, and setting the image UV value to correspond one-to-one with the end face information of the obj model, so as to link the captured image texture to the obj model. At this point, a target object model with texture and color has been made.

[0101] Step S12: In order to realize the model mask information of the subsequent data set, it is necessary to realize the reprojection mapping into pixels. Therefore, it is also necessary to consider the point cloud of the modeled 3D obj file. Here, MeshLab software is used to import the obj file, perform point cloud sampling and color mapping of the material texture information to the point cloud. At this time, a large number of color point clouds have been obtained, and the model point cloud is downsampled to retain the point cloud data with feature information. However, this part of the point cloud is based on the Solidworks modeling coordinate system of step S11. This coordinate system may not be on the target object. It is necessary to re-establish the target object coordinate system at the center of mass of the target object, remap the point cloud to the target object coordinate system, export the target object point cloud as a ply file, and use the FPS farthest point algorithm to calculate the feature points of the target object model in the coordinate system and save the coordinate values ​​in the target object coordinate system.

[0102] Specifically, if Figure 2 As shown, in step S21, the target object needs to be placed in the white object-carrying area of ​​the pose calibration plate. At this time, the rotating platform fixedly connected to the pose calibration plate and the robotic arm with a depth camera at the end are both in a stationary state. The pose of the end of the robotic arm in the stationary state is recorded and recorded as the initial pose matrix of the end of the robotic arm. The depth camera is turned on, and the sizes of the color image and the depth image are set to 640x480. While ensuring that the pose calibration plate and the target object information are all clearly visible in the field of view of the depth camera, the color frame and depth frame of the depth camera are obtained at the same time.

[0103] Step S22: There are two types of feature markers on the pose calibration board: the first type is the 6X6 Aruco two-dimensional markers of ID1-ID12, and the second type is the hemispherical three-dimensional markers with a radius of r. The Depth image obtained in S21 is used to reconstruct the spherical point cloud of the second type. The point clouds are preset to satisfy the equations of N spatial fitting spherical planes SP:

[0104]

[0105] Obtain the center set of N spherical landmarks through the PCL point cloud sphere fitting algorithm The three-dimensional sphere center point cloud obtained in this way is disordered, so it is also necessary to perform Aruco code detection on the RGB image in step S21.

[0106] Specifically, according to Figure 3 As shown in the figure, there are Aruco codes with ID1-12 arranged in a clockwise order starting from the upper left corner on the pose calibration board. The aruco module in the open source library OpenCV is used to detect the ID number and corner point coordinates of each Aruco annotation image on the RGB image using the interface function detectMarkers. At the same time, the estimatePoseSingleMarkers interface function is used for position detection, and finally the Aruco code coordinate system position set of M IDs is obtained. ;

[0107] An Aruco sign with a different ID is preset between each two spherical markers. Centroid set with spherical markers The Euclidean distance is:

[0108]

[0109] According to the Euclidean distance , the Aruco code ID that is closer to the center of the ball can be determined. If the Aruco code is not blocked, it should be two consecutive ID numbers. According to the pre-set sequence of the posture calibration board, the serial number of the center of the ball marker can be determined;

[0110] The corresponding serial numbers of the sphere centers are pushed into the stack in sequence to complete the sorting of the sphere center set of the spherical markers, that is, the initial spherical marker point set under the initial posture of the posture calibration plate is obtained. .

[0111] Step S31: According to step S12, in the target object coordinate system The S real feature point sets of the target object have been preset in , observe the visible area of ​​the target object in the RGB image captured in step S21;

[0112] After the TCP calibration of the tool end of the robot arm is completed, according to Figure 2 As shown in the figure, the tool end of the robot arm is controlled to reach the preset real feature points of the target object placed on the pose calibration board, and the TCP coordinate position of the stamping point is recorded in real time, which is recorded point by point as the real stamping point set. ;

[0113] This set of TCP points is actually in the robot base coordinate system The coordinates of the first photo are recorded in the terminal coordinate system Relative to the base coordinate system Rigid transformation of , first transfer the real stamp point set to the coordinate system relative to the end , continue with the hand-eye matrix after hand-eye calibration Transfer to the camera coordinate system , satisfying the following relationship;

[0114]

[0115] Combined with S12 preset real feature point set , the initial pose of the target object on the pose calibration plate can be obtained by the least squares algorithm, that is, the coordinate system of the target object can be obtained. Relative to the camera coordinate system The first rigid transformation matrix under .

[0116] Step S41: After completing the initial pose of the pose calibration plate and the target object relative to the camera, the Realsense depth camera's interface function for continuously acquiring image frames is used to collect RGB-D images at a fixed frequency; when the acquisition starts, the rotating platform is started to rotate the pose calibration plate and the target object at a fixed speed, and the robot arm with a depth camera at the end is started synchronously to ensure that the pose is continuously changed in a certain position space and range angle. The pose change also requires real-time recording of the end pose of the robot arm, and calculation of the relative matrix relationship between the camera coordinate systems at the interval end pose. The method for calculating the relative pose includes:

[0117] By using the installation method of fixing the depth camera to the end of the robotic arm, when the end of the robotic arm changes its posture, the camera coordinate system has changed along with the end, so the following matrix transformation relationship is satisfied:

[0118]

[0119] in is the matrix relationship between the end and the base of the robot arm, which can be obtained by reading the teach pendant and is a known quantity, and is the result of hand-eye calibration, and , is a known quantity. By using the matrix transformation relationship between the depth camera and the end of the robotic arm, the relative matrix relationship between the camera coordinate systems at the interval end pose can be obtained. .

[0120] Step S51: After the set 2-minute pose data collection time is over, the turntable and the robot arm are stopped from swinging, and each frame of RGB and Depth image during the pose data collection period is subjected to a series of processing including:

[0121] First, detect the position of the Aruco code of ID1-12 on the pose calibration board of the RGB image, and use the interface function detectMarkers to detect the ID number and corner point coordinates from the upper left corner. At the same time, use the estimatePoseSingleMarkers interface function for position detection, and use all the one-to-one corresponding ID and position information as the calculation result of each frame;

[0122] If the ID acquisition position is missing due to occlusion, fill it with 0, and finally obtain the Aruco code coordinate system position set of 12 IDs After processing all the S collected RGB images, S different matrices containing 12 columns of Aruco ID and position are obtained, namely:

[0123]

[0124] Step S52: First, convert the depth map to point cloud, convert the depth map of the 12 hemispherical features preset on the pose calibration board into point cloud information. This type of point cloud meets the spatial sphere plane characteristics. Use the point cloud sphere fitting algorithm in PCL to fit the spherical feature center of the visible part of each frame collected;

[0125] At the same time, in order to deal with the occlusion problem, the ball platform features with too few point clouds are eliminated to obtain robust point cloud feature points;

[0126] According to the Aruco information of different IDs and positions extracted synchronously in RGB in step S51, the Euclidean distance calculation of step S22 is performed synchronously, and finally the order of the spherical feature centers is sorted, and the same The corresponding sphere center matrix for:

[0127]

[0128] Step S53: After the RGB and Depth images between the interval frames are processed by steps S51 and S52, the ball feature center data set in the interval frame can be obtained. In ;

[0129] The local coordinate system of the ball feature on the preset pose acquisition board is: , the origin of the coordinates is at the center of the ball feature. Since the size of the ball feature and other parameters are known, a set of relative local coordinates can be obtained. Point set , so the ball feature center data of each frame satisfies the following coordinate system transformation relationship:

[0130]

[0131] According to the camera coordinate system constantly transforming when collecting the pose mentioned in step S41, the center data set of the above ball features The point cloud in is also based on the transformed camera coordinate system, so it is necessary to transform the coordinate system of the point cloud of the next frame of the interval frame into the camera coordinate system of the previous frame for discussion, that is, to perform the following matrix transformation:

[0132]

[0133] When converted to the same camera coordinate system, the interval between two sets of sphere centers will satisfy the following matrix transformation relationship:

[0134]

[0135] in , s is the total number of images collected. This matrix transformation relationship represents: when converted to the same camera coordinate system, the homogeneous transformation matrix of the sphere center point cloud in the next frame image relative to the sphere center point cloud in the previous frame becomes a rigid transformation problem between two sets of points, which can be solved by applying the point cloud registration method;

[0136] At the same time, since the pose calibration plate and the target object do not undergo relative transformation during the pose collection process, the relative homogeneous transformation matrix of the sphere center point cloud in the previous and next frames also appears in the target object pose. Therefore, the initial pose of the target object obtained in step S31 can be recursively multiplied by the relative pose between the interval frames, that is, the true pose of the target object coordinate system relative to the camera coordinate system of each frame is obtained, referring to the following coordinate system conversion relationship:

[0137]

[0138] Step S61: Based on all the point cloud coordinate values ​​in the target object coordinate system obtained in step S12 and the real target object pose obtained in step S53, that is, the rigid body transformation between the target object coordinate system and the camera coordinate system, all the point cloud coordinate values ​​in the camera coordinate system can be obtained according to the coordinate system transfer;

[0139] Then the intrinsic reference information K obtained by camera calibration is combined to finally map the real target back to the pixel coordinate system and obtain the mask map of the target. This process mainly involves the transfer between multiple coordinate systems, namely:

[0140]

[0141] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A device for constructing a 6D pose dataset of a texture object based on a depth camera, characterized in that: include: Robotic arm vision device and rotating platform; The robot arm vision device includes a robot arm, a camera mounting frame, a depth camera of model RealsenseD455 and a TCP tool end. The robot arm base is fixed on the object-carrying table, the camera mounting frame is fixed on the end joint of the robot arm, the depth camera is fixed on the camera mounting frame, and the TCP tool end is fixed on the end flange of the robot arm. The rotating platform includes a target object, an Aruco code, a posture calibration plate, a rotating turntable and an adapter plate. The adapter plate is fixedly connected to the object-carrying table, the rotating turntable is fixed on the adapter plate, the posture calibration plate is fixed on the rotatable surface of the rotating turntable, the Aruco code is fixed on the posture calibration plate, and the target object is fixed on the blank surface of the posture calibration plate.

2. The device for constructing a 6D pose dataset of a texture object based on a depth camera according to claim 1, characterized in that: The robotic arm is connected to the object-carrying table by bolts, the camera mounting frame is fixed to the end joint of the robotic arm by bolts, the depth camera is connected to the camera mounting frame by bolts and nuts, the TCP tool end is connected to the end of the robotic arm by bolts, and the depth camera changes with the posture change of the end of the robotic arm; the adapter plate is fixed to the object-carrying table by bolts, and the rotating turntable is also fixed to the adapter plate by bolts. The posture calibration plate is adhered to the rotatable surface of the rotating turntable, and the target object is also adhered to the blank placement surface of the posture calibration plate, and the target object rotates with the rotatable surface of the rotating turntable.

3. A method for constructing a 6D pose dataset of texture objects based on a depth camera, characterized in that: The device for constructing a 6D pose data set of a texture object based on a depth camera according to any one of claims 1 to 2 is implemented, comprising: Step S1: Use Solidworks to model the target object, perform UV texture mapping, load it into MeshLab to convert the model into point cloud, perform point cloud processing operations in sequence, and finally output the target object model point cloud with texture, and preset the farthest point of the model in the target object coordinate system as the target object feature point; Step S2: put the rotating platform in a static state, use the depth camera to capture the RGB and Depth images of the pose acquisition board and the target object, record the initial pose of the end of the robot arm, and use the Aruco code to obtain the ordered initial ball feature points; Step S3: Control the robot arm to reach the visible feature points of the target object in the captured image one by one, and record the TCP coordinates reached, and calculate the initial position and posture of the target object based on the target feature points and TCP coordinate values ​​combined with the coordinate system conversion relationship; Step S4: Start the rotating platform to rotate at a fixed rate, set the robot program to control the end pose transformation, and record the end pose of each transformation. During the operation, use the depth camera to collect RGB-D images at a fixed frequency, record the end pose of each transformation, and calculate the relative matrix relationship between the camera coordinate systems at the interval end pose. Step S5: After completing the data acquisition, the image output is used to solve the spherical center feature point set on the pose calibration plate of each frame pose, and the relative pose of the spherical feature points between the interval frames is solved, and then the initial pose of the target object is used to calculate the true pose of the target object in each frame of the image; Step S6: According to the point cloud model of the target object and its true pose, reproject it into each frame image corresponding to the true pose to obtain the pose data set content such as the minimum bounding box and mask image.

4. The method for constructing a 6D pose dataset of texture objects based on a depth camera according to claim 3, characterized in that: In step S2, the method for obtaining the pose feature point data according to the Aruco code and the spherical marker collected by the depth camera in a static state includes: Step S22: reconstruct the spherical marker point cloud, obtain the pose feature points through point cloud algorithm processing, and sort the pose feature points according to the Aruco marker code in the static RGB image to obtain an ordered initial spherical marker point set; The equations of these spherical marker point clouds that satisfy N spatial fitting spherical planes SP are assumed to be: ； Obtain the center set of N spherical landmarks through the PCL point cloud sphere fitting algorithm , using the Aruco module in the open source library OpenCV, use the interface function detectMarkers to detect the ID number and corner coordinates of each Aruco annotation map on the RGB image, and use the estimatePoseSingleMarkers interface function for position detection, and finally obtain the Aruco code coordinate system position set of M IDs ; An Aruco sign with a different ID is preset between each two spherical markers. Centroid set with spherical markers The Euclidean distance is: According to the Euclidean distance , we can determine the Aruco code ID that is closer to the center of the ball, determine the serial number of the center of the ball marker, and push the corresponding serial number of the center into the stack in sequence to complete the sorting of the center set of the spherical marker, that is, to obtain the initial spherical marker point set under the initial posture of the posture calibration plate. .

5. The method for constructing a 6D pose dataset of a texture object based on a depth camera according to claim 4, characterized in that: In step S31, the method of calculating the initial posture of the target object according to the static target image data and the data of the TCP end of the control robot arm reaching the feature point of the target object includes: First, complete the TCP calibration of the tool end of the robot arm, and then calibrate the TCP according to the preset real feature point set of the target object. , observe the visible area of ​​the target object in the RGB image captured in step S21; Control the tool end of the robot arm to reach the preset real feature points of the target object placed on the pose calibration board, and record the TCP coordinate position of the stamping point in real time, which is recorded point by point as the real stamping point set. Using coordinate system transfer relations: Combined preset real feature point set , the initial pose of the target object on the pose calibration plate can be obtained through the PnP algorithm, that is, the coordinate system of the target object can be obtained. Relative to the camera coordinate system The first rigid transformation matrix under .

6. The method for constructing a 6D pose dataset of texture objects based on a depth camera according to claim 5, characterized in that: Step S4: Start the rotating platform to rotate at a fixed rate, and set the robot arm program to control the end pose transformation. During the operation, use the depth camera to collect RGB-D images at a fixed frequency, and record the end pose of each transformation. The relative matrix relationship between the camera coordinate systems at the interval end pose is calculated. The method includes: By using the installation method of fixing the depth camera to the end of the robotic arm, when the end of the robotic arm changes its posture, the camera coordinate system also changes along with the end, thus satisfying the following matrix transformation relationship: in is the matrix relationship between the end and the base of the robot arm, which can be obtained by reading the teach pendant and is a known quantity, and is the result of hand-eye calibration, and , is a known quantity. By using the matrix transformation relationship between the depth camera and the end of the robotic arm, the relative matrix relationship between the camera coordinate systems at the interval end pose can be obtained. .

7. The method for constructing a 6D pose dataset of texture objects based on a depth camera according to claim 6, characterized in that: The step S5, after completing the image output after data acquisition, solves the relative posture of the ball feature points between the interval frames according to the ball center feature point set on the posture calibration plate of each frame posture, and then uses the initial posture of the target object to calculate the real posture of the target object in the frame-by-frame image. The method includes: Step S51: After completing the image acquisition of the pose calibration plate and the target object within the preset time, perform Aruco code pose estimation on each frame of the acquired image, and record the ID and coordinate values ​​obtained by the algorithm; Step S52: Convert the depth map of the spherical feature set next to the Aruco code to a point cloud, use the point cloud sphere fitting algorithm to obtain the centroid of each visible part of the spherical feature, and record the sorted spherical feature center point set according to the Aruco code ID matching principle; Step S53 calculates the registration between the point sets according to the ball feature center point set with the same ID between the interval frames, obtains the relative pose transformation of the pose calibration plate in the interval frame image, and multiplies the initial pose of the target object by the relative pose multiple times to obtain the real pose of the target object in each frame.

8. The method for constructing a 6D pose dataset of texture objects based on a depth camera according to claim 7, characterized in that: In step S52, the method for solving and sorting the spherical feature center point set includes: The depth map of the 12 hemispherical features preset on the pose calibration board is converted into point cloud information. In order to deal with the occlusion problem, the ball platform features with too few point clouds are eliminated. Such point clouds meet the spatial ball plane characteristics. The point cloud sphere fitting algorithm in PCL is used to fit the spherical feature center of the visible part of each frame collected to obtain robust point cloud feature points. According to the Aruco information of different IDs and positions extracted synchronously in RGB, the Euclidean distance is calculated, and finally the order of the spherical feature center is sorted, which can be obtained with The corresponding sphere center matrix for:

9. The method for constructing a 6D pose dataset of texture objects based on a depth camera according to claim 8, characterized in that: In step S53, the method for calculating the true pose of the target object coordinate system relative to the camera coordinate system in each frame includes: First, the local coordinate system of the ball feature on the preset pose acquisition board is , set the origin of coordinates at the center of the ball feature. Since the size of the ball feature and other parameters are known, we get a set of parameters relative to the local coordinate system. Point set Finally, the ball feature center data of each frame satisfies the following coordinate system transformation relationship: ， Since the camera coordinate system is constantly changing when collecting poses, the ball feature center dataset obtained in the collected images other than the initial frame The point clouds are all based on the camera coordinate system after the movement, so it is necessary to transform the coordinate system of the point cloud of the next frame of the interval frame into the camera coordinate system of the previous frame for discussion, and perform the following matrix transformation to the same camera coordinate system: ， The interval between two sets of sphere center sets satisfies the following matrix transformation relationship: ， This becomes a rigid transformation problem between two sets of points, which can be solved by applying the point cloud registration method to obtain the homogeneous transformation matrix of the sphere center point cloud in the next frame image outside the initial frame relative to the sphere center point cloud in the previous frame. ; At the same time, since the pose calibration plate and the target object do not undergo relative transformation during the pose collection process, refer to the following coordinate system transformation relationship: ， The initial pose of the target object is multiplied by the relative pose between interval frames to obtain the true pose of the target object coordinate system relative to the camera coordinate system in each frame. .

10. The method for constructing a 6D pose dataset of texture objects based on a depth camera according to claim 9, characterized in that: In step S6, according to the point cloud model of the target object and its real posture, the method of reprojecting it to each frame image corresponding to the real posture to obtain the posture data set content such as the minimum bounding box and the mask image includes: According to the predicted point set in the target object coordinate system and the obtained target object coordinate system relative to The rigid body transformation between the camera coordinate systems and the internal parameter K information obtained by camera calibration are combined to finally map the real target back to the pixel coordinate system and obtain the mask map of the target. This process mainly involves the transfer between multiple coordinate systems, namely: 。

Images