Method for estimating pose of workpiece, electronic equipment and robot

Abstract

The invention provides a method for estimating the pose of a workpiece, electronic equipment and a robot, and the method comprises the steps: carrying out the plane segmentation of a workpiece point cloud of a target workpiece, and obtaining a segmentation plane set of the point cloud of the target workpiece; and constructing a coordinate system of a target segmentation plane in the segmentation plane set, and determining the pose of a target workpiece on the target segmentation plane based on the coordinate system of the target segmentation plane. The method can overcome environmental influence and accurately extract the workpiece point cloud, can directly construct the workpiece coordinate system on the workpiece point cloud, does not need to construct a workpiece template in advance and depend on a specific workpiece shape and structure, is suitable for pose estimation of workpieces of different types, sizes and shapes, and has good universality.

Description

Technical Field

[0001] The present invention relates to the field of robots, and more specifically to a method for estimating the position and posture of a workpiece, an electronic device and a robot. Background Art

[0002] With the increasing popularity of industrial robots in automated production, it is becoming increasingly important to provide accurate visual guidance for robot operations in areas such as robot automatic sorting and assembly. Using 3D vision to estimate the pose of workpieces can not only help robots identify the position, posture and direction of objects, but also help robots plan the optimal path and action, thereby improving sorting efficiency and accuracy.

[0003] Existing 3D visual pose estimation methods mainly use 3D cameras to obtain RGB images and point clouds of the work scene, then extract the target workpiece mask through the RGB image to obtain the workpiece point cloud, and then align the workpiece point cloud and the workpiece template point cloud to obtain the rigid transformation between the two, thereby realizing the workpiece pose estimation. However, the environment in the disordered sorting scene is complex, and the RGB image is easily affected by factors such as lighting changes and background interference. It is difficult to guarantee the accuracy of the extracted target workpiece point cloud by relying solely on the texture features provided by the image. At the same time, the workpiece poses, sizes and shapes are diverse, and there are different degrees of occlusion between the workpieces, which requires the workpiece to be calibrated in advance or relies on specific shapes and structures. Workpiece pose estimation methods face challenges when dealing with complex disordered sorting scenes.

[0004] The above information disclosed in this background section is only for further understanding of the background of the invention and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0005] The present invention provides a method, electronic device and robot for estimating the pose of a workpiece. In view of the problem that RGB images are easily affected by changes in the illumination of the working scene, background interference, etc., and it is difficult to ensure the accuracy of the point cloud extraction of the target workpiece by relying solely on the texture features of the workpiece, the present invention generates a color point cloud by fusing point cloud data and RGB image data, and then builds a point cloud instance segmentation model based on deep learning technology to learn the structure and texture features of the workpiece, so as to achieve accurate extraction of workpiece point clouds of various shapes and sizes in scene point cloud data under different lighting conditions. In addition, in view of the problem that the types, sizes and shapes of workpieces in disordered sorting scenes are diverse and the template library construction is complex, the present invention can directly construct a workpiece coordinate system on the workpiece point cloud to obtain the pose of the workpiece suitable for robot suction, without the need to calibrate the workpiece in advance and without relying on the specific shape and structure of the workpiece.

[0006] The first aspect of the present invention provides 1. a method for estimating the posture of a workpiece, characterized in that the method comprises the following steps: S1: performing planar segmentation on the workpiece point cloud of the target workpiece to obtain a segmentation plane set of the target workpiece point cloud; S2: constructing a coordinate system of the target segmentation plane in the segmentation plane set, and determining the posture of the target workpiece on the target segmentation plane based on the coordinate system of the target segmentation plane.

[0007] According to an embodiment of the present invention, before step S1, the method further includes: S0: selecting a workpiece point cloud of a target workpiece to be estimated from the instance objects of the workpiece point cloud. Before step S0, the method further includes: S01: generating a color point cloud of the work scene based on the work scene image where the workpiece is located, and preprocessing the color point cloud of the work scene to obtain the workpiece point cloud of the workpiece; S02: inputting the workpiece point cloud into a point cloud instance segmentation model for instance segmentation to obtain the instance object of the workpiece point cloud.

[0008] According to an embodiment of the present invention, the working scene where the workpiece is located is a disordered sorting scene.

[0009] According to an embodiment of the present invention, the target segmentation plane is all the segmentation planes in the segmentation plane set, or the target segmentation plane is all the segmentation planes in the segmentation plane set.

[0010] According to an embodiment of the present invention, in S2, constructing the coordinate system of each segmentation plane in the segmentation plane set includes: S21: determining the center point and normal vector of each target segmentation plane, and constructing the coordinate system of the workpiece on each target segmentation plane based on the center point and normal vector of each target segmentation plane.

[0011] According to an embodiment of the present invention, the S21 also includes: S211: obtaining the center point of each plane in the segmentation plane set, and determining a normal vector with a preset radius with the center point as the center; S212: projecting the point cloud of each plane to a two-dimensional plane formed by the center point and the normal vector, and obtaining the first principal component of the point cloud projected onto the two-dimensional plane, and using the first principal component as the long side of the two-dimensional plane; S213: using the center point as the origin, the long side as the x-axis, and the normal vector as the z-axis, to construct the workpiece coordinate system based on a right-handed system or a left-handed system.

[0012] According to an embodiment of the present invention, in S211, obtaining the center point of each target segmentation plane in the segmentation plane set includes: calculating the centroid point of each target segmentation plane, and selecting the point closest to the centroid point and whose distance is less than a preset threshold as the center point of the plane; or obtaining the center point of the maximum inscribed circle of each target segmentation plane, and selecting the point closest to the center point and whose distance is less than a preset threshold as the center point of the plane; or obtaining the center point of the maximum inscribed rectangle of each target segmentation plane, and selecting the point closest to the center point and whose distance is less than a preset threshold as the center point of the plane.

[0013] According to an embodiment of the present invention, in S213, the workpiece coordinate system is brought into an optimal state by fine-tuning the angle of the z-axis multiple times, and the optimal state is that the long side of the suction cup at the end of the robot's mechanical arm is parallel to the long side of the target workpiece.

[0014] According to an embodiment of the present invention, in said S213, the angle of the z-axis is fine-tuned for multiple times, including: after the workpiece coordinate system rotates around the z-axis at a preset adjustment angle, the following operations are performed: S41: Keeping the center point of the coordinate system and the z-axis unchanged, according to the angle of rotation of the z-axis, calculating the new x-axis and y-axis coordinate axes, and constructing a new coordinate system; S42: Determining whether the rotation direction of the z-axis is correct, judging the change in the length of the point cloud plane in the y-axis direction of the workpiece coordinate system before and after the rotation, if the length becomes shorter, it means that the direction of rotation of the z-axis is correct, otherwise the z-axis needs to be rotated in the opposite direction; S43: Calculating the contour points of each point cloud plane, extracting the new workpiece The point set near the intersection of the x-axis of the coordinate system and the contour point is used to obtain the point set of two contour points; S44: perform straight line fitting on the point set respectively, and calculate the angle between the fitting line and the x-axis of the workpiece coordinate system and the coordinates of the intersection; S45: With the coordinates of the intersection as the center point, the points in the point set of the contour point and the coordinates of the intersection form vectors pointing to the contour point in pairs, and the angle between two adjacent vectors is calculated in a clockwise direction, and the maximum angle is obtained after sorting; S46: Repeat S41-S45, when the angle between the fitting line and the x-axis reaches the first target angle, and the maximum angle reaches the second target angle, it is determined that the posture in the currently constructed workpiece coordinate system is the best posture. The first angle is preferably about 90 degrees, and the second angle is preferably about 180 degrees.

[0015] According to an embodiment of the present invention, the posture is a 6D posture, which represents the posture of the workpiece in 6 degrees of freedom, and the 6 degrees of freedom include: 3 degrees of freedom displacement of the workpiece coordinate system relative to the base coordinate system and 3 degrees of freedom spatial rotation.

[0016] A second aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, wherein when the processor executes the computer program, the method for estimating the workpiece posture described above is implemented.

[0017] A third aspect of the present invention provides a robot that estimates the pose of a workpiece on a workbench using the method described above.

[0018] The solution of the present invention fuses the scene point cloud and RGB image acquired by the 3D camera to generate a color point cloud, and uses the point cloud instance segmentation model to learn the workpiece structure and texture features. It can overcome the environmental influence and accurately extract the workpiece point cloud. At the same time, it can directly construct the workpiece coordinate system on the workpiece point cloud without pre-building a workpiece template and relying on a specific workpiece shape and structure. It is suitable for pose estimation of workpieces of different types, sizes and shapes, and has good versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flow chart showing a method for estimating a workpiece pose according to an example of the present invention.

[0021] Figure 2 Detailed description is a flow chart showing the instance segmentation of a workpiece point cloud according to an example of the present invention.

[0022] Figure 3 is a flow chart showing obtaining a workpiece point set on a workbench according to an embodiment of the present invention.

[0023] Figure 4 is a flow chart showing workpiece posture estimation according to an embodiment of the present invention.

[0024] Figure 5 is a flowchart showing construction of a workpiece coordinate system on each target segmentation plane in a segmentation plane set according to an embodiment of the present invention.

[0025] Figure 6 is a flow chart showing a method for determining an optimal posture of a workpiece according to an embodiment of the present invention.

[0026] Figure 7 is a block diagram showing a device for estimating the pose of a workpiece on a workbench according to an embodiment of the present invention. Specific embodiments

[0027] As used herein, the words "first", "second", etc. may be used to describe elements in exemplary embodiments of the present invention. These words are only used to distinguish one element from another, and the inherent characteristics or order of the corresponding elements, etc. are not limited by the words. Unless otherwise defined, all terms (including technical or scientific terms) used in this article have the same meaning as those commonly understood by ordinary technicians in the technical field to which the present invention belongs. Those terms defined in common dictionaries are interpreted as having the same meaning as the contextual meaning in the relevant technical field, and are not interpreted as having ideal or overly formal meanings, unless they are clearly defined as having such meanings in the present invention.

[0028] It will be understood by those skilled in the art that the apparatus and methods of the present invention described herein and illustrated in the accompanying drawings are non-limiting exemplary embodiments, and the scope of the present invention is limited only by the claims. Features illustrated or described in conjunction with one exemplary embodiment may be combined with features of other embodiments. Such modifications and variations are included within the scope of the present invention.

[0029] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the accompanying drawings, detailed descriptions of related known functions or configurations are omitted to avoid unnecessary obscuration of the technical points of the present invention. In addition, throughout the description, the same reference numerals always refer to the same circuits, modules or units, and for the sake of brevity, repeated descriptions of the same circuits, modules or units are omitted.

[0030] In addition, it should be understood that one or more of the following methods or aspects thereof can be performed by at least one control unit or controller. The terms "control unit", "controller", "control module" or "main control module" can refer to a hardware device including a memory and a processor. The memory or computer readable storage medium is configured to store program instructions, and the processor is specifically configured to execute the program instructions to perform one or more processes that will be further described below. Moreover, it should be understood that, as will be appreciated by those of ordinary skill in the art, the following methods can be performed by including a processor in combination with one or more other components.

[0031] The solution of the present invention firstly fuses the depth image and RGB image of the working scene acquired by the 3D camera to generate a color point cloud, and then uses the point cloud instance segmentation network to learn the structure and texture features of the workpiece, overcomes the influence of the environment and accurately extracts the workpiece point cloud, and then directly constructs the workpiece coordinate system on the extracted target workpiece point cloud without relying on pre-built templates and specific workpiece shapes, so as to achieve the purpose of posture estimation suitable for workpieces of different types, sizes and shapes, and realize intelligent perception of different workpiece postures.

[0032] Figure 1 is a flow chart showing a method for estimating a workpiece pose according to an example of the present invention.

[0033] like Figure 1 As shown, the method for estimating the workpiece pose includes the following steps:

[0034] S1: performing plane segmentation on the workpiece point cloud of the target workpiece to obtain a segmentation plane set of the target workpiece point cloud;

[0035] S2: constructing a coordinate system of each segmentation plane in the segmentation plane set, and determining the position and posture of the target workpiece on each target segmentation plane based on the coordinate system of each segmentation plane.

[0036] According to one or more embodiments of the present invention, before step S1, the method further includes step S0: selecting a workpiece point cloud of a target workpiece to be estimated from instance objects of the workpiece point cloud.

[0037] Before S0, the workpiece pose estimation method includes the following steps:

[0038] S01: generating a color point cloud of a work scene based on an image of a work scene where a workpiece is located, and preprocessing the color point cloud of the work scene to obtain a workpiece point cloud of the workpiece.

[0039] According to one or more embodiments of the present invention, the present invention uses a 3D camera to collect RGB image data and depth image data of a work scene to generate a color point cloud of the work scene, and then uses a pre-built workbench point cloud model to remove the workbench point cloud and background point cloud in the color point cloud of the work scene, retaining the workpiece point cloud on the workbench.

[0040] S02: Inputting the workpiece point cloud into a point cloud instance segmentation model for instance segmentation to obtain instance objects of the workpiece point cloud.

[0041] According to one or more embodiments of the present invention, the present invention creates a point cloud instance segmentation model training and test data set, trains a segmentation model, uses the trained segmentation model to perform instance segmentation on the workpiece point cloud in the scene, and obtains the workpiece point cloud instance object.

[0042] According to one or more embodiments of the present invention, after the present invention segments the workpiece point cloud data into workpiece instance objects, it selects the target workpiece point cloud from the workpiece instance objects in combination with actual needs, then segments the point cloud plane from the workpiece point cloud, and constructs a workpiece coordinate system to obtain the workpiece pose.

[0043] In addition, after S2, the workpiece pose estimation method also includes:

[0044] S3: Perform over-limit judgment and collision detection screening on the estimated pose of the target workpiece to obtain the final estimated pose.

[0045] According to one or more embodiments of the present invention, after obtaining the position of the workpiece on the splitting plane, some of the postures may cause the robot's mechanical arm to issue an over-limit alarm, or may cause the path planned by the mechanical arm to collide with an obstacle. Therefore, it is necessary to perform over-limit judgment and collision detection on the posture to avoid the above situation.

[0046] The workpiece point cloud generation, workpiece point cloud instance segmentation, and workpiece pose estimation of the workpiece pose estimation solution of the present invention are described in detail below.

[0047] Workpiece point cloud generation:

[0048] In the S01, it is necessary to generate a color point cloud for the work scene of the workpiece and pre-process the color point cloud. Specifically, the S01 includes:

[0049] S011: Generate a color point cloud of the working scene. Use a 3D camera to obtain an RGB image and a depth image of the working scene, perform data conversion on the RGB image and the depth image to generate an RGBD image, and generate a color point cloud of the working scene based on the RGBD image.

[0050] S012: Preprocessing of the color point cloud of the working scene. The model of the working platform is pre-built, and for the generated color point cloud of the working scene, the point cloud within the working platform is regarded as the workpiece point cloud and retained, and the work platform model and the point cloud outside it are regarded as the background point cloud and removed or eliminated, so as to obtain the preprocessed workpiece point cloud data.

[0051] Workpiece point cloud instance segmentation:

[0052] According to one or more embodiments of the present invention, the cloud instance segmentation model in S02 requires data collection and data production and training of the point cloud instance segmentation model.

[0053] Data collection and data production include:

[0054] According to one or more embodiments of the present invention, the specific data collection and data production process includes: collecting and generating color point cloud data of a certain number of work scenes, manually annotating workpiece instance labels based on the workpiece point cloud data obtained after preprocessing, wherein the labels of the workpiece instances need to be divided according to the actual types of workpieces in the sorting scene. For example, in the express sorting scene, the workpieces are mainly different types of express delivery, so the workpiece instance labels can be divided into three categories: boxes, bags and others, and then combined with point cloud data enhancement and other methods to produce model training and test data sets.

[0055] According to one or more embodiments of the present invention, whether the collected point cloud data contains other objects needs to be determined according to the type of workpiece being sorted. For example, in an express sorting scenario, generally the workpiece is only an express parcel. However, if the parcel is damaged and items are mixed in the express parcel, the collected point cloud at this time will include the point cloud of items other than the parcel. If the workpiece being sorted is a standard industrial component, the collected point cloud at this time will only include the workpiece point cloud.

[0056] Among them, the point cloud instance segmentation model training includes:

[0057] The present invention is based on the point cloud instance segmentation model required for HAI S network training, wherein the HAI S network is a voxel-based model, which consists of four parts: a point-by-point prediction network, a point aggregation module, a point set aggregation module, and an intra-instance prediction network. According to one or more embodiments of the present invention, the point cloud instance segmentation model training is completed on two Geoforce 3090 GPUs. During training, the batch size is set to 16, with a total of 120 iterations, the initial learning rate is set to 0.001, and the cosine annealing function is used to decay the learning rate. The voxel size is set to 0.02m based on empirical values, and the point aggregation bandwidth is set to 0.03m. The HAIS network can achieve efficient reasoning on the point cloud and meet the application requirements of workpiece point cloud instance segmentation.

[0058] Figure 2 Detailed description is a flow chart showing the instance segmentation of a workpiece point cloud according to an example of the present invention.

[0059] like Figure 2 As shown, the point cloud instance segmentation process includes the following steps:

[0060] S021: Input the acquired workpiece point cloud data into a point-by-point prediction network to predict the point-by-point semantic label and center offset vector of the point cloud workpiece;

[0061] S022: Segmenting the workpiece point cloud in the work scene based on the point-by-point semantic labels and the center offset to obtain a point set of the workpiece on the workbench;

[0062] S023: dividing the point set of the workpiece into a main instance object and fragmented secondary instance objects around the workpiece, and aggregating the main instance object and the secondary instance objects to obtain a workpiece instance;

[0063] S024: removing noise points generated by aggregation by smoothing the workpiece instance to obtain an instance object of the workpiece point cloud.

[0064] Specifically, in S021, it includes: inputting the workpiece point cloud data obtained by preprocessing in S01 into the trained point-by-point prediction network, extracting the structure and texture features of the workpiece from the point cloud and predicting the point-by-point semantic label and the center offset vector Δx .

[0065] Figure 3 is a flow chart showing obtaining a workpiece point set on a workbench according to an embodiment of the present invention.

[0066] Specifically, in S022, it includes:

[0067] S0221: According to the point-by-point center movement vector, each point in the workpiece point cloud is moved toward the center of the point instance, so that the points of the same instance are close to each other. Specifically, the semantic label and center offset vector Δ of each point obtained by the point-by-point prediction network are x Based on this, the point aggregation module is used to move the vector point by point Each point Move towards the center of its instance so that the points of the same instance are close to each other in space. The coordinates of the points after the move are calculated as:

[0068]

[0069] in, is the coordinate of the point after the move. The point-by-point prediction network will give each point an offset vector relative to the center of its instance. Each point moves toward the center of its instance according to the offset vector predicted by the network to achieve the goal of being close to each other in space. Its quantitative index can be considered as the center offset vector.

[0070] S0222: define each foreground point in the workpiece point cloud as a node, establish edges between nodes with consistent semantic labels and distances less than the point aggregation bandwidth, and segment the workpiece point cloud to obtain a point set of the workpiece on the workbench by traversing all nodes of the workpiece point cloud and connecting the edges. After the traversal, the same workpiece instance is connected by the established edges, and points between different workpiece instances are separated from each other.

[0071] Specifically, each foreground point is regarded as a node, and an edge is established between nodes with consistent semantic labels and a distance less than the point aggregation bandwidth. After traversing all nodes, the points of the same instance are connected by the established edges, and the points between different instances are separated from each other. Thus, the point cloud is divided into multiple point sets, forming a preliminary instance segmentation prediction result of the workpiece point cloud, and completing point-level aggregation. The workpiece point cloud data obtained by the preprocessing in S01 may contain noise points and other object point clouds. Here, the foreground point refers to the real workpiece point cloud, and the corresponding background points refer to points other than the workpiece point cloud. Taking the express sorting scene as an example, the foreground point refers to the "box" and "bag" express point cloud, and the background point refers to the "other" type point cloud. In addition, the points in the same instance are connected to each other by the established edges, and the points between different instances are separated from each other, and there is no connecting edge. By judging whether there are edges connecting the point cloud points, different instance point sets are distinguished.

[0072] S023 includes: dividing the point set of the workpiece into a primary instance object and a secondary instance object, and merging some of the secondary instance objects in the workpiece point set into the primary instance object according to the preset semantic label and dynamic clustering bandwidth condition.

[0073] Specifically, the point set of the workpiece is further aggregated and divided into primary instance objects and secondary instance objects according to the number of points. For secondary instance m, the primary instance n that is closest to it and has the same semantic label is found. If the distance between primary instance n and secondary instance m is less than the dynamic clustering bandwidth, the secondary instance is merged into the primary instance. The dynamic clustering bandwidth is defined as:

[0074] r set =max(r size ,r cls ) (2)

[0075]

[0076] Among them, r size is the dynamic clustering bandwidth related to the size of the master instance n, r cl s is the average dynamic clustering bandwidth of the class instances of the master instance n, calculated from the training data, α is the scaling factor, The size of the primary instance n.

[0077] According to one or more embodiments of the present invention, the secondary instance that meets the dynamic clustering bandwidth condition is merged with the primary instance to form a higher quality artifact instance segmentation result.

[0078] According to one or more embodiments of the present invention, step S024 further inputs the workpiece instance point cloud object obtained after point set aggregation into the intra-instance prediction network to learn the internal features of the workpiece instance, eliminate the incorrectly merged point sets, and output the final workpiece instance segmentation object.

[0079] Workpiece pose estimation

[0080] Figure 4 is a flow chart showing workpiece posture estimation according to an embodiment of the present invention.

[0081] like Figure 4 As shown, workpiece posture estimation S2 includes the following steps:

[0082] After preprocessing the workpiece point cloud of the target workpiece, the following steps are performed:

[0083] S20: performing plane segmentation on the preprocessed workpiece point cloud of the target workpiece by using a random sampling consistency algorithm to obtain a segmentation plane set;

[0084] S21: determining the center point and normal vector of each target segmentation plane in the segmentation plane set, and constructing a coordinate system of the workpiece on each target segmentation plane based on the center point and normal vector of each target segmentation plane;

[0085] S22: Determine the position and posture of the target workpiece on each target segmentation plane based on the coordinate system of the workpiece on each target segmentation plane.

[0086] Specifically, in the preprocessing of the workpiece point cloud of the target workpiece, according to the needs of the robot to absorb the workpiece, such as the need to preferentially absorb the largest workpiece or the smallest workpiece, the target workpiece point cloud is extracted from the prediction result of the point cloud instance segmentation model. And the extracted workpiece point cloud of the target workpiece is preprocessed, and the preprocessing includes: for the extracted target workpiece point cloud, data preprocessing is performed on it, the number of point clouds is reduced by downsampling the point cloud grid, the discrete point clouds around the workpiece are removed by the point cloud statistical filtering algorithm, and then the point cloud smoothing algorithm is used to optimize the point cloud quality.

[0087] Specifically, in S20, for the pre-processed target workpiece point cloud, based on the random sampling consistency algorithm, the possible planes in the workpiece are segmented, and the inscribed circle of the plane is calculated, and the point cloud planes that satisfy the inscribed circle radius greater than the suction cup are listed as segmentation planes. The obtained segmentation plane set is: plane set P = {p 1 ,...,p n}.

[0088] Figure 5 is a flowchart showing construction of a workpiece coordinate system on each target segmentation plane in a segmentation plane set according to an embodiment of the present invention.

[0089] As shown in the figure, step S21 includes:

[0090] S211: For each target segmentation plane in the segmentation plane set, calculate the centroid of the plane, select a point that is closest to the centroid and whose distance is less than a preset threshold as the center point of the plane, and determine a normal vector of a preset radius with the center point as the center;

[0091] S212: Projecting the point cloud of the plane onto a two-dimensional plane formed by the center point and the normal vector, and calculating a first principal component of the point cloud projected onto the two-dimensional plane using a principal component analysis algorithm, and using the first principal component as the long side of the two-dimensional plane;

[0092] S213: Taking the center point as the origin, the long side as the x-axis, and the normal vector as the z-axis, construct the workpiece coordinate system based on a right-handed system or a left-handed system.

[0093] Specifically, in S211, for the splitting plane p i, calculate its centroid point, and select the point on the point cloud plane that is closest to the centroid point and whose distance is less than the threshold as the center point of the segmentation plane Taking this point as the center, calculate the normal vector of the neighborhood plane with a radius of r as the normal vector of the plane

[0094] According to one or more embodiments of the present invention, in S211, the center point may not be the center point of the largest inscribed circle of the point cloud plane or the center point of the largest inscribed rectangle. The center point is calculated preferentially because the center of mass is generally located near the center of the candidate point cloud plane. Finally, after the plane pose is sent to the robotic arm, the robotic arm absorbs the center of the workpiece plane.

[0095] Specifically, in S212, based on the segmentation plane p i The center point and the normal vector Project the segmentation plane point cloud to the center point and the normal vector For the segmented plane point cloud projected onto the two-dimensional plane, the principal component analysis algorithm (PCA) is used to calculate the first principal component of the candidate point cloud plane as the long side of the plane.

[0096] Specifically, in S213, the split plane p i The center point is used as the origin of the workpiece coordinate system, and the long side As the x-axis, the normal vector As the z-axis, the workpiece coordinate system is constructed based on the right-hand system or the left-hand system.

[0097] According to one or more embodiments of the present invention, for the constructed workpiece coordinate system, to determine whether it is in the best posture, the workpiece coordinate system is iteratively fine-tuned multiple times by rotating a small angle around the z-axis (iteration refers to rotating a small angle around the z-axis and rotating the z-axis multiple times) to make the workpiece coordinate system reach the best state. The specific steps include: after the workpiece coordinate system is initially obtained, the relative position of the end tool of the robot arm and the workpiece during the suction stage can be known, and whether it is the best posture is determined based on the relative position. When the end tool of the robot arm is a suction cup, the long side of the suction cup should be parallel to the long side of the workpiece. By determining whether the long side of the suction cup is parallel to the long side of the workpiece, it can be determined whether the currently constructed workpiece coordinate system is the best posture for the workpiece.

[0098] Figure 6 is a flow chart showing a method for determining an optimal posture of a workpiece according to an embodiment of the present invention.

[0099] After the workpiece coordinate system rotates around the z-axis by the preset fine-tuning angle:

[0100] S41: The center point and z-axis of the workpiece coordinate system remain unchanged (normal vector). According to the rotation angle, the new x-axis and y-axis coordinate axes are calculated to construct a new workpiece coordinate system.

[0101] S42: Determine whether the rotation direction of the z-axis is correct (judgment is sufficient once in the first iteration), and determine the length change of the point cloud plane in the y-axis direction of the workpiece coordinate system before and after the rotation. If the length becomes shorter, it means that the rotation direction of the z-axis is correct, otherwise the z-axis needs to be rotated in the opposite direction;

[0102] S43: Calculate the plane contour points of the point cloud according to the AC (Angle Cr iter ion) algorithm (only need to be calculated once in the first iteration), extract the point set near the intersection of the x-axis of the new workpiece coordinate system and the contour point, and obtain the point set of two contour points;

[0103] S44: performing straight line fitting on the point sets respectively, and calculating the angle between the fitting straight line and the x-axis of the workpiece coordinate system and the coordinates of the intersection point;

[0104] S45: With the intersection coordinate as the center point, the points in the contour point set and the intersection coordinate form vectors pointing to the contour point in pairs, and the angle between two adjacent vectors is calculated in a clockwise direction. After sorting, the maximum angle is obtained.

[0105] S46: Repeat the above steps S41-S45. When the angle between the fitting line and the x-axis reaches the first angle of about 90 degrees (a threshold value can be set, such as plus or minus 5 degrees), and the maximum angle reaches the second angle of about 180 degrees (a threshold value can be set, such as plus or minus 5 degrees), the posture expressed by the currently constructed workpiece coordinate system is considered to be the best posture. Among them, the first angle is preferably about 90 degrees (a threshold value can be set, such as plus or minus 5 degrees); the second angle is preferably: about 180 degrees (a threshold value can be set, such as plus or minus 5 degrees).

[0106] Specifically, according to one or more embodiments of the present invention, the plane center point p i center The constructed workpiece coordinate system is converted to the robot base coordinate system, and the Euler angle of the segmentation plane is calculated based on the constructed workpiece coordinate system to obtain the pose of the segmentation plane. Finally, the 6D pose of each segmentation plane is obtained, and the pose that best suits the suction plane is output as the final pose estimation result of the target workpiece, combined with the robot's optimal path planning and suction action.

[0107] According to one or more embodiments of the present invention, in an unordered sorting scenario, the 6D pose of the workpiece must be given, and the 3D pose cannot meet the sorting requirements. 6D refers to 6 degrees of freedom, representing 3 degrees of freedom of displacement (also called translation) and 3 degrees of freedom of spatial rotation, which are collectively expressed as pose. Among them, pose refers to the displacement and rotation transformation between two coordinate systems. The 6D pose of the workpiece referred to in the present invention refers to the translation and rotation of the workpiece coordinate system relative to the camera coordinate system.

[0108] According to one or more embodiments of the present invention, in posture screening, some postures in the split plane may cause the robot arm to have an over-limit alarm, or may cause the path planned by the robot arm to collide with an obstacle. Therefore, it is necessary to perform over-limit judgment and collision detection on the posture to avoid the above situation.

[0109] Figure 7 is a block diagram showing a device for estimating the pose of a workpiece on a workbench according to an embodiment of the present invention.

[0110] According to one or more embodiments of the present invention, the present invention also provides a device for estimating the posture of a workpiece on a workbench, the device comprising: an image processing device, configured to: generate a color point cloud of the work scene based on an image of the work scene where the workpiece is located, and pre-process the color point cloud of the work scene to obtain a workpiece point cloud; a point cloud instance segmentation model, configured to: perform instance segmentation on the workpiece point cloud to obtain an instance object of the workpiece point cloud; a posture estimation device, configured to: select a workpiece point cloud of a target workpiece to be estimated from the workpiece instance object, and perform plane segmentation on the workpiece point cloud of the target workpiece, and estimate the posture of the target workpiece by constructing a coordinate system of each segmentation plane.

[0111] According to one or more embodiments of the present invention, the present invention further provides an electronic device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to implement the above-mentioned Figure 1 The method for estimating the workpiece posture.

[0112] According to one or more embodiments of the present invention, the present invention further provides a robot, using Figure 1 The method described estimates the pose of the workpiece on the workbench.

[0113] According to one or more embodiments of the present invention, the control logic of the workpiece pose estimation device or apparatus of the present invention can use the encoded instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium (e.g., a hard disk drive, a flash memory, a read-only memory, an optical disk, a digital versatile disk, a cache, a random access memory and / or any other storage device or storage disk) to implement the processing constructed by the SDK as described above in the present invention, and store information for any time period (e.g., an extended time period, permanent, a short-term instance, a temporary cache and / or an information cache) in a non-transitory computer and / or machine readable medium. As used herein, the term "non-transitory computer readable medium" is explicitly defined to include any type of computer readable storage device and / or storage disk, and excludes propagation signals and excludes transmission media.

[0114] According to one or more embodiments of the present invention, the control circuit (control logic, main control system or control module) of the workpiece pose estimation device or apparatus of the present invention may include one or more processors, and may also include non-temporary computer-readable media inside. Specifically, the device (main control system or control module) may include a microcontroller MCU, which is used to implement various operations and implement multiple functions in workpiece pose estimation. The processor used to implement the abnormal detection of the blast furnace tuyere may be such as but not limited to one or more single-core or multi-core processors. (One or more) processors may include any combination of general-purpose processors and special-purpose processors (for example, graphics processors, application processors, etc.). The processor may be coupled thereto and / or may include a computer memory / storage device, and may be configured to execute instructions stored in the memory / storage device to implement various applications and / or operating systems running on the controller in the present invention.

[0115] The following are further examples of the present invention:

[0116] Example 1. A method for estimating the pose of a workpiece, characterized in that the method comprises the following steps: S1: performing planar segmentation on the workpiece point cloud of the target workpiece to obtain a segmentation plane set of the target workpiece point cloud; S2: constructing a coordinate system of the target segmentation plane in the segmentation plane set, and determining the pose of the target workpiece on the target segmentation plane based on the coordinate system of the target segmentation plane.

[0117] Example 2: According to the method of Example 1, before S1, the method further includes: S0: selecting the workpiece point cloud of the target workpiece to be estimated from the instance objects of the workpiece point cloud. Before step S0, the method further includes: S01: generating a color point cloud of the work scene based on the work scene image where the workpiece is located, and preprocessing the color point cloud of the work scene to obtain the workpiece point cloud of the workpiece; S02: inputting the workpiece point cloud into the point cloud instance segmentation model for instance segmentation to obtain the instance object of the workpiece point cloud.

[0118] Example 3. The method according to Example 1, wherein, in S01, the image of the working scene is an RGB image and a depth image of the working scene.

[0119] Example 4. The method according to Example 3, wherein S01 includes: converting the RGB image and the depth image to generate an RGBD image, and generating a color point cloud of the working scene based on the RGBD image.

[0120] Example 5. The method according to Example 4, wherein, in S01, the preprocessing includes: based on a pre-built workbench point cloud model, removing the workbench point cloud and the background point cloud in the colored point cloud to obtain the workpiece point cloud of the workpiece.

[0121] Example 6. According to the method described in Example 1, S02 includes: labeling instance labels of the workpiece based on the data of the workpiece point cloud, and generating training and test data sets in the point cloud instance segmentation model according to the instance labels of the workpiece.

[0122] Example 7. The method according to Example 6, wherein in S02, the point cloud instance segmentation model is trained based on the HAIS network. Example 8. The method according to Example 6, wherein in S02, the point cloud instance segmentation model is trained based on the HAIS network, including:

[0123] S021: Input the acquired workpiece point cloud data into a point-by-point prediction network to predict the point-by-point semantic label and center offset vector of the point cloud workpiece;

[0124] S022: Segmenting the workpiece point cloud in the work scene based on the point-by-point semantic labels and the center offset to obtain a point set of the workpiece on the workbench;

[0125] S023: dividing the point set of the workpiece into a main instance object and fragmented secondary instance objects around the workpiece, and aggregating the main instance object and the secondary instance objects to obtain a workpiece instance;

[0126] S024: removing noise points generated by aggregation by smoothing the workpiece instance to obtain an instance object of the workpiece point cloud.

[0127] Example 9. The method according to Example 8, wherein in S022, further comprising:

[0128] S0221: according to the point-by-point center movement vector, each point in the workpiece point cloud is moved toward the center of the point instance, so that the points of the same instance are close to each other;

[0129] S0222: Define each foreground point in the workpiece point cloud as a node, establish edges between nodes with consistent semantic labels and distances less than the point aggregation bandwidth, and segment the workpiece point cloud by traversing all nodes of the workpiece point cloud and connecting the edges to obtain a point set of the workpiece on the workbench; wherein, after the traversal, the same workpiece instance is connected by the established edges, and points between different workpiece instances are separated from each other.

[0130] Example 10. The method according to Example 9, wherein in S023, further comprising:

[0131] The point set of the workpiece is divided into a primary instance object and a secondary instance object, and according to the preset semantic label and dynamic clustering bandwidth condition, some secondary instance objects in the workpiece point set are merged into the primary instance object.

[0132] Example 11. The method according to Example 9, wherein in S023, the preset semantic label and dynamic clustering bandwidth conditions are:

[0133] Among all secondary instance objects, find the primary instance object that is closest to it and has the same semantic label. If the distance between the primary instance object and the secondary instance object is less than the dynamic clustering bandwidth, merge the secondary instance object into the primary instance object.

[0134] Example 12. The method of Example 11, wherein the dynamic clustering bandwidth is defined as:

[0135] r set =max(r size ,r cls ),in

[0136] Among them, r si ze is the dynamic clustering bandwidth related to the size of the main instance object n, r cls is the average dynamic clustering bandwidth of the instance objects in the category where the main instance object n belongs, α is the scaling factor, The size of the primary instance n.

[0137] Example 13. The method according to Example 1, wherein S2 comprises:

[0138] S201: preprocessing the workpiece point cloud of the target workpiece;

[0139] S202: performing plane segmentation on the preprocessed workpiece point cloud of the target workpiece by using a random sampling consistency algorithm to obtain a segmentation plane set;

[0140] S21: determining the center point and normal vector of each target segmentation plane in the segmentation plane set, and constructing a coordinate system of the workpiece on each target segmentation plane based on the center point and normal vector of each target segmentation plane, wherein the target segmentation plane is all the segmentation planes in the segmentation plane set, or the target segmentation plane is all the segmentation planes in the segmentation plane set;

[0141] S22: Determine the position and posture of the target workpiece on each target segmentation plane based on the coordinate system of the workpiece on each target segmentation plane.

[0142] Example 14. The method according to Example 13, wherein in S21, further comprising:

[0143] S211: For each target segmentation plane in the segmentation plane set, calculate the centroid of the plane, select a point that is closest to the centroid and whose distance is less than a preset threshold as the center point of the plane, and determine a normal vector of a preset radius with the center point as the center;

[0144] S212: Projecting the point cloud of the plane onto a two-dimensional plane formed by the center point and the normal vector, and calculating a first principal component of the point cloud projected onto the two-dimensional plane using a principal component analysis algorithm, and using the first principal component as the long side of the two-dimensional plane;

[0145] S213: Taking the center point as the origin, the long side as the x-axis, and the normal vector as the z-axis, construct the workpiece coordinate system based on a right-handed system or a left-handed system.

[0146] Example 15. The method according to Example 13, wherein in S22, the workpiece coordinate system is converted to the base coordinate system of the robot, and based on the constructed workpiece coordinate system, the position and posture of the target workpiece on the plane is obtained by calculating the Euler angle of the plane on the base coordinate system.

[0147] Example 16. The method according to Example 15, wherein, in S22, it also includes: based on the robot's optimal path planning and suction action conditions, outputting the final posture of the robot processing the workpiece on the plane.

[0148] Example 17. A method according to Example 15, wherein the posture is a 6D posture of the plane, the 6D posture represents the posture of the workpiece in 6 degrees of freedom, and the 6-degree-of-freedom workpiece coordinate system has 3 degrees of freedom of displacement and 3 degrees of freedom of spatial rotation relative to a base coordinate system.

[0149] Example 18. The method according to Example 1, wherein after S2, the method further comprises:

[0150] S3: Perform over-limit judgment and collision detection screening on the estimated pose of the target workpiece to obtain the final estimated pose.

[0151] Example 19. A method according to Example 18, wherein in S213, the workpiece coordinate system is brought into an optimal state by fine-tuning the angle of the z-axis multiple times, wherein the optimal state is that the long side of the suction cup at the end of the robot's robotic arm is parallel to the long side of the target workpiece.

[0152] Example 20. The method according to Example 19, wherein in S213, fine-tuning the angle of the z-axis for multiple times comprises: performing the following operations after the workpiece coordinate system rotates around the z-axis by a preset adjustment angle:

[0153] S41: Keeping the center point of the coordinate system and the z-axis unchanged, calculating new x-axis and y-axis coordinate axes according to the rotation angle of the z-axis, and constructing a new coordinate system;

[0154] S42: Determine whether the rotation direction of the z-axis is correct, and judge the change in the length of the point cloud plane in the y-axis direction of the workpiece coordinate system before and after the rotation. If the length becomes shorter, it means that the rotation direction of the z-axis is correct, otherwise the z-axis needs to be rotated in the opposite direction;

[0155] S43: Calculate the contour points of each point cloud plane, extract the point set near the intersection of the x-axis of the new workpiece coordinate system and the contour point, and obtain the point set of two contour points;

[0156] S44: performing straight line fitting on the point sets respectively, and calculating the angle between the fitting straight line and the x-axis of the workpiece coordinate system and the coordinates of the intersection point;

[0157] S45: With the coordinates of the intersection point as the center point, points in the point set of the contour point and the coordinates of the intersection point form vectors pointing to the contour point in pairs, and the angle between two adjacent vectors is calculated in a clockwise direction, and the maximum angle is obtained after sorting.

[0158] S46: Repeat S41-S45, when the angle between the fitting straight line and the x-axis reaches 90 degrees, and the maximum angle reaches 180 degrees, determine that the posture in the currently constructed workpiece coordinate system is the optimal posture.

[0159] Example 21. A device for estimating the pose of a workpiece on a workbench, characterized in that the device comprises:

[0160] The image processing device is configured to: generate a color point cloud of the work scene based on an image of the work scene where the workpiece is located, and pre-process the color point cloud of the work scene to obtain a workpiece point cloud;

[0161] The point cloud instance segmentation model is configured to: perform instance segmentation on the workpiece point cloud to obtain instance objects of the workpiece point cloud;

[0162] The pose estimation device is configured to: select a workpiece point cloud of a target workpiece to be estimated from a workpiece instance object, perform plane segmentation on the workpiece point cloud of the target workpiece, and estimate the pose of the target workpiece by constructing a coordinate system of each segmentation plane.

[0163] Example 22. An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, wherein when the processor executes the computer program, the method for estimating the workpiece posture as described above is implemented.

[0164] Example 22. A robot estimates the pose of a workpiece on a workbench using the above method.

[0165] The accompanying drawings and detailed description of the present invention referred to above as examples of the present invention are used to explain the present invention, but do not limit the meaning or scope of the present invention described in the claims. Therefore, those skilled in the art can easily realize modifications from the above description. In addition, those skilled in the art can delete some components described herein without degrading performance, or can add other components to improve performance. In addition, those skilled in the art can change the order of the steps of the method described herein according to the environment of the process or equipment. Therefore, the scope of the present invention should not be determined by the embodiments described above, but by the claims and their equivalents.

[0166] While the invention has been described in connection with what are presently considered to be capable of implementation, it is to be understood that the invention is not limited to the disclosed embodiments, but on the contrary is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for estimating the pose of a workpiece, characterized in that: The method comprises the following steps: S1: performing plane segmentation on the workpiece point cloud of the target workpiece to obtain a segmentation plane set of the target workpiece point cloud; S2: constructing a coordinate system of a target segmentation plane in the segmentation plane set, and determining a position and posture of a target workpiece on the target segmentation plane based on the coordinate system of the target segmentation plane.

2. The method according to claim 1, characterized in that Wherein before S1, it also includes: S0: Selecting the workpiece point cloud of the target workpiece to be estimated from the instance objects of the workpiece point cloud.

3. The method according to claim 1, characterized in that Wherein the S0 front also includes: S01: generating a color point cloud of the work scene based on an image of the work scene where the workpiece is located, and preprocessing the color point cloud of the work scene to obtain a workpiece point cloud of the workpiece; S02: Inputting the workpiece point cloud into a point cloud instance segmentation model for instance segmentation to obtain instance objects of the workpiece point cloud.

4. The method according to claim 1, characterized in that in, The work scene where the workpiece is located is a disordered sorting scene.

5. The method according to claim 1, characterized in that in, The target segmentation planes are all segmentation planes in the segmentation plane set, or The target segmentation plane is a segmentation plane whose radius of the inscribed circle in the segmentation plane set meets a set condition.

6. The method according to claim 1, wherein: In S2, constructing a coordinate system of the target segmentation plane in the segmentation plane set includes: S21: Determine the center point and normal vector of each target segmentation plane, and construct a coordinate system of the workpiece on each target segmentation plane based on the center point and normal vector of each target segmentation plane.

7. The method according to claim 6, characterized in that Wherein said S21 further comprises: S211: Obtain the center point of each target segmentation plane in the segmentation plane set, and determine a normal vector of a preset radius with the center point as the center; S212: Projecting the point cloud of each target segmentation plane onto a two-dimensional plane formed by the center point and the normal vector, obtaining a first principal component of the point cloud projected onto the two-dimensional plane, and using the first principal component as the long side of the two-dimensional plane; S213: Taking the center point as the origin, the long side as the x-axis, and the normal vector as the z-axis, construct the workpiece coordinate system based on a right-handed system or a left-handed system.

8. The method according to claim 7, characterized in that in, In S211, obtaining the center point of each target segmentation plane in the segmentation plane set includes: Calculate the centroid of each target segmentation plane, and select the point closest to the centroid and whose distance is less than a preset threshold as the center point of the plane; or Obtain the center point of the largest inscribed circle of each target segmentation plane, and select the point closest to the center point and whose distance is less than a preset threshold as the center point of the plane; or Get the center point of the largest inscribed rectangle of each target segmentation plane, and select the point closest to the center point and whose distance is less than a preset threshold as the center point of the plane.

9. The method according to claim 7, wherein in S213, the workpiece coordinate system is brought into an optimal state by fine-tuning the angle of the z-axis for multiple times, and the optimal state is that the long side of the suction cup at the end of the robot's mechanical arm is parallel to the long side of the target workpiece.

10. The method according to claim 9, wherein in S213, fine-tuning the angle of the z-axis for multiple times comprises: After the workpiece coordinate system rotates around the z-axis to the preset adjustment angle, perform the following operations: S41: Keeping the center point of the coordinate system and the z-axis unchanged, calculating new x-axis and y-axis coordinate axes according to the rotation angle of the z-axis, and constructing a new coordinate system; S42: Determine whether the rotation direction of the z-axis is correct, and judge the change in the length of the point cloud plane in the y-axis direction of the workpiece coordinate system before and after the rotation. If the length becomes shorter, it means that the rotation direction of the z-axis is correct, otherwise the z-axis needs to be rotated in the opposite direction; S43: Calculate the contour points of each point cloud plane, extract the point set near the intersection of the x-axis of the new workpiece coordinate system and the contour point, and obtain the point set of two contour points; S44: performing straight line fitting on the point sets respectively, and calculating the angle between the fitting straight line and the x-axis of the workpiece coordinate system and the coordinates of the intersection point; S45: With the coordinates of the intersection point as the center point, points in the point set of the contour point and the coordinates of the intersection point form vectors pointing to the contour point in pairs, and the angle between two adjacent vectors is calculated in a clockwise direction, and the maximum angle is obtained after sorting; S46: Repeat S41-S45, when the angle between the fitting straight line and the x-axis reaches the first target angle, and the maximum angle reaches the second target angle, determine that the pose in the currently constructed workpiece coordinate system is the optimal pose.

11. The method according to claim 1, characterized in that: in, The posture is a 6D posture, which represents the posture of the workpiece in 6 degrees of freedom, and the 6 degrees of freedom include: a displacement of 3 degrees of freedom of the workpiece coordinate system relative to the base coordinate system and a spatial rotation of 3 degrees of freedom.

12. An electronic device comprising a memory and a processor, wherein the memory is used to store a computer program, wherein: When the processor executes the computer program, the method for estimating the workpiece posture according to any one of claims 1 to 11 is implemented.

13. A robot, using the method according to any one of claims 1 to 11 to estimate the pose of a workpiece on a workbench.

Images