Workpiece pose estimation method, apparatus, device, medium, and product

Abstract

The application discloses a workpiece pose estimation method, device, equipment, medium and product. The method comprises the following steps: acquiring a first shooting image of a plurality of workpieces in a disorderly stack, wherein the first shooting image comprises a first depth image and a first RGB image; determining first target point cloud data of a target workpiece according to the first depth image and the first RGB image; calculating a grabbing position and a grabbing angle of a mechanical arm based on the first target point cloud data of the target workpiece; grabbing the target workpiece to a transfer table according to the grabbing position and the grabbing angle of the mechanical arm, and acquiring a second shooting image, wherein the second shooting image comprises a second depth image and a second RGB image; obtaining second target point cloud data and a first pose of the target workpiece according to the second depth image and the second RGB image; and performing pose estimation according to the first pose, the second target point cloud data and preset point cloud data to obtain a target pose of the target workpiece. The accuracy of workpiece pose estimation in a stacking scene can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a workpiece pose estimation method, device, equipment, medium and product. BACKGROUND

[0002] With the development of scientific level and the continuous transformation and upgrading of manufacturing industry, machine vision is more and more widely used in the field of intelligent manufacturing represented by industrial robots. As a typical application scenario in industrial production, unordered picking uses a mechanical arm to replace manual work to complete the task of picking and assembling target objects, which can not only liberate labor, but also optimize production efficiency. In the process of unordered picking of the mechanical arm, a high-efficiency and stable pose estimation algorithm is very important. In recent years, due to the popularity of 3D vision, many 6D pose estimation algorithms have been proposed, such as deep learning algorithms. However, in the face of complex stacking scenes, these algorithms still have some problems. For example, in a complex industrial scene, due to the changeable stacking scene of workpieces and the occlusion between workpieces, the accuracy of workpiece pose estimation is poor. SUMMARY

[0003] The embodiments of the application provide a workpiece pose estimation method, device, equipment, medium and product, which can improve the accuracy of workpiece pose estimation.

[0004] In a first aspect, the embodiments of the application provide a workpiece pose estimation method, which comprises:

[0005] obtaining a first shooting image of a plurality of unordered stacked workpieces, the first shooting image comprising a first depth image and a first RGB image of the plurality of workpieces;

[0006] determining first target point cloud data of a target workpiece according to the first depth image and the first RGB image, the target workpiece being a workpiece located at the uppermost layer in the plurality of unordered stacked workpieces;

[0007] calculating a grabbing position and a grabbing angle of a mechanical arm based on the first target point cloud data of the target workpiece;

[0008] grabbing the target workpiece to a transfer table according to the grabbing position and the grabbing angle of the mechanical arm, and obtaining a second shooting image, the second shooting image comprising a second depth image and a second RGB image of the target workpiece;

[0009] obtaining second target point cloud data of the target workpiece and a first pose of the target workpiece according to the second depth image and the second RGB image;

[0010] According to the first pose of the target workpiece, the second target point cloud data, and preset point cloud data, pose estimation is performed on the target workpiece, to obtain a target pose of the target workpiece.

[0011] In an embodiment of the present application, the first target point cloud data of the target workpiece is determined according to the first depth image and the first RGB image, including:

[0012] The first depth image and the first RGB image are preprocessed to obtain a workpiece mask.

[0013] The first workpiece point cloud data is segmented according to the workpiece mask to obtain second workpiece point cloud data, wherein the first workpiece point cloud data is obtained by point cloud conversion based on the first depth image.

[0014] The second workpiece point cloud data is analyzed to determine the first target point cloud data of the target workpiece.

[0015] In an embodiment of the present application, the first depth image and the first RGB image are preprocessed to obtain a workpiece mask, including:

[0016] The first depth image and the first RGB image are aligned to obtain first aligned image data.

[0017] The first aligned image data is image segmented to obtain the workpiece mask.

[0018] In an embodiment of the present application, the first workpiece point cloud data is segmented according to the workpiece mask to obtain second workpiece point cloud data, including:

[0019] Each first point in the first workpiece point cloud data is projected to a pixel point of the first RGB image.

[0020] For each first pixel point projected to the pixel point of the first RGB image, if a second pixel point corresponding to the first pixel point in the workpiece mask satisfies a first preset value, a first point corresponding to the first pixel point is retained.

[0021] Second workpiece point cloud data is obtained according to all retained first points.

[0022] In an embodiment of the present application, the second workpiece point cloud data is analyzed to determine the first target point cloud data of the target workpiece, including:

[0023] The second workpiece point cloud data is analyzed to obtain point cloud data belonging to each workpiece in the second workpiece point cloud data.

[0024] calculating a depth mean value of the point cloud data belonging to each workpiece;

[0025] taking the point cloud data of the workpiece with the minimum depth mean value as first target point cloud data of the target workpiece.

[0026] In an embodiment of the present application, the grabbing position and the grabbing angle of the mechanical arm are calculated based on the first target point cloud data of the target workpiece, comprising:

[0027] performing coordinate conversion on the first target point cloud data to obtain intermediate point cloud data of the first target point cloud data in a first coordinate system;

[0028] determining the grabbing position of the target workpiece in the first coordinate system according to the position of each point in the intermediate point cloud data;

[0029] performing principal component analysis on the intermediate point cloud data to determine the grabbing angle of the target workpiece in the first coordinate system.

[0030] In an embodiment of the present application, the principal component analysis on the intermediate point cloud data to determine the grabbing angle of the target workpiece in the first coordinate system comprises:

[0031] performing principal component analysis on the intermediate point cloud data to obtain a plurality of principal component components;

[0032] sorting the plurality of principal component components, and taking the direction indicated by the principal component component ranked first as a first direction of a second coordinate system;

[0033] taking the normal direction of the projection of the first direction on the XY plane of the first coordinate system as a second direction of the second coordinate system;

[0034] determining a third direction of the second coordinate system according to the first direction and the second direction;

[0035] determining the grabbing angle of the target workpiece in the first coordinate system according to the first direction, the second direction and the third direction.

[0036] In an embodiment of the present application, the determination of the grabbing angle of the target workpiece in the first coordinate system according to the first direction, the second direction and the third direction comprises:

[0037] if the Z component of the third direction in the first coordinate system is less than a target preset value, rotating the second direction and the third direction;

[0038] determining the grabbing angle of the target workpiece in the first coordinate system according to the first direction, the second direction and the third direction.

[0039] In an embodiment of the present application, the second target point cloud data of the target workpiece and the first pose of the target workpiece are obtained according to the second depth image and the second RGB image, comprising:

[0040] The second depth image and the second RGB image are preprocessed to obtain a mask of the target workpiece;

[0041] The first pose of the target workpiece is estimated according to the mask of the target workpiece, the second RGB image, and preset model data of the target workpiece.

[0042] The second target point cloud data of the target workpiece is obtained by segmenting third workpiece point cloud data according to the mask of the target workpiece, wherein the third workpiece point cloud data is obtained by point cloud conversion based on the second depth image.

[0043] In an embodiment of the present application, the second target point cloud data of the target workpiece is obtained by segmenting third workpiece point cloud data according to the mask of the target workpiece, comprising:

[0044] Each second point in the third workpiece point cloud data is projected to a pixel point of the second RGB image;

[0045] For each third pixel point projected to the pixel point of the second RGB image, if a fourth pixel point corresponding to the third pixel point in the mask of the target workpiece satisfies a second preset value, the second point corresponding to the third pixel point is retained;

[0046] The second target point cloud data is obtained according to all retained second points.

[0047] In an embodiment of the present application, the target pose of the target workpiece is estimated according to the first pose of the target workpiece, the second target point cloud data, and preset point cloud data, comprising:

[0048] The preset point cloud data of the target workpiece is obtained from a preset model file;

[0049] The first pose of the target workpiece is used as an initial pose of a preset point cloud registration algorithm;

[0050] The target pose of the target workpiece is obtained by performing point cloud registration on the second target point cloud data and the preset point cloud data using the preset point cloud registration algorithm according to the initial pose.

[0051] In a second aspect, embodiments of the present application provide a workpiece pose estimation device, comprising:

[0052] The first acquisition module is configured to acquire a first photographed image of the unordered stacked multiple workpieces, the first photographed image comprising a first depth image and a first RGB image of the multiple workpieces.

[0053] The determination module is configured to determine first target point cloud data of a target workpiece according to the first depth image and the first RGB image, the target workpiece being a workpiece located at the uppermost layer in the unordered stacked multiple workpieces.

[0054] The calculation module is configured to calculate a grabbing position and a grabbing angle of a mechanical arm based on the first target point cloud data of the target workpiece.

[0055] The grabbing module is configured to grab the target workpiece to a transfer table according to the grabbing position and the grabbing angle of the mechanical arm.

[0056] The second acquisition module is configured to acquire a second photographed image, the second photographed image comprising a second depth image and a second RGB image of the target workpiece.

[0057] The third acquisition module is configured to obtain second target point cloud data of the target workpiece and a first pose of the target workpiece according to the second depth image and the second RGB image.

[0058] The estimation module is configured to perform pose estimation on the target workpiece according to the first pose of the target workpiece, the second target point cloud data and preset point cloud data, to obtain a target pose of the target workpiece.

[0059] In a third aspect, an electronic device is provided, which includes a processor and a memory storing computer program instructions; the processor implements the workpiece pose estimation method according to the first aspect when executing the computer program instructions.

[0060] In a fourth aspect, a computer readable storage medium is provided, which stores computer program instructions; the computer program instructions are executed by a processor to implement the workpiece pose estimation method according to the first aspect.

[0061] In a fifth aspect, a computer program product is provided, which includes instructions executed by a processor of an electronic device to cause the electronic device to perform the workpiece pose estimation method according to the first aspect.

[0062] The workpiece pose estimation method provided by the embodiments of the present application first determines a target workpiece according to a first shooting image, and transfers the target workpiece to a transfer table, then shoots the target workpiece on the transfer table to obtain a second shooting image, and coarsely estimates the pose of the target workpiece according to the second shooting image, and then finely estimates the pose of the target workpiece. In this way, on the one hand, the success rate of identification of the slender workpiece in the matching process under the disordered stacking scene is improved, and then the reliability of the workpiece grasped by the mechanical arm is improved; on the other hand, the target workpiece is transferred from the disordered stacking scene to the pose estimation scene of a single object, and the mechanism of coarse pose estimation and fine pose estimation is adopted, so that high-precision and robust estimation of the workpiece is realized, and the accuracy of the workpiece pose estimation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly introduced. Those skilled in the art can obtain other drawings according to these drawings without creating any creative labor.

[0064] Figure 1 is a flowchart of the workpiece pose estimation method provided by the embodiments of the present application;

[0065] Figure 2 is another flowchart of the workpiece pose estimation method provided by the embodiments of the present application;

[0066] Figure 3 is a second coordinate system diagram of the target workpiece provided by the embodiments of the present application;

[0067] Figure 4 is a structural diagram of the workpiece pose estimation device provided by the embodiments of the present application;

[0068] Figure 5 is a structural diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0069] The features and exemplary embodiments of various aspects of the present application will be described in detail below, in order to make the purposes, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. The present application can be implemented without some of these specific details by those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0070] It should be noted that, in the present document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0071] In various specific embodiments of the present application, when it is necessary to perform relevant processing according to user information, user behavior data, user history data, and user location information, and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present disclosure to function normally will be obtained.

[0072] To solve the problems in the prior art, the embodiments of the present application provide a workpiece pose estimation method, device, equipment, medium and product. First, the workpiece pose estimation method provided by the embodiments of the present application is introduced.

[0073] Figure 1 A flowchart of the workpiece pose estimation method provided by an embodiment of the present application is shown. As shown in Figure 1 The workpiece pose estimation method provided by the embodiments of the present application is applied to an electronic device, and includes the following steps 101-106, wherein:

[0074] Step 101: Obtain a first shooting image of a plurality of workpieces stacked in disorder, wherein the first shooting image includes a first depth image and a first RGB image of the plurality of workpieces.

[0075] The first camera is arranged above the material frame and used to capture the workpieces in the disordered stack in the material frame to obtain a first captured image. The electronic device can establish a communication connection with the first camera through an open software development kit (SDK) provided by a camera manufacturer, set resolution, frame rate, exposure time and gain parameters of the first camera using an SDK interface, and write a script (such as based on Python or C++) to call the SDK interface to realize image acquisition of the workpieces in the disordered stack in the material frame. The first camera can be an RGB-D camera used to acquire a first RGB image (including color information) and a first depth image.

[0076] In step 102, first target point cloud data of a target workpiece is determined according to the first depth image and the first RGB image, and the target workpiece is a workpiece located at the uppermost layer in the disordered stack of workpieces.

[0077] The first depth image and the first RGB image are analyzed, for example, the first depth image and the first RGB image are image-aligned, and the aligned images are image-segmented to obtain point cloud data of each workpiece, and the target workpiece is determined according to the point cloud data of each workpiece, for example, the depth mean of the point cloud data of each workpiece is calculated; the workpiece corresponding to the minimum depth mean is taken as the target workpiece, and the point cloud data of the target workpiece is the first target point cloud data.

[0078] In step 103, a grabbing position and a grabbing angle of a mechanical arm are calculated based on the first target point cloud data of the target workpiece.

[0079] The grabbing position and the grabbing angle can be a grabbing position and a grabbing angle of the mechanical arm in a mechanical arm coordinate system to grab the target workpiece. The first target point cloud data is point cloud data in a camera coordinate system, which can be converted to the mechanical arm coordinate system to determine the grabbing position and the grabbing angle.

[0080] In step 104, the target workpiece is grabbed to a transfer table according to the grabbing position and the grabbing angle of the mechanical arm, and a second captured image is acquired, the second captured image including a second depth image and a second RGB image of the target workpiece.

[0081] After the grabbing position and the grabbing angle are determined, the target workpiece is transferred to the transfer table by the determined grabbing position and grabbing angle of the target workpiece, which can improve the recognition success rate of the slender workpiece in the matching process in the disordered stack scenario, and further improve the reliability of the mechanical arm in grabbing the workpiece.

[0082] The second camera is arranged above the transfer table and used to capture the target workpiece on the transfer table to obtain a second captured image. The electronic device can establish a communication connection with the second camera through an open software development kit (SDK) provided by a camera manufacturer, set resolution, frame rate, exposure time and gain parameters of the second camera using an SDK interface, and write a script (such as based on Python or C++) to call the SDK interface to realize image acquisition of the target workpiece on the transfer table. The second camera can be an RGB-D camera used to acquire a second RGB image (including color information) and a second depth image.

[0083] In step 105, second target point cloud data of the target workpiece and a first pose of the target workpiece are obtained according to the second depth image and the second RGB image.

[0084] The second depth image and the second RGB image are analyzed, for example, the second depth image and the second RGB image are image-aligned, and the aligned images are image-segmented (i.e., the target workpiece and the background region are segmented) to obtain a mask of the target workpiece; and a first pose of the target workpiece is determined according to the mask of the target workpiece and the second RGB image, which can be regarded as a rough pose estimation result.

[0085] Since the workpiece on the transfer table is only the target workpiece, the workpiece in the captured second depth image and second RGB image is also only the target workpiece, and when the pose of the workpiece is estimated according to the second depth image and the second RGB image, the interference of other workpieces can be excluded, so that the subsequent pose estimation of the target workpiece is more accurate.

[0086] In step 106, a target pose of the target workpiece is obtained by estimating the pose of the target workpiece according to the first pose of the target workpiece, the second target point cloud data and preset point cloud data.

[0087] The preset point cloud data can be a cad model point cloud of the target workpiece obtained from a preset cad model file, and the target pose can be a 6D pose, i.e., the pose of the target workpiece is further finely estimated to obtain the target pose.

[0088] Through the above process, the grabbing position and the grabbing angle of the target workpiece are determined according to the first shooting image, and the target workpiece is grabbed to the transfer table by the mechanical arm through the determined grabbing position and the grabbing angle of the target workpiece. Through this step, the identification success rate of the slender workpiece in the matching process under the disordered stacking scene can be improved, and then the reliability of the mechanical arm in grabbing the workpiece can be improved. Then, the target workpiece on the transfer table is shot to obtain a second shooting image, and the pose of the target workpiece is first roughly estimated and then finely estimated according to the second shooting image. In this way, the target workpiece can be transferred from the disordered stacking scene to the pose estimation scene of a single object, and through the mechanism of combining rough pose estimation and fine pose estimation, not only the pose estimation of the workpiece under the disordered stacking scene is realized, but also the accuracy of the pose estimation of the target workpiece is improved.

[0089] Figure 2 A flowchart of a workpiece pose estimation method provided by an embodiment of the application is shown. As shown in Figure 2 The workpiece pose estimation method provided by the embodiment of the application is applied to an electronic device and includes the following steps 201-208, wherein:

[0090] Step 201: Obtain a first shooting image of a plurality of workpieces stacked in disorder, wherein the first shooting image includes a first depth image and a first RGB image of the plurality of workpieces.

[0091] Step 201 can refer to the description in step 101, which is not repeated here.

[0092] Step 202: Preprocess the first depth image and the first RGB image to obtain a workpiece mask.

[0093] Exemplarily, the first depth image and the first RGB image are aligned to obtain first aligned image data, and the first aligned image data is subjected to image segmentation to obtain the workpiece mask.

[0094] Specifically, the intrinsic parameters, the extrinsic rotation parameters and the offset parameters of the depth camera and the RGB camera in the first camera are obtained respectively, each pixel point coordinate in the first depth image is traversed, the pixel point coordinate in the first RGB image corresponding to the pixel point in the first depth image is calculated, and each pixel point in the first depth image is projected into the coordinate system of the first RGB image, so that the first depth image and the first RGB image are aligned to obtain the first aligned image data of the workpiece, so that the resolution of the first depth image and the first RGB image is consistent and the pixel points are one-to-one corresponding, thereby ensuring the accurate matching of the color information and the depth information in space.

[0095] The first aligned image data is subjected to image segmentation, for example, the first aligned image data is subjected to image segmentation by using a yolo11-seg model, to obtain segmented workpiece masks (mask). The workpiece mask can be regarded as a mask image, and the pixel points of the workpiece mask correspond one-to-one to the pixel points of the first RGB image. In the workpiece mask, the pixel points of the same workpiece have the same pixel value, the pixel points of different workpieces have different pixel values, and the pixel points of each workpiece and the pixel points of the background region have different pixel values. For example, if the first aligned image data includes two workpiece data (i.e., first workpiece data and second workpiece data) and background region data, in the workpiece mask, the pixel value of the pixel point belonging to the first workpiece can be set to a first value (such as red), the pixel value of the pixel point belonging to the second workpiece can be set to a second value (such as yellow), and the pixel value of the pixel point belonging to the background region can be set to a third value (such as black).

[0096] Through the above processing, the workpiece mask is obtained, which facilitates subsequent determination of the target workpiece according to the workpiece mask.

[0097] In step 203, the first workpiece point cloud data is segmented according to the workpiece mask to obtain second workpiece point cloud data, wherein the first workpiece point cloud data is obtained by point cloud conversion based on the first depth image.

[0098] In step 2031, the first depth image is subjected to point cloud conversion to obtain the first workpiece point cloud data.

[0099] In step 2032, each first point in the first workpiece point cloud data is projected to a pixel point of the first RGB image, and for each first pixel point in the pixel point of the first RGB image, if a second pixel point corresponding to the first pixel point in the workpiece mask satisfies a first preset value, the first point corresponding to the first pixel point is retained.

[0100] In step 2033, the second workpiece point cloud data is obtained according to all retained first points.

[0101] For example, the first workpiece point cloud data includes a plurality of points (for the sake of description, the points in the first workpiece point cloud data are referred to as first points), and for a first point in the first workpiece point cloud data, the first point is projected to a pixel coordinate (the pixel coordinate can be understood as the position of the pixel point) of the first RGB image;

[0102] For the first point which has been projected to the pixel coordinate of the first RGB image, find the pixel value corresponding to the pixel coordinate of the first point projected to the first RGB image in the workpiece mask. Specifically, the pixel points of the workpiece mask correspond one-to-one to the pixel points of the first RGB image. If the first point can be projected to the first pixel point of the first RGB image, the second pixel point corresponding to the first pixel point can be found in the workpiece mask. If the pixel value of the second pixel point is a first preset value, for example, the first preset value can be set as a value other than 0 (0 represents a background area), it is considered that the first workpiece point cloud data has been segmented using the workpiece mask, and the first point is saved.

[0103] Repeat the above process until all first points included in the first workpiece point cloud data are traversed. Finally, the second workpiece point cloud data is obtained according to all retained first points.

[0104] Through the above processing, the point cloud data of the stacked workpiece can be segmented from the first workpiece point cloud data. This segmentation method does not consider the occlusion between workpieces, so the point cloud data of the stacked workpiece will not be segmented into discontinuous segments, which can improve the point cloud data segmentation effect and provide a basis for subsequent workpiece pose estimation.

[0105] In step 204, the second workpiece point cloud data is analyzed to determine the first target point cloud data of the target workpiece.

[0106] The second workpiece point cloud data is the point cloud data of multiple workpieces in a stacked state, and the workpiece located on the uppermost layer, i.e., the target workpiece, needs to be segmented from the multiple workpieces. Specifically, the second workpiece point cloud data is analyzed to obtain the point cloud data belonging to each workpiece in the second workpiece point cloud data. The depth mean of the point cloud data belonging to each workpiece is calculated. The point cloud data of the workpiece with the smallest depth mean is taken as the first target point cloud data of the target workpiece.

[0107] For example, the first point in the second workpiece point cloud data is analyzed, wherein the first point in the second workpiece point cloud data is obtained by filtering all first points included in the first workpiece point cloud data through steps 2031-2033. At the same time, the first point needs to be labeled. The labeling method is: adding a label to the first point retained through step 2032, and adding the same label to the first points belonging to the same workpiece. Since the pixel values of the pixel points belonging to the same workpiece in the workpiece mask are the same, the labels of the first points belonging to the same workpiece are also the same through the above labeling method. When analyzing the first points in the second workpiece point cloud data, whether each first point belongs to the same workpiece can be directly determined according to the label of each first point.

[0108] After obtaining the point cloud data of each workpiece, the point cloud data of the workpiece is converted, each point cloud data is converted into a depth value, or a pixel point in a first depth map corresponding to each point in the point cloud data of the workpiece is obtained, and a pixel value corresponding to the pixel point is the depth value; the depth mean of the point cloud data belonging to each workpiece is calculated; the point cloud data of the workpiece with the smallest depth mean is taken as the first target point cloud data of the target workpiece, wherein the target workpiece is the workpiece stacked on the topmost layer, that is, the workpiece to be grabbed.

[0109] In the above manner, the first target point cloud data of the target workpiece is determined by the depth mean of each workpiece, so that the target workpiece stacked on the topmost layer can be screened out, facilitating the mechanical arm to grab and transfer it to the transfer table.

[0110] In step 205, the grabbing position and grabbing angle of the mechanical arm are calculated based on the first target point cloud data of the target workpiece.

[0111] Specifically, the first target point cloud data is subjected to coordinate conversion to obtain intermediate point cloud data of the first target point cloud data in a first coordinate system; the grabbing position of the target workpiece in the first coordinate system is determined according to the position of each point in the intermediate point cloud data; principal component analysis is performed on the intermediate point cloud data to determine the grabbing angle of the target workpiece in the first coordinate system.

[0112] The first target point cloud data is data in a camera coordinate system and needs to be converted into the first coordinate system for calculation, wherein the first coordinate system can be a mechanical arm base coordinate system. For ease of description, the point cloud data after the first target point cloud data is converted into the first coordinate system is referred to as intermediate point cloud data.

[0113] For example, the mean value coordinates are obtained by calculating the mean value of the coordinates of each point in the intermediate point cloud data in the first coordinate system, and the mean value coordinates are taken as the grabbing position of the mechanical arm to grab the target workpiece. Principal component analysis is performed on the second point cloud data to determine the grabbing angle of the target workpiece in the first coordinate system.

[0114] The principal component analysis on the intermediate point cloud data to determine the grabbing angle of the target workpiece in the first coordinate system comprises:

[0115] The principal component analysis on the intermediate point cloud data obtains a plurality of principal component components;

[0116] The plurality of principal component components are sorted, and the first direction indicated by the principal component component ranked first is taken as the first direction of a second coordinate system;

[0117] The normal direction of the projection of the first direction on the XY plane of the first coordinate system is taken as the second direction of the second coordinate system.

[0118] determining a third direction of the second coordinate system according to the first direction and the second direction;

[0119] determining a grasping angle of the target workpiece in the first coordinate system according to the first direction, the second direction and the third direction.

[0120] In the above, each of the principal component components can be a coordinate in the first coordinate system, the coordinate corresponding to a vector, the vector indicating a direction; the second coordinate system can be a self-defined three-dimensional coordinate system. The plurality of principal component components can be sorted in descending order of importance, and the direction indicated by the principal component component ranked first can be taken as the first direction of the second coordinate system.

[0121] If the Z component of the third direction in the first coordinate system is less than a target preset value, the second direction and the third direction are rotated; and a grasping angle of the target workpiece in the first coordinate system is determined according to the first direction, the second direction and the third direction after rotation, wherein the Z component is a depth direction component of the first coordinate system.

[0122] The three-dimensional representation of the first coordinate system adopts XYZ, and the three-dimensional representation of the second coordinate system adopts xyz (uniformly using the right-hand rotation coordinate system order), wherein XYZ and xyz respectively represent the horizontal direction, the vertical direction and the depth direction of the first coordinate system and the second coordinate system. For example, in the second coordinate system, the first direction can be the x direction (horizontal direction), the second direction can be the y direction (vertical direction), and the third direction can be the z direction (depth direction).

[0123] For example, the direction indicated by the principal component component with the largest importance is taken as the x direction of the second coordinate system, as shown in FIG. 1B, that is, the direction of the main shaft of the target workpiece in FIG. 1B; the normal direction of the projection of the x direction on the XY plane of the first coordinate system is taken as the y direction of the second coordinate system; and the z direction of the second coordinate system is determined according to the cross product result of the vector indicating the x direction and the vector indicating the y direction, for example, the unit vector of the x direction and the unit vector of the y direction are cross-multiplied to obtain the unit vector of the z direction. Figure 3 Figure 3 For example, the direction indicated by the principal component component with the largest importance is taken as the x direction of the second coordinate system, as shown in FIG. 1B, that is, the direction of the main shaft of the target workpiece in FIG. 1B; the normal direction of the projection of the x direction on the XY plane of the first coordinate system is taken as the y direction of the second coordinate system; and the z direction of the second coordinate system is determined according to the cross product result of the vector indicating the x direction and the vector indicating the y direction, for example, the unit vector of the x direction and the unit vector of the y direction are cross-multiplied to obtain the unit vector of the z direction.

[0124] If the Z component of the z direction in the first coordinate system is greater than or equal to 0 (for example, the Z component of the unit vector of the z direction is greater than or equal to 0), in this case, since the orientation of the robot arm when grasping the workpiece is from top to bottom, the grasping angle of the target workpiece in the first coordinate system can be directly determined according to the x direction, the y direction and the z direction.

[0125] ​If the Z component of the z direction in the first coordinate system is less than 0 (for example, the Z component of the unit vector of the z direction is less than 0), to ensure that the mechanical arm is gripped towards the workpiece from top to bottom, the y direction and the z direction are rotated by 180° at this time; then the grabbing angle of the target workpiece in the first coordinate system is determined according to the x direction, the rotated y direction and the rotated z direction.

[0126] In step 206, the target workpiece is gripped to the transfer table according to the grabbing position and the grabbing angle of the target workpiece, and a second shooting image is obtained, the second shooting image including a second depth image and a second RGB image of the target workpiece. The second shooting image is obtained by shooting the target workpiece on the transfer table.

[0127] The mechanical arm can grip the target workpiece to the transfer table according to the grabbing position and the grabbing angle of the target workpiece calculated in step 205.

[0128] The second camera is arranged above the transfer table and is used to shoot the target workpiece on the transfer table to obtain the second shooting image. The electronic device can establish a communication connection with the second camera through an open software development kit (SDK) provided by a camera manufacturer; set the resolution, frame rate, exposure time and gain parameters of the second camera using the SDK interface; and write a script (such as based on Python or C++) to call the SDK interface to realize image acquisition of the target workpiece on the transfer table. The second camera can be an RGB-D camera used to acquire a second RGB image (including color information) and a second depth image.

[0129] Since the workpiece on the transfer table is only the target workpiece, the second depth image and the second RGB image shot also only have the target workpiece, so when the pose of the workpiece is estimated according to the second depth image and the second RGB image, the interference of other workpieces can be excluded, and the pose estimation of the target workpiece is more accurate.

[0130] In step 207, the second target point cloud data of the target workpiece and the first pose of the target workpiece are obtained according to the second depth image and the second RGB image.

[0131] Specifically, the second depth image and the second RGB image are preprocessed to obtain a mask of the target workpiece; the first pose of the target workpiece is estimated according to the mask of the target workpiece, the second RGB image and preset model data of the target workpiece; and the second target point cloud data of the target workpiece is obtained by segmenting third workpiece point cloud data according to the mask of the target workpiece, wherein the third workpiece point cloud data is obtained by point cloud conversion based on the second depth image.

[0132] Exemplarily, the second depth image and the second RGB image are aligned to obtain second aligned image data; and the second aligned image data is segmented to obtain the workpiece mask.

[0133] Specifically, the intrinsic parameters, the extrinsic rotation parameters and the offset parameters of the depth camera and the RGB camera in the second camera are obtained respectively, each pixel point coordinate in the second depth image is traversed, the pixel point coordinate in the second RGB image corresponding to the pixel point in the second depth image is calculated, each pixel point in the second depth image is projected into the coordinate system of the second RGB image, so that the second depth image and the second RGB image are aligned, and the second aligned image data of the workpiece is obtained, that is, the resolutions of the second depth image and the second RGB image are consistent, and the pixel points are one-to-one corresponding, so as to ensure the accurate matching of the color information and the depth information in space.

[0134] The second aligned image data is segmented, for example, the yolo11-seg model is used to segment the second aligned image data to obtain the mask of the target workpiece.

[0135] According to the mask of the target workpiece, the second RGB image and the preset model data of the target workpiece, a first pose of the target workpiece is estimated, for example, the mask of the target workpiece, the second RGB image and the preset model data of the target workpiece are input into the FoundationPose model for pose estimation to obtain the first pose of the target workpiece, wherein the preset model data can be a cad model file of the target workpiece, the first pose belongs to the pose estimation of coarse matching, and the FoundationPose model is a model based on a large language model and a diffusion model and used for 6D pose estimation and tracking.

[0136] According to the mask of the target workpiece, the third workpiece point cloud data is segmented to obtain second target point cloud data of the target workpiece, specifically including: projecting each second point in the third workpiece point cloud data to a pixel point of the second RGB image; for each third pixel point projected to the pixel point of the second RGB image, if a fourth pixel point corresponding to the third pixel point in the mask of the target workpiece satisfies a second preset value, the second point corresponding to the third pixel point is retained; and the second target point cloud data is obtained according to all the retained second points.

[0137] The second depth image is subjected to point cloud data conversion to obtain third workpiece point cloud data. For example, the third workpiece point cloud data includes a plurality of points. For ease of description, the points in the third workpiece point cloud data are referred to as second points. For a second point in the third workpiece point cloud data, the second point is projected to a pixel coordinate of the second RGB image (the pixel coordinate can be understood as the position of a pixel). Since the pixel points of the mask of the target workpiece are in one-to-one correspondence with the pixel points of the second RGB image, for the second point that has been projected to the pixel coordinate of the second RGB image, the second point is projected to a third pixel point of the pixel point of the second RGB image. For each third pixel point, a fourth pixel point corresponding to the third pixel point coordinate is searched for in the mask of the target workpiece. If the pixel value of the fourth pixel point is a second preset value, the second point corresponding to the third pixel point is retained, where the second preset value can be set to a value other than 0 (0 represents a background area). In this case, it is considered that the third workpiece point cloud data has been segmented using the mask of the target workpiece.

[0138] The above process is repeated until all second points included in the third workpiece point cloud data are traversed. The second target point cloud data is obtained according to all retained second points.

[0139] Through the above processing, the second target point cloud data can be segmented from the third workpiece point cloud data. Since no other workpiece shields the target workpiece in the third workpiece point cloud data, the second target point cloud data obtained by segmentation is complete and unshielded, which is beneficial to subsequent pose estimation of the target workpiece based on the second target point cloud data and improves the accuracy of workpiece pose estimation.

[0140] In step 208, the target workpiece is subjected to pose estimation according to the first pose of the target workpiece, the second target point cloud data, and preset point cloud data to obtain a target pose of the target workpiece.

[0141] Specifically, the preset point cloud data of the target workpiece is obtained from a preset model file; the first pose of the target workpiece is taken as an initial pose of a preset point cloud registration algorithm; and the second target point cloud data and the preset point cloud data are subjected to point cloud registration by using the preset point cloud registration algorithm according to the initial pose to obtain the target pose of the target workpiece, so as to realize more refined estimation of the pose of the target workpiece.

[0142] The preset point cloud registration algorithm can be an Iterative Closest Point (ICP) algorithm or other point cloud registration algorithms, which are not limited herein. The preset model file can be a cad model file of the target workpiece or the like.

[0143] The workpiece pose estimation method provided by the above embodiment is divided into two stages for pose estimation of workpieces in a stacked scene. In the first stage, for multiple workpieces in an unordered stacked scene, a workpiece located at the top layer is determined as a target workpiece, and then the target workpiece is grabbed to a transfer table through a calculated grabbing position and grabbing angle of the robot arm for grabbing the target workpiece. In the second stage, for a single target workpiece on the transfer table, a rough estimation is first performed on the pose of the target workpiece, and then a higher-precision pose estimation is realized through the rough estimation result, which has real-time performance.

[0144] The workpiece pose estimation method provided by the above embodiment can solve the reliability of grabbing an elongated workpiece, and can also solve the problem that, in a complex scene of unordered stacking of elongated workpieces, the point cloud data collected by the camera is cut into segments by the stacked workpieces due to occlusion between the workpieces, so that the elongated workpieces are prone to recognition failure in the pose matching process, the segmentation and matching of the target workpiece are poor, and an accurate pose cannot be given.

[0145] Figure 4 A structure diagram of a workpiece pose estimation device provided by an embodiment of the present application is shown. As shown in Figure 4 The workpiece pose estimation device 400 includes:

[0146] A first acquisition module 401 is configured to acquire a first photographed image of multiple workpieces stacked in disorder, the first photographed image including a first depth image and a first RGB image of the multiple workpieces;

[0147] A determination module 402 is configured to determine first target point cloud data of a target workpiece according to the first depth image and the first RGB image, the target workpiece being a workpiece located at the uppermost layer among the multiple workpieces stacked in disorder;

[0148] A calculation module 403 is configured to calculate a grabbing position and a grabbing angle of a robot arm based on the first target point cloud data of the target workpiece;

[0149] A grabbing module 404 is configured to grab the target workpiece to a transfer table according to the grabbing position and the grabbing angle of the robot arm;

[0150] A second acquisition module 405 is configured to acquire a second photographed image, the second photographed image including a second depth image and a second RGB image of the target workpiece;

[0151] A third acquisition module 406 is configured to obtain second target point cloud data of the target workpiece and a first pose of the target workpiece according to the second depth image and the second RGB image;

[0152] The estimation module 407 is configured to perform pose estimation on the target workpiece according to the first pose of the target workpiece, the second target point cloud data, and preset point cloud data, to obtain a target pose of the target workpiece.

[0153] The workpiece pose estimation apparatus 400 provided by the embodiments of the present application can implement each process of the workpiece pose estimation method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0154] Figure 5 A hardware structure schematic diagram of an electronic device provided by the embodiments of the present application is shown.

[0155] The electronic device can include a processor 601 and a memory 602 storing computer program instructions.

[0156] Specifically, the processor 601 described above can be configured to implement one or more integrated circuits of the embodiments of the present application.

[0157] The memory 602 can include a mass storage for data or instructions. The memory can include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions that, when executed (e.g., by one or more processors), are operable to perform the operations described with reference to the method according to the first aspect or the second aspect of the present disclosure.

[0158] The processor 601 implements any one of the workpiece pose estimation methods in the above embodiments by reading and executing the computer program instructions stored in the memory 602.

[0159] In one example, the electronic device can further include a communication interface 603 and a bus 610. As shown, the processor 601, the memory 602, and the communication interface 603 are connected through the bus 610 and complete communication among each other. Figure 5

[0160] The communication interface 603 is mainly used to realize the communication between the modules, apparatuses, units, and / or devices in the embodiments of the present application.

[0161] The bus 610 includes hardware, software, or both, which couples components of hardware and software to each other. The bus 610 can include one or more buses, as appropriate. The present application contemplates any suitable bus or interconnect.

[0162] ​In addition, in combination with the workpiece pose estimation method in the above embodiments, an embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium has computer program instructions stored thereon. The computer program instructions are executed by a processor to implement any of the workpiece pose estimation methods in the above embodiments.

[0163] An embodiment of the present application can provide a computer program product. Instructions in the computer program product are executed by a processor of an electronic device to cause the electronic device to perform any of the workpiece pose estimation methods in the above embodiments.

[0164] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted. In the above embodiments, several specific steps are described as examples. However, the method processes of the present application are not limited to the specific steps described, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0165] The functional blocks shown in the structural block diagrams described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The code segments can be downloaded via a computer network such as the Internet, an intranet, etc.

[0166] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0167] The above is only a specific implementation of the present application, and those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, modules and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here. It should be understood that the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A workpiece pose estimation method, characterized by, The method comprises: acquiring a first shooting image of a plurality of workpieces in a disordered stack, the first shooting image comprising a first depth image and a first RGB image of the plurality of workpieces; determining first target point cloud data of a target workpiece according to the first depth image and the first RGB image, the target workpiece being a workpiece located at the uppermost layer in the plurality of workpieces in the disordered stack; calculating a grabbing position and a grabbing angle of a mechanical arm based on the first target point cloud data of the target workpiece; grabbing the target workpiece to a transfer table according to the grabbing position and the grabbing angle of the mechanical arm, and acquiring a second shooting image, the second shooting image comprising a second depth image and a second RGB image of the target workpiece; obtaining second target point cloud data of the target workpiece and a first pose of the target workpiece according to the second depth image and the second RGB image; performing pose estimation on the target workpiece according to the first pose of the target workpiece, the second target point cloud data and preset point cloud data to obtain a target pose of the target workpiece; wherein the calculating of the grabbing position and the grabbing angle of the mechanical arm based on the first target point cloud data of the target workpiece comprises: performing coordinate conversion on the first target point cloud data to obtain intermediate point cloud data of the first target point cloud data in a first coordinate system; determining the grabbing position of the target workpiece in the first coordinate system according to the position of each point in the intermediate point cloud data; performing principal component analysis on the intermediate point cloud data to determine the grabbing angle of the target workpiece in the first coordinate system; the performing of the pose estimation on the target workpiece according to the first pose of the target workpiece, the second target point cloud data and the preset point cloud data to obtain the target pose of the target workpiece comprises: obtaining the preset point cloud data of the target workpiece from a preset model file; taking the first pose of the target workpiece as an initial pose of a preset point cloud registration algorithm; performing point cloud registration on the second target point cloud data and the preset point cloud data by using the preset point cloud registration algorithm according to the initial pose to obtain the target pose of the target workpiece.

2. The workpiece pose estimation method of claim 1, wherein, The determining of the first target point cloud data of the target workpiece according to the first depth image and the first RGB image comprises: performing preprocessing on the first depth image and the first RGB image to obtain a workpiece mask; segmenting first workpiece point cloud data according to the workpiece mask to obtain second workpiece point cloud data, wherein the first workpiece point cloud data is obtained by point cloud conversion based on the first depth image; performing analysis on the second workpiece point cloud data to determine the first target point cloud data of the target workpiece.

3. The workpiece pose estimation method of claim 2, wherein, The performing of the preprocessing on the first depth image and the first RGB image to obtain a workpiece mask comprises: aligning the first depth image and the first RGB image to obtain first aligned image data; performing image segmentation on the first aligned image data to obtain the workpiece mask.

4. The workpiece pose estimation method of claim 2, wherein, The segmenting of the first workpiece point cloud data according to the workpiece mask to obtain second workpiece point cloud data comprises: Projecting each first point in the first workpiece point cloud data to a pixel point of the first RGB image; For each first pixel point projected to the pixel point of the first RGB image, if a second pixel point corresponding to the first pixel point in the workpiece mask satisfies a first preset value, retaining the first point corresponding to the first pixel point; Obtaining second workpiece point cloud data according to all retained first points.

5. The workpiece pose estimation method of claim 2, wherein, Analyzing the second workpiece point cloud data to determine first target point cloud data of the target workpiece, including: Analyzing the second workpiece point cloud data to obtain point cloud data belonging to each workpiece in the second workpiece point cloud data; Calculating a depth mean value of the point cloud data belonging to each workpiece; Taking the point cloud data of the workpiece with the smallest depth mean value as the first target point cloud data of the target workpiece.

6. The workpiece pose estimation method of claim 1, wherein, Performing principal component analysis on the intermediate point cloud data to determine a grabbing angle of the target workpiece in a first coordinate system, including: Performing principal component analysis on the intermediate point cloud data to obtain a plurality of principal component components; Ranking the plurality of principal component components, and taking a direction indicated by a principal component component ranked first as a first direction of a second coordinate system; Taking a normal direction of a projection of the first direction on an XY plane of the first coordinate system as a second direction of the second coordinate system; Determining a third direction of the second coordinate system according to the first direction and the second direction; Determining the grabbing angle of the target workpiece in the first coordinate system according to the first direction, the second direction, and the third direction.

7. The workpiece pose estimation method of claim 6, wherein, Determining the grabbing angle of the target workpiece in the first coordinate system according to the first direction, the second direction, and the third direction, including: If a Z component of the third direction in the first coordinate system is less than a target preset value, rotating the second direction and the third direction; Determining the grabbing angle of the target workpiece in the first coordinate system according to the first direction, the second direction obtained after rotation, and the third direction.

8. The workpiece pose estimation method of claim 1, wherein, Obtaining second target point cloud data of the target workpiece and a first pose of the target workpiece according to the second depth image and the second RGB image, including: Preprocessing the second depth image and the second RGB image to obtain a mask of the target workpiece; Performing pose estimation according to the mask of the target workpiece, the second RGB image, and preset model data of the target workpiece to obtain the first pose of the target workpiece; Segmenting third workpiece point cloud data according to the mask of the target workpiece to obtain the second target point cloud data of the target workpiece, wherein the third workpiece point cloud data is obtained by point cloud conversion based on the second depth image.

9. The workpiece pose estimation method of claim 8, wherein, Segmenting third workpiece point cloud data according to the mask of the target workpiece to obtain the second target point cloud data of the target workpiece, including: Projecting each second point in the third workpiece point cloud data to a pixel point of the second RGB image; For each third pixel point projected into the pixel points of the second RGB image, if a fourth pixel point corresponding to the third pixel point in a mask of the target workpiece satisfies a second preset value, a second point corresponding to the third pixel point is retained; Second target point cloud data is obtained according to all retained second points.

10. A workpiece pose estimation apparatus, characterized by comprising: The device comprises: A first acquisition module is configured to acquire a first captured image of a plurality of workpieces in disorderly stacking, the first captured image comprising a first depth image and a first RGB image of the plurality of workpieces; A determination module is configured to determine first target point cloud data of a target workpiece according to the first depth image and the first RGB image, the target workpiece being a workpiece located at the uppermost layer among the plurality of workpieces in disorderly stacking; A calculation module is configured to calculate a grabbing position and a grabbing angle of a mechanical arm based on the first target point cloud data of the target workpiece; A grabbing module is configured to grab the target workpiece to a transfer table according to the grabbing position and the grabbing angle of the mechanical arm; A second acquisition module is configured to acquire a second captured image, the second captured image comprising a second depth image and a second RGB image of the target workpiece; A third acquisition module is configured to obtain second target point cloud data of the target workpiece and a first pose of the target workpiece according to the second depth image and the second RGB image; An estimation module is configured to perform pose estimation on the target workpiece according to the first pose of the target workpiece, the second target point cloud data, and preset point cloud data, to obtain a target pose of the target workpiece; The calculation of the grabbing position and the grabbing angle of the mechanical arm based on the first target point cloud data of the target workpiece comprises: coordinate conversion is performed on the first target point cloud data to obtain intermediate point cloud data of the first target point cloud data in a first coordinate system; a grabbing position of the target workpiece in the first coordinate system is determined according to the position of each point in the intermediate point cloud data; principal component analysis is performed on the intermediate point cloud data to determine a grabbing angle of the target workpiece in the first coordinate system; The pose estimation on the target workpiece according to the first pose of the target workpiece, the second target point cloud data, and preset point cloud data to obtain a target pose of the target workpiece comprises: preset point cloud data of the target workpiece is obtained from a preset model file; the first pose of the target workpiece is taken as an initial pose of a preset point cloud registration algorithm; point cloud registration is performed on the second target point cloud data and the preset point cloud data by using the preset point cloud registration algorithm according to the initial pose, to obtain the target pose of the target workpiece.

11. An electronic device, comprising: comprise: a processor and a memory having computer program instructions stored thereon; the processor, when executing the computer program instructions, implements the workpiece pose estimation method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer program instructions are stored on the computer readable storage medium, and when executed by the processor, implement the workpiece pose estimation method according to any one of claims 1-9.

13. A computer program product, characterised in that, The instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the workpiece pose estimation method of any one of claims 1-9.

Images