Workpiece pose estimation method, device, equipment, medium and product

Abstract

The invention discloses a workpiece pose estimation method and device, equipment, a medium and a product, and the method comprises the steps: obtaining a first shot image of a plurality of workpieces stacked disorderly, the first shot image comprising a first depth image and a first RGB image; determining first target point cloud data of the target workpiece according to the first depth image and the first RGB image; the grabbing position and the grabbing angle of the mechanical arm are calculated based on the first target point cloud data of the target workpiece; the target workpiece is grabbed to the transfer table according to the grabbing position and the grabbing angle of the mechanical arm, a second shot image is obtained, and the second shot image comprises a second depth image and a second RGB image; obtaining second target point cloud data and a first pose of the target workpiece according to the second depth image and the second RGB image; and performing pose estimation according to the first pose, the second target point cloud data and preset point cloud data to obtain a target pose of the target workpiece. The accuracy of workpiece pose estimation in a stacking scene can be improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a workpiece pose estimation method, device, equipment, medium and product. Background Technology

[0002] With the development of science and the continuous transformation and upgrading of the manufacturing industry, machine vision is being increasingly widely used in intelligent manufacturing, represented by industrial robots. Unordered grasping, as a typical application scenario in industrial production, uses robotic arms to replace manual labor in picking up and assembling target objects, thus freeing up labor and optimizing production efficiency. In the process of unordered grasping by robotic arms, an efficient and stable pose estimation algorithm is crucial. Although in recent years, due to the popularization of 3D vision, many 6D pose estimation algorithms have been proposed, such as deep learning algorithms, these algorithms still have some problems when facing complex stacking scenarios. For example, in complex industrial scenarios, due to the variable stacking conditions of workpieces and occlusion between workpieces, the accuracy of workpiece pose estimation is poor. Summary of the Invention This application provides a workpiece pose estimation method, apparatus, device, medium, and product, which can improve the accuracy of workpiece pose estimation. In a first aspect, embodiments of this application provide a workpiece pose estimation method, the method comprising: Acquire a first image of multiple workpieces stacked in a disordered manner, the first image including a first depth image and a first RGB image of the multiple workpieces; Based on the first depth image and the first RGB image, the first target point cloud data of the target workpiece is determined, wherein the target workpiece is the workpiece located on the top layer among the disordered stacked multiple workpieces. The gripping position and gripping angle of the robotic arm are calculated based on the first target point cloud data of the target workpiece; The robotic arm grasps the target workpiece to the transfer platform according to the grasping position and grasping angle, and acquires a second image, which includes a second depth image and a second RGB image of the target workpiece. Based on the second depth image and the second RGB image, the second target point cloud data of the target workpiece and the first pose of the target workpiece are obtained; The target pose of the target workpiece is estimated by performing pose estimation on the first pose of the target workpiece, the second target point cloud data, and the preset point cloud data.

[0003] In one embodiment of this application, determining the first target point cloud data of the target workpiece based on the first depth image and the first RGB image includes: The first depth image and the first RGB image are preprocessed to obtain the workpiece mask; The first workpiece point cloud data is segmented according to the workpiece mask to obtain the second workpiece point cloud data, wherein the first workpiece point cloud data is obtained by point cloud transformation based on the first depth image; The point cloud data of the second workpiece is analyzed to determine the first target point cloud data of the target workpiece.

[0004] In one embodiment of this application, preprocessing the first depth image and the first RGB image to obtain a workpiece mask includes: Align the first depth image and the first RGB image to obtain the first aligned image data; The first aligned image data is segmented to obtain the workpiece mask.

[0005] In one embodiment of this application, segmenting the first workpiece point cloud data according to the workpiece mask to obtain the second workpiece point cloud data includes: Project each first point in the first workpiece point cloud data onto a pixel in the first RGB image; For each first pixel in the pixels projected onto the first RGB image, if the second pixel corresponding to the first pixel in the workpiece mask satisfies a first preset value, then the first point corresponding to the first pixel is retained. Based on all the retained first points, the second workpiece point cloud data is obtained.

[0006] In one embodiment of this application, analyzing the point cloud data of the second workpiece to determine the first target point cloud data of the target workpiece includes: The point cloud data of the second workpiece is analyzed to obtain the point cloud data belonging to each workpiece in the point cloud data of the second workpiece; The depth mean of the point cloud data belonging to each workpiece is calculated; The point cloud data of the workpiece with the smallest average depth is used as the first target point cloud data of the target workpiece.

[0007] In one embodiment of this application, calculating the gripping position and gripping angle of the robotic arm based on the first target point cloud data of the target workpiece includes: Perform coordinate transformation on the first target point cloud data to obtain the intermediate point cloud data of the first target point cloud data in the first coordinate system; Based on the position of each point in the intermediate point cloud data, determine the gripping position of the target workpiece in the first coordinate system; Principal component analysis is performed on the intermediate point cloud data to determine the grasping angle of the target workpiece in the first coordinate system.

[0008] In one embodiment of this application, principal component analysis is performed on the intermediate point cloud data to determine the grasping angle of the target workpiece in the first coordinate system, including: Principal component analysis was performed on the intermediate point cloud data to obtain multiple principal component components; The principal component components are sorted, and the direction indicated by the principal component component ranked first is taken as the first direction of the second coordinate system; The normal direction of the projection of the first direction onto the XY plane of the first coordinate system is taken as the second direction of the second coordinate system; The third direction of the second coordinate system is determined based on the first direction and the second direction; The grasping angle of the target workpiece in the first coordinate system is determined based on the first direction, the second direction, and the third direction.

[0009] In one embodiment of this application, determining the gripping angle of the target workpiece in a first coordinate system based on the first direction, the second direction, and the third direction includes: If the Z component of the third direction in the first coordinate system is less than the target preset value, then the second direction and the third direction are rotated. The grasping angle of the target workpiece in the first coordinate system is determined based on the first direction, the second direction obtained after rotation, and the third direction.

[0010] In one embodiment of this application, obtaining the second target point cloud data and the first pose of the target workpiece based on the second depth image and the second RGB image includes: The second depth image and the second RGB image are preprocessed to obtain a mask for the target workpiece; Based on the mask of the target workpiece, the second RGB image, and the preset model data of the target workpiece, pose estimation is performed to obtain the first pose of the target workpiece; The third workpiece point cloud data is segmented based on the mask of the target workpiece to obtain the second target point cloud data of the target workpiece, wherein the third workpiece point cloud data is obtained by point cloud transformation based on the second depth image.

[0011] In one embodiment of this application, segmenting the third workpiece point cloud data according to the mask of the target workpiece to obtain the second target point cloud data of the target workpiece includes: Project each second point in the third workpiece point cloud data onto a pixel in the second RGB image; For each third pixel in the pixels projected onto the second RGB image, if the fourth pixel corresponding to the third pixel in the mask of the target workpiece satisfies the second preset value, then the second point corresponding to the third pixel is retained. Based on all the retained second points, the second target point cloud data is obtained.

[0012] In one embodiment of this application, the target pose of the target workpiece is estimated based on the first pose of the target workpiece, the second target point cloud data, and preset point cloud data to obtain the target pose of the target workpiece, including: Obtain the preset point cloud data of the target workpiece from the preset model file; The first pose of the target workpiece is used as the initial pose of the preset point cloud registration algorithm. Based on the initial pose, the second target point cloud data and the preset point cloud data are registered using the preset point cloud registration algorithm to obtain the target pose of the target workpiece.

[0013] Secondly, embodiments of this application provide a workpiece pose estimation device, the device comprising: The first acquisition module is used to acquire a first captured image of multiple workpieces stacked in a disordered manner. The first captured image includes a first depth image and a first RGB image of the multiple workpieces. The determination module is used to determine the first target point cloud data of the target workpiece based on the first depth image and the first RGB image, wherein the target workpiece is the workpiece located on the top layer among the disordered stacked multiple workpieces; The calculation module is used to calculate the gripping position and gripping angle of the robotic arm based on the first target point cloud data of the target workpiece; The gripping module is used to grip the target workpiece to the transfer platform according to the gripping position and gripping angle of the robotic arm; The second acquisition module is used to acquire a second captured image, the second captured image including a second depth image and a second RGB image of the target workpiece; The third acquisition module is used to obtain the second target point cloud data of the target workpiece and the first pose of the target workpiece based on the second depth image and the second RGB image. The estimation module is used to estimate the pose of the target workpiece based on the first pose of the target workpiece, the second target point cloud data, and the preset point cloud data, so as to obtain the target pose of the target workpiece.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the workpiece pose estimation method as described in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the workpiece pose estimation method as described in the first aspect.

[0016] Fifthly, embodiments of this application provide a computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the workpiece pose estimation method as described in the first aspect.

[0017] The workpiece pose estimation method provided in this application first determines the target workpiece based on a first captured image and transfers the target workpiece to a transfer platform. Then, a second captured image is obtained by capturing the target workpiece on the transfer platform. Based on the second captured image, the pose of the target workpiece is first roughly estimated and then finely estimated. In this way, on the one hand, the success rate of identifying slender workpieces in the matching process in disordered stacking scenarios is improved, thereby improving the reliability of the robotic arm grasping the workpiece. On the other hand, by transferring the target workpiece from the disordered stacking scenario to the pose estimation scenario of a single object and adopting a mechanism that combines coarse and fine pose estimation, high-precision and robust estimation of the workpiece is achieved, improving the accuracy of workpiece pose estimation. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the workpiece pose estimation method provided in an embodiment of this application; Figure 2 This is another schematic flowchart of the workpiece pose estimation method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the second coordinate system of the target workpiece provided in the embodiments of this application; Figure 4 This is a schematic diagram of the workpiece pose estimation device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0022] In all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. Additionally, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments obtained.

[0023] To address the problems of the prior art, embodiments of this application provide a workpiece pose estimation method, apparatus, device, medium, and product. The workpiece pose estimation method provided in this application embodiment will be described first below.

[0024] Figure 1 A flowchart illustrating a workpiece pose estimation method according to an embodiment of this application is shown. Figure 1 As shown, the workpiece pose estimation method provided in this application embodiment is applied to electronic devices and includes the following steps 101-106, wherein: Step 101: Obtain a first image of multiple workpieces stacked in disorder, the first image including a first depth image and a first RGB image of the multiple workpieces.

[0025] The first camera is positioned above the material frame to capture images of the randomly stacked workpieces within the frame, obtaining a first image. Electronic devices can establish a communication connection with the first camera using the open-source software development kit (SDK) provided by the camera manufacturer; the SDK interface is used to set the first camera's resolution, frame rate, exposure time, and gain parameters; and scripts (such as those based on Python or C++) are written to call the SDK interface to acquire images of the randomly stacked workpieces within the frame. The first camera can be an RGB-D camera, used to acquire a first RGB image (including color information) and a first depth image.

[0026] Step 102: Based on the first depth image and the first RGB image, determine the first target point cloud data of the target workpiece, wherein the target workpiece is the workpiece located on the top layer among the disordered stacked workpieces.

[0027] The first depth image and the first RGB image are analyzed. For example, the first depth image and the first RGB image are aligned, and the aligned image is segmented to obtain the point cloud data of each workpiece. The target workpiece is determined based on the point cloud data of each workpiece. For example, the depth mean of the point cloud data of each workpiece is calculated. The workpiece corresponding to the minimum depth mean is taken as the target workpiece, and the point cloud data of the target workpiece is the first target point cloud data.

[0028] Step 103: Calculate the gripping position and gripping angle of the robotic arm based on the first target point cloud data of the target workpiece.

[0029] The gripping position and gripping angle refer to the gripping position and angle of the robotic arm in the robotic arm coordinate system when gripping the target workpiece. The first target point cloud data is the point cloud data in the camera coordinate system. The first target point cloud data can be transformed into the robotic arm coordinate system before determining the gripping position and gripping angle.

[0030] Step 104: Grab the target workpiece to the transfer platform according to the gripping position and gripping angle of the robotic arm, and acquire a second captured image, the second captured image including a second depth image and a second RGB image of the target workpiece.

[0031] After determining the gripping position and angle, the robotic arm transfers the target workpiece to the transfer station based on the determined gripping position and angle. This can improve the success rate of identifying slender workpieces in the matching process in disordered stacking scenarios, thereby improving the reliability of the robotic arm in gripping the workpiece.

[0032] The second camera is positioned above the transfer platform to capture images of the target workpiece on the platform, obtaining a second image. Electronic devices can establish a communication connection with the second camera using the open-source software development kit (SDK) provided by the camera manufacturer; the SDK interface is used to set the resolution, frame rate, exposure time, and gain parameters of the second camera; and scripts (such as those based on Python or C++) are written to call the SDK interface to acquire images of the target workpiece on the transfer platform. The second camera can be an RGB-D camera, used to acquire a second RGB image (including color information) and a second depth image.

[0033] Step 105: Based on the second depth image and the second RGB image, obtain the second target point cloud data of the target workpiece and the first pose of the target workpiece.

[0034] The second depth image and the second RGB image are analyzed. For example, the second depth image and the second RGB image are aligned, and the aligned image is segmented (that is, the target workpiece is separated from the background region) to obtain the mask of the target workpiece. Based on the mask of the target workpiece and the second RGB image, the first pose of the target workpiece is determined. This first pose can be regarded as a coarse pose estimation result.

[0035] Since the workpiece on the transfer platform is only the target workpiece, the workpiece in the captured second depth image and second RGB image is also only the target workpiece. When estimating the pose of the workpiece based on the second depth image and second RGB image, the interference of other workpieces can be eliminated, making the subsequent pose estimation of the target workpiece more accurate.

[0036] Step 106: Based on the first pose of the target workpiece, the second target point cloud data, and the preset point cloud data, perform pose estimation on the target workpiece to obtain the target pose of the target workpiece.

[0037] The preset point cloud data can refer to the point cloud of the target workpiece CAD model obtained from a preset CAD model file, and the target pose can refer to the 6D pose, that is, the target pose is obtained by further refining the pose of the target workpiece.

[0038] Through the above process, the grasping position and angle of the target workpiece are first determined based on the first captured image. The robotic arm then grasps the target workpiece to the transfer platform using the determined grasping position and angle. This step improves the success rate of recognizing slender workpieces in the matching process in disordered stacking scenarios, thereby enhancing the reliability of the robotic arm's workpiece grasping. Next, a second captured image is obtained by capturing the target workpiece on the transfer platform. Based on the second captured image, the pose of the target workpiece is first roughly estimated and then finely estimated. This allows the target workpiece to be transferred from a disordered stacking scenario to a single-object pose estimation scenario. Furthermore, by employing a mechanism combining coarse and fine pose estimation, not only is pose estimation of the workpiece in disordered stacking scenarios achieved, but the accuracy of the target workpiece's pose estimation is also improved.

[0039] Figure 2 A flowchart illustrating a workpiece pose estimation method according to an embodiment of this application is shown. Figure 2 As shown, the workpiece pose estimation method provided in this application embodiment is applied to electronic devices and includes the following steps 201-208, wherein: Step 201: Obtain a first image of multiple workpieces stacked in disorder, the first image including a first depth image and a first RGB image of the multiple workpieces.

[0040] For details of step 201, please refer to the description in step 101, which will not be repeated here.

[0041] Step 202: Preprocess the first depth image and the first RGB image to obtain the workpiece mask.

[0042] For example, the first depth image and the first RGB image are aligned to obtain first aligned image data; the first aligned image data is then segmented to obtain the workpiece mask.

[0043] Specifically, the intrinsic parameters, extrinsic rotation parameters, and offset parameters of the depth camera and RGB camera in the first camera are obtained respectively. The coordinates of each pixel in the first depth image are traversed, and the coordinates of the corresponding pixel in the first RGB image are calculated. Each pixel in the first depth image is projected onto the coordinate system of the first RGB image to align the first depth image and the first RGB image, thereby obtaining the first aligned image data of the workpiece. This ensures that the resolution of the first depth image and the first RGB image are consistent and that the pixels correspond one-to-one, thus guaranteeing the accurate spatial matching of color information and depth information.

[0044] The first aligned image data is segmented, for example, using the YOLO11-SEG model to obtain a segmented workpiece mask. The workpiece mask can be considered a mask image, with each pixel corresponding one-to-one with a pixel in the first RGB image. In the workpiece mask, pixels of the same workpiece have the same pixel value, pixels of different workpieces have different pixel values, and the pixel values ​​of each workpiece and the background region are also different. For example, if the first aligned image data includes two workpiece data (i.e., first workpiece data and second workpiece data) and background region data, then in the workpiece mask, the pixel value of the first workpiece can be set to a first value (e.g., red), the pixel value of the second workpiece can be set to a second value (e.g., yellow), and the pixel value of the background region can be set to a third value (e.g., black).

[0045] The above process yields a workpiece mask, which facilitates subsequent determination of the target workpiece based on the workpiece mask.

[0046] Step 203: Segment the first workpiece point cloud data according to the workpiece mask to obtain the second workpiece point cloud data, wherein the first workpiece point cloud data is obtained by point cloud conversion based on the first depth image.

[0047] Step 2031: Perform point cloud conversion on the first depth image to obtain the point cloud data of the first workpiece.

[0048] Step 2032: Project each first point in the first workpiece point cloud data onto the pixels of the first RGB image; for each first pixel projected onto the first RGB image, if the second pixel corresponding to the first pixel in the workpiece mask satisfies a first preset value, then retain the first point corresponding to the first pixel. Step 2033: Obtain the second workpiece point cloud data based on all the retained first points.

[0049] For example, the first workpiece point cloud data includes multiple points (for ease of description, the points in the first workpiece point cloud data are referred to as the first point). For the first point in the first workpiece point cloud data, the first point is projected onto the pixel coordinates of the first RGB image (pixel coordinates can be understood as the position of a pixel). For a first point whose pixel coordinates have been projected onto the first RGB image, the pixel value corresponding to the first point's pixel coordinates projected onto the first RGB image is found in the workpiece mask. Specifically, the pixels of the workpiece mask correspond one-to-one with the pixels of the first RGB image. If the first point can be projected onto the first pixel of the first RGB image, then the second pixel corresponding to the first pixel can be found in the workpiece mask. If the pixel value of the second pixel is a first preset value, for example, the first preset value can be set to a value other than 0 (0 represents the background area), then it is considered that the first workpiece point cloud data has been segmented using the workpiece mask, and the first point is saved.

[0050] Repeat the above process until all the first points included in the first workpiece point cloud data have been traversed, and finally obtain the second workpiece point cloud data based on all the retained first points.

[0051] Through the above processing, the point cloud data of the stacked workpieces can be separated from the point cloud data of the first workpiece. This segmentation method does not consider the occlusion between workpieces, so it will not divide the point cloud data of the stacked workpieces into discontinuous small segments, which can improve the point cloud data segmentation effect and provide a basis for subsequent accurate estimation of workpiece pose.

[0052] Step 204: Analyze the second workpiece point cloud data to determine the first target point cloud data of the target workpiece.

[0053] The second workpiece point cloud data consists of point cloud data from multiple workpieces stacked together. The workpiece located at the top layer, i.e., the target workpiece, needs to be segmented from these multiple workpieces. Specifically, the second workpiece point cloud data is analyzed to obtain the point cloud data belonging to each workpiece; the depth mean of the point cloud data belonging to each workpiece is calculated; and the point cloud data of the workpiece with the smallest depth mean is used as the first target point cloud data of the target workpiece.

[0054] For example, the first point in the second workpiece point cloud data is analyzed. This first point is obtained by filtering all first points included in the first workpiece point cloud data through steps 2031-2033. Simultaneously, this first point needs to be marked. The marking method is as follows: a mark is added to the first points retained through step 2032, and the same mark is added to first points belonging to the same workpiece. Since pixels belonging to the same workpiece in the workpiece mask have the same pixel value, marking in this way ensures that the marks of first points belonging to the same workpiece are also the same. When analyzing the first points in the second workpiece point cloud data, it is possible to directly determine whether each first point belongs to the same workpiece based on the mark of each first point.

[0055] After obtaining the point cloud data of each workpiece, the point cloud data of the workpiece is transformed into a depth value, or the pixel value corresponding to each point in the point cloud data of the workpiece is obtained in the first depth map, and the pixel value corresponding to the pixel value is the depth value; the average depth value is calculated for the point cloud data of each workpiece; the point cloud data of the workpiece with the smallest average depth value is taken as the first target point cloud data of the target workpiece, wherein the target workpiece is the workpiece stacked on the top layer, that is, the workpiece to be grasped.

[0056] In the above method, the first target point cloud data of the target workpiece is determined by the average depth of each workpiece, which can filter out the target workpieces stacked on the top layer, making it easier for the robotic arm to grasp and transfer them to the transfer station.

[0057] Step 205: Calculate the gripping position and gripping angle of the robotic arm based on the first target point cloud data of the target workpiece.

[0058] Specifically, coordinate transformation is performed on the first target point cloud data to obtain intermediate point cloud data of the first target point cloud data in the first coordinate system; based on the position of each point in the intermediate point cloud data, the gripping position of the target workpiece in the first coordinate system is determined; principal component analysis is performed on the intermediate point cloud data to determine the gripping angle of the target workpiece in the first coordinate system.

[0059] The first target point cloud data is data in the camera coordinate system and needs to be transformed to the first coordinate system for calculation. The first coordinate system can be the coordinate system of the robot arm base. For ease of description, the point cloud data after the first target point cloud data is transformed to the first coordinate system is called intermediate point cloud data.

[0060] For example, the mean coordinates of each point in the intermediate point cloud data in the first coordinate system are calculated to obtain the mean coordinates, which are then used as the gripping position for the robotic arm to grasp the target workpiece. Principal component analysis is performed on the second point cloud data to determine the gripping angle of the target workpiece in the first coordinate system.

[0061] Principal component analysis is performed on the intermediate point cloud data to determine the grasping angle of the target workpiece in the first coordinate system, including: Principal component analysis was performed on the intermediate point cloud data to obtain multiple principal component components; The principal component components are sorted, and the direction indicated by the principal component component ranked first is taken as the first direction of the second coordinate system; The normal direction of the projection of the first direction onto the XY plane of the first coordinate system is taken as the second direction of the second coordinate system; The third direction of the second coordinate system is determined based on the first direction and the second direction; The grasping angle of the target workpiece in the first coordinate system is determined based on the first direction, the second direction, and the third direction.

[0062] In the above, each principal component can be a coordinate in a first coordinate system, which corresponds to a vector indicating a direction; the second coordinate system can be a custom three-dimensional coordinate system. Multiple principal components can be sorted from most important to least important, and the direction indicated by the principal component ranked first can be used as the first direction of the second coordinate system.

[0063] If the Z component of the third direction in the first coordinate system is less than the target preset value, then the second direction and the third direction are rotated; the gripping angle of the target workpiece in the first coordinate system is determined according to the first direction, the second direction obtained after rotation, and the third direction, wherein the Z component is the depth direction component of the first coordinate system.

[0064] The three-dimensional representation of the first coordinate system is XYZ, and the three-dimensional representation of the second coordinate system is xyz (using the same right-handed rotation coordinate system order). XYZ and xyz represent the horizontal, vertical, and depth directions of the first and second coordinate systems, respectively. For example, in the second coordinate system, the first direction can be the x-direction (horizontal), the second direction can be the y-direction (vertical), and the third direction can be the z-direction (depth).

[0065] For example, the direction indicated by the principal component with the highest importance is taken as the x-direction of the second coordinate system, such as... Figure 3 As shown, that is Figure 3 The principal axis direction of the target workpiece is determined; the normal direction of the projection of the x-direction onto the XY plane of the first coordinate system is taken as the y-direction of the second coordinate system; the z-direction of the second coordinate system is determined based on the cross product of the vector indicating the x-direction and the vector indicating the y-direction. For example, the unit vector in the x-direction and the unit vector in the y-direction are cross-producted to obtain the unit vector in the z-direction.

[0066] If the Z component of the z direction in the first coordinate system is greater than or equal to 0 (for example, the Z component of the unit vector in the z direction is greater than or equal to 0), in this case, since the robotic arm grips the workpiece from top to bottom, the gripping angle of the target workpiece in the first coordinate system can be directly determined based on the x, y and z directions.

[0067] If the Z component of the z direction in the first coordinate system is less than 0 (for example, the Z component of the unit vector in the z direction is less than 0), in order to ensure that the robotic arm grips the workpiece from top to bottom, the y and z directions are rotated by 180°; then the gripping angle of the target workpiece in the first coordinate system is determined based on the x direction, the y direction obtained after rotation, and the z direction obtained after rotation.

[0068] Step 206: Grasp the target workpiece onto the transfer platform according to the gripping position and angle of the robotic arm, and acquire a second image. The second image includes a second depth image and a second RGB image of the target workpiece. The second image is obtained by photographing the target workpiece on the transfer platform.

[0069] The robotic arm can grasp the target workpiece and place it on the transfer platform according to the grasping position and grasping angle of the target workpiece calculated in step 205.

[0070] The second camera is positioned above the transfer platform to capture images of the target workpiece on the platform, obtaining a second image. Electronic devices can establish a communication connection with the second camera using the open software development kit (SDK) provided by the camera manufacturer; the SDK interface is used to set the resolution, frame rate, exposure time, and gain parameters of the second camera; and scripts (such as those based on Python or C++) are written to call the SDK interface to acquire images of the target workpiece on the transfer platform. The second camera can be an RGB-D camera, used to acquire a second RGB image (including color information) and a second depth image.

[0071] Since the workpiece on the transfer platform is only the target workpiece, the second depth image and the second RGB image captured also contain only the target workpiece. When estimating the pose of the workpiece based on the second depth image and the second RGB image, the interference from other workpieces can be eliminated, making the pose estimation of the target workpiece more accurate.

[0072] Step 207: Based on the second depth image and the second RGB image, obtain the second target point cloud data of the target workpiece and the first pose of the target workpiece.

[0073] Specifically, the second depth image and the second RGB image are preprocessed to obtain a mask of the target workpiece; pose estimation is performed based on the mask of the target workpiece, the second RGB image, and the preset model data of the target workpiece to obtain the first pose of the target workpiece; the third workpiece point cloud data is segmented based on the mask of the target workpiece to obtain the second target point cloud data of the target workpiece, wherein the third workpiece point cloud data is obtained by point cloud transformation based on the second depth image.

[0074] For example, the second depth image and the second RGB image are aligned to obtain second aligned image data; the second aligned image data is then segmented to obtain the workpiece mask.

[0075] Specifically, the intrinsic parameters, extrinsic rotation parameters, and offset parameters of the depth camera and RGB camera in the second camera are obtained respectively. The coordinates of each pixel in the second depth image are traversed, and the coordinates of the corresponding pixel in the second RGB image are calculated. Each pixel in the second depth image is projected into the coordinate system of the second RGB image to align the second depth image and the second RGB image, thereby obtaining the second aligned image data of the workpiece. That is, the resolution of the second depth image and the second RGB image are consistent, and the pixels correspond one-to-one, thus ensuring the accurate spatial matching of color information and depth information.

[0076] The second aligned image data is segmented, for example, using the YOLO11-SEG model to segment the second aligned image data to obtain a mask for the target workpiece.

[0077] Pose estimation is performed based on the mask of the target workpiece, the second RGB image, and the preset model data of the target workpiece to obtain the first pose of the target workpiece. For example, the mask of the target workpiece, the second RGB image, and the preset model data of the target workpiece are input together into the FoundationPose model for pose estimation to obtain the first pose of the target workpiece. The preset model data can be the CAD model file of the target workpiece. The first pose belongs to coarse matching pose estimation. The FoundationPose model is a model based on a large language model and a diffusion model and is used for 6D pose estimation and tracking.

[0078] The third workpiece point cloud data is segmented according to the mask of the target workpiece to obtain the second target point cloud data of the target workpiece. Specifically, this includes: projecting each second point in the third workpiece point cloud data onto a pixel in the second RGB image; for each third pixel in the pixels projected onto the second RGB image, if the fourth pixel corresponding to the third pixel in the mask of the target workpiece satisfies a second preset value, then the second point corresponding to the third pixel is retained; and the second target point cloud data is obtained based on all retained second points.

[0079] The second depth image is transformed into point cloud data to obtain the third workpiece point cloud data. For example, the third workpiece point cloud data includes multiple points. For ease of description, these points are referred to as the second points. The second points in the third workpiece point cloud data are projected onto the pixel coordinates of the second RGB image (pixel coordinates can be understood as the position of a pixel). Since the pixels of the target workpiece mask correspond one-to-one with the pixels of the second RGB image, the second points already projected onto the second RGB image are recorded as the third pixels. For each third pixel, the fourth pixel corresponding to its coordinates is found in the target workpiece mask. If the pixel value of the fourth pixel is a second preset value, the second point corresponding to that third pixel is retained. The second preset value can be set to any value other than 0 (0 represents the background area). In this case, it is considered that the third workpiece point cloud data has been segmented using the target workpiece mask.

[0080] Repeat the above process until all the second points included in the third workpiece point cloud data have been traversed. Based on all the retained second points, the second target point cloud data is obtained.

[0081] Through the above processing, the second target point cloud data can be segmented from the third workpiece point cloud data. Since there are no other workpieces in the third workpiece point cloud data that occlude the target workpiece, the segmented second target point cloud data is complete and unoccluded, which is beneficial for subsequent pose estimation of the target workpiece based on the second target point cloud data and improves the accuracy of workpiece pose estimation.

[0082] Step 208: Perform pose estimation on the target workpiece based on the first pose of the target workpiece, the second target point cloud data, and the preset point cloud data to obtain the target pose of the target workpiece.

[0083] Specifically, the preset point cloud data of the target workpiece is obtained from the preset model file; the first pose of the target workpiece is used as the initial pose of the preset point cloud registration algorithm; the second target point cloud data and the preset point cloud data are registered using the preset point cloud registration algorithm based on the initial pose to obtain the target pose of the target workpiece, so as to achieve a more refined estimation of the pose of the target workpiece.

[0084] The aforementioned preset point cloud registration algorithm can be the Iterative Closest Point (ICP) algorithm, or other point cloud registration algorithms; no restrictions are placed here. The preset model file can be, for example, a CAD model file of the target workpiece.

[0085] The workpiece pose estimation method provided in the above embodiments is divided into two stages for workpiece pose estimation in stacked scenarios. In the first stage, for multiple workpieces in a disordered stacked scenario, the workpiece located on the top layer is first determined as the target workpiece. Then, the target workpiece is grasped onto the transfer platform by calculating the grasping position and grasping angle of the robotic arm. In the second stage, for a single target workpiece on the transfer platform, the pose of the target workpiece is first roughly estimated. Then, a higher accuracy pose estimation is achieved through the result of the rough estimation, which has real-time performance.

[0086] The workpiece pose estimation method provided in the above embodiments can solve the reliability problem of grasping slender workpieces on the one hand, and solve the problem of the point cloud data collected by the camera being segmented by the stacked workpieces due to occlusion between workpieces in complex scenarios where slender workpieces are stacked in disorder. This makes it easy for slender workpieces to fail to be recognized during pose matching, resulting in poor segmentation and matching of the target workpiece and the inability to give an accurate pose.

[0087] Figure 4 A structural diagram of the workpiece pose estimation device provided in an embodiment of this application is shown. Figure 4 As shown, the workpiece pose estimation device 400 includes: The first acquisition module 401 is used to acquire a first captured image of multiple workpieces stacked in a disordered manner. The first captured image includes a first depth image and a first RGB image of the multiple workpieces. The determining module 402 is used to determine the first target point cloud data of the target workpiece based on the first depth image and the first RGB image, wherein the target workpiece is the workpiece located on the top layer among the disordered stacked multiple workpieces; The calculation module 403 is used to calculate the gripping position and gripping angle of the robotic arm based on the first target point cloud data of the target workpiece; The gripping module 404 is used to grip the target workpiece to the transfer platform according to the gripping position and gripping angle of the robotic arm; The second acquisition module 405 is used to acquire a second captured image, the second captured image including a second depth image and a second RGB image of the target workpiece; The third acquisition module 406 is used to obtain the second target point cloud data of the target workpiece and the first pose of the target workpiece based on the second depth image and the second RGB image. The estimation module 407 is used to estimate the pose of the target workpiece based on the first pose of the target workpiece, the second target point cloud data, and the preset point cloud data, so as to obtain the target pose of the target workpiece.

[0088] The workpiece pose estimation device 400 provided in this application embodiment can realize the various processes implemented in the aforementioned workpiece pose estimation method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0089] Figure 5 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0090] The electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0091] Specifically, the processor 601 described above can be configured to implement one or more integrated circuits according to the embodiments of this application.

[0092] Memory 602 may include a large-capacity memory for data or instructions. Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to the first or second aspect of this disclosure.

[0093] The processor 601 reads and executes computer program instructions stored in the memory 602 to implement any of the workpiece pose estimation methods in the above embodiments.

[0094] In one example, the electronic device may also include a communication interface 603 and a bus 610. For example, Figure 5 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.

[0095] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0096] Bus 610 includes hardware, software, or both, coupling hardware and software components together. Where appropriate, bus 610 may include one or more buses. This application considers any suitable bus or interconnect.

[0097] Furthermore, in conjunction with the workpiece pose estimation methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the workpiece pose estimation methods in the above embodiments.

[0098] This application provides a computer program product in which the instructions are executed by the processor of an electronic device, causing the electronic device to perform any of the workpiece pose estimation methods described in the above embodiments.

[0099] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described as examples. However, the method process of this application is not limited to the specific steps described. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0100] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. The code segments can be downloaded via computer networks such as the Internet or intranets.

[0101] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0102] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A workpiece pose estimation method, characterized by, The method comprises: acquiring a first shooting image of a plurality of workpieces in a disordered stack, the first shooting image comprising a first depth image and a first RGB image of the plurality of workpieces; determining first target point cloud data of a target workpiece according to the first depth image and the first RGB image, the target workpiece being a workpiece located at the uppermost layer in the plurality of workpieces in the disordered stack; calculating a grabbing position and a grabbing angle of a mechanical arm based on the first target point cloud data of the target workpiece; grabbing the target workpiece to a transfer table according to the grabbing position and the grabbing angle of the mechanical arm, and acquiring a second shooting image, the second shooting image comprising a second depth image and a second RGB image of the target workpiece; obtaining second target point cloud data of the target workpiece and a first pose of the target workpiece according to the second depth image and the second RGB image; performing pose estimation on the target workpiece according to the first pose of the target workpiece, the second target point cloud data and preset point cloud data to obtain a target pose of the target workpiece.

2. The workpiece pose estimation method of claim 1, wherein, According to the first depth image and the first RGB image, the first target point cloud data of the target workpiece is determined, comprising: preprocessing the first depth image and the first RGB image to obtain a workpiece mask; segmenting first workpiece point cloud data according to the workpiece mask to obtain second workpiece point cloud data, wherein the first workpiece point cloud data is obtained by point cloud conversion based on the first depth image; analyzing the second workpiece point cloud data to determine the first target point cloud data of the target workpiece.

3. The workpiece pose estimation method of claim 2, wherein, The first depth image and the first RGB image are preprocessed to obtain a workpiece mask, comprising: aligning the first depth image and the first RGB image to obtain first aligned image data; performing image segmentation on the first aligned image data to obtain the workpiece mask.

4. The workpiece pose estimation method of claim 2, wherein, The first workpiece point cloud data is segmented according to the workpiece mask to obtain the second workpiece point cloud data, comprising: projecting each first point in the first workpiece point cloud data to a pixel point of the first RGB image; for each first pixel point projected to the pixel point of the first RGB image, if a second pixel point corresponding to the first pixel point in the workpiece mask satisfies a first preset value, the first point corresponding to the first pixel point is retained; obtaining the second workpiece point cloud data according to all retained first points.

5. The workpiece pose estimation method of claim 2, wherein, The second workpiece point cloud data is analyzed to determine the first target point cloud data of the target workpiece, comprising: analyzing the second workpiece point cloud data to obtain point cloud data belonging to each workpiece in the second workpiece point cloud data; calculating the depth mean value of the point cloud data belonging to each workpiece; taking the point cloud data of the workpiece with the smallest depth mean value as the first target point cloud data of the target workpiece.

6. The workpiece pose estimation method of any one of claims 1-5, wherein, The first target point cloud data is converted to obtain intermediate point cloud data of the first target point cloud data in a first coordinate system, and the grabbing position and the grabbing angle of the mechanical arm are calculated based on the intermediate point cloud data. ​ According to the position of each point in the intermediate point cloud data, a grasping position of the target workpiece in the first coordinate system is determined; Principal component analysis is performed on the intermediate point cloud data to determine a grasping angle of the target workpiece in the first coordinate system.

7. The workpiece pose estimation method of claim 6, wherein, The principal component analysis on the intermediate point cloud data to determine the grasping angle of the target workpiece in the first coordinate system comprises: Perform principal component analysis on the intermediate point cloud data to obtain a plurality of principal component components; Sort the plurality of principal component components, and take the direction indicated by the principal component component ranked first as the first direction of the second coordinate system; Take the normal direction of the projection of the first direction on the XY plane of the first coordinate system as the second direction of the second coordinate system; Determine the third direction of the second coordinate system according to the first direction and the second direction; Determine the grasping angle of the target workpiece in the first coordinate system according to the first direction, the second direction and the third direction.

8. The workpiece pose estimation method of claim 7, wherein, The determination of the grasping angle of the target workpiece in the first coordinate system according to the first direction, the second direction and the third direction comprises: If the Z component of the third direction in the first coordinate system is less than a target preset value, rotate the second direction and the third direction; Determine the grasping angle of the target workpiece in the first coordinate system according to the first direction, the second direction and the third direction.

9. The workpiece pose estimation method of claim 1, wherein, According to the second depth image and the second RGB image, the second target point cloud data of the target workpiece and the first pose of the target workpiece are obtained, comprising: Preprocessing the second depth image and the second RGB image to obtain a mask of the target workpiece; According to the mask of the target workpiece, the second RGB image, and the preset model data of the target workpiece, pose estimation is performed to obtain the first pose of the target workpiece; According to the mask of the target workpiece, the third workpiece point cloud data is segmented to obtain the second target point cloud data of the target workpiece, wherein the third workpiece point cloud data is obtained by point cloud conversion based on the second depth image.

10. The workpiece pose estimation method of claim 9, wherein, According to the mask of the target workpiece, the third workpiece point cloud data is segmented to obtain the second target point cloud data of the target workpiece, comprising: Project each second point in the third workpiece point cloud data to a pixel point of the second RGB image; For each third pixel point projected to the pixel point of the second RGB image, if the corresponding fourth pixel point of the third pixel point in the mask of the target workpiece satisfies a second preset value, the second point corresponding to the third pixel point is retained; According to all the retained second points, the second target point cloud data is obtained.

11. The workpiece pose estimation method of claim 1, wherein, According to the first pose of the target workpiece, the second target point cloud data and the preset point cloud data, pose estimation is performed on the target workpiece to obtain the target pose of the target workpiece, comprising: Obtain the preset point cloud data of the target workpiece from a preset model file; Take the first pose of the target workpiece as the initial pose of the preset point cloud registration algorithm; The second target point cloud data and the preset point cloud data are point cloud registered according to the initial pose and by using the preset point cloud registration algorithm, to obtain a target pose of the target workpiece.

12. A workpiece pose estimation apparatus, characterized by comprising: The device comprises: A first acquisition module is configured to acquire a first captured image of a plurality of workpieces in a disordered stack, the first captured image comprising a first depth image and a first RGB image of the plurality of workpieces. A determination module is configured to determine first target point cloud data of a target workpiece according to the first depth image and the first RGB image, the target workpiece being a workpiece located at an uppermost layer in the plurality of workpieces in the disordered stack. A calculation module is configured to calculate a grabbing position and a grabbing angle of a mechanical arm based on the first target point cloud data of the target workpiece. A grabbing module is configured to grab the target workpiece to a transfer table according to the grabbing position and the grabbing angle of the mechanical arm. A second acquisition module is configured to acquire a second captured image, the second captured image comprising a second depth image and a second RGB image of the target workpiece. A third acquisition module is configured to obtain second target point cloud data of the target workpiece and a first pose of the target workpiece according to the second depth image and the second RGB image. An estimation module is configured to perform pose estimation on the target workpiece according to the first pose of the target workpiece, the second target point cloud data, and preset point cloud data, to obtain a target pose of the target workpiece.

13. An electronic device, comprising: A processor and a memory having computer program instructions stored therein; The processor executes the computer program instructions to implement the workpiece pose estimation method according to any one of claims 1-11. The computer program instructions are stored on the computer readable storage medium, and when executed by a processor, implement the workpiece pose estimation method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The instructions in the computer program product are executed by a processor of an electronic device, so that the electronic device performs the workpiece pose estimation method according to any one of claims 1-11.

15. A computer program product, characterised in that, ​

Images