Method for six-degree-of-freedom (6D) object pose estimation using artificial intelligence (AI)

Abstract

A method for estimating a 6D pose of an object includes: (1) obtaining a low-resolution image and a high-resolution image from a viewpoint of a scene using a camera; (2) performing pose detection on the low-resolution image to obtain a category label, a region mask and an initial pose of an object in the scene; (3) performing pose refinement on each of the detected objects on the high-resolution image comprises: a) generating a cropped image from the high-resolution image based on a region mask of each detected object, and generating a rendered image of each detected object based on a CAD model, a current pose of each detected object, and camera data; b) calculating a refined object pose for each of the detected objects by comparing the cropped image and the rendered image, and c) updating the current pose of each detected object with the refined object pose, and (4) repeating step (3) until a criterion is met.

Description

Technical Field

[0001] This disclosure relates to a method for estimating the pose of a six-degree-of-freedom (6D) object, and more specifically, to a method for estimating the pose of a 6D object using artificial intelligence (AI). Background Technology

[0002] Six-degree-of-freedom (6D) object pose estimation is applied in computer vision and robotics to determine the 6D pose of an object in three-dimensional (3D) space. For example, determining the 6D pose of an object in 3D space involves estimating the object's position and orientation within a scene. In applications such as random bin picking and / or pick-and-place, current vision algorithms are sensitive to scene lighting conditions and object occlusion. Additionally, adjustments are needed for each type of object in the scene. This affects the accuracy of 6D object pose estimation for images of that object in the scene, especially those that are randomly placed or stacked.

[0003] Therefore, it is necessary to solve the problem of accuracy in 6D object pose estimation under different lighting conditions, and further improve the robustness of 6D object pose estimation in various applications. Summary of the Invention

[0004] In an exemplary embodiment, this disclosure provides a method for estimating the six-degree-of-freedom (6D) pose of objects in a scene. The method includes:

[0005] (1) Obtain a first image and a second image from the viewpoint of the scene using a camera, wherein the second image has a higher resolution than the first image; (2) Perform pose detection on the first image to obtain the category labels, region masks and initial poses of objects in the scene; (3) Perform pose refinement on each of the detected objects on the second image, wherein performing pose refinement includes: a) generating a cropped image from the second image based on the region mask of each detected object; and generating a rendered image of each detected object based on a computer-aided design (CAD) model, the current pose of each detected object and intrinsic and extrinsic parameter data of the camera associated with the second image; b) calculating the refined object pose of each of the detected objects by comparing the cropped image and the rendered image; and c) updating the current pose of each detected object with the refined object pose; and (4) repeating step (3) until the criteria are met.

[0006] Obtaining the first image and the second image includes: capturing the second image from the viewpoint of the scene using a camera, and downscaling the second image to the first image.

[0007] The region mask for each detected object is obtained by performing pose detection via at least one object detection process based on pose detection, detecting the class labels and bounding boxes of objects in a first image; and using the bounding box of each detected object as a region mask.

[0008] The region mask for each detected object is obtained by performing pose detection via at least one object detection process based on pose detection, detecting the class label and instance mask of the object in the first image; and using the instance mask of each detected object as a region mask.

[0009] The region mask for each detected object is obtained by performing pose detection by rendering each object using the corresponding CAD model and the current pose of each object.

[0010] Pose detection is performed based on deep neural networks or multiple deep neural networks running sequentially or in parallel.

[0011] The rendered image for each detected object includes at least one of the following: a color image, a grayscale image, a depth image, a point cloud, or a Normalized Object Coordinate Space (NOCS) graph.

[0012] Calculating a refined object pose for each of the detected objects by comparing a cropped image and a rendered image includes: extracting two-dimensional (2D) keypoints on the cropped and rendered images; matching the 2D keypoints; locating the three-dimensional (3D) position of each matched 2D keypoint from the rendered image; and applying a perspective n-point solver to the matched keypoints to calculate the refined object pose.

[0013] The criteria include: the number of times step (3) is repeated is greater than a first predefined threshold, or the difference in object pose between two consecutive iterations is less than a second predefined threshold.

[0014] The method further includes: (5) obtaining a depth image or point cloud image of the scene; (6) aligning the rendered image with the depth image or the point cloud image for each detected object to obtain a refined object pose; and (7) outputting the refined object pose for each detected object. The method also includes controlling the robot to perform at least one of bin picking and bin placement based on the refined object pose.

[0015] In another exemplary embodiment, this disclosure provides a method for estimating the six-degrees-of-freedom (6D) pose of one or more objects in a scene. The method includes:

[0016] Using one or more cameras, one or more first images are obtained from multiple viewpoints of the scene; (1) pose detection is performed on the one or more first images from each viewpoint to obtain object category labels, region masks, and initial poses; (2) the detected objects are matched across multiple viewpoints; (3) a multi-view pose triangulation process is performed on the pose of each matched object across multiple viewpoints to compute a refined object pose using intrinsic and extrinsic parameter data from one or more cameras associated with the one or more first images from each viewpoint; and the refined object pose of each matched object is output. The method also includes controlling a robot to perform bin picking and / or bin placement based on the refined object poses.

[0017] Matching detected objects across multiple viewpoints includes one or more of the following: (1) determining whether the category labels of the detected objects from each view are the same; (2) calculating the Euclidean translation distance of the detected objects from each viewpoint in the common world coordinate system; (3) calculating the rotational angular distance of the detected objects from each viewpoint in the common world coordinate system; and (4) calculating the triangulation pose inconsistency error index by performing a multi-view pose triangulation procedure for each matched object.

[0018] The multi-view pose triangulation process for each matched object includes solving the optimization problem of the refined object pose for each matched object.

[0019] The method further includes: (4) acquiring one or more second images from multiple viewpoints of the scene using one or more cameras, wherein the one or more second images have a higher resolution than one or more first images; (5) refining the pose of each matched object using the one or more second images from each viewpoint by: a) generating a cropped image from the one or more second images based on a region mask of each matched object, and generating each... (a) Rendered image of a matched object; b) Calculate the refined object pose of each of the matched objects by comparing the cropped image and the rendered image; and c) Update the current pose using the refined object pose; (6) Repeat step (5) iteratively until the first criterion is met; (7) Perform a multi-view pose triangulation procedure on the updated current pose across multiple viewpoints for each matched object to calculate the refined object pose using intrinsic and extrinsic parameter data from one or more cameras associated with one or more second images from each viewpoint; and (8) Repeat steps (5) to (7) until the second criterion is met.

[0020] Step (4) of obtaining one or more second images from multiple viewpoints of the scene includes: acquiring a high-resolution image via a camera; downscaling the high-resolution image to a low-resolution image; using the low-resolution image for steps prior to step (4); and using the high-resolution image for steps after step (4).

[0021] In another exemplary embodiment, this disclosure provides an apparatus for estimating the six-degree-of-freedom (6D) pose of objects in a scene. The apparatus includes one or more processors and is configured to:

[0022] (1) Obtain a first image and a second image from the viewpoint of the scene, wherein the second image has a higher resolution than the first image; (2) Perform pose detection on the first image to obtain the category labels, region masks, and initial poses of objects in the scene; (3) Perform pose refinement on each of the detected objects on the second image, wherein performing pose refinement includes: a) generating a cropped image from the second image based on the region mask of each detected object; and generating a rendered image of each detected object based on a computer-aided design (CAD) model, the current pose of each detected object, and intrinsic and extrinsic parameter data of the camera associated with the second image; b) calculating the refined object pose of each of the detected objects by comparing the cropped image and the rendered image; and c) updating the current pose of each detected object using the refined object pose; and (4) repeating step (3) until the criteria are met.

[0023] The region mask for each detected object is obtained by performing pose detection via the following steps: at least one object detection process based on pose detection, detecting the class labels and bounding boxes of objects in a first image; and using the bounding box of each detected object as a region mask.

[0024] The region mask of each detected object is obtained by performing pose detection via the following steps: at least one object detection process based on pose detection, detecting the class label and instance mask of the object in the first image; and using the instance mask of each detected object as a region mask. Attached Figure Description

[0025] Figure 1 This is an application of a six-degree-of-freedom (6D) object pose estimation method according to exemplary embodiments of the present disclosure;

[0026] Figure 2 This is a schematic diagram of a 6D object pose estimation process or system for a single view according to exemplary embodiments of the present disclosure;

[0027] Figure 3aThis is a schematic diagram of a 6D object pose estimation process or system for multiple views according to an exemplary embodiment of the present disclosure;

[0028] Figure 3b This is a schematic diagram of a 6D object pose estimation process or system for multiple views according to another exemplary embodiment of this disclosure;

[0029] Figure 4 This is a schematic flowchart of a 6D object pose estimation process for a single view according to an exemplary embodiment of the present disclosure;

[0030] Figure 5a This is a schematic flowchart of a 6D object pose estimation process for multiple views according to an exemplary embodiment of the present disclosure;

[0031] Figure 5b This is a schematic flowchart of a 6D object pose estimation process for multiple views according to another exemplary embodiment of the present disclosure;

[0032] Figure 5c This is a schematic flowchart of a 6D object pose estimation process for multiple views according to another exemplary embodiment of this disclosure; and

[0033] Figure 6 This is a schematic diagram of an apparatus for a 6D object pose estimation process according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0034] Exemplary embodiments of this disclosure provide a method, apparatus, and non-transitory computer-readable medium for estimating the six-degree-of-freedom (6D) pose of objects in a scene. The objects in the scene may be of the same class or come from different classes.

[0035] Applications of computer vision and robotics, such as bin picking and other pick-and-place applications, require robust and accurate 6D object pose estimation. Robustness and accuracy of 6D object pose estimation are particularly important in scenes where objects are randomly placed and / or stacked and ready for picking and placing. Current practice commonly involves providing computer-aided design (CAD) models of objects in a scene as 3D textured or untextured mesh models for object identification. Traditional vision algorithms operate on high-resolution images and provide accurate 6D object pose estimation. However, traditional vision algorithms are highly sensitive to ambient lighting conditions and require careful tuning for each type of object being identified in the scene. Other traditional vision algorithms rely on 3D vision cameras to capture red, green, and blue depth (RGBD) information of the scene; however, these algorithms often require high-end, expensive 3D vision cameras to achieve high accuracy.

[0036] Embodiments of this disclosure provide a vision algorithm that combines artificial intelligence (AI)-based vision techniques with conventional vision techniques. AI-based vision techniques can be used for initial object pose detection based on low-resolution images. Conventional vision techniques can be used for object pose refinement based on high-resolution images to achieve higher accuracy. This vision algorithm, combining AI-based and conventional vision techniques, addresses robustness and accuracy issues in applications such as random bin picking and other pick-and-place applications.

[0037] Figure 1 This is an illustrative application of a six-degree-of-freedom (6D) object pose estimation method according to exemplary embodiments of the present disclosure.

[0038] Random bin picking and other pick-and-place applications ensure key milestones in industrial automation processes. These applications enable faster and more efficient material handling and provide significant flexibility for component lifting and conveying in production environments. For example, Figure 1 As shown, in the factory process, random bin picking and other pick and place applications 100 are able to identify items 106 in any orientation and / or in any configuration, select items 106 and then place them in bins 108 and / or on pallets.

[0039] like Figure 1 As shown, such random bin picking and other pick-and-place applications 100 employ a robot 102 and a vision system 104. The vision system 104 can be based on various vision technologies. Generally, these various vision technologies use one or more cameras with the same or different modalities (e.g., camera 210 shown) to record two-dimensional monochrome, color, or depth images of a scene from one or more viewpoints. For example, the orientation and / or configuration of object 106 is determined based on images of object 106 captured by the vision system 104 from one or more viewpoints. The robot 102 is then commanded to pick up object 106 from bin 108 and / or place object 106 into bin 108 based on the determined orientation and / or configuration.

[0040] In exemplary embodiments of this disclosure, artificial intelligence (AI) vision technology is combined with conventional vision technology and used in the vision system 104 of random bin picking and other pick-and-place applications 100. AI vision technology offers high detection rates and strong robustness to ambient lighting conditions, overcoming the shortcomings of conventional vision technology. However, AI vision technology is less efficient at processing high-resolution images, thus failing to provide high accuracy in object pose. This shortcoming of AI vision technology can be overcome by applying conventional vision technology to high-resolution images. Note that high resolution and low resolution are relative. For example, a 5MP color image of 2560x1920 pixels is considered high resolution compared to a 0.3MP color image of 640x480 pixels; however, it is considered low resolution compared to a 12MP color image of 4256x2832 pixels. In this disclosure, we will use 5MP as an example of high resolution and 0.3MP as an example of low resolution. It should be understood that this is for illustrative purposes only. This should in no way be construed as a precise definition of high resolution and low resolution. In extreme cases, high resolution and low resolution can refer to the same resolution.

[0041] Figure 2 This is a schematic diagram of a 6D object pose estimation process or system for a single view according to exemplary embodiments of the present disclosure.

[0042] like Figure 2 As shown, Figure 1 The vision system 104 of the random bin picking and other pick-and-place applications 100 combines AI-based vision with traditional vision and employs one or more cameras 210 to capture images from a single viewpoint. Cameras 210 are selected to be capable of capturing high-resolution images. For example, camera 210 could be a 5-megapixel (5MP) color camera. A 5MP color camera produces a resolution of 2560 × 1920 pixels per unit. Alternatively or additionally, other cameras, such as grayscale monochrome cameras and / or cameras with different resolutions, may also be used. The 5MP color image captured from the single viewpoint is then downscaled to a lower resolution image, as shown in image downscaling process 220. For example, a 5MP color image captured from a single viewpoint can be downscaled to a 0.3MP color image. The 5MP color image captured from a single viewpoint can also be downscaled to images of different resolutions, such as a 1.2MP color image or a 2MP color image.

[0043] Then, a 0.3MP low-resolution color image from a single viewpoint is input into the 6D pose detection process 230. The output of 230 is a set of pose detection results, each containing information about the detected objects, such as, but not limited to, category labels, location and orientation, and detection confidence scores. Additional object information (e.g., object masks and object boundaries) can be obtained directly from 230 or can be computed. Various 6D methods can be used for the pose detection process. The pose detection process 230 can be a single-stage pose predictor based on a neural network, or it can be a two-stage pose detection process consisting of an object detection stage and a pose estimation stage.

[0044] For a two-stage pose detection process employing an object detector, the image is generally input into a deep neural network (such as a mask-based convolutional neural network (R-CNN) or "You Only Look Once" (YOLO)) to generate a list of detected objects. The output contains at least information about the identity and location of each detected object. The object identity is represented by a category label. The location of the object in the image can be represented by an object region mask, which indicates the approximate image region to which the object belongs. The object region mask can be a bounding box, boundary contour, instance mask, or other forms. The neural network-based 2D object detection outputs the category labels and region masks of possible objects from a 0.3MP color image of a single-viewpoint image. Then, in the second stage, the detected category labels and object region masks are further processed to obtain the poses of all detected objects. Thus, the initial poses of these detected objects are estimated.

[0045] In the single-stage pose detection process 230 (i.e., the neural network-based single-stage pose predictor), the output includes at least a class label, a region mask, and the poses of all possible objects. If the region mask is not directly output, it can be calculated from the predicted pose and the object's CAD model. Thus, the class label, object region mask, and initial object pose for each detected object are obtained from the 6D pose detection process 230 for further processing of the 5MP color image captured from a single viewpoint and / or a downscaled 0.3MP color image.

[0046] The initial object pose from the neural network-based or AI-based pose detection process 230 may not be accurate enough for the application. 250 provides a pose refinement process that uses high-resolution images and conventional vision techniques to improve pose accuracy. The pose refinement process 250 receives two types of input: a high-resolution camera image and the output of the pose detection process 230, which includes category labels, region masks, and the pose of each detected object. The pose refinement process 250 can be applied to each detected object. For each detected object, the object region mask and a high-resolution 5MP color image captured from a single viewpoint are input into an image cropping process 240 to crop the captured 5MP color image to obtain a cropped camera image of each detected object. Since the object region mask is obtained from a low-resolution 0.3MP image, it needs to be upsampled to obtain a high-resolution 5MP image. Compared to the high-resolution 5MP camera image, the cropped camera image will have the same pixel density (pixels per inch) but a smaller image size. As is understood, multiple cropped camera images will be obtained, each corresponding to an object detected from 230. Generally, the neural network-based 6D pose detection process 230 and the image cropping process 240 process images of different resolutions.

[0047] For each detected object, a category label, object region mask, and initial object pose are input into rendering process 252 to obtain a rendered object image. If the objects in the scene come from different categories with different CAD models, the correct object CAD model is found from a pre-stored CAD model library using the category label. The rendered object image can include a color image, a monochrome image, a depth image, a point cloud, a Normalized Object Coordinate Space (NOCS) map, or other types of images. The rendered object image has the same pixel density as the captured high-resolution 5MP image, but with a smaller image size. The actual implementation of rendering process 252 can incorporate several optimizations for better computational and memory efficiency. For example, the entire scene with all detected objects can be rendered into a 5MP scene color image and a 5MP scene depth image, and then the rendered scene color image and scene depth image can be cropped using an object region mask to generate smaller object color images and object depth images for each detected object. Alternatively, a smaller field of view defined by the object region mask can be used to directly render an object color image and an object depth image for each detected object. These images are smaller in size but have the same pixel density as the captured 5MP color image.

[0048] In one exemplary embodiment, a rendered object color image with the same pixel density as the captured high-resolution 5MP color image and a cropped camera image generated from image cropping process 240 for each detected object are input into feature extraction and matching process 254. 2D keypoints are extracted from these two images in real time, and then these extracted 2D keypoints are compared with each other to obtain matched 2D corresponding keypoints for each detected object. Various 2D keypoint extraction methods can be used. For higher accuracy, sub-pixel keypoint extraction methods may be used.

[0049] Furthermore, the matched 2D keypoints and rendered high-resolution images of each detected object are input into the 3D CAD model localization process 256. In this 3D CAD model localization process 256, the 3D position of each matched 2D keypoint on each detected object is calculated from the rendered image and the 2D coordinates of the matched 2D keypoints. To perform this calculation, the rendered image may include a rendered depth image, a rendered point cloud, or a rendered Normalized Object Coordinate Space (NOCS) map. The 3D position is relative to the origin of the CAD model of the detected object.

[0050] Furthermore, the 3D positions and 2D image coordinates of all matched 2D corresponding keypoints for each detected object are input into the perspective n-point (PnP) solving process 258 to calculate the refined object pose. PnP solving is commonly used in computer vision to estimate the object pose from an object image given a set of 3D points and their corresponding 2D projections in an image. Higher accuracy in object pose can be achieved by using high-resolution camera images, high-resolution rendered images, and possible sub-pixel feature extraction methods.

[0051] In an exemplary embodiment of this disclosure, the high-resolution pose refinement process 250 (which consists of an image cropping process 240, a rendering process 252, a feature extraction and matching process 254, a 3D localization process on the CAD model 256, and a PnP solving process 258) can be iterated more than once until a stopping criterion is reached. This stopping criterion may be the number of iterations, the difference in object pose from two consecutive iterations being less than a preset threshold, or other conditions.

[0052] Thus, high-accuracy object pose estimation can be achieved from images using both AI-based vision and traditional vision. This object pose estimation is more robust to the lighting conditions of the surrounding environment in which the image is captured. This accurate and robust object pose estimation is crucial for bin picking and / or pick-and-place applications, where bins and / or other objects are randomly placed and / or stacked.

[0053] Figure 2 Only an image from a single viewpoint of the scene is used. While combining AI-based vision with traditional vision improves object pose accuracy, relying on a single-view color image has accuracy limitations in the camera's viewing direction. For example, if the Z-axis is chosen as the camera's viewing direction, then... Figure 2 The pose estimation process shown outputs an object pose with high accuracy on the X and Y axes but poor accuracy on the Z axis. One way to improve accuracy on the Z axis is to obtain additional, potentially highly accurate, depth images. A better object pose can be obtained using an algorithm commonly known as Iterative Closest Point (ICP), which registers the object's CAD model to a depth image or point cloud (given an initial object pose). Another way to improve accuracy on the Z axis is to process scene images from multiple viewpoints and then combine the results from each viewpoint using triangulation principles.

[0054] Figure 3a This is a schematic diagram of a 6D object pose estimation process or system for multiple views according to an exemplary embodiment of the present disclosure.

[0055] In such Figure 3a In the exemplary embodiment of this disclosure shown, two cameras 210 are arranged at two different viewpoints, each viewpoint associated with a single-view object pose estimation system (310 and 320). 310 and 320 can operate sequentially or in largely parallel to reduce process time. Note that... Figure 3a Only two cameras and two single-view pose estimation processes are shown in the diagram. It should be understood that this is for illustrative purposes only. The vision system can have more than two cameras, thus implementing more than two single-view pose estimation processes. For each pose estimation process (310 or 320) at each viewpoint, the camera captures one or more images. These one or more images are then fed into the pose detection process 230, which is similar to... Figure 2 The process in the pose detection process 230 outputs a list of pose detections, each pose detection containing at least the category label, region mask, and initial pose of possible objects in the scene.

[0056] After the pose detection process 230 for each camera at each viewpoint is completed, the object mapping process 260 is performed. Generally, the object mapping process 260 matches all detected objects across all viewpoints based on the information obtained from the pose detection process 230 of each single-view pose estimation 310 and 320.

[0057] The object correspondence process 260 uses several matching criteria to determine whether the set of object candidates detected across all viewpoints corresponds to the same object instance. These criteria include, but are not limited to: a) identical category labels; b) the estimated object position's Euclidean distance in the common world coordinate system is less than a threshold; c) the difference in estimated object orientation is less than a threshold; and d) triangulation pose consistency is less than a threshold, wherein triangulation pose inconsistency is calculated by performing the multi-view pose triangulation process 270, which is further described below. The matching criteria can be applied sequentially, such that the criteria applied first will filter out unlikely correspondence candidates.

[0058] For each object matched according to the object correspondence process 260, a multi-view pose triangulation process 270 is performed to refine the object pose. Since the object pose detection process 230 performs object pose estimation only once per view, a set of object pose estimates is obtained from multiple views. Under calibrated multi-view conditions (i.e., the relative position and orientation of each camera are known), triangulation can further refine the pose of each matched object. For each matched object, the multi-view pose triangulation process 270 takes as input a set of initial estimated poses from each viewpoint and a set of camera intrinsic and extrinsic parameter data from each viewpoint, and uses triangulation principles to calculate the triangulated pose. This process also outputs an error index called triangulation pose inconsistency, which indicates the deviation of the initial estimated poses from all viewpoints from the triangulation principles. Triangulation pose inconsistency is a good measure for determining whether pose estimates from different viewpoints belong to the same object. It can be used as one of the matching criteria in the object correspondence process 260. The pose triangulation process 270 can be implemented in different ways. One approach is to formulate it as an optimization problem:

[0059]

[0060] Where T is the triangulation pose of the matched object in the public world coordinate system. It represents the 3D positions of a set of model points in the object model coordinate system. Is the object in the first i The view relative to the first i pose of each camera coordinate system It is the first i The external data of the first camera is defined as the first... i The pose of each camera coordinate system relative to the common world coordinate system, where π is the camera projection matrix, is defined as follows: , It is the first i The intrinsic parameters of a camera are defined as follows: 'e' is the triangulation pose inconsistency error index. (Model points) These can be any fixed points whose position is known relative to the origin of the object's CAD model. For example, they can be the eight vertices of a cube with arbitrary side length, or some vertices on the surface of the object's CAD model.

[0061] exist Figure 3a In this process, camera images are fed into pose detection process 230. Therefore, only a single-resolution image is needed. When using a deep neural network in pose detection process 230, image resolution is typically low due to hardware and computation time limitations. Figure 3a It can provide good accuracy in object pose estimation, but the accuracy may be limited due to the use of low-resolution images.

[0062] Figure 3b This is a schematic diagram of a 6D object pose estimation process or system for multiple views according to another exemplary embodiment of this disclosure. It uses both a single-view high-resolution pose refinement process 250 and a multi-view pose triangulation process 270 to further improve the object pose accuracy.

[0063] and Figure 3a Similarly, two cameras 210 are positioned at two different viewpoints, each associated with a single-view object pose estimation system (310 and 320). 310 and 320 can run sequentially or in largely parallel to reduce processing time. Note that in Figure 3b Only two cameras and two single-view pose estimation processes are shown in the diagram. It should be understood that this is for illustrative purposes only. The vision system can have more than two cameras, and thus more than two single-view pose estimation processes. For the pose estimation process (310 or 320) at each viewpoint, the camera captures one or more high-resolution images. Through a downscaling process 220, one or more low-resolution images are obtained at each viewpoint. These one or more low-resolution images are then fed into the pose detection process 230, which is similar to... Figure 2 The process in the pose detection process 230 outputs a pose detection list, where each pose detection includes at least the category label, region mask, and initial pose of possible objects in the scene.

[0064] After completing pose detection process 230 for each camera at each viewpoint, object mapping process 260 is performed, followed by multi-view pose triangulation process 270 to calculate the refined object pose of each matched object in the scene. Figure 3b Processes 230, 260, and 270 in the middle Figure 3a The process is similar, except that the former processes low-resolution images.

[0065] For each matched object, the refined object pose from the multi-view pose triangulation process 270 is then transformed into the camera coordinate system of each viewpoint. At each viewpoint, the high-resolution image obtained from that viewpoint, along with the new object pose, object category label, and object region mask, is fed into the high-resolution pose refinement process 250 to calculate another refined object pose with better accuracy. The high-resolution pose refinement process 250 is executed independently at each viewpoint for each matched object. Figure 2 Similarly, the high-resolution pose refinement process 250 consists of an image cropping process 240, a rendering process 252, a feature extraction and matching process 254, a 3D localization process on the CAD model 256, and a PnP solving process 258. This process can be iterated more than once until a stopping criterion is met. The stopping criterion may be the number of iterations, the difference between the object poses from two consecutive iterations being less than a preset threshold, or other conditions.

[0066] For each matched object, the set of new object poses obtained at each viewpoint from the high-resolution pose refinement process 250 is input into another multi-view pose triangulation process 270 to calculate another triangulation pose. The triangulated object pose can then be output as the final pose for each matched object.

[0067] The single-view high-resolution pose refinement process 250 and the multi-view pose triangulation process 270 can be repeated once or multiple times until another stopping criterion is reached. The stopping criterion may be the number of iterations, the difference between the object poses from two consecutive iterations being less than a preset threshold, or other conditions.

[0068] Generally speaking, Figure 3b The initial pose obtained from the pose detection process 230 is improved using a series of potentially repeated pose refinement processes. The pose refinement process can be a single-view high-resolution refinement process 250 or a multi-view pose triangulation process 270.

[0069] Figure 4 This is a schematic flowchart of a 6D object pose estimation process for a single view according to an exemplary embodiment of the present disclosure.

[0070] As shown in the figure, the pose estimation process or method 400 for a 6D object in a single view includes the following steps:

[0071] At 402, the device acquires both low-resolution and high-resolution images from the viewpoint of the scene.

[0072] The device typically includes one or more processors that perform 6D object pose estimation processes or methods.

[0073] Low-resolution and high-resolution images can be acquired directly from the camera. Alternatively, a high-resolution image can be acquired from the camera and then downscaled to create a low-resolution image.

[0074] At position 404, the device performs a pose detection process on the low-resolution viewpoint image to detect the category labels, region masks, and initial poses of objects in the scene.

[0075] Pose detection is performed based on a neural network pose detection model. Before performing pose detection, the viewpoint image is downscaled. For example, the viewpoint image can be downscaled from a 5MP color image to a 0.3MP color image. Additionally and / or alternatively, other resolutions of the viewpoint image can also be used.

[0076] At point 406, the equipment is checked to determine if it meets the standards.

[0077] This standard includes: whether the processing of high-resolution viewpoint images has been repeated more than a first predefined threshold, and / or whether the difference in object pose between two consecutive iterations is less than a second predefined threshold. Additionally and / or alternatively, other standards may also apply. (Processing of high-resolution viewpoint images) Figure 4 (416) includes steps 408, 410 and 412 as described below.

[0078] If the criterion is met, the device outputs the category label and current pose of the detected object, such as... Figure 4 As shown at position 414. Therefore, the pose estimation process for a 6D object in a single view is complete.

[0079] If the criterion is not met, the high-resolution viewpoint image processing procedure 416 begins. This processing 416 includes the following steps:

[0080] At position 408, the device generates both the cropped and rendered images.

[0081] The cropped image is generated by cropping a high-resolution viewpoint image based on a region mask for each detected object. The rendered image is generated using the object's CAD model, the current pose of each detected object, and intrinsic and extrinsic parameter data of the camera associated with the high-resolution image. The rendered image may include a rendered high-resolution color image, a rendered high-resolution depth image, and / or a high-resolution NOCS image.

[0082] At 410, the device calculates the refined object pose of each detected object in the high-resolution viewpoint image by comparing the cropped image and the rendered image.

[0083] The process of calculating the refined object pose of each detected object in a high-resolution viewpoint image may include: extracting 2D keypoints on the cropped and rendered images; matching these 2D keypoints; locating the 3D position of each matched 2D keypoint from the rendered image; and solving the view PnP equation.

[0084] At position 412, the device uses refined object poses to update the current pose of each detected object.

[0085] The device then checks again at point 406 to determine if the standard is met.

[0086] If the criterion is met, the device outputs the category label and current pose of the detected object, such as... Figure 4 Step 414 is shown. Therefore, the 6D object pose estimation process for a single view ends. Then, the device controls the robot (e.g., as shown in step 414). Figure 1 The robot 102 shown performs bin picking and / or bin placement based on output data (e.g., refined object pose). For example, the device provides one or more instructions to control the robot (e.g., robot 102) by selecting an item (e.g., item 106) in a factory process and placing it into a bin (e.g., bin 108) and / or onto a pallet.

[0087] If the condition is not met, the processing of the high-resolution viewpoint image 416 is repeated.

[0088] Figure 5a This is a schematic flowchart of a 6D object pose estimation process for multiple views according to an exemplary embodiment of the present disclosure.

[0089] As shown in the figure, the pose estimation process or method 500a for a 6D object with multiple views includes the following steps:

[0090] At position 502, the device acquires one or more images from multiple viewpoints of the scene. In an exemplary embodiment, at least one image of the scene is acquired at each viewpoint.

[0091] One or more images from multiple viewpoints of the scene may be provided as input from an external device. Additionally and / or alternatively, one or more images from multiple viewpoints of the scene may be captured by an integrated camera.

[0092] At position 506, the device performs pose detection on the scene image at each viewpoint to detect the object's category label, region mask, and initial pose.

[0093] Steps 503, 504, and 507 ensure that one or more images at each viewpoint are processed by pose detection at 506.

[0094] At 508, once all images at each viewpoint have been processed by pose detection at 506, the device matches the detected objects across multiple viewpoints.

[0095] At 510, the device performs a multi-view pose triangulation process across multiple viewpoints on a set of poses for each matched object to compute a refined object pose using intrinsic and extrinsic camera data associated with one or more images from each viewpoint.

[0096] At position 512, the device outputs the category label and refined object pose for each matched object. The device then controls the robot (e.g., as...). Figure 1 The robot 102 shown performs bin picking and / or bin placement based on output data (e.g., refined object pose). For example, the device provides one or more instructions to control the robot (e.g., robot 102) by selecting an item (e.g., item 106) in a factory process and placing it into a bin (e.g., bin 108) and / or onto a pallet.

[0097] Figure 5b and Figure 5c A schematic flowchart of a 6D object pose estimation process for multiple views is shown according to another exemplary embodiment of the present disclosure.

[0098] The pose estimation process or method 500b for 6D objects with multiple views includes the following steps:

[0099] like Figure 5b As shown, steps 502, 503, 504, 506, 507, 508, and 510 are related to... Figure 5a The steps shown are the same.

[0100] Figure 5c Further steps in the 6D object pose estimation process or method for multiple views shown include:

[0101] At point 522, the device acquires one or more high-resolution images from multiple viewpoints of the scene. In an exemplary embodiment, at least one high-resolution image of the scene is acquired at each viewpoint.

[0102] At point 524, the device determines whether the first criterion is met.

[0103] In an exemplary embodiment, the first criterion includes that the difference between the object poses from two consecutive iterations is less than a preset threshold. Additionally and / or alternatively, other criteria may also apply.

[0104] If the first criterion is met, the device outputs the category label and current pose of the detected object at position 532.

[0105] If the first criterion is not met, the device performs high-resolution pose refinement on the high-resolution image at each viewpoint at point 528 to obtain an updated object pose. The device repeats the high-resolution pose refinement at point 528 for each viewpoint one or more iterations until the second criterion is met.

[0106] In an exemplary embodiment, the second criterion includes an iteration count greater than a preset threshold. Other criteria may also apply.

[0107] Steps 525, 526, and 529 ensure that one or more images at each viewpoint are processed by high-resolution pose refinement at 528. Once the device has processed the high-resolution pose refinement for all images at each viewpoint at step 528, the 6D object pose estimation process or method 500b for multiple views will continue as follows:

[0108] At 530, the device performs a multi-view pose triangulation process on a set of current poses of each matched object at multiple viewpoints to compute updated object poses using intrinsic and extrinsic parameter data from cameras associated with one or more high-resolution images from each viewpoint.

[0109] The device repeats the multi-view pose triangulation process at step 530 until the first criterion is met.

[0110] At point 532, once the first criterion is met, the device outputs the category label and current pose of the detected object. The device then controls the robot (e.g., as...). Figure 1 The robot 102 shown performs bin picking and / or bin placement based on output data (e.g., refined object pose). For example, the device provides one or more instructions to control the robot (e.g., robot 102) by selecting an item (e.g., item 106) in a factory process and placing it into a bin (e.g., bin 108) and / or onto a pallet.

[0111] Figure 6 This is a schematic diagram of an apparatus for performing a 6D object pose estimation process according to an exemplary embodiment of the present disclosure.

[0112] like Figure 6 As shown, device 600, which uses AI-based vision technology and traditional vision technology to perform a 6D object pose estimation process, includes: bus 610, processor 602, communication interface 604, and memory 606. Alternatively or additionally, device 600 may also include camera 608. For example, processor 602, communication interface 604, memory 606, and camera 608 can communicate with each other via bus 610.

[0113] Processor 602 may include one or more general-purpose processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or a tensor processing unit (TPU), or a combination of a CPU, GPU, or TPU with hardware chips. The hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The programmable logic device (PLD) may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a general-purpose array logic (GAL), or any combination thereof.

[0114] Memory 606 may include volatile memory, such as random access memory (RAM). Memory 606 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD). Memory 606 may also include combinations of the types discussed above.

[0115] The memory 606 may store computer-readable program code. The processor 602 may read the computer-readable program code stored in the memory 606 to execute, for example... Figure 2 and Figures 3a-3b The 6D object pose estimation shown and described above is for single-view and / or multi-view purposes. Processor 602 can also read computer-readable program code stored on memory 606 to implement the above-described... Figure 4 Method 400 and / or Figure 5a 500a and / or Figure 5b and Figure 5c The 500b is used to perform 6D object pose estimation for single-view and / or multi-view purposes. Additionally, the processor 602 can also read computer-readable program code stored on memory 606 to implement one or more other functions and / or combinations of these functions.

[0116] The processor 602 can also communicate with another computing device via the communication interface 604. For example, the processor 602 can communicate with another computing device to obtain preset values ​​and / or thresholds to determine whether the estimated object pose for a single view and / or multiple views is sufficiently satisfactory. For example, the processor 602 can communicate with an external camera to obtain images captured from a single viewpoint and / or multiple viewpoints of the scene.

[0117] The processor 602 can also trigger the camera 608 to capture images from the viewpoint of the scene for 6D object pose estimation in a single view. The processor 602 can also trigger the camera 608 to capture at least one image from each of n viewpoints of the scene, where n is an integer greater than 1, for 6D object pose estimation in multiple views.

[0118] Those skilled in the art should understand that, Figure 6 The device 600 shown can communicate with one or more other computing devices via communication interface 604 or wireless connection to enable more functions and / or combinations of functions. The device 600 may also include one or more other functional components to perform and / or trigger more functions or combinations of functions.

[0119] All references cited in this article (including publications, patent applications, and patents) are incorporated herein by reference, with the same effect as if each reference were individually and explicitly cited and listed in full in this article.

[0120] Unless otherwise stated herein or the context clearly specifies otherwise, the terms “a,” “an,” “the,” “at least one,” and similar designations used in the context of describing the invention (especially in the context of the following claims) should be interpreted to cover both singular and plural forms. Unless otherwise stated herein or the context clearly specifies otherwise, the term “at least one” followed by a list of one or more items (e.g., “at least one of A and B”) should be interpreted to mean one item selected from the listed items (A or B) or any combination of two or more of the listed items (A and B). Unless otherwise stated, the terms “comprising,” “having,” and “including” should be interpreted as open-ended terms (i.e., meaning “including but not limited to”). Unless otherwise stated herein, the numerical ranges listed herein are intended only as a method of abbreviation for individually referring to each individual value falling within the range, and each individual value is incorporated into the specification as if it were individually listed herein. Unless otherwise stated herein or the context clearly specifies otherwise, all methods described herein can be performed in any suitable order. Unless otherwise required, any examples or exemplary language (e.g., "such as") provided herein are intended only to better illustrate this disclosure and do not constitute a limitation on the scope of this disclosure. No language in the specification should be construed as indicating that any unrequired element is necessary for practicing this disclosure.

[0121] This document describes exemplary embodiments of the present disclosure, including the best mode of implementation known to the inventors. Various variations of these exemplary embodiments will become apparent to those skilled in the art upon reading the foregoing description. The inventors anticipate that those skilled in the art will employ such variations as appropriate, and the inventors intend that the present disclosure be practiced in a manner different from that specifically described herein. Therefore, this disclosure includes all modifications and equivalents of the subject matter set forth in the appended claims permitted by applicable law. Furthermore, unless otherwise stated herein or expressly provided by the context, this disclosure covers any combination of the elements described above in all possible variations.

Claims

1. A method for estimating six degrees of freedom (6D) poses of one or more objects in a scene, comprising: (1) obtaining a first image and a second image from a viewpoint of the scene using a camera, wherein the second image has a higher resolution than the first image; (2) performing pose detection on the first image to obtain class labels, region masks, and initial poses of objects in the scene; (3) performing pose refinement on each of the detected objects on the second image, wherein performing the pose refinement comprises: a) generating a cropped image from the second image based on a region mask of each detected object, and generating a rendered image of each detected object based on a computer aided design (CAD) model, a current pose of each detected object, and intrinsic and extrinsic data of the camera associated with the second image; b) computing a refined object pose for each of the detected objects by comparing the cropped image and the rendered image; and c) updating the current pose of each detected object with the refined object pose; and (4) repeating step (3) until a criterion is satisfied.

2. The method of claim 1, wherein obtaining the first image and the second image comprises: capturing the second image from the viewpoint of the scene by the camera, and downscaling the second image to the first image.

3. The method of claim 1, wherein the region mask of each detected object is obtained by performing the pose detection via: detecting the class labels and bounding boxes of the objects in the first image based on at least one object detection procedure of the pose detection; and using the bounding box of each detected object as the region mask.

4. The method of claim 1, wherein the region mask of each detected object is obtained by performing the pose detection via: detecting the class labels and instance masks of the objects in the first image based on at least one object detection procedure of the pose detection; and using the instance mask of each detected object as the region mask.

5. The method of claim 1, wherein the region mask of each detected object is obtained by performing the pose detection via rendering each object with a corresponding CAD model and a current pose of each object.

6. The method of claim 1, wherein performing the pose detection is based on a deep neural network, or multiple deep neural networks running in sequence or in parallel.

7. The method of claim 1, wherein the rendered image of each detected object comprises at least one of: a color image, a grayscale image, a depth image, a point cloud, or a normalized object coordinate space (NOCS) map.

8. The method of claim 1, wherein computing the refined object pose for each of the detected objects by comparing the cropped image and the rendered image comprises: extracting two-dimensional (2D) keypoints on the cropped image and the rendered image; matching the 2D keypoints; localizing a three-dimensional (3D) position for each matched 2D keypoint from the rendered image; and applying a perspective-n-point solution for the matched keypoint to compute a refined object pose.

9. The method of claim 1, wherein the criteria include: Step (3) is repeated a number of times greater than a first predefined threshold, or the difference in object poses at two consecutive iterations is less than a second predefined threshold.

10. The method of claim 1, further comprising: (5) obtaining a depth image or a point cloud image of the scene; (6) aligning, for each detected object, the rendered image with the depth image or with the point cloud image to obtain a refined object pose; and (7) outputting the refined object pose for each detected object.

11. The method of claim 10, wherein outputting the refined object pose for each detected object comprises: controlling a robot to perform at least one of bin picking and / or bin placement based on the refined object pose.

12. A method for estimating six degree of freedom (6D) poses of one or more objects in a scene, comprising: obtaining one or more first images from a plurality of viewpoints of the scene using one or more cameras; (1) performing pose detection on the one or more first images from each viewpoint to obtain a class label, a region mask, and an initial pose of an object; (2) matching the detected objects across the plurality of viewpoints; (3) performing a multi-view pose triangulation process on the pose of each matched object across the plurality of viewpoints to compute a refined object pose using intrinsic and extrinsic data of the one or more cameras associated with the one or more first images from each viewpoint; and outputting the refined object pose of each matched object.

13. The method of claim 12, wherein outputting the refined object pose of each matched object comprises: controlling a robot to perform at least one of bin picking and / or bin placement based on the refined object pose.

14. The method of claim 12, wherein matching the detected objects across the plurality of viewpoints comprises one or more of: (1) deciding whether the class label of the detected object from each view is the same, (2) computing the Euclidean translational distance of the detected object from each viewpoint in a common world coordinate system; (3) computing the rotational angular distance of the detected object from each viewpoint in a common world coordinate system; (4) computing a triangulation pose inconsistency error indicator by performing a multi-view pose triangulation process for each matched object.

15. The method of claim 14, wherein performing the multi-view pose triangulation process for each matched object comprises: solving an optimization problem for the refined object pose of each matched object.

16. The method of claim 12, further comprising: (4) obtaining one or more second images from the plurality of viewpoints of the scene using the one or more cameras, wherein the one or more second images have a higher resolution than the one or more first images; (5) The pose of each matched object is refined using the one or more second images from each viewpoint in the following manner: a) Generate a cropped image from the one or more second images based on the region mask of each matched object, and generate a rendered image of each matched object based on the computer-aided design (CAD) model, the current pose of each matched object, and the intrinsic and extrinsic parameter data of the one or more cameras associated with the one or more second images; b) Calculate the refined object pose for each matched object by comparing the cropped image and the rendered image; as well as c) Update the current pose using the refined object pose; (6) Repeat step (5) multiple times until the first criterion is met; (7) For each matched object, perform a multi-view pose triangulation process across the multiple viewpoints on the updated current pose to compute a refined object pose using the intrinsic and extrinsic parameter data of the one or more cameras associated with the one or more second images from each viewpoint; as well as (8) Repeat steps (5) to (7) until the second criterion is met.

17. The method of claim 16, wherein step (4) of obtaining the one or more second images from the plurality of viewpoints of the scene comprises: High-resolution images are acquired using the camera; Downscale the high-resolution image to a low-resolution image; Use the low-resolution image in steps prior to step (4); as well as The high-resolution image is used in steps following step (4).

18. An apparatus for estimating the six-degrees-of-freedom (6D) pose of one or more objects in a scene, wherein the apparatus includes one or more processors, and wherein the apparatus is configured to: (1) Obtain a first image and a second image from the viewpoint of the scene, wherein the second image has a higher resolution than the first image; (2) Perform pose detection on the first image to obtain the category labels, region masks, and initial poses of objects in the scene; (3) Perform pose refinement on the second image for each of the detected objects, wherein performing the pose refinement includes: a) Generate a cropped image from the second image based on the region mask of each detected object, and generate a rendered image of each detected object based on the computer-aided design (CAD) model, the current pose of each detected object, and the intrinsic and extrinsic parameter data of the camera associated with the second image; b) By comparing the cropped image and the rendered image, calculate the refined object pose for each of the detected objects; and c) Update the current pose of each detected object using the refined object pose; and (4) Repeat step (3) until the standard is met.

19. The device of claim 18, wherein the region mask for each detected object is obtained by performing the pose detection in such a way as: Based on at least one object detection process of the pose detection, the category label and bounding box of the object in the first image are detected; and The bounding box of each detected object is used as the region mask.

20. The device of claim 18, wherein the region mask for each detected object is obtained by performing the pose detection in such a way as: Based on at least one object detection process of the pose detection, the category label and instance mask of the object in the first image are detected; and The instance mask of each detected object is used as the region mask.

Images