Random bin picking and precise placement using a multi-view system
By combining a multi-view system and an AI pose estimation model, the problems of pose estimation accuracy and cost in random bin picking systems are solved, enabling low-cost, high-precision picking and placement of mechanical parts and reducing cycle time.
Patent Information
- Application Number
- CN202480091090.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing random bin picking systems struggle to balance pose estimation accuracy and cost. High-precision systems are expensive, while low-cost systems have limited picking performance, leading to unsuccessful placement of mechanical parts and increased cycle time.
A multi-view system combined with an AI pose estimation model is used to capture images of mechanical parts through multiple 2D cameras, predict the coarse pose using a deep learning-based neural network, refine it through multi-view triangulation, and determine the precise position for picking and placing by combining the tool center point of the robotic arm.
This approach improves the accuracy of picking up and placing mechanical parts while maintaining low system costs, reduces cycle time, and eliminates the need for intermediate fixtures.
Smart Images

Figure CN123125084A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a random bin picking system. More specifically, this disclosure relates to precise picking and placement using a multi-view system in robotic arm-based random bin picking applications. Background Technology
[0002] For industrial machine tool loading and unloading applications, robotic arms are required to accurately pick up industrial parts randomly placed in standard containers for subsequent processing, including purging, cleaning, deburring, measurement, inspection, and / or precise placement onto conveyor belts. Accurate pose estimation of the mechanical parts is necessary to place them accurately after the initial pick-up process. Most current random bin picking applications are based on 3D vision systems. These systems can achieve high-precision pose estimation for automating the picking of industrial parts, but integrating them into random bin picking systems is costly. Alternatively, low-cost single RGB or grayscale cameras can be used for such applications, but these systems have limited picking performance due to poor pose estimation accuracy. In some random bin picking applications, precise placement is required. Due to the gripping contact and interaction between the gripper and the mechanical part, the pose of the picked part cannot be accurately determined for precise placement. Current industrial implementations involve using an intermediate fixture where the robotic arm places the picked part and then picks it up again after the fixture corrects the pose. This repositioning step will affect the cycle time of such pick-and-place systems. Summary of the Invention
[0003] A first aspect of this disclosure provides a method for random bin picking using a multi-view system. The method includes: capturing a first set of images of a collection of mechanical parts randomly distributed in a bin; determining a subset of target parts from the set of mechanical parts based on the captured images; predicting a first coarse position for each target part in the subset of target parts using an artificial intelligence (AI)-based pose estimation model; refining the predicted first coarse position of each target part in the set of target parts, wherein refining the predicted coarse position of each target part includes: determining the position of each target part in a combined coordinate system generated by incorporating information from the first set of images; providing instructions to a robotic arm to pick up a target part from the set of target parts and to move the robotic arm carrying the target part to a preset position; capturing a second set of images of the target parts while the target part is held in the preset position by the robotic arm; predicting a second coarse position of the target part using the pose estimation model; and refining the second predicted coarse position of the target part, wherein refining the second predicted coarse position of the target part for precise placement includes: determining the position of the target part in a second combined coordinate system generated by incorporating information from the second set of images.
[0004] According to one embodiment of the first aspect, the method further includes placing the target part on a conveyor belt for further processing based on the refined second predicted coarse position.
[0005] According to one embodiment of the first aspect, the method further includes placing the target part in a preform mold for further processing based on the refined second predicted coarse position.
[0006] According to one embodiment of the first aspect, a first image set is captured simultaneously using a first camera and a second camera, wherein the first camera and the second camera are color cameras.
[0007] According to one embodiment of the first aspect, the first camera and the second camera are arranged at different heights relative to the hopper.
[0008] According to one embodiment of the first aspect, the known position of the tool center tip of the robotic arm is used to perform prediction of the second coarse position and refinement of the second coarse position.
[0009] A second aspect of this disclosure provides a system for random bin picking using a multi-view system. The system includes: a controller configured to: capture a first set of images of a collection of mechanical parts randomly distributed in a bin; determine a subset of target parts from the collection of mechanical parts based on the captured images; predict a first coarse position for each target part in the subset of target parts using an AI-based pose estimation model; refine the predicted first coarse position for each target part in the set of target parts, wherein refining the predicted coarse position for each target part includes: determining the position of each target part in a combined coordinate system generated by incorporating information from the first set of images; instructing a robotic arm to pick up a target part from the set of target parts and move the robotic arm carrying the target part to a preset position; capturing a second set of images of the target parts while the target part is held in the preset position by the robotic arm; predicting a second coarse position for the target parts using the pose estimation model; and refining the second predicted coarse position for the target parts, wherein refining the second predicted coarse position for precise placement includes: determining the position of the target part in a second combined coordinate system generated by incorporating information from the second set of images.
[0010] According to one embodiment of the second aspect, the controller is also configured to place the target part on the conveyor belt for further processing based on the refined second predicted coarse position.
[0011] According to one embodiment of the second aspect, the controller is also configured to place the target part in a preform mold for further processing based on the refined second predicted coarse position.
[0012] According to one embodiment of the second aspect, a first image set is captured simultaneously using a first camera and a second camera, wherein the first camera and the second camera are color cameras.
[0013] According to one embodiment of the second aspect, the first camera and the second camera are arranged at different heights relative to the hopper.
[0014] According to one embodiment of the second aspect, the known position of the tool center tip of the robotic arm is used to perform prediction of the second coarse position and refinement of the second coarse position. Attached Figure Description
[0015] The subject matter of this disclosure will be described in more detail below with reference to exemplary accompanying drawings. All features described and / or illustrated herein may be used alone or in different combinations. Features and advantages of various embodiments will become apparent from the following detailed description with reference to the accompanying drawings, which illustrate the following:
[0016] Figure 1 A simplified system diagram of a random bin picking system according to one or more examples of this disclosure is shown;
[0017] Figure 2 An exemplary system diagram related to a refinement technique based on multi-view triangulation is shown, according to one or more examples of this disclosure;
[0018] Figure 3 Exemplary illustrations are shown in relation to a pose estimation process for precise placement after picking up a workpiece, according to one or more examples of this disclosure;
[0019] Figure 4 Based on one or more examples of this disclosure Figure 1 A simplified block diagram of one or more devices or systems within an exemplary environment; and
[0020] Figure 5 The process performed by a controller as part of a random bin picking system is shown according to one or more examples of this disclosure. Detailed Implementation
[0021] Examples of this application will now be described more fully below with reference to the accompanying drawings, which illustrate some, but not all, examples of this application. In practice, this application may be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these examples are provided to enable this application to meet applicable legal requirements. Where possible, unless expressly stated otherwise, any term expressed herein in the singular is intended to include the plural form as well, and vice versa. Furthermore, as used herein, the terms “a” and / or “an” should mean “one or more,” although the phrase “one or more” is also used herein. Additionally, when something is said herein to be “based on” another thing, it may also be based on one or more other things. In other words, unless expressly indicated otherwise, “based on” as used herein means “at least partially based on” or “at least partially based on.”
[0022] This disclosure focuses on solving the problem of random bin picking and precise placement using a multi-view vision system. This disclosure provides an AI-based pose estimation vision system comprising two or more two-dimensional (2D) cameras and utilizing a robotic arm to perform precise pose estimation of mechanical parts for picking and placement applications. Images from each 2D camera are fed to the pose estimation vision system to perform different pose estimations on the target mechanical part using different images for picking. Subsequently, triangulation-based pose and / or pose refinement is performed for precise picking of the mechanical part.
[0023] In some cases, the pose of the picked-up mechanical part may be slightly offset due to contact between the robotic arm's end effector and the part, potentially leading to unsuccessful placement during subsequent machine tool loading and unloading. Traditionally, this pose deviation is corrected by placing the part in a fixture at an intermediate position and picking it up again. However, this process can negatively impact cycle time and requires complex setup. To achieve satisfactory performance while maintaining low system cost, this system employs a cost-effective multi-view system based on a 2D camera for random bin picking.
[0024] To avoid unsuccessful placement, this disclosure describes adjusting the pose estimate of a mechanical part after it has been picked up to compensate for any changes in its pose after being picked up and held by a robotic arm, thereby achieving successful placement of the mechanical part. In some embodiments, the pose estimation adjustment is performed using a set of secondary 2D images. The set of secondary 2D images is captured after the part is picked up and attached to the pick-up end effector. The set of secondary 2D images is used to predict a second coarse pose of the mechanical part using a deep learning-based algorithm. The second coarse pose is refined using a triangulation-based pose refiner module. To predict the second coarse pose and perform a second fine refinement on it, the tool center point (TCP) position of the robotic arm needs to be determined. The TCP position is used as a reference point, which helps in determining the second coarse pose, which is then refined, and the mechanical part is placed.
[0025] Figure 1A simplified schematic diagram of a random bin picking system according to one or more examples of this disclosure is shown. System 100 includes a controller 104, a vision system 108, a robotic arm 102, and a memory 106. The controller 104 instructs the robotic arm 102 to precisely pick up a mechanical part randomly placed in a bin and then place the mechanical part at a location for further processing. The controller 104 determines a first coarse position of the mechanical part randomly placed in the bin and performs a first pose refinement on the first coarse position using a pose refinement module. In some embodiments, the first coarse position is acquired by an artificial intelligence (AI)-based deep learning neural network. The AI-based deep learning neural network receives multiple RGB or monochrome 2D images of the mechanical parts in the bin. Multiple mechanical parts are identified from the multiple 2D images. The AI-based deep learning neural network predicts the 6-DOF (6D) pose of each detected mechanical part in the bin using the AI-based deep learning neural network. The predicted 6D pose can be represented in terms of 3D translation and 3D orientation. The AI-based deep learning model is applied to each view of each image in the multiple images. In some examples, the predicted 6D pose of each part is refined. This refinement is based on multi-view triangulation. To perform multi-view triangulation, a set of 3D points is predefined or sampled from the CAD mesh model of each detected mechanical part in the bin. In some cases, the CAD mesh model can be stored in memory 106. The 3D points of the CAD mesh model are converted to 2D points in image space. Using the coarse pose of the mechanical part in each view, the 2D points (pixel coordinates) of the selected 3D points from the CAD model are calculated in image space via back projection. Multi-view triangulation is applied to each point in the sampled set. Using the multi-view triangulation of the 2D points, the 3D coordinates of the selected points in the camera view are calculated. Using rigid body constraints, the refined pose of the target part can be easily calculated using the 3D coordinates in the camera view.
[0026] Coarse pose determination and pose refinement can be used to efficiently pick up industrial parts with a robotic arm for further assembly processing.
[0027] The pose estimation vision process, as shown in system 100, is implemented in two stages. In the first stage, vision system 108 is used to photograph a bin containing randomly distributed parts. In some cases, vision system 108 may include at least two 2D cameras for capturing images of the bin filled with randomly distributed parts. In other cases, at least two cameras of vision system 108 may be positioned at different angles and different heights to capture images of the industrial parts in the bin from different perspectives. For example, a first camera 202 may be positioned at a first height above the bin, and a second camera 204 may be positioned at a second height above the bin, where the first and second heights are different.
[0028] In some embodiments, the first image set is captured by at least two cameras of the vision system 108. From the first image set of the parts bin, the target mechanical part to be picked up is identified. In some cases, an off-the-shelf Mask R-CNN model can be used to identify the target part. The Mask R-CNN model can be retrained or fine-tuned using a generated synthetic dataset, given ground truth information such as part identifiers (IDs), 2D bounding boxes, etc. Other state-of-the-art object detection models, such as YOLO variants, can also be used.
[0029] For each target mechanical part, a first coarse pose is estimated using a neural network model 110. Then, a pose refinement module combines a first set of images from at least two cameras of the vision system to refine the first coarse pose of each target mechanical part. Using the first refined pose generated by refining the first coarse pose, the controller 104 instructs the robotic arm 102 to pick up one of the target mechanical parts from the bin. After the target mechanical part is picked up, the robotic arm 102 moves the target mechanical part to a preset position. The vision system 108 can capture a second set of images of the mechanical part now held by the robotic arm 102. Using the second set of images, a second coarse pose of the part is determined, and the second coarse pose is further refined by combining information from the second set of images. The controller 104 can adjust the second coarse pose of the mechanical part while the robotic arm 102 holds the mechanical part based on the determination of the second coarse pose and the output of the pose refinement module. Once the second coarse pose of the part is refined in the robotic arm 102, the robotic arm places the mechanical part in a pre-made mold or on a conveyor belt for processing.
[0030] As described above, in the first stage, the vision system 108 captures a first set of images related to a real-world random bin scene. In some embodiments, the bin may include a collection of industrial parts randomly distributed throughout the bin. The vision system 108 can be used to photograph the bin. Using an AI-based neural network 110, the first set of images captured by the at least two cameras of the vision system is used to estimate the coarse position of each target mechanical part in the bin. Subsequently, the first set of images is combined to perform a first pose refinement. As described above, the first coarse position is obtained by an AI-based deep learning neural network. The AI-based deep learning neural network receives 2D images of mechanical parts in the bin and predicts the 6-DOF (6D) pose of each detected mechanical part in the bin. The estimated 6D pose can be represented in terms of 3D translation and 3D orientation. A set of 3D points is predefined or sampled from the mesh model of each detected mechanical part in the bin. Using the coarse pose of the mechanical parts in each view, the 2D (2D) pixel coordinates of the selected 3D points in the CAD model in the image space are calculated by back projection. Multi-view triangulation is applied to each point in the selected set of points. Therefore, the 3D coordinates of the selected points in the camera view are calculated. Using rigid body constraints, the refined pose of the target part can be easily calculated using the 3D coordinates in the camera view. Then, the controller 104 instructs the robotic arm 102 to pick up the mechanical part based on the first coarse pose and the first refined pose.
[0031] In some embodiments, 6D pose estimation, including two steps, can be performed twice. A first 6D pose estimation step can be used to determine a coarse position, and a second 6D pose estimation step can be used to refine the determined coarse position. The first 6D pose estimation can be used to perform a pick-up operation. The second 6D pose estimation can be used to determine the pose of the picked-up part and place it onto a mold or conveyor belt.
[0032] An AI-based neural network model 110, which can be used to perform a first coarse pose estimation for each target part randomly distributed in a bin, can be stored in memory 106. The controller applies the AI-based neural network 110 to an image of a bin filled with parts captured by a vision system 108. The neural network model 110 is used to predict the first coarse pose of mechanical parts in a set of industrial parts randomly distributed in the bin. In this case, the neural network model 110 can receive an image of the bin captured by the vision system 108 and process the image to predict the first coarse pose of the parts randomly distributed in the bin.
[0033] The neural network 110 can be trained using a training dataset 112 stored in memory 106. The training dataset 112 may include computer-aided design (CAD) mesh models of mechanical parts randomly distributed in a bin. In some embodiments, a large physically based rendering (PBR) synthetic dataset and ground truth labels may be generated for training the 2D-based visual model. Physically based rendering synthetic images are photorealistic and better reflect real lighting and reflections. Physically based rendering is a shading and rendering technique that provides a more accurate representation of how light interacts with different material properties. Ground truth labels may include a part identifier (ID), a 2D bounding box of the part, the part's visibility in the view, and the part's pose, consisting of xyz-axis translation and orientation represented by a 3x3 rotation matrix.
[0034] The coarse pose estimation model implemented using Neural Network 110 can be a single-stage pose estimator or a two-stage pose estimator including an object detector and a pose predictor. In some cases, the mechanical parts are detected by an object detector, which is then fine-tuned or retrained to identify the picking target using deep learning. The object detector used here is based on Mask R-CNN. However, the object detector can be easily changed to other algorithms based on You Only Look Once (YOLO). In some cases, the part detector is trained using a 2D image dataset and ground truth labels consisting of mechanical part identifiers and 2D bounding boxes.
[0035] In some embodiments, the object detector may first detect objects from an image captured by the vision system 108. A pose predictor can then be used to determine the pose of the objects detected by the object detector. In some embodiments, the pose predictor may be a coarse pose predictor implemented using a neural network 110. In some embodiments, the object detector and the coarse pose estimator may be combined into a single-stage pose estimator, implemented by the neural network 110.
[0036] Following the initial coarse pose estimation, multi-view triangulation is applied to refine the grasp pose and / or object pose for accurate picking of the mechanical part. In some embodiments, multi-view triangulation involves combining data acquired by different cameras of the vision system 108. In some cases, combining information from a first image set may involve combining the coordinate systems of the individual images in the first image set. The position of the mechanical part in the bin can then be generated in a common coordinate system. Combining information from different images may help determine the depth of the target mechanical part in the bin.
[0037] Based on the first refined pose, controller 104 instructs robotic arm 102 to pick up a mechanical part from the hopper. Once the part is picked up, robotic arm 102 moves to a preset position for secondary 2D imaging while holding the mechanical part. Different cameras of vision system 108 can capture a second set of images of the target mechanical part while it is held in robotic arm 102. While the target mechanical part is held by robotic arm 102, a second coarse pose of the target mechanical part is determined. The second coarse pose can be determined by providing the second set of images to neural network model 110. In some embodiments, the second coarse position of the mechanical part can be represented by a six-degree-of-freedom (6D) pose consisting of three-dimensional translation and three-dimensional rotation. The information from the second set of images is then combined to perform a second pose refinement. In some embodiments, the second coarse position and the second pose refinement of the second pose of the mechanical part are determined relative to the known tool center point (TCP) of robotic arm 102. The known tool center point location helps determine the pose range used for generating synthetic data to train a close-up view AI model that estimates the coarse pose of the part when it is picked up in the robotic arm. Since the part is in a known TCP position, the part association step can be omitted in a multi-view vision system. Two different AI models can be used for coarse pose estimation when the part is in the bin and when it is picked up in the robotic arm. In some embodiments, the second estimated pose is not determined relative to the TCP.
[0038] Once the pose of the mechanical part in the robotic arm 102 is refined, the controller 102 instructs the robotic arm 102 to place the mechanical part for subsequent processing. In some embodiments, the mechanical part may be placed on a preform mold or conveyor belt for further processing.
[0039] Figure 2 An exemplary system diagram related to a refinement technique based on multi-view triangulation is shown, according to one or more examples of this disclosure. Figure 2 System 200 includes a first camera 202 and a second camera 204. The first camera 202 and the second camera 204 can be as follows: Figure 1This is part of the vision system 108. In some embodiments, the first camera 202 and the second camera 204 are both color cameras or monochrome cameras. As described above, the first camera 202 and the second camera 204 can be positioned at different heights above the bin to better capture the depth of the mechanical parts arranged in the bin. The first camera 202 and the second camera 204 are separated by a fixed distance 'b' 206. Both the first camera 202 and the second camera 204 are used to simultaneously capture images of the mechanical parts randomly distributed in the bin. Coordinate system 208 is associated with the image captured by the first camera 202, and a second coordinate system 210 is associated with the image captured by the second camera 204. The images from the first camera 202 and the second camera 204 are used to determine a coarse pose associated with the mechanical parts in the two images. In some embodiments, the images from the first camera 202 and the second camera 204 are provided to a neural network model 110 to determine a coarse position of the mechanical parts. The coordinate system 208 of the first image and the coordinate system 210 of the second image are in a 2D image space. A combined coordinate system 212, generated using the coordinate system 208 of the first image and the coordinate system 210 of the second image, consists of 3D coordinates from the main camera view. The combined coordinate system 212 is used to perform pose refinement on the coarse position of the mechanical part determined in the previous step. To perform pose refinement on the coarse position, the position of the mechanical part in the bin can be generated in the combined coordinate system. Using a transformation matrix describing the camera position in the robot base, the part pose can then be described in a common robot base (common 3D coordinate system).
[0040] Figure 3 Exemplary diagrams are shown relating to a pose estimation process for precise placement after picking up a workpiece, according to one or more examples of this disclosure. Figure 3 The first image 302 depicts a random distribution of mechanical parts in a bin. The second image 304 depicts the arrangement of a robotic arm 310 and a vision system 312 above the bin. The robotic arm 310 is similar to robotic arm 102, and the vision system 312 is similar to vision system 108. The vision system 312 may include, for example... Figure 2 The first camera 202 and the second camera 204 are shown. As described above, the vision system 108 captures a first set of images of randomly distributed mechanical parts in a bin. The first set of images is used by the neural network 110 to determine a first coarse location of a target set of parts from the randomly distributed mechanical parts in the bin. Subsequently, the information from the first set of images is combined to perform a first pose refinement on the first coarse location, thereby generating a first refined location. The controller 104 instructs the robotic arm 102 to pick up a target part from the target set of parts using the first refined location.
[0041] Image 306 shows the state of the robotic arm 310 after picking up a mechanical part from the bin. After the mechanical part is picked up, it is moved to position 314 for secondary pose estimation. To perform secondary pose estimation, vision system 312 captures a second set of images. After picking up the target part using a first refined position, controller 102 uses the second set of images captured by vision system 108 to perform a second coarse position determination and a second pose refinement. Image 308 shows the position of the robotic arm 310 after performing the second pose refinement on the mechanical part held in the robotic arm 310.
[0042] Figure 4 This is a block diagram of an exemplary system or device 400 within system 100 (e.g., controller 104). System 400 includes a processor 104, such as a central processing unit (CPU), and / or logic that executes computer-executable instructions to perform the functions, processes, and / or methods described herein. In some examples, the computer-executable instructions are stored locally and accessed from a non-transitory computer-readable medium (e.g., storage device 410, which may be a hard disk drive or flash drive). Read-only memory (ROM) 406 includes computer-executable instructions for initializing processor 404, while random access memory (RAM) 408 is main memory for loading and processing the instructions executed by processor 404. Network interface 412 can be connected to a wired or cellular network and to a local area network or wide area network. System 400 may also include a bus 402 connecting processor 404, ROM 406, RAM 408, storage device 410, and / or network interface 412. Components within system 400 may communicate with each other using bus 402. The components within system 400 are merely exemplary and may not include every component within controller 104. Additionally and / or alternatively, system 400 may further include components that may not be included in every entity of system 400. For example, in some examples, controller 104 may not include network interface 412. In some embodiments, system 400 may include a computing accelerator (GPU) 414 for running artificial intelligence (AI) deep learning neural network models.
[0043] Figure 5 The process 500 is illustrated as one or more examples of a random bin picking system executed by a controller according to this disclosure. However, it will be appreciated that any of the following blocks can be executed in any suitable order, and that process 500 can be executed in any environment and by any suitable computing device and / or controller.
[0044] At position 502, the vision system 108 captures an image of at least one mechanical part in the bin. In some embodiments, the vision system 108 may include a first camera 202 and a second camera 204. The first camera 202 and the second camera 204 may be color cameras or monochrome cameras, which may be mounted at different heights relative to the bin to accurately measure the depth of the mechanical part placed in the bin. In some embodiments, more than two cameras may be used to capture images of at least one mechanical part in the bin.
[0045] At position 504, a first coarse pose of each target part is estimated. In some embodiments, a first set of images captured by the first camera 202 and the second camera 204 is provided to the neural network 110. The neural network 110 selects target parts and determines a first coarse position of the mechanical parts in the bin.
[0046] At point 506, mechanical parts are associated in the multi-view system. In some embodiments, information from the first camera 202 and the second camera 204 is combined. Figure 2 As discussed herein, the coordinates of the image captured by the first camera 202 and the coordinates of the image captured by the second camera 204 are combined to generate a combined coordinate system. A first approximate position of one or more mechanical parts in the bin can then be generated in this combined coordinate system.
[0047] At point 508, the first coarse position of the target mechanical part is refined. In some embodiments, the refinement of the first coarse position is performed using combined information from multiple camera views.
[0048] At 510, the controller 104 instructs the robotic arm 102 to pick up the mechanical part based on the first refined pose of the mechanical part. In some embodiments, after the part is picked up, the robotic arm 102 moves with the mechanical part to a preset position for secondary pose estimation.
[0049] At point 512, after the mechanical part is picked up, the pose of the mechanical part is estimated and refined. In some embodiments, a secondary pose estimation is performed after the robotic arm 102 picks up the part and is moved to a preset position. The secondary pose estimation is performed using a second set of images of the robotic arm 102 and the mechanical part captured by the vision system 108. Using the second set of images captured by the vision system 108, a second coarse position of the mechanical part is determined while the robotic arm 102 holds the mechanical part. The images captured by the vision system 108 are then combined to perform a second refinement of the pose of the mechanical part while the robotic arm 102 holds the mechanical part.
[0050] At point 514, the mechanical part is placed on a preform mold or conveyor belt for further processing. For example, the controller 104 uses the second refined pose of the robotic arm 102 while holding the mechanical part to guide the robotic arm 102 to place the mechanical part in the correct pose on the preform mold or conveyor belt for further processing.
[0051] Although the subject matter of this disclosure has been illustrated and described in detail in the accompanying drawings and foregoing description, such illustrations and descriptions should be considered illustrative or exemplary, and not restrictive. Any statements made herein characterizing the invention should also be considered illustrative or exemplary, and not restrictive, as the invention is defined by the claims. It will be understood that changes and modifications can be made by those skilled in the art without departing from the scope of the appended claims, which may include any combination of features from the different embodiments described above.
[0052] The terms used in the claims should be interpreted as having the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the articles “a” or “the” when introducing an element should not be interpreted as excluding multiple elements. Similarly, references to “or” should be interpreted as inclusive, such that references to “A or B” do not exclude “A and B” unless it is clearly indicated from the context or the foregoing description that it is intended to refer only to one of A and B. Furthermore, references to “at least one of A, B, and C” should be interpreted as one or more of the group of elements consisting of A, B, and C, and should not be interpreted as requiring at least one of each of the listed elements A, B, and C, whether A, B, and C are related as a category or otherwise. Moreover, references to “A, B, and / or C” or “at least one of A, B, or C” should be interpreted as including any single entity from the listed elements, such as A, any subset from the listed elements, such as A and B, or the entire list of elements A, B, and C.
Claims
1. A method for random bin picking using a multi-view system, the method comprising: Capture the first set of images of a collection of mechanical parts randomly distributed in a bin; A subset of target parts is determined from the set of mechanical parts based on the captured images; A first coarse position is predicted for each target part in the subset of target parts using an AI-based pose estimation model; Refining the predicted first coarse position of each target part in the target part set, wherein refining the predicted coarse position of each target part includes: determining the position of each target part in a combined coordinate system generated by combining information from the first image set; Instructions are given to the robotic arm to pick up a target part from the target part set and to move the robotic arm carrying the target part to a preset position; When the target part is held in the preset position by the robotic arm, a second set of images of the target part is captured; The pose estimation model is used to predict a second coarse position of the target part; and Refining the second predicted coarse position of the target part, wherein refining the second predicted coarse position of the target part for precise placement includes: determining the position of the target part in a second combined coordinate system generated by combining information from the second image set.
2. The method according to claim 1, wherein the method further comprises: Based on the refined second predicted coarse position, the target part is placed on the conveyor belt for further processing.
3. The method according to claim 1, wherein the method further comprises: Based on the refined second predicted coarse position, the target part is placed in a preform mold for further processing.
4. The method of claim 1, wherein the first image set is captured simultaneously using a first camera and a second camera, wherein the first camera and the second camera are color cameras.
5. The method of claim 6, wherein the first camera and the second camera are arranged at different heights relative to the hopper.
6. The method of claim 1, wherein the prediction and refinement of the second coarse position are performed using the known position of the tool center tip of the robotic arm.
7. A system for random bin picking using a multi-view system, the system comprising: Controller, wherein the controller is configured to: Capture the first set of images of a collection of mechanical parts randomly distributed in a bin; A subset of target parts is determined from the set of mechanical parts based on the captured images; A first coarse position is predicted for each target part in the subset of target parts using an AI-based pose estimation model; Refining the predicted first coarse position of each target part in the target part set, wherein refining the predicted coarse position of each target part includes: determining the position of each target part in a combined coordinate system generated by combining information from the first image set; Instructions are given to the robotic arm to pick up a target part from the target part set and to move the robotic arm carrying the target part to a preset position; When the target part is held in the preset position by the robotic arm, a second set of images of the target part is captured; The pose estimation model is used to predict a second coarse position of the target part; and Refining the second predicted coarse position of the target part, wherein refining the second predicted coarse position of the target part for precise placement includes: determining the position of the target part in a second combined coordinate system generated by combining information from the second image set.
8. The system of claim 7, wherein the controller is further configured to place the target part on a conveyor belt based on the refined second predicted coarse position for further processing.
9. The system of claim 7, wherein the controller is further configured to place the target part in a preform mold based on the refined second predicted coarse position for further processing.
10. The system of claim 7, wherein the first image set is captured simultaneously using a first camera and a second camera, wherein the first camera and the second camera are color cameras.
11. The system of claim 10, wherein the first camera and the second camera are arranged at different heights relative to the hopper.
12. The system of claim 7, wherein the prediction and refinement of the second coarse position are performed using the known position of the tool center tip of the robotic arm.
