Method for estimating posture of object, method for robot to grasp object, control device, and robot system

Abstract

To provide a technique of a method for estimating a posture of an object in consideration of the depth of the object.SOLUTION: A method for estimating a posture of an object includes: an image acquisition step of acquiring, by an imaging unit configured to image an area in which a plurality of objects are arranged, a two-dimensional image including at least one of the plurality of objects; a small area creation step of creating, for at least one object among the at least one object in the two-dimensional image, a small area surrounding the object; a type estimation step of estimating a type of the object surrounded by the small area; a selection step of selecting one of the small areas in which the type of the object has been estimated in the type estimation step; a depth estimation step of estimating a depth of the one of the small areas based on the two-dimensional image; and a posture estimation step of estimating a three-dimensional posture of the object included in the one of the small areas based on the type of the object and the depth estimated in depth estimation step.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a method for estimating the posture of an object, a method for a robot to grasp an object, a control device, and a robot system. [Background technology]

[0002] Patent Document 1 discloses an object gripping system that uses a machine-learned inference model to output the position and rotation angle of an object from an image of the object and allows a robot to grip the object. In the technology of Patent Document 1, a rectangular bounding box is set in advance to fit the periphery of a two-dimensional image of a workpiece to be detected, and an inference model that has learned the position and rotation angle of the bounding box is prepared. The inference model further learns a picking point, which is a position on the surface of the workpiece for gripping by the robot.

[0003] A plurality of workpieces are stacked and arranged in a basket in the working area of ​​the robot. A two-dimensional image and a depth image of the plurality of workpieces are acquired by an imaging unit equipped with a camera capable of acquiring a two-dimensional image and a depth camera capable of acquiring a depth image having distance information to an object. Based on the two-dimensional image, the inference model outputs the positions and rotation angles of the plurality of workpieces in the basket and the coordinates of the picking points. Furthermore, based on the depth image, the inclination and surface roughness of the workpieces at the picking points, which are positions for each of the plurality of workpieces to be grasped by the robot, and the distance to the picking point are calculated. Next, from among the plurality of workpieces, the workpiece that is most likely to be grasped is determined based on the inclination and surface roughness of the workpiece at the picking point and the distance to the picking point. Then, the determined workpiece is grasped by the robot. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-197978 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, only the positions and rotation angles of multiple workpieces in a two-dimensional image are estimated by an inference model, and the depth image is used to estimate the distance to the picking point of the workpiece. In other words, the posture of the workpiece in Patent Document 1 is not estimated taking into account the depth. For example, in the case of a long and thin screw, when only the tip is displayed in a two-dimensional image, or in the case of an object with unevenness, the robot may not be able to accurately grasp the object unless it estimates the posture of the object taking into account the depth of each part of the screw or object in order to grasp it. Therefore, a method for estimating the posture of an object while taking into account the depth of each part of the object is required. [Means for solving the problem]

[0006] According to a first embodiment of the present disclosure, there is provided a method for estimating the orientation of an object, which includes an image acquisition step of acquiring a two-dimensional image including at least one of the objects by an imaging unit that captures an image of an area in which the objects are arranged, a small area creation step of creating a small area surrounding at least one of the objects among the one or more objects in the two-dimensional image, a type estimation step of estimating the type of the object surrounded by the small area, a selection step of selecting one of the small areas whose type of the object has been estimated in the type estimation step, a depth estimation step of estimating the depth of the one small area based on the two-dimensional image, and an orientation estimation step of estimating a three-dimensional orientation of the object included in the one small area based on the type of the object and the depth estimated in the depth estimation step.

[0007] According to a second aspect of the present disclosure, there is provided a method for a robot to grasp an object. The method for a robot to grasp an object includes the instruction step according to the method for estimating the posture of an object according to claim 2, an arrangement step in which the robot grasps one object based on the instruction and arranges it in a predetermined place, and an end determination step after the arrangement step, for determining whether all of the objects surrounded by the small areas created in the small area creation step have been arranged in the predetermined place by the robot, and if it is determined that all of the objects surrounded by the small areas created in the small area creation step have not been arranged in the predetermined place by the robot, the process proceeds to the selection step, where an unselected small area is selected, and if it is determined that all of the objects surrounded by the small areas created in the small area creation step have been arranged in the predetermined place by the robot, the process ends.

[0008] According to a third aspect of the present disclosure, there is provided a control device, which controls an imaging unit that captures an image of an area in which a plurality of objects are arranged and obtains a two-dimensional image including at least one of the plurality of objects, and a robot capable of grasping an object, creates a small region surrounding at least one of the one or more objects in the two-dimensional image, estimates a type of the object surrounded by the small region, selects one of the small regions in which the type of the object is estimated, estimates a depth of the one small region based on the two-dimensional image, estimates a three-dimensional orientation of the object included in the one small region based on the type of the object and the estimated depth, and instructs the robot to grasp the object based on the three-dimensional orientation of the object.

[0009] According to a fourth aspect of the present disclosure, there is provided a robot system, comprising: an imaging unit that images an area in which a plurality of objects are arranged, the imaging unit acquiring a two-dimensional image including at least one of the plurality of objects; a robot capable of grasping an object; and a control device that controls the imaging unit and the robot, the control device creating a small region surrounding at least one of the one or more objects in the two-dimensional image, estimating a type of the object surrounded by the small region, selecting one of the small regions in which the type of the object is estimated, estimating a depth of the one small region based on the two-dimensional image, estimating a three-dimensional orientation of the object included in the one small region based on the type of the object and the estimated depth, and instructing the robot to grasp the object based on the three-dimensional orientation of the object. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic configuration diagram of a robot system according to an embodiment of the present invention. [Diagram 2] Block diagram of the robot system. [Diagram 3] 3A and 3B are diagrams showing images of a plurality of objects captured by an imaging section. [Figure 4] 4 is a flowchart showing an example of a method for estimating the posture of an object and a method for the robot to grasp an object in the first embodiment. [Diagram 5] The output confidence table for each of the three subregions. [Figure 6] FIG. 11 is a diagram showing an image acquired by capturing an image of a working area in which a plurality of objects are placed in the second embodiment. [Figure 7] FIG. 13 is a diagram showing four objects arranged in a working area in the third embodiment. [Figure 8] 13 is a flowchart showing an example of a method for estimating the posture of an object and a method for a robot to grasp an object according to the third embodiment. [Figure 9] A diagram showing the placement rules. [Figure 10]8 is a diagram showing the four objects shown in FIG. 7 arranged in predetermined locations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A. First embodiment: A1. Configuration of the first embodiment: FIG. 1 is a schematic configuration diagram of a robot system 1 according to the present embodiment. FIG. 2 is a block diagram of the robot system 1. The robot system 1 can estimate three-dimensional postures of a plurality of objects and grasp the objects based on the estimated postures. As shown in FIG. 1, the robot system 1 includes an imaging unit 10, a robot 20, a display 30, an input device 40, and a control device 50. As shown in FIG. 2, the imaging unit 10, the robot 20, the display 30, and the input device 40 are electrically connected to the control device 50.

[0012] FIG. 3 is a diagram showing an image of a plurality of objects captured by the imaging unit 10. The imaging unit 10 shown in FIG. 1 captures an image of an area in which a plurality of objects are arranged. In this embodiment, the imaging unit 10 captures an image of an area in which a plurality of objects are arranged by capturing an image of a range including the area in which a plurality of objects are arranged, thereby capturing an image of the area in which a plurality of objects are arranged. Before describing the details of the imaging unit 10, the area in which a plurality of objects are arranged will be described. In this specification, the area in which a plurality of objects are arranged and the area in which a plurality of objects are arranged will be commonly referred to as the "working area WA". The working area WA is the area surrounded by a dashed frame in FIG. 1 and FIG. 3. In FIG. 1, no object is arranged in the working area WA. In FIG. 3, a state in which a plurality of objects are arranged in the working area WA is shown. In FIG. 3, three objects are arranged in the working area WA. As shown in FIG. 3, the three objects are respectively referred to as part 1, part 2, and part 3.

[0013] The imaging unit 10 captures an image of the working area WA, thereby acquiring a two-dimensional image including at least one of the multiple objects. In this embodiment, the imaging unit 10 captures a two-dimensional image including two or more of the multiple objects. Specifically, in this embodiment, a two-dimensional image including all of the three objects is acquired. In addition, when all of the multiple objects are not included in the image, it is a case where some objects are not reflected in the two-dimensional image because they are overlapped with other objects. The imaging unit 10 is controlled by the control device 50 to capture an image of the working area WA. In this embodiment, the imaging unit 10 is a stereo camera that is composed of two cameras and outputs each image. Note that FIG. 3 shows an image captured by one of the two cameras. The image captured by which camera is used can be arbitrarily changed by the worker. Note that both images captured by the two cameras may be used.

[0014] The robot 20 shown in FIG. 1 is capable of grasping an object. The robot 20 grasps an object upon receiving an instruction to grasp the object from the control device 50. In detail, the robot 20 grasps a specific object arranged in the working area WA upon receiving an instruction from the control device 50. Then, the robot 20 moves the grasped object to a predetermined arrangement location in accordance with an instruction from the control device 50. Note that the arrangement location is not shown in FIG. 1. In this embodiment, the robot 20 grasps and moves all of the three objects shown in FIG. 3 one by one upon receiving an instruction from the control device 50. Details of the instructions from the control device 50 will be described later.

[0015] In this embodiment, the robot 20 is a six-axis robot. As shown in FIG. 1, the robot 20 includes an arm 21 and a robot control unit 22. The arm 21 is equipped with a hand 210 as an end effector. The hand 210 can be realized as a gripper or a suction pad capable of gripping an object. The term "grasping" refers to the robot 20's actions of gripping and suctioning an object. In this embodiment, the hand 210 is a suction pad. A TCP (Tool Center Point) is set at the tip of the hand 210 as a control point of the robot 20. The control point TCP can be set at any position by the operator. The robot control unit 22 receives an instruction from the control device 50 and operates the arm 21. The type of object and a gripping method according to the three-dimensional posture of the object are input in advance to the robot control unit 22. The robot control unit 22 receives a gripping instruction including information on the three-dimensional posture of the object from the control device 50 and operates the arm 21 based on the posture information.

[0016] The display 30 displays an image captured by the imaging unit 10, an input image for the user to input various information to the control device 50, information estimated by the control device 50, etc. The information estimated by the control device 50 will be described later. The input device 40 is a device for the user to input various information to the control device 50.

[0017] The control device 50 controls the imaging unit 10 and the robot 20. The control device 50 also estimates the posture of an object and instructs the robot 20 to grasp the object. As shown in FIG. 2, the control device 50 includes a processor 51, a memory 52, an interface unit 53, and a communication unit 54. The processor 51 realizes various functions by using programs stored in the memory 52. ​​The memory 52 stores a first machine learning model 521, an object selection program 522, a second machine learning model 523, a third machine learning model 524, and a robot control program 525.

[0018] The first machine learning model 521 is a machine learning model that uses data of a two-dimensional image captured by the imaging unit 10 as an input value, creates a small area surrounding at least one of one or more objects included in the image, and outputs the type of object surrounded by the small area and the two-dimensional coordinates of the small area. The control device 50 uses the first machine learning model 521 to create a small area, and estimates the type of object surrounded by the small area and the coordinates of the small area. In this embodiment, a small area surrounding an object is created for each of two or more objects among a plurality of objects included in the image. Specifically, as shown in FIG. 3, a small area surrounding an object is created for each of three objects in the image. Then, the type of object surrounded by the created small area and the two-dimensional coordinates are estimated. The small area is represented by a rectangular frame surrounding the object, or a line along the outer contour of the object. In this embodiment, the small area is represented as a rectangular frame surrounding the object, as shown in FIG. 3. The type of object is output by ID. For example, part 1 will be output as ID1, part 2 as ID2, and part 3 as ID3.

[0019] The two-dimensional coordinates of the small area will be described. As an example, the two-dimensional coordinates of the first small area SA1, which is a small area surrounding the component 1 in FIG. 3, will be described. As a premise, coordinates are previously determined for pixels constituting an image captured by the imaging unit 10. As shown in FIG. 3, the upper left corner of the screen is the origin, the direction toward the right of the paper is the X axis, and the direction toward the top of the paper is the Y axis, and the coordinates of the pixels are expressed as real numbers. Note that the coordinates shown in FIG. 3 are shown for the purpose of explanation, and are not actually shown in the image. The coordinates of the origin are (X axis 0 pixel, Y axis 0 pixel). The first small area SA1 ranges from 100 pixels to 900 pixels in the X axis direction, and from 100 pixels to 800 pixels in the Y axis direction. Therefore, the coordinates of the first small area SA1 are expressed as "X axis 100 to 900 pixels, Y axis 100 to 800 pixels". Note that, for convenience, the size and coordinates of the pixels have been described as above, but the size and coordinates of the pixels can be changed arbitrarily. For convenience, the created small regions are shown in FIG. 3, but the small regions are not shown in the image captured by the imaging unit 10.

[0020] Furthermore, in this embodiment, the control device 50 uses the first machine learning model 521 to output the reliability of the type of object surrounded by the small region. When the types of objects surrounded by different small regions are the same, the reliability of each small region may differ depending on the light environment of the working area WA and the posture of the object. In addition, for example, when multiple objects are overlapped, the reliability of the type estimated for the object with the most exposed surface in the two-dimensional image captured by the imaging unit 10 tends to be the highest. The reliability is expressed as a normalized value from 0 to 1, with the value 1 being the highest reliability.

[0021] The first machine learning model 521 is created by performing machine learning including deep learning in advance outside the robot system 1. The first machine learning model 521 is a trained model that estimates the type of object and generates a small area. As a machine learning method, for example, an object detection algorithm such as R-CNN (Region Based Convolutional Neural Networks), SSD, or YOLO (You Look Only Once) is adopted. As training data for machine learning, an image in which the type of object and the small area surrounding the object are labeled with correct answer data is used. The training data may be created by an operator labeling the type and the size of the small area based on an image of an object, or may be created by generating a two-dimensional image including the object and labeling the type and the size of the small area through a simulation using a CAD model.

[0022] The object selection program 522 is a program for selecting one small region from among the small regions for which the type of object has been estimated by the control device 50 using the first machine learning model 521. In detail, the object selection program 522 is a program for selecting an object for which depth estimation is performed using the second machine learning model 523 described later. In this embodiment, the control device 50 uses the object selection program 522 to select one small region based on the reliability of the type of object surrounded by the small region output by the first machine learning model 521. Specifically, the control device 50 uses the object selection program 522 to select one small region having the highest reliability of the reliability of the estimated type of the object surrounded by the small region output by the first machine learning model 521.

[0023] The second machine learning model 523 is a machine learning model that outputs the depth of one small region based on a two-dimensional image. In detail, the second machine learning model 523 is a machine learning model that outputs the depth of the small region using the type of object surrounded by one small region and a two-dimensional image captured by the imaging unit 10 as input data. As input data, two-dimensional images acquired by two cameras at different positions constituting the imaging unit 10 are used. By using the second machine learning model 523, the depth of the small region is estimated based on the type of object estimated by the first machine learning model 521 and the two-dimensional image.

[0024] The second machine learning model 523 is a combination of trained models in which machine learning has been performed in advance for each type of object. The processor 51 of the control device 50 selects an appropriate trained model according to the type of the estimated object. As input data for machine learning, images of small areas including the object captured by the two cameras are used. As training data for machine learning, the depth of the image of the small area including the object is used. Note that the training data may be an image captured using a real object and a depth created by a depth sensor, or may be created by an ideal depth of a virtual image created by two cameras in a simulation. Machine learning including deep learning has been performed in advance on the trained models constituting the second machine learning model 523 outside the robot system 1. As a machine learning method for the second machine learning model 523, GC-Net (End to End Learning of Geometry and Context for Deep Stereo Regression), which is a depth estimation algorithm, is used.

[0025] The third machine learning model 524 is a machine learning model used to estimate the three-dimensional posture of an object included in one small region based on the type of object and the estimated depth of the small region. In this embodiment, the third machine learning model 524 is a trained machine learning model that outputs the three-dimensional posture of an object using the type of object, the estimated depth of the small region, and a two-dimensional image captured by the imaging unit 10 as input data. The control device 50 uses the third machine learning model 524 to estimate the coordinates of each of the important points, which are one or more points predetermined for the type of input object, based on the input two-dimensional image. The important points are, for example, a part that is easily grasped by the robot 20, an end of the object, etc. Then, the control device 50 uses the third machine learning model 524 to estimate the three-dimensional posture of the object based on the deviation between the coordinates of a reference posture predetermined for the object and the coordinates of each of the one or more important points, and the depth of the small region. The estimated coordinates of the three-dimensional posture are converted by the processor 51 of the control device 50 into coordinates based on the control point TCP of the robot 20 .

[0026] The third machine learning model 524 is a combination of trained models in which machine learning for each type of object has been performed in advance. The processor 51 of the control device 50 selects an appropriate trained model according to the estimated type of object included in the small area whose depth has been estimated using the second machine learning model 523. The trained models constituting the third machine learning model 524 are trained in advance by machine learning including deep learning outside the robot system 1. As a machine learning method, PVN3D (A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation) is used. Since PVN3D uses image information in addition to depth information, it is considered that the accuracy of estimating a three-dimensional posture is higher than that of an algorithm that does not use image information. As training data for machine learning, a posture created from an image and depth of a small area including an object is used. The training data may be created by an operator manually measuring an image and a depth, as in the creation of the second machine learning model 523, or may be created by simulation.

[0027] The robot control program 525 is made up of a number of commands for operating the robot 20 .

[0028] The interface unit 53 receives input from a user via the input device 40. The communication unit 54 transmits instructions to the imaging unit 10 and the robot control unit 22.

[0029] A2. Method for Estimating Object Pose and Method for Robot 20 Grasping Object: Fig. 4 is a flowchart showing an example of a method for estimating the posture of an object and a method for the robot 20 to grasp an object according to the first embodiment. As a premise, it is assumed that three objects shown in Fig. 3 are placed in the working area WA shown in Fig. 1. First, in step S10 of Fig. 4, the imaging unit 10 captures a two-dimensional image including the three objects placed in the working area WA. The imaging unit 10 transmits the captured two-dimensional image to the control device 50. The process of step S10 is referred to as an "image capture process".

[0030] In step S20 of FIG. 4, the control device 50 creates a small area surrounding at least one of the one or more objects in the two-dimensional image. The control device 50 creates a small area surrounding the object for one of the two two-dimensional images captured by each of the two cameras. In this embodiment, a small area surrounding the object is created for each of the three objects included in the image. Specifically, the control device 50 uses the first machine learning model 521 to create a small area surrounding the object for each of the three objects shown in FIG. 3. In FIG. 3, the first small area surrounding part 1 is labeled SA1, the second small area surrounding part 2 is labeled SA2, and the third small area surrounding part 3 is labeled SA3. The process of step S20 is referred to as a "small area creation process."

[0031] In step S30 of FIG. 4, the control device 50 estimates the type of object surrounded by the small region and the two-dimensional coordinates of the small region. In this embodiment, the control device 50 uses the first machine learning model 521 to estimate the type of object surrounded by the small region and the two-dimensional coordinates of the small region for each of the first small region SA1 to the third small region SA3. In this embodiment, the control device 50 uses the first machine learning model 521 to output the reliability of the object type estimated for each of the three small regions. The process of step S30 is referred to as a "type estimation process."

[0032] FIG. 5 is a table of the reliability output for each of the three small regions. In step S40 of FIG. 4, the control device 50 selects one small region from among the small regions in which the type of object has been estimated in the type estimation process. In this embodiment, the control device 50 uses the object selection program 522 to select one small region based on the reliability of the estimated type of the object surrounded by each of the three small regions. As shown in FIG. 5, among parts 1 to 3, part 1 has the highest reliability. Therefore, the first small region SA1 surrounding part 1 is selected. The process of step S40 is referred to as a "selection process."

[0033] In step S50 of FIG. 4, the control device 50 estimates the depth of the selected small region using the second machine learning model 523. In this embodiment, the control device 50 estimates the depth of the small region based on the type of object surrounded by the selected small region and the two-dimensional image acquired by the imaging unit 10, using a trained model prepared in advance according to the type of object, as described above. Since the first small region SA1 is selected in the selection process, as described above, the control device 50 measures the depth using a trained model according to the estimated type of object included in the first small region SA1. The process of step S50 is referred to as a "depth estimation process".

[0034] In step S60, the control device 50 estimates the three-dimensional orientation of an object included in one small area based on the estimated object type, the depth estimated in the depth estimation process, and the two-dimensional image. The control device 50 estimates the three-dimensional orientation of the part 1 included in the first small area SA1 by using the third machine learning model 524. The process of step S60 is referred to as a "pose estimation process."

[0035] In step S70, the control device 50 instructs the robot 20 to grasp the object based on the three-dimensional posture of the object estimated in the posture estimation process. In detail, the communication unit 54 of the control device 50 transmits to the robot control unit 22 of the robot 20 instructions including the estimated posture of the part 1 whose posture was estimated in the posture estimation process, the coordinates of the part to be grasped, the placement position in the placement location, and the operation method of the arm 21. The coordinates of the object are converted into coordinates based on the control point TCP of the robot 20 and transmitted. The process of step S70 is referred to as an "instruction process".

[0036] In step S80, the robot 20 grasps the part 1 based on the instruction and the information on the posture of the part 1. The robot 20 places the part 1 at the placement location. The process of step S80 is referred to as the "placement process." After placing the object at the placement location, the robot control unit 22 transmits a signal to the control device 50 that the placement of the object instructed to be placed has been completed.

[0037] In step S90, the control device 50 judges whether all of the objects surrounded by the small areas created in the small area creation step have been placed in the predetermined locations by the robot 20. In this embodiment, it is judged whether all of the three objects have been placed in the predetermined locations. This step is referred to as a "completion judgment step." First, after the placement step, the imaging unit 10 captures an image of a range including the working area WA. The control device 50 compares the image acquired in step S10 with the image to judge whether the object surrounded by the small area is included in the image. If the image contains an object surrounded by the small area, it is judged that all of the objects surrounded by the small area have not been placed in the placement location by the robot 20. In this case, the process proceeds to the selection step. As described above, since the placement of the part 1 in the predetermined location has been completed, only the part 2 and the part 3 are placed in the working area WA. In the second selection step, the second small area SA2, which has a high reliability as shown in FIG. 5, is selected from the second small area SA2 and the third small area SA3 that have not been selected. If the control device 50 determines that all of the objects enclosed in the small regions have been placed in the predetermined locations by the robot 20, the process ends.

[0038] In this embodiment, the posture of an object can be estimated while taking into account the depth of the object. In addition, the area in which the depth is estimated is smaller than in a mode in which the depths of all of the objects are calculated based on the depth of the area in which all of the objects are included. There is a small possibility that an object other than the object for which the depth is to be estimated is included in the area. Therefore, the accuracy of the depth estimation is improved. In addition, instructions based on the posture of the object estimated while taking into account the depth of the object can be given to the robot 20.

[0039] In this embodiment, the depth is estimated using a trained model corresponding to the type of object. For example, based on the depth of an area including all of the multiple different types of objects, the depths of all of the multiple objects can be compared with the aspect required by the trained model, thereby reducing the amount of trained data and the training time. In addition, for example, when the type of object is changed, a trained model for only the new object only needs to be created, and there is no need to retrain the trained model to include the depths of all of the multiple different types of objects. Therefore, the method for estimating the posture of an object can be efficiently operated.

[0040] Furthermore, in this embodiment, one small area selected in the selection process is selected based on the reliability of the estimated object type. Since the posture is estimated for an object with high reliability, the accuracy of the estimated posture is likely to be higher than for an object with low reliability. Therefore, there is a high possibility that the robot 20 can grasp it reliably. Furthermore, in this embodiment, after the end determination process, an unselected small area is selected again in the selection process. All of the objects surrounded by the small areas created in the small area creation process can be placed in the placement location by the robot 20.

[0041] In this embodiment, the imaging unit 10 is a stereo camera. For example, when RGBD is used to acquire depth information of a two-dimensional image, the timing of shooting between the RGB camera and the depth camera may differ. By using a stereo camera, the depth can be estimated based on multiple two-dimensional images captured simultaneously. Therefore, images and depths captured at the same time can be acquired. Therefore, the posture of an object can be accurately estimated even if the object is being transported on a belt conveyor.

[0042] B. Second embodiment: The second embodiment differs from the first embodiment in the method of creating a small area in the small area creation process, the method of selecting a small area in the selection process, the number of objects arranged in the working area WA, and the state of arrangement of the objects. Since the other configurations are the same as those of the first embodiment, the same reference numerals are used and detailed description will be omitted.

[0043] In the second embodiment, the control device 50 uses the object selection program 522 to select one small region based on the coordinates of each of the multiple small regions. In particular, the control device 50 selects a small region located at coordinates closest to predetermined coordinates. In this embodiment, the predetermined coordinates are the origin (0 pixel on the X-axis, 0 pixel on the Y-axis). The smallest value of the coordinates of the small region is compared with the origin. Note that, for example, when multiple small regions are created with the same distance between the minimum coordinates on the X-axis and the Y-axis and the origin, such as when the coordinates of one small region are (100-1000 pixels on the X-axis, 200-500 pixels on the Y-axis) and the coordinates of another small region are (200-300 pixels on the X-axis, 100-500 pixels on the Y-axis), the small region with the smallest value on the Y-axis is selected. The method of selecting the predetermined coordinates and the closest coordinates can be changed arbitrarily.

[0044] Fig. 6 is a diagram showing an image acquired by imaging the working area WA in which a plurality of objects are arranged in the second embodiment. In the second embodiment, imaging is performed so that the imaging range of the imaging unit 10 coincides with the working area WA. In step S10 of Fig. 4, as shown in Fig. 6, the imaging unit 10 captures a two-dimensional image including the plurality of objects arranged in the working area WA. In the second embodiment, the plurality of objects are arranged so as to overlap each other as shown in Fig. 6.

[0045] In step S20 of FIG. 4, the control device 50 creates a small region surrounding at least one of the multiple objects in the image. In FIG. 6, a rectangular frame representing the small region is shown for ease of understanding. In the second embodiment, as shown in FIG. 6, small regions surrounding some of the multiple objects included in the image are created, not all of the objects. The control device 50 uses the first machine learning model 521 to create a small region for the detected object. As shown in FIG. 6, there are objects that are overlapped by other objects, for which no small region has been created. Also, as shown in FIG. 6, there are cases where two or more objects are included in the same small region, but the first machine learning model 521 is set so that only one object is recognized as being included in the small region by the control device 50. Note that in the second embodiment, in step S30 of FIG. 4, the reliability is not output for each estimated type of object surrounded by the small region.

[0046] In step S50 of FIG. 4, the control device 50 selects one small region. In the second embodiment, the control device 50 uses the object selection program 522 to select one small region based on the coordinates of each of the multiple small regions. Of the multiple small regions shown in FIG. 6, a fourth small region SA4 closest to the origin (0 pixel on the X-axis, 0 pixel on the Y-axis) is selected. In step S70 of FIG. 4, the robot 20 places the object in the placement location. In the second embodiment, the robot 20 places the objects side by side in the placement location.

[0047] In step S90 in FIG. 4, the control device 50 determines whether all of the objects surrounded by the small areas have been placed in the placement locations by the robot 20. In the second embodiment, the control device 50 determines whether all of the objects surrounded by the small areas shown in FIG. 6 have been placed in the placement locations. Then, after the process is completed, the worker causes the control device 50 to execute the process of step S10 in FIG. 4 again. By moving some of the objects placed in the working area WA to the placement locations, objects that were not surrounded by small areas in the previous process are surrounded by the small areas. Therefore, by repeating the process, all of the objects are placed in the placement locations. Note that the position or posture of the object may be changed by the worker or the robot 20 before the process of step S10 is performed again.

[0048] In the second embodiment, the robot 20 can grasp the object based on the coordinates of a small area surrounding the object. The robot 20 can quickly grasp the object.

[0049] C. Third embodiment: The third embodiment differs from the first embodiment in the method of selecting small areas in the selection process, the type and number of objects arranged in the working area, and the arrangement method in the arrangement process. Since the other configurations are the same as those in the first embodiment, the same reference numerals are used and detailed descriptions are omitted.

[0050] The control device 50 in the third embodiment uses an object selection program 522 to select one small area based on a predetermined order of objects to be grasped by the robot 20. The order of objects to be placed at the placement locations and the method of placement by the robot 20 are input in advance into the memory 52 of the control device 50 by an operator. The order of objects to be placed at the placement locations corresponds to the order of objects to be grasped by the robot 20. For example, in order to have the robot 20 perform an assembly task, if the order of placing objects is determined or the objects need to be placed depending on the front / back or orientation of the objects, the order is input in advance by the operator.

[0051] FIG. 7 is a diagram showing four objects arranged in the working area WA in the third embodiment. FIG. 7 is not a diagram showing a diagram captured by the imaging unit 10. In the third embodiment, a worker inputs in advance to the object selection program 522 of the control device 50 a "placement rule" indicating the type of object whose orientation is to be estimated and the type of object whose orientation is not to be estimated. Specifically, the types of parts A and B are input as the types of objects whose orientation is to be estimated. Part A is an object that needs to be placed overlapping part B at the placement location. The types of parts C and D are input as the types of objects whose orientation is not to be estimated.

[0052] Fig. 8 is a flowchart showing an example of a method for estimating the posture of an object and a method for the robot 20 to grasp an object according to the third embodiment. In step S10C in Fig. 8, a placement rule is input by an operator to the object selection program 522 of the control device 50. In step S20C, an image including the four parts placed in the working area WA is acquired by the imaging unit 10. The process of step 20C is the same as step S10C in Fig. 4. In step S30C in Fig. 8, a small area creation step and a type estimation step are executed.

[0053] Fig. 9 is a diagram showing the arrangement rules. In step S40C in Fig. 8, the control device 50 uses the object selection program 522 to select small regions based on the order of grasping of the objects. As shown in Fig. 9, the order is determined as follows: part B, part A, part C, and part D. The control device 50 selects a small region surrounding part B, which has the earliest order. The order is determined in advance by the operator depending on whether or not it is necessary to estimate the posture, whether or not it is necessary to estimate the depth, etc.

[0054] In step S50C of FIG. 8, the control device 50 judges whether or not it is necessary to estimate the depth of the small region selected in the selection process. The judgment is made according to the arrangement rule. In the arrangement rule shown in FIG. 9, the part B, which is the object with the earliest order, needs to be overlapped with the part A as described above. Therefore, the posture needs to be estimated. The same is true for the part A. It is considered that it is not necessary to estimate the posture of the part C for the robot 20 to grasp it. However, since estimating the depth makes it easier for the robot 20 to grasp it, it is considered that it is necessary for the operator to estimate the depth. For the part D, since it can be picked up by the hand 210 by bringing the robot 20 close to the part D, it is considered that it is not necessary for the operator to estimate the depth. If it is judged that it is necessary to estimate the depth, the process proceeds to step S60C. If it is judged that it is not necessary to estimate the depth, the process proceeds to step S70C.

[0055] In step S60C of FIG. 8, the control device 50 executes a depth estimation process. In step S70C, the control device 50 determines whether or not it is necessary to estimate the posture of the small region selected in the selection process. If it is determined that it is necessary to estimate the posture, the process proceeds to step S80C. If it is determined that it is not necessary to estimate the posture, the process proceeds to step S90C. In step S80C, the posture estimation process is executed. In step S90C, the control device 50 determines coordinates for grasping by the robot 20. In this embodiment, the control device 50 determines the center of the small region as the coordinates for grasping by the robot 20. In step S90C, if the depth estimation process is executed in step S60C, the estimated depth is determined as information of the depth required for grasping by the robot 20. In step 90C, if the depth estimation process is not executed in step S60C, a preset initial value is determined as information of the depth required for grasping by the robot 20. In this embodiment, the depth of a work table on which no object is placed is set as the initial value, based on the imaging unit 10. Note that the initial value can be changed arbitrarily.

[0056] In step S100C, the placement process is executed by the robot 20. The robot 20 grasps the object and places it at the placement location according to the picking location and placement location predetermined for each type of object as shown in FIG. 9. As described above, the postures in which parts A and B are placed are predetermined. In grasping part A or part B, the robot 20 grasps the object based on an instruction received from the control device 50. On the other hand, in the case of part C or part D, the robot 20 places the object at the placement location regardless of the posture of the object.

[0057] In step S110C, the control device 50 executes an end determination step. If it is determined in the end determination step that all of parts A through D have been placed in the predetermined locations, the process ends. If it is determined in the end determination step that all of parts A through D have not been placed in the predetermined locations, the process moves to step S40C again. In step S40C again, a small region containing an object is selected that is next in order to the object contained in the small region selected in the previous process.

[0058] Fig. 10 is a diagram showing a state in which the four objects shown in Fig. 7 are arranged at the arrangement location. As shown in Fig. 10, part A is stacked on part B. In this way, in the third embodiment, the robot 20 can arrange a plurality of objects in a stacked state at the arrangement location.

[0059] D. Other embodiments: D1. Alternative embodiment 1: (1) In the first embodiment, the imaging unit 10 captures an image of a range including the area where the multiple objects are placed, thereby capturing an image of the area where the multiple objects are placed. Note that the imaging unit may capture an image so as not to include an area other than the work area.

[0060] (2) In the first embodiment, small regions are created for all three detected objects. Note that, for example, in a case where all three objects are the same type and an instruction is input to the control device to have the robot place two objects in the placement locations, the control device may use the first machine learning model to select two objects from the three detected objects, create small regions for the two objects, and have the robot grasp the two objects in turn.

[0061] (3) In the above embodiment, the first machine learning model 521, the second machine learning model 523, and the third machine learning model 524 are created by performing machine learning including deep learning in advance outside the robot system 1. Note that the control device may include a teacher data creation unit that creates teacher data and a learning execution unit that executes machine learning, and the first machine learning model, the second machine learning model, and the third machine learning model may be created within the control device.

[0062] (4) In the above embodiment, the imaging unit 10 captures a two-dimensional image including a plurality of objects. Note that, for example, in a mode in which a plurality of objects are arranged in the working area, the imaging unit may capture a two-dimensional image including only one of the objects. In this mode, a small area surrounding the one object is created in the small area creation step, and the small area is selected in the selection step.

[0063] (5) In the above embodiment, in the type estimation step, the two-dimensional coordinates of the small region are output. Note that, for example, in a case where the object is not placed in the placement location by the robot, the first machine learning model does not output the two-dimensional coordinates of the small region, and the two-dimensional coordinates of the small region do not need to be estimated in the type estimation step.

[0064] (6) In the above embodiment, the second machine learning model 523 is a machine learning model that takes the type of object surrounded by one small region and a two-dimensional image captured by the imaging unit 10 as input data and outputs the depth of the small region. For example, the second machine learning model may be a machine learning model that takes the type of object surrounded by one small region and a two-dimensional image captured by the imaging unit as input data and outputs the depth of a plurality of points predetermined according to the estimated type of object. In this aspect, the second machine learning model has learned feature points, which are a plurality of points necessary for the third machine learning model to estimate the three-dimensional posture. The feature points are predetermined by the operator according to the type of object, and are, for example, a portion to which the hand of the robot can adhere, a portion that the hand may come into contact with when the robot grasps, etc. This allows the robot to grasp with high accuracy. In addition, for example, in an aspect in which all objects included in the two-dimensional image are of the same type, the depth of one small region may be estimated based on the two-dimensional image.

[0065] (7) In the above embodiment, the control device 50 performs a posture estimation process based on the type of object, the estimated depth of the small region, and the two-dimensional image by using the third machine learning model 524. For example, in a mode in which the coordinates of each of the important points, which are one or more points predetermined for the input object type, are not estimated, the posture estimation process may be performed based on the type of object and the estimated depth of the small region.

[0066] (8) In the above embodiment, the robot is a six-axis robot. However, the robot may be a SCARA robot.

[0067] D2. Alternative embodiment 2: In the above embodiment, the robot system 1 includes the robot 20, and an instruction is given to the robot 20 to grasp an object based on the three-dimensional orientation of the object. Note that, for example, in a system not including a robot, an object does not have to be grasped based on the three-dimensional orientation of the object.

[0068] D3. Alternative embodiment 3: In the above embodiment, the depth is estimated by using a trained model prepared in advance according to the type of object. Note that the depth may be estimated by an operator based on the type of object and the depth of the region.

[0069] D4. Alternative embodiment 4: In the above embodiment, one small region is selected from the plurality of small regions in the selection step. For example, in an aspect in which one small region is created in the small region creation step, the one small region may be selected by the control device in the selection step.

[0070] D5. Alternative embodiment 5: In the above embodiment, the image capturing unit 10 is a stereo camera. For example, the image capturing unit may be a monocular camera, and the depth of the small region may be estimated by using a technique for estimating depth using the monocular camera. In addition, the two-dimensional image may be either color or grayscale.

[0071] D6. Alternative embodiment 6: (1) In the above embodiment, the placement step is included. For example, in a system that does not include a robot, the placement step does not need to be executed.

[0072] (2) In the above embodiment, a termination determination step is provided. For example, in a mode in which one small area is created in the small area creation step, the process may be terminated without executing the termination determination step. Note that in a mode in which one small area is created in the small area creation step, the termination determination step may be performed and the process may be terminated when it is determined that all of the objects surrounded by the small areas have been placed in the placement locations by the robot.

[0073] (3) In the first embodiment described above, in the termination determination step, if an image includes objects surrounded by small areas and it is determined that all of the objects surrounded by the small areas have not been placed in the placement locations by the robot 20, the process proceeds to a selection step. Note that, for example, if it is determined in the termination determination step that all of the objects surrounded by the small areas have not been placed in the placement locations by the robot, the process may proceed to an image acquisition step, an image is acquired again, and a small area is created.

[0074] For example, when moving from the end determination step to the image acquisition step again, the position or orientation of the object placed in the working area may be changed by the operator or the robot. For a small area that had a reliability lower than a predetermined value, changing the position or orientation by the robot or the operator may enable the estimation of the orientation of the object with a high reliability.

[0075] D7. Alternative Embodiment 7: (1) In the above embodiment, the robot system 1 includes the robot 20. Note that the system may include, for example, an imaging unit and a control device without including a robot. Also, the control device is not limited to one that controls the robot system.

[0076] (2) For example, the robot may be equipped with a control device, or the imaging unit may be equipped with a control device.

[0077] (3) In the above embodiment, the robot system 1 includes the display 30 and the input device 40. However, for example, the robot system may be configured not to include a display and an input device.

[0078] E. Other forms: The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various forms without departing from the spirit of the present disclosure. For example, the present disclosure can be realized in the following forms. The technical features in the above-mentioned embodiments corresponding to the technical features in each form described below can be appropriately replaced or combined in order to solve some or all of the problems of the present disclosure, or to achieve some or all of the effects of the present disclosure. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted.

[0079] (1) According to one embodiment of the present disclosure, there is provided a method for estimating the orientation of an object, the method for estimating the orientation of an object including an image acquisition step of acquiring a two-dimensional image including at least one of the objects by an imaging unit that captures an image of an area in which the objects are arranged, a small area creation step of creating a small area surrounding at least one of the objects among the one or more objects in the two-dimensional image, a type estimation step of estimating a type of the object surrounded by the small area, a selection step of selecting one of the small areas in which the type of the object has been estimated in the type estimation step, a depth estimation step of estimating a depth of the one small area based on the two-dimensional image, and an orientation estimation step of estimating a three-dimensional orientation of the object included in the one small area based on the type of the object and the depth estimated in the depth estimation step. According to this type of method for estimating the orientation of an object, the orientation of the object can be estimated while taking into account the depth of the object.

[0080] (2) The method for estimating the orientation of an object having the above-described configuration may further include an instruction step of instructing a robot to grasp the object based on the three-dimensional orientation of the object. According to this method for estimating the orientation of an object, it is possible to give instructions to the robot based on the orientation of the object estimated while taking into account the depth of the object.

[0081] (3) In the method for estimating the posture of an object of the above-mentioned form, in the depth estimation step, the depth of the one small region may be estimated using a trained model prepared in advance according to the type of the object. According to this method for estimating the orientation of an object, the depth is estimated using a trained model corresponding to the type of object. For example, based on the depth of an area including all of the different types of objects, the depths of all of the objects can be compared with the aspect required by the trained model, thereby reducing the amount of training data and the training time. In addition, for example, when the type of object is changed, a trained model for only the new object only needs to be created, and there is no need to retrain the trained model to include the depths of all of the different types of objects. Therefore, the method for estimating the orientation of an object can be efficiently operated.

[0082] (4) In the method for estimating the posture of an object of the above-mentioned form, in the image acquisition step, a two-dimensional image including two or more of the plurality of objects is acquired, in the small area creation step, a small area surrounding each of the two or more objects included in the image is created, and in the selection step, one small area is selected based on a high reliability of the type of object surrounded by the small area, which is output by a trained model that performs object type estimation and small area generation. According to this method for estimating the posture of an object, the posture of an object with high reliability is estimated, so the accuracy of the estimated posture is likely to be higher than that of an object with low reliability, and therefore the robot is likely to be able to grasp the object reliably.

[0083] (5) In the method for estimating the posture of an object of the above-mentioned form, the image acquisition step may acquire a two-dimensional image including two or more of the plurality of objects, the small area creation step may create a small area surrounding each of two or more of the objects included in the image, the type estimation step may further estimate coordinates of the small area, and the selection step may select one small area based on the coordinates of each of the small areas. According to this method for estimating the orientation of an object, it is possible to cause a robot to grasp an object based on the coordinates of a small area surrounding the object, and it is possible to cause the robot to grasp the object quickly.

[0084] (6) In the method for estimating the posture of an object of the above-mentioned form, in the image acquisition step, a two-dimensional image including two or more of the plurality of objects is acquired, in the small area creation step, a small area surrounding each of the two or more objects included in the image is created, and in the selection step, one small area is selected based on a predetermined order of the objects to be grasped by the robot. According to this type of method for estimating the orientation of an object, it is possible to have the robot place a plurality of objects in a stacked state in a predetermined location.

[0085] (7) In the method for estimating the orientation of an object described above, the imaging unit may be a stereo camera. For example, when depth information of a two-dimensional image is acquired using RGBD, the timing of shooting with the RGB camera and the depth camera may differ. According to this method of estimating the object's posture, the stereo camera is used, so that the depth can be estimated based on multiple two-dimensional images captured simultaneously. Therefore, images and depths captured at the same time can be acquired. Therefore, the posture of the object can be accurately estimated even if the object is being transported on a conveyer belt, for example.

[0086] (8) According to another aspect of the present disclosure, there is provided a method for a robot to grasp an object. The method for a robot to grasp an object includes the instruction step according to the method for estimating the posture of an object according to claim 2, an arrangement step in which the robot grasps one of the objects based on the instruction and arranges it in a predetermined place, and an end determination step after the arrangement step, in which it is determined whether all of the objects surrounded by the small areas created in the small area creation step have been arranged in the predetermined place by the robot, and if it is determined that all of the objects surrounded by the small areas created in the small area creation step have not been arranged in the predetermined place by the robot, the process proceeds to the selection step, in which an unselected small area is selected, and if it is determined that all of the objects surrounded by the small areas created in the small area creation step have been arranged in the predetermined place by the robot, the process ends. According to this form of method for a robot to grasp an object, the robot can be made to grasp a plurality of objects.

[0087] (9) According to another aspect of the present disclosure, there is provided a control device that controls an imaging unit that captures an image of an area in which a plurality of objects are arranged and obtains a two-dimensional image including at least one of the plurality of objects, and a robot capable of grasping an object, creates a small region surrounding at least one of the one or more objects in the two-dimensional image, estimates a type of the object surrounded by the small region, selects one of the small regions in which the type of the object is estimated, estimates a depth of the one small region based on the two-dimensional image, estimates a three-dimensional orientation of the object included in the one small region based on the type of the object and the estimated depth, and instructs the robot to grasp the object based on the three-dimensional orientation of the object.

[0088] (10) According to another aspect of the present disclosure, a robot system is provided, the robot system including: an imaging unit that images an area in which a plurality of objects are arranged, the imaging unit acquiring a two-dimensional image including at least one of the plurality of objects; a robot capable of grasping an object; and a control device that controls the imaging unit and the robot, the control device creating a small region surrounding at least one of the one or more objects in the two-dimensional image, estimating a type of the object surrounded by the small region, selecting one of the small regions in which the type of the object is estimated, estimating a depth of the one small region based on the two-dimensional image, estimating a three-dimensional orientation of the object included in the one small region based on the type of the object and the estimated depth, and instructing the robot to grasp the object based on the three-dimensional orientation of the object.

[0089] (11) In the robot system of the above aspect, the imaging unit may be a stereo camera.

[0090] The present disclosure may be realized in various forms other than a method for estimating the posture of an object, a method for a robot to grasp an object, a control device, or a robot system. For example, the present disclosure may be realized in the form of a control device or a manufacturing method for a robot system, a control method for a control device or a robot system, a computer program for implementing the control method, a method for verifying whether an estimated posture is correct, or a method for a robot to grasp an object, a non-transitory recording medium on which the computer program is recorded, or the like. [Explanation of symbols]

[0091] 1...robot system, 10...imaging unit, 20...robot, 21...arm, 22...robot control unit, 30...display, 40...input device, 50...control unit, 51...processor, 52...memory, 53...interface unit, 54...communication unit, 210...hand, 521...first machine learning model, 522...object selection program, 523...second machine learning model, 524...third machine learning model, 525...robot control program, SA1...first small area, SA2...second small area, SA3...third small area, SA4...fourth small area, TCP...control point

Claims

1. 1. A method for estimating a pose of an object, comprising: an image acquisition step of acquiring a two-dimensional image including at least one of the plurality of objects by an imaging unit that images an area in which the plurality of objects are arranged; a small region creating step of creating a small region surrounding at least one of the one or more objects in the two-dimensional image; a type estimation step of estimating a type of the object surrounded by the small region; a selection step of selecting one of the small regions for which the type of the object has been estimated in the type estimation step; a depth estimation step of estimating a depth of the one small region based on the two-dimensional image; A method for estimating the posture of an object, comprising: a posture estimation process for estimating a three-dimensional posture of the object contained in one small region based on the type of the object and the depth estimated in the depth estimation process.

2. A method for estimating an object pose according to claim 1, comprising: The method for estimating the pose of an object further includes an instruction step of instructing a robot to grasp the object based on the three-dimensional pose of the object.

3. A method for estimating an object pose according to claim 2, comprising the steps of: A method for estimating the posture of an object, in which, in the depth estimation process, the depth of the one small region is estimated using a trained model prepared in advance according to the type of the object.

4. A method for estimating an object pose according to claim 1, comprising: In the image acquisition step, a two-dimensional image including two or more of the plurality of objects is acquired; In the small region creating step, a small region surrounding each of two or more objects among the objects included in the image is created; In the selection step, the one small region is A method for estimating the pose of an object, in which a small region is selected based on the reliability of the type of object enclosed by the small region, as output by a trained model that estimates the type of object and generates the small region.

5. A method for estimating an object pose according to claim 1, comprising: In the image acquisition step, a two-dimensional image including two or more of the plurality of objects is acquired; In the small region creating step, a small region surrounding each of two or more objects among the objects included in the image is created; The type estimation step further comprises estimating coordinates of the small region; A method for estimating the pose of an object, wherein in the selecting step, the one small region is selected based on coordinates of each of the small regions.

6. A method for estimating an object pose according to claim 2, comprising the steps of: In the image acquisition step, a two-dimensional image including two or more of the plurality of objects is acquired; In the small region creating step, a small region surrounding each of two or more objects among the objects included in the image is created; A method for estimating the posture of an object, wherein in the selection step, the one small region is selected based on a predetermined order in which the object is grasped by the robot.

7. A method for estimating an object pose according to claim 1, comprising: The method for estimating the pose of an object, wherein the imaging unit is a stereo camera.

8. A method for a robot to grasp an object, comprising: The method for estimating the pose of an object according to claim 2, further comprising: a placement step in which the robot grasps the one object and places it at a predetermined location based on the instruction; a completion determination step of determining whether or not all of the objects surrounded by the small areas created in the small area creation step have been placed in the predetermined locations by the robot, after the placement step; When it is determined that all of the objects surrounded by the small regions created in the small region creation step have not been placed in the predetermined locations by the robot, the method proceeds to the selection step, in which an unselected small region is selected; A method for a robot to grasp an object, the method including terminating processing when it is determined that all of the object enclosed in the small area created in the small area creation process has been placed in the predetermined location by the robot.

9. A control device, an imaging unit that captures an image of an area in which a plurality of objects are arranged, the imaging unit acquiring a two-dimensional image including at least one of the plurality of objects; A robot capable of grasping an object; Control the creating a small region surrounding at least one of the one or more objects in the two-dimensional image; Estimating the type of the object that the small region encloses; selecting one of the small regions in which the type of the object has been estimated; Estimating a depth of the one small region based on the two-dimensional image; estimating a three-dimensional orientation of the object included in one of the small regions based on the type of the object and the estimated depth; A control device that instructs the robot to grasp the object based on the three-dimensional orientation of the object.

10. 1. A robot system comprising: an imaging unit that captures an image of an area in which a plurality of objects are arranged, the imaging unit acquiring a two-dimensional image including at least one of the plurality of objects; A robot capable of grasping an object; A control device that controls the imaging unit and the robot, creating a small region surrounding at least one of the one or more objects in the two-dimensional image; Estimating the type of the object that the small region encloses; selecting one small region from among the small regions in which the type of the object has been estimated; Estimating a depth of the one small region based on the two-dimensional image; estimating a three-dimensional orientation of the object included in one of the small regions based on the type of the object and the estimated depth; a control device that instructs the robot to grasp the object based on a three-dimensional posture of the object.

11. The robot system according to claim 10, The robot system, wherein the imaging unit is a stereo camera.

Images