ROBOT CONTROL DEVICE, ROBOT CONTROL METHOD, AND PROGRAM

Abstract

Provided are a device and a method for enabling reliable execution of grasping processing for an object by a robot. In this invention, there are generated an overlooking-camera reference containing box containing an object to be grasped contained in a captured image by an overlooking camera mounted to a robot and a hand-end camera reference containing box containing the object to be grasped contained in a captured image of a hand-end camera mounted to the robot. Moreover, there is calculated a relative position of a target grasping position of the object to be grasped with respect to the overlooking-camera reference containing box in the captured image by the overlooking camera, there is calculated a target grasping position with respect to the hand-end camera reference containing box in the captured image by the hand-end camera on the basis of the calculated relative position, and the calculated position is set to a corrected target grasping position of the object to be grasped contained in the captured image by the hand-end camera. Further, there is generated control information for causing a hand of the robot to grasp the corrected target grasping position in the captured image by the hand-end camera to cause the robot to execute grasping processing.

Description

[Technical field]

[0001] The present disclosure relates to a robot control device, a robot control method, and a program, and more particularly to a robot control device, a robot control method, and a program that control a gripping process of an object by a robot. [Background technology]

[0002] In recent years, the use of robots has been increasing in various fields. For example, robots are used to assemble products in factories, and nursing care robots are also being developed.

[0003] These robots perform a variety of tasks, one example of which is the task of grasping and moving an object. For example, in the case of an assembly robot used in a factory, a hand with a gripping mechanism connected to the robot's arm is used to grasp a part to be used in product assembly, and while still holding the part, the robot moves to a specified position and then releases the grip, allowing the part to be attached to another object.

[0004] Also, for example, in the case of a nursing care robot, a process is performed in which a cup placed on a table out of reach of the user is grasped by the robot's hand, carried to a position within reach of the user, and handed to the user.

[0005] When having a robot grasp such an object, for example, an image of the surroundings is captured using an overhead camera with a field of view that allows an overall understanding of the robot's surroundings, the captured image is analyzed to confirm the position of the object to be grasped, and the hand is then moved to the position of the object to be grasped to grasp it.

[0006] However, if there is an error in the position information obtained based on the analysis of the image captured by the overhead camera, the destination of the hand may deviate from the target grasping position, resulting in a failure of the grasping process.

[0007] Furthermore, mechanical errors in the robot's arm (deflection or backlash) can cause the robot's hand to deviate from the intended position, and these errors can also cause grasping failure.

[0008] A conventional technique that discloses a method for solving such problems is Patent Document 1 (JP Patent Publication 2007-319938A). Patent Document 1 discloses a configuration in which a robot is equipped with an overhead camera and a hand camera attached to a hand section that performs object grasping processing, and these two cameras are used. The hand performing the object grasping process is photographed with an overhead camera, the positional relationship between the overhead camera and the hand is determined, and the object to be grasped is recognized by the hand camera.

[0009] However, when this method is applied, the overhead camera needs to capture an image of the hand unit in addition to capturing an image of the object to be grasped, which creates the problem that the grasping range is limited to objects that are in the vicinity of the hand unit.

[0010] In addition, the configuration disclosed in Patent Document 1 stores shape data of the object to be grasped in advance in a storage unit, and recognizes the object to be grasped by using this shape data. Therefore, there is a problem that it cannot be applied to the grasping process of unknown objects for which shape data is not stored. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] JP 2007-319938 A [Patent Document 2] JP 2013-184257 A Summary of the Invention [Problem to be solved by the invention]

[0012] The present disclosure has been made in consideration of, for example, the above-mentioned problems, and aims to provide a robot control device, a robot control method, and a program that enable reliable execution of a grasping process in a configuration in which a robot is used to grasp an object even if there is no registered data on the shape of the object to be grasped. [Means for solving the problem]

[0013] A first aspect of the present disclosure is a bounding box generation unit that generates a first camera reference bounding box that encompasses a target object to be grasped included in an image captured by a first camera attached to a robot, and a second camera reference bounding box that encompasses the target object to be grasped included in an image captured by a second camera attached to the robot; a gripping position calculation unit that calculates a relative position of a target gripping position of the object to be grasped with respect to the first camera reference bounding box in the captured image of the first camera, calculates the target gripping position with respect to the second camera reference bounding box in the captured image of the second camera based on the calculated relative position, and sets the calculated position as a corrected target gripping position of the object to be grasped included in the captured image of the second camera; The robot control device includes a control information generation unit that generates control information for causing the robot hand to grasp the corrected target grasping position in the image captured by the second camera.

[0014] Furthermore, a second aspect of the present disclosure is A robot control method executed in a robot control device, A bounding box generation step in which a bounding box generation unit generates a first camera reference bounding box that encompasses a grasp target object included in an image captured by a first camera attached to the robot, and a second camera reference bounding box that encompasses the grasp target object included in an image captured by a second camera attached to the robot; a gripping position calculation step in which a gripping position calculation unit calculates a relative position of a target gripping position of the object to be grasped with respect to the first camera reference bounding box in the captured image of the first camera, calculates the target gripping position with respect to the second camera reference bounding box in the captured image of the second camera based on the calculated relative position, and sets the calculated position as a corrected target gripping position of the object to be grasped included in the captured image of the second camera; The robot control method includes a control information generating step in which a control information generating unit generates control information for causing the hand of the robot to grasp the corrected target grasping position in the image captured by the second camera.

[0015] Furthermore, a third aspect of the present disclosure is A program for causing a robot control device to execute a robot control process, a bounding box generation step of causing a bounding box generation unit to generate a first camera reference bounding box that encompasses a target object to be grasped included in an image captured by a first camera attached to the robot, and a second camera reference bounding box that encompasses the target object to be grasped included in an image captured by a second camera attached to the robot; a gripping position calculation step of causing a gripping position calculation unit to calculate a relative position of a target gripping position of the object to be grasped with respect to the first camera reference bounding box in the image captured by the first camera, calculate the target gripping position with respect to the second camera reference bounding box in the image captured by the second camera based on the calculated relative position, and set the calculated position as a corrected target gripping position of the object to be grasped included in the image captured by the second camera; The program executes a control information generating step of causing a control information generating unit to generate control information for causing the robot's hand to grasp the corrected target grasping position in the image captured by the second camera.

[0016] The program of the present disclosure is, for example, a program that can be provided by a storage medium or a communication medium in a computer-readable format to an information processing device or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or the computer system.

[0017] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. In this specification, a system refers to a logical collective configuration of multiple devices, and each device is not necessarily located in the same housing.

[0018] According to the configuration of one embodiment of the present disclosure, an apparatus and method are realized that enable a robot to reliably grasp an object. Specifically, for example, an overhead camera reference bounding box that encompasses the object to be grasped included in an image captured by an overhead camera attached to the robot, and a hand camera reference bounding box that encompasses the object to be grasped included in an image captured by a hand camera attached to the robot are generated. Furthermore, a relative position of a target gripping position of the object to be grasped with respect to the overhead camera reference bounding box in the image captured by the overhead camera is calculated, and a target gripping position with respect to the hand camera reference bounding box in the image captured by the hand camera is calculated based on the calculated relative position, and the calculated position is set as a corrected target gripping position of the object to be grasped included in the image captured by the hand camera. Furthermore, control information is generated to cause the hand of the robot to grasp the corrected target gripping position in the image captured by the hand camera, and a gripping process is performed by the robot. This configuration realizes an apparatus and method that enables a robot to reliably grasp an object. It should be noted that the effects described in this specification are merely examples and are not limiting, and additional effects may also be provided. [Brief description of the drawings]

[0019] [Figure 1]1A to 1C are diagrams illustrating an example of the operation and control processing of a robot. [Diagram 2] 1A to 1C are diagrams illustrating an example of the operation and control processing of a robot. [Diagram 3] 1A and 1B are diagrams for explaining problems with robot operation and control processing. [Figure 4] 1A to 1C are diagrams illustrating an example of solving problems in the operation and control processing of a robot. [Diagram 5] FIG. 2 is a diagram illustrating a configuration example of a robot control device according to the present disclosure. [Figure 6] FIG. 2 is a flowchart illustrating a processing sequence executed by a robot control device according to the present disclosure. [Figure 7] 11A and 11B are diagrams illustrating a specific example of a process for inputting designation information of an object to be grasped and designation information of a target grasping position. [Figure 8] 11A to 11C are diagrams illustrating a specific example of a point cloud extraction process for an object to be grasped. [Figure 9] FIG. 11 is a diagram showing a flowchart for explaining a sequence of a process for generating a bounding box using an image captured by an overhead camera, which is executed by a robot control device according to the present disclosure. [Figure 10] 11A to 11C are diagrams illustrating a specific example of a bounding box generation process. [Figure 11] 11A to 11C are diagrams illustrating a specific example of a bounding box generation process. [Figure 12] 11A to 11C are diagrams illustrating a specific example of a bounding box generation process. [Figure 13] 11A to 11C are diagrams illustrating a specific example of a bounding box generation process. [Figure 14] 11A to 11C are diagrams illustrating a specific example of a bounding box generation process. [Figure 15] 11A and 11B are diagrams illustrating a specific example of a process for calculating a relative relationship between a bounding box based on an overhead camera and a target gripping position. [Figure 16] FIG. 11 is a diagram showing a flowchart for explaining a sequence of a process for generating a bounding box by applying an image captured by a hand-end camera, which is executed by a robot control device according to the present disclosure. [Figure 17] 11A to 11C are diagrams illustrating a specific example of a calculation process of a corrected target gripping position executed by a robot control device according to the present disclosure. [Figure 18] FIG. 2 is a diagram illustrating an example of a hardware configuration of a robot control device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The robot control device, the robot control method, and the program according to the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items. 1. Overview of object grasping processing by robots 2. Problems with robot gripping 3. Configuration example of the robot control device according to the present disclosure 4. Details of the processing executed by the robot control device of the present disclosure 5. Modifications and application examples of the robot control device of the present disclosure 6. Hardware configuration example of the robot control device according to the present disclosure 7. Summary of the Disclosure

[0021] [1. Overview of object grasping processing by robots] First, an overview of object grasping processing by a robot will be described with reference to FIG. FIG. 1 is a diagram illustrating a processing sequence when a robot 10 grasps an object 50 that is a target object to be grasped. The robot 10 grasps the object 50 by performing the operations in the order of steps S01 to S03 shown in the figure.

[0022] The robot 10 has a head 20, a hand 30, and an arm 40. The hand 30 is connected to the robot body by the arm 40, and is configured such that the position, orientation, etc. of the hand 30 can be changed by controlling the arm 40. The hand 30 has rotatable movable parts on both sides that correspond to human fingers, and is configured to be capable of performing an action of gripping an object and an action of releasing an object.

[0023] The robot 10 moves by being driven by driving parts such as the legs and wheels, and further, the arm 40 is controlled to move the hand 30 to a position where the object 50 can be grasped. Alternatively, the robot body 10 may not move, and the hand 30 may be moved closer to the object by controlling only the arm 40. The process of the present disclosure is applicable to either configuration. In the embodiment described below, a configuration example in which the main body of the robot 10 is also movable will be described as an example.

[0024] The robot 10 has two cameras for confirming the position etc. of an object 50 to be grasped. One is an overhead camera 21 attached to the head 20 , and the other is a hand camera 31 attached to the hand 30 .

[0025] The overhead camera 21 and the hand camera 31 are not limited to cameras for capturing visible light images, but also include sensors capable of acquiring distance images, etc. However, it is preferable to use a camera or sensor capable of acquiring three-dimensional information. For example, a stereo camera, a ToF sensor, a Lidar sensor, or a combination of these sensors with a monocular camera may be used. It is preferable to use a camera or sensor capable of acquiring data capable of analyzing the three-dimensional position of the object to be grasped.

[0026] FIG. 1 (step S01) shows a step of confirming the position of an object 50 by the overhead camera 21. The data processing unit in the robot 10 detects the object 50, which is the object to be grasped, from the image captured by the overhead camera 21, and calculates the three-dimensional position of the object 50. After confirming the position, the data processing unit of the robot 10 moves so as to approach the object 50.

[0027] (Step S02) shows a process in which the robot 10 approaches the object 50 and moves the hand 30 to a position where the object 50 can be grasped. This control of the hand position is executed based on an analysis of an image captured by a hand camera 31 attached to the hand 30.

[0028] The data processing unit in the robot 10 detects the object 50, which is the object to be grasped, from the image captured by the hand camera 31, and calculates the three-dimensional position of the object 50. After confirming this position, the data processing unit of the robot 10 performs an adjustment process to set the position and orientation of the hand 30 to a state in which the object 50 can be grasped.

[0029] (Step S03) shows the gripping process after the adjustment process of the hand 30 in step S02. The movable parts on both sides of the hand 30 are operated to grasp the object 50.

[0030] The detailed sequence of this object grasping process will be described with reference to FIG. FIG. 2 is a diagram showing the gripping sequence of the object 50 by the robot 10 described above with reference to FIG. 1 in more detail in processing units. The process is executed in the order of steps S11 to S15 shown in FIG. Each processing step will be explained in turn below.

[0031] (Step S11) First, in step S11, a target gripping position determination process is executed. First, the image captured by the overhead camera 21 attached to the head 20 of the robot 10 is analyzed to detect the object 50 to be grasped, and the position of the object 50 is analyzed.

[0032] (Step S12) Step S12 is a trajectory planning step. Based on the position information of the object 50, which is the object to be grasped, calculated in step S11, the data processing unit of the robot 10 generates a movement path for the robot or the hand to approach the calculated position of the object 50, i.e., a trajectory plan. Note that the position of the hand 30 after movement may be anywhere as long as the object to be grasped can be observed by the hand end camera 31 attached to the hand 30.

[0033] (Step S13) Next, in step S13, the robot and the hand are moved according to the trajectory generated in step S12. As described above, the position of the hand 30 after the movement is a position where the object to be grasped can be observed by the hand end camera 31 attached to the hand 30.

[0034] (Step S14) Next, in step S14, the position and orientation of the hand 30 are finely adjusted. This hand position control is performed based on the analysis of images captured by a hand camera 31 attached to the hand 30.

[0035] The data processing unit in the robot 10 detects the object 50, which is the object to be grasped, from the image captured by the hand camera 31, and calculates the position of the object 50. After confirming this position, the data processing unit of the robot 10 performs an adjustment process to set the position and orientation of the hand 30 to a state in which the object 50 can be grasped.

[0036] (Step S15) Finally, the movable parts on both sides of the hand 30 are operated to grasp the object 50.

[0037] In this manner, the robot 10 first confirms the position of the object 50 to be grasped based on the image captured by the overhead camera 21 attached to the head 20. After that, after the hand 30 approaches the object 50, the image captured by the hand camera 31 attached to the hand 30 is analyzed, and the position and orientation of the hand 30 are fine-tuned to grasp the object 50.

[0038] [2. Problems with robot gripping] Next, a problem in the gripping process of the robot described with reference to FIGS. 1 and 2 will be described.

[0039] Problems in the gripping process of a robot will be described with reference to FIG. FIG. 3, like FIG. 1 described above, is a diagram illustrating a processing sequence when the robot 10 grasps an object 50 that is a target object to be grasped. The robot 10 grasps the object 50 by performing the steps S01 to S03 shown in the figure in that order.

[0040] Here, the difference from FIG. 1 is the shape of an object 50 to be grasped. In the configuration described with reference to FIG. 1, the object 50 to be grasped has a spherical or cylindrical shape, but the object 50 to be grasped shown in FIG. 3 has a rectangular parallelepiped shape.

[0041] FIG. 3 (step S01) shows a step of confirming the position of the object 50 by the overhead camera 21. The data processing unit in the robot 10 detects the object 50, which is the object to be grasped, from the image captured by the overhead camera 21, and calculates the three-dimensional position of the object 50. After confirming the position, the data processing unit of the robot 10 moves so as to approach the object 50.

[0042] (Step S02) shows a process in which the robot 10 approaches the object 50 and moves the hand 30 to a position where the object 50 can be grasped. This control of the hand position is executed based on an analysis of an image captured by a hand camera 31 attached to the hand 30.

[0043] The data processing unit in the robot 10 detects the object 50, which is the object to be grasped, from the image captured by the hand camera 31, and calculates the position of the object 50. After confirming this position, the data processing unit of the robot 10 performs an adjustment process to set the position and orientation of the hand 30 to a state in which the object 50 can be grasped.

[0044] (Step S03) shows the gripping process after the adjustment process of the hand 30 in step S02. An attempt is made to grasp an object 50 by operating the movable parts on both sides of the hand 30.

[0045] However, in the example shown in Fig. 3, object 50, which is the object to be grasped, has a rectangular parallelepiped shape. When grasping object 50 having such a shape, unless the orientation of hand 30 is set to a specific direction with respect to object 50 that enables stable grasping processing, object 50 may rotate in hand 30 as shown in Fig. 3 (S03), and the grasping processing may fail.

[0046] For example, if the object 50 is a container filled with water, a situation will occur in which the water spills out of the container.

[0047] FIG. 4 is a diagram showing an example of a control process of the robot 10 for stably holding the object 50 having such a rectangular parallelepiped shape.

[0048] FIG. 4, like FIG. 3, shows a process sequence for grasping an object 50, which is an object to be grasped, by the robot 10 when the object 50 has a rectangular parallelepiped shape.

[0049] 4 (S03), the difference from Fig. 3 is that the orientation of the hand 30 is set to a direction that enables stable gripping with respect to the object 50, and gripping is performed. By controlling the position and direction of the hand 30 in this way, it becomes possible to stably grip objects having various shapes.

[0050] The robot control device of the present disclosure has a configuration that enables processing as shown in FIG. 4, that is, control for stably grasping objects having various shapes. The configuration and processing of the robot control device according to the present disclosure will be described below.

[0051] [3. Configuration Example of the Robot Control Device of the Present Disclosure] Next, a configuration example of the robot control device according to the present disclosure will be described.

[0052] FIG. 5 is a block diagram showing an example of a configuration of a robot control device 100 according to the present disclosure. A robot control device 100 according to the present disclosure shown in FIG. 5 is configured, for example, inside the robot 10 shown in FIGS.

[0053] As shown in FIG. 5, the robot control device 100 of the present disclosure includes a data processing unit 110, a robot head unit 120, a robot hand unit 130, a robot movement unit 140, a communication unit 150, and an input / output unit (user terminal) 180. The input / output unit (user terminal) 180 may be located inside the robot body, or may be configured as a user terminal that is an independent device different from the robot body. Moreover, the data processing unit 110 may be located within the robot body, or may be configured within an independent device separate from the robot body.

[0054] The data processing unit 110 includes a grasp target object point group extraction unit 111 , a grasp target object bounding box generation unit 112 , a grasp position calculation unit 113 , and a control information generation unit 114 . The robot head 120 has a drive unit 121 and an overhead camera 122 . The robot hand unit 130 includes a drive unit 131 and a hand camera 132 . The robot movement unit 140 includes a drive unit 141 and a sensor 142 .

[0055] Note that the components shown in FIG. 5 are the main components applied to the processing of the present disclosure, and there are various other components inside the robot, such as a memory unit, for example.

[0056] A driving unit 121 of the robot head 120 drives the robot head 120 and controls the orientation of the robot head. Through this control, the image capturing direction of the overhead camera 122 is controlled. The overhead camera 122 captures images observed from the robot head 120 . The overhead camera 122 is not limited to a camera for capturing visible light images, and may be a sensor capable of acquiring distance images, etc. However, it is preferable to use a camera or sensor capable of acquiring three-dimensional information. For example, a stereo camera, a ToF sensor, Lidar, or other sensor, or a combination of these sensors with a monocular camera, may be used. It is preferable to use a camera or sensor capable of acquiring data capable of analyzing the three-dimensional position of the object to be grasped.

[0057] However, even with only a monocular camera, it is possible to analyze the three-dimensional position of an object in a captured image by performing analysis processing of the continuously captured images, such as SLAM (simultaneous localization and mapping) processing, and if the robot control device 100 has a configuration for performing such analysis processing, the overhead camera 122 may be a monocular camera.

[0058] Note that SLAM (simultaneous localization and mapping) processing is a process that performs self-position identification (localization) and environmental map creation (mapping) in parallel.

[0059] A driving unit 131 of the robot hand unit 130 controls the orientation of the robot hand and the gripping operation. The hand camera 132 is a camera that captures an image immediately in front of the robot hand unit 130.

[0060] The hand camera 132 is not limited to a camera for capturing visible light images, and may be a sensor capable of acquiring distance images, etc. However, it is preferable to use a camera or sensor capable of acquiring three-dimensional information. For example, a stereo camera, a ToF sensor, Lidar, or other sensor, or a combination of these sensors with a monocular camera, may be used. It is preferable to use a camera or sensor capable of acquiring data capable of analyzing the three-dimensional position of the object to be grasped.

[0061] However, similar to the overhead camera 122 described above, if the robot control device 100 has a configuration for performing analysis processing of the three-dimensional position of an object in a captured image, for example, by SLAM processing, the hand camera 132 may also be a monocular camera.

[0062] The drive unit 141 of the robot movement unit 140 is, for example, a drive unit that drives the legs and wheels of the robot, and performs drive processing for moving the robot body. The sensor 142 is a sensor for detecting obstacles in the direction of movement of the robot, and is composed of a camera, a ToF sensor, Lidar, etc.

[0063] The data processing unit 110 includes a grasp target object point group extraction unit 111 , a grasp target object bounding box generation unit 112 , a grasp position calculation unit 113 , and a control information generation unit 114 .

[0064] The grasp target object point cloud extraction unit 111 executes extraction processing of a point cloud (three-dimensional point cloud) indicating the grasp target object included in the image captured by the overhead camera 122 and the image captured by the hand-end camera 132. The point cloud corresponds to a point cloud (three-dimensional point cloud) indicating the outer shape of the object to be grasped, that is, the three-dimensional shape of the object. A specific processing example will be described later.

[0065] The grasp target object enclosing box generating unit 112 generates, based on the three-dimensional point cloud of the grasp target object generated by the grasp target object point cloud extracting unit 111, a "grasp target object enclosing box" that encloses this three-dimensional point cloud.

[0066] The "grasp target object containing box" is a box containing a group of points indicating the three-dimensional shape of the grasp target object, and the shape is not particularly limited, such as a rectangular parallelepiped, a cylinder, a cone, a torus, etc. In the following embodiment, an example will be described in which a rectangular parallelepiped containing solid (bounding box) is used as the "grasp target object containing box".

[0067] The grip position calculation unit 113 executes, for example, the following process. (1) A process of calculating a relative relationship between a bounding box of an object to be grasped in an image captured by an overhead camera and a target grasping position; (2) A process of calculating a corrected target grip position, which is a relative position of the target grip position with respect to the target object inclusion box (bounding box) of the object to be grasped in the image captured by the hand-tip camera, by applying a relative relationship between the target grip position and the target object inclusion box (bounding box) of the object to be grasped in the image captured by the overhead camera; These processes are executed.

[0068] The target gripping position is, for example, a target gripping position set by a user while viewing an image captured by an overhead camera using the input / output unit (user terminal) 180. It is a gripping position where the object to be gripped can be stably gripped by the hand of a robot, and corresponds to, for example, a contact position between the hand and the object to be gripped when the object to be gripped is gripped by the hand.

[0069] For example, when using a gripper-type hand that pinches object 50 from both sides as described with reference to Figures 1 to 4, target gripping positions are set at two locations on both sides of object 50. Specific examples and details will be described later.

[0070] The grip position calculation unit 113 is (a) A bounding box of the object to be grasped in an image captured by an overhead camera; (b) The bounding box of the object to be grasped in the image captured by the hand camera Two bounding boxes for the object to be grasped are generated for the object to be grasped in the images captured by the different cameras, and the relative positions of the bounding boxes and the grasping position are matched to calculate which position of the object to be grasped in the image captured by the hand camera corresponds to the target grasping position set by the user. This calculated position is set as the corrected target grasping position.

[0071] The control information generating unit 114 generates control information for causing the robot hand to grasp the "corrected target grasping position" calculated by the grasping position calculating unit 113. This control information is output to the driving unit 141 of the robot moving unit 140 and the driving unit 131 of the robot hand unit 130.

[0072] The drive unit 141 of the robot moving unit 140 and the drive unit 131 of the robot hand unit 130 perform drive processing in accordance with the control information generated by the control information generation unit 114, i.e., control information that causes the robot's hand to grasp the "corrected target grasping position" calculated by the grasping position calculation unit 113. This drive process enables the robot hand to grasp the "corrected target grasping position." This "corrected target grip position" is a grip position that matches the target grip position specified by the user while viewing the overhead image, and is a grip position that is set on the object to be grasped that is included in the image captured by the hand camera. By grasping the corrected target grip position that is set on the object to be grasped that is included in the image captured by the hand camera with the robot hand, it becomes possible to stably grasp the object. Note that it is assumed that the target grip position specified by the user while viewing the overhead image does not include [0] recognition errors or machine errors.

[0073] [4. Details of the processing executed by the robot control device of the present disclosure] Next, the processing executed by the robot control device 100 of the present disclosure will be described in detail.

[0074] FIG. 6 is a flowchart illustrating a process sequence for calculating the gripping position (corrected target gripping position) of the object to be gripped, which is executed by the data processing unit 110 of the robot control device 100 shown in FIG.

[0075] As described above, the "corrected target grasping position" is a grasping position at which the object to be grasped included in the image captured by the hand camera can be grasped stably, and is a grasping position that matches the target grasping position specified by the user while looking at the overhead image.

[0076] Hereinafter, the details of the processing executed by the robot control device 100 of the present disclosure will be described according to the flowchart shown in FIG. The processing according to the flow shown in FIG. 6 is processing that can be executed under the control of a control unit (data processing unit) composed of a CPU or the like having a program execution function of an information processing device, in accordance with a program stored in a storage unit (memory) of the robot control device 100. The process of each step in the flow shown in FIG. 6 will be described below.

[0077] In the flow shown in FIG. 6, the processes of steps S111 to S114 are executed based on images (including distance images) captured by the overhead camera 122 of the robot head 120. On the other hand, in the flow shown in FIG. 6, the processes of steps S121 to S123 are executed based on images (including distance images) captured by the hand camera 132 of the robot hand section .

[0078] First, the processes executed based on the images captured by the overhead camera 122 in steps S111 to S114 will be described.

[0079] (Step S111) First, the data processing unit 110 of the robot control device 100 receives as input designation information of an object to be grasped using an image captured by an overhead camera, and designation information of a target grasping position.

[0080] The user inputs designation information for the object to be grasped and designation information for the target grasping position, for example, using the input / output unit (user terminal) 180 while viewing an image captured by the overhead camera. A specific example of this process will be described with reference to FIG.

[0081] Figure 7 shows the following diagrams: (1) Example of the process for specifying an object to be grasped (2) Example of specifying the gripping position of the object to be gripped

[0082] 7(1) and (2) are both images captured by the overhead camera 122 and displayed on the display unit of the input / output unit (user terminal) 180. That is, the image captured by the overhead camera 122 is an image of a rectangular parallelepiped object to be grasped placed on a table. In this way, the user inputs designation information for the object to be grasped and designation information for the target grasping position while viewing the image captured by the overhead camera displayed on the input / output unit (user terminal) 180.

[0083] FIG. 7(1) is an example of input of designation information for an object to be grasped. For example, as shown in FIG. 7(1), the user specifies the object to be grasped by a method such as setting a rectangular area surrounding the object to be grasped, which has a rectangular parallelepiped shape.

[0084] Furthermore, as shown in FIG. 7(2), the user specifies a gripping position (target gripping position) for stably gripping the object to be grasped. As a method for specifying the gripping position, as shown in the figure, there is a method of directly specifying the gripping position on the surface of the object to be grasped, and a method of setting an arrow indicating the gripping position. In the example shown in FIG. 7(2), the gripping positions on the surface of a rectangular parallelepiped object to be gripped are to be set at approximately the center positions of two opposing faces on both sides.

[0085] Of the two opposing surfaces on both sides of the object to be grasped, one surface is located in a position that can be observed from the image captured by the overhead camera, and this point allows the grasping position on the surface of the object to be grasped to be directly specified. However, one of the faces is in a position that cannot be seen in the image captured by the overhead camera. In such a case, the user sets an arrow indicating the gripping position as shown in the figure. In addition, a method may be applied in which a three-dimensional image of the object to be grasped is displayed on the display unit, and a marker whose position information can be interactively set by the user is displayed on the display data, and the user moves the marker to directly specify the grasping position.

[0086] When a user directly specifies a gripping position on the surface of an object to be grasped, the data processing unit 110 of the robot control device 100 determines this specified position as the target gripping position and stores this position information (a relative position with respect to the object to be grasped, or a three-dimensional position of the target gripping position) in a memory unit. In addition, if the user does not directly specify a gripping position on the surface of the object to be grasped but instead sets an arrow, the data processing unit 110 of the robot control device 100 calculates the intersection point between the arrow set by the user and the object to be grasped, determines this intersection point as the target gripping position, and stores this position information (the relative position with respect to the object to be grasped, or the three-dimensional position of the target gripping position) in the memory unit.

[0087] The gripping positions are points at which the object to be grasped can be stably grasped and lifted by the robot's hand, and the number of gripping positions varies depending on the configuration of the robot's hand. In this embodiment, as described with reference to Figs. 1 to 4, the robot's hand is of a gripper type having two movable parts that can rotate left and right. In this gripper type hand configuration, the two movable parts grasp the object to be grasped by sandwiching it from the left and right, so that the two points at which each of the two movable parts on the left and right comes into contact with the object to be grasped can be set as the gripping positions.

[0088] For example, if the robot's hand has three fingers, the three points where each finger comes into contact with the object to be grasped are designated as the grasping positions.

[0089] As shown in Figures 7(1) and (2), when a user inputs designation information for an object to be grasped and designation information for a target grasping position while viewing an image captured by an overhead camera displayed on an input / output unit (user terminal) 180, a data processing unit 110 of the robot control device 100 determines an object to be grasped and a target grasping position on the object to be grasped based on this input information.

[0090] (Step S112) Next, the process of step S112 in the flow of FIG. 6 will be described. In step S112, a point cloud extraction process of the object to be grasped in the image captured by the overhead camera is executed.

[0091] This process is executed by the grasp target object point group extraction unit 111 of the data processing unit 110 of the robot control device 100. The grasp target object point cloud extraction unit 111 executes extraction processing of a point cloud (three-dimensional point cloud) indicating the grasp target object based on the grasp target object selected from within the image captured by the overhead camera 122. The point cloud corresponds to a point cloud (three-dimensional point cloud) indicating the outer shape of the grasp target object, that is, the three-dimensional shape of the object.

[0092] An example of a three-dimensional point cloud extraction process of a grasp target object executed by the grasp target object point cloud extraction unit 111 will be described with reference to FIG. Figure 8 shows the following diagrams: (1) a target object to be grasped and information specifying the target object to be grasped; (2) Example of point cloud (3D point cloud) of the object to be grasped

[0093] FIG. 8(1) shows an object to be grasped selected from the image captured by the overhead camera 122 and a rectangular area as designation information of the object to be grasped designated by the user. The grasp target object point cloud extraction unit 111 determines an object in a specified rectangular area as a grasp target object, and extracts a point cloud corresponding to the object.

[0094] Incidentally, initially, point clouds are generated for all object regions in the image. Therefore, the grasp target object point cloud extraction unit 111 needs to perform a process of removing point clouds other than the point cloud related to the grasp target object included in the rectangular region specified by the user.

[0095] For example, if there is a point cloud corresponding to an object on a support plane (such as a table) of the object to be grasped, the point cloud corresponding to the support plane (table) is removed. As a method for removing point clouds corresponding to objects other than the object to be grasped and extracting only the point clouds corresponding to the object to be grasped, for example, a clustering process that classifies point clouds for individual objects is effective.

[0096] The point cloud is divided into clusters for each object, and then the point cloud consisting of the cluster that contains the most clusters within a rectangular area that is the specified area of ​​the object to be grasped set by the user is extracted as the point cloud corresponding to the object to be grasped. The other point cloud clusters are point clouds of objects other than the object to be grasped, so they are deleted. By performing such processing, it is possible to extract a point cloud (three-dimensional point cloud) of the object to be grasped, for example, as shown in FIG. 8(2).

[0097] For the detection process of the support plane such as a table on which the object to be grasped is placed, an existing method such as the RANSAC method can be applied. For the clustering, an existing method such as Euclidean Clustering can be applied.

[0098] (Step S113) Next, the process of step S113 in the flow of FIG. 6 will be described. In step S113, a process of generating a bounding box of the object to be grasped in the image captured by the overhead camera is performed.

[0099] This process is executed by the grasp target object enclosing box generating unit 112 of the data processing unit 110 of the robot control device 100.

[0100] The grasp target object enclosing box generating unit 112 generates, based on the three-dimensional point cloud of the grasp target object generated by the grasp target object point cloud extracting unit 111, a "grasp target object enclosing box" that encloses this three-dimensional point cloud. As described above, the shape of the "grasp target object containing box" is not particularly limited, and it can be various shapes such as a rectangular parallelepiped, a cylinder, a cone, a torus, etc. However, in this embodiment, an example will be described in which a bounding box having a rectangular parallelepiped shape is used as the "grasp target object containing box".

[0101] The detailed sequence of the process of step S113, that is, the process of generating a bounding box (bounding box) of the object to be grasped in the image captured by the overhead camera executed by the object to be grasped bounding box generating unit 112, will be described with reference to the flow shown in FIG. 9 and FIG. 10.

[0102] The process of each step in the flow shown in FIG. 9 will be described. (Step S201) First, in step S201, the grasp target object bounding box generating unit 112 inputs the following pieces of information: (a) Point cloud of objects to be grasped based on an overhead camera (b) Target gripping position

[0103] (a) The graspable object point cloud based on the overhead camera is point cloud data generated by the graspable object point cloud extraction unit 111 and is input from the graspable object point cloud extraction unit 111. (b) The target grip position is the target grip position input by the user, that is, the target grip position input by the user in step S111 of the flow in FIG.

[0104] (Step S302) Next, in step S302, the grasp target object bounding box generating unit 112 performs processing to set one side of the bounding box (bounding box) parallel to a vertical plane (yz plane) perpendicular to the approach direction (x direction) of the target grasp position.

[0105] This process will be described with reference to FIG. FIG. 10(1) shows an example of a coordinate system and input information in the process of generating a bounding box.

[0106] In the coordinate system, the approach direction in which the hand 30 approaches the object 50 to be grasped is defined as the x-direction, and the movement direction perpendicular to the x-direction for moving the movable part of the hand 30 closer to the target grasp position during the grasping process of the hand 30 is defined as the y-axis. Furthermore, this is a right-handed coordinate system in which the direction perpendicular to the x-axis and y-axis is set as the z-axis.

[0107] In step S202, a process is performed in which one side of a bounding box is set parallel to a vertical plane (yz plane) perpendicular to the approach direction (x direction) of the target grip position.

[0108] A specific example is shown in Fig. 10(2). As shown in Fig. 10(2), an edge parallel to a vertical plane (yz plane) perpendicular to the approach direction (x direction) of the target grip position is set as one side of the containing box (bounding box).

[0109] By setting these four sides, the faces of the bounding box in a rectangular parallelepiped shape are defined. However, as a bounding box with sides parallel to the yz plane, various bounding boxes can be generated, for example, as shown in Figure 11 (2a) and (2b).

[0110] FIG. 11(2a) is an example of an undesirable bounding box generation, and FIG. 11(2b) is an example of a desirable bounding box generation. In step S202, the grasp target object bounding box generating unit 112 further sets one surface of the bounding box so as to face directly against the approach direction (x direction) of the hand 30, as shown in FIG. 11(2b). That is, the rotation (yaw angle) around the z-axis of a bounding box having sides parallel to the yz plane is adjusted so that one face of the bounding box faces directly toward the approach direction (x-direction) of the hand 30.

[0111] (Step S203) Next, in step S203, the grasp target object bounding box generating unit 112 determines whether or not a supporting plane of the grasp target object exists. The support plane is, for example, a plane such as a table on which an object to be grasped is placed.

[0112] If a supporting plane for the object to be grasped exists, the process proceeds to step S204. On the other hand, if a supporting plane for the object to be grasped does not exist, the process proceeds to step S211.

[0113] (Step S204) If it is determined in step S203 that a supporting plane for the object to be grasped exists, the process proceeds to step S204. In step S204, the grasp target object bounding box generating unit 112 generates a bounding box by setting one face of the bounding box on the support plane.

[0114] The process of step S204 will be described with reference to Fig. 12. The example shown in Fig. 12(3a) shows a state in which the object to be grasped is placed on a table, which is a supporting plane.

[0115] In step S204, the graspable object bounding box generating unit 112 sets one surface of the bounding box (bounding box) on the supporting plane (table) as shown in FIG. 12(3a).

[0116] Furthermore, a bounding box is generated by connecting the surface set on the support plane (table) to the edge previously generated in step S202, i.e., the edge set parallel to the vertical plane (yz plane) perpendicular to the approach direction (x direction) of the target grasping position described with reference to Figures 10(2) and 11(2b). As a result, a bounding box such as that shown in FIG. 13(3b) is generated.

[0117] (Step S211) On the other hand, if it is determined in step S203 that a supporting plane for the object to be grasped does not exist, the process proceeds to step S211. In step S211, the grasp target object bounding box generation unit 112 projects the grasp target object point group onto a vertical plane (zx plane) parallel to the approach direction (y direction) of the target grasping position, and sets this projection plane as the constituent plane of the bounding box.

[0118] A specific example of this process will be described with reference to FIG. As shown in Figure 14 (4), the grasp target object bounding box generation unit 112 projects the grasp target object point cloud onto a vertical plane (zx plane) parallel to the approach direction (y direction) of the target grasping position, and uses this projection plane as the constituent surface of the bounding box. The projection plane generated by this projection process is the "projection plane of the grasp target object point group onto the xz plane" shown in FIG. 14(4).

[0119] (Step S212) Next, in step S212, the grasp target object bounding box generating unit 112 executes a two-dimensional principal component analysis on the projected point group to determine the orientation of the bounding box around the pitch axis (y axis).

[0120] The point cloud of the object to be grasped projected onto the xz plane is a point cloud obtained by projecting onto a two-dimensional plane (xz plane) a point cloud that originally spreads in a three-dimensional space of the object to be grasped having a three-dimensional shape. A two-dimensional principal component analysis is performed on the point cloud developed on the two-dimensional plane to determine a bounding box that contains the three-dimensional object to be grasped. Specifically, the orientation of the bounding box around the pitch axis (y-axis) is determined by performing a two-dimensional principal component analysis on the projected point cloud.

[0121] This process makes it possible to generate a bounding box, such as that shown in FIG. 14(5). Instead of the two-dimensional principal component analysis on the projected point group, three-axis principal component analysis may be directly applied.

[0122] In addition, since three-dimensional position information of the target gripping position is available, a more accurate bounding box generation process can be achieved by generating a bounding box so that the three-dimensional position of the target gripping position is contained within the bounding box.

[0123] So far, the details of the process of step S113 in the flow shown in FIG. 6, that is, the process of generating the bounding box (bounding box) of the object to be grasped in the image captured by the overhead camera by the object to be grasped bounding box generating unit 112, have been described.

[0124] As described above, the grasp target object containing box generation unit 112 generates a "grasp target object containing box (bounding box)" that contains the three-dimensional point cloud of the grasp target object generated by the grasp target object point cloud extraction unit 111 based on the three-dimensional point cloud of the grasp target object.

[0125] Next, the process of step S114 in the flow of FIG. 6 will be described. (Step S114) In step S114, a calculation process is executed for the relative positional relationship between a bounding box of the object to be grasped in the image captured by the overhead camera and the target grasping position.

[0126] This process is executed by the grip position calculation unit 113 of the data processing unit 110 of the robot control device 100.

[0127] The gripping position calculation unit 113 executes a process of calculating the relative positional relationship between a gripping target object containing box (bounding box) of the gripping target object in the image captured by the overhead camera and a target gripping position.

[0128] As described above, the target grasping position is a grasping position set by the user while viewing an image captured by an overhead camera using, for example, the input / output unit (user terminal) 180, and is a grasping position that the user judges to be able to stably grasp the object to be grasped by the robot's hand.

[0129] The process of step S114 will be described with reference to FIG. FIG. 15 shows object 50, which is an object to be grasped, and overhead camera reference bounding box (bounding box) 201 that encompasses object 50 generated by object to be grasped bounding box generation unit 112 in step S113 of the flow shown in FIG.

[0130] The overhead camera reference bounding box 201 is a bounding box generated based on an image captured by the overhead camera 122.

[0131] The grip position calculation unit 113 generates a coordinate system (overhead camera reference bounding box coordinate system) with one vertex of the overhead camera reference bounding box (bounding box) 201 as the origin. As shown in Figure 15, the overhead camera reference bounding box coordinate system is a coordinate system in which one vertex of the bounding box 201 is set as the origin (O(bb1)) and each side of the overhead camera reference bounding box (bounding box) 201, which has a rectangular parallelepiped shape, is set on the X, Y, and Z axes.

[0132] The overhead camera reference bounding box (bounding box) 201 is defined as a rectangular parallelepiped having four vertices: the origin (O(bb1)) on the overhead camera reference bounding box coordinate system, a point (X(bb1)) on the X-axis, a point (Y(bb1)) on the Y-axis, and a point (Z(bb1)) on the Z-axis.

[0133] Fig. 15 further shows three-dimensional position coordinates of the target gripping position in the bird's-eye camera reference bounding box coordinate system. These are the following two points shown in Fig. 15. Target gripping position L((X(L1),Y(L1),Z(L1)),211L Target gripping position R((X(R1),Y(R1),Z(R1)),211R These are the two points.

[0134] As described above, the target gripping position is a target gripping position set by the user using the input / output unit (user terminal) 180 while viewing an image captured by an overhead camera, for example.

[0135] In step S114, the grip position calculation unit 113 calculates the target grip position set by the user as a three-dimensional position on the coordinate system (bird's-eye camera reference bounding box coordinate system) shown in Fig. 15. That is, it is the three-dimensional position coordinates of the following two points shown in Fig. 15. Target gripping position L((X(L1),Y(L1),Z(L1)),211L Target gripping position R((X(R1),Y(R1),Z(R1)),211R

[0136] The coordinates of this target gripping position are in a coordinate system in which one vertex of the bounding box is set as the origin and each side of the overhead camera reference bounding box 201, which has a rectangular parallelepiped shape, is set as the X, Y, and Z axes. Therefore, the coordinates of the target gripping position shown in FIG. 15 are coordinates that indicate the relative positional relationship between the containing box (bounding box) of the object to be gripped in the image captured by the overhead camera and the target gripping position.

[0137] In this way, in step S114, the grip position calculation unit 113 calculates the relative positional relationship between the grip target object inclusion box (bounding box) of the grip target object in the overhead camera captured image and the target grip position by calculating the three-dimensional position of the target grip position on the coordinate system (overhead camera reference bounding box coordinate system) shown in FIG. 15, that is, Target gripping position L((X(L1),Y(L1),Z(L1)),211L Target gripping position R((X(R1),Y(R1),Z(R1)),211R These calculation processes are executed.

[0138] Next, the processes in steps S121 to S123 shown in FIG. 6 will be described. The processes in steps S121 to S123 are executed based on images captured by the hand camera 132 of the robot hand section .

[0139] (Step S121) In step S121, a point cloud extraction process is executed for the object to be grasped in the image captured by the hand-end camera.

[0140] This process is executed by the grasp target object point group extraction unit 111 of the data processing unit 110 of the robot control device 100. The grasp target object point cloud extraction unit 111 extracts a point cloud (three-dimensional point cloud) indicating the grasp target object included in the image captured by the hand-end camera 132. As described above, the point cloud corresponds to a point cloud (three-dimensional point cloud) indicating the outer shape of the object to be grasped, that is, the three-dimensional shape of the object.

[0141] That is, as described above with reference to FIG. 8, a point cloud (three-dimensional point cloud) of the object to be grasped as shown in FIG. 8(2) is generated. In the process described above with reference to FIG. 8, point cloud extraction is performed using a rectangular area that specifies the object to be grasped specified by the user.

[0142] In the process of extracting an object to be grasped from an image captured by the hand camera 132, the image captured by the hand camera 132 may be displayed on the input / output unit (user terminal) 180, and the user may specify an object to be grasped and set a rectangular area to perform a similar process, but it is also possible to perform the process without the user specifying a rectangular area. In other words, the process of extracting an object to be grasped from the image captured by the hand camera 132 can be executed autonomously by referring to the shape and size of a bounding box generated based on the image captured by the overhead camera 122.

[0143] Specifically, for example, the process of extracting the object to be grasped is performed by applying Min-Cut Based Segmentation processing, which is known as a method of extracting a specific object from an image. That is, the process of extracting the object to be grasped from the image captured by the hand camera 132 is performed by applying a method such as setting the seed point and size to the point group and size included in the containing box (bounding box) based on the overhead camera and extracting the foreground by the Min-Cut Based Segmentation processing.

[0144] During this process, if necessary, processing such as detection of a support plane, such as a table on which the object to be grasped is placed, and clustering may be added. As described above, the detection process of the support plane such as a table on which the object to be grasped is placed can be performed using existing methods such as the RANSAC method, and the clustering process can be performed using existing methods such as Euclidean Clustering.

[0145] If the size of the extracted point cloud based on the hand camera 132 obtained by this process is significantly different from the containing box (bounding box) based on the overhead camera generated in step S113, it is preferable to change the parameters again and perform processing such as changing the point cloud to be extracted. By performing such processing, a point cloud (three-dimensional point cloud) of the object to be grasped, such as that shown in Figure 8 (2), can be extracted as a point cloud (three-dimensional point cloud) indicating the object to be grasped contained in the image captured by the hand camera 132.

[0146] (Step S122) Next, the process of step S122 in the flow of FIG. 6 will be described. In step S122, a process of generating a containing box (bounding box) of the object to be grasped in the image captured by the hand-end camera is performed.

[0147] This process is executed by the grasp target object enclosing box generating unit 112 of the data processing unit 110 of the robot control device 100.

[0148] The graspable object containing box generation unit 112 generates a "graspable object containing box" that contains the 3D point cloud of the graspable object in the image captured by the hand camera 132 generated by the graspable object point cloud extraction unit 111. The bounding box generated by the graspable object bounding box generation unit 112, that is, the bounding box that encompasses the graspable object in the image captured by the hand-end camera, is a bounding box of the same shape as the bounding box previously generated in step S113.

[0149] As mentioned above, the "object to be grasped containing box" is not limited to a particular shape and can be a variety of shapes, such as a rectangle, cylinder, cone, or torus, but the "object to be grasped containing box" previously generated in step S113 is a bounding box having a rectangular parallelepiped shape, and in this step S122, a bounding box having a rectangular parallelepiped shape is also generated.

[0150] The process of step S122, that is, the detailed sequence of the process of generating a bounding box (encompassing box) of the object to be grasped in the image captured by the hand-end camera executed by the graspable object bounding box generating unit 112 will be described with reference to the flow shown in FIG.

[0151] The process of each step in the flow shown in FIG. 16 will be described. (Step S301) First, in step S301, the grasp target object bounding box generating unit 112 inputs the following pieces of information: (a) Point cloud of the object to be grasped based on the hand camera (b) Enclosing box (bounding box) based on the overhead camera (c) Target gripping position

[0152] (a) The hand-end camera-based grasp target object point cloud is point cloud data generated by the grasp target object point cloud extraction unit 111 based on the image captured by the hand-end camera in step S121, and is input from the grasp target object point cloud extraction unit 111. (b) The bounding box based on the overhead camera is the bounding box generated in step S113 of the flow in FIG. 6, and is input from the grasp target object point bounding box generation unit 112. (c) The target gripping position is the target gripping position input by the user in step S111 of the flow in FIG.

[0153] (Step S302) Next, in step S302, the grasp target object bounding box generating unit 112 performs processing to set one side of the bounding box (bounding box) parallel to a vertical plane (yz plane) perpendicular to the approach direction (x direction) of the target grasp position.

[0154] This process is similar to the process in step S202 in the flow of FIG. 9 described above. That is, this is the process previously described with reference to FIGS. In step S302, a process is performed in which one side of a bounding box is set parallel to a vertical plane (yz plane) perpendicular to the approach direction (x direction) of the target grip position. Specifically, as shown in FIG. 10(2), a side parallel to a vertical plane (yz plane) perpendicular to the approach direction (x direction) of the target gripping position is set as one side of a bounding box.

[0155] By setting these four sides, the faces of the bounding box in a rectangular parallelepiped shape are defined. However, as a bounding box with sides parallel to the yz plane, various bounding boxes can be generated, for example, as shown in Figure 11 (2a) and (2b).

[0156] FIG. 11(2a) is an example of an undesirable bounding box generation, and FIG. 11(2b) is an example of a desirable bounding box generation. In step S202, the grasp target object bounding box generating unit 112 further sets one surface of the bounding box so as to face directly against the approach direction (x direction) of the hand 30, as shown in FIG. 11(2b). That is, the rotation (yaw angle) around the z-axis of a bounding box having sides parallel to the yz plane is adjusted so that one face of the bounding box faces directly toward the approach direction (x-direction) of the hand 30.

[0157] (Step S303) Next, in step S303, the grasp target object bounding box generating unit 112 determines whether or not a supporting plane of the grasp target object exists. The support plane is, for example, a plane such as a table on which an object to be grasped is placed.

[0158] If a supporting plane for the object to be grasped exists, the process proceeds to step S304. On the other hand, if a supporting plane for the object to be grasped does not exist, the process proceeds to step S311.

[0159] (Step S304) In step S303, if it is determined that a supporting plane for the object to be grasped exists, the process proceeds to step S304. In step S304, the grasp target object bounding box generating unit 112 generates a bounding box by setting one face of the bounding box on the support plane.

[0160] The process in step S304 is similar to the process in step S204 in the flow chart of FIG. 9 described above. That is, this is the process previously described with reference to FIGS. The example shown in FIG. 12(3a) shows a state in which an object to be grasped is placed on a table, which is a supporting plane.

[0161] In step S304, the graspable object bounding box generating unit 112 sets one surface of the bounding box (bounding box) on the supporting plane (table) as shown in FIG. 12(3a).

[0162] Furthermore, a bounding box is generated by connecting the surface set on the support plane (table) to the edge previously generated in step S302, i.e., the edge set parallel to the vertical plane (yz plane) perpendicular to the approach direction (x direction) of the target grasping position described with reference to Figures 10(2) and 11(2b). As a result, a bounding box such as that shown in FIG. 13(3b) is generated.

[0163] (Step S311) On the other hand, if it is determined in step S303 that a supporting plane for the object to be grasped does not exist, the process proceeds to step S311. In step S311, the graspable object bounding box generation unit 112 sets a bounding box (bounding box) having the same orientation as the bounding box (bounding box) based on the image captured by the overhead camera 122 that has already been generated as a bounding box (bounding box) based on the image captured by the hand camera 132.

[0164] That is, a bounding box having the same orientation as the bounding box based on the image captured by the overhead camera 122 generated in step S113 of the flow shown in FIG. 6 is set as the bounding box based on the image captured by the hand-end camera 132.

[0165] Next, the process of step S123 in the flow of FIG. 6 will be described. (Step S123) In step S123, the relative positional relationship between the bounding box of the object to be grasped in the image captured by the overhead camera and the target grasping position is applied to perform a calculation process of a corrected target grasping position, which is the relative position of the target grasping position with respect to the bounding box of the object to be grasped in the image captured by the hand-end camera.

[0166] In other words, the relative position of the target grasping position of the object to be grasped with respect to the containing box (bounding box) of the object to be grasped in the image captured by the overhead camera is calculated, and based on the calculated relative position, the target grasping position is calculated with respect to the hand camera reference containing box in the image captured by the hand camera, and the calculated position is set as the corrected target grasping position of the object to be grasped included in the image captured by the hand camera.

[0167] This process is executed by the grip position calculation unit 113 of the data processing unit 110 of the robot control device 100 shown in FIG.

[0168] The grip position calculation unit 113 applies the relative positional relationship between the containing box (bounding box) of the object to be grasped in the image captured by the overhead camera and the target grip position, and performs a calculation process of a corrected target grip position, which is the relative position of the target grip position with respect to the containing box (bounding box) of the object to be grasped in the image captured by the hand camera.

[0169] As described above, the target grasping position is a grasping position set by the user while viewing an image captured by an overhead camera using, for example, the input / output unit (user terminal) 180, and is a grasping position that the user judges to be able to stably grasp the object to be grasped by the robot's hand.

[0170] In step 114 of the flow shown in FIG. 6, which is a process based on the image captured by the overhead camera 122 described above with reference to FIG. 15, the relative position between the bounding box of the object to be grasped in the image captured by the overhead camera and the target grasping position is calculated. In step S123, the relative positional relationship between the bounding box of the object to be grasped in the image captured by the overhead camera and the target grasping position is used to calculate the position of the target grasping position in the image captured by the overhead camera relative to the bounding box of the object to be grasped in the image captured by the hand camera.

[0171] That is, a calculation process is executed for a corrected target grip position, which is a relative position of the target grip position with respect to a containing box (bounding box) of the object to be grasped in the image captured by the hand-end camera.

[0172] In step S123, the grip position calculation unit 113 (a) A bounding box of the object to be grasped in an image captured by the overhead camera 122; (b) A bounding box of the object to be grasped in the image captured by the hand camera 132 Two bounding boxes for the object to be grasped are generated for the object to be grasped in the images captured by the different cameras, and the relative positions of the bounding boxes and the grasping position are matched to calculate which position of the object to be grasped in the image captured by the hand camera corresponds to the target grasping position set by the user. This calculated position is set as the corrected target grasping position.

[0173] The corrected target grip position in the image captured by the hand camera 132 that is set by this process corresponds to the target grip position that the user set while looking at the image captured by the overhead camera 122. Therefore, the robot control device 100 observes the image captured by the hand camera 132 and performs a grasping process of the object 50 by abutting the gripper of the hand at a corrected target grasping position, which is the relative position of the target grasping position with respect to the containing box (bounding box) of the object to be grasped in the image captured by the hand camera, thereby making it possible to stably grasp the object 50.

[0174] The process of step S123 will be described with reference to FIG. Figure 17 shows the following two diagrams. (1) Analysis data based on overhead camera (2) Analysis data based on the hand camera

[0175] (1) The overhead camera-based analysis data is data generated based on images captured by the overhead camera 122. That is, it is data generated by the processing of steps S111 to S114 in the flow shown in FIG. 6, and corresponds to the data previously described with reference to FIG.

[0176] (2) Hand-end camera-based analysis data is data generated based on images captured by the hand-end camera 132. That is, the data is generated by the processing in steps S121 to S123 of the flow shown in FIG.

[0177] (1) The analysis data based on the overhead camera includes: (1a) an object 50 that is a target object to be grasped; (1b) In step S113 of the flow shown in FIG. 6, the grasp target object bounding box generating unit 112 generates an overhead camera reference bounding box (bounding box) 201 that encompasses the object 50; (1c) In step S111 of the flow shown in FIG. 6, the user sets the target grip positions 211L and 211R based on the image captured by the overhead camera 122. Each of these data is shown.

[0178] As explained above with reference to FIG. 15, each of these data is shown on the overhead camera reference bounding box coordinate system. As mentioned above, the overhead camera reference bounding box coordinate system is a coordinate system in which one vertex of the bounding box 201 is set as the origin (O(bb1)) and each side of the overhead camera reference bounding box (bounding box) 201, which has a rectangular parallelepiped shape, is set on the X, Y, and Z axes.

[0179] The overhead camera reference bounding box (bounding box) 201 is defined as a rectangular parallelepiped having four vertices: the origin (O(bb1)) of the overhead camera reference bounding box coordinate system, a point (X(bb1)) on the X-axis, a point (Y(bb1)) on the Y-axis, and a point (Z(bb1)) on the Z-axis.

[0180] Fig. 17(1) further shows three-dimensional position coordinates of the target gripping position in the bird's-eye camera reference bounding box coordinate system. These are the following two points shown in Fig. 17(1). Target gripping position L((X(L1),Y(L1),Z(L1)),211L Target gripping position R((X(R1),Y(R1),Z(R1)),211R These are the two points. As described above, these target gripping positions are gripping positions set by the user using the input / output unit (user terminal) 180 while viewing an image captured by an overhead camera, for example.

[0181] As previously explained with reference to Figure 15, the coordinates of the target grasping positions shown in Figure 17(1) are coordinates that indicate the relative positional relationship between the containing box (bounding box) of the object to be grasped in the image captured by the overhead camera and the target grasping position.

[0182] In step S123 of the flow in FIG. 6, the gripping position calculation unit 113 executes the following process. In other words, by applying the relative positional relationship between the containing box (bounding box) of the object to be grasped in the image captured by the overhead camera and the target grasping position, a calculation process is performed for the corrected target grasping position, which is the relative position of the target grasping position to the containing box (bounding box) of the object to be grasped in the image captured by the hand camera.

[0183] Specifically, the relative position of the target grasping position of the object to be grasped relative to the containing box (bounding box) of the object to be grasped in the image captured by the overhead camera is calculated, and based on the calculated relative position, the target grasping position is calculated relative to the hand camera reference containing box in the image captured by the hand camera, and the calculated position is set as the corrected target grasping position of the object to be grasped included in the image captured by the hand camera.

[0184] The corrected target gripping position is the corrected target gripping position shown in FIG. That is, the corrected target grip position shown in FIG. 17(2) Correction target gripping position L((X(L2),Y(L2),Z(L2)),231L Correction target gripping position R((X(R2),Y(R2),Z(R2)),231R These calculation processes are executed.

[0185] In Figure 17(2), (2a) an object 50 that is a target object to be grasped; (2b) In step S122 of the flow shown in FIG. 6, the hand camera reference bounding box (bounding box) 221 that encompasses the object 50 generated by the grasp target object bounding box generation unit 112; (2c) Corrected target gripping positions 231L, 231R, Each of these data is shown.

[0186] Each of these data is shown on the hand camera reference bounding box coordinate system. The hand camera reference bounding box coordinate system is a coordinate system in which one vertex of the hand camera reference bounding box (bounding box) 221 is set as the origin (O(bb2)) and each side of the hand camera reference bounding box (bounding box) 221, which has a rectangular parallelepiped shape, is set on the X, Y, and Z axes.

[0187] The hand camera reference bounding box (bounding box) 221 is defined as a rectangular parallelepiped having four vertices: the origin (O(bb1)) on the hand camera reference bounding box coordinate system, a point (X(bb1)) on the X axis, a point (Y(bb1)) on the Y axis, and a point (Z(bb1)) on the Z axis.

[0188] In step S123 of the flow in FIG. 6, the grip position calculation unit 113 applies the relative positional relationship between the encompassing box (bounding box) of the object to be grasped in the image captured by the overhead camera and the target grip position to perform a calculation process of a corrected target grip position, which is the relative position of the target grip position with respect to the encompassing box (bounding box) of the object to be grasped in the image captured by the hand-end camera.

[0189] That is, as shown in FIG. Correction target gripping position L((X(L2),Y(L2),Z(L2)),231L Correction target gripping position R((X(R2),Y(R2),Z(R2)),231R These calculation processes are executed.

[0190] The grip position calculation unit 113 executes the process of calculating the corrected target grip position as follows. First, the target grip position in the bird's-eye camera-based bounding box coordinate system included in the analysis data based on the bird's-eye camera shown in FIG. 17(1), that is, Target gripping position L((X(L1),Y(L1),Z(L1)),211L Target gripping position R((X(R1),Y(R1),Z(R1)),211R A relational expression indicating the coordinates of these two points is generated using the vertex data (X(bb1), Y(bb1), Z(bb1)) of the overhead camera reference bounding box 201.

[0191] That is, the following relational expressions (Relational Expression 1) and (Relational Expression 2) are generated. Target gripping position L((X(L1),Y(L1),Z(L1)) =((lx)·(X(bb1)),(ly)·(Y(bb1)),(lz)·(Z(bb1)))···(Equation 1) Target gripping position R((X(R1),Y(R1),Z(R1)) =((rx)·(X(bb1)),(ry)·(Y(bb1)),(rz)·(Z(bb1)))···(Equation 2) Two such equations (Equation 1) and (Equation 2) are generated.

[0192] In addition, lx, ly, and lz shown in (Relationship Equation 1) are coefficients that indicate the ratio of each of the x, y, and z coordinate positions of the coordinates ((X(L1), Y(L1), Z(L1))) of the target grasping position L to the length of each side of the overhead camera reference bounding box (201). Similarly, rx, ry, and rz in (Relationship Equation 2) are coefficients indicating the ratio of each of the x, y, and z coordinate positions of the coordinates ((X(R1), Y(R1), Z(R1)) of the target grasping position R to the length of each side of the overhead camera reference bounding box 201.

[0193] Based on these two relational expressions (Relational Expression 1) and (Relational Expression 2), the coefficients lx, ly, and lz and the coefficients rx, ry, and rz are calculated.

[0194] Next, the lengths of each side of the hand camera reference bounding box 221 (X(bb2), Y(bb2), Z(bb2)) shown in the analysis data based on the hand camera reference shown in FIG. 17(2) are multiplied by the coefficients (lx, ly, lz, rx, ry, rz) calculated based on the above Relational Expressions 1 and 2, Correction target gripping position L((X(L2),Y(L2),Z(L2)),231L Correction target gripping position R((X(R2),Y(R2),Z(R2)),231R These corrected target gripping positions L and R are calculated.

[0195] That is, the corrected target gripping positions L and R are calculated using the following calculation formulas (Calculation Formula 1) and (Calculation Formula 2). Correction target gripping position L((X(L2),Y(L2),Z(L2)) =((lx)·(X(bb2)),(ly)·(Y(bb2)),(lz)·(Z(bb2)))···(Formula 1) Corrected target gripping position R((X(R2),Y(R2),Z(R2)) =((rx)·(X(bb2)),(ry)·(Y(bb2)),(rz)·(Z(bb2)))·(Formula 2) The corrected target gripping positions L and R are calculated using these two calculation formulas (Calculation Formula 1) and (Calculation Formula 2).

[0196] By the above-mentioned processing, in step S123 of the flow in FIG. 6, the grip position calculation unit 113 executes a calculation process of a corrected target grip position, which is the relative position of the target grip position with respect to the containing box (bounding box) of the object to be grasped in the image captured by the hand-end camera.

[0197] The corrected target grip position in the image captured by the hand camera 132 that is set by this process corresponds to the target grip position that the user set while looking at the image captured by the overhead camera 122. Therefore, the robot control device 100 observes the image captured by the hand camera 132 and performs a grasping process of the object 50 by abutting the gripper of the hand at a corrected target grasping position, which is the relative position of the target grasping position with respect to the containing box (bounding box) of the object to be grasped in the image captured by the hand camera, thereby making it possible to stably grasp the object 50.

[0198] As described above, the robot control device 100 of the present disclosure has the following features: (a) A bounding box of the object to be grasped in an image captured by an overhead camera; (b) The bounding box of the object to be grasped in the image captured by the hand camera Two bounding boxes for the object to be grasped are generated for the object to be grasped in the images captured by the different cameras, and the relative positions of the bounding boxes and the grasping position are matched to calculate which position of the object to be grasped in the image captured by the hand camera corresponds to the target grasping position set by the user. This calculated position is set as the corrected target grasping position.

[0199] The control information generating unit 114 generates control information for causing the hand of the robot to grasp the "corrected target grasping position" calculated by the grasping position calculating unit 113. This control information is output to the driving unit 141 of the robot moving unit 140 and the driving unit 131 of the robot hand unit 130.

[0200] The drive unit 141 of the robot moving unit 140 and the drive unit 131 of the robot hand unit 130 perform drive processing in accordance with the control information generated by the control information generation unit 114, i.e., control information that causes the robot's hand to grasp the "corrected target grasping position" calculated by the grasping position calculation unit 113. This drive process enables the robot hand to grasp the "corrected target grasping position." This "corrected target grip position" is a grip position that coincides with the target grip position specified by the user while viewing the overhead image, and is a grip position that is set on the object to be grasped that is included in the image captured by the hand camera. By grasping the corrected target grip position that is set on the object to be grasped that is included in the image captured by the hand camera with the hand of the robot, it becomes possible to stably grasp the object.

[0201] [5. Modifications and Applications of the Robot Control Device of the Present Disclosure] Next, modifications and applications of the robot control device of the present disclosure described above will be described.

[0202] Modifications and application examples of each of the following items will be described in order. (1) Regarding the processing steps shown in Figure 6 (2) Regarding the process of step S111 in the flow shown in FIG. 6 (3) Regarding the process of step S112 in the flow shown in FIG. 6 (4) Regarding the process from step S113 onward in the flow shown in FIG. 6 (5) Regarding the processing in steps S114 and S123 in the flow shown in FIG. 6

[0203] (1) Regarding the processing steps shown in Figure 6 As described above, in the flow shown in FIG. 6, the processes of steps S111 to S114 are executed based on images (including distance images) captured by the overhead camera 122 of the robot head 120. On the other hand, in the flow shown in FIG. 6, the processes of steps S121 to S123 are executed based on images (including distance images) captured by the hand camera 132 of the robot hand section .

[0204] During these processes, the processes of steps S111 to S114 executed based on the images (including distance images) captured by the overhead camera 122 and the processes of steps S121 to S122 executed based on the images (including distance images) captured by the hand-held camera 132 can be executed in parallel. After the processes of steps S111 to S114 are completed, the processes of steps S121 to S122 may be executed.

[0205] However, the processing of step S123 is executed after the processing of steps S111 to S114, which are executed based on the images captured by the overhead camera 122 (including distance images), and the processing of steps S121 to S122, which are executed based on the images captured by the hand-end camera 132 (including distance images), are completed.

[0206] It is preferable to set the processing procedure so that it is performed at a timing when the object 50 to be grasped can be reliably observed in the images captured by the overhead camera 122 and the hand camera 132. By controlling the processing procedure, it is possible to avoid processing in a state where an occlusion occurs on the object to be grasped, for example, when a part such as an arm or a hand blocks the view of the overhead camera 122.

[0207] (2) Regarding the process of step S111 in the flow shown in FIG. 6 In step S111 of the flow shown in FIG. 6, processing is performed to input designation information of an object to be grasped using an image captured by an overhead camera and designation information of a target grasping position.

[0208] That is, as explained above with reference to Figure 7, the user uses the input / output unit (user terminal) 180 to input designation information for the object to be grasped and designation information for the target grasping position while viewing the image captured by the overhead camera.

[0209] When specifying an object to be grasped, a rectangular area surrounding the object is input, but this is just one example. It is also possible to perform various object designation processes, such as touching part of the object, or as described in Patent Document 2 (JP Patent Publication No. 2013-184257), a previous patent application filed by the applicant.

[0210] In addition, after object detection using deep learning, which is a machine learning process (R-CNN, YOLO, SSD, etc.), a process may be performed in which the user selects a rectangle that corresponds to the object. Alternatively, a method may be applied in which objects are extracted pixel by pixel using semantic segmentation and the user selects the object. Furthermore, in a case where the target gripping position has already been determined, a process may be performed to automatically select the object closest to the target gripping position. As for the method of determining the target gripping position, instead of the user directly determining the position and orientation, the target gripping position may be determined autonomously by designating only the object to be grasped and making a grasp plan.

[0211] (3) Regarding the process of step S112 in the flow shown in FIG. 6 In step S112 of the flow shown in FIG. 6, a process of extracting a point cloud of the object to be grasped in the image captured by the overhead camera is performed.

[0212] As described above with reference to FIG. 8, this process is executed as a process for extracting an object point group corresponding to a rectangular area designated by the user. This point cloud extraction process may also be performed by applying foreground extraction such as Min-cut based segmentation, similar to the process of specifying the object to be grasped in step S111.

[0213] In this case, the foreground can be extracted based on a point that represents the point cloud of the object indicated by the user, or a point that corresponds to the center of a rectangular area, and a predetermined rough estimate of the size of the object being held, making it possible to roughly remove irrelevant point clouds.

[0214] (4) Regarding the process from step S113 onward in the flow shown in FIG. 6 In the processing from step S113 onwards in the flow shown in Figure 6, an example has been described in which the shape of the robot's hand is a gripper-type hand. However, for other types of hands, such as a multi-fingered hand with three or more fingers or a suction hand, by defining a representative point of the hand according to the hand shape, it is possible to calculate a corrected target gripping position corresponding to the target gripping position and execute stable gripping processing, as in the above-mentioned example.

[0215] For example, in the case of a five-fingered hand, five target grip positions are prepared for the contact points of each finger, and the relative positions of all the points between the image captured by the overhead camera and the bounding box set for each image captured by the hand camera are calculated. By performing such processing, it is possible to calculate corrected target grip positions corresponding to the target grip positions corresponding to the contact points of each finger of the five-fingered hand.

[0216] (5) Regarding the processing in steps S114 and S123 in the flow shown in FIG. 6 In steps S114 and S123 of the flow shown in FIG. 6, as previously explained with reference to FIG. 15 and FIG. 17, the process of calculating the relative positional relationship between the bounding box of the object to be grasped and the grasping position is performed by calculating the relative positional relationship in all of the x, y, and z coordinates.

[0217] In addition, in step S123, the relative positional relationship between the containing box (bounding box) of the object to be grasped in the image captured by the overhead camera and the target grasping position is applied to perform a calculation process of a corrected target grasping position, which is the relative position of the target grasping position with respect to the containing box (bounding box) of the object to be grasped in the image captured by the hand camera.

[0218] However, for example, if there is a component whose bounding box shape differs significantly due to differences in the viewpoints of the overhead camera and the hand camera, the calculation process of the corrected target gripping position may not be performed for that component, and only the component whose bounding box shape is similar may be calculated. For example, if it is desired to correct the deviation in the y direction in order to place the object in the hand so that the gripper can pinch the object, it is sufficient to calculate only the y component. In addition, if it is desired to reflect a user's instruction to hold the object as high as possible as possible, the z component may be preferentially calculated.

[0219] [6. Hardware configuration example of the robot control device disclosed herein] Next, an example of a hardware configuration of the robot control device according to the present disclosure will be described.

[0220] The robot control device explained in the above embodiment can be configured as a device separate from the robot itself, or can be configured as a device within the robot. The robot control device can also be realized by utilizing an information processing device such as a PC. An example of the configuration of an information processing device that constitutes a robot control device according to the present disclosure will be described with reference to FIG.

[0221] A CPU (Central Processing Unit) 301 functions as a control unit or data processing unit that executes various processes according to programs stored in a ROM (Read Only Memory) 302 or a storage unit 308. For example, it executes processes according to the sequences described in the above-mentioned embodiments. A RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304.

[0222] CPU 301 is connected to input / output interface 305 via bus 304, and input / output interface 305 is connected to input unit 306 including various switches, a keyboard, a mouse, a microphone, a sensor, etc., and output unit 307 including a display, a speaker, etc. CPU 301 executes various processes in response to commands input from input unit 306, and outputs the process results to output unit 307, for example.

[0223] A storage unit 308 connected to the input / output interface 305 is formed of, for example, a hard disk, and stores various data and programs executed by the CPU 301. A communication unit 309 functions as a transmission / reception unit for Wi-Fi communication, Bluetooth (registered trademark) (BT) communication, and other data communication via a network such as the Internet or a local area network, and communicates with external devices.

[0224] A drive 310 connected to the input / output interface 305 drives removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, and executes recording or reading of data.

[0225] [7. Summary of the configuration of this disclosure] The embodiments of the present disclosure have been described in detail above with reference to specific embodiments. However, it is obvious that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of an example, and should not be interpreted as being limited. In order to judge the gist of the present disclosure, the claims should be taken into consideration.

[0226] The technology disclosed in this specification can have the following configurations. (1) a bounding box generation unit that generates a first camera reference bounding box that encompasses a target object to be grasped included in an image captured by a first camera attached to a robot, and a second camera reference bounding box that encompasses the target object to be grasped included in an image captured by a second camera attached to the robot; a gripping position calculation unit that calculates a relative position of a target gripping position of the object to be grasped with respect to the first camera reference bounding box in the captured image of the first camera, calculates the target gripping position with respect to the second camera reference bounding box in the captured image of the second camera based on the calculated relative position, and sets the calculated position as a corrected target gripping position of the object to be grasped included in the captured image of the second camera; a robot control device having a control information generation unit that generates control information for causing a hand of the robot to grasp the corrected target grasping position in the image captured by the second camera.

[0227] (2) the first camera is an overhead camera that captures an overhead image; The robot control device according to (1), wherein the second camera is a hand camera that captures an image from the hand performing the gripping process of the object to be gripped or from a position close to the hand.

[0228] (3) The robot control device according to (2), wherein the first camera is an overhead camera attached to a head of the robot and captures an overhead image from the head.

[0229] (4) The target gripping position is The robot control device according to any one of (1) to (3), wherein the gripping position is designated by a user by looking at an image on a display unit that displays an image captured by the first camera.

[0230] (5) The target gripping position is The robot control device according to (4), wherein the grasping position is determined by a user viewing an image on a display unit showing an image captured by the first camera as a position where the object to be grasped can be grasped stably.

[0231] (6) The robot control device further comprises: The robot control device according to any one of (1) to (5), further comprising a point cloud extraction unit that executes an extraction process of a three-dimensional point cloud indicating the object to be grasped that is included in the image captured by the first camera and the image captured by the second camera.

[0232] (7) The bounding box generation unit: The robot control device according to (6), wherein the point cloud extraction unit generates a bounding box that encompasses the generated three-dimensional point cloud.

[0233] (8) The bounding box generation unit: The robot control device according to (6) or (7), wherein the point cloud extraction unit generates a bounding box that is a rectangular parallelepiped containing box that contains the generated 3D point cloud.

[0234] (9) The bounding box generation unit: A robot control device according to any one of (1) to (8), which generates the first camera reference bounding box in an image captured by the first camera and the second camera reference bounding box in an image captured by the second camera as bounding boxes of the same shape.

[0235] (10) The bounding box generation unit The robot control device according to any one of (1) to (9), which generates a bounding box having sides parallel to a vertical plane perpendicular to an approach direction of the hand of the robot with respect to the object to be grasped.

[0236] (11) The bounding box generation unit The robot control device according to any one of (1) to (10), wherein, when a support plane supporting the object to be grasped exists, a bounding box having the support plane as a constituent plane is generated.

[0237] (12) The bounding box generation unit: A robot control device according to any one of (1) to (11), wherein when there is no support plane supporting the object to be grasped, a bounding box is generated with the projection plane generated by projecting the object to be grasped onto a vertical plane parallel to the approach direction of the robot's hand as a constituent plane.

[0238] (13) A robot control method executed in a robot control device, comprising: A bounding box generation step in which a bounding box generation unit generates a first camera reference bounding box that encompasses a grasp target object included in an image captured by a first camera attached to the robot, and a second camera reference bounding box that encompasses the grasp target object included in an image captured by a second camera attached to the robot; a gripping position calculation step in which a gripping position calculation unit calculates a relative position of a target gripping position of the object to be grasped with respect to the first camera reference bounding box in the captured image of the first camera, calculates the target gripping position with respect to the second camera reference bounding box in the captured image of the second camera based on the calculated relative position, and sets the calculated position as a corrected target gripping position of the object to be grasped included in the captured image of the second camera; A robot control method comprising: a control information generating step in which a control information generating unit generates control information for causing the robot's hand to grasp the corrected target grasping position in the image captured by the second camera.

[0239] (14) A program for causing a robot control device to execute a robot control process, a bounding box generation step of causing a bounding box generation unit to generate a first camera reference bounding box that encompasses a target object to be grasped included in an image captured by a first camera attached to the robot, and a second camera reference bounding box that encompasses the target object to be grasped included in an image captured by a second camera attached to the robot; a gripping position calculation step of causing a gripping position calculation unit to calculate a relative position of a target gripping position of the object to be grasped with respect to the first camera reference bounding box in the image captured by the first camera, calculate the target gripping position with respect to the second camera reference bounding box in the image captured by the second camera based on the calculated relative position, and set the calculated position as a corrected target gripping position of the object to be grasped included in the image captured by the second camera; A program that causes a control information generating unit to execute a control information generating step of generating control information for causing the robot hand to grasp the corrected target grasping position in the image captured by the second camera.

[0240] The series of processes described in the specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in a memory in a computer built into dedicated hardware, or the program can be installed and executed in a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program from the recording medium to the computer, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

[0241] In addition, the various processes described in the specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as necessary. In addition, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices in the same housing. [Industrial Applicability]

[0242] As described above, according to the configuration of one embodiment of the present disclosure, an apparatus and method are realized that enable a robot to reliably perform a process of grasping an object. Specifically, for example, an overhead camera reference bounding box that encompasses the object to be grasped included in an image captured by an overhead camera attached to the robot, and a hand camera reference bounding box that encompasses the object to be grasped included in an image captured by a hand camera attached to the robot are generated. Furthermore, a relative position of a target gripping position of the object to be grasped with respect to the overhead camera reference bounding box in the image captured by the overhead camera is calculated, and a target gripping position with respect to the hand camera reference bounding box in the image captured by the hand camera is calculated based on the calculated relative position, and the calculated position is set as a corrected target gripping position of the object to be grasped included in the image captured by the hand camera. Furthermore, control information is generated to cause the hand of the robot to grasp the corrected target gripping position in the image captured by the hand camera, and a gripping process is performed by the robot. This configuration realizes an apparatus and method that enables a robot to reliably grasp an object. [Explanation of symbols]

[0243] 10. Robot 20 heads 21 Bird's-eye view camera 30 Hands 31 Hand Camera 50 Object (object to be grasped) 100 Robot control device 110 Data processing section 111 Grip target object point group extraction unit 112 Grasp target object bounding box generation unit 113 Grip position calculation unit 114 Control information generation unit 120 Robot Head 121 Drive unit 122 Bird's-eye view camera 130 Robot Hand Division 131 Drive unit 132 Hand Camera 140 Robot Movement Unit 141 Drive unit 142 Sensors 201 Bird's-eye view camera reference bounding box 211 Target gripping position 221 Hand camera reference bounding box 231 Correction target gripping position 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / Output Interface 306 Input section 307 Output section 308 Storage section 309 Communications Department 310 Drive 311 Removable Media

Claims

1. a bounding box generation unit that generates a first camera reference bounding box that encompasses a target object to be grasped included in an image captured by a first camera attached to a robot, and a second camera reference bounding box that encompasses the target object to be grasped included in an image captured by a second camera attached to the robot; a gripping position calculation unit that calculates a relative position of a target gripping position of the object to be grasped with respect to the first camera reference bounding box in the captured image of the first camera, calculates the target gripping position with respect to the second camera reference bounding box in the captured image of the second camera based on the calculated relative position, and sets the calculated position as a corrected target gripping position of the object to be grasped included in the captured image of the second camera; A robot control device having a control information generation unit that generates control information for causing the robot's hand to grasp the corrected target grasping position in the image captured by the second camera.

2. The first camera is an overhead camera that captures an overhead image, The robot control device according to claim 1 , wherein the second camera is a hand camera that captures an image from the hand performing the gripping process of the object to be gripped or from a position close to the hand.

3. The robot control device according to claim 2 , wherein the first camera is an overhead camera that is attached to a head of the robot and captures an overhead image from the head.

4. The target gripping position is The robot control device according to claim 1 , wherein the gripping position is specified by a user by looking at an image on a display unit that displays the image captured by the first camera.

5. The target gripping position is The robot control device according to claim 4 , wherein the grasping position is determined by a user viewing an image on a display unit that displays the image captured by the first camera, as a position at which the object to be grasped can be grasped stably.

6. The robot control device further includes: The robot control device according to claim 1 , further comprising a point cloud extraction unit that executes a process of extracting a three-dimensional point cloud that indicates the object to be grasped and is included in the image captured by the first camera and the image captured by the second camera.

7. The bounding box generator includes: The robot control device according to claim 6 , wherein the point cloud extraction unit generates a bounding box that encompasses the generated three-dimensional point cloud.

8. The bounding box generator includes: The robot control device according to claim 6 , wherein the point cloud extraction unit generates a bounding box that is a rectangular parallelepiped containing box that contains the generated three-dimensional point cloud.

9. The bounding box generator includes: The robot control device according to claim 1 , wherein the first camera reference bounding box in the image captured by the first camera and the second camera reference bounding box in the image captured by the second camera are generated as bounding boxes of the same shape.

10. The bounding box generator includes: The robot control device according to claim 1 , wherein a bounding box is generated having sides parallel to a vertical plane perpendicular to an approach direction of the hand of the robot toward the object to be grasped.

11. The bounding box generator includes: The robot control device according to claim 1 , wherein, when a support plane that supports the object to be grasped exists, a bounding box having the support plane as a configuration plane is generated.

12. The bounding box generator includes: The robot control device according to claim 1, wherein when there is no support plane supporting the object to be grasped, a bounding box is generated with the projection plane generated by projecting the object to be grasped onto a vertical plane parallel to the approach direction of the robot's hand as a constituent plane.

13. A robot control method executed in a robot control device, a bounding box generating step in which a bounding box generating unit generates a first camera reference bounding box that encompasses a grasp target object included in an image captured by a first camera attached to the robot, and a second camera reference bounding box that encompasses the grasp target object included in an image captured by a second camera attached to the robot; a gripping position calculation step in which a gripping position calculation unit calculates a relative position of a target gripping position of the object to be grasped with respect to the first camera reference bounding box in the captured image of the first camera, calculates the target gripping position with respect to the second camera reference bounding box in the captured image of the second camera based on the calculated relative position, and sets the calculated position as a corrected target gripping position of the object to be grasped included in the captured image of the second camera; A robot control method comprising: a control information generating step in which a control information generating unit generates control information for causing the robot's hand to grasp the corrected target grasping position in the image captured by the second camera.

14. A program for causing a robot control device to execute a robot control process, a bounding box generation step of causing a bounding box generation unit to generate a first camera reference bounding box that encompasses a target object to be grasped included in an image captured by a first camera attached to the robot, and a second camera reference bounding box that encompasses the target object to be grasped included in an image captured by a second camera attached to the robot; a gripping position calculation step of causing a gripping position calculation unit to calculate a relative position of a target gripping position of the object to be grasped with respect to the first camera reference bounding box in the image captured by the first camera, calculate the target gripping position with respect to the second camera reference bounding box in the image captured by the second camera based on the calculated relative position, and set the calculated position as a corrected target gripping position of the object to be grasped included in the image captured by the second camera; A program that causes a control information generating unit to execute a control information generating step of generating control information for causing the robot's hand to grasp the corrected target grasping position in the image captured by the second camera.

Images