Robot control device, representative object information generation device, manipulation information assigning device, method, and program

Abstract

To cause a robot to manipulate objects, even when causing the robot to manipulate the objects of mutually different shapes or sizes, without preparing a three-dimensional model for each object.SOLUTION: A robot control device 14 performs: a process of making a representative shape available for each object classification, which is a classification of objects; and a process of assigning manipulation information indicating a manipulation of a robot with respect to the representative shape to a manipulation portion being a portion to be manipulated. The robot control device 14 acquires an observed shape that represents a shape of a target object being an object to be manipulated, and an object classification to which the target object belongs. The robot control device 14 assigns manipulation information associated with the representative shape that corresponds to the object classification to which the target object belongs to a portion of the observed shape that corresponds to the manipulation portion. The robot control device 14 controls the manipulation of the robot with respect to the target object on the basis of the manipulation information assigned to the observed shape.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a robot control device, a representative object information generation device, an operation information imparting device, a method, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for grasping an object to be grasped by a robot hand. This grasping method involves preparing a three-dimensional model of the object to be grasped, in which the position where the robot should grasp the object is set, determining the position and orientation of the object to be grasped by matching it with the three-dimensional model, and specifying the grasping position at that position and orientation from the three-dimensional model, thereby causing the robot to grasp the object to be grasped. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-46937 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when a robot is made to perform an operation such as gripping a target object, the target object may have a variety of shapes and sizes. However, the operating part, which is the part of the robot that performs the operation such as gripping the target object, may have a specific shape characteristic.

[0005] For example, if the target object is a cup with a handle, even if the shape or size of the handle or the shape and size of the cup itself varies, the part that the robot's hand grasps may always be the handle. Also, even if the shape or size of each part varies during assembly work in a manufacturing process, when a robot grasps parts, the operation part specified for the assembly work may be a protrusion, recess, outer side surface, or the like that is provided in common to all parts.

[0006] In the method of Patent Document 1, since the target object is matched with a 3D model, even if the shape characteristics of the operation parts of each target object are common, if the shapes or sizes of the target objects are different, a 3D model must be prepared for each target object.

[0007] Furthermore, after the robot's operation on a target object has been set and the robot is able to execute that operation, the method of operation on the target object may be changed. For example, if the target object is a cup with a handle, the operation part may be the handle and the robot may be made to grasp the handle, and then the operation part may be changed to the cup itself and the robot may be made to grasp the cup itself.

[0008] The present disclosure aims to provide a robot control device, a representative object information generation device, an operation information imparting device, a method, and a program that allow a robot to operate objects of different shapes or sizes without preparing a 3D model for each object, and that can easily change the operation after the robot is able to perform the operation. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the robot control device according to the present disclosure is a robot control device that controls the operation of a robot on an object, and includes: a representative object information generation unit that performs a process of making a representative shape available for each object classification that is a classification of the object, and a process of assigning operation information that indicates the operation of the robot on the representative shape to an operation part of the representative shape that is the part that is the target of the operation; a target object information acquisition unit that acquires an observed shape that represents a shape observed from a target object that is the object to be operated, and the object classification to which the target object belongs; an operation information assignment unit that assigns the operation information associated with the representative shape that corresponds to the object classification to which the target object belongs to a part of the observed shape that corresponds to the operation part; and a robot control unit that controls the operation of the robot on the target object based on the operation information assigned to the observed shape.

[0010] In addition, the representative object information generation device disclosed herein is a representative object information generation device that includes a representative object information generation unit that performs a process of making a representative shape available for each object classification, which is a classification of objects, and a process of assigning operation information indicating the operation of a robot on the representative shape to an operation part of the representative shape that is the part that is the target of the operation.

[0011] The operation information imparting device of the present disclosure is an operation information imparting device including: a representative object information acquisition unit that acquires information including representative shapes for each object classification that is a classification of objects, where operation information indicating an operation of a robot on the representative shapes is assigned to an operation part of the representative shapes that is the part that is the target of the operation; a target object information acquisition unit that acquires an observation shape that represents the shape of a target object that is the object to be operated and the object classification to which the target object belongs; and an operation information imparting unit that assigns the operation information associated with the representative shape that corresponds to the object classification to which the target object belongs to a part of the observation shape that corresponds to the operation part. [Effects of the Invention]

[0012] According to the robot control device, representative object information generation device, operation information imparting device, method, and program disclosed herein, even when a robot is to operate objects of different shapes or sizes, it is possible to have the robot operate the objects without preparing a 3D model for each object, and to easily change the operation after the robot is able to perform the operation. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a schematic configuration of a robot control system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the hardware configuration of the robot control device according to the present embodiment. [Figure 3] FIG. 1 is a diagram for explaining an overview of the present embodiment. [Figure 4] FIG. 2 is a diagram for explaining various types of information that can be stored in a representative object information storage unit. [Figure 5] FIG. 1 is a diagram for explaining a trained model of the present embodiment. [Figure 6] FIG. 10 is a diagram for explaining a registration process. [Figure 7] FIG. 10 is a diagram for explaining a registration process. [Figure 8] 10A and 10B are diagrams illustrating an example of transfer of operation information when the registration process is completed. [Figure 9] 10A and 10B are diagrams for explaining a process of approximating a graspable region of a target object by a combination of basic shapes such as a cylinder or a rectangular parallelepiped. [Figure 10] 10A and 10B are diagrams for explaining a process of approximating a graspable region of a target object by a combination of basic shapes such as a cylinder or a rectangular parallelepiped. [Figure 11] FIG. 10 is a diagram for explaining a provisional holding region. [Figure 12] FIG. 10 is a diagram for explaining a provisional holding region. [Figure 13] FIG. 10 is a diagram for explaining symmetry information. [Figure 14]10 is a flowchart showing the flow of representative object information generation processing in the embodiment. [Figure 15] 10 is a flowchart showing the flow of a robot control process in the embodiment. [Figure 16] FIG. 10 is a diagram showing a modified example of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of the present disclosure will be described below with reference to the drawings. In this embodiment, a robot control system equipped with a robot control device according to the present disclosure will be described as an example. Note that the same reference numerals are used in each drawing to designate the same or equivalent components and parts. Furthermore, the dimensions and proportions of the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0015] Fig. 1 is a diagram for explaining this embodiment. As shown in Fig. 1, a robot control system 10 of this embodiment includes a robot 12 and a robot control device 14. The robot 12 includes a camera 12A. The robot control device 14 of the robot control system 10 controls the robot 12 to manipulate a target object, for example, by outputting a control command to the robot 12.

[0016] Fig. 2 is a block diagram showing the hardware configuration of the robot control device 14 according to this embodiment. As shown in Fig. 2, the robot control device 14 has a CPU (Central Processing Unit) 42, memory 44, a storage device 46, an input / output I / F (Interface) 48, a storage medium reader 50, and a communication I / F 52. Each component is connected to each other via a bus 54 so as to be able to communicate with each other.

[0017] The storage device 46 stores robot control programs for executing the various processes described below. The CPU 42 is a central processing unit that executes various programs and controls each component. That is, the CPU 42 reads the programs from the storage device 46 and executes the programs using the memory 44 as a work area. The CPU 42 controls each component and performs various arithmetic operations in accordance with the programs stored in the storage device 46.

[0018] The memory 44 is made up of RAM (Random Access Memory) and serves as a working area to temporarily store programs and data. The storage device 46 is made up of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including the operating system and various data.

[0019] The input / output I / F 48 is an interface for inputting data from an external device and outputting data to an external device. Also, input devices for various inputs, such as a keyboard or a mouse, and output devices for various information outputs, such as a display or a printer, may be connected. A touch panel display may be used as the output device, allowing it to function as an input device.

[0020] The storage medium reader 50 reads data stored in various storage media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray Disc, and USB (Universal Serial Bus) memory, and writes data to the storage media.

[0021] The communication I / F 52 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0022] FIG. 3 is a diagram for explaining an overview of this embodiment. As shown in FIG. 3, the robot control device 14 of this embodiment extracts an object region, which is a region where a target object exists, from an RGB image and a range image acquired by a camera 12A, which is an example of a sensor. The robot control device 14 then estimates the three-dimensional object position and orientation of the target object (denoted as "6D object pose" in FIG. 3). Note that, when estimating the three-dimensional object position and orientation of the target object, it is possible to use, for example, the technology disclosed in Reference 1 below. In this embodiment, a case will be described where the camera 12A is attached to the robot 12, but the camera 12A may be installed in a location different from the robot 12.

[0023] Reference 1: He Wang, et.al, “Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation”, CVPR 2019.

[0024] Reference 1 discloses a technique for estimating NOCS (Normalized Object Coordinate Space), which is an example of an object coordinate system, and by using this NOCS, it is possible to estimate the three-dimensional object position and orientation of a target object. In this embodiment, an example will be described in which the NOCS disclosed in Reference 1 is used to estimate the three-dimensional object position and orientation of a target object.

[0025] Process C shown in FIG. 3 is realized, for example, by the technology disclosed in Reference 1. Specifically, as shown in FIG. 3, the robot control device 14 identifies the object category to which the target object belongs from an RGB image containing the target object and estimates a NOCS map of the target object. The NOCS map is a map in which, when the position of each point on the surface of the target object is expressed as a coordinate value in a unit cubic space (NOCS coordinate value), each pixel representing the target object is associated with the coordinate value in the unit cubic space of each point corresponding to each pixel. At this time, the position and orientation of the object in the unit cubic space are set for each category. Then, the robot control device 14 extracts an object region corresponding to the target object in the range image based on the target object's category (e.g., "Cup" or "Bottle" in FIG. 3), a mask image showing the area occupied by the target object in the RGB image, and a range image containing the target object. Then, the robot control device 14 associates the NOCS map of the target object with the object region, thereby estimating the three-dimensional object position and orientation of the target object.

[0026] Next, the robot control device 14 assigns operation information representing an operation on the object to the observed shape, which is a point cloud representing the target object. The representative object information storage unit 20 shown in Fig. 3 stores, for each category to which objects belong, representative object information, which is a 3D model of a representative object (hereinafter simply referred to as a "representative object") belonging to that category. Furthermore, the representative object information stored in the representative object information storage unit 20 is assigned operation information representing an operation part for the representative object.

[0027] The robot control device 14 associates the representative object information for each category stored in the representative object information storage unit 20 with the target object information representing the target object, thereby assigning operation information to the target object information.

[0028] For example, in the example of FIG. 3, the representative object information storage unit 20 stores a plurality of pieces of representative object information, including representative object information for a cup and representative object information for a bottle. Furthermore, each of the plurality of pieces of representative object information is assigned operation information that indicates an operation part for the representative object. For example, as shown in FIG. 3, gripping operation area information that indicates an area to be gripped by the robot and pouring operation area information that indicates an area of ​​the spout are stored in the representative object information storage unit 20 as operation information. Furthermore, the representative object information storage unit 20 also stores information regarding the relative posture between the robot's hand and the gripping operation part.

[0029] The robot control device 14 refers to various pieces of information stored in the representative object information storage unit 20 to control the robot 12 so that an operation such as grasping or moving a target object is performed.

[0030] Therefore, in this embodiment, for each category to which objects belong, representative object information, which is a 3D model of a representative object belonging to that category, is stored in the representative object information storage unit 20 together with operation information. Then, in this embodiment, by associating the point cloud of the target object that is the target of operation with the representative object information, the operation information assigned to the representative object information is transferred to the point cloud of the target object. Even when a robot is to operate objects of different shapes or sizes, it is possible to have the robot operate the objects without preparing a 3D model for each object. This will be explained in detail below.

[0031] Next, the functional configuration of the robot control device 14 will be described. As shown in Fig. 1, the robot control device 14 functionally includes a representative object information generation unit 18, a representative object information storage unit 20, a target object information acquisition unit 22, an operation information assignment unit 24, and a robot control unit 26. The representative object information storage unit 20 is also provided in a predetermined storage area of ​​the robot control device 14. Each functional configuration is realized by the CPU 42 reading out each program stored in the storage device 46, expanding it into the memory 44, and executing it.

[0032] The representative object information generation unit 18 performs a process of making a representative shape available for each category, which is a classification of objects. The representative object information generation unit 18 also performs a process of assigning operation information, which indicates an operation of the robot 12 on the representative shape, to an operation part, which is a part of the representative shape that is the target of the operation. The categories in the present disclosure are an example of object classification.

[0033] Specifically, when targeting multiple categories, the representative object information generation unit 18 stores representative object information, which is a three-dimensional model representing a representative shape that is the shape of a representative object belonging to each of the multiple categories, in the representative object information storage unit 20 described later, thereby making the representative shape available for each of the multiple categories.

[0034] In addition, when storing the representative object information in the representative object information storage unit 20, the representative object information generation unit 18 assigns operation information indicating the operation of the robot 12 on the representative shape to the operation part, which is the part of the representative shape represented by the representative object information that is the target of the operation.

[0035] The representative object information storage unit 20 stores, for each of a plurality of categories, representative object information that is a three-dimensional model of a representative object that belongs to the category, with operation information added to the representative object information.

[0036] FIG. 4 is a diagram illustrating various types of information that may be stored in the representative object information storage unit 20. As shown in FIG. 4, the representative object information storage unit 20 may store a category name, representative object information, operation information, an OBB (Oriented Bounding Box) of the operation area, approximation information of the shape of the operation area, and relative position and orientation information of the hand. Note that the representative object information storage unit 20 stores, for example, a category name, a representative object shape, and operation information for each of a plurality of representative objects. The representative object information storage unit 20 may additionally store at least one of the OBB of the operation information, approximation information of the operation information, and relative position and orientation information of the hand for each of a plurality of representative objects. The OBB will be described later.

[0037] For example, a user managing the robot control device 14 performs an operation to store various information including representative object information and operation information for each of multiple categories in the representative object information storage unit 20, thereby storing various information in the representative object information storage unit 20.

[0038] The target object information acquisition unit 22 acquires an observed shape that represents the shape of a target object that is an object to be operated, and a category to which the target object belongs.

[0039] Specifically, the target object information acquisition unit 22 acquires the NOCS map of the target object and the category to which the target object belongs by inputting an image of the target object into a pre-generated trained model for acquiring the NOCS map of the object and the category to which the object belongs.

[0040] FIG. 5 is a diagram for explaining the trained model of this embodiment. The trained model of this embodiment can be generated in advance using, for example, known machine learning technology and known artificial intelligence technology. For example, the trained model is a known neural network model. Details of the trained model are disclosed, for example, in Reference 1 above.

[0041] 5, for example, when an RGB image of a target object is captured by camera 12A, the RGB image is input to the trained model, which outputs the target object's category, a mask image in which the background of the target object is masked, and a NOCS map of the target object. The mask image identifies the area occupied by the object in the RGB image.

[0042] Therefore, the target object information acquisition unit 22 inputs an RGB image containing the target object into the trained model, thereby acquiring the category of the target object, a mask image in which the background of the target object is masked, and a NOCS map of the target object.

[0043] Then, the target object information acquisition unit 22 performs a pose fitting process using the distance image showing the target object, the mask image in which the background of the target object is masked, and the NOCS map of the target object, for example, by utilizing the technology disclosed in Reference 1 above, and estimates the three-dimensional object position and orientation of the target object in the world coordinate system.

[0044] Specifically, the target object information acquisition unit 22 identifies the position and orientation of the target object in world coordinates by associating each point on the surface of the target object represented by the NOCS map with world coordinates. The coordinate values ​​in cubic space for each point on the surface of the target object are determined from the NOCS map of the target object output from the trained model. Then, the 3D object position and orientation of the target object are estimated from the correspondence between the coordinate values ​​in cubic space for each point on the surface of the target object and the distance obtained from the range image.

[0045] The operation information assigning unit 24 assigns operation information associated with an operation part of a representative shape corresponding to the category to which the target object belongs to a part of the observed shape corresponding to that operation part.

[0046] Specifically, the operation information assignment unit 24 assigns operation information associated with the representative shape to the corresponding part of the observed shape by associating each point on the surface of the representative object represented by the representative shape with each point on the surface of the target object represented by the observed shape.

[0047] FIG. 6 is a diagram for explaining the registration process for determining the deviation between each point on the surface of the representative object represented by the representative shape and each point on the surface of the target object represented by the observed shape.

[0048] 6, when associating the point cloud of the representative shape R with the point cloud of the target object O, the position of each point included in the point cloud of the target object O is changed to associate the point cloud of the representative shape R with the point cloud of the target object O. More specifically, the operation information assignment unit 24 associates the point cloud of the representative shape R with the point cloud of the target object O by performing a registration process as shown in FIG.

[0049] As shown in FIG. 7, for example, the center of the Gaussian kernel G is set at the position of each point included in the point cloud of the target object O, and the position of each point included in the point cloud of the target object O is changed according to the value calculated by the Gaussian kernel. For example, as shown in FIG. 7, consider a case where the center of the Gaussian kernel G1 is set to the position of point p1 included in the point cloud of the target object O. In this case, when the probability of output from the Gaussian kernel G1 for point p2 included in the point cloud of the representative shape R is compared with the probability of output from the Gaussian kernel G1 for point p3 included in the point cloud of the representative shape R, the probability of point p3 is greater than the probability of point p2. Therefore, the operation information assigning unit 24 updates the position of point p1 of the target object so as to move point p1 of the target object in the direction of point p3.

[0050] The operation information providing unit 24 performs the above-described process on each of all points included in the point cloud of the target object O. In this case, for example, the operation information providing unit 24 updates the position of each point on the target object according to the following equation: y j is the 3D coordinate of the jth point in the point cloud of the target object O before updating, and v j is the three-dimensional deviation of the j-th point in the point cloud of the target object O, and x i is the updated 3D coordinate of the i-th point included in the point cloud of the representative shape R. The deviation v of each point included in the point cloud is calculated according to the probability of each point calculated by the Gaussian kernel. j is estimated.

[0051]

number

[0052] The operation information attaching unit 24 repeatedly executes the above-described process. In this case, the operation information attaching unit 24 updates the size of the Gaussian kernel (for example, the variance of the Gaussian kernel) so that the size of the Gaussian kernel gradually decreases.

[0053] For example, as shown in FIG. 7, the operation information providing unit 24 sets the Gaussian kernel to G1, G2, G3 as the number of iterations increases from the first iteration to the second iteration and then to the Kth iteration.

[0054] 7 shows an example in which the center of the Gaussian kernel G is set at the position of each point included in the point cloud of the target object O, and the position of each point included in the point cloud of the target object O is changed according to the value calculated by the Gaussian kernel, but it is also possible to set the center of the Gaussian kernel G at the position of each point included in the point cloud of the representative shape R, and then acquire the value calculated by the Gaussian kernel. In this case, there may be cases in which some points in the point cloud of the symmetric object O do not exist because they cannot be observed from the camera, as will be described later. In such cases, it is necessary to perform processing such as not including points in the point cloud of the symmetric object O that are a certain distance away from points in the point cloud of the representative shape R as targets for correspondence.

[0055] 7 may be real world space or NOCS space. In the case of world space, a process of projecting the representative object into the world space is executed before the registration process. In the case of NOCS space, a process of projecting the representative object into the NOCS space is executed before the registration process.

[0056] Fig. 8 is a diagram showing an example of the transfer of operation information when the registration process is completed. As shown in Fig. 8, the operation information (e.g., gripping area information, spout area information, etc.) previously assigned to the point cloud of the representative shape R is transferred to the point cloud of the target object O.

[0057] In the example shown in FIG. 8, the relative position and orientation information between the grasped region of the object and the hand of the robot is also transferred from the point cloud of the representative shape R to the point cloud of the target object O.

[0058] In this case, for each part of the representative shape R to which operation information is assigned, a circumscribed rectangular box posture (OBB: Oriented Bounding Box) is calculated in advance, and the grasping position and posture are defined by the relative position and posture between the OBB of the part to which operation information is assigned and the origin of the robot's hand coordinate system.

[0059] The operation information providing unit 24 may calculate the OBB after transferring the operation information to the point cloud of the target object. In this case, the relative position and orientation from the OBB becomes the gripping position and orientation in the observation point cloud.

[0060] The robot control unit 26 controls the operation of the robot on the target object O based on the operation information assigned to the point cloud of the target object O. For example, based on the gripping region information included in the operation information assigned to the point cloud of the target object O, the robot control unit 26 outputs a control command to the robot 12 to cause the hand of the robot 12 to grip the part represented by the gripping region information. Also, for example, based on the spout region information included in the operation information assigned to the point cloud of the target object O, the robot control unit 26 outputs a control command to the robot 12 to tilt the part represented by the spout region information and pour liquid present in the target object.

[0061] In this way, since the robot performs operations in accordance with the operation information for the representative object information stored in the representative object information storage unit 20, even after the robot has been enabled to perform operations, the user can easily change the operations to be performed by the robot by changing the operation information stored in the representative object information storage unit 20.

[0062] Note that when gripping region information is added to the point cloud of the target object O, the size of the point cloud to which the gripping region information is added may not satisfy the size required for robot operation. For example, it is possible that a part of the operation region cannot be observed because it is behind the observation direction of the camera 12A, or that the observation of the operation region by the camera is blocked by another part of the target object. In anticipation of such cases, the operation information adding unit 24 may perform the following processes.

[0063] (basic geometric approximation) For example, the operation information providing unit 24 may approximate the grip area information by a predetermined geometric shape (for example, a rectangular parallelepiped or a cylinder). Note that this processing corresponds to processing in the case where information such as a circumscribed rectangular parallelepiped orientation (OBB: Oriented Bounding Box) is not provided in advance for the representative shape R.

[0064] The point cloud of the target object acquired by the camera 12A is often not a point cloud of the entire periphery of the target object. Therefore, it may be difficult for the robot 12 to manipulate the object based on a point cloud of only a portion of the target object. Therefore, the operation information providing unit 24 approximates the grasping area of ​​the target object with a combination of basic shapes such as a cylinder or a rectangular parallelepiped. The robot control unit 26 then controls the robot 12 to grasp the basic shapes.

[0065] 9 and 10 are diagrams illustrating a process of approximating a gripping region of a target object by a combination of basic shapes such as a cylinder or a rectangular parallelepiped. For example, as shown in FIG. 9, the gripping region of the point cloud of the representative shape R is approximated in advance by a rectangular parallelepiped or a cylinder. Then, as shown in FIG. 10, when operation information previously assigned to the point cloud of the representative shape R is transferred to the point cloud of the target object, information on the gripping region approximated by a rectangular parallelepiped or a cylinder is also transferred to the point cloud of the target object. As a result, a basic geometric shape is assigned to the point cloud of the target object. Therefore, even if a point cloud of the entire periphery of the target object has not been obtained, the robot 12 only needs to grip a basic geometric shape such as a cylinder or a rectangular parallelepiped projected onto the point cloud of the target object.

[0066] (temporary grip) Alternatively, for example, the operation information may include provisional grasping region information indicating a region to be provisionally grasped by the robot 12. In this case, the operation information assigning unit 24 transfers the grasping region information and provisional grasping region information previously assigned to the point cloud of the representative shape R to the point cloud of the target object O, thereby adding the grasping region information and provisional grasping region information to the point cloud of the target object O.

[0067] Then, if the point cloud area of ​​the target object O reflected in the grasping area information is less than a predetermined size, the robot control unit 26 controls the operation of the robot 12 so that the robot 12 grasps the part represented by the temporary grasping area information assigned to the target object.

[0068] The robot control unit 26 controls the operation of the robot 12 so as to grasp the part of the target object O represented by the provisional grasping area information, and if, after changing the position or posture of the target object O, the area of ​​the point cloud of the target object O reflected in the grasping area information becomes larger than a predetermined size, controls the operation of the robot 12 so as to grasp the grasping area information of the target object O.

[0069] As described above, the point cloud obtained by camera 12A does not include a point cloud covering the entire periphery of target object O, which may make it difficult for the robot 12 to manipulate the object. For this reason, the robot control unit 26 determines whether the area represented by the provisional grasping area information can be grasped, based on the point cloud of the target object O obtained by camera 12A. If the area of ​​the point cloud of the target object O captured in the grasping area information is equal to or larger than a predetermined size, the robot control unit 26 determines that the area represented by the provisional grasping area information is graspable. The robot control unit 26 controls the robot 12 to grasp the area represented by the provisional grasping area information of the target object O. After the robot 12 changes the position or posture of the target object O, the robot control unit 26 determines whether the area of ​​the point cloud captured in the grasping area information of the target object O is equal to or larger than a predetermined size, based on the point cloud obtained by camera 12A. The robot control unit 26 controls the operation of the robot 12 so that the target object O is grasped in the grasped region information when the region of the point cloud reflected in the grasped region information of the target object O is equal to or larger than a predetermined size.

[0070] 11 and 12 are diagrams illustrating a temporary gripping region. As shown on the left side of FIG. 11, the point cloud of the target object O is assigned with gripping region information and provisional gripping region information. The right side of FIG. 11 illustrates a case where the point cloud captured by the camera 12A does not include the gripping region. In this case, as shown on the left side of FIG. 12, the robot 12 is caused to grip the provisional gripping region, the position or orientation of the target object O is changed, and the point cloud of the target object O is again captured by the camera 12A as shown on the right side of FIG. 12. At this time, the robot control unit 26 updates the point cloud information of the hidden target object O. The operation information assigning unit 24 then assigns gripping region information to the point cloud of the target object O by transferring the operation information assigned to the point cloud of the representative shape R to the newly acquired point cloud of the target object O. The robot control unit 26 controls the robot 12 to grip the target object O based on the newly assigned gripping region information. When a new image is obtained by the camera 12A, the category of the target object O may be identified again.

[0071] (Photographing the target object from multiple viewpoints) When it is possible to photograph the target object O from a plurality of viewpoints, the operation information assigning unit 24 transfers the gripping region information that has been assigned in advance to the point cloud of the representative shape R to the point cloud of the target object O based on the point clouds obtained from the plurality of viewpoints. For example, when the operation information assigning unit 24 assigns the gripping region information to the point cloud of the target object O, if the area of ​​the point cloud that appears in the gripping region information is smaller than a predetermined size, the operation information assigning unit 24 assigns the gripping region information to the point cloud of the target object O acquired from a different viewpoint.

[0072] (Use of symmetry information) If the target object O is an object having symmetry, the hidden points of the target object O are reproduced based on symmetry information that indicates the symmetry of the target object O.

[0073] In this case, for example, symmetry information is assigned in advance to the point group of the representative shape R. Then, the operation information assigning unit 24 transfers the symmetry information assigned in advance to the point group of the representative shape R to the point group of the target object O.

[0074] FIG. 13 is a diagram for explaining symmetry information. FIG. 13 is also a diagram showing an example of a point cloud of a target object O to which symmetry information has been transferred. As shown in FIG. 13, S1 and S2 in the point cloud of the target object O are represented as symmetrical parts. By adding symmetry information to the point cloud of the target object O, the robot 12 can reproduce the hidden point cloud in the point cloud of the target object O, and can grasp the part corresponding to the grasping region.

[0075] Next, the operation of the robot control system 10 according to this embodiment will be described.

[0076] When the robot control device 14 receives a predetermined instruction signal, the CPU 42 of the robot control device 14 reads out the representative object information generation program from the storage device 46, and loads and executes it in the memory 44. As a result, the CPU 42 functions as each functional component of the robot control device 14, and the representative object information generation process shown in FIG.

[0077] In step S100, the representative object information generation unit 18 stores, for each of the multiple categories, representative object information, which is a 3D model of a representative shape belonging to that category, in the representative object information storage unit 20 described later, thereby making the representative shape available for each of the multiple categories.

[0078] In step S102, the representative object information generation unit 18 receives, for each of a plurality of categories, representative object information that is a 3D model of a representative shape belonging to the category, and operation information. For example, the representative object information generation unit 18 receives a plurality of pieces of representative object information and operation information input by the user.

[0079] In step S104, the representative object information generating unit 18 stores the representative object information and operation information received in step S100 in the representative object information storage unit 20, thereby making the representative shape available for each of the multiple categories.

[0080] Next, when the robot control device 14 receives a predetermined instruction signal, the CPU 42 of the robot control device 14 reads the robot control program from the storage device 46, loads it into the memory 44, and executes it. As a result, the CPU 42 functions as each functional component of the robot control device 14, and the robot control process shown in FIG. 15 is executed.

[0081] In step S200, the target object information acquisition unit 22 acquires an RGB image and a range image of the target object O captured by the camera 12A.

[0082] In step S202, the target object information acquisition unit 22 inputs the RGB image received in step S100 into a trained model generated in advance, thereby acquiring the category to which the target object O belongs, the NOCS map of the target object O, and a mask image of the target object O. Then, the object region of the target object O is extracted using the distance image and the mask image.

[0083] In step S204, the target object information acquisition unit 22 performs a pose fitting process using the distance image acquired in step S200, the NOCS map of the target object O acquired in step S202, and the extracted object region of the target object O, for example, by utilizing the technology disclosed in Reference 1 above, and estimates the three-dimensional object position and orientation of the target object in the world coordinate system.

[0084] In step S206, the operation information providing unit 24 acquires, from the representative object information storage unit 20, representative object information corresponding to the category of the target object O identified in step S202.

[0085] In step S208, the operation information assigning unit 24 assigns operation information associated with the representative shape corresponding to the representative object information read out in step S206 to each part of the point cloud of the target object O acquired in step S202.

[0086] In step S210, the robot control unit 26 controls the robot 12 to operate the target object O in accordance with the point cloud of the target object O with the operation information obtained in step S208.

[0087] As described above, the robot control device according to this embodiment controls the operation of a robot on an object. The robot control device performs a process of making a representative shape available for each object classification, which is a classification of objects, and a process of assigning operation information indicating the robot's operation on the representative shape to an operation portion of the representative shape, which is the portion that is the target of the operation. The robot control device acquires an observed shape that represents the shape of a target object, which is the object to be operated, and an object classification to which the target object belongs. The robot control device assigns operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape that corresponds to the operation portion. The robot control device controls the operation of the robot on the target object based on the operation information assigned to the observed shape. This makes it possible to have the robot operate objects of different shapes or sizes without preparing a 3D model for each object.

[0088] The technology of the present disclosure is not limited to the above-described embodiment, and various modifications and applications are possible within the scope of the gist of this disclosure.

[0089] For example, the robot control system 10 of the above embodiment may be configured as a robot control system 210 as shown in Fig. 16. The robot control system 210 shown in Fig. 16 includes a representative object information generation device 215, a robot task management device 216, an operation information assignment device 217, and a robot 12. The representative object information generation unit 18 of the representative object information generation device 215 transmits the representative object information to the robot task management device 216, which stores the representative object information in the representative object information storage unit 20. The representative object information acquisition unit 218 of the operation information assignment device 217 acquires the representative object information and operation information stored in the representative object information storage unit 20 of the robot task management device 216. Then, the operation information assignment unit 24 assigns the operation information assigned to the representative object information to the point cloud of the target object, as in the above embodiment.

[0090] In the above embodiment, the case where the technology disclosed in Reference 1 is used has been described as an example, but the present invention is not limited to this. A point cloud and a three-dimensional object position and orientation of a target object may be acquired using a technology different from the technology disclosed in Reference 1.

[0091] In addition, modified examples are described for the following points.

[0092] (Acquisition of RGB images and range images (or point clouds)) The image is acquired by a sensor capable of simultaneously capturing an RGB image (or grayscale image) and a distance image (or point cloud). A sensor known as an RGB-D sensor is generally used, but a homemade sensor in which an RGB camera and a 3D measurement camera are separately prepared and calibrated may also be used. Image acquisition methods include the global shutter method and the rolling shutter method. Distance image acquisition methods include triangulation using multiple cameras such as a stereo camera, phase shift methods and gray code methods using a camera and projector, and the ToF (Time of Flight) method, which consists of a projector that emits infrared light and a photoreceiver that receives the reflected light. This disclosure does not limit the image acquisition method or distance image measurement method.

[0093] (Object category and NOCS map estimation) The above embodiment employs processing using the technology disclosed in Reference 1, and simultaneously estimates a segmentation mask (the above-mentioned mask image) in addition to the target object category and NOCS map. In the present embodiment, the object category and NOCS map are estimated from only an RGB image according to the algorithm of an existing method, but the object category and NOCS map may also be estimated by simultaneously learning a range image.

[0094] (Point cloud extraction of object regions) In this embodiment, only the target object region is extracted from the point cloud or range image using the segmentation mask (the above-mentioned mask image) that is output simultaneously when the object category and NOCS map are estimated, but other methods may also be used. Note that by using the method used in the above embodiment, it is possible to prevent erroneous correspondence with the background in the registration process with a 3D model of the representative shape that is performed in subsequent processing.

[0095] (Point cloud registration between the target object's point cloud and a 3D model of its representative shape) In the above embodiment, the position and orientation recognition results of the object at the category level are used as initial values ​​to perform the registration process between the 3D model of the representative shape and the point cloud of the target object. However, other methods may be used. In the above embodiment, for example, non-rigid registration can be used. Non-rigid registration solves the problem of determining the position of each 3D point between the 3D model of the representative shape and the point cloud of the target object. This allows for the determination of transformation parameters for each 3D point, making registration possible even if the shapes of the 3D model of the representative shape and the point cloud of the target object differ. Furthermore, by determining the transformation parameters for each 3D point, it becomes possible to transfer operation information assigned to the 3D model of the representative shape to the point cloud of the target object.

[0096] (category classification of individual 3D models) In the above embodiment, various shapes of objects to be recognized are classified into categories in order to generate NOCS maps and 3D models of representative shapes for each category. There are various ways to define the categories to be classified, such as defining categories by general object names such as "Cup," "Bottle," and "Bowl," defining categories by 3D models with similar geometric 3D shapes, or defining categories by the degree of similarity in the configuration or number of basic shape parts when the shape of a 3D model is approximated by a combination of basic shape parts such as a cylinder, sphere, plane, and cube. In the above embodiment, the method of defining the categories to be classified is not important.

[0097] (Generate a 3D model of a representative shape for each category) In the above embodiment, the individual 3D models are classified into categories, and then a 3D model of a representative shape is generated for each category. The 3D model of the representative shape may be a 3D model that has an average shape of the individual 3D models in the category, or a shape that has a principal component of the individual 3D models using a statistical analysis method such as Principal Component Analysis (PCA). Alternatively, one of the individual 3D models in the category may be selected and used as the representative 3D model. If there is a large variation in shape among the individual 3D models in the category, two or more 3D models of the representative shape may be retained in the same category.

[0098] (Operation information is registered in a 3D model of a representative shape for each category) In the above embodiment, information for the robot to operate the generated 3D model of the representative shape is registered as operation information. The operation information to be registered varies depending on the robot's application, but examples include a grasping area for the robot to grasp an object, an area for inserting other objects or substances, or an area where the object can be placed stably (for example, on a desk). The operation information may be registered manually by a human, or automatically using the shape characteristics or symmetry of the object as an index. Furthermore, the operation information may be registered using a method for recognizing an object operation area called functional attribute estimation (for example, affordance estimation). In the above embodiment, the type of operation information to be registered and the registration method are not important.

[0099] (Separation of trained models) In the above embodiment, an example has been described in which one trained model is used to perform object classification, NOCS map estimation, mask image generation, etc., but this is not limited to this. For example, the trained model may be separated into a first trained model that performs object classification and a second trained model that performs NOCS map estimation, mask image generation, etc. In this case, the target object information acquisition unit 22 inputs an image of the target object into the first trained model that has been trained to input an image of the target object and output a category that is an object classification, thereby acquiring the category to which the target object belongs. The target object information acquisition unit 22 acquires distance information (e.g., a distance image) about an image containing the target object, and inputs the image containing the target object to a second trained model that has been trained to output a NOCS map of a representative shape corresponding to the category to which the target object belongs, the NOCS map being aligned with the target object in the image, and a mask image showing the area occupied by the target object. By inputting the image containing the target object, the target object information acquisition unit 22 acquires a NOCS map of a representative shape corresponding to the category to which the target object belongs, aligned with the image containing the target object, and a mask image. The target object information acquisition unit 22 identifies an observed shape positioned in a world coordinate system from the mask image and distance information, identifies the position, orientation, and size of the observed shape in the world coordinate system from the distance information and NOCS map, positions the representative shape in the world coordinate system based on the position, orientation, and size of the observed shape in the world coordinate system, and calculates the deviation for each part between the representative shape positioned in the world coordinate system and the observed shape. The operation information assignment unit 24 assigns operation information to parts of the observed shape based on the deviation.

[0100] The robot control device 14 may further include a target object shape estimation unit. In this case, the operation information further includes approximate shape information indicating an approximate shape that approximates the shape of the operation portion. When the portion of the observed shape corresponding to the operation portion is a part of the portion of the target object corresponding to the operation portion, the target object shape estimation unit estimates the overall shape of the portion of the target object corresponding to the operation portion by approximating it to the approximate shape based on the observed shape and approximate shape information. For example, the overall shape of the portion of the target object is approximated by a cube, a rectangular parallelepiped, a sphere, a cylinder, or the like.

[0101] Furthermore, the processes executed by the CPU after reading the software (program) in the above-described embodiments may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) whose circuit configuration can be changed after fabrication, such as field-programmable gate arrays (FPGAs), and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. Each process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0102] In the above embodiment, the programs are pre-stored (installed) in a storage device, but the present invention is not limited to this. The programs may be provided in a form stored on a storage medium such as a CD-ROM, DVD-ROM, Blu-ray disc, or USB memory. The programs may also be downloaded from an external device via a network.

[0103] (Addendum) The following additional notes are provided regarding aspects of the present disclosure.

[0104] (Appendix 1) A robot control device that controls the operation of a robot on an object, a representative object information generation unit that performs a process of making a representative shape available for each object classification that is a classification of the object, and a process of assigning operation information that indicates an operation of the robot with respect to the representative shape to an operation part that is a part of the representative shape that is a target of the operation; a target object information acquisition unit that acquires an observed shape representing a shape of a target object that is an object to be operated and the object classification to which the target object belongs; an operation information assigning unit that assigns the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; a robot control unit that controls the operation of the robot to the target object based on the operation information assigned to the observed shape, Robot control device. (Appendix 2) a representative object information generation unit that performs a process of making a representative shape available for each object classification that is a classification of objects, and a process of assigning operation information that indicates an operation of a robot with respect to the representative shape to an operation part that is a part of the representative shape that is a target of the operation; A representative object information generating device comprising: (Appendix 3) a representative object information acquisition unit that acquires information including a representative shape for each object classification, which is a classification of objects, and in which operation information indicating an operation of the robot with respect to the representative shape is assigned to an operation part of the representative shape that is a part that is a target of the operation; a target object information acquisition unit that acquires an observed shape representing a shape of a target object that is an object to be operated and the object classification to which the target object belongs; an operation information assigning unit that assigns the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape that corresponds to the operation portion, Operation information providing device. (Appendix 4) the target object information acquisition unit inputs an image of the target object into a first trained model for acquiring the object classification, and acquires an object classification to which the target object belongs; 4. An operation information providing device according to claim 3. (Appendix 5) The target object information acquisition unit acquiring distance information for an image in which the target object appears; a second trained model that is trained to input an image of the target object and output a NOCS map of the representative shape corresponding to the object classification to which the target object belongs, the NOCS map being aligned with the target object in the image, and a mask image showing the area occupied by the target object; and Identifying the observed shape positioned in the world coordinate system from the mask image and the distance information; Identifying the position, orientation, and size of the observed shape in a world coordinate system from the distance information and the NOCS map; positioning the representative shape in the world coordinate system based on the position, orientation, and size of the observed shape in the world coordinate system; Calculating a deviation for each part between the representative shape positioned in the world coordinate system and the observed shape; the operation information assigning unit assigns the operation information to a portion of the observed shape based on the deviation. 4. An operation information providing device according to claim 3. (Appendix 6) the operation information further includes approximate shape information indicating an approximate shape that approximates the shape of the operation area, a target object shape estimation unit that, when a portion of the observed shape corresponding to the operation portion is a part of the portion of the target object corresponding to the operation portion, estimates an entire shape of the portion of the target object corresponding to the operation portion by approximating the approximate shape based on the observed shape and the approximate shape information; 2. A robot control device as described in appendix 1. (Appendix 7) the operation information includes gripping area information representing an area to be gripped by the robot, and temporary gripping area information representing an area to be temporarily gripped by the robot; the operation information adding unit adds the gripping region information and the provisional gripping region information to the observed shape; the robot control unit controls the operation of the robot so as to grasp a portion of the target object represented by the provisional grasping region information when the region of the observed shape reflected in the grasping region information is smaller than a predetermined size. 10. The robot control device according to claim 1 or 6. (Appendix 8) the control unit controls the operation of the robot so as to grasp a portion of the target object represented by the provisional grasping region information, and when an area of ​​the observed shape reflected in the grasping region information becomes equal to or larger than a predetermined size after changing the position or posture of the target object, controls the operation of the robot so as to grasp the grasping region information of the target object. 8. The robot control device according to claim 7. (Appendix 9) the operation information includes gripping area information representing an area to be gripped by the robot; when the area of ​​the observed shape captured in the gripping area information is smaller than a predetermined size when the gripping area information is assigned to the observed shape, the operation information assigning unit assigns the gripping area information to the observed shape acquired from a different viewpoint. 10. The robot control device according to claim 1, 7, or 8. (Appendix 10) the operation information includes gripping area information representing an area to be gripped by the robot; When the area of ​​the observed shape reflected in the gripping area information is smaller than a predetermined size, the control unit refers to symmetry information previously assigned to the representative shape, estimates the shape of a part of the target object corresponding to the operation part, and controls the operation of the robot to grip the part. 10. The robot control device of claim 1, 7, 8, or 9. (Appendix 11) A robot control method for controlling the operation of a robot on an object, comprising: performing a process of making a representative shape available for each object classification that is a classification of the object, and a process of assigning operation information that indicates an operation of the robot with respect to the representative shape to an operation part that is a part of the representative shape that is a target of the operation; acquiring an observed shape representing the shape of a target object that is the object to be operated and the object classification to which the target object belongs; assigning the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; controlling the operation of the robot to the target object based on the operation information assigned to the observed shape; A robot control method that causes a computer to execute processing. (Appendix 12) A process of making a representative shape available for each object classification, which is a classification of objects, and a process of assigning operation information indicating an operation of the robot with respect to the representative shape to an operation part, which is a part of the representative shape that is a target of the operation, A representative object information generating method that causes a computer to execute processing. (Appendix 13) acquire information including a representative shape for each object classification, which is a classification of objects, in which operation information indicating an operation of the robot with respect to the representative shape is assigned to an operation part that is a part of the representative shape that is a target of the operation; acquiring an observed shape representing the shape of a target object that is the object to be operated and the object classification to which the target object belongs; assigning the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; A method for providing operation information that causes a computer to execute a process. (Appendix 14) A robot control program for controlling the operation of a robot on an object, performing a process of making a representative shape available for each object classification that is a classification of the object, and a process of assigning operation information that indicates an operation of the robot with respect to the representative shape to an operation part that is a part of the representative shape that is a target of the operation; acquiring an observed shape representing the shape of a target object that is the object to be operated and the object classification to which the target object belongs; assigning the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; controlling the operation of the robot to the target object based on the operation information assigned to the observed shape; A robot control program that causes a computer to execute a process. (Appendix 15) A process of making a representative shape available for each object classification, which is a classification of objects, and a process of assigning operation information indicating an operation of the robot with respect to the representative shape to an operation part, which is a part of the representative shape that is a target of the operation, A representative object information generation program for causing a computer to execute processing. (Appendix 17) acquire information including a representative shape for each object classification, which is a classification of objects, in which operation information indicating an operation of the robot with respect to the representative shape is assigned to an operation part that is a part of the representative shape that is a target of the operation; acquiring an observed shape representing the shape of a target object that is the object to be operated and the object classification to which the target object belongs; assigning the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; A program that provides operational information to cause a computer to execute a process. [Explanation of symbols]

[0105] 10 Robot Control System 12. Robot 12A Camera 14 Robot control device 18 Representative object information generation unit 20 Representative object information storage unit 22 Target object information acquisition unit 24 Operation information assignment unit 26 Robot control unit 210 Robot Control System 215 Representative object information generation device 216 Robot Work Management Device 217 Operation information providing device 218 Representative object information acquisition unit

Claims

1. A robot control device that controls the operation of a robot on an object, a representative object information generation unit that performs a process of making a representative shape available for each object classification that is a classification of the object, and a process of assigning operation information that indicates an operation of the robot with respect to the representative shape to an operation part that is a part of the representative shape that is a target of the operation; a target object information acquisition unit that acquires an observed shape representing a shape of a target object that is an object to be operated and the object classification to which the target object belongs; an operation information assigning unit that assigns the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; a robot control unit that controls the operation of the robot to the target object based on the operation information assigned to the observed shape, Robot control device.

2. a representative object information generation unit that performs a process of making a representative shape available for each object classification that is a classification of objects, and a process of assigning operation information that indicates an operation of the robot with respect to the representative shape to an operation part that is a part of the representative shape that is the target of the operation; A representative object information generating device comprising:

3. a representative object information acquisition unit that acquires information including a representative shape for each object classification, which is a classification of objects, and in which operation information indicating an operation of the robot with respect to the representative shape is assigned to an operation part of the representative shape that is a part that is a target of the operation; a target object information acquisition unit that acquires an observed shape representing a shape of a target object that is an object to be operated and the object classification to which the target object belongs; an operation information assigning unit that assigns the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape that corresponds to the operation portion, Operation information providing device.

4. the target object information acquisition unit inputs an image of the target object into a first trained model that has been trained to input an image of the target object and output the object classification, thereby acquiring an object classification to which the target object belongs; The operation information providing device according to claim 3 .

5. The target object information acquisition unit acquiring distance information for an image in which the target object appears; a second trained model that is trained to input an image of the target object and output a NOCS map of the representative shape corresponding to the object classification to which the target object belongs, the NOCS map being aligned with the target object in the image, and a mask image showing the area occupied by the target object; and by inputting an image of the target object into the second trained model, the second trained model is trained to input an image of the target object and output a NOCS map of the representative shape corresponding to the object classification to which the target object belongs, the NOCS map being aligned with the image of the target object in the image, and a mask image showing the area occupied by the target object; Identifying the observed shape positioned in a world coordinate system from the mask image and the distance information; Identifying the position, orientation, and size of the observed shape in a world coordinate system from the distance information and the NOCS map; positioning the representative shape in world coordinates based on the position, orientation, and size of the observed shape in the world coordinate system; Calculating a deviation for each part between the representative shape positioned in the world coordinate system and the observed shape; the operation information assigning unit assigns the operation information to a portion of the observed shape based on the deviation. The operation information providing device according to claim 3 .

6. the operation information further includes approximate shape information indicating an approximate shape that approximates the shape of the operation portion, a target object shape estimation unit that, when a portion of the observed shape corresponding to the operation portion is a part of the portion of the target object corresponding to the operation portion, estimates an entire shape of the portion of the target object corresponding to the operation portion by approximating the approximate shape based on the observed shape and the approximate shape information; The robot control device according to claim 1 .

7. the operation information includes gripping area information representing an area to be gripped by the robot, and temporary gripping area information representing an area to be temporarily gripped by the robot; the operation information adding unit adds the gripping region information and the provisional gripping region information to the observed shape; the robot control unit controls the operation of the robot so as to grasp a portion of the target object represented by the provisional grasping region information when the region of the observed shape reflected in the grasping region information is smaller than a predetermined size. The robot control device according to claim 1 .

8. the robot control unit controls the operation of the robot so as to grasp a portion of the target object represented by the provisional grasping region information, and when, after changing the position or posture of the target object, the region of the observed shape reflected in the grasping region information becomes equal to or larger than a predetermined size, controls the operation of the robot so as to grasp the grasping region information of the target object. The robot control device according to claim 7.

9. the operation information includes gripping area information representing an area to be gripped by the robot; when the area of ​​the observed shape captured in the gripping area information is smaller than a predetermined size when the gripping area information is assigned to the observed shape, the operation information assigning unit assigns the gripping area information to the observed shape acquired from a different viewpoint. The robot control device according to claim 1 .

10. the operation information includes gripping area information representing an area to be gripped by the robot; When the area of ​​the observed shape reflected in the gripping area information is smaller than a predetermined size, the robot control unit refers to symmetry information previously assigned to the representative shape, estimates the shape of a part of the target object corresponding to the operation part, and controls the operation of the robot so as to grip the part. The robot control device according to claim 1 .

11. A robot control method for controlling the operation of a robot on an object, comprising: performing a process of making a representative shape available for each object classification that is a classification of the object, and a process of assigning operation information that indicates an operation of the robot with respect to the representative shape to an operation part that is a part of the representative shape that is a target of the operation; acquiring an observed shape representing the shape of a target object that is the object to be operated and the object classification to which the target object belongs; assigning the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; controlling the operation of the robot to the target object based on the operation information assigned to the observed shape; A robot control method that causes a computer to execute processing.

12. A process of making a representative shape available for each object classification, which is a classification of objects, and a process of assigning operation information indicating an operation of the robot with respect to the representative shape to an operation part, which is a part of the representative shape that is a target of the operation, A representative object information generating method that causes a computer to execute processing.

13. acquire information including a representative shape for each object classification, which is a classification of objects, in which operation information indicating an operation of the robot with respect to the representative shape is assigned to an operation part that is a part of the representative shape that is a target of the operation; acquiring an observed shape representing the shape of a target object that is the object to be operated and the object classification to which the target object belongs; assigning the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; A method for providing operation information that causes a computer to execute a process.

14. A robot control program for controlling the operation of a robot on an object, performing a process of making a representative shape available for each object classification that is a classification of the object, and a process of assigning operation information that indicates an operation of the robot with respect to the representative shape to an operation part that is a part of the representative shape that is a target of the operation; acquiring an observed shape representing the shape of a target object that is the object to be operated and the object classification to which the target object belongs; assigning the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; controlling the operation of the robot to the target object based on the operation information assigned to the observed shape; A robot control program that causes a computer to execute a process.

15. A process of making a representative shape available for each object classification, which is a classification of objects, and a process of assigning operation information indicating an operation of the robot with respect to the representative shape to an operation part, which is a part of the representative shape that is a target of the operation, A representative object information generation program for causing a computer to execute processing.

16. acquire information including a representative shape for each object classification, which is a classification of objects, in which operation information indicating an operation of the robot with respect to the representative shape is assigned to an operation part that is a part of the representative shape that is a target of the operation; acquiring an observed shape representing the shape of a target object that is the object to be operated and the object classification to which the target object belongs; assigning the operation information associated with the representative shape corresponding to the object classification to which the target object belongs to a portion of the observed shape corresponding to the operation portion; A program that provides operational information to cause a computer to execute a process.

Images