Robot control device and robot control method

Abstract

A robot control device comprises a control unit that is capable of estimating the manner-of-holding objects to be held, such estimation performed according to: a database storing reference information including object information on a plurality of objects and manner-of-holding information of the plurality of objects; and an estimation model enabling estimation of the manner-of-holding the objects. The control unit controls a robot on the basis of the estimated manner-of-holding. The control unit acquires recognition information for the objects to be held, and in a case where, based on the recognition information, it is determined that the manner-of-holding the objects to be held cannot be estimated from the database, estimates the manner-of-holding by using the estimation model.

Description

Cross-reference to related applications

[0001] This application claims the priority of Japanese Patent Application No. 2021-211022 (filed on December 24, 2021), and the entire disclosure of the said application is incorporated herein by reference for that purpose.

Technical Field

[0002] This disclosure relates to a robot control device and a robot control method.

Background Art

[0003] Conventionally, an object gripping device for gripping an object is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] A robot control device according to an embodiment of the present disclosure includes a control unit. The control unit is configured to be able to estimate the holding mode of an object to be held by a database storing reference information including object information of a plurality of objects and holding mode information of the plurality of objects, and an inference model capable of estimating the holding mode of an object. The control unit is configured to control a robot based on the estimated holding mode. When the control unit acquires recognition information of an object to be held and determines that the holding mode of the object to be held based on the recognition information cannot be estimated from the database, the control unit is configured to estimate the holding mode by the inference model.

[0006] The robot control method according to an embodiment of the present disclosure is executed by a robot control device. The robot control device is configured to be able to estimate the holding mode of an object to be held by a database storing reference information including object information of a plurality of objects and holding mode information of the plurality of objects, and an inference model capable of estimating the holding mode of an object. The robot control device is configured to control the robot based on the estimated holding mode. The robot control method includes the robot control device acquiring recognition information of an object to be held. When the robot control device determines that the holding mode of the object to be held based on the recognition information cannot be estimated from the database, the robot control method includes estimating the holding mode by the inference model.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0008] When a robot holds an object, the holding position can be determined based on general-purpose AI (Artificial Intelligence) or rules. Since the holding position is not uniformly determined, there is a possibility that reliable holding or holding according to the user's wishes may not be executed every time. That is, since the holding position is not uniformly determined, the executability of holding may decrease. The robot control system 1 (see FIG. 1) according to the present disclosure can improve the executability of holding.

[0009] (Configuration example of robot control system 1) As shown in FIGS. 1 and 2, a robot control system 1 according to an embodiment of the present disclosure includes a robot 2, an information acquisition unit 4, a robot control device 10, and a database 20. The robot 2 is configured to be able to hold the object to be held 8 by the end effector 2B. The robot control device 10 controls the robot 2 to cause the robot 2 to execute an operation of holding the object to be held 8 by the end effector 2B. The robot control device 10 determines, as a holding mode, the position where the robot 2 contacts the object to be held 8 when holding the object to be held 8 and the posture when the robot 2 holds the object to be held 8 based on the information stored in the database 20, and outputs the determined position and posture to the robot control device 10.

[0010] In the present embodiment, the robot control device 10 controls the robot 2 to hold the object to be held 8 on the work start table 6, for example. Further, the robot control device 10 controls the robot 2 to move the object to be held 8 from the work start table 6 to the work target table 7, for example. The object to be held 8 is also referred to as a work target. The robot 2 operates inside the operation range 5.

[0011] <Robot 2> Robot 2 includes an arm 2A and an end effector 2B. The arm 2A may be configured as, for example, a 6-axis or 7-axis vertically articulated robot. The arm 2A may also be configured as a 3-axis or 4-axis horizontally articulated robot or a scalar robot. The arm 2A may be configured as a 2-axis or 3-axis orthogonal robot. The arm 2A may be configured as a parallel link robot or the like. The number of axes constituting the arm 2A is not limited to those exemplified. In other words, the robot 2 has an arm 2A connected by a plurality of joints and operates by driving the joints.

[0012] The end effector 2B may include, for example, a gripper configured to hold the object 8 to be held. The gripper may have at least one finger. The finger of the gripper may have one or more joints. The finger of the gripper may have a suction portion that holds the object 8 by suction. The end effector 2B may be configured as two or more fingers that sandwich and hold (grip) the object 8 to be held. The end effector 2B may be configured as at least one nozzle having a suction portion. The end effector 2B may include a scoop hand configured to scoop up the object 8 to be held. The end effector 2B is not limited to these examples and may be configured to be able to perform various other operations. In the configuration illustrated in FIG. 1, it is assumed that the end effector 2B includes a gripper.

[0013] The end effector 2B may be provided with a sensor. The sensor may include a contact force sensor that detects the contact force when the finger of the gripper of the end effector 2B contacts the object 8 to be held. The sensor may include a force sensor that detects the force or torque acting on the end effector 2B, the gripper, or the finger. The contact force sensor or the force sensor may be configured as a piezoelectric sensor or a strain gauge or the like. The sensor may include a current sensor that detects the current flowing through the motor that drives the arm 2A, the end effector 2B, the gripper, or the finger.

[0014] The robot control device 10 can control the position of the end effector 2B by operating the arm 2A of the robot 2. The end effector 2B may have an axis that serves as a reference for the direction in which it acts on the object 8 to be held. When the end effector 2B has an axis, the robot control device 10 can control the direction of the axis of the end effector 2B by operating the arm 2A of the robot 2. The robot control device 10 controls the robot 2 so as to start or end the operation in which the end effector 2B acts on the object 8 to be held. While controlling the position of the end effector 2B or the direction of the axis of the end effector 2B, the robot control device 10 controls the operation of the end effector 2B to operate the robot 2 so as to move or process the object 8 to be held. In the configuration illustrated in FIG. 1, the robot control device 10 controls the robot 2 so that the end effector 2B holds the object 8 to be held on the work start table 6 and moves the end effector 2B to the work target table 7. The robot control device 10 controls the robot 2 so that the end effector 2B releases the object 8 to be held on the work target table 7. By doing so, the robot control device 10 can move the object 8 to be held from the work start table 6 to the work target table 7 by the robot 2.

[0015] <Information acquisition unit 4> The information acquisition unit 4 acquires recognition information. The information acquisition unit 4 may be configured to include a camera. The camera as the information acquisition unit 4 captures an image of the object 8 to be held as recognition information. The information acquisition unit 4 may be configured to include a depth sensor. The depth sensor as the information acquisition unit 4 acquires depth data of the object 8 to be held. The depth data may be converted into point cloud information of the object 8 to be held.

[0016] <Robot control device 10> As shown in FIG. 2, the robot control device 10 includes a control unit 12 and an interface 14. The interface 14 acquires information or data regarding the object to be held 8 or the like from an external device or outputs information or data to the external device. Further, the interface 14 acquires an image obtained by photographing the object to be held 8 from the information acquisition unit 4. The interface 14 may receive an input from the user. The interface 14 may output information or data so as to be recognized by the user. The control unit 12 determines a mode in which the robot 2 holds the object to be held 8 based on the information or data acquired by the interface 14. The mode in which the robot 2 holds the object to be held 8 is also simply referred to as the holding mode. Further, the robot control device 10 controls the robot 2 so that the robot 2 holds the object to be held 8 in the determined holding mode.

[0017] The control unit 12 may be configured to include at least one processor in order to provide control and processing capabilities for executing various functions. The processor may execute a program that realizes various functions of the control unit 12. The processor may be realized as a single integrated circuit. The integrated circuit is also referred to as an IC (Integrated Circuit). The processor may be realized as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be realized based on various other known technologies.

[0018] The control unit 12 may include a storage unit. The storage unit may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The storage unit stores various information. The storage unit stores programs and the like executed by the control unit 12. The storage unit may be configured as a non-transitory readable medium. The storage unit may function as a work memory of the control unit 12. At least a part of the storage unit may be configured separately from the control unit 12.

[0019] Interface 14 may be configured to include a communication device that can communicate either wired or wirelessly. The communication device may be configured to communicate using communication methods based on various communication standards. The communication device can be configured using known communication technologies.

[0020] Interface 14 may also be configured to include an input device that receives input such as information or data from a user. The input device may be configured to include, for example, a touch panel or touch sensor, or a pointing device such as a mouse. The input device may be configured to include physical keys. The input device may be configured to include a voice input device such as a microphone.

[0021] Interface 14 is configured to include an output device that outputs information or data, etc. to the user. The output device may include, for example, a display device that outputs visual information such as images, characters, or graphics. The display device may be configured to include, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display or an inorganic EL display, or a PDP (Plasma Display Panel), etc. The display device is not limited to these displays and may be configured to include various other types of displays. The display device may be configured to include a light-emitting device such as an LED (Light Emission Diode) or an LD (Laser Diode). The display device may be configured to include various other devices. The output device may include, for example, an audio output device such as a speaker that outputs auditory information such as sound. The output device is not limited to these examples and may include various other devices.

[0022] The robot control device 10 may be configured as a server device. The server device may include at least one computer. The server device may be configured to execute parallel processing on a plurality of computers. The server device does not necessarily include a physical housing and may be configured based on virtualization technologies such as virtual machines or container orchestration systems. The server device may be configured using cloud services. When the server device is configured using cloud services, it may be configured by combining managed services. That is, the functions of the robot control device 10 can be realized as cloud services.

[0023] The server device may include at least one server group. The server group functions as the control unit 12. The number of server groups may be one or two or more. When the number of server groups is one, the functions realized by one server group include the functions realized by each server group. Each server group is communicably connected to each other by wire or wirelessly.

[0024] Although the robot control device 10 is described as one configuration in FIGS. 1 and 2, a plurality of configurations can be regarded as one system and operated as needed. That is, the robot control device 10 is configured as a platform with variable capacity. When one configuration becomes inoperable due to an unexpected event such as a natural disaster when a plurality of configurations are used as the robot control device 10, the system operation is continued using other configurations. In this case, each of the plurality of configurations is connected by a line regardless of whether it is wired or wireless and is configured to be communicable with each other. The plurality of configurations may be constructed across cloud services and an on-premises environment.

[0025] Also, the robot control device 10 is connected to the robot 2 or the database 20 by a line regardless of whether it is wired or wireless, for example. The robot control device 10, the database 20, or the robot 2 is equipped with a communication device using a standard protocol and can communicate bidirectionally.

[0026] <Database 20> Database 20 corresponds to a storage device configured separately from the robot control device 10. Database 20 may be configured to include an electromagnetic storage medium such as a magnetic disk, or may be configured to include a memory such as a semiconductor memory or a magnetic memory. Database 20 may be configured as an HDD or an SSD or the like. As will be described later, Database 20 stores information used to estimate the holding mode of the object to be held 8. In other words, the robot control device 10 registers information used to estimate the holding mode of the object to be held 8 in Database 20. Database 20 may be in the cloud. Even when Database 20 is in the cloud, the robot control device 10 may be at a site such as a factory. Note that the robot control device 10 and Database 20 may be integrally configured.

[0027] Database 20 may include at least one database group. The number of database groups may be one or two or more. The number of database groups may be appropriately increased or decreased based on the capacity of the data managed by the server device functioning as the robot control device 10 and the availability requirements required for the server device functioning as the robot control device 10. The database group may be connected to the server device functioning as the robot control device 10 or each server group so as to be communicable by wire or wirelessly.

[0028] (Operation example of the robot control system 1) The robot control system 1 controls the robot 2 by the robot control device 10 to cause the robot 2 to execute work. In the present embodiment, the work to be executed by the robot 2 includes an operation of holding the object to be held 8. The control unit 12 of the robot control device 10 determines the holding mode of the object to be held 8 by the robot 2. The control unit 12 controls the robot 2 so that the robot 2 holds the object to be held 8 in the determined holding mode. The holding mode includes the position where the robot 2 contacts the object to be held 8 when holding the object to be held 8, and the posture of the robot 2 such as an arm or an end effector when the robot 2 holds the object to be held 8.

[0029] The control unit 12 acquires information by recognizing the object to be held 8. The information acquired by recognizing the object to be held 8 is also referred to as recognition information. The control unit 12 is configured to extract candidates for the holding mode of the object to be held 8 based on the recognition information and the information registered in the database 20, and to be able to estimate the holding mode. The means for estimating the holding mode based on the recognition information and the information registered in the database 20 is also referred to as the first estimation means. Further, the control unit 12 is configured to be able to estimate the holding mode based on an inference model that receives the recognition information as an input and outputs an estimation result of the holding mode of the object to be held 8. The means for estimating the holding mode based on the inference model is also referred to as the second estimation means. When the control unit 12 estimates the holding mode using the first estimation means, it can estimate the holding mode at a faster processing speed or with a lighter calculation load than when using the second estimation means. When the control unit 12 estimates the holding mode using the second estimation means, it can estimate the holding mode with higher versatility than when using the first estimation means.

[0030] In the robot control system 1 according to the present embodiment, the control unit 12 attempts to estimate the holding mode by the first estimation means. When the control unit 12 can estimate the holding mode by the first estimation means, it controls the robot 2 so that the robot 2 holds the object to be held 8 in the holding mode estimated by the first estimation means. When the control unit 12 cannot estimate the holding mode by the first estimation means, it estimates the holding mode by the second estimation means, and controls the robot 2 so that the robot 2 holds the object to be held 8 in the holding mode estimated by the second estimation means.

[0031] <Information registered in the database 20> As described above, when the control unit 12 estimates the holding mode of the object to be held 8 using the first estimation means, it estimates the holding mode of the object to be held 8 based on the recognition information and the information registered in the database 20. The information registered in the database 20 is information referred to for estimating the holding mode, and is also referred to as reference information. The reference information includes information associating information representing the holding mode of an object with information regarding the object. The information representing the holding mode of the object is also referred to as holding mode information. The information regarding the object is also referred to as object information.

[0032] The control unit 12 collates the object information included in the reference information with the recognition information of the object to be held 8. When the reference information includes object information that matches or is similar to the recognition information (when it is registered in the database 20), the control unit 12 acquires the holding mode information associated with the object information that matches or is similar to the recognition information.

[0033] An example of the reference information is shown in FIG. 3. Hereinafter, the reference information will be described based on the example of FIG. 3.

[0034] <<Object Information>> The object information may include information for specifying the type of the object. The column of "ID" in the first column from the left in the table of FIG. 3 corresponds to the information for specifying the type of the object. The type of the object may be, for example, a specific article name such as a spring or a screw, or a model number or the like. The object information of a certain object includes one piece of information for specifying the type of the object.

[0035] The object information may include information for specifying the posture of the visible object, as shown in the second column from the left in the table of FIG. 3. When the viewpoint for viewing the object is fixed, the posture of the visible object corresponds to the posture of the object itself. Also, when the posture of the object is fixed, the posture of the visible object varies depending on the viewpoint from which the object is viewed (from which direction the object is viewed). The information for specifying the posture of the visible object is also referred to as posture information. The posture information includes information for specifying the posture of the object with respect to a fixed viewpoint, or information for specifying the viewpoint with respect to a fixed posture of the object. One object can be viewed in multiple ways. Therefore, one object can be viewed in multiple ways each specified by a respective one of a plurality of posture information. That is, the object information of a certain object includes one or more pieces of posture information.

[0036] In the table of FIG. 3, the posture information is represented as P_1, P_2, or P_n. In the present embodiment, the posture information can be represented by the coordinates of points on the spherical surface centered on an object such as the object 8 to be held, as shown in FIG. 4 for example, so as to specify the viewpoint with respect to the fixed posture of the object. The spherical surface centered on an object such as the object 8 to be held is also referred to as the information acquisition spherical surface 30. As examples of viewpoints for viewing the object, a point 31, a point 32, and a point 33 are shown on the information acquisition spherical surface 30. The posture information can be represented by the angles by which the object rotates around each of the X, Y, and Z axes when, for example, the viewpoint for viewing the object located at the center of the information acquisition spherical surface 30 in FIG. 4 is fixed to the point 31, so as to specify the posture of the object with respect to a fixed viewpoint.

[0037] The object information may include the type of the posture information, as shown in the third column from the left in the table of FIG. 3. The posture information may include reference posture information and other normal posture information. The reference posture information is information that identifies a point that is an important reference for calculating the position or direction at which the information acquisition unit 4 acquires the recognition information. The reference posture information may include the posture information extracted from the feature amounts used for estimating the posture information using the inference model. The reference posture information is posture information that is not similar to other reference posture information. The normal posture information is the posture information calculated later when a sufficient number of posture information is registered in the database 20. The normal posture information is information that is similar to at least one piece of reference posture information. The normal posture information is used auxiliary for fine adjustment of the posture information, or calculation of holding results including the success rate or frequency of holding, etc.

[0038] The object information may include information regarding the features of the object. The information regarding the features of the object is, for example, information regarding feature amounts representing the features of the object, as shown in the fourth column from the left in the table of FIG. 3. The feature amounts may differ depending on the appearance of the object such as the holding target 8. Therefore, when the object information includes a plurality of pieces of posture information, the object information includes information regarding the feature amounts corresponding to each of the plurality of pieces of posture information.

[0039] The feature quantity may be a set of feature points 8A extracted from an image representing the appearance of an object such as the object 8 to be held, as shown in FIG. 5 for example. In this case, the feature quantity may be, for example, numerical information indicating the number or distribution of the feature points 8A. Further, the feature quantity may be the image itself representing the appearance of the object. The image representing the appearance of the object may include images of the object in a specific posture taken from various viewpoints. In this case, the feature quantity may be, for example, pixel values constituting the image. The image representing the appearance of the object may include images of the object in various postures taken from a specific viewpoint. Further, the feature quantity may be a set of at least some points included in the point cloud information of the object measured from a certain viewpoint, or the point cloud information of the object itself. In this case, the feature quantity may be, for example, numerical information indicating the number or distribution of the point cloud, or numerical information indicating color or luminance included in the point cloud information. Further, the feature quantity may be data obtained by displaying the modeling data of the object as seen from a certain viewpoint. In the table of FIG. 3, the feature quantity of the object is represented as V_1, V_2 or V_n so as to correspond to each of the posture information (P_1, P_2 or P_n).

[0040] The feature quantity can be obtained by a method such as AKAZE (Accelerated-KAZE), ORB (Oriented FAST and Rotated BRIEF), or SIFT (Scale-Invariant Feature Transform), but is not limited thereto and may be represented by various other methods. However, in one piece of object information, the information indicating the feature quantity needs to be stored in the same format. In the database 20, the feature quantity itself may be registered in its original format, or may be registered as information indicating the feature quantity in another format. As another format, for example, the feature quantity may be restored to a 3D model using a technology such as VSLAM (Visual Simultaneous Localization and Mapping) and then registered in the database 20.

[0041] <<Retention mode information>> The holding mode information specifies the holding mode of an object in a certain view. The holding mode information may be configured to specify the holding mode with five parameters of [x, y, w, h, θ] as shown in the fifth column from the left in the table of FIG. 3. x and y represent the coordinates of the holding position in the plane corresponding to the view of the object. That is, x and y represent the position coordinates of the end effector 2B when holding the object. w and h represent the finger width and interval of the end effector 2B to be held. θ represents the angle of the finger of the end effector 2B with respect to the object. In other words, the holding mode may be represented in a format that specifies the center position and angle of the finger when holding, as well as the opening width of the finger.

[0042] The holding mode information may be configured to specify the holding mode of the object in the form of 6DoF (Degrees of Freedom). The 6DoF form represents the holding position as three-dimensional coordinates (x, y, z) and represents the posture of the finger holding the object as the rotation angles (θx, θy, θz) about each axis of XYZ. That is, the holding mode information may be configured to specify the holding mode with six parameters of [x, y, z, θx, θy, θz] in the 6DoF form. The holding mode information may be represented in a form that shows the range where the fingers of the end effector 2B exist as a polygon as the grasping rectangle angle.

[0043] The holding mode information may include the grasping force when holding the object by grasping. The holding mode information may include information specifying the posture of the end effector 2B or fingers when holding the object. The holding mode information is not limited to these and may include information specifying various aspects related to holding.

[0044] The holding mode information can be different information depending on the view of the same object. Therefore, the holding mode information is associated with the posture information. That is, the holding mode information is associated with the object information. Also, when one object information includes a plurality of posture information, the holding mode information can be associated with at least some of the posture information. In other words, one or more holding mode information can be associated with one object information.

[0045] The holding mode of an object seen in a certain view is not limited to only one mode but can be two or more modes. Therefore, one or more holding mode information can be associated with one posture information. In the table illustrated in FIG. 3, three holding mode information of [x, y, w, h, θ]_1 to 3 are associated with one way of seeing an object (P_1). Also, n holding mode information of [x, y, w, h, θ]_1 to n are associated with one way of seeing an object (P_2).

[0046] For example, as shown in FIG. 6, assuming that the holding mode information specifying the first candidate mode 41, the second candidate mode 42, and the third candidate mode 43 as candidates for the mode of holding an object such as the holding object 8 is associated with the posture information of the holding object 8 shown in the photograph of FIG. 6. The first candidate mode 41 corresponds to the positions of two fingers when holding the position near the center of the cylindrical spring as the holding object 8 so as to sandwich it in the diameter direction of the cylindrical spring (the short side direction of the spring that looks like a rectangle in plan view). The second candidate mode 42 corresponds to the positions of two fingers when holding a position deviated from the center of the holding object 8 so as to sandwich it in the diameter direction. The third candidate mode 43 corresponds to the positions of two fingers when holding the cylindrical spring as the holding object 8 so as to sandwich it in the axial direction of the cylindrical spring (the long side direction of the spring that looks like a rectangle in plan view). In the holding mode information specifying each candidate mode, x and y represent the coordinates of the midpoint of the positions of the two fingers in the image plane representing the way of seeing the object as the holding object 8. w and h represent the width of the finger and the interval between the two fingers. θ represents the angle by which the direction in which the two fingers are aligned is rotated around the normal line of the image plane. Assuming that the view represented by P_1 in the table illustrated in FIG. 3 corresponds to the image illustrated in FIG. 6, the three holding mode information ([x, y, w, h, θ]_1 to 3) associated with P_1 can be information representing the first candidate mode 41 to the third candidate mode 43, respectively.

[0047] The holding mode information may include the success rate when the object is held in each holding mode, as shown in the sixth column from the left in the table of FIG. 3. The success rate is calculated based on whether the object was successfully held or failed when the object was actually held in various holding modes for each appearance of the object. As will be described later, the control unit 12 may estimate the holding mode based on the success rate.

[0048] <Acquisition of Recognition Information> The control unit 12 acquires the recognition information of the object 8 to be held in order to estimate the holding mode of the object 8 to be held. The recognition information may include information regarding the characteristics of the object 8 to be held. The control unit 12 may, for example, acquire information regarding the feature point 8A of the object 8 to be held as information regarding the characteristics of the object 8 to be held. Specifically, the feature amount of the object 8 to be held may be acquired. The control unit 12 may acquire an image of the object 8 to be held taken by a camera, extract the feature point 8A from the image, and acquire it as recognition information including the feature amount of the object 8 to be held. The control unit 12 may acquire the point cloud information of the object 8 to be held detected by the depth sensor, extract the feature point 8A from the point cloud information, and acquire it as recognition information including the feature amount of the object 8 to be held. The control unit 12 may also acquire the image or point cloud information of the object 8 to be held as recognition information including the feature amount of the object 8 to be held.

[0049] As will be described later, the control unit 12 queries the reference information registered in the database 20 to determine whether object information that matches or is similar to the recognition information is registered in the database 20. The control unit 12 may acquire the recognition information in a format comparable to the object information. For example, when the object information includes a feature amount, the control unit 12 may acquire the feature amount of the object 8 to be held as the recognition information. In this case, the control unit 12 may determine whether object information that matches or is similar to the recognition information is registered in the database 20 by comparing the feature amount of the object information with the feature amount of the recognition information. Further, when the object information includes posture information, the control unit 12 may acquire the posture information of the object 8 to be held as the recognition information. Note that the posture information may include information regarding the characteristics of the posture of the object 8 to be held.

[0050] When the position or direction at which the information acquisition unit 4 acquires the recognition information of the object to be held 8 is fixed, the control unit 12 may estimate the posture of the object to be held 8 based on the recognition information of the object to be held 8 and generate posture information. The control unit 12 may estimate the posture of the object to be held 8 based on information regarding the characteristics of the object to be held 8 and information specifying the position of the information acquisition unit 4, and generate posture information.

[0051] The control unit 12 may acquire, as recognition information of the object to be held 8, posture information specifying the position or direction at which the information acquisition unit 4 acquires the recognition information of the object to be held 8. The control unit 12 may estimate the posture information of the position or direction at which the information acquisition unit 4 acquires the recognition information based on the recognition information of the object to be held 8 and generate posture information. The control unit 12 may estimate the posture information of the position or direction at which the information acquisition unit 4 acquires the recognition information using a posture estimation method such as epipolar geometry.

[0052] The control unit 12 may finely adjust the estimation result of the posture information of the position or direction at which the information acquisition unit 4 acquires the recognition information. The control unit 12 may finely adjust the estimation result of the posture information by a method using peripheral shooting points. In this case, the control unit 12 extracts the posture information around the estimated posture information from the posture information registered in the database 20, and finely adjusts the posture information by estimating the posture information using the feature amounts associated with the extracted posture information. Further, the control unit 12 may finely adjust the estimation result of the posture information by a method using all shooting points. In this case, the control unit 12 finely adjusts the estimation result of the posture information while searching all the posture information registered in the database 20.

[0053] <Determination of the holding mode by querying reference information> The control unit 12 queries the reference information registered in the database 20 and compares the recognition information with the object information included in the reference information to determine whether object information that matches or is similar to the recognition information is registered in the database 20. When the object information includes feature amounts, the control unit 12 may compare the feature amounts acquired as recognition information with the feature amounts included in the object information.

[0054] Also, when the object information includes the posture information, the control unit 12 may compare the posture information of the object 8 to be held acquired as the recognition information with the posture information included in the object information. When the posture information includes the feature amount of the posture, the control unit 12 may compare the feature amount acquired as the recognition information with the feature amount included in the object information.

[0055] <<Estimation of the holding mode using the database 20>> When there is reference information including object information that matches or is similar to the recognition information among the reference information registered in the database 20, the control unit 12 determines that the holding mode of the object 8 to be held can be estimated using the database 20. The control unit 12 acquires the holding mode information associated with the object information that matches or is similar to the recognition information from the database 20.

[0056] <<<When there is object information that matches the recognition information>>> When the reference information including the object information that matches the recognition information is registered in the database 20, the control unit 12 acquires the holding mode information associated with the object information that matches the recognition information. The control unit 12 may estimate the holding mode specified by the acquired holding mode information as the holding mode of the object 8 to be held. When a plurality of holding mode information is associated with the object information that matches the recognition information, the control unit 12 acquires the plurality of holding mode information. The control unit 12 may select one piece of holding mode information from the plurality of holding mode information and estimate the holding mode specified by the selected holding mode information as the holding mode of the object 8 to be held. The control unit 12 may select one piece of holding mode information, for example, based on the holding results associated with each of the plurality of holding mode information. The control unit 12 may select the holding mode information with the best holding result. The control unit 12 may select the holding mode information with the highest success rate as the holding mode information with the best holding result. The control unit 12 may select the holding mode information with the highest holding frequency as the holding mode information with the best holding result.

[0057] For example, the control unit 12 acquires the probability (success rate) of successful retention when retained in each candidate mode illustrated in FIG. 6. Assume that the success rate when retained in the first candidate mode 41 is 90%. Assume that the success rate when retained in the second candidate mode 42 is 60%. Assume that the success rate when retained in the third candidate mode 43 is 40%. The control unit 12 may select retention mode information that identifies the candidate mode with a high success rate. In this case, the control unit 12 may select the retention mode information that identifies the first candidate mode 41. The control unit 12 may acquire the retention frequency instead of the success rate and select the retention mode information that identifies the candidate mode with a high retention frequency.

[0058] <<<When there is object information similar to the recognition information>>> When reference information including object information similar to the recognition information is registered in the database 20, the control unit 12 acquires the retention mode information associated with the object information similar to the recognition information. In other words, the control unit 12 may be configured to search the database 20 for object information similar to the recognition information and extract the retention mode associated with the searched object information.

[0059] When the numerical value representing the difference between the recognition information and the object information is less than a predetermined threshold, the control unit 12 may determine that the recognition information and the object information are similar. For example, the control unit 12 calculates the difference between the posture information of the object 8 to be retained and the posture information included in the object information as a numerical value, and when the calculated numerical value is less than a predetermined threshold, the control unit 12 may determine that the recognition information and the object information are similar. When the posture information is represented as parameters in a spherical coordinate system, the control unit 12 may calculate the difference between the parameters of the posture information of the object 8 to be retained and the parameters of the posture information included in the object information. When the posture information is represented as a rotation angle around a predetermined axis, the control unit 12 may calculate the difference between the rotation angle representing the posture information of the object 8 to be retained and the rotation angle representing the posture information included in the object information.

[0060] Specifically, the control unit 12 may calculate the difference between the feature amount of the object to be held 8 and the feature amount included in the object information as a numerical value, and determine that the recognition information and the object information are similar when the calculated numerical value is less than a predetermined threshold. When the feature amount is a set of feature points, the control unit 12 may calculate the difference between the number of feature points of the object to be held 8 and the number of feature points included in the object information. The control unit 12 may also calculate the difference between the coordinates of the feature points of the object to be held 8 and the coordinates of the feature points included in the object information. When the feature amount is point cloud information representing the object to be held 8, the control unit 12 may calculate the difference between the number of points included in the point cloud information representing the object to be held 8 and the number of points included in the point cloud information representing the object in the object information. The control unit 12 may also calculate the difference between the coordinates of each point included in the point cloud information representing the object to be held 8 and the coordinates of each point included in the point cloud information representing the object in the object information. When the feature amount is an image of the object, the control unit 12 may calculate the difference between the image of the object to be held 8 and the image of the object specified by the object information.

[0061] When the number of object information similar to the recognition information is one, the control unit 12 acquires the holding mode information associated with the one object information. The control unit 12 may estimate the holding mode specified by the acquired holding mode information as the holding mode of the object to be held 8. When a plurality of holding mode information is associated with one object information, the control unit 12 may select one holding mode information from the plurality of holding mode information, and estimate the holding mode specified by the selected holding mode information as the holding mode of the object to be held 8.

[0062] When a plurality of object information is similar to the recognition information, the control unit 12 may acquire the holding mode information associated with each of the plurality of object information. When the control unit 12 acquires a plurality of holding mode information, the control unit 12 may select one holding mode information from the plurality of holding mode information, and estimate the holding mode specified by the selected holding mode information as the holding mode of the object to be held 8.

[0063] When the control unit 12 selects one piece of holding mode information from a plurality of pieces of holding mode information, regardless of whether the number of object information is one or plural, it may select one piece of holding mode information based on the success rates associated with each of the plurality of pieces of holding mode information. The control unit 12 may select the holding mode information with the highest success rate.

[0064] As described above, even when the object information that matches the recognition information is not registered in the database 20, the control unit 12 may estimate the holding mode based on the holding mode information associated with the object information similar to the recognition information. Even when the object information that matches the recognition information is not registered in the database 20, the control unit 12 may estimate the holding mode information associated with the object information that matches the recognition information based on a plurality of pieces of object information similar to the recognition information and the holding mode information associated with each of the object information.

[0065] For example, when the recognition information of the object 8 to be held includes attitude information, the control unit 12 determines whether the attitude information in the vicinity of the attitude information of the object 8 to be held is registered in the database 20 on the information acquisition spherical surface 30. The control unit 12 may determine, for example, the attitude information that specifies a point located within a predetermined distance (within a predetermined range) from the point specified by the attitude information on the information acquisition spherical surface 30 as the attitude information in the vicinity. The attitude information in the vicinity is also referred to as reference attitude information.

[0066] When the attitude information (reference attitude information) in the vicinity of the attitude information of the object 8 to be held is registered in the database 20, the control unit 12 acquires the holding mode information associated with each of the plurality of pieces of reference attitude information from the database 20. The control unit 12 can interpolate and generate the holding mode information estimated to be associated with the attitude information of the object that matches the attitude information of the object 8 to be held based on the holding mode information associated with the reference attitude information. The control unit 12 can estimate the holding mode in the attitude specified by the attitude information of the object 8 to be held based on the holding mode information generated by interpolation.

[0067] Hereinafter, a specific example of interpolation will be described with reference to FIG. 7. The control unit 12 acquires recognition information generated by recognizing the object 8 to be held from the point 34 on the information acquisition spherical surface 30. The recognition information includes attitude information for specifying the point 34. Here, it is assumed that reference information including the attitude information for specifying the point 34 is not registered in the database 20. On the other hand, it is assumed that reference information including the attitude information for specifying the point 35 located in the vicinity of the point 34 is registered. In other words, although the attitude information that matches the attitude information of the object 8 to be held is not registered in the database 20, it is assumed that the attitude information (reference attitude information) in the vicinity of the attitude information of the object 8 to be held is registered in the database 20.

[0068] The control unit 12 acquires holding mode information associated with object information including the attitude information for specifying each of the four points 35. It is assumed that the attitude information for specifying the point 35 is associated with three holding mode information for specifying the first candidate mode 41, the second candidate mode 42, and the third candidate mode 43 in FIG. 6, respectively. In this case, it is estimated that the three holding mode information for specifying the first candidate mode 41, the second candidate mode 42, and the third candidate mode 43, respectively, are associated with the attitude information for specifying the point 34. The control unit 12 can select any one of the first candidate mode 41, the second candidate mode 42, and the third candidate mode 43 as the holding mode of the object 8 to be held.

[0069] The control unit 12 may select a holding mode of the object to be held 8 from the first candidate mode 41, the second candidate mode 42, and the third candidate mode 43 based on the success rate of holding for each mode. For example, assume that the holding success rates of the first candidate mode 41 at each of the four points 35 are 90%, 90%, 100%, and 80%. In this case, the control unit 12 may consider that the holding success rate of the first candidate mode 41 at the point 34 is 90% which is the average of the holding success rates at each of the four points 35. Also, assume that the holding success rates of the second candidate mode 42 at each of the four points 35 are 60%, 70%, 70%, and 70%. In this case, the control unit 12 may consider that the holding success rate of the second candidate mode 42 at the point 34 is 67% which is the average of the holding success rates at each of the four points 35. Also, assume that the holding success rates of the third candidate mode 43 at each of the four points 35 are 40%, 30%, 40%, and 0%. In this case, the control unit 12 may consider that the holding success rate of the third candidate mode 43 at the point 34 is 37% which is the average of the holding success rates at each of the four points 35.

[0070] Even when the control unit 12 acquires information from the point 34, if the holding success rate of the first candidate mode 41 is considered to be higher than the holding success rates of the second candidate mode 42 and the third candidate mode 43, the control unit 12 may determine the first candidate mode 41 as the holding mode of the object to be held 8. Alternatively, the control unit 12 may determine the holding mode by regarding the holding success rate of any one of the points 35 located in the vicinity of the point 34 as the holding success rate of the point 34 as it is.

[0071] As described above, even when the object information that matches the recognition information is not registered in the database 20, the control unit 12 can interpolate using the object information in the vicinity of the object information that matches the recognition information. By doing so, it becomes easier to determine the holding mode of the object to be held 8 based on the database 20.

[0072] When a plurality of object information that matches or is similar to the recognition information is registered in the database 20, the control unit 12 may acquire the holding mode information associated with each object information. The control unit 12 may select one piece of object information from among the plurality of object information and acquire the holding mode information associated with the selected object information. The control unit 12 may calculate the degree of match between the recognition information and each of the plurality of object information, select the object information with a high degree of match, and acquire the holding mode information associated with the selected object information.

[0073] When a plurality of holding mode information is associated with object information that matches or is similar to the recognition information, the control unit 12 may select and acquire one piece of holding mode information from among the plurality of holding mode information. The control unit 12 may select the holding mode information based on the success rate associated with the holding mode information. For example, in the reference information shown in the table of FIG. 3, the object information indicating that the object to be held 8 is a spring and the appearance of the spring is P_1 is associated with three pieces of holding mode information represented as [x, y, w, h, θ]_1 to 3, respectively. Here, the success rates associated with each holding mode information are 0%, 20%, and 50%, respectively. The control unit 12 may select the holding mode information represented as [x, y, w, h, θ]_3, which is associated with the highest success rate (50%) among these.

[0074] The control unit 12 estimates the holding mode of the object to be held 8 based on the acquired holding mode information. When the control unit 12 acquires one piece of holding mode information, it may estimate the mode specified by the holding mode information as the holding mode of the object to be held 8. When the control unit 12 acquires one or more pieces of holding mode information, it may estimate the holding mode of the object to be held 8 based on the holding mode information. When the control unit 12 acquires a plurality of pieces of holding mode information, it may estimate the holding mode of the object to be held 8 based on the success rate associated with each holding mode information. The control unit 12 may select one piece of holding mode information from among the plurality of holding mode information based on the success rate and estimate the holding mode of the object to be held 8 based on the selected holding mode information.

[0075] As described above, in the estimation of the holding mode using the database 20, the reference information has the posture information for each of a plurality of objects and the holding mode information related to the posture information. The control unit 12 may acquire the posture information of the object 8 to be held as recognition information, and estimate the holding mode of the object 8 to be held based on the acquired posture information of the object 8 to be held. Further, the control unit 12 may estimate the holding mode of the object 8 to be held based on at least one piece of holding target information related to the reference posture information similar to the posture information of the object 8 to be held. Further, the control unit 12 may estimate the holding mode based on a plurality of pieces of holding mode information related to each of a plurality of reference posture information similar to the posture information. Further, when a plurality of pieces of holding mode information are associated with one piece of reference posture information, the control unit 12 may select the holding mode information having the highest success rate among the plurality of pieces of holding mode information, and estimate the holding mode of the object 8 to be held.

[0076] <<Estimation of Holding Mode Using Inference Model>> When the object information that matches or is similar to the recognition information is not registered in the database 20, the control unit 12 estimates the holding mode of the object 8 to be held using the inference model. In other words, when the object information similar to the recognition information is not registered in the database 20, the control unit 12 estimates the holding mode of the object 8 to be held by the inference model. The inference model is configured to receive the recognition information as an input and output an estimation result of the holding mode of the object 8 to be held. The inference model may include a model generated by machine learning such as deep learning. The inference model may be configured to estimate the holding mode based on the technology of AI (Artificial Intelligence). The inference model may be configured to estimate the holding mode based on rules. When configured to estimate the holding mode based on rules, the executability or certainty of holding can be improved as compared with AI. The inference model may use a 3D model. The inference model may be configured to estimate the holding mode by various methods not limited to these examples. Further, the robot control device 10 may have different types of inference models.

[0077] The input information such as recognition information used in the estimation of the object to be held based on database queries (first estimation means) and the estimation of the holding mode based on the inference model (second estimation means) may be common. When the holding mode is estimated by the inference model, the inference model may be configured to output the estimation result in a format comparable to the input information to the database query or the inference model. Specifically, the estimation result output from the inference model can be handled as reference information in the database query, and in this embodiment, it is in a data format that enables comparison of the feature amounts of the input information and the reference information in the database query.

[0078] <<Determination of the holding mode based on the estimation result>> The control unit 12 determines the holding mode of the object to be held 8 based on the estimation result of the holding mode using the database 20 or the estimation result of the holding mode using the inference model. The control unit 12 may determine the estimation result of the holding mode as the holding mode as it is. The control unit 12 may generate a holding mode based on the estimation result of the holding mode and determine the generated mode as the holding mode. The control unit 12 may also determine, as the holding mode, a mode obtained by modifying or changing the estimation result of the holding mode. When the formats of the holding modes estimated using the database 20 and the inference model are different, the control unit 12 may convert them to the same format. Further, the control unit 12 may convert the holding mode estimated using the database 20 or the inference model to match the format of the holding mode used to control the robot 2.

[0079] <Operation based on the determined holding mode> The control unit 12 controls the robot 2 so that the robot 2 holds the object to be held 8 in the determined holding mode. The control unit 12 acquires, as the holding result, whether the holding by the robot 2 was successful.

[0080] <<Registration of the holding result>> When the control unit 12 estimates the holding mode of the object to be held 8 from the inference model, it may be configured to register the estimation result in the database 20. Further, the control unit 12 controls the robot 2 based on the estimation result of the holding mode of the object to be held 8 by the inference model, and when the holding of the object to be held 8 is successful in the holding mode corresponding to the adopted estimation result, it may be configured to register the estimation result in the database 20. For example, when the holding of the object to be held 8 is successful in the holding mode determined based on the estimation result of the holding mode obtained from the database 20, the control unit 12 may update the success rate associated with the holding mode information registered in the database 20. When the holding of the object to be held 8 is successful in the holding mode determined based on the estimation result of the holding mode obtained from the inference model, the control unit 12 may generate reference information associating the holding mode information representing the successful holding mode with the object information of the object to be held 8 and register it in the database 20. In this case, since the database 20 is constructed based on the successful holding mode, the reliability of the estimation result of the holding mode by the introduction of the database 20 can be improved.

[0081] Each time the control unit 12 obtains the result of successful holding for one object to be held 8, it may register the holding mode information representing the successful holding mode in the database 20. The control unit 12 may summarize the results of successful holding of a plurality of objects to be held 8 and register the holding mode information representing the successful holding mode in the database 20, or may summarize the results of successful holding of one object to be held 8 a plurality of times and register the holding mode information representing the successful holding mode in the database 20. The control unit 12 may extract some results based on the number of posture information corresponding to the result of successful holding or the density of the points represented by the posture information on the information acquisition spherical surface 30, and register in the database 20 the holding mode information including only the posture information corresponding to the extracted results.

[0082] If the holding fails in the mode specified by the holding mode information registered in the database 20, the control unit 12 may update it to reduce the success rate associated with the holding mode information. Even if the holding fails in a mode not registered in the database 20, the control unit 12 may register in the database 20 the success rate associated with the holding mode information specifying that mode as 0%.

[0083] When the control unit 12 determines the holding mode for the posture information and obtains the holding result, and there is no posture information that matches or is similar to the recognition information registered in the database 20, the control unit 12 may register that posture information in the database 20 as the reference posture information. Note that the control unit 12 may register a plurality of pieces of reference posture information. When the control unit 12 determines the holding mode for the posture information in which posture information similar to the recognition information is registered in the database 20 and obtains the holding result, the control unit 12 may register that posture information in the database 20 as the normal posture information. Note that the control unit 12 may register a plurality of pieces of normal posture information.

[0084] <<Determination of Holding Success>> The control unit 12 may determine whether the holding has succeeded based on the detection result of the sensor of the robot 2. The control unit 12 may estimate the holding state of the object 8 to be held by the end effector 2B. The state in which the end effector 2B is normally holding the object 8 is also referred to as the holding normal state. It is assumed that when the end effector 2B can hold the object 8 so that the object 8 does not slip from the gripper or finger, the holding state is estimated to be the holding normal state. Conversely, it is assumed that when the object 8 slips or falls from the gripper or finger, the holding state is not the holding normal state.

[0085] The control unit 12 may estimate the holding state based on the position information of the end effector 2B. For example, when the position where the gripper or finger of the end effector 2B holds the object 8 to be held is within a predetermined distance from the position defined in the holding mode, the control unit 12 may estimate that the holding state is the normal holding state. Also, when the force or torque acting on the end effector 2B detected by the contact force sensor or the force sensor is within a predetermined range, the control unit 12 may estimate that the holding state is the normal holding state.

[0086] Even if the position where the gripper or finger of the end effector 2B holds the object 8 to be held is within a predetermined distance from the position defined in the holding mode, if the contact force sensor or the force sensor does not detect the force or torque acting on the end effector 2B or the like, the control unit 12 may estimate that the holding state is not the normal holding state. Also, when the value of the force or torque detected by the contact force sensor or the force sensor is outside the predetermined range, the control unit 12 may estimate that the holding state is not the normal holding state. Conversely, when the conditions for estimating that the holding state is not the normal holding state are not satisfied, the control unit 12 may estimate that the holding state is the normal state.

[0087] While the end effector 2B is holding the object 8 (from the start to the end of holding), the control unit 12 may continuously estimate the holding state, may estimate the holding state at a predetermined interval, or may estimate the holding state at irregular timings. When the control unit 12 continuously estimates that the holding state is the normal holding state while the end effector 2B is holding the object 8, the control unit 12 may determine that the holding has succeeded. When the control unit 12 estimates that the holding state is not the normal holding state one or more times while the end effector 2B is holding the object 8, the control unit 12 may determine that the holding has not succeeded. Conversely, when the control unit 12 does not estimate that the holding state is not the normal holding state while the end effector 2B is holding the object 8, the control unit 12 may determine that the holding has succeeded.

[0088] <Example of the procedure of the robot control method> The control unit 12 may execute the operations described above as a robot control method including the procedures of the flowchart illustrated in FIG. 8. The robot control method may be realized as a robot control program to be executed by a processor constituting the control unit 12. The robot control program may be stored in a non-transitory computer-readable medium.

[0089] The control unit 12 acquires recognition information of the object 8 to be held (step S1). The control unit 12 queries the reference information (step S2). The control unit 12 determines whether it is possible to estimate the holding mode using the database 20 based on the result of querying the reference information (step S3).

[0090] When the control unit 12 determines that it is possible to estimate the holding mode using the database 20 (step S3: YES), it estimates the holding mode using the database 20 (step S4). In this case, the control unit 12 estimates the holding mode based on the holding mode information acquired from the database 20. After the procedure of step S4, the control unit 12 proceeds to the procedure of step S6.

[0091] When the control unit 12 determines that it is not possible to estimate the holding mode using the database 20 (step S3: NO), it estimates the holding mode using the inference model (step S5). In this case, the control unit 12 inputs the recognition information to the inference model and acquires the estimation result of the holding mode from the inference model, thereby estimating the holding mode. After the procedure of step S5, the control unit 12 proceeds to the procedure of step S6.

[0092] The control unit 12 determines the holding mode of the object 8 to be held based on the holding mode estimated by the database 20 or the inference model (step S6). The control unit 12 controls the robot 2 so as to execute the holding operation on the robot 2 in the determined holding mode (step S7). The control unit 12 acquires the holding result by the robot 2 (step S8). The control unit 12 registers new reference information or updated reference information in the database 20 (step S9). After executing the procedure of step S9, the control unit 12 ends the execution of the procedure of the flowchart in FIG. 8.

[0093] <Small brackets> As described above, according to the robot control device 10 or the robot control method according to the present embodiment, the holding position is determined based on the comparison between the information of the object registered in the database 20 and the recognition information. On the other hand, even when the information that matches or is similar to the recognition information is not registered in the database 20, the holding position is determined based on the inference model. That is, the control unit 12 estimates the holding mode of the object 8 to be held using the first estimation means, and when the first estimation means cannot estimate, it uses the second estimation means to estimate the holding mode of the object 8 to be held.

[0094] Here, as a comparative example, when the information that matches or is similar to the recognition information is not registered in the database, a method of registering the recognition information in the database and then determining the holding position can be considered. However, a great deal of workload and time are required to newly register the recognition information in the database. Therefore, in the comparative example, the holding position of the object to be held that is not registered in the database cannot be easily determined.

[0095] On the other hand, the control unit 12 of the robot control device 10 according to the present embodiment is configured to be able to estimate the holding mode of the object to be held 8 by means of a database 20 storing reference information including object information of a plurality of objects and holding mode information of the plurality of objects, and each of the inference models capable of estimating the holding mode of the object. Further, the control unit 12 is configured to control the robot 2 based on the estimated holding mode. When the control unit 12 acquires the recognition information of the object to be held 8 and determines that the holding mode of the object to be held 8 based on the recognition information cannot be estimated from the database 20, the control unit 12 estimates the holding mode by means of the inference model. In other words, according to the robot control device 10 or the robot control method according to the present embodiment, the holding position of the object to be held 8 not registered in the database 20 can be determined based on the inference model. Further, the holding position of the object to be held 8 can be easily determined by changing the algorithm for determining the holding position. By doing so, both an improvement in the speed of the estimation process of the holding mode and ensuring the versatility of the estimation process of the holding mode can be achieved. As a result, the feasibility of holding can be improved. Further, the convenience of the robot 2 for holding the object to be held 8 can be improved.

[0096] Note that, in the above example, an example is described in which when the holding mode of the object to be held 8 cannot be estimated by the first estimation means, the second estimation means is used to estimate the holding mode of the object to be held 8. The present invention is not limited to this example, and when the object to be held 8 cannot be held in accordance with the holding mode estimated by the first estimation means, the second estimation means may estimate the object to be held.

[0097] (Other Embodiments) Other embodiments will be described below.

[0098] <Method of Referring to the Category of the Object to be Held 8> The control unit 12 may determine whether it is possible to estimate the holding mode based on the database 20 by referring to the category of the object to be held 8. For example, the holding mode may differ depending on whether the object to be held 8 is a bolt or a spring. Also, the holding mode may differ depending on whether the object to be held 8 is a bolt that is an industrial part or a ballpoint pen that is a writing instrument. However, the control unit 12 may not be able to identify the type of the object to be held 8 based only on the feature amount of the object to be held 8. If the control unit 12 recognizes the object to be held 8 as another object such as a spring or a ballpoint pen even though the object to be held 8 is a bolt, the estimated holding mode may not be an appropriate mode. Therefore, by referring to the category of the object to be held 8, the recognition accuracy of the type of the object to be held 8 can be improved. As a result, the object to be held 8 can be held appropriately.

[0099] In other words, the reference information may include category information indicating the category of each of the plurality of objects. The control unit 12 may be configured to be able to estimate the holding mode from the database 20 based on the object information having the category information retrieved from the reference information stored in the database 20 after acquiring the category information of the object to be held 8. Also, the control unit 12 may be configured to estimate the holding mode of the object to be held 8 by an inference model when the category information of the object to be held 8 is not registered in the database 20.

[0100] Specifically, the control unit 12 may acquire, as recognition information of the object to be held 8, not only the feature amount of the object to be held 8 but also information for specifying the category of the object to be held 8. The information for specifying the category of the object to be held 8 is also referred to as category information. The category may correspond to the type of the object to be held 8. The category information may include, for example, classification information indicating that the object to be held 8 is an industrial part or a writing instrument. The category information may include, for example, individual information constituting the classification information. The individual information refers to, for example, bolts or nuts if the classification information is an industrial part, or pencils or erasers if the classification information is a writing instrument. The control unit 12 may acquire the classification information or may acquire the individual information. The category information may be the name of the object or may be an ID or number assigned to the object. The category information may correspond to the cell of the ID in the first column from the left in the table of FIG. 3 exemplifying the reference information. Note that the category information may be acquired through user input.

[0101] When the control unit 12 acquires the category information of the object to be held 8, it queries the reference information in the database 20 and determines whether reference information including the same category information is registered in the database 20. When the same category information as that of the object to be held 8 is not registered in the database 20, the control unit 12 determines that the holding mode of the object to be held 8 cannot be estimated by the database 20 and may estimate the holding mode by an inference model. When the same category information as that of the object to be held 8 is registered in the database 20, the control unit 12 may continue to collate the feature amount of the object to be held 8 with the feature amount included in the reference information. When reference information including a feature amount that matches or is similar to the feature amount of the object to be held 8 is registered in the database 20, the control unit 12 may estimate the holding mode by the database 20. When reference information including a feature amount that matches or is similar to the feature amount of the object to be held 8 is registered in the database 20, the control unit 12 may estimate the holding mode by an inference model.

[0102] When AI or template matching is used in advance to recognize the object to be held 8, the control unit 12 may estimate information such as a label or category indicating what kind of object the object to be held 8 is. That is, the control unit 12 may estimate the category information of the object to be held 8. The category information may be included in the recognition information of the object to be held 8. When the control unit 12 acquires the recognition information of the object to be held 8, the control unit 12 may determine whether the recognition of the object to be held 8 using AI or template matching or the like has been executed in advance by using the category information included in the recognition information. When the recognition of the object to be held 8 using AI or template matching or the like has been executed in advance, the control unit 12 may proceed to the collation of feature amounts. When the recognition of the object to be held 8 using AI or template matching or the like has not been executed in advance, the control unit 12 may proceed to the estimation of the object to be held using the inference model.

[0103] As a procedure for the control unit 12 to refer to the category of the object to be held 8, the procedure illustrated in the flowchart of FIG. 9 may be executed before the robot control method including the procedure illustrated in the flowchart of FIG. 8.

[0104] The control unit 12 acquires the category information of the object to be held 8 (step S11). The control unit 12 determines whether the acquired category information is registered in the database 20 (step S12). When the acquired category information is not registered in the database 20 (step S12: NO), the control unit 12 determines that the holding mode cannot be estimated using the database 20, and proceeds to the procedure of estimating the holding mode using the inference model in step S5 of FIG. 8. When the acquired category information is registered in the database 20 (step S12: YES), the control unit 12 acquires the feature amounts of the object to be held 8 (step S13), and proceeds to the procedure of inquiring the reference information in step S2 of FIG. 8.

[0105] If the recognition information of the object 8 to be held can match or be similar to the object information registered in the database 20, specifically, for example, if the feature amount of the object 8 to be held can match or be similar to a plurality of feature amounts included in the object information registered in the database 20, the accuracy of the collation by the feature amount may decrease. By collating the category of the object 8 to be held, the accuracy of the collation by the feature amount can be enhanced.

[0106] <Clustering> The control unit 12 may integrate a plurality of pieces of reference information that are similar to each other and registered in the database 20. For example, the control unit 12 may be configured to cluster the reference information registered in the database 20 and execute an integration process for at least two pieces of reference information included in each cluster. The integration process may include a deletion process for the reference information before integration. When the reference information includes reference posture information, the control unit 12 may be configured to cluster the reference posture information registered in the database 20 and execute an integration process for at least two pieces of reference posture information included in each cluster. The integration process may include a deletion process for the reference posture information before integration. By the integration process, the data amount of the database 20 can be reduced. Also, the workload when querying the database 20 can be reduced. The integrated reference information is also referred to as integrated information. The integrated reference posture information is also referred to as integrated posture information. It can be said that the control unit 12 generates integrated information or integrated posture information by integrating the reference information or the reference posture information.

[0107] As illustrated in FIG. 10, assume that points 36, 37, 38, and 39 corresponding to the attitude information are located on the information acquisition spherical surface 30. Point 36 is represented by a solid-line circle. Point 37 is represented by a solid-line triangle. Point 38 is represented by a solid-line quadrilateral. Point 39 is represented by a dashed-line circle in the sense that it is located on the back side of the information acquisition spherical surface 30. In this example, the control unit 12 may cluster points 36, 37, 38, and 39 into different clusters respectively. The control unit 12 may perform clustering of points corresponding to the attitude information based on various algorithms such as, for example, the k-means method or the Gaussian mixture model.

[0108] Among the five points 36 clustered into one cluster, the control unit 12 may determine a point 36C that represents the cluster. By deleting the four points 36 other than the point 36C from the database 20, the control unit 12 may integrate the points included in the cluster into one. The point 36C is represented by a black-filled circle. Similarly for other clusters, the control unit 12 may determine points 37C, 38C, and 39C that represent each cluster. The point 37C is represented by a black-filled triangle. The point 38C is represented by a black-filled quadrilateral. The point 39C is represented by a dashed-line circle with cross-hatching. By deleting the points 37, 38, and 39 other than the points 37C, 38C, and 39C that represent each cluster from the database 20, the control unit 12 may integrate the points included in each cluster into one. By doing so, the reference information registered in the database 20 is organized. By organizing the reference information, the load of the process of collating the reference information in the database 20 can be reduced.

[0109] The control unit 12 may perform integration processing so that the difference in the density of the integrated attitude information becomes small by the integration processing. The control unit 12 may perform integration processing so that the density distribution of the reference information after the integration processing becomes uniform. The control unit 12 may perform integration processing so that the difference in the interval between two points among the points representing the integrated attitude information on the information acquisition spherical surface 30 becomes small by the integration processing. By doing so, it becomes easier for object information similar to the recognition information to remain on the information acquisition spherical surface 30.

[0110] The control unit 12 may perform integration processing when the number of reference attitude information included in the reference information of a certain object exceeds the integration determination threshold. Also, the control unit 12 may perform integration processing when the density of the points representing a plurality of reference attitude information included in the reference information of a certain object on the information acquisition spherical surface 30 exceeds the density determination threshold. The integration determination threshold or the density determination threshold may be set based on the data capacity of the database 20 or the calculation load required for the control unit 12 to query the reference information in the database 20.

[0111] In other words, the control unit 12 may perform integration processing when the number or density of the reference attitude information satisfies the integration condition. The integration condition is considered to be satisfied when the number of the reference attitude information exceeds the integration determination threshold, or when the density of the points representing the reference attitude information on the information acquisition spherical surface 30 exceeds the density determination threshold, etc. By setting the integration condition, the integration of the reference information can be executed in a timely manner.

[0112] <Another example of the procedure of the robot control method> The control unit 12 may execute a robot control method including the following procedure.

[0113] <<When there is no registration in the database 20 at all>> When the category of the object 8 to be held is not registered in the database 20, the control unit 12 estimates the holding mode using the inference model. The control unit 12 controls the robot 2 so that the end effector 2B holds the object 8 to be held in the estimated holding mode. Further, the control unit 12 extracts a feature amount from the recognition information of the object 8 to be held. The control unit 12 acquires the position or direction at which the recognition information is obtained, or the posture information indicating the posture of the object 8 to be held. The control unit 12 registers, in the database 20, reference information associating the holding mode information specifying the estimated holding mode with the acquired posture information. In this case, the control unit 12 registers the acquired posture information in the database 20 as the reference posture information.

[0114] <<When there is sufficient registration in the database 20>> When the category of the object 8 to be held is registered in the database 20, the control unit 12 queries the database 20 for the reference information registered for the recognition information of the object 8 to be held, and searches for matching or similar object information. The control unit 12 extracts the feature amount of the object 8 to be held.

[0115] The control unit 12 estimates the posture information of the object 8 to be held. For example, the control unit 12 extracts the feature amount of the object 8 to be held. Then, the control unit 12 may estimate the posture information that identifies the position or direction at which the recognition information is obtained by searching for feature amounts that match or are similar to the extracted feature amounts. The control unit 12 searches for object information that matches or is similar to the estimation result of the posture information of the object 8 to be held. When the control unit 12 discovers object information that matches the posture information of the object 8 to be held, it estimates the mode specified by the holding mode information associated with the object information as the holding mode of the object 8 to be held. When the control unit 12 discovers object information including reference posture information in the vicinity of the posture information of the object 8 to be held, it may generate and acquire the holding mode information in the posture information of the object 8 to be held by interpolating the holding mode information associated with the reference posture information. When the control unit 12 discovers object information including reference posture information in the vicinity of the posture information of the object 8 to be held, it may acquire the holding mode information associated with the reference posture information as the holding mode information in the posture information of the object 8 to be held. When using the holding mode information associated with the reference posture information as it is, since the object 8 to be held is held based on posture information different from the posture information of the object 8 to be held, there is a possibility that the object 8 to be held is held obliquely.

[0116] The control unit 12 controls the robot 2 so that the end effector 2B holds the object 8 to be held in the holding mode specified by the acquired holding mode information. The control unit 12 estimates the holding state based on the detection result of the sensor of the robot 2 and determines whether the holding is successful. When the holding is successful, the control unit 12 may register in the database 20 the object information associating the posture information and feature amount of the object 8 to be held with the holding mode information. Further, when the control unit 12 holds the object 8 to be held using the already registered holding mode information, it may update the success rate associated with the holding mode information based on the holding result.

[0117] <<When there is registration in the database 20 but it is not sufficient registration>> When the category of the object to be held 8 is registered in the database 20, the control unit 12 queries the reference information registered in the database 20 about the recognition information of the object to be held 8 and searches for matching or similar object information. The control unit 12 extracts the feature amount of the object to be held 8. The control unit 12 estimates the posture information that specifies the position or direction where the recognition information was obtained by searching for feature amounts that match or are similar to the extracted feature amount. In this example, assume that the recognition information of the object to be held 8 does not match or is not similar to the reference information in the database 20. For example, when only the feature amount of the surface of the object is registered in the database 20, if the feature amount included in the recognition information is the feature amount of the back surface of the object, it does not match or is not similar to the feature amount registered in the database 20. In this case, the control unit 12 cannot estimate the posture information from the feature amount, and there is a high possibility that the holding mode information cannot be extracted from the feature amount of the recognition information. Therefore, the control unit 12 estimates the holding mode using the inference model. The control unit 12 may determine whether to use the inference model based on the degree of match between the feature amount of the recognition information and the feature amount of the reference information.

[0118] The control unit 12 estimates the holding mode using the inference model. The control unit 12 controls the robot 2 so that the end effector 2B holds the object to be held 8 in the estimated holding mode. Also, the control unit 12 extracts the feature amount from the recognition information of the object to be held 8.

[0119] When the recognition information and the reference information do not match at all, specifically, when the feature amounts of the recognition information and the reference information do not match at all (when the degree of match is extremely low or when the degree of match is less than the first lower limit value), the control unit 12 cannot estimate the holding mode from the reference information registered in the database 20. Therefore, the control unit 12 temporarily generates new reference attitude information as attitude information that is not similar to the existing reference attitude information. Specifically, the control unit 12 temporarily generates reference attitude information corresponding to a position on the information acquisition spherical surface 30 that is not similar to the first reference attitude information included in the reference information, and registers it in the database 20. The non-similar position may be, for example, the opposite position. The control unit 12 may finely adjust the temporarily generated attitude information based on the attitude information acquired in subsequent operations.

[0120] When the recognition information and the reference information match to some extent, specifically, when the feature amounts of the recognition information and the reference information match to some extent (when the degree of match is equal to or greater than the first lower limit value and less than the second lower limit value), the control unit 12 can estimate the attitude information from the reference information registered in the database 20 with low accuracy. The control unit 12 generates the estimated attitude information as temporary normal attitude information even with low estimation accuracy, and registers it in the database 20. The control unit 12 may finely adjust the temporarily generated attitude information based on the attitude information acquired in subsequent operations.

[0121] After generating a plurality of temporary reference attitude information or temporary normal attitude information, the control unit 12 may finely adjust the temporary attitude information. The control unit 12 may finely adjust the attitude information by a method using the peripheral shooting points. Also, the control unit 12 may finely adjust the estimation result of the attitude information by a method using all the shooting points. In this case, for example, the control unit 12 may finely adjust the positional relationship on the information acquisition spherical surface 30 according to the degree of similarity with the attitude information located around the attitude information among the attitude information registered in the database 20.

[0122] Although the embodiments of the robot control device 10 have been described above, as embodiments of the present disclosure, in addition to a method or program for implementing the device, a storage medium (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.) on which the program is recorded can also be taken as an embodiment.

[0123] Further, the implementation form of the program is not limited to application programs such as object code compiled by a compiler and program code executed by an interpreter, and may be in the form of program modules incorporated into an operating system. Furthermore, the program may or may not be configured such that all processing is performed only on the CPU on the control board. The program may be configured such that a part or all of it is performed by another processing unit mounted on an expansion board or an expansion unit added to the board as necessary.

[0124] Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions etc. included in each component etc. can be rearranged so as not to be logically contradictory, and a plurality of components etc. can be combined into one or divided.

[0125] All of the constituent elements described in the present disclosure, and / or all of the disclosed methods, or all of the steps of the processing, can be combined in any combination except for combinations in which these features are mutually exclusive. Also, each of the features described in the present disclosure can be replaced with an alternative feature that serves for the same purpose, an equivalent purpose, or a similar purpose, unless explicitly negated. Therefore, unless explicitly negated, each of the disclosed features is merely an example of a comprehensive series of identical or equivalent features.

[0126] Furthermore, the embodiments according to the present disclosure are not limited to any specific configurations of the above-described embodiments. The embodiments according to the present disclosure can be extended to all novel features described in the present disclosure, or combinations thereof, or all novel methods described, or processing steps, or combinations thereof.

[0127] In the present disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configurations. The configurations distinguished by the descriptions such as "first" and "second" in the present disclosure can have their numbers exchanged in the configuration. For example, the first candidate aspect 41 can have the identifiers "first" and "second" exchanged with the second candidate aspect 42. The exchange of the identifiers is performed simultaneously. The configurations are still distinguishable after the exchange of the identifiers. The identifiers can be deleted. The configurations with the identifiers deleted are distinguished by reference numerals. Based only on the descriptions of the identifiers such as "first" and "second" in the present disclosure, the order of the configurations should not be interpreted, nor should it be used as the basis for the existence of a smaller-numbered identifier.

Explanation of Reference Numerals

[0128] 1 Robot control system (2: Robot, 2A: Arm, 2B: End effector, 4: Information acquisition unit, 5: Operating range of the robot, 6: Work start platform, 7: Work target platform) 8 Object to be held (8A: Feature point) 10 Robot control device (12: Control unit, 14: Interface) 20 Database 30 Information acquisition spherical surface (31 to 39, 36C to 39C: Points) 41 to 43 First to Third candidate aspects

Claims

1. A robot control device comprising a database storing reference information including object information of a plurality of objects and holding mode information of the plurality of objects, and a control unit capable of estimating a holding mode of the robot when holding an object to be held by each of inference models capable of estimating a holding mode of the object, and controlling the robot based on the estimated holding mode. The control unit: Acquires recognition information of the object to be held. When it is determined that the holding mode of the object to be held based on the recognition information cannot be estimated from the database, the holding mode is estimated by the inference model.

2. The control unit: Searches the database for object information similar to the recognition information. The robot control device according to claim 1, wherein a holding mode related to the searched object information can be extracted.

3. The reference information includes category information indicating the category of each of the plurality of objects. The control unit: Acquires the category information of the object to be held. The robot control device according to claim 1, wherein a holding mode can be estimated from the database based on the object information having the category information.

4. The reference information includes category information indicating the category of each of the plurality of objects. The control unit: Acquires the category information of the object to be held. When the category information is not registered in the database, the holding mode of the object to be held is estimated by the inference model.

5. The reference information has posture information for each of the plurality of objects and holding mode information related to the posture information. The control unit: Acquires the posture information of the object to be held as the recognition information. The robot control device according to claim 1, which estimates the holding mode based on the posture information of the object to be held.

6. The control unit The robot control device according to claim 1, which estimates the holding mode based on at least one piece of holding object information related to reference posture information similar to the posture information of the object to be held.

7. The control unit The robot control device according to claim 6, which estimates the holding mode based on a plurality of holding mode information items each related to reference posture information similar to the posture information.

8. One piece of the reference posture information is associated with a plurality of holding mode information items, The control unit The robot control device according to claim 7, which selects holding mode information based on the holding results associated among the plurality of holding mode information items.

9. The control unit The robot control device according to claim 7, wherein when the holding mode is estimated from the inference model, the estimation result can be registered in the database.

10. The control unit The robot control device according to claim 9, which generates reference information associating holding mode information representing the successful holding mode and object information of the object successfully held when the robot is controlled based on the estimation result of the holding mode and the holding of the object to be held is successful, and registers the reference information in the database.

11. The control unit The robot control device according to claim 7, which is capable of performing integration processing on at least two pieces of information among the plurality of reference posture information items registered in the database.

12. The control unit The robot control device according to claim 11, wherein the integration process is executed such that the difference in density of the integrated posture information generated by the integration process becomes small.

13. The control unit The robot control device according to claim 11, wherein the integration process is executed when the number or density of the plurality of pieces of reference posture information satisfies an integration condition.

14. The control unit The robot control device according to any one of claims 1 to 13, wherein when object information similar to the recognition information is not registered in the database, the holding mode of the object to be held is estimated by the inference model.

15. A robot control device capable of estimating the holding mode of a robot when holding an object to be held by a database storing reference information including object information of a plurality of objects and holding mode information of the plurality of objects, and an inference model capable of estimating the holding mode of the object, obtains recognition information of the object to be held, A robot control method, wherein when the robot control device determines that the holding mode of the object to be held based on the recognition information cannot be estimated from the database, the holding mode is estimated by the inference model.

Images