Information processing method, robot control method, information processing device, and robot control system

Abstract

This information processing method involves recognizing the position or posture of a holding object on the basis of multidimensional data of the holding object and the position or posture of a template of the holding object. The information processing method comprises calculating, on the basis of a residual between the template of the holding object and the multidimensional data of the holding object at a point where the residual deviates from an allowable residual, a relative movement amount of the template of the holding object so as to make the template of the holding object relatively close to the multidimensional data of the holding object.

Description

Information processing method, robot control method, information processing device, and robot control system Cross-reference to related applications

[0001] This application claims priority from Japanese Patent Application No. 2024-36190 (filed March 8, 2024), the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to an information processing method, a robot control method, an information processing device, and a robot control system.

[0003] As described in Patent Document 1, a method is known in which the position and orientation of an object are recognized by matching a template with the object shown in an image.

[0004] US Patent Application Publication No. 2016 / 0335496

[0005] An information processing method according to an embodiment of the present disclosure is a method for recognizing a position or orientation of a held object based on multidimensional data of the held object and a position or orientation of a template of the held object, the information processing method including calculating a relative movement amount of the template of the held object so as to move the template of the held object relatively closer to the multidimensional data of the held object, based on a residual at a point where a residual between the template of the held object and the multidimensional data of the held object is outside an allowable residual.

[0006] A robot control method according to an embodiment of the present disclosure includes controlling the robot to make the robot hold the object to be held based on holding information generated by executing the information processing method.

[0007] An information processing apparatus according to an embodiment of the present disclosure includes a processor that recognizes a position or orientation of a held object based on multidimensional data of the held object and a position or orientation of a template of the held object, and the processor calculates a relative movement amount of the template of the held object so as to move the template of the held object relatively closer to the multidimensional data of the held object, based on a residual at a point where the residual between the template of the held object and the multidimensional data of the held object is outside an allowable residual.

[0008] A robot control system according to an embodiment of the present disclosure includes the information processing device and a robot controlled based on a processing result of the information processing device.

[0009] It is a block diagram showing an example of the configuration of a robot control system according to the present disclosure.It is a flowchart showing an example of the procedure of a robot control method according to the present disclosure.It is a flowchart showing an example of the procedure of an information processing method according to the present disclosure.

[0010] When an iterative closest point (ICP) algorithm is used to match a template to point cloud data of an object, optimization stagnates as the number of points at which the residual decreases to near zero increases, and convergence takes time. Improvements in convergence speed are required. According to an information processing method, a robot control method, an information processing device, and a robot control system according to an embodiment of the present disclosure, the convergence of the template residual to the point cloud data of the object is accelerated.

[0011] (Configuration Example of Robot Control System 1) A robot control system 1 according to an embodiment of the present disclosure includes an information processing device 10, a photographing device 20, a robot controller 30, and a robot 40. The robot control system 1 recognizes the position and posture of a holding object using the photographing device 20 and the information processing device 10, and controls the robot 40 using the robot controller 30 to cause the robot 40 to hold the holding object. The holding object is an object that the robot 40 is intended to hold. The holding object may be an industrial product such as a nut or a bolt. The holding object may also be food. The holding object is not limited to the objects illustrated, and may be various other objects. The robot 40 may hold a single holding object. The robot 40 may hold at least one object selected from a set of multiple objects as the holding object. The multiple objects including the holding object may be stacked in a disorderly manner or may be aligned.

[0012] An example of the configuration of the robot control system 1 will be described below.

[0013] <Photographing device 20> The photographing device 20 photographs a held object and generates multidimensional data of the held object. The multidimensional data may include point cloud data or depth data, etc. The multidimensional data may include, for example, two-dimensional data or three-dimensional data. In this embodiment, the photographing device 20 may generate, for example, three-dimensional data of the held object. In the following description, three-dimensional data may be read as multidimensional data or two-dimensional data. Each point data of the point cloud data is, for example, data having position information in two-dimensional space or three-dimensional space. Furthermore, each point data of the point cloud data may have, for example, color information.

[0014] The image capturing device 20 may generate three-dimensional data of a collection of multiple objects including the held object. The collection of multiple objects includes the held object and objects existing around it. The image capturing device 20 may be configured to include a stereo camera, a depth camera, or a distance measuring camera that measures the distance to at least some points on the held object. The image capturing device 20 may generate three-dimensional data of the held object by analyzing an image of the held object captured with visible light or infrared light.

[0015] The camera device 20 may be attached to the robot 40. The camera device 20 may be fixed in a position where it can capture an image of the held object or a collection of multiple objects including the held object. The coordinates of the three-dimensional data generated by the camera device 20 may be determined based on the position and attitude of the camera device 20. The attitude of the camera device 20 may be represented by an angle that specifies the direction in which the camera device 20 captures images. The attitude of the camera device 20 is not limited to an angle and may be represented in various other ways, such as a spatial vector. The camera device 20 may also be attached to something other than the robot 40. In this case, for example, the camera device 20 may be installed on a stand or the like that supports the robot 40 so that it can capture an image of the held object from above. Specifically, for example, the camera device 20 may be attached to a pillar portion that extends upward from a support portion of the stand that supports the robot 40.

[0016] When the image capturing device 20 is fixed, it may generate three-dimensional data in three-dimensional coordinates determined based on the fixed position and orientation. When the image capturing device 20 is attached to the robot 40, it may generate three-dimensional data in three-dimensional coordinates determined based on the position and orientation of the robot 40 when the image capturing device 20 captures the image. After acquiring the three-dimensional data from the image capturing device 20, the information processing device 10 may convert the coordinate system of the three-dimensional data based on the position and orientation of the robot 40.

[0017] The image capturing device 20 may be configured to be able to output RAW images, which may then be converted into RGB data or three-dimensional data by the information processing device 10. The image capturing device 20 may also be, for example, an infrared camera. In this case, for example, when handling fruit as the object to be held, it is possible to determine when it is ripe to eat and select the object to be held, or when handling beverages as the object to be held, it is possible to select the object to be held based on the amount of mineral components they contain.

[0018] <Information Processing Device 10> The information processing device 10 recognizes at least one of the position and posture of the holding object from the three-dimensional data of the holding object. The information processing device 10 generates holding information based on the recognition result of at least one of the position and posture of the holding object. The holding information is information that specifies at least one of the position and posture of the robot 40 when the robot 40 holds the holding object. In this embodiment, the information processing device 10 can recognize the position and posture of the holding object. Furthermore, in this embodiment, the holding information includes information that specifies the position and posture of the robot 40.

[0019] The information processing device 10 includes a processor 12 , a storage unit 14 , and a communication unit 16 .

[0020] The processor 12 may execute a program that realizes the functions of the information processing device 10. The processor 12 may be realized as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor 12 may be realized as a plurality of communicatively connected integrated circuits and discrete circuits. The processor 12 may be configured to include a CPU (Central Processing Unit). The processor 12 may be configured to include a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). The processor 12 may be realized based on various other known technologies.

[0021] The storage unit 14 stores programs executed by the processor 12, or data or information used by the information processing device 10. The storage unit 14 may be configured to include an electromagnetic storage medium such as a magnetic disk, or may be configured to include a memory such as a semiconductor memory or a magnetic memory. The storage unit 14 may be configured as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 14 may function as a work memory for the processor. At least a portion of the storage unit 14 may be included in the processor 12. At least a portion of the storage unit 14 may be configured as a storage device separate from the information processing device 10.

[0022] The communication unit 16 may include a communication interface for wired or wireless communication with the image capture device 20 or the robot controller 30. The communication interface may include a communication device based on various communication standards such as a local area network (LAN), a wide area network (WAN), RS-232C, or RS-485. The communication unit 16 acquires three-dimensional data from the image capture device 20. The communication unit 16 outputs the retained information to the robot controller 30.

[0023] The information processing device 10 may be configured to include one or more servers. The information processing device 10 may be configured to cause multiple servers to execute parallel processing. The information processing device 10 does not need to be configured to include a physical housing, and may be configured based on virtualization technology such as a virtual machine or a container orchestration system. The information processing device 10 may be configured using cloud services. When the information processing device 10 is configured using cloud services, it may be configured by combining managed services. In other words, the functions of the information processing device 10 may be realized as cloud services.

[0024] The information processing device 10 may include at least one server group and at least one database group. The server group functions as the processor 12. The number of server groups may be one or two or more. When the number of server groups is one, the functions realized by one server group encompass the functions realized by each server group. The server groups are connected to each other so as to be able to communicate with each other via wired or wireless communication. The database group functions as the storage unit 14. The number of database groups may be one or two or more. The number of database groups may be increased or decreased as appropriate based on the amount of data managed by the information processing device 10 and the availability requirements required of the information processing device 10. The database group is connected to each server group so as to be able to communicate with each other via wired or wireless communication. The information processing device 10 may be connected to an external database. An information processing system may be configured including the information processing device 10 and an external database.

[0025] Although the information processing device 10 is depicted as a single configuration in FIG. 1 , multiple configurations can be operated as a single system as needed. In other words, the information processing device 10 is configured as a platform with variable capacity. By using multiple configurations as the information processing device 10, even if one configuration becomes inoperable due to an unforeseen event such as a natural disaster, the system can continue to operate using the other configurations. In this case, each of the multiple configurations is connected by a line, whether wired or wireless, and is configured to be able to communicate with each other. The multiple configurations may be configured across cloud services and on-premises environments.

[0026] The information processing device 10 is also communicatively connected via a line, whether wired or wireless, to at least one component of the robot control system 1. The information processing device 10 and at least one component of the robot control system 1 are provided with an interface that uses a mutually standard protocol, enabling two-way communication.

[0027] <Robot 40> The robot 40 is equipped with an end effector capable of holding an object to be held. The end effector may include a gripping hand. The gripping hand may have multiple fingers. The number of fingers on the gripping hand may be two or more. The gripping hand may also have a cloth-like member and be configured to be able to grasp the object to be held by wrapping it around it. The fingers of the gripping hand may have one or more joints. The end effector may include a suction hand. The suction hand may have one or more suction portions. The end effector may include a scooping hand. The end effector may be configured to hold the object to be held in various ways, without being limited to gripping, suction, or scooping.

[0028] The robot 40 may further include an arm for changing the position and orientation of the end effector. The end effector is attached to the tip of the arm or to a predetermined position. The arm includes joints and links. The arm may include a motor for driving the joints. The arm may be configured as, for example, a six- or seven-axis vertical articulated robot. The arm may be configured as a three- or four-axis horizontal articulated robot or a SCARA robot. The arm may be configured as a two- or three-axis Cartesian robot. The arm may be configured as a parallel link robot, etc. The number of axes configuring the arm is not limited to those illustrated.

[0029] The robot 40 can control the position and posture of the end effector by operating the arm. The posture of the end effector may be represented by an angle specifying the direction in which the end effector acts on the object to be held. The posture of the end effector is not limited to an angle and may be represented by various other modes, such as a spatial vector. The direction in which the end effector acts on the object to be held may be, for example, the direction in which the end effector approaches the object to be held when it picks up and holds the object to be held, or the direction in which it picks up the object to be held. The direction in which the end effector acts on the object to be held may be the direction in which the end effector approaches the object to be held when it grasps the object to be held with multiple fingers, or the direction in which it grasps the object to be held. The direction in which the end effector acts on the object to be held is not limited to these examples and may be various directions.

[0030] The robot 40 may further include sensors that detect the state of the arm including joints, links, etc., or the state of the end effector. The sensors may detect information related to the actual position or posture of the arm, or the actual velocity or acceleration of the arm, as the state of the arm. The sensors may detect information related to the actual position or posture of the end effector, or the actual velocity or acceleration of the end effector, as the state of the end effector. The sensors may detect forces acting on the arm or the end effector. The sensors may detect currents flowing through motors that drive the joints of the arm, or the torque of the motors. The sensors can detect information obtained as a result of the actual operation of the robot 40.

[0031] The robot 40 may be provided with a communication interface for wired or wireless communication with the robot controller 30. The communication interface may be configured identically or similarly to the communication interface of the communication unit 16 of the information processing device 10. The robot 40 may output detection results of the state of the arm or end effector to the robot controller 30 through the communication interface. The robot controller 30 can grasp the results of the actual operation of the robot 40 by obtaining the detection results of the sensors.

[0032] <Robot controller 30> The robot controller 30 acquires holding information from the information processing device 10 and controls the robot 40 to make the robot 40 hold an object to be held based on the holding information. The robot controller 30 may acquire sensor detection results from the robot 40 and control the robot 40 based on the sensor detection results. The robot controller 30 may control the robot 40 based on processing results from the information processing device 10. In other words, the robot 40 may be controlled based on processing results from the information processing device 10.

[0033] The robot controller 30 may generate information for controlling the end effector of the robot 40 and output it to the robot 40. If the end effector is a gripping hand, the information for controlling the end effector may include information for controlling the opening and closing of the gripping hand. If the end effector is a suction hand, the information for controlling the end effector may include information for controlling the start and end of suction by the suction hand.

[0034] The robot controller 30 may generate information for controlling the arm of the robot 40 and output it to the robot 40. The information for controlling the arm may include information for controlling the angle of the joint of the arm or the torque of the joint.

[0035] The robot controller 30 may be configured to include at least one processor that generates information for controlling the robot 40. The processor may execute a program that realizes the functions of the robot controller 30. The processor of the robot controller 30 may be configured identically or similarly to the processor 12 of the information processing device 10.

[0036] The robot controller 30 may further include a storage unit. The storage unit of the robot controller 30 stores programs executed by the processor of the robot controller 30, or data or information used by the robot controller 30. The storage unit of the robot controller 30 may function as a work memory for the processor of the robot controller 30. The storage unit of the robot controller 30 may be configured identically or similarly to the storage unit 14 of the information processing device 10. At least a portion of the storage unit of the robot controller 30 may be included in the processor of the robot controller 30. At least a portion of the storage unit of the robot controller 30 may be configured as a storage device separate from the robot controller 30.

[0037] The robot controller 30 may include a communication interface for wired or wireless communication with the robot 40 or the information processing device 10. The communication interface of the robot controller 30 may be configured identically or similarly to the communication interface of the communication unit 16 of the information processing device 10. The robot controller 30 acquires stored information from the information processing device 10 through the communication interface. The robot controller 30 outputs control information to the robot 40 through the communication interface. The robot controller 30 may acquire sensor detection results from the robot 40 through the communication interface.

[0038] In the robot control system 1 illustrated in Fig. 1, one robot controller 30 is connected to one robot 40. One robot controller 30 may be connected to two or more robots 40. One robot controller 30 may control only one robot 40, or may control two or more robots 40. The number of robot controllers 30 and robots 40 is not limited to one, and may be two or more.

[0039] The robot control system 1 may include an input device. The input device is, for example, communicably connected to the information processing device 10. The input device can receive operation input from, for example, a user of the robot control system 1. The input device may include, for example, a touch panel or touch sensor, or a pointing device such as a mouse. The input device may include physical keys. The input device may include an audio input device such as a microphone. The input device is not limited to these examples and may include various other devices.

[0040] The robot control system 1 may also include an output device. The output device may be communicatively connected to, for example, the information processing device 10. The output device may include a display device. The output device may display data or information of the robot control system 1 so that the user can recognize it. The display device may include, for example, a liquid crystal display (LCD), an organic electroluminescence (EL) display, an inorganic electroluminescence (EL) display, or a plasma display panel (PDP). The display device is not limited to these displays and may include various other types of displays. The display device may include a light-emitting device such as an LED (light-emitting diode). The display device may include various other devices. The output device may include an audio output device such as a speaker that outputs auditory information such as sound. The output device is not limited to these examples and may include various other devices.

[0041] The robot control system 1 may also include a terminal device that can be communicatively connected to the information processing device 10. The terminal device may also function as at least one of an input device and an output device.

[0042] (Example of Operation of Robot Control System 1) An example of operation in which the robot control system 1 recognizes an object to be held and causes the robot 40 to hold the object will be described below.

[0043] The robot control system 1 may execute a robot control method including the steps of the flowchart illustrated in Fig. 2. The robot control method may be realized as a robot control program executed by a processor included in the robot control system 1. The robot control program may be stored in a non-transitory computer-readable medium.

[0044] The image capturing device 20 generates three-dimensional data of a holding object or a collection of multiple objects including the holding object (step S1). Assume that point cloud data is generated as the three-dimensional data. The image capturing device 20 outputs the generated three-dimensional data to the information processing device 10.

[0045] The information processing device 10 recognizes objects from the three-dimensional data (step S2). When the three-dimensional data is point cloud data of a set of multiple objects, the processor 12 of the information processing device 10 may recognize individual objects by, for example, performing segmentation on the point cloud data of the set of multiple objects. The segmentation may be performed, for example, by an AI (Artificial Intelligence) model that has undergone machine learning. The processor 12 may also recognize individual objects by performing template matching on the point cloud data of the set of multiple objects. The method of recognizing objects is not limited to the above-mentioned example, and various other methods may be used.

[0046] The information processing device 10 selects a holding object from the recognized objects (step S3). The processor 12 of the information processing device 10 may evaluate the ease of holding each object recognized in the procedure of step S2 by the robot 40, for example, and calculate an index representing the ease of holding. The processor 12 may select an object that is easy for the robot 40 to hold as the holding object.

[0047] The processor 12 may estimate, for example, the probability that the robot 40 will successfully hold each object as an index representing the ease of holding. When multiple objects overlap, the processor 12 may evaluate the object located on top as being easier to hold. The processor 12 may evaluate, as a result of performing segmentation, an object that is not hidden by other objects or an object that is only slightly hidden by other objects as being easier to hold. The processor 12 may evaluate, as an object with a low obscuration rate calculated by segmentation, an object as being easier to hold. The processor 12 may evaluate, as an object with a surrounding space that allows the end effector of the robot 40 to enter, an object as being easier to hold. The processor 12 may evaluate, as an object that protrudes more in the direction in which the end effector of the robot 40 enters than surrounding objects, an object as being easier to hold. The processor 12 may represent the recognition results of each object as a bounding box. The processor 12 may associate an index representing the ease of holding with the bounding box.

[0048] The processor 12 of the information processing device 10 may select each object recognized in the procedure of step S2 as a retention target. The processor 12 may also select each object recognized in the procedure of step S2 as a retention target without evaluation. In other words, the objects may be selected as retention targets without evaluation.

[0049] Furthermore, in step S3, the processor 12 may select a holding target based on, for example, past holding records. Specifically, for example, a recognized object may be selected as a holding target if it has a position or orientation similar to or identical to a position or orientation previously held by the robot 40. The processor 12 may select a holding target based on, for example, the probability that the robot 40 will successfully hold each object. The processor 12 may select a holding target based on, for example, the overlap of multiple objects. The processor 12 may select, as a holding target, an object that is not hidden by other objects or that is only slightly hidden by other objects as a result of performing segmentation. The processor 12 may select, as a holding target, an object with a low occlusion rate calculated by segmentation. The processor 12 may select, as a holding target, an object whose segmentation area calculated by segmentation is larger than a predetermined standard. The processor 12 may select, as a holding target, an object that has a space around it that the end effector of the robot 40 can enter. The processor 12 may determine, as a holding target, an object that protrudes further in the direction in which the end effector of the robot 40 enters than surrounding objects. The processor 12 may represent the recognition result of each object as a bounding box. The processor 12 may display, in the bounding box, an indicator indicating the basis for selecting the object as a holding target.

[0050] The information processing device 10 recognizes the position and orientation of the object (step S4). The processor 12 of the information processing device 10 may recognize the position and orientation of the object by performing template matching to match point cloud data of a set of multiple objects including the object with a template corresponding to the object. The template may be, for example, reference information including information indicating the three-dimensional shape of the object, or point cloud data indicating the three-dimensional shape of the object. In the present disclosure, the processor 12 uses an ICP (Iterative Closest Point) algorithm as the template matching method. The processor 12 is not limited to the ICP algorithm as the template matching method, and may employ various other algorithms. The processor 12 may perform template matching based on the point cloud data of an object selected as the object from the point cloud data, rather than on all point cloud data.

[0051] When recognizing the position and orientation of the held object using the ICP algorithm, the processor 12 repeatedly changes the position and orientation of the template, for example, so as to reduce the residual between each point of the template, which is point cloud data, and each point of the point cloud data of the held object. The processor 12 repeatedly changes the position and orientation of the template until an evaluation value based on the residual between the template and the point cloud data of the held object becomes less than a convergence threshold. The processor 12 determines that the position and orientation of the template match the position and orientation of the held object represented by the point cloud data of the held object when the evaluation value based on the residual between the template and the point cloud data of the held object becomes less than the convergence threshold. In other words, the processor 12 determines that the operation of matching the template with the point cloud data of the held object has converged when the evaluation value based on the residual between the template and the point cloud data of the held object becomes less than the convergence threshold. The processor 12 uses the converged position and orientation of the template as the recognition result of the position and orientation of the held object.

[0052] The information processing device 10 generates holding information for the holding object based on the recognition result of the position and orientation of the holding object (step S5). The processor 12 of the information processing device 10 may generate, as the holding information, a path along which the end effector of the robot 40 approaches the holding object. The processor 12 may generate, as the holding information, the position and orientation at which the end effector of the robot 40 starts to hold the holding object. When the holding object is a nut and the end effector of the robot 40 is a gripping hand, the processor 12 may generate the holding information so that the fingers of the gripping hand are inserted perpendicularly into the top surface of the nut. The processor 12 outputs the holding information to the robot controller 30 via the communication unit 16.

[0053] The robot controller 30 controls the robot 40 based on the holding information to make the robot 40 hold the object to be held (step S6). After executing the procedure of step S6, the robot control system 1 ends the execution of the flowchart of FIG.

[0054] (Example of Operation for Recognizing Position and Posture of Held Object) An example of operation in which the processor 12 of the information processing device 10 recognizes the position and posture of a held object using the ICP algorithm in the recognition procedure of step S4 in Fig. 2 will be described below. The processor 12 may execute an information processing method including the procedure of the flowchart exemplified in Fig. 3. The information processing method may be realized as an information processing program executed by the processor 12. The information processing program may be stored in a non-transitory computer-readable medium.

[0055] The processor 12 sets initial values ​​for the position and orientation of the template corresponding to the held object (step S11). The processor 12 may set the initial values ​​based on the result of recognizing the held object by segmentation.

[0056] The processor 12 assigns weights to the observation points based on factors that cause errors when recognizing the position and orientation of the held object (step S12). The observation points are the object selected as the target for matching (matching) the template, i.e., the points included in the point cloud data of the held object.

[0057] The point cloud data of the retained object may include outliers. In other words, some of the observation points may be outliers. Outliers are points with large residuals from the initial values ​​of the template. Outliers affect the evaluation value when matching the template to the point cloud data of the retained object, causing errors. To prevent outliers from affecting the evaluation value, the processor 12 may set the weight of points whose residuals from the initial values ​​are equal to or greater than the outlier processing threshold to 0, and set the weight of points whose residuals from the initial values ​​are less than the outlier processing threshold to 1.

[0058] The observation points may contain noise. The noise may affect the evaluation value and cause errors when matching the template to the point cloud data of the target object. The processor 12 may set the noise variance as a weight for the observation points in the least squares method to reduce the effect of the noise on the evaluation value.

[0059] The processor 12 weights the observation points based on the residuals from the template points (step S13). When matching the template to the point cloud data of the held object, the processor 12 does not try to make the residuals between the template points and the observation points zero, but calculates the amount of movement of the position and orientation of the template so that the residuals between the template points and the observation points are within an allowable residual, and moves the template. The allowable residual is the allowable range of the residuals between the template points and the observation points.

[0060] When the processor 12 attempts to make the residual between the template points and the observation points zero, it calculates the amount of movement of the position and orientation of the template based on the value of the residual from the template points for all observation points.

[0061] On the other hand, in the present disclosure, the processor 12 does not attempt to make the residual between the template point and the observation point zero, but calculates the amount of movement of the position and orientation of the template under the condition that the residual between the template point and the observation point does not have to approach zero as long as it is within the allowable residual. In this case, the processor 12 sets a smaller weight to an observation point whose residual from the template point is within the allowable residual than a weight to be set to an observation point whose residual from the template point is not within the allowable residual. The processor 12 may set a weight to zero to an observation point whose residual from the template point is within the allowable residual so that the residual value of the observation point whose residual from the template point is within the allowable residual is not reflected in the amount of movement of the template.

[0062] When the residuals of all observation points are reflected, i.e., when the movement amount of the template position and orientation is calculated by also reflecting the residuals of observation points with small residuals, there is an influence from observation points whose residuals are already sufficiently small, and when the position and orientation of the template are moved by the calculated movement amount, the movement amount of observation points with large residuals becomes small. Because the movement amount of observation points with large residuals is small, it is difficult for the observation points with large residuals to approach the template points. As a result, the number of times the template is moved increases until the template and the point cloud data of the retained object match. In other words, convergence is slow.

[0063] On the other hand, when calculating the amount of movement of the template position and orientation by reducing the contribution of the residuals of observation points that fall within the allowable residual and increasing the contribution of the residuals of observation points that do not fall within the allowable residual, as in the present disclosure, the influence of observation points with large residuals can be increased compared to when calculating the amount of movement of the template position and orientation by also reflecting the residuals of observation points with small residuals. Therefore, the amount of movement of observation points with large residuals can be increased, and observation points with large residuals are more likely to approach the template points. As a result, the number of times the template must be moved is reduced until the template and the point cloud data of the retained object match. In other words, convergence is faster.

[0064] The processor 12 may use an expected value of the residual between the template point and the observation point to calculate the amount of movement of the position and orientation of the template, reflecting the residual of the observation point that is not within the allowable residual. In other words, the processor 12 may use the probability that the true value of the observation point is within the allowable residual from the template point to calculate the amount of movement of the position and orientation of the template, reflecting the residual of the observation point that is not within the allowable residual.

[0065] Specifically, the observation point contains noise. Therefore, the true value of a point corresponding to the observation point in the held object does not necessarily coincide with the observation point. The true value of the point corresponding to the observation point is a value shifted from the observation point by the noise component. Assuming that the noise is represented by a probability distribution, the true value of the point corresponding to the observation point is represented using the probability distribution of the noise. In the present disclosure, the probability distribution of the true value of the observation point is approximated by a normal distribution. The probability distribution of the true value of the observation point is not limited to a normal distribution and may be approximated by other distributions. Note that, for example, if the noise or observation error of the observation point is based on a camera, the variance of the observation results, such as depth data acquired by the camera, may be identified.

[0066] When the true value of an observation point is represented by a probability distribution, the residual between the template point and the observation point is represented by an expected value. The greater the deviation of the measurement value of the observation point from the template value, the higher the probability that the true value of the observation point will not fall within the allowable residual from the template value. By setting the probability that the true value of the observation point will not fall within the allowable residual from the template value as the weight of the observation point, the processor 12 can make the residual of the observation point from the template point contribute significantly to the calculation of the movement amount of the position and orientation of the template.

[0067] When the processor 12 matches the template to the point cloud data of the retained object, a local solution may exist near the true solution where the template matches the point cloud data of the retained object, i.e., the true value, due to erroneous surface correspondence. The processor 12 may set the value of the dot product of the normal vector of the template surface and the normal vector of the retained object surface as a weight for the observation point, so as to make it less likely to converge to a local solution, i.e., to make it less likely that an error will occur in the correspondence between the template surface and the retained object surface. The processor 12 may set the cosine distance as a weight for the observation point. The processor 12 may also set a value calculated by projecting a residual vector onto the normal vector as a weight for the observation point.

[0068] The processor 12 calculates the amount of movement of the position and orientation of the template and moves the template (step S14). Specifically, the processor 12 calculates the amount of movement of the position and orientation of the template based on a value obtained by reflecting the weight set for the observation point in the procedures of steps S12 and S13 on the residual from the point of the template. The processor 12 may calculate the amount of movement of the position and orientation of the template based on the Gauss-Newton method. Alternatively, the processor 12 may calculate the amount of movement of the position and orientation of the template based on SVD (singular value decomposition).

[0069] If the retained object is a smooth object, most points are incorrectly matched in the process of matching the template to the point cloud data of the retained object. In this case, residuals of observation points near the edge of the retained object contribute significantly to moving the template toward the correct match. The processor 12 may match multiple observation points to one point on the template to promote correct matching between the template and the point cloud data of the retained object near the edge of the retained object. When multiple observation points are matched to one point on the template, the processor 12 may set a weight for each correspondence in the procedure of step S13. The processor 12 may calculate the movement amount of the position and orientation of the template based on a value reflecting a weight on the residual of each of the multiple observation points matched to one point on the template.

[0070] The processor 12 calculates an evaluation value (step S15). The processor 12 may calculate the sum of residuals between the points of the template after the movement and the observation points as the evaluation value. The processor 12 may calculate a value that reflects the weight set for the observation points in the residuals between the points of the template after the movement and the observation points, and calculate the sum of these values ​​as the evaluation value.

[0071] When the processor 12 associates a plurality of observation points with one point on the template, the processor 12 may add the residual error for each association to the evaluation value.

[0072] The processor 12 determines whether the evaluation value is less than the convergence determination threshold (step S16). If the evaluation value is equal to or greater than the convergence determination threshold, i.e., if the evaluation value is not less than the convergence determination threshold (step S16: NO), the processor 12 returns to the procedure of step S13 and executes the procedures of steps S13 to S15 for the moved template.

[0073] If the evaluation value is less than the convergence determination threshold (step S16: YES), the processor 12 determines the position and orientation of the template based on the position and orientation of the template after movement in the template movement procedure of step S14 (step S17). After executing step S17, the processor 12 ends execution of the flowchart in Fig. 3. The processor 12 uses the position and orientation of the template determined in step S17 in Fig. 3 as the recognition result of the position and orientation of the held object in the procedure of step S4 in the flowchart in Fig. 2.

[0074] As described above, in the robot control system 1, the information processing device 10 recognizes the position and orientation of the held object by matching a template to point cloud data of the held object using the ICP algorithm. An example of recognizing the position and orientation of the held object will be described below.

[0075] When the object to be held was a nut, the position and orientation of the nut were recognized using the nut's top surface as a template. The nut's top surface is a surface perpendicular to the axis of the nut's hole. In a comparative example, when the amount of movement of the template's position and orientation was calculated by reflecting the residuals of observation points with small residuals, the residuals of the template's surface and position relative to the nut's top surface were eliminated after 640 repetitions of template movement, but the residuals of the template's rotational direction relative to the nut's top surface were not eliminated even after 1,000 repetitions of template movement. On the other hand, in an information processing method according to the present disclosure, when the amount of movement of the template's position and orientation was calculated without reflecting the residuals of observation points with small residuals, the residuals of the template's surface and position relative to the nut's top surface were eliminated after 40 repetitions of template movement, and the residuals of the template's rotational direction relative to the nut's top surface were eliminated after 75 repetitions of template movement. In other words, the information processing method according to the present disclosure converges more quickly to match the template with the nut's top surface as the object to be held than the method according to the comparative example.

[0076] Furthermore, the robot 40 actually held nuts based on the results of recognizing the position and orientation of the nuts using the information processing method disclosed herein. The success rate when the robot 40 held an M2 nut was 98% when multiple nuts were placed flat so as not to touch each other, and 77% when approximately 3,000 nuts were piled up in a messy manner. The success rate when the robot 40 held an M1.2 nut was 96% when multiple nuts were placed flat so as not to touch each other, and 77% when approximately 3,000 nuts were piled up in a messy manner. The number of holding attempts to calculate the success rate was 100.

[0077] (Summary) As described above, according to the information processing method disclosed herein, the convergence of the template residual to the point cloud data of the object to be held is accelerated. The rapid convergence of the template residual reduces the time required to recognize the position and orientation of the object to be held. The reduction in the time required to recognize the position and orientation of the object to be held reduces the time required for the robot 40 to start holding the object to be held. As a result, the efficiency of the holding operation by the robot 40 is improved.

[0078] In the robot control system 1 described above, at least some of the functions of the information processing device 10 may be realized by the robot controller 30. For example, the robot controller 30 may execute the procedure of selecting a holding object in step S3 of Fig. 2, the procedure of recognizing the position and orientation of the holding object in step S4, and the procedure of generating holding information in step S5. The robot controller 30 may also execute the procedure of recognizing an object in step S2 of Fig. 2. The robot controller 30 may also acquire RAW data from the imaging device 20 and execute the procedure of generating 3D data in step S1 of Fig. 2.

[0079] Furthermore, in the above-described robot control system 1, a holding object is selected from a plurality of recognized objects. However, a plurality of holding objects may be selected. In this case, for example, a predetermined number of holding objects may be selected from objects that are easy to hold based on an evaluation value. Furthermore, when a plurality of holding objects are selected, an execution holding object to actually hold may be selected from the plurality of holding objects after the step of recognizing the position and orientation of the holding object or the step of generating holding information. Note that the execution holding object may be selected based on, for example, the distance from the end effector to the holding object. Furthermore, the execution holding object may be selected based on an evaluation value of ease of holding or the evaluation value calculated in step S16.

[0080] Furthermore, in the above-described robot control system 1, after the position and orientation of the holding object are recognized, the ease of holding may be evaluated or re-evaluated based on at least one of the position and orientation of the holding object. Note that the evaluation or re-evaluation of the ease of holding may be performed before or after generating the holding information.

[0081] Furthermore, in the robot control system 1 described above, after generating the holding information, the ease of holding may be evaluated or re-evaluated based on at least one of the position and the orientation of the end effector.

[0082] In the above description, the information processing device 10 may be read as an image processing device, and the information processing method may be read as an image processing method.

[0083] In the above description, the processor 12 moves the template of the held object so as to approach the three-dimensional data of the held object, but the three-dimensional data of the held object may also be moved so as to approach the template of the held object. The processor 12 may move the template of the held object relatively closer to the multidimensional data of the held object. The processor 12 may calculate the relative movement amount of the template of the held object so as to move the template of the held object relatively closer to the multidimensional data of the held object.

[0084] The above has described an embodiment of the robot control system 1, but embodiments of the present disclosure can also be embodied as a method or program for implementing a system or device, as well as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card).

[0085] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may also be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed.

[0086] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to cause logical inconsistencies, and multiple components can be combined or divided into one.

[0087] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0088] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein.

[0089] (1) An information processing method according to one embodiment of the present disclosure is an information processing method for recognizing a position or orientation of a held object based on multidimensional data of the held object and a position or orientation of a template of the held object, and includes calculating a relative movement amount of the template of the held object so as to bring the template of the held object relatively closer to the multidimensional data of the held object based on a residual at a point where the residual between the template of the held object and the multidimensional data of the held object is outside an allowable residual.

[0090] (2) The information processing method described in (1) above may further include recognizing at least one object from multidimensional data of a set of multiple objects, selecting one of the at least one object as the object to be held, and setting the position and orientation of the object selected as the object to be held as initial values ​​for the position and orientation of a template of the object to be held.

[0091] (3) The information processing method described in (1) or (2) above may further include calculating an evaluation value based on the residual between the template of the held object and the multidimensional data of the held object, repeatedly moving the position and orientation of the template of the held object until the evaluation value becomes less than a convergence determination threshold, and recognizing the position and orientation of the template of the held object when the evaluation value becomes less than the convergence determination threshold as the position and orientation of the held object.

[0092] (4) The information processing method described in any one of (1) to (3) above may further include generating holding information for causing the robot to hold the holding object from the recognition results of the position and posture of the holding object.

[0093] (5) A robot control method according to one embodiment of the present disclosure includes controlling the robot to hold the object to be held based on the holding information generated by executing the information processing method described above in (4).

[0094] (6) An information processing device according to an embodiment of the present disclosure includes a processor that recognizes a position or orientation of a held object based on multidimensional data of the held object and a position or orientation of a template of the held object, and calculates a relative movement amount of the template of the held object so as to move the template of the held object relatively closer to the multidimensional data of the held object, based on a residual at a point where the residual between the template of the held object and the multidimensional data of the held object is outside an allowable residual.

[0095] (7) A robot control system according to an embodiment of the present disclosure includes the information processing device described in (6) above, and a robot controlled based on a processing result of the information processing device.

[0096] REFERENCE SIGNS LIST 1 Robot control system 10 Information processing device (12: processor, 14: storage unit, 16: communication unit) 20 Imaging device 30 Robot controller 40 Robot

Claims

1. An information processing method for recognizing the position or orientation of a held object based on multidimensional data of the held object and the position or orientation of a template of the held object, the information processing method including calculating a relative movement amount of the template of the held object so as to bring the template of the held object relatively closer to the multidimensional data of the held object based on a residual at a point where the residual between the template of the held object and the multidimensional data of the held object is outside an allowable residual.

2. The information processing method according to claim 1, further comprising: recognizing at least one object from multidimensional data of a set of multiple objects; selecting one of the at least one object as the object to be held; and setting the position and orientation of the object selected as the object to be held as initial values ​​for the position and orientation of a template of the object to be held.

3. The information processing method according to claim 1 or 2, further comprising: calculating an evaluation value based on a residual between the template of the held object and the multidimensional data of the held object; repeatedly moving the position and orientation of the template of the held object until the evaluation value becomes less than a convergence determination threshold; and recognizing the position and orientation of the template of the held object when the evaluation value becomes less than the convergence determination threshold as the position and orientation of the held object.

4. An information processing method according to any one of claims 1 to 3, further comprising generating holding information for causing the robot to hold the object to be held from the results of recognizing the position and orientation of the object to be held.

5. A robot control method comprising controlling the robot to hold the object to be held based on the holding information generated by executing the information processing method according to claim 4.

6. An information processing device comprising a processor that recognizes the position or orientation of a held object based on multidimensional data of the held object and the position or orientation of a template of the held object, wherein the processor calculates a relative movement amount of the template of the held object so as to bring the template of the held object relatively closer to the multidimensional data of the held object based on a residual at a point where the residual between the template of the held object and the multidimensional data of the held object is outside an allowable residual.

7. A robot control system comprising the information processing device according to claim 6 and a robot controlled based on the processing results of said information processing device.

Images