Information processor, information processing method, robot system, article manufacturing method, program, and recording medium

Abstract

To provide a technique advantageous for improvement in deduction accuracy.SOLUTION: An information processor comprises an information processing unit for performing information processing. The information processing unit acquires sensing data from a vision sensor which senses a work-piece, deducts the sensing data as input data of a learned model, and acquires information concerning an area including a plurality of positions which may be a position of a first finger part of at least two finger parts when causing at least two finger parts included in a robot to execute a holding operation for holding the work-piece by deduction of the learned model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to robot technology. [Background technology]

[0002] In a system that uses a robot to pick up a workpiece, it is necessary to calculate the position of the robot where the workpiece can be held accurately and quickly.

[0003] Patent document 1 discloses that a control device uses a prediction model that uses an image as input to predict multiple candidate gripping positions for an object, and by referring to the multiple candidate gripping positions, determines the gripping position at which the gripping device will grip the object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-021147 Summary of the Invention [Problem to be solved by the invention]

[0005] Machine learning in the learning phase requires a large number of input data. The large number of input data includes two or more identical or similar input data, and each of the two or more input data may have a large number of correct answers. In this way, when there are a large number of correct answers for one input data, in the method of Patent Document 1, it may occur that only a part of the large number of correct answers is linked to the input data as correct answer data, or different correct answer data between two or more input data is linked to each input data, so the accuracy of learning decreases. As a result, the accuracy of inference in the inference phase decreases.

[0006] For example, if a workpiece has a cylindrical part and is imaged from a direction perpendicular to the bottom surface of the cylindrical part, when an image of the workpiece imaged from such a direction is used as input data, the center of the bottom surface of the cylindrical part is set as the gripping center, and all positions in 360 degrees around the gripping center can be the correct gripping positions of the two fingers.

[0007] In addition, for example, there may be cases where a workpiece has an upper portion and a lower portion with a width different from that of the upper portion, and the workpiece is imaged from a direction in which the upper portion and the lower portion are captured. When an image of the workpiece imaged from such a direction is used as input data, multiple positions, such as a position for gripping the upper portion and a position for gripping the lower portion, may be correct gripping positions for the two fingers.

[0008] In this way, when there are multiple correct answers for the input data and it is not possible to determine a single correct answer, the accuracy of machine learning decreases, and as a result, the accuracy of inference decreases.

[0009] The present disclosure provides techniques that are advantageous in improving the accuracy of inference. [Means for solving the problem]

[0010] A first aspect of the present disclosure is an information processing device that includes an information processing unit that performs information processing, and the information processing unit acquires sensing data from a vision sensor that senses a workpiece, performs inference using the sensing data as input data for a trained model, and acquires, through the inference of the trained model, information on an area that includes multiple positions that can be the position of a first finger of at least two fingers included in a robot when the robot is caused to perform a holding operation to hold the workpiece.

[0011] A second aspect of the present disclosure is an information processing method for processing information, comprising the steps of acquiring sensing data from a vision sensor that senses a workpiece, performing inference using the sensing data as input data for a trained model, and acquiring, through inference from the trained model, information on an area that includes multiple positions that can be the position of a first finger of at least two fingers included in a robot when a holding operation is performed to hold the workpiece with the at least two fingers. Effect of the Invention

[0012] According to the present disclosure, a technique is provided that is advantageous for improving the accuracy of inference. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is an explanatory diagram illustrating a schematic configuration of a robot system according to a first embodiment. [Diagram 2] 1A is a block diagram showing the configuration of an information processing device according to a first embodiment, and FIG. [Diagram 3] FIG. 2 is an explanatory diagram of an image according to the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram of an inference process of an inference unit according to the first embodiment; [Diagram 5] FIG. 2 is an explanatory diagram of an image according to the first embodiment. [Figure 6] FIG. 2 is an explanatory diagram of an image according to the first embodiment. [Figure 7] FIG. 2 is an explanatory diagram of an image according to the first embodiment. [Figure 8] FIG. 2 is an explanatory diagram of an image according to the first embodiment. [Figure 9] FIG. 2 is an explanatory diagram of an image according to the first embodiment. [Figure 10] FIG. 4 is an explanatory diagram of a process for determining a holding position according to the first embodiment. [Figure 11] FIG. 4 is an explanatory diagram of a process for determining a holding position according to the first embodiment. [Figure 12]6 is an explanatory diagram of a process of a holding position calculation unit according to the first embodiment. FIG. [Figure 13] 6 is an explanatory diagram of a process of a holding position calculation unit according to the first embodiment. FIG. [Figure 14] FIG. 2 is an explanatory diagram of a method for determining a position in a three-dimensional space according to the first embodiment. [Figure 15] FIG. 2 is an explanatory diagram of a method for determining a position in a three-dimensional space according to the first embodiment. [Figure 16] FIG. 2 is an explanatory diagram of a method for determining a position in a three-dimensional space according to the first embodiment. [Figure 17] FIG. 2 is an explanatory diagram of a control unit according to the first embodiment. [Figure 18] FIG. 11 is an explanatory diagram showing a schematic configuration of a robot system according to a second embodiment. [Figure 19] FIG. 11 is an explanatory diagram of an information processing unit according to the second embodiment. [Figure 20] FIG. 11 is an explanatory diagram of a holding position calculation unit according to the second embodiment. [Figure 21] FIG. 11 is an explanatory diagram of a holding position calculation unit according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. The following embodiments are illustrative of preferred configurations of the present disclosure, and for example, those skilled in the art can appropriately modify the detailed configurations without departing from the spirit of the present disclosure. In addition, in the drawings referred to in the following description of the embodiments, elements indicated by the same reference numbers have the same functions unless otherwise noted. In addition, these drawings are represented diagrammatically for convenience of illustration and explanation, and therefore shapes, sizes, arrangements, etc. may not strictly match between the drawings.

[0015] [First embodiment] 1 is an explanatory diagram showing a schematic configuration of a robot system 1 according to a first embodiment. The robot system 1 includes a base 5, a vision sensor 8, a robot 150, an information processing device 100, and a display device 400. The information processing device 100 includes an information processing unit 200 that processes information, and a control unit 300 that controls the robot 150.

[0016] The robot 150 is an industrial robot and is arranged in a logistics line or a production line. The robot 150 is a manipulator. The robot 150 is fixed to, for example, a stand (not shown). A pedestal 5 is arranged around the robot 150. A plurality of workpieces can be bulk-stacked on the pedestal 5, and in the example of FIG. 1, two workpieces 3 and 4 are placed on the pedestal 5. In the first embodiment, the workpieces 3 and 4 are the same type of workpieces and have the same shape and size. The workpiece 3 has, for example, an upper portion 31 that is a cylindrical portion having a cylindrical shape, and a lower portion 32 that is a cylindrical portion having a cylindrical shape.

[0017] Note that objects other than the pedestal 5 may be placed around the robot 150. Furthermore, the workpieces 3 and 4 may have the same shape, or may have different shapes. Furthermore, the number of workpieces placed on the pedestal 5 may be one, or three or more. When multiple workpieces are placed on the pedestal 5, the multiple workpieces may be in a bulk stacked state.

[0018] The robot 150 and the information processing device 100 are connected to each other by wire or wirelessly, directly or via a network. The vision sensor 8 and the information processing device 100 are connected to each other by wire or wirelessly, directly or via a network.

[0019] The display device 400 is an example of a display unit. The display device 400 is a display such as a liquid crystal monitor, and is connected to the information processing device 100. The display device 400 may be included in the information processing device 100.

[0020] The robot 150 has a robot arm 151 and a robot hand 152 which is an example of an end effector. The robot arm 151 is, for example, a vertical multi-joint robot arm. The base of the robot 150 is a fixed end and is fixed to a stand (not shown). The tip of the robot 150 is a free end. That is, the tip of the robot 150 moves to an arbitrary position by the operation of the robot 150. Note that the robot arm 151 is not limited to a vertical multi-joint robot arm, and may be, for example, an orthogonal robot, a parallel link robot, or a SCARA (Selective Compliance Assembly Robot Arm) as long as it has a degree of freedom required for the robot hand 152 to hold the workpiece 3 or 4.

[0021] The robot hand 152 is supported by the robot arm 151. The robot hand 152 is attached to a predetermined portion of the robot arm 151, for example, to the tip of the robot arm 151. The robot hand 152 is configured to be able to hold each of the workpieces 3 and 4.

[0022] The robot hand 152 has at least two fingers, and in the first embodiment, two fingers 153 and 154. When the finger 153 is the first finger, the finger 154 is the second finger, and when the finger 154 is the first finger, the finger 153 is the second finger.

[0023] The finger portions 153, 154 may perform any operation such as a rotation operation, a translation operation, or an operation of approaching or moving away from the holding center, as long as the finger portions 153, 154 can hold a workpiece. The robot hand 152 is not limited to having two finger portions 153, 154, and may have three or more fingers. A proximity sensor may be provided on the side of the finger portion 153 facing the finger portion 154.

[0024] With the above configuration, the robot arm 151 can move the robot hand 152 to a desired position, and the robot 150 can be made to hold the workpiece 3 or 4.

[0025] In a production line for manufacturing goods, the robot 150 can hold a workpiece with the robot hand 152 and carry it out a transporting operation, assemble it to another workpiece, or grip a tool and process the workpiece. Alternatively, the robot 150 can also perform an operation by attaching an actuator other than the robot hand 152 to the tip of the robot arm 151 instead of the robot hand 152, depending on the work content of the manufacturing process.

[0026] For example, a plurality of workpieces are arranged around the robot 150, for example on the pedestal 5. An article that is an assembly can be manufactured by a manufacturing method in which the robot 150 holds one workpiece and assembles the workpiece to another workpiece. The assembly can be an intermediate product or a final product.

[0027] The vision sensor 8 has a function of sensing the top of the pedestal 5. The vision sensor 8 outputs sensing data to the information processing device 100 as a sensing result. The sensing data of the vision sensor 8 is preferably digital data that digitizes the state of the top of the pedestal 5, such as a grayscale image or a point cloud. In particular, the sensing data is preferably tensor-format data that is suitable for input data in machine learning, such as a grayscale image. The grayscale image is, for example, a monochrome image or a color image. The color image is, for example, an RGB image.

[0028] In the first embodiment, the vision sensor 8 is a digital camera having an image sensor (not shown). The image sensor is, for example, a CMOS image sensor or a CCD image sensor. The vision sensor 8 captures an image of a subject, and generates a captured image, for example, a monochrome image or a color image, in which the subject appears as sensing data, and outputs the captured image to the information processing device 100.

[0029] The vision sensor 8 is disposed at a position where it can capture images of the workpieces 3 and 4 placed on the pedestal 5. For example, the vision sensor 8 is disposed above the pedestal 5. The vision sensor 8 is fixed to a frame (not shown) that is disposed around the robot 150.

[0030] The installation position of the vision sensor 8 is not limited to the frame, and the vision sensor 8 may be disposed on the bottom of the robot 150 or the base 5. The vision sensor 8 may be a monocular camera or a stereo camera.

[0031] In the first embodiment, the information processing device 100 is configured with a computer. The information processing device 100 can send an imaging command to the vision sensor 8 to cause the vision sensor 8 to capture an image. The information processing device 100 is configured to be able to acquire image information (image data) generated by the vision sensor 8, and is configured to be able to process the acquired image information. Furthermore, the information processing device 100 can execute control processing for controlling the robot 150 in addition to information processing.

[0032] 2(a) is a block diagram showing the configuration of an information processing device 100 according to the first embodiment. The information processing device 100 is configured as a computer, and includes a CPU (Central Processing Unit) 121, which is an example of a processor. The information processing device 100 also includes, as storage units, a ROM (Read Only Memory) 122, a RAM (Random Access Memory) 123, and an HDD (Hard Disk Drive) 124. The information processing device 100 also includes a recording disk drive 125, and an interface 126, which is an input / output interface.

[0033] The CPU 121, the ROM 122, the RAM 123, the HDD 124, the recording disk drive 125, and the interface 126 are connected to each other by a bus so that data can be transmitted between them. The ROM 122 stores a basic program related to the operation of the computer. The RAM 123 is a storage device that temporarily stores various data such as the results of the calculation processing of the CPU 121. The HDD 124 records the results of the calculation processing of the CPU 121 and various data acquired from the outside, and also records a program 127 for making the CPU 121 execute the information processing of the information processing unit 200 and the control processing of the control unit 300. The program 127 is application software for making the CPU 121 function as the information processing unit 200 and the control unit 300. Thus, the CPU 121 functions as the information processing unit 200 and the control unit 300 by executing the program 127, and can execute the information processing and control processing in the information processing method.

[0034] The recording disk drive 125 can read out various data and programs recorded on the recording disk 128. In the first embodiment, the non-transient recording medium readable by the computer is the HDD 124, and the program 127 is recorded on the HDD 124, but this is not limiting. The program 127 may be recorded on any recording medium as long as it is a non-transient recording medium readable by the computer. As a recording medium for supplying the program 127 to the computer, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a non-volatile memory, etc. can be used.

[0035] Furthermore, the information processing device 100 may be configured to be able to communicate with an external device via a network, and the program 127 may be downloaded from the external device via the network.

[0036] Furthermore, the information processing device 100 is configured to execute the processes of the information processing unit 200 and the control unit 300, but is not limited to this. For example, the control process of the control unit 300 may be executed by a device other than the information processing device 100 or a CPU other than the CPU 121.

[0037] The information processing unit 200 calculates the holding position of the robot 150 for holding the workpiece 3 or 4, based on the input data acquired from the vision sensor 8 and the trained model 250.

[0038] The control unit 300 executes a workpiece holding operation and a workpiece transport operation by operating the robot arm 151 and the robot hand 152 based on the holding position information acquired from the information processing unit 200. The holding position includes information on a position in a three-dimensional space defined by a robot coordinate system. The robot coordinate system is a three-dimensional Cartesian coordinate system based on the robot 150, and has the base of the robot 150 as the origin, for example.

[0039] 1 in the learning phase, and in the inference phase, obtains a holding position using the trained model 250. The process of the information processing unit 200 in the inference phase will be specifically described below.

[0040] Fig. 2(b) is an explanatory diagram of the information processing unit 200 according to the first embodiment. The information processing unit 200 has an imaging control unit 210, an inference unit 220, and a holding position calculation unit 230. Fig. 3 is an explanatory diagram of an image 11 according to the first embodiment.

[0041] The imaging control unit 210 makes the vision sensor 8 capture an image of the base 5 to sense the workpieces 3 and 4, and acquires an image 11, which is an image captured by the vision sensor 8. The image 11 is an example of sensing data from the vision sensor 8, and is image data. That is, the sensing data output by the vision sensor 8 is the image 11 captured by the vision sensor 8 of the workpieces 3 and 4. The imaging direction of the vision sensor 8 is, for example, the vertical direction. A vision coordinate system is defined for the vision sensor 8. The X direction and the Y direction in the vision coordinate system are directions that intersect (are perpendicular to) the imaging direction. The X direction and the Y direction are directions that intersect (are perpendicular to) each other. The image 11 is a digital image having a plurality of pixels arranged in a row and column direction, i.e., in the XY direction.

[0042] The image 11 captures the workpieces 3 and 4 placed on the pedestal 5 as seen from the vision sensor 8. That is, the image 11 includes a workpiece image 3I composed of two or more pixels corresponding to the workpiece 3 and a workpiece image 4I composed of two or more pixels corresponding to the workpiece 4. The image 11 is an image in which the scenery as seen from the vision sensor 8 is projected onto the XY plane, which is a predetermined plane. The XY plane is a virtual plane.

[0043] 4 is an explanatory diagram of an inference process of the inference unit 220 according to the first embodiment. The inference unit 220 applies the trained model 250 to the machine learning model 12 and performs inference using the image 11 as input data.

[0044] A model that applies semantic segmentation is used as the machine learning model 12. Semantic segmentation is a machine learning algorithm that has a function of taking an image as input and taking an image as output, and has a feature of being able to output an output image in which label data is added to each pixel of the input image.

[0045] Machine learning models for performing semantic segmentation include U-Net and HRNet, but any machine learning model may be used as long as it is possible to assign label data to each pixel of the image 11.

[0046] In the first embodiment, a U-Net model having three output layers is used as the machine learning model 12. In the inference phase, a trained model 250 created in advance in the learning phase is applied to the machine learning model 12.

[0047] The inference unit 220 performs inference using the image 11 as input data for the trained model 250, and acquires three images 13, 14, and 15 as output data that is the result of the inference. Each of the three images 13 to 15 corresponds to the image 11, and is defined in the same vision coordinate system as the image 11. Therefore, the images 13, 14, and 15 are defined in the same XY plane as the image 11. In other words, the images 13, 14, and 15 acquired by inference of the trained model 250 are information that is defined in the same XY plane as the image 11. Each of the three images 13 to 15 has a plurality of pixels in the same arrangement as the arrangement of the image 11, and the same number of pixels as the number of pixels of the image 11.

[0048] The inference unit 220 generates two images 16 and 17 based on the three images 13, 14, and 15, and outputs the two images 16 and 17 to the holding position calculation unit 230. Each of the two images 16 and 17 corresponds to the image 11, and is defined in the same vision coordinate system as the image 11. Therefore, the images 16 and 17 are defined in the same XY plane as the image 11. Each of the two images 16 and 17 has a plurality of pixels in the same arrangement as the arrangement of the image 11 and the same number of pixels as the number of pixels of the image 11.

[0049] FIG. 5 is an explanatory diagram of image 13 according to the first embodiment. Note that FIG. 5 illustrates an image (11+13) in which image 11 and image 13 are superimposed. Specifically, image 13 is superimposed on image 11. The inference unit 220 displays an image based on image (11+13) shown in FIG. 5 on the display device 400. Note that the inference unit 220 may display an image based on image 11, image 13, or image (11+13) on the display device 400.

[0050] Image 13 shows areas that may be the positions of finger parts 153 or 154 when robot 150 is caused to perform a holding operation to hold workpiece 3 or 4, and areas other than these areas, in different colors (labels).

[0051] In image (11+13), areas 222, 223, 224, and 225 corresponding to workpiece 3 are located around workpiece image 3I, and areas 226 and 227 corresponding to workpiece 4 are located around workpiece image 4I. Areas 222 to 225 are areas that include multiple positions that can be the positions of fingers 153 or 154 when fingers 153 and 154 included in robot 150 start to execute a holding operation to hold workpiece 3. Areas 226 and 227 are areas that include multiple positions that can be the positions of fingers 153 or 154 when fingers 153 and 154 included in robot 150 start to execute a holding operation to hold workpiece 4.

[0052] In FIG. 5, the areas 222 to 227 shown in black are two-dimensional areas in the image 13, but correspond to areas viewed from the vision sensor 8 in the three-dimensional space of the real space in which the robot 150 is placed. The areas 222 to 227 are areas into which the finger 153 or 154 of the robot hand 152 can enter in the three-dimensional space of the real space. The areas 222 to 225 are set away from the workpiece 3 so that the finger 153 or 154 does not come into contact with the workpiece 3 when it enters. Similarly, the areas 226 and 227 are set away from the workpiece 4 so that the finger 153 or 154 does not come into contact with the workpiece 4 when it enters.

[0053] Moreover, the regions 222 to 225 are regions in the three-dimensional space of the real space where the finger portion 153 or 154 will not collide with objects around the workpiece 3 when the finger portion 153 or 154 enters a position within the regions 222 to 225. Moreover, the regions 226 and 227 are regions in the three-dimensional space of the real space where the finger portion 153 or 154 will not collide with objects around the workpiece 4 when the finger portion 153 or 154 enters a position within the regions 226 and 227.

[0054] Each pixel of the image 13 can be expressed by binary label data of "1" indicating that the finger 153 or 154 can enter and "0" indicating that the finger 153 and 154 cannot enter. In the display device 400, an area with label data of "1" is black, and an area with label data of "0" is white. The area to which label data of "1" is assigned is the areas 222 to 227. One pixel in the areas 222 to 227 represents the position of a predetermined part of the finger 153 or 154. The position of each pixel in the areas 222 to 227 can be the position of the finger 153 or 154 when the robot 150 is made to perform a holding operation. Note that each pixel of the image 13 may be expressed by label data of multiple gradations according to the margin of entry and the likelihood of the finger 153 or 154.

[0055] FIG. 6 is an explanatory diagram of image 14 according to the first embodiment. Note that FIG. 6 illustrates an image (11+14) in which image 11 and image 14 are superimposed. Specifically, image 14 is superimposed on image 11. The inference unit 220 displays an image based on image (11+14) shown in FIG. 6 on the display device 400. Note that the inference unit 220 may display an image based on image 11, image 14, or image (11+14) on the display device 400.

[0056] In image (11+14), areas 222, 223, 224, and 225 corresponding to workpiece 3 are located around workpiece image 3I, and areas 226 and 227 corresponding to workpiece 4 are located around workpiece image 4I.

[0057] The image 14 is an image in which the distance in the X direction of the finger part 154 (153) relative to the finger part 153 (154) is expressed by label data indicating gradation such as light and shade at each position (pixel) in the regions 222 to 227. In the first embodiment, label data indicating gradation is given to each position (pixel) in the regions 222 to 227, and is expressed by light and shade of black in the example of FIG.

[0058] 6, the whiter the color of each of the regions 222-227, that is, the closer the label data (grayscale value) of the pixel in each of the regions 222-227 is to "1", the greater the intensity of the X-direction component in the movement direction (vector) of the finger portion 153 or 154. Here, when focusing on one pixel in the regions 222-227, "intensity" is information that represents the distance from the first finger portion (e.g., finger portion 153) to the second finger portion (e.g., finger portion 154) at the position of that one pixel.

[0059] In this way, the image 14 is information on the X-direction component indicating the distance in the X-direction of the finger portion 154 (153) relative to the finger portion 153 (154) at each position within the regions 222 to 227 of the finger portions 153, 154. The X-direction is an example of a first direction. The X-direction component is an example of a first component. As described above, when the finger portion 153 is the first finger portion, the finger portion 154 is the second finger portion, and when the finger portion 154 is the first finger portion, the finger portion 153 is the second finger portion.

[0060] The strength is normalized, for example, by the distance between the two fingers 153, 154 required to grip the workpiece. In the example of FIG. 6, the strength is normalized by a value between "1" and "-1", with the X direction being positive. For example, when the strength is "0", the distance in the X direction of the second finger relative to the first finger is 0. In this case, the first finger and the second finger in the open state face each other in the Y direction. When the sign of the strength is positive, the second finger is located in the positive direction of the X axis relative to the first finger, and when the sign of the strength is negative, the second finger is located in the negative direction of the X axis relative to the first finger.

[0061] In the example of FIG. 6, at the position of each pixel in area 222, the second finger is positioned at a distance of about 30% of the opening width in the positive direction of the X-axis relative to the first finger, so label data (shading value) of “0.3” is assigned.

[0062] FIG. 7 is an explanatory diagram of image 15 according to the first embodiment. Note that FIG. 7 illustrates an image (11+15) in which image 11 and image 15 are superimposed. Specifically, image 15 is superimposed on image 11. The inference unit 220 displays an image based on image (11+15) shown in FIG. 7 on the display device 400. Note that the inference unit 220 may display an image based on image 11, image 15, or image (11+15) on the display device 400.

[0063] In image (11+15), areas 222, 223, 224, and 225 corresponding to workpiece 3 are located around workpiece image 3I, and areas 226 and 227 corresponding to workpiece 4 are located around workpiece image 4I.

[0064] In image 14, label data relating to distance in the X direction is given, whereas in image 15, label data relating to distance in the Y direction is given.

[0065] The image 15 is an image in which the distance in the Y direction of the finger part 154 (153) relative to the finger part 153 (154) is expressed by label data indicating gradation such as light and shade at each position (pixel) in the regions 222 to 227. In the first embodiment, label data indicating gradation is assigned to each point (pixel) in the regions 222 to 227, and is expressed by light and shade of black in the example of Fig. 7.

[0066] In the example of FIG. 7, the whiter the color of each of the regions 222-227, that is, the closer the label data (gray value) of the pixels in each of the regions 222-227 is to "1", the greater the strength of the Y-direction component of the movement direction (vector) of the finger 153 or 154.

[0067] In this way, the image 14 is information on the Y-direction component indicating the distance in the Y direction of the finger portion 154 (153) relative to the finger portion 153 (154) at each position within the regions 222 to 227 of the finger portions 153, 154. The Y direction is an example of the second direction. The Y-direction component is an example of the second component. As described above, when the finger portion 153 is the first finger portion, the finger portion 154 is the second finger portion, and when the finger portion 154 is the first finger portion, the finger portion 153 is the second finger portion.

[0068] FIG. 8 is an explanatory diagram of image 16 according to the first embodiment. Image 16 indicates the moving direction of finger portion 153 or 154. Note that FIG. 8 illustrates an image (11+16) in which image 11 and image 16 are superimposed. Specifically, image 16 is superimposed on image 11. The inference unit 220 displays an image based on image (11+16) shown in FIG. 8 on the display device 400. Note that the inference unit 220 may display an image based on image 11, image 16, or image (11+16) on the display device 400.

[0069] The inference unit 220 obtains an image 16 using the images 13, 14, and 15. The image 16 is an image in which the moving direction of the finger portion 153 or 154 is represented by label data indicating gradation such as light and dark when the position of each pixel in the areas 222 to 227 is set as the starting point of moving the finger portion 153 or 154 in a holding motion. In this way, the inference unit 220 causes the image displayed on the display device 400 to differ depending on the moving direction of the finger portion 153 or 154.

[0070] The movement direction of the fingers is the direction in which the fingers are moved to hold the workpiece after entering any position in the area shown in image 13 when the robot hand 152 is to hold the workpiece.

[0071] The movement direction is calculated by defining a vector in which label data indicating the distance given to pixels in regions 222-227 in image 14 is defined as the X coordinate, and label data indicating the distance given to pixels in regions 222-227 in image 15 is defined as the Y coordinate. For example, in region 226, when finger portion 153 is inserted on the left side of workpiece 4, the movement direction of finger portion 153 is to the right, and when finger portion 153 is inserted on the right side of workpiece 4, the movement direction of finger portion 153 is to the left.

[0072] In this way, the image 16 includes information on the movement direction of the finger 153 or 154 with each position in the areas 222 to 227 as the starting point of the holding motion. Note that, in the first embodiment, the inference unit 220 generates the image 16 from the images 13 to 15, but this is not limited to this. For example, the inference unit 220 may directly generate the image 16 by inference of the trained model 250. As described above, the inference unit 220 of the first embodiment acquires the image 16 based on inference of the trained model 250.

[0073] FIG. 9 is an explanatory diagram of image 17 according to the first embodiment. Note that FIG. 9 illustrates an image (11+17) in which image 11 and image 17 are superimposed. Specifically, image 17 is superimposed on image 11. The inference unit 220 displays an image based on image (11+17) shown in FIG. 9 on the display device 400. Note that the inference unit 220 may display an image based on image 11, image 17, or image (11+17) on the display device 400.

[0074] The inference unit 220 acquires an image 17 using the images 13, 14, and 15. The image 17 is an image in which, when the position of each pixel in the areas 222 to 227 is set as a starting point for moving the finger part 153 (154), the distance of the finger part 154 (153) to the finger part 153 (154) is expressed by label data indicating gradation such as light and dark.

[0075] The distance of the second finger part from the first finger part is the distance from the first finger part to the second finger part when the first finger part is inserted into any position in the area shown in the image 13.

[0076] The movement direction is calculated by defining a vector in which label data indicating the distance given to the pixels in the regions 222-227 in the image 14 is the X coordinate, and label data indicating the distance given to the pixels in the regions 222-227 in the image 15 is the Y coordinate. The distance is expressed by label data indicating a gradation such as light and dark according to the magnitude of the calculated vector. In the example of FIG. 9, the whiter the color of the pixels in the regions 222-227, the longer the distance. In this way, the inference unit 220 causes the image displayed on the display device 400 to differ depending on the distance of one finger part from the other finger part of the two finger parts 153, 154.

[0077] Note that, although the above description concerns a case in which the inference unit 220 performs inference processing on the entire image 11 as one area, this is not limited to this. For example, the inference unit 220 may divide the input image 11 into multiple areas and perform inference processing for each area.

[0078] In this way, the image 17 includes information on the distance of the finger portion 154 (153) relative to the finger portion 153 (154) at each position in the regions 222 to 227. Note that, in the first embodiment, the inference unit 220 generates the image 17 from the images 13 to 15, but this is not limited to this. For example, the inference unit 220 may directly generate the image 17 by inference of the trained model 250. As described above, the inference unit 220 of the first embodiment acquires the image 17 based on inference of the trained model 250.

[0079] Here, the learning process of the inference unit 220 in the learning phase will be described. The inference unit 220 generates a trained model 250 by having the machine learning model 12 perform supervised learning. To generate the trained model 250, a large number of pairs of the image 11 and the correct answer data (images 13, 14, 15) are prepared. Two examples of methods for creating the images 11, 13, 14, 15 used in machine learning will be described below. 1. The worker creates training data. 2. Create training data through computer simulation.

[0080] In the first method, the worker creates images 11, 13, 14, and 15. The worker randomly places the workpieces 2 on the pedestal 5 shown in FIG. 1, and inputs a command to the information processing device 100 to cause the vision sensor 8 of the information processing device 100 to capture an image of the workpiece 2. As a result, the information processing device 100 acquires the image 11. The worker creates images 13 to 15 by opening the file of the image 11 with a paint tool or the like and adding label data by coloring corresponding pixels. In this case, the worker creates images 13 to 15 while making decisions such as not coloring areas that are interfering with other workpieces or areas that are difficult to remove.

[0081] The second method is a method in which the information processing device 100 creates the images 11, 13, 14, and 15 by computer simulation. The simulation is performed in a virtual environment that reproduces the real environment. The information processing device 100 performs a simulation in which a virtual model corresponding to the workpiece 2 is randomly arranged on a virtual model corresponding to the pedestal 5, and generates an image 11 viewed from a position corresponding to the position of the vision sensor 8. Information on the positions where the finger portions 153 and 154 can grasp the workpiece is set in advance in the information processing device 100. This allows the information processing device 100 to calculate positions where the finger portions do not collide with objects around the workpiece to be held. Therefore, the information processing device 100 can automatically generate the images 13, 14, and 15.

[0082] In this way, in the learning phase, the information processing unit 200 generates the trained model 250 by supervised learning using data such as images 11, 13, 14, and 15.

[0083] Next, a description will be given of the processing of the holding position calculation unit 230. The holding position calculation unit 230 calculates information (holding position) for causing the fingers 153, 154 of the robot hand 152 to hold a workpiece by using various information calculated by the inference unit 220, i.e., at least one of the images 13 to 17.

[0084] The holding position includes information on the three-dimensional spatial position of the fingers 153, 154 of the robot hand 152 when the fingers 153, 154 start the holding operation to hold a workpiece, information on the direction of the fingers 153, 154, and information on the width (holding width) of the fingers 153, 154.

[0085] FIG. 10 is an explanatory diagram of the process of determining the holding position according to the first embodiment. In FIG. 10, an image (11+13) is illustrated. The holding position calculation unit 230 samples position candidates of the finger portion 153 or 154 when determining the holding position. In the first embodiment, the holding position calculation unit 230 samples a plurality of position candidates from within each of the regions 222 to 227 in the image 13. The work 3 will be described below. The process for the work 4 is similar to the process for the work 3, and therefore a detailed description will be omitted.

[0086] FIG. 11 is an explanatory diagram of a process for determining a holding position according to the first embodiment. A plurality of position candidates 2221, 2222, 2223, and 2224 are sampled from within the region 222. A plurality of position candidates 2231 and 2232 are sampled from within the region 223. A plurality of position candidates 2241 and 2242 are sampled from within the region 224. A plurality of position candidates 2251, 2252, and 2253 are sampled from within the region 225. From the region 222, the position candidates 2221, 2222, and 2223 are sampled to a predetermined density. From the region 223, the position candidates 2231 and 2232 are sampled to a predetermined density. From the region 224, the position candidates 2241 and 2242 are sampled to a predetermined density. From the region 225, the position candidates 2251, 2252, and 2253 are sampled to a predetermined density.

[0087] The holding position calculation unit 230 obtains a holding vector having each of the sampled position candidates as its starting point. For example, the holding position calculation unit 230 obtains a holding vector V1 having the position candidate 2221 as its starting point, and obtains a holding vector V2 having the position candidate 2251 as its starting point.

[0088] The retained vector V1 is a vector whose X coordinate is label data indicating the distance assigned to the pixel of the position candidate 2221 in image 14 and whose Y coordinate is label data indicating the distance assigned to the pixel of the position candidate 2221 in image 15. The retained vector V2 is a vector whose X coordinate is label data indicating the distance assigned to the pixel of the position candidate 2251 in image 14 and whose Y coordinate is label data indicating the distance assigned to the pixel of the position candidate 2251 in image 15.

[0089] In this way, the retained vector can be obtained from images 14 and 15. The retained vector is a vector that indicates the direction from the first finger part to the second finger part starting from the position candidate and the distance from the first finger part to the second finger part. The retained vector can also be obtained from images 16 and 17.

[0090] Next, the holding position calculation unit 230 extracts pairs that satisfy a predetermined condition from among the multiple holding vectors as candidate pairs. Specifically, the holding position calculation unit 230 extracts pairs that satisfy the following two conditions as candidate pairs. 1. The angle between the two holding vectors is greater than or equal to a specified angle. 2. Of the two retention vectors, the distance between the end point of one retention vector and the start point of the other retention vector is less than a certain value.

[0091] As described above, the holding position calculation unit 230 extracts two holding vectors that satisfy two conditions from among the multiple holding vectors, and extracts a pair of two position candidates corresponding to the two holding vectors as a candidate pair. In the example of FIG. 11, the position candidate 2221 corresponding to the holding vector V1 and the position candidate 2251 corresponding to the holding vector V2 are extracted as a candidate pair P1. Also, the pair of the position candidates 2222 and 2252 is extracted as a candidate pair P2. Also, the pair of the position candidates 2223 and 2253 is extracted as a candidate pair P3. Also, the pair of the position candidates 2231 and 2241 is extracted as a candidate pair P4. Also, the pair of the position candidates 2232 and 2242 is extracted as a candidate pair P5. In this way, the holding position calculation unit 230 finds the candidate pairs P1 to P5 for the work 3. Also, the holding position calculation unit 230 finds the candidate pairs for the work 3.

[0092] In this way, the holding position calculation unit 230 executes an extraction process to extract candidate pairs P1 to P5 including two position candidates corresponding to the two finger parts 153 and 154 that satisfy a predetermined condition from among a plurality of positions in the areas 222 to 227, using the images 14 and 15 or the images 16 and 17. Each of the candidate pairs P1 to P5 is an example of a group. The images 16 and 17 are information related to the images 14 and 15.

[0093] In addition, when the robot hand 152 has three or more fingers as the at least two fingers, the extracted group includes three or more position candidates as the at least two position candidates corresponding to the at least two fingers. The number of position candidates included in the group is the same as the number of fingers.

[0094] Fig. 12 is an explanatory diagram of the process of the holding position calculation unit 230 according to the first embodiment. In Fig. 5, a dashed line indicating a candidate pair is superimposed on the image 11. The holding position calculation unit 230 displays the image 11 indicating the dashed line indicating the candidate pair shown in Fig. 12 on the display device 400.

[0095] In addition, there may be cases where a candidate pair that satisfies the above two conditions cannot be found. Fig. 13 is an explanatory diagram of the process of the holding position calculation unit 230 according to the first embodiment. Fig. 13 illustrates a case where a candidate pair cannot be found.

[0096] 13, if there are no areas 224, 225 as shown in FIG. 11, extraction of a candidate pair fails. In such a case, there is a possibility that extraction of a candidate pair may be successful by changing the posture of the workpiece 3. Therefore, if the holding position calculation unit 230 fails to extract a candidate pair in the extraction process, the control unit 300 controls the robot 150 so that the robot 150 contacts the workpiece 3.

[0097] 13, in an area corresponding to area 225 in FIG. 11, there is no area 225 due to interference with another workpiece, and extraction of a candidate pair fails. On the other hand, in a position candidate 2221 corresponding to holding vector V1, entry of finger 153 or 154 is possible. In the first embodiment, information on the holding vector is used to change the posture of the workpiece 3. For example, the control unit 300 causes the finger 153 of the robot hand 152 to enter area 222 and moves the finger 153 in the direction of holding vector V1. This makes it possible to change the position and / or posture of the workpiece 3, and there is a possibility that a new candidate pair may be successfully extracted.

[0098] For example, when the extraction of the candidate pair P1 is successful in the extraction process, the holding position calculation unit 230 determines information on the positions of the fingers 153, 154 based on the extracted candidate pair P1. That is, when the extraction of the candidate pair P1 is successful, the holding position calculation unit 230 sets information on the position candidates of the fingers 153, 154 included in the extracted candidate pair P1 as information on the positions of the fingers 153, 154 (holding pair). For example, the holding position calculation unit 230 sets one determined candidate pair P1 as the holding pair P1 from among the multiple candidate pairs P1 to P5. Then, the holding position calculation unit 230 calculates the holding position from the determined holding pair P1. That is, the holding pair P1 is information on the positions in the XY directions, and the holding position calculation unit 230 acquires information on the positions in the direction intersecting the XY plane to acquire information on the positions in the three-dimensional space corresponding to the determined holding pair P1. The information on the positions in the three-dimensional space is information on the positions in the robot coordinate system.

[0099] Figures 14 to 16 are explanatory diagrams of a method for determining a position in a three-dimensional space according to the first embodiment. Figures 14 to 16 show mutually different methods, and any one of the methods is implemented.

[0100] The method shown in Fig. 14 will be described. The holding pair P1 is only a holding pair in the image 11. In order to convert the holding pair P1, which is information on the two-dimensional positions 2221, 2251, into the holding position 236 from which the robot 150 can pick up the workpiece 3, it is necessary to calculate the three-dimensional depth direction.

[0101] In the first embodiment, the inference unit 220 acquires information on a position in a direction intersecting the XY plane based on the image 11, which is sensing data. Specifically, the inference unit 220 calculates a distance 241 in the depth direction from the vision sensor 8 by inference using the trained model 250. The inference unit 220 outputs, as output data, the three-dimensional distance 241 from the vision sensor 8 to a three-dimensional position corresponding to the position candidates 2221, 2251 of the retention pair P1.

[0102] The holding position calculation unit 230 converts the two-dimensional information of the positions 2221, 2251 of the holding pair P1 into three-dimensional information from the data of the distance 241, thereby calculating the holding position 236. As described above, the holding position 236 includes information on the three-dimensional spatial positions of the fingers 153, 154 at which the fingers 153, 154 of the robot hand 152 start a holding operation to hold a workpiece, information on the orientation of the fingers 153, 154, and information on the width (holding width) of the fingers 153, 154.

[0103] The method shown in Fig. 15 will be described. In this method, the holding position calculation unit 230 calculates the holding position 236 by assuming that the three-dimensional position where the workpiece 3 exists corresponding to the positions 2221 and 2251 is a virtual surface 242 which is a virtual plane or a virtual curved surface.

[0104] Alternatively, if the vision sensor 8 has a three-dimensional distance measurement function, the holding position calculation unit 230 acquires the results of measuring the distance to the workpiece 3 from the vision sensor 8 (distance measurement information), and acquires position information (holding position 236) in the direction intersecting the XY plane based on the distance measurement information.

[0105] The method shown in Fig. 16 will be described. The control unit 300 controls the robot 150 so that the finger portions 153, 154 move in a three-dimensional space, and the holding position calculation unit 230 acquires information on the position in a direction intersecting the XY plane based on the detection result of a sensor 155 such as a proximity sensor provided on one of the finger portions 153, 154. The sensor 155 is a sensor capable of detecting the presence or absence of a workpiece, and is provided, for example, on the inner side of the finger portion 153 (the side facing the finger portion 154).

[0106] Specifically, the holding position calculation unit 230 first sets an upper limit plane 243 and a lower limit plane 244 of the workpiece to be taken out. The planes 243 and 244 are virtual planes set in a three-dimensional space. After that, the holding position calculation unit 230 calculates a holding position 245 and a holding position 246 when the holding pair P1 is projected onto each of the planes 243 and 244.

[0107] In an actual operation of holding the workpiece 3, the control unit 300 first moves the robot hand 152 to the holding position 245, and then moves the robot hand 152 at a low speed from the holding position 245 to the holding position 246. At that time, the control unit 300 detects the presence or absence of the workpiece 3 using a sensor such as a proximity sensor attached to the inner side of the fingers of the robot hand 152, stops the robot hand 152 when it reaches a predetermined holding position 236, and performs the holding operation from the stopped position. The approach direction of the robot hand 152 is preferably a direction going linearly from the vision sensor 8 to the holding position 236.

[0108] 17 is an explanatory diagram of the control unit 300 according to the first embodiment. The control unit 300 has a robot control unit 310 and a robot hand control unit 320. The robot control unit 310 controls the operation of the robot arm 151 based on information on the holding position 236 calculated by the information processing unit 200. As described above, the holding position 236 includes information on a position in three-dimensional space. The robot hand control unit 320 controls the robot hand 152 based on information on the holding width included in the holding position 236 calculated by the information processing unit 200, thereby causing the robot hand 152 to hold a workpiece or release the held workpiece.

[0109] As described above, according to the first embodiment, in the learning phase, instead of treating each individual position as correct answer data, the information processing unit 200 performs machine learning using an area including many positions as correct answer data to generate the trained model 250, thereby reducing the variability in the correct answer data, and thus improving the accuracy of learning. Then, in the inference phase, the information processing unit 200 performs inference using such a trained model 250, and thus a unique inference result is obtained as an area including many positions, thereby improving the accuracy of inference.

[0110] For example, when the workpiece 4 is imaged from the bottom side of the cylindrical portion, as shown in FIG. 3, the workpiece 4 becomes a cylindrical workpiece image 4I in the image 11 obtained by imaging. In this case, the robot hand 152 having two fingers 153, 154 can hold the workpiece 4 at multiple positions over 360 degrees around the holding center point of the cylindrical portion. In the first embodiment, the information processing unit 200 is made to machine-learn the regions 226, 227 including multiple positions that the fingers 153, 154 can enter, instead of the holding center, so that the information processing unit 200 can machine-learn the same input data among multiple input data as unique correct answer data expressing all circumferential directions of the cylindrical portion. Since the information processing unit 200 performs inference using such a learned model 250, the inference result is uniquely obtained as the regions 226, 227 including multiple positions, and the accuracy of inference can be improved.

[0111] For example, when the upper and lower parts 31 and 32 of the workpiece 3 are imaged from the side, as shown in FIG. 3, the workpiece 3 is imaged as a workpiece image 3I consisting of two parts with different widths in the image 11 obtained by imaging. In such a case, the robot hand 152 having two finger parts 153 and 154 can hold the workpiece 3 at a plurality of positions of the upper and lower parts 31 and 32. In the first embodiment, the information processing unit 200 is made to machine-learn the regions 222-225 including a plurality of positions into which the finger parts 153 and 154 can enter, so that the information processing unit 200 can machine-learn the same input data among a plurality of input data as unique correct answer data expressing both the upper and lower parts 31 and 32. Then, the information processing unit 200 performs inference using such a trained model 250, and therefore a unique inference result is obtained as the regions 222-225 including a large number of positions, thereby improving the accuracy of inference.

[0112] In this way, since correct answer data for the input data is uniquely determined, the accuracy of machine learning is improved, and as a result, the accuracy of inference is improved. Also, since the area where the finger portion 153 or 154 can enter is calculated, the accuracy of interference determination can be improved. This increases the probability that the robot 150 will succeed in holding the workpiece, and improves the workability of the robot 150.

[0113] In the above description, the image 16 is generated by the inference unit 220. However, if information on the direction of the fingers is not necessary for controlling the robot hand 152, the generation of the image 16 can be omitted.

[0114] In the above description, the image 17 is generated by the inference unit 220. However, if information on the width of the fingers is not necessary for controlling the robot hand 152, the generation of the image 17 can be omitted.

[0115] [Second embodiment] Even in a workpiece holding method that uses 3D shape matching technology and point cloud-based interference detection technology, the accuracy of interference detection is low. In this method, the holding position for removing the workpiece is determined by combining the information on the holding position set for the workpiece in advance with the position and orientation of the workpiece calculated by 3D shape matching technology. At that time, point cloud information is used to prevent the fingers of the robot hand from interfering with workpieces other than the workpiece to be held or with surrounding objects. However, point cloud information is easily affected by missing parts and blind spots, so interference detection may not be performed accurately. Note that "interference" includes "collision."

[0116] The second embodiment will be described below. In the following, elements with the same reference symbols as those in the first embodiment will have substantially the same configurations and functions as those described in the first embodiment unless otherwise specified, and differences from the first embodiment will be mainly described.

[0117] The hardware configuration of the robot system in the second embodiment is similar to the hardware configuration of the robot system 1 in the first embodiment. That is, the processing of the information processing unit in the second embodiment is partially different from the processing of the information processing unit 200 described in the first embodiment. The parts that differ from the first embodiment will be described below.

[0118] Fig. 18 is an explanatory diagram showing a schematic configuration of a robot system 1A according to the second embodiment. The robot system 1A includes a pedestal 5, a vision sensor 8, a robot 150, an information processing device 100, and a display device 400. The information processing device 100 has an information processing unit 200A that processes information, and a control unit 300 that controls the robot 150. The information processing unit 200A and the control unit 300 are configured, for example, by the CPU 121 shown in Fig. 2(a) executing the program 127. Workpieces 3 and 4 are placed on the pedestal 5.

[0119] In the above first embodiment, a method for calculating a holding position from information such as the position of the finger parts is described. In the second embodiment, a method is described in which the information processing unit 200A determines whether or not the finger parts 153, 154 can be removed without interfering with surrounding objects at a holding position calculated by a method different from the method described in the first embodiment, based on, for example, the image 13 shown in Fig. 4. This makes it possible to solve the interference problem.

[0120] 19 is an explanatory diagram of an information processing unit 200A according to the second embodiment. The information processing unit 200A has an imaging control unit 210, an inference unit 220, a workpiece recognition unit 410, and a holding position calculation unit 420.

[0121] The functions of the imaging control unit 210 and the inference unit 220 are as described in the first embodiment.

[0122] The workpiece recognition unit 410 recognizes the three-dimensional positions and orientations of the workpieces 3 and 4 from the image 11 obtained from the imaging control unit 210, and outputs the information on the positions and orientations. For example, the workpiece recognition unit 410 recognizes the three-dimensional positions and orientations of the workpieces 3 and 4 by performing matching using a workpiece model, which is three-dimensional information of the workpieces 3 and 4.

[0123] The workpiece model may be any data as long as it is a model corresponding to the workpieces 3 and 4, and is, for example, contour information of a CAD model or three-dimensional point cloud information. The workpiece recognition unit 410 recognizes the position and orientation of the workpiece in the three-dimensional space by template matching using the contour information of the CAD model or three-dimensional matching using the three-dimensional point cloud information.

[0124] The method for recognizing the positions and orientations of the workpieces 3 and 4 is not limited to these methods, and any other known method may be used.

[0125] Fig. 20 and Fig. 21 are explanatory diagrams of the holding position calculation unit 420 according to the second embodiment. Fig. 20 shows a schematic diagram of the workpiece 3 recognized by the workpiece recognition unit 410 and holding positions 431, 432 in a three-dimensional space relative to the position and orientation of the workpiece 3.

[0126] A method for calculating the holding positions 431 and 432 will be described. First, the holding position calculation unit 420 calculates the two-dimensional holding position using a known technique. After that, the holding position calculation unit 420 multiplies the spatial position and orientation of the workpiece 3 calculated using the workpiece recognition unit 410 by the two-dimensional holding position to calculate the spatial holding positions 431 and 432.

[0127] That is, the holding position calculation unit 420 acquires holding positions 431, 432 based on the three-dimensional positions and orientations of the workpieces 3, 4 recognized by the workpiece recognition unit 410. Each of the holding positions 431, 432 is a group including two three-dimensional position candidates corresponding to the two finger portions 153, 154. In this manner, the holding position calculation unit 420 acquires one or more groups.

[0128] At the holding positions 431, 432 calculated by this method, there is a possibility that the finger portions 153, 154 of the robot hand 152 may come into contact with workpieces around the workpiece 3 to be held when taking out the workpiece. For example, among the holding positions 431, 432, at the holding position 431, the finger portions 153 or 154 of the robot hand 152 do not interfere with the workpieces around the workpiece 3, but at the holding position 432, the finger portions 153 or 154 of the robot hand 152 interfere with the workpieces around the workpiece 3.

[0129] 21 shows an image in which holding positions 431, 432 are superimposed on, for example, image 13 shown in FIG. 4, and holding position calculation unit 420 displays the image shown in FIG. 21 on display device 400. Note that image 13 differs from that in the first embodiment in that, of areas 222-225 shown in FIG. 5, another workpiece exists in area 222, and therefore areas 223-225 are found by inference. In other words, inference unit 220 outputs image 13 including areas 223-225 where finger portion 153 or 154 does not collide with surrounding objects as output data.

[0130] 21, holding position 431 overlaps with areas 223 and 224 into which finger portion 153 or 154 enters, and holding position 432 has area 225 into which one of two fingers 153 and 154 can enter, but does not have an area into which the other finger can enter.

[0131] As described above, the holding position calculation unit 420 extracts the holding position 431 included in the areas 223 to 225 from the holding positions 431, 432, and obtains information on the positions of the finger portions 153, 154 in the three-dimensional space based on the extracted holding position 431.

[0132] By performing such processing, it is possible to obtain a holding position 431 where the fingers 153 and 154 of the robot hand 152 do not interfere with objects other than the workpiece to be held, even in a state where a plurality of workpieces are randomly piled on the pedestal 5. In such interference determination, information on the images 16 and 17 of the fingers may be added in addition to the image 13 to determine whether the holding position satisfies the conditions.

[0133] As described above, according to the second embodiment, the area into which the finger portion 153 or 154 can enter is calculated by inference by the inference unit 220. By combining the machine learning technology with the matching technology, interference determination can be performed with higher accuracy, and as a result, the success rate of holding the workpiece can be increased.

[0134] Note that the second embodiment or the modified example of the second embodiment may be combined with at least one of the first embodiment and the modified example of the first embodiment.

[0135] The present disclosure is not limited to the above-described embodiments, and the embodiments can be modified in many ways within the technical concept of the present disclosure. For example, at least two of the above-described embodiments and modifications may be combined. In addition, the effects described in the present embodiment are merely a list of the most favorable effects resulting from the embodiments of the present disclosure, and the effects of the embodiments of the present disclosure are not limited to those described in the present embodiment.

[0136] In the above embodiment, the robot arm 151 is a vertically multi-jointed robot arm, but the present invention is not limited to this. The robot arm 151 may be, for example, a horizontally multi-jointed robot arm, a parallel link robot arm, or an orthogonal robot. The present disclosure is also applicable to a machine that can automatically perform operations such as extension and contraction, bending and stretching, vertical movement, horizontal movement, or rotation, or a combination of these operations, based on information in a storage device provided in a control device.

[0137] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0138] The disclosure of the above embodiments includes the following sections.

[0139] (Section 1) An information processing unit that processes information, The information processing unit includes: The sensing data is acquired from the vision sensor that senses the workpiece, Performing inference using the sensing data as input data for a trained model; acquiring information about an area including a plurality of positions that may be positions of a first finger of at least two fingers included in a robot when the robot is caused to perform a holding operation to hold the workpiece by inference of the trained model; 23. An information processing apparatus comprising:

[0140] (Section 2) The information processing unit includes: acquiring information on a moving direction of the first finger part with each position within the region as a starting point of the holding motion based on an inference of the trained model; 2. The information processing device according to item 1,

[0141] (Section 3) The information processing unit includes: acquiring information on a distance of a second finger part of the at least two finger parts relative to the first finger part at each position within the region based on an inference of the trained model; 3. The information processing device according to item 1 or 2,

[0142] (Section 4) The sensing data is an image of a workpiece captured by the vision sensor. 4. The information processing device according to any one of items 1 to 3.

[0143] (Section 5) The region is a region into which the first finger portion can enter. 5. The information processing device according to any one of items 1 to 4.

[0144] (Section 6) The region is a region in which the first finger portion does not collide with objects around the workpiece when the first finger portion is inserted. 6. The information processing device according to item 5,

[0145] (Section 7) The output data acquired by inference of the trained model is information defined on the same predetermined plane as the captured image. 5. The information processing device according to item 4,

[0146] (Section 8) The information processing unit includes: As output data of the trained model, information on the region, information on a first component indicating a distance in a first direction of a second finger part relative to the first finger part at each position in the region among the at least two finger parts, and information on a second component indicating a distance in a second direction intersecting with the first direction of the second finger part relative to the first finger part at each position in the region are acquired. 8. The information processing device according to item 7,

[0147] (Section 9) the information processing unit acquires information on a moving direction of the first finger portion with each position within the area as a starting point of the holding motion, using the information on the area, the information on the first component, and the information on the second component. 9. The information processing device according to item 8,

[0148] (Section 10) the information processing unit obtains information on a distance of a second finger portion of the at least two fingers relative to the first finger portion at each position within the region, using information on the region, information on the first component, and information on the second component. 10. The information processing device according to item 9,

[0149] (Section 11) the information processing unit executes an extraction process to extract, from a plurality of positions within the region, a group including at least two position candidates corresponding to the at least two finger portions that satisfy a predetermined condition, using the output data or information related to the output data; determining information on the positions of the at least two finger portions based on the extracted group when the extraction process is successful in extracting the group; 11. The information processing device according to any one of items 8 to 10,

[0150] (Section 12) A control unit for controlling the robot is further provided. When the information processing unit fails to extract the group in the extraction process, the control unit controls the robot so that the robot contacts the workpiece. 12. The information processing device according to item 11,

[0151] (Section 13) the information processing unit acquires position information in a direction intersecting the predetermined plane, and acquires position information in a three-dimensional space corresponding to the determined position information of each of the at least two finger portions. 13. The information processing device according to item 11 or 12,

[0152] (Section 14) The information processing unit acquires position information in a direction intersecting the predetermined plane based on the sensing data. 14. The information processing device according to item 13,

[0153] (Section 15) The information processing unit acquires position information in a direction intersecting the predetermined plane based on a result of measuring the distance to the workpiece. 14. The information processing device according to item 13,

[0154] (Section 16) A control unit for controlling the robot is further provided. The control unit controls the robot to move the at least two fingers, The information processing unit acquires position information in a direction intersecting the predetermined plane based on a detection result of a sensor capable of detecting a workpiece, the sensor being provided on one of the at least two finger portions. 14. The information processing device according to item 13,

[0155] (Section 17) The information processing unit includes: Recognizing the position and orientation of the workpiece in a three-dimensional space from the sensing data; obtaining one or more groups including at least two three-dimensional space position candidates corresponding to the at least two finger portions based on the recognized three-dimensional space position and orientation of the workpiece; Extracting a group included in the region from the one or more groups; acquiring information on the three-dimensional space positions of the at least two finger portions based on the extracted groups; 11. The information processing device according to any one of items 1 to 10.

[0156] (Section 18) A control unit for controlling the robot is further provided. The control unit controls the robot using at least the position information of the three-dimensional space. 18. The information processing device according to any one of items 13 to 17,

[0157] (Section 19) The information processing unit recognizes a position and orientation of the workpiece in a three-dimensional space by matching the position and orientation of the workpiece with a workpiece model corresponding to the workpiece. 20. The information processing device according to item 17,

[0158] (Section 20) The information processing unit includes: an image based on the information of the region is displayed on a display unit; 20. The information processing device according to any one of items 1 to 19,

[0159] (Section 21) The information processing unit includes: an image based on the sensing data and an image based on the information on the region are displayed on the display unit in a superimposed manner; 21. The information processing device according to item 20,

[0160] (Section 22) The information processing unit includes: acquiring information on a moving direction of the first finger part with each position within the region as a starting point of the holding motion based on an inference of the trained model; an image based on the information of the region is displayed on the display unit in a different manner depending on the moving direction; 22. The information processing device according to item 20 or 21,

[0161] (Section 23) The information processing unit includes: acquiring information on a distance of a second finger part of the at least two finger parts relative to the first finger part at each position within the region based on an inference of the trained model; an image based on the information of the region is displayed on the display unit in a manner that varies depending on the distance; 23. The information processing device according to any one of items 20 to 22.

[0162] (Section 24) The information processing unit generates the trained model by supervised learning. 24. The information processing device according to any one of items 1 to 23,

[0163] (Section 25) An information processing method for performing information processing, The sensing data is acquired from the vision sensor that senses the workpiece, Performing inference using the sensing data as input data for a trained model; acquiring information about an area including a plurality of positions that may be positions of a first finger of at least two fingers included in a robot when the robot is caused to perform a holding operation to hold the workpiece by inference of the trained model; 23. An information processing method comprising:

[0164] (Section 26) An information processing device according to any one of items 1 to 24, A robot having at least two fingers, the robot being controlled by the information processing device. A robot system comprising:

[0165] (Section 27) Item 27. A method for manufacturing an article, comprising the steps of: manufacturing an article using the robot system according to item 26.

[0166] (Section 28) Item 26. A program for causing a computer to execute the information processing method according to item 25.

[0167] (Section 29) Item 29. A computer-readable recording medium having the program according to item 28 recorded thereon. [Explanation of symbols]

[0168] 1... robot system, 8... vision sensor, 100... information processing device, 150... robot, 153... finger unit, 154... finger unit, 200... information processing unit, 250... trained model

Claims

1. An information processing unit that processes information, The information processing unit includes: The sensing data is acquired from the vision sensor that senses the workpiece, Performing inference using the sensing data as input data for a trained model; acquiring information on an area including a plurality of positions that may be positions of a first finger of at least two fingers included in a robot when the robot is caused to perform a holding operation to hold the workpiece by inference of the trained model; 23. An information processing apparatus comprising:

2. The information processing unit includes: acquiring information on a moving direction of the first finger portion with each position within the region as a starting point of the holding motion based on an inference of the trained model; 2. The information processing apparatus according to claim 1,

3. The information processing unit includes: acquiring information on a distance of a second finger part of the at least two finger parts relative to the first finger part at each position within the region based on an inference of the trained model; 2. The information processing apparatus according to claim 1,

4. The sensing data is an image of a workpiece captured by the vision sensor.

2. The information processing apparatus according to claim 1,

5. The region is a region into which the first finger portion can enter.

2. The information processing apparatus according to claim 1,

6. The region is a region in which the first finger portion does not collide with objects around the workpiece when the first finger portion is inserted.

6. The information processing apparatus according to claim 5,

7. The output data acquired by inference of the trained model is information defined on the same predetermined plane as the captured image.

5. The information processing apparatus according to claim 4.

8. The information processing unit includes: As output data of the trained model, information on the region, information on a first component indicating a distance in a first direction of a second finger part relative to the first finger part at each position in the region among the at least two finger parts, and information on a second component indicating a distance in a second direction intersecting with the first direction of the second finger part relative to the first finger part at each position in the region are acquired.

8. The information processing apparatus according to claim 7,

9. the information processing unit acquires information on a moving direction of the first finger portion with each position within the area as a starting point of the holding motion, using information on the area, information on the first component, and information on the second component.

9. The information processing apparatus according to claim 8,

10. the information processing unit obtains information on a distance of a second finger portion of the at least two fingers relative to the first finger portion at each position within the region, using information on the region, information on the first component, and information on the second component.

10. The information processing apparatus according to claim 9,

11. the information processing unit executes an extraction process to extract, from a plurality of positions within the region, a group including at least two position candidates corresponding to the at least two finger portions that satisfy a predetermined condition, using the output data or information related to the output data; determining information on the positions of the at least two finger portions based on the extracted group when the extraction process is successful in extracting the group; 9. The information processing apparatus according to claim 8,

12. A control unit for controlling the robot is further provided. When the information processing unit fails to extract the group in the extraction process, the control unit controls the robot so that the robot contacts the workpiece.

12. The information processing apparatus according to claim 11 .

13. the information processing unit acquires position information in a direction intersecting the predetermined plane, and acquires position information in a three-dimensional space corresponding to the determined position information of each of the at least two finger portions.

12. The information processing apparatus according to claim 11 .

14. The information processing unit acquires position information in a direction intersecting the predetermined plane based on the sensing data.

14. The information processing apparatus according to claim 13,

15. The information processing unit acquires position information in a direction intersecting the predetermined plane based on a result of measuring the distance to the workpiece.

14. The information processing apparatus according to claim 13,

16. A control unit for controlling the robot is further provided. The control unit controls the robot to move the at least two fingers, The information processing unit acquires position information in a direction intersecting the predetermined plane based on a detection result of a sensor capable of detecting a workpiece, the sensor being provided on one of the at least two finger portions.

14. The information processing apparatus according to claim 13,

17. The information processing unit includes: Recognizing a three-dimensional position and orientation of the workpiece from the sensing data; acquiring one or more groups including at least two three-dimensional space position candidates corresponding to the at least two finger portions based on the recognized three-dimensional space position and orientation of the workpiece; Extracting a group included in the region from the one or more groups; acquiring information on the three-dimensional space positions of the at least two finger portions based on the extracted groups; 2. The information processing apparatus according to claim 1,

18. A control unit for controlling the robot is further provided. The control unit controls the robot using at least the position information of the three-dimensional space.

14. The information processing apparatus according to claim 13,

19. The information processing unit recognizes a position and orientation of the workpiece in a three-dimensional space by matching the position and orientation of the workpiece with a workpiece model corresponding to the workpiece.

18. The information processing apparatus according to claim 17,

20. The information processing unit includes: an image based on the information of the region is displayed on a display unit; 2. The information processing apparatus according to claim 1,

21. The information processing unit includes: an image based on the sensing data and an image based on the information on the region are displayed on the display unit in a superimposed manner; 21. The information processing apparatus according to claim 20,

22. The information processing unit includes: acquiring information on a moving direction of the first finger portion with each position within the region as a starting point of the holding motion based on an inference of the trained model; an image based on the information of the region is displayed on the display unit in a different manner depending on the moving direction; 21. The information processing apparatus according to claim 20,

23. The information processing unit includes: acquiring information on a distance of a second finger part of the at least two finger parts relative to the first finger part at each position within the region based on an inference of the trained model; an image based on the information of the region is displayed on the display unit in a manner that varies depending on the distance; 21. The information processing apparatus according to claim 20,

24. The information processing unit generates the trained model by supervised learning.

2. The information processing apparatus according to claim 1,

25. An information processing method for performing information processing, The sensing data is acquired from the vision sensor that senses the workpiece, Performing inference using the sensing data as input data for a trained model; acquiring information on an area including a plurality of positions that may be positions of a first finger of at least two fingers included in a robot when the robot is caused to perform a holding operation to hold the workpiece by inference of the trained model; 23. An information processing method comprising:

26. An information processing device according to any one of claims 1 to 24; A robot having at least two fingers, the robot being controlled by the information processing device. A robot system comprising:

27. A method for manufacturing an article, comprising the steps of: manufacturing an article by using the robot system according to claim 26;

28. A program for causing a computer to execute the information processing method according to claim 25.

29. A computer-readable recording medium having the program according to claim 28 recorded thereon.

Images