Method for generating recognizers and holding deformable objects

Abstract

To provide a method for generating a recognition unit which can easily generate a highly accurate recognition unit.SOLUTION: The method for generating a recognition unit according to an embodiment of the present invention includes the steps in which: a three-dimensional scan unit 10 performs a three-dimensional scan of the whole periphery of a flexible object in a constant shape; a three-dimensional model of the flexible object is generated from the result of the three-dimensional scan; deformation processing is performed on the three-dimensional model so that a plurality of pieces of synthesis image data are generated; and the recognition unit is trained by using the synthesis image data and the three-dimensional model as learning data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a method for generating a recognizer. [Background technology]

[0002] Patent Document 1 discloses an object recognition device having a training image generation unit that creates a training image of the shape of an object. The training image generation unit generates a 3DCG image of the object. The training image generation unit generates a 2D appearance image by projecting a 3D CG image onto a 2D plane. Furthermore, the object recognition device changes the 3DCG basic image by applying blurring. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-211732 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, a learning image is created by blurring a three-dimensional CG base image. However, since the learning image is generated from a CG image, it is difficult to improve the recognition accuracy of the recognizer.

[0005] The present disclosure has been made to solve such problems, and provides a recognizer generation method that can easily generate a highly accurate recognizer. [Means for solving the problem]

[0006] The method for generating a recognizer in this embodiment includes the steps of: using a 3D scanner to 3D scan the entire circumference of a flexible object while the shape of the flexible object is constant; generating a 3D model of the flexible object from the 3D scan results; performing a deformation process on the 3D model to generate multiple composite image data; and training a recognizer using the 3D model and the composite image data as learning data. Effect of the Invention

[0007] The present disclosure provides a method for generating a recognizer that can learn a recognizer with high recognition performance. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a schematic diagram for explaining the configuration of a learning system and its processing. [Diagram 2] FIG. 2 is a schematic diagram for explaining the configuration of a learning system and its processing. [Diagram 3] FIG. 1 is a block diagram showing a configuration of a learning system. [Figure 4] FIG. 13 is a diagram for explaining a correspondence relationship in a transformation process. [Diagram 5] 1 is a flowchart illustrating a method for generating a recognizer. [Figure 6] FIG. 1 is a schematic diagram showing a configuration of a robot arm equipped with a recognizer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems.

[0010] (System Configuration) The learning system according to the present embodiment is a learning system for generating a recognizer for an object. Specifically, the learning system generates learning data based on sensing data obtained by sensing an object. Then, the learning system generates a recognizer for the object by machine learning using the learning data. The recognizer is a machine learning model that recognizes the name, type, position, orientation, shape, and the like of an object. Here, the object to be recognized by the recognizer can be a deformable flexible object. For example, the object is a flexible object that deforms due to an external force. The object is deformed when it is grasped by a robot, or when it is positioned by changing its orientation.

[0011] A learning system 1 according to an embodiment and a method for generating a recognizer by the learning system 1 will be described with reference to Figs. 1 to 3. Figs. 1 and 2 are schematic diagrams for explaining the learning system 1 and its processing flow. Fig. 3 is a block diagram showing the configuration of the learning system 1. As shown in Figs. 1 to 3, the learning system 1 includes a three-dimensional scanner 10 and a processing device 20.

[0012] The processing device 20 can be implemented as a device capable of executing a program, such as a central processing unit (CPU) of a computer, and various functions can also be realized by a program.

[0013] First, the generation of learning data will be described. The 3D scanner 10 performs a 3D scan of the object O. The 3D scanner 10 includes a sensor unit 11 and an arm mechanism 12. The arm mechanism 12 has an actuator that changes the position and orientation of the sensor unit 11. Specifically, the arm mechanism 12 is provided with a plurality of joint mechanisms having a motor and the like. The sensor unit 11 is attached to the tip of the arm mechanism 12. The arm mechanism 12 moves the sensor unit 11, so that the object O can be captured from various directions.

[0014] 3, the sensor unit 11 includes a camera 111 and a depth sensor 112. The camera 111 is a color camera that captures a color image of an object. In other words, RGB data of the object O is obtained based on the detection result of the camera 111. The object O is, for example, a plastic packaging bag for packaging food or the like.

[0015] The depth sensor 112 measures the depth (distance) to an object. For example, it is a LiDAR distance measuring sensor. Specifically, the depth sensor 112 includes a laser light source that generates laser light and a photodetector that detects the laser light reflected by the object O. The depth sensor 112 measures the distance to the object O based on the round-trip time it takes for the laser light to reflect off the object O and return to the depth sensor 112.

[0016] Three-dimensional shape data of an object is obtained based on the detection result of the depth sensor 112. That is, the depth sensor 112 obtains the three-dimensional shape data by detecting the distance to the surface of the object. The three-dimensional shape data is a depth image showing the surface shape of the object O, that is, the boundary between the object O and space.

[0017] Here, object O is suspended by wires 15. Therefore, the 3D scanner 10 can 3D scan the entire circumference of object O. Specifically, when object O is suspended by wires 15, the shape of object O does not change. With the shape of object O remaining constant, the 3D scanner 10 3D scans the entire circumference of object O.

[0018] Specifically, the arm mechanism 12 changes the position and orientation of the sensor unit 11. For example, the arm mechanism 12 operates so that the sensor unit 11 moves around the object O. This allows a three-dimensional scan of the entire circumference of the object O. Furthermore, since the object O is suspended by the wire 15, the entire object O can be three-dimensionally scanned. The arm mechanism 12 moves the sensor unit 11 below the object O, allowing the sensor unit 11 to sense the lower surface of the object O.

[0019] It is preferable that the wire 15 has a width equal to or smaller than the detection resolution of the sensor unit 11. For example, by making the width equal to or smaller than the detection resolution of the depth sensor 112, the wire 15 is not reflected in the three-dimensional shape data of the depth sensor 112. In this way, the three-dimensional shape of the object O can be measured more accurately. The wire 15 may also be made of a transparent material so as not to be detected by the camera 111. This allows the RGB data of the object O to be measured more accurately. can be obtained. Of course, the wire 15 may be made of a metal material or a resin material.

[0020] Although the object O is suspended by the wire 15, the method of holding the object O is not limited to this. For example, the object O may be placed on a mesh. In this way, the entire circumference of the object O can be three-dimensionally scanned while the shape of the object O remains constant. In this case, the mesh may be formed with wires of a resolution equal to or less than the detection resolution. Of course, if the wire 15 is detected, the processing device 20 may perform post-processing to remove the wire 15 by post-processing.

[0021] The 3D scanner 10 performs a 3D scan. The sensor data acquisition unit 21 acquires sensor data obtained by the 3D scan. For example, the sensor data acquisition unit 21 acquires depth data and RGB. The sensor data acquisition unit 21 acquires 3D scan data indicated by the 3D scan result of the 3D scanner 10.

[0022] The three-dimensional model generating unit 22 generates a three-dimensional model M from the three-dimensional scanning result of the three-dimensional scanner 10. In the three-dimensional model M, for example, RGB data and three-dimensional shape data are associated with each other. That is, the three-dimensional shape data indicates the surface shape of the object O. The RGB data includes information such as the color, pattern, and shading of the surface of the object O. In the three-dimensional model M, the gradation of the RGB data is associated with the three-dimensional coordinates of the surface shape. In this manner, the three-dimensional model generating unit 22 generates the three-dimensional model M of the object O.

[0023] The three-dimensional model is data that indicates the three-dimensional shape and color gradation of the object O. The three-dimensional model M is data that indicates a virtual model of the object O. The three-dimensional model M is generated from scan data of the real domain. The three-dimensional model M includes three-dimensional coordinates of each point corresponding to the surface of the object O. In other words, the three-dimensional model M includes data of the three-dimensional coordinates of multiple points. Furthermore, RGB data is attached to the three-dimensional coordinates of each point.

[0024] The transformation processing unit 23 performs a transformation process on the three-dimensional model M to generate a plurality of synthetic image data S. Here, the transformation process refers to performing an arithmetic process corresponding to twisting, bending, and the like on the three-dimensional model M of the flexible object O. The transformation processing unit 23 can transform the virtual three-dimensional model of the flexible object O into various shapes. Therefore, it is possible to generate synthetic image data S according to the number of deformations. In this way, it is possible to generate a large amount of data involving deformation of the flexible object O. As shown in FIG. 2, a model obtained by performing a transformation process on the three-dimensional model M is called a deformed model M1. In other words, the deformed model M1 is a model generated by deforming the three-dimensional model M in a virtual domain.

[0025] Deformation processing is the process of deforming an object O by bending or twisting a 3D model in a virtual domain. Since it is processing in a virtual domain, the amount of bending, bending direction, amount of twisting, and twisting direction can be arbitrarily set. change The parameters of the deformation process include, for example, a bending amount, a bending direction, a twist amount, and a twist direction. For example, by changing at least one of the bending amount, the bending direction, the twist amount, and the twist direction, the object O can be deformed into various shapes.

[0026] The transformation processing unit 23 transforms the object O in a virtual domain. The transformation processing unit 23 can generate a rendering image of the deformed object O as composite image data S. Therefore, the transformation processing unit 23 can generate composite image data of the object O transformed into various shapes, as shown in FIG.

[0027] The transformation processing unit 23 performs N transformation processes by changing the parameters of the transformation processes, where N is an integer equal to or greater than 2. The transformation processing unit 23 then generates synthetic image data for each of transformations 1, 2, ..., N. In this way, a plurality of synthetic image data S can be generated. Each of the plurality of synthetic image data corresponds to data of the object O in a state where the shape of the object O is different.

[0028] The transformation processing unit 23 generates one or more synthetic image data S for each transformation model. For example, multiple synthetic image data can be generated for one transformation model M1 by changing the imaging direction. The synthetic image data S may include depth image and RGB image data, similar to a three-dimensional model, that is, three-dimensional scan data. The synthetic image data S may be only data that can be obtained at the time of recognition. For example, the synthetic image data S may be RGB data.

[0029] The transformation processing unit 23 can virtually transform the object O based on a three-dimensional model. Since the transformation processing unit 23 transforms the object O in a virtual domain, it can easily generate a large amount of synthetic image data. In other words, it is possible to generate multiple pieces of synthetic image data without having to transform the object O in the real domain by a human or a robot. In other words, it is possible to generate a large number of transformed models by changing the parameters of the transformation processing.

[0030] For the three-dimensional transformation process, CG (computer graphics) software such as Blender can be used. An example of software for performing the transformation process of the transformation processing unit 23 will be shown below. https: / / docs.blender.org / manual / en / latest / modeling / modifiers / deform / simple_deform.html https: / / www.youtube.com / watch?v=AyuSN4XSNEA https: / / blender.community / c / rightclickselect / c1cbbc / https: / / blender.stackexchange.com / questions / 168909 / code-behind-the-simple-deform-modifier

[0031] An example of a bending or twisting algorithm is given below. https: / / smartech.gatech.edu / bitstream / handle / 1853 / 63 / 04-15.pdf

[0032] Each point that composes a 3D model of an object has 3D xyz information. The XYZ coordinates of each point before deformation are (x, y, z), and the XYZ coordinates of each point before deformation are (x', y', z'). If the XYZ coordinates of each point are changed using the following equations (1) to (3), the 3D model will change to a curved shape.

[0033] x'=x+α x sin(β xy) ···(1) y' = y (2) z' = z (3)

[0034] The above formulas (1) to (3) show an example of bending along the y-axis in the x-axis direction using a sine wave, where α is a parameter that controls the bending and β is a parameter that controls the amplitude.

[0035] The learning device 24 trains the recognizer using the three-dimensional model M and the multiple synthetic image data S as learning data. That is, the learning device 24 learns the recognizer by performing machine learning using the learning data. The learning device 24 generates the recognizer by, for example, deep learning.

[0036] In this way, the recognizer can be trained using a sufficient number of synthetic image data S. That is, since the transformation processing unit 23 transforms the three-dimensional model M in a virtual domain, the synthetic image data S is automatically generated. This allows a large number of synthetic image data S to be generated easily. A recognizer with high recognition accuracy can be generated. A sufficient number of learning data can be automatically generated for machine learning with high recognition accuracy. Furthermore, since the three-dimensional model M in the real domain is used, it is possible to obtain higher accuracy.

[0037] Furthermore, the learning device 24 trains the recognizer based on the correspondence relationship when the transformation processing unit 23 performs the transformation processing. Here, the correspondence relationship means that when the transformation processing is performed, the coordinate information before and after the transformation of the 3D model is associated with each other. In other words, each point of the 3D model is associated with each point after the transformation processing. This makes it possible to generate a recognizer with high recognition accuracy.

[0038] The correspondence is specified by the amount of change in position and posture, the amount of bending, the amount of twisting, the bending direction, the twisting direction, etc., during the transformation process. The correspondence when the transformation process is performed will be explained with reference to Fig. 4. In 3D model M, an arbitrary location of object O is designated as location P. In Fig. 4, location P is a corner of a packaging bag. In Fig. 4, the transformed models obtained by transforming 3D model M into different shapes are designated as transformed models M1, M2, and M3.

[0039] Since the transformation processing unit 23 performs displacement processing based on the 3D scan data, it is possible to maintain the correspondence of the point P between transformation processes. In other words, the transformation processing unit 23 can identify the position coordinates of the point P after transformation from the position coordinates of the point P before transformation. The transformation processing unit 23 then generates the synthetic image data S by adding information on the position coordinates of the point P. Therefore, in the synthetic image data S of the deformation model M1, the pixel address of the point P, etc. are identified. The learning device 24 can generate a more accurate recognizer.

[0040] Furthermore, when generating the composite image data S, the processing device 20 may change environmental conditions such as lighting. For example, the processing device 20 generates composite image data S while changing the intensity of the illumination light. For example, for one deformed model M1, the processing device 20 generates composite image data when the illumination light intensity is high and composite image data when the illumination light intensity is low. Furthermore, the processing device 20 may generate the composite image data S by changing the light source of the illumination light (sunlight, fluorescent light, etc.) or the illumination direction of the illumination light.

[0041] Furthermore, the processing device 20 may include information such as the material of the object O in the three-dimensional model M or the deformed model M1. The processing device 20 generates a rendering image according to the material and surface roughness of the object as synthetic image data S. Optical characteristics such as the reflection characteristics and scattering characteristics of the illumination light can be determined according to the material and surface roughness of the object.

[0042] In this manner, the processing device 20 sets material information such as the material of the object O. The processing device 20 generates synthetic image data S by rendering the three-dimensional model M while changing environmental conditions such as the illumination light. This makes it possible to generate appropriate synthetic image data S. In other words, by setting the optical characteristics of the object O, it is possible to generate synthetic image data when parameters such as the illumination light environmental conditions are changed. This makes it possible to further improve the accuracy of the recognizer.

[0043] Hereinafter, a method for generating a recognizer according to the present embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the method for generating a recognizer. Here, the method for generating a recognizer includes a process for generating learning data and a process for performing machine learning using the learning data.

[0044] First, the 3D scanner 10 3D scans a flexible object O (S101). For example, while the shape of the object O is constant, the 3D scanner 10 3D scans the entire circumference of the object O. The 3D model generator 22 generates a 3D model M of the object O from the 3D scan results (S102).

[0045] The transformation processing unit 23 generates synthetic image data by transformation processing (S103). For example, the transformation processing unit 23 performs transformation processing on a three-dimensional model to generate a plurality of synthetic image data. The learning device 24 trains a recognizer using learning data (S104). For example, the learning device 24 uses the three-dimensional model M and the synthetic image data S as learning data to machine-train the recognizer. In this way, a highly accurate recognizer can be generated.

[0046] (Robot Arm) Next, a robot arm 200 equipped with a recognizer will be described with reference to Fig. 6. The robot arm 200 includes an arm mechanism 201, an end effector 202, a control unit 203, and a camera 204.

[0047] The end effector 202 holds a flexible object O. For example, the end effector 202 is a robot hand that grasps the object O. Alternatively, the end effector 202 may have a suction mechanism that holds the object O by suction.

[0048] The arm mechanism 201 is provided with a plurality of joint mechanisms each having a motor and the like. An end effector 202 is attached to the tip of the arm mechanism 201. The arm mechanism 201 changes the position or posture of the end effector 202 by driving each joint. By moving the end effector 202, the arm mechanism 201 can approach the object O from various directions.

[0049] The camera 204 captures an image of an object O. The camera 204 may be a color camera that captures a color image, or may be a depth camera that captures a depth image. Of course, the camera 204 may be both a color camera and a depth camera.

[0050] The control unit 203 controls the arm mechanism 201 and the end effector 202. The control unit 203 is, for example, an information processing device having a processor, a memory, etc. For example, the control unit 203 stores a control program for controlling the arm mechanism 201 and the end effector 202.

[0051] The control unit 203 is equipped with the above-mentioned recognizer. The recognizer of the control unit 203 recognizes the object O based on the image pickup result by the camera 204. This allows the control unit 203 to recognize the position and orientation of the object O. The control unit 203 controls the arm mechanism 201 so that the end effector 202 moves to the holding position specified by the recognizer.

[0052] For example, a part of the object O that is easy to hold is specified in advance as a holding position. For example, a corner or a flat surface of the object O in a three-dimensional model M may be specified by the user as a holding position. Then, the recognizer recognizes the holding position based on the image pickup result by the camera 204. Then, the arm mechanism 201 operates so that the end effector 202 is directed to the holding position specified by the recognizer. In this manner, the robot arm 200 can reliably hold the flexible object O.

[0053] The method for generating a recognizer in this embodiment includes the steps of: using a 3D scanner to 3D scan the entire circumference of a flexible object while the shape of the flexible object is constant; generating a 3D model of the flexible object from the 3D scan results; performing a deformation process on the 3D model to generate multiple composite image data; and training a recognizer using the 3D model and the composite image data as learning data.

[0054] When transformation processing is performed using the above generation method of A recognizer may be trained based on the correspondences. In the above generating method, the 3D scan may be performed while the flexible object is held by a wire having a line width equal to or less than the detection resolution of a depth sensor provided in the 3D scanner.

[0055] The learning system of this embodiment includes a 3D scanner that 3D scans the entire circumference of a flexible object while the shape of the flexible object is constant, a 3D model generation unit that creates a 3D model of the flexible object from the 3D scan results of the 3D scanner, a transformation processing unit that performs a transformation process on the 3D model to generate multiple pieces of composite image data, and a learning device that uses the 3D model and the composite image data as learning data to train a recognizer.

[0056] The above-mentioned method of generating a recognizer can be implemented by a computer program or hardware. The processing device 20 and the control unit 203 are provided with a memory for storing a program, a processor for executing the program, and the like. The processing device 20 etc. executes the program, thereby enabling the method according to this embodiment to be executed. The processing device 20, the control unit 203, and the recognizer may be distributed and arranged in a plurality of devices.

[0057] A part or all of the above processes may be executed by a computer program. That is, the control of the processing device 20 is executed by a control computer that constitutes the processing device 20 executing the program. The above-mentioned program includes a group of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or the tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0058] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention. [Explanation of symbols]

[0059] 1. Learning System 10 3D Scanner 11 Sensor unit 12 Arm mechanism 111 Camera 112 Depth Sensor 15 Wire 20 Processing equipment 21 Sensor data acquisition unit 22 3D model generation section 23 Transformation Processing Section 24 Learning Unit 200 Robot Arm 201 Arm mechanism 202 End Effector 203 Control Unit

Claims

1. A method for generating a recognizer that recognizes a position and orientation of a flexible object based on an image of the flexible object captured by a camera, and identifies a holding position, comprising: A step of three-dimensionally scanning the entire circumference of the flexible object with a three-dimensional scanner while the shape of the flexible object is constant; generating a three-dimensional model of the flexible object from the three-dimensional scan results; performing a transformation process on the three-dimensional model to generate a plurality of synthetic image data; training a recognizer using the three-dimensional model and the synthetic image data as training data; the three-dimensional model of the flexible object includes three-dimensional shape data indicating a surface shape; the three-dimensional model includes data of three-dimensional coordinates of a plurality of points corresponding to a surface of the flexible object; In the step of generating the composite image data, performing a deformation process of bending or twisting the three-dimensional model to generate a deformed model; generating a plurality of deformation models having different bending directions, bending amounts, twist directions, and twist amounts, thereby generating the plurality of synthetic image data; the three-dimensional coordinates of the plurality of points vary depending on the bending direction, bending amount, twist direction, and twist amount; a three-dimensional coordinate of each point of the three-dimensional model is associated with a three-dimensional coordinate of each point of the deformed model based on the bending direction, bending amount, twist direction, and twist amount; the plurality of points of the deformed model are identified by pixel addresses of the composite image data; In the step of training the recognizer, A method for generating a recognizer that uses the synthetic image data as input to recognize the position and orientation of the deformed model and identify the holding position by training a recognizer based on the correspondence between the 3D model and the deformed model when the deformation process is performed.

2. A step of capturing an image of the flexible object with a color camera to obtain a color image; 2. The method of generating a recognizer according to claim 1, wherein the three-dimensional coordinates are accompanied by data indicating a color gradation obtained from the color image.

3. A method for generating a recognizer as described in claim 2, which generates multiple synthetic image data with different orientations for a single deformation model.

4. The flexible object is suspended by a wire having a width equal to or less than the detection resolution of the 3D scanner, and the 3D scanner performs a 3D scan, 4. The method for generating a recognizer according to claim 1, further comprising sensing the underside surface of the flexible object by moving the three-dimensional scanner to the underside of the flexible object.

5. A method for generating a recognizer described in any one of claims 1 to 3, in which the three-dimensional scanner performs a three-dimensional scan while the flexible object is placed on a mesh formed of wires having a width equal to or less than the detection resolution of the three-dimensional scanner.

6. A method for holding a target object, which is a flexible object, with an end effector, comprising: The target object is captured by a camera. A recognizer generated by the generation method according to any one of claims 1 to 5 recognizes the position and orientation of the target object based on the image capture result of the camera, and specifies a holding position of the target object; A holding method for driving the end effector so that the end effector holds the target object at the holding position.

Images