Method, system, and storage medium for generating training data

Abstract

A method for generating training data, a system, and a storage medium, the method being for learning of a machine learning model, the method including: a step of performing pre-learning of the machine learning model using simulation data of an object; a step of capturing a first image of the object from a first photographing direction; a step of identifying a first pose of the object from the first image using the machine learning model whose pre-learning is completed; a step of making a correct / incorrect determination of the first pose; a step of, in a case where the first pose is determined to be correct, capturing a second image of the object from a second photographing direction different from the first photographing direction, calculating a second pose by performing a transformation on the first pose corresponding to a change from the first photographing direction to the second photographing direction, and generating training data by assigning the second pose to the second image; and a step of changing a real pose of the object and repeating the steps b to e.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a method, a system, and a computer program that generate training data for learning of a machine learning model. BACKGROUND

[0002] A technique is disclosed in Patent Literature 1 that recognizes a position and a pose of an object by matching an object feature calculated from three-dimensional model data with an image obtained by photographing the object. In addition, a technique is disclosed in Patent Literature 2 that uses a machine learning model to recognize a position and a pose of an object. In general, in a case where a machine learning model is applied to recognition of a position and a pose of an object, training data labeled with a position and a pose is required, and preparation of generating the training data requires a large amount of work. Therefore, in Patent Literature 2, a method of generating training data by simulation is used.

[0003] Patent Literature 1: Japanese Patent Application Publication No. 2019-185239

[0004] Patent Literature 2: Japanese Patent Application Publication No. 2020-87310

[0005] When training data is generated by simulation, an image of an object is generated for a certain scene, and a label of a position and a pose is given to the object in the scene. However, there is a difference between an image obtained in simulation and an actual environment, and there is a problem that recognition accuracy of a position and a pose in an actual environment tends to decrease in a machine learning model learned by training data generated by simulation. SUMMARY

[0006] According to a first aspect of the present disclosure, there is provided a method of generating training data for learning of a machine learning model that recognizes a pose of an object, the method including: a process of performing pre-learning of the machine learning model using simulation data of the object; a process of photographing a first image of the object from a first photographing direction using a camera; a process of recognizing a first pose of the object from the first image using the machine learning model on which the pre-learning is completed; a process of making a correct / incorrect determination on the first pose; a process of, in a case where the first pose is determined to be correct, photographing a second image of the object from a second photographing direction different from the first photographing direction using the camera, calculating a second pose by performing a transformation on the first pose corresponding to a change from the first photographing direction to the second photographing direction, and generating training data by giving the second pose to the second image; and a process of changing a real pose of the object and repeating the processes from the process of photographing the first image to the process of generating the training data.

[0007] According to a second aspect of the present disclosure, there is provided a system for generating training data for learning of a machine learning model that identifies a pose of an object, the system including: a camera that captures the object; and a training data generation unit that generates the training data, the training data generation unit performing: an a-process that performs pre-learning of the machine learning model using simulation data of the object; a b-process that captures a first image of the object from a first capturing direction using the camera; a c-process that identifies a first pose of the object from the first image using the machine learning model for which the pre-learning is completed; a d-process that determines whether the first pose is correct or incorrect; an e-process that captures a second image of the object from a second capturing direction different from the first capturing direction using the camera, calculates a second pose by performing a transformation of the first pose corresponding to a change from the first capturing direction to the second capturing direction, and generates training data by assigning the second pose to the second image; and an f-process that changes a real pose of the object and repeats the b-process to the e-process.

[0008] According to a third aspect of the present disclosure, there is provided a storage medium storing a computer program that causes a processor to perform a process of generating training data for learning of a machine learning model that identifies a pose of an object, the computer program causing the processor to perform: an a-process that performs pre-learning of the machine learning model using simulation data of the object; a b-process that captures a first image of the object from a first capturing direction using a camera; a c-process that identifies a first pose of the object from the first image using the machine learning model for which the pre-learning is completed; a d-process that determines whether the first pose is correct or incorrect; an e-process that captures a second image of the object from a second capturing direction different from the first capturing direction using the camera, calculates a second pose by performing a transformation of the first pose corresponding to a change from the first capturing direction to the second capturing direction, and generates training data by assigning the second pose to the second image; and an f-process that changes a real pose of the object and repeats the b-process to the e-process. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 is an explanatory diagram indicating a configuration of a robot system in an embodiment.

[0010] Figure 2 is a functional block diagram of an information processing apparatus in an embodiment.

[0011] Figure 3 is an explanatory diagram indicating a pose identification function of an object of a machine learning model.

[0012] Figure 4is a flowchart showing the overall steps of the processing in the first embodiment.

[0013] Figure 5 is an explanatory diagram of a generation process of training data for pre-learning performed by simulation.

[0014] Figure 6 is a flowchart showing detailed steps of the step S130 of the first embodiment.

[0015] Figure 7 is an explanatory diagram showing a state of a change of a photographing direction in the first embodiment.

[0016] Figure 8 is a flowchart showing the overall steps of the processing in the second embodiment.

[0017] Figure 9 is a flowchart showing detailed steps of the step S125 of the second embodiment.

[0018] Figure 10 is an explanatory diagram showing a method of determining a second photographing direction using statistical data.

[0019] Figure 11 is a flowchart showing detailed steps of the step S135 of the second embodiment.

[0020] Explanation of Reference Numerals

[0021] 100 … robot; 110 … base; 120 … arm; 122 … arm tip; 150 … hand; 180 … current sensor; 190 … joint encoder; 200 … control device; 300 … information processing device; 310 … processor; 311 … training data generation section; 312 … simulation execution section; 314 … learning execution section; 316 … object recognition section; 318 … right / wrong determination section; 320 … memory; 330 … interface circuit; 340 … input device; 350 … display device; 400 … camera; 410 … first camera; 420 … second camera; 430 … illumination section; 500 … stand; 510 … first tray; 520 … second tray. DETAILED DESCRIPTION

[0022] A. First Embodiment:

[0023] Figure 1 is an explanatory diagram showing an example of a robot system in one embodiment. The robot system is provided with a robot 100, a control device 200 that controls the robot 100, an information processing device 300, a camera 400, and a stand 500. The information processing device 300 is, for example, a personal computer. In the robot system, the robot 100 is controlled by the control device 200 based on a result of processing performed by the information processing device 300. Figure 1In the figure, three axes X, Y, Z of an orthogonal coordinate system that defines a three-dimensional space are drawn. The X axis and the Y axis are axes in the horizontal direction, and the Z axis is an axis in the vertical direction. These X, Y, Z axes are coordinate axes of a robot coordinate system Σr with a predetermined position of the robot 100 as the origin.

[0024] The robot 100 is provided with a base 110 and a robot arm 120. A hand 150 that is an end effector is installed at a front end of the robot arm 120, that is, an arm tip 122. The hand 150 can be implemented as a gripper that can hold an object OB or a suction pad. A TCP (Tool Center Point) that is a control point of the robot 100 is provided at a front end of the hand 150. Note that the control point TCP can be provided at an arbitrary position.

[0025] The robot arm 120 is connected in series through six joints J1 to J6. Among these joints J1 to J6, three joints J2, J3, J5 are bend joints, and the other three joints J1, J4, J6 are twist joints. In the present embodiment, a six-axis robot is exemplified, but a robot including an arbitrary robot mechanism having one or more joints can be used. In addition, the robot 100 of the present embodiment is a vertical multi-joint robot, but a horizontal multi-joint robot can be used.

[0026] The rack 500 is provided with a first tray 510 and a second tray 520. A plurality of objects OB are bulk-packed on the first tray 510. The second tray 520 serves as a place where the objects OB taken out from the first tray 510 are placed. The robot 100 performs a work of taking out the objects OB from the first tray 510 and placing them on the second tray 520.

[0027] A camera 400 that captures an image of the objects OB in the first tray 510 is provided at the front end of the robot arm 120. The image captured by the camera 400 is used to find a three-dimensional position and an attitude of the objects OB. Hereinafter, the three-dimensional position and the attitude are referred to as "pose". As the camera 400, for example, an RGBD camera, a stereo camera can be used. The RGBD camera is a camera having an RGB camera that captures an RGB image and a D camera that captures a depth image. Instead of the RGB camera, a monochrome camera that captures a grayscale image can be used. In addition, a monocular camera can be used as the camera 400. Note that the camera 400 can be provided separately from the robot 100 without being provided at the robot arm 120. However, if the camera 400 is provided at the robot arm 120, it has an advantage that it is easy to change a direction in which the camera 400 captures the objects OB.

[0028] Figure 2is a block diagram showing the functions of the information processing apparatus 300. The information processing apparatus 300 has a processor 310, a memory 320, an interface circuit 330, an input device 340 connected to the interface circuit 330, and a display device 350. The control apparatus 200 and the camera 400 are also connected to the interface circuit 330. The measurement results of the current sensor 180 and the joint encoder 190 of the robot 100 are supplied to the information processing apparatus 300 via the control apparatus 200. The current sensor 180 is a sensor that measures the current of the motor provided at each joint of the robot 100. The joint encoder 190 is a sensor that detects the action position of each joint.

[0029] In the present embodiment, the camera 400 has a first camera 410 that captures a two-dimensional image such as an RGB image or a grayscale image, a second camera 420 that captures a depth image, and an illumination section 430 that irradiates the second camera 420 with illumination light. The illumination section 430 is a projector that irradiates an infrared pattern used for capturing a depth image.

[0030] The processor 310 has a function as a training data generation section 311 that generates training data for learning a machine learning model that identifies the pose of the object OB. The training data generation section 311 includes the functions of a simulation execution section 312, a learning execution section 314, an object recognition section 316, and a right or wrong determination section 318. The simulation execution section 312 performs processing of generating training data for pre-learning of a machine learning model by simulating a scenario in which the object OB is present on the first tray 510. The learning execution section 314 performs pre-learning and formal learning of the machine learning model. The object recognition section 316 performs processing of identifying the pose of the object OB from an image captured by the camera 400 using the machine learning model that has completed learning. The right or wrong determination section 318 performs determination of whether the pose of the object OB identified using the machine learning model is correct or incorrect. The functions of the training data generation section 311 are implemented by the processor 310 executing a computer program stored in the memory 320. However, part or all of the functions of the training data generation section 311 can also be implemented by a hardware circuit.

[0031] In the memory 320, a machine learning model MM that identifies the pose of the object OB, simulation data SD, training data PTD for pre-learning, training data RTD for formal learning, and a robot control program RP are stored. The simulation data SD is data for simulating a scenario in which the object OB is bulked on the first tray 510, and includes CAD data that is three-dimensional model data of the object OB, robot data that indicates the position and shape of the robot 100, and camera parameters of the camera 400. The robot control program RP is constituted by a plurality of commands that cause the robot 100 to act.

[0032] Figure 3is an explanatory diagram showing a pose recognition function of an object OB indicating a machine learning model MM. The camera 400 has been pre-calibrated, and the relative relationship between the camera coordinate system Σc and the robot coordinate system Σr is known. The camera 400 generates a two-dimensional image Ml and a depth image M2 by photographing the object OB. Hereinafter, these images Ml, M2 will also be collectively referred to as "images IM". The machine learning model MM is a regression model that outputs the pose PA of the object OB in accordance with the input of the images IM. The pose PA is expressed by a position (x, y, z) and an attitude (w, p, r) in the robot coordinate system Σr, for example. The attitude is expressed by the rotation angles (w, p, r) around the three axes.

[0033] As to which of the plurality of objects in the bulk state is taken as a target to recognize the pose PA, the following three ways can be considered.

[0034] (A1) Only the object located at the uppermost position is taken as a target.

[0035] (A2) Not only the object located at the uppermost position but also the object partially overlapping with other objects is taken as a target. Specifically, for example, an object having a hidden area ratio of 20% or less is taken as a target.

[0036] (A3) All objects are taken as targets.

[0037] In consideration of actual bulk sorting work, it is preferable to generate training data in a manner that the above (A2) object can be taken as a target to recognize the pose. In this case, with respect to the bulk state generated in the simulation, an image to which the poses of a plurality of objects satisfying the above (A2) condition are attached as labels is generated as a training data group. However, hereinafter, for the sake of convenience of explanation, it is assumed that the machine learning model MM recognizes the pose PA of only one object OB in accordance with the above (A1).

[0038] Figure 4 is a flowchart showing the overall steps of the processing in the first embodiment. In step S110, the simulation execution unit 312 generates training data for pre-learning by simulation.

[0039] Figure 5is a diagram illustrating a generation process of training data for pre-learning by simulation. The simulation execution section 312 inputs simulation data SD including CAD data of an object OB to be recognized and various parameters to the scene simulator SS. The scene simulator SS generates a scene in a bulk state by physically simulating, for example, a plurality of objects OB represented by CAD data falling randomly from a certain height. Since it is simulation, the poses of the objects OB in bulk are known. The simulation execution section 312 renders the scene based on camera information set with various parameters and generates an image IM including a two-dimensional image M1 and a depth image M2. The simulation execution section 312 generates training data PTD for pre-learning by assigning the poses of the objects OB as labels to the image IM. By generating a plurality of scenes in bulk, a large amount of training data PTD can be generated. In the present embodiment, the simulation of the bulk state of the objects OB is performed, but a state other than the bulk state, for example, a state in which one object OB is placed on the tray 510 in an arbitrary pose, can be simulated.

[0040] In step S120, the learning execution section 314 performs pre-learning of the machine learning model MM using the training data PTD for pre-learning.

[0041] In step S130, the training data generation section 311 generates training data RTD for formal learning taking into account specular reflection light of the object OB. The reason for taking into account the specular reflection light of the object OB is as follows. That is, in the simulation performed in step S110, it is difficult to accurately reproduce an image IM captured in a state in which the specular reflection light of the object OB is incident on the camera 400 in an actual environment. Therefore, in the machine learning model MM learned using the training data for pre-learning, there is a tendency that the recognition accuracy of the pose of the object OB is reduced in a state in which the specular reflection light is incident on the camera 400 in an actual environment. In particular, in the case where the object OB is a glossy object, since it is difficult to accurately reflect the reflection condition of light in the simulation, there is a tendency that the recognition accuracy is greatly reduced. Therefore, in step S130, the training data for formal learning is generated for a state in which the camera 400 receives the specular reflection light of the object OB.

[0042] Figure 6 is a flowchart illustrating detailed steps of step S130. In step S310, the training data generation section 311 captures a first image of the object OB in bulk using the camera 400. The "first image" includes a two-dimensional image M1 and a depth image M2 as illustrated in FIG. 6. The direction of the camera 400 at this time is referred to as a "first capturing direction". Figure 3

[0043] ​In step S320, the object recognition section 316 recognizes the first pose of the object OB from the first image using the machine learning model MM for which the pre-learning is completed. In step S330, the correct / incorrect determination section 318 determines the correctness / incorrectness of the recognition result. This correct / incorrect determination can be performed using the reliability score, for example. The reliability score can be calculated using the recognized first pose of the object OB, by generating a simulation image of the object OB in the first pose by simulation, and as an index indicating how much the simulation image coincides with the first image captured by the camera 400. The reliability score RS is calculated by the following expression, for example.

[0044] RS = a x S1 + (1 - a) x S2 (1)

[0045] Here, a is a coefficient of 0 ≤ a ≤ 1, S1 is a similarity between a two-dimensional image included in the simulation image and a two-dimensional image included in the first image, and S2 is a similarity between a depth image included in the simulation image and a depth image included in the first image. The similarities S1, S2 can be calculated as image similarity, cosine similarity. According to the above expression (1), the reliability score RS can be calculated by weighted addition of the similarities S1, S2.

[0046] The correct / incorrect determination section 318 determines the correctness / incorrectness of the recognition result by comparing the reliability score RS with a reliability threshold value set in advance. That is, in a case where the reliability score RS is equal to or higher than the reliability threshold value, it is determined that the recognition result is correct, and in a case where it is lower than the reliability threshold value, it is determined that the recognition result is incorrect. As another method, for example, the outlines of the object OB in the simulation image and the first image can be superimposed and displayed on the display device 350, and the user can determine the correctness / incorrectness. In a case where the recognition result is correct, it proceeds to step S340 described later. On the other hand, in a case where the recognition result is incorrect, it proceeds to step S390, changes the real pose of the object OB, and returns to step S310. The change of the pose of the object OB is performed by, for example, the user re-dispensing the object OB.

[0047] In step S340, the training data generation section 311 determines the specular reflection direction of the illumination light as the second photographing direction, and in step S350, a second image is generated by capturing the object OB from the second photographing direction using the camera 400.

[0048] Figure 7 is an explanatory diagram indicating a state of a change in the photographing direction in the first embodiment. Figure 7The image above shows the state in step S310 where the object OB is being photographed in the first shooting direction. In this state, diffuse light from the illumination light emitted from the illumination unit 430 to the second camera 420 is incident on the second camera 420. The depth image captured by the second camera 420 is an image obtained from this diffuse light. In step S340, as... Figure 7 As shown below, the pose of camera 400 is changed so that specular reflection of the illumination light from illumination unit 430 is incident on second camera 420. Since the emission direction of the illumination light from illumination unit 430 is known, the direction of specular reflection of the object OB can be calculated if the pose of object OB is known. Furthermore, the pose change of camera 400 can be performed by moving robotic arm 120. As described above, in simulation, it is difficult to accurately reproduce images captured when specular reflection of object OB is incident on camera 400 in a real environment, especially when object OB is a glossy object. Therefore, in step S340, a second shooting direction is determined such that specular reflection of object OB is incident on camera 400 in a real environment. If training data is generated using images captured from this second shooting direction, training data that is difficult to generate through simulation can be obtained.

[0049] exist Figure 7 In the example, the state of the specular reflected light from object OB incident on the second camera 420 for the depth image is determined as the second shooting direction. However, alternatively, the state of the specular reflected light from object OB incident on the first camera 410 for the two-dimensional image can also be determined as the second shooting direction. If the position of the illumination source for the two-dimensional image is known, the latter can be performed. Alternatively, the states of the specular reflected light from object OB incident on both the first camera 410 and the second camera 420 can be set as the second shooting directions for shooting.

[0050] In step S360, the training data generation unit 311 calculates the second pose of object OB in the second image. The second pose can be calculated according to the following formula.

[0051] Pb = B H A ×Pa (2)

[0052] Here, Pb is a 4×4 matrix representing the second pose, and Pa is a 4×4 matrix representing the first pose. B H A This is the homogeneous transformation matrix representing the transformation from the first pose Pa to the second pose Pb. B H AThe same as the matrix representing the transformation from the first photographing direction Da of the camera 400 to the second photographing direction Db. In other words, the second pose Pb is calculated by performing a transformation on the first pose Pa corresponding to the change from the first photographing direction Da to the second photographing direction Db.

[0053] In step S370, the training data generation section 311 generates an image with a pose as training data for formal learning. That is, the training data for formal learning is generated by assigning the second pose as a label to the second image photographed in step S350. At this time, the training data for formal learning can also be added by assigning the first pose as a label to the first image photographed in step S310.

[0054] In step S380, the training data generation section 311 determines whether a predetermined number of training data for formal learning has been obtained. In the case where a sufficient number of training data has not been obtained, the process proceeds to step S390, the real pose of the object OB is changed, and the above-described steps S310 to S380 are executed again. On the other hand, in the case where a sufficient number of training data has been obtained, the process of the present embodiment ends. Figure 6

[0055] Figure 6 After the process of step S130 shown in FIG. 13 ends, in step S140 shown in FIG. 14, the learning execution section 314 executes formal learning of the machine learning model MM using the training data for formal learning generated in step S130. At the time of formal learning, not only the training data generated in step S130 but also the training data for pre-learning generated in step S110 can be used. In step S140, the robot 100 performs a job using the machine learning model MM on which the formal learning has been completed. The job is executed in accordance with the robot control program RP generated in advance. Figure 4

[0056] As described above, in the first embodiment, the first pose of the object OB is recognized from the first image using the machine learning model MM on which the pre-learning has been completed, in the case where the first pose is determined to be correct, a second image is photographed from a second photographing direction different from the first photographing direction using a camera, and the training data for formal learning is generated by assigning a second pose of the object OB to the second image. As a result, it is possible to easily generate training data for machine learning in an actual environment, and since it is possible to fill in the difference between simulation and actual environment, it is possible to improve the performance of the machine learning model MM. In addition, in the first embodiment, since the specular reflection direction is set as the second photographing direction, it is possible to generate training data for correctly recognizing the pose of a glossy object.

[0057] B. Second Embodiment: ​​

[0058] Figure 8 This is a flowchart illustrating the overall steps of the processing in the second embodiment. Figure 4 The only difference in the first embodiment shown is the addition of step S125 and the replacement of step S130 with step S135; all other steps are the same as in the first embodiment. Furthermore, the device structure of the second embodiment is the same as that of the first embodiment.

[0059] In step S125, the training data generation unit 311 generates statistical data using the pre-learned machine learning model MM. This statistical data is used to determine the second shooting direction.

[0060] Figure 9 This is a flowchart illustrating the detailed steps of step S125. In step S210, the training data generation unit 311 uses the camera 400 to capture an image of the bulk object OB. This image includes... Figure 3 The two-dimensional image M1 and depth image M2 are shown. In step S220, the object recognition unit 316 uses the pre-learned machine learning model MM to recognize the pose of object OB from the image obtained in step S210. In step S230, the accuracy determination unit 318 determines the accuracy of the recognition result. (The last sentence appears to be incomplete and possibly refers to a different context.) Figure 6 The same as step S330 is used to perform the correctness determination using a reliability score. If the identification result is correct, proceed to step S240 (described later). On the other hand, if the identification result is incorrect, proceed to step S260, change the actual pose of the object OB, and return to step S210. The change of the pose of the object OB is performed, for example, by the user reassembling the object OB.

[0061] In step S240, the training data generation unit 311 generates an image with pose as statistical data. That is, statistical data is generated by assigning the pose identified in step S220 to the image captured in step S210. In step S250, the training data generation unit 311 determines whether a predetermined number of statistical data has been obtained. If a sufficient number of statistical data has not been obtained, the process proceeds to step S260, changes the actual pose of object OB, and returns to step S210, repeating steps S210 to S250. On the other hand, if a sufficient number of statistical data has been obtained, the process proceeds to step S270.

[0062] In step S270, the training data generation unit 311 calculates the insufficient pose with reference to the statistical data and registers it in the statistical data.

[0063] Figure 10is an explanatory diagram showing insufficient poses in the statistical data. The "insufficient pose" refers to a pose that is not included in the statistical data generated in the processing of steps S210 to S260. In Figure 10 above, only the rotation angles (w, p) among the poses (x, y, z, w, p, r) included in the statistical data are shown in two dimensions. Ideally, the poses are roughly evenly included in the statistical data, but in Figure 9 , the poses determined to be incorrect in step S230 are not included in the statistical data. In Figure 10 above, the black dots represent the poses determined to be correct and included in the statistical data. In the incorrect solution area, the poses determined to be correct are missing. The training data generation section 311 identifies one or more poses as "insufficient poses" in the incorrect solution area and registers them in the statistical data. In Figure 10 below, a state in which the insufficient poses (Desired position and attitude) are added with white circles is shown. Note that the "insufficient pose" can be found, for example, by analyzing the results of identifying a certain region when a full spherical object OB is divided into a certain region and the poses included in the statistical data are mapped. The insufficient pose is a pose that cannot be accurately identified in the machine learning model MM that has been pre-learned through simulation, and can be considered to be a pose for which the difference between the simulation and the reality is large. Therefore, in the incorrect solution area in which the poses determined to be correct in the statistical data are missing, one or more poses are registered as "insufficient poses", and appropriate poses for bridging the difference between the simulation and the reality can be effectively collected.

[0064] Thus, when the processing of step S125 ends, the processing proceeds to step S135 of Figure 8 , in which the training data generation section 311 generates training data for formal learning using the statistical data.

[0065] Figure 11 is a flowchart showing the detailed steps of step S135. Figure 11 The step is a step in which step S340 is replaced with step S345 in the detailed steps of step S130 shown in Figure 6 in the first embodiment. Figure 6

[0066] In step S345, the training data generation section 311 determines a second photographing direction using the statistical data. This second photographing direction is a direction in which the object OB is photographed in a manner different from the first photographing direction in Figure 10 ​The second photographing direction is determined in step S345 of the second embodiment is a direction in which the pose recognized by the pre-learned machine learning model MM is not determined to be correct. In this way, if the second photographing direction is set to a direction that is not determined to be correct in the pre-learned machine learning model MM by simulation, it is possible to generate training data that fills in the difference between simulation and the actual environment.

[0067] The second photographing direction determined in step S345 of the second embodiment is a direction in which the pose recognized by the pre-learned machine learning model MM is not determined to be correct. In this way, if the second photographing direction is set to a direction that is not determined to be correct in the pre-learned machine learning model MM by simulation, it is possible to generate training data that fills in the difference between simulation and the actual environment.

[0068] As described above, in the second embodiment, as in the first embodiment, the first pose of the object OB is recognized from the first image using the pre-learned machine learning model MM, and in a case where the first pose is determined to be correct, a second image is captured from a second photographing direction different from the first photographing direction using a camera, and training data for formal learning is generated by assigning a second pose of the object OB to the second image. As a result, it is possible to easily generate training data for machine learning in the actual environment, and since it is possible to fill in the difference between simulation and the actual environment, it is possible to improve the performance of the machine learning model MM. In addition, in the second embodiment, since the second photographing direction is set to a direction that is not determined to be correct in the pre-learned machine learning model MM, it is possible to generate training data that fills in the difference between simulation and the actual environment.

[0069] In the above-described first embodiment and second embodiment, a case where an object is recognized in a work using the robot 100 is assumed, but the present disclosure can also be applied to a case where an object is recognized in a system that does not use a robot.

[0070] • Other modes:

[0071] The present disclosure is not limited to the above-described embodiments, and can be implemented in various ways within the scope of the gist thereof. For example, the present disclosure can also be implemented by the following aspects. In order to solve part or all of the technical problems of the present disclosure, or in order to achieve part or all of the effects of the present disclosure, the technical features of the above-described embodiments can be appropriately replaced, combined, or deleted corresponding to the technical features of each of the aspects described below. In addition, as long as the technical features are not described as essential contents in the present specification, they can be appropriately deleted.

[0072] (1) According to a first aspect of the present disclosure, there is provided a method of generating training data for learning of a machine learning model that identifies a pose of an object. The method includes: (a) a process of performing pre-learning of the machine learning model using simulation data of the object; (b) a process of capturing a first image of the object from a first photographing direction using a camera; (c) a process of identifying a first pose of the object from the first image using the machine learning model that has completed the pre-learning; (d) a process of making a correct / incorrect determination of the first pose; (e) in a case where the first pose is determined to be correct, a process of capturing a second image of the object from a second photographing direction different from the first photographing direction using the camera, calculating a second pose by performing a transformation of the first pose corresponding to a change from the first photographing direction to the second photographing direction, and generating training data by assigning the second pose to the second image; and (f) a process of changing a real pose of the object and repeating the processes (b) to (e).

[0073] According to this method, it is possible to easily generate training data for machine learning in an actual environment, and since it is possible to fill in the difference between simulation and an actual environment, it is possible to improve the performance of the machine learning model.

[0074] (2) In the above method, the process (e) can also include: (e1) a process of calculating a specular reflection direction from the object based on a direction in which illumination light is radiated and the first pose; and (e2) a process of setting the specular reflection direction as the second photographing direction.

[0075] According to this method, since the specular reflection direction is set as the second photographing direction, it is possible to generate training data for correctly performing pose identification of a glossy object.

[0076] (3) In the above method, the second photographing direction can also be set as a direction in which a pose identified by the machine learning model that has completed the pre-learning using an image captured from the second photographing direction is not determined to be correct.

[0077] According to this method, since the second photographing direction is set as a direction that is not determined to be correct in the machine learning model that has completed the pre-learning by simulation, it is possible to generate training data that fills in the difference between simulation and an actual environment.

[0078] (4) In the above method, before the processes (b) to (f), a process of performing a plurality of times a process of identifying a pose of the object using the machine learning model whose pre-learning is completed and performing a correct / incorrect determination, and registering a history of the pose determined to be correct, can be further included, and the second photographing direction can also be set to a direction in which there is no pose of the history determined to be correct.

[0079] According to this method, since the second photographing direction is set to a direction in which there is no pose of the history determined to be correct in the machine learning model whose pre-learning is completed based on the simulation data, it is possible to generate training data that fills the difference between the simulation and the actual environment.

[0080] (5) In the above method, the process (d) can also include: (d1) a process of generating a simulation image of the object in the first pose by simulation; (d2) a process of calculating a reliability score of the first pose using the first image and the simulation image; and (d3) a process of determining the correct / incorrect of the first pose by comparing the reliability score with a threshold value.

[0081] According to this method, it is possible to determine the correct / incorrect of the first pose by the reliability score calculated from the first image and the simulation image.

[0082] (6) According to a second aspect of the present disclosure, there is provided a system that generates training data for learning of a machine learning model that identifies a pose of an object. The system includes: a camera that photographs the object; and a training data generation unit that performs generation of the training data. The training data generation unit performs: (a) a process of performing pre-learning of the machine learning model using simulation data of the object; (b) a process of photographing a first image of the object from a first photographing direction using the camera; (c) a process of identifying a first pose of the object from the first image using the machine learning model whose pre-learning is completed; (d) a process of determining the correct / incorrect of the first pose; (e) a process of photographing a second image of the object from a second photographing direction different from the first photographing direction using the camera, calculating a second pose by performing a transformation corresponding to a change from the first photographing direction to the second photographing direction on the first pose, and generating training data by assigning the second pose to the second image; and (f) a process of changing a real pose of the object and repeating the processes (b) to (e).

[0083] (7) According to a third aspect of the present disclosure, there is provided a computer program that causes a processor to execute processing of generating training data for learning of a machine learning model that identifies a pose of an object. The computer program causes the processor to execute: (a) processing of performing pre-learning of the machine learning model using simulation data of the object; (b) processing of capturing a first image of the object from a first photographing direction using a camera; (c) processing of identifying a first pose of the object from the first image using the machine learning model after the pre-learning; (d) processing of making a true or false determination of the first pose; (e) processing of capturing a second image of the object from a second photographing direction different from the first photographing direction using the camera, calculating a second pose by performing a transformation of the first pose corresponding to a change from the first photographing direction to the second photographing direction, and generating training data by assigning the second pose to the second image; and (f) processing of changing a real pose of the object and repeating the processing of (b) to (e).

[0084] The present disclosure can also be realized by various modes other than the above. For example, it can be realized by a robot system that includes a robot and a robot control device, a computer program for realizing the functions of the robot control device, a non-transitory storage medium in which the computer program is recorded, and the like.

Claims

1. A method for generating training data for learning a machine learning model, the machine learning model recognizing the pose of an object, the method comprising: Step a: Pre-learning of the machine learning model is performed using simulated data of the object; Step b: Using a camera to capture a first image of the object from a first shooting direction; Step c: Using the pre-learned machine learning model, identify the first pose of the object from the first image; Step d involves determining the correctness of the first pose. In step e, if the first pose is determined to be correct, the camera is used to capture a second image of the object from a second shooting direction different from the first shooting direction. The second pose is calculated by performing a transformation on the first pose that corresponds to the change from the first shooting direction to the second shooting direction. Training data is generated by assigning the second pose to the second image. as well as In step f, the actual pose of the object is changed, and steps b through e are repeated. The e-process includes: In step e1, the specular reflection direction of the illumination light from the object is determined based on the direction of the illumination light illuminating the object and the first pose. as well as In step e2, the mirror reflection direction is set as the second shooting direction so that the mirror reflection of the object is incident on the camera.

2. The method according to claim 1, wherein, The d-step includes: Step d1 involves generating a simulated image of the object in the first pose through simulation. In step d2, the reliability score of the first pose is calculated using the first image and the simulated image; and In step d3, the correctness of the first pose is determined by comparing the reliability score with a threshold.

3. A system for generating training data for learning a machine learning model, the machine learning model recognizing the pose of an object, the system comprising: Camera, to photograph the object; and The training data generation unit generates the training data. The training data generation unit performs the following: A process involves using simulated data of the object to perform pre-learning of the machine learning model; b. Processing: Using the camera, capture a first image of the object from a first shooting direction; c. Processing: Using the pre-learned machine learning model, identify the first pose of the object from the first image; d processing, determining the correctness of the first pose; e-processing: using the camera to capture a second image of the object from a second shooting direction different from the first shooting direction; calculating the second pose by performing a transformation on the first pose corresponding to the change from the first shooting direction to the second shooting direction; and generating training data by assigning the second pose to the second image. as well as In step f, the actual pose of the object is changed, and steps b through e are repeated. The e-process includes: e1 processing: Based on the direction of the illumination light illuminating the object and the first pose, determine the specular reflection direction of the illumination light from the object; as well as e2 processing sets the specular reflection direction as the second shooting direction, so that the specular reflection of the object is incident on the camera.

4. A storage medium storing a computer program that causes a processor to perform processing to generate training data for learning a machine learning model, the machine learning model recognizing the pose of an object, the computer program causing the processor to perform: A process involves using simulated data of the object to perform pre-learning of the machine learning model; b. Processing: Using a camera to capture a first image of the object from a first shooting direction; c. Processing: Using the pre-learned machine learning model, identify the first pose of the object from the first image; d processing, determining the correctness of the first pose; e-processing: using the camera to capture a second image of the object from a second shooting direction different from the first shooting direction; calculating the second pose by performing a transformation on the first pose corresponding to the change from the first shooting direction to the second shooting direction; and generating training data by assigning the second pose to the second image. as well as In step f, the actual pose of the object is changed, and steps b through e are repeated. The e-process includes: e1 processing: Based on the direction of the illumination light illuminating the object and the first pose, determine the specular reflection direction of the illumination light from the object; as well as e2 processing sets the specular reflection direction as the second shooting direction, so that the specular reflection of the object is incident on the camera.

Images