Method, system, and computer program for creating training data for use in machine learning models

Abstract

To provide a technique for easily creating teacher data used for learning a machine learning model in a real environment.SOLUTION: A method of the present disclosure comprises steps of: (a) performing pre-learning of a machine learning model using simulation data of an object; (b) imaging a first image of the object from a first imaging direction; (c) recognizing a first position / orientation of the object from the first image using a pre-learned machine learning model; (d) performing correctness / incorrectness determination for the first position / orientation; (e) imaging, when the first position / orientation is determined to be correct, a second image of the object from a second imaging direction different from the first imaging direction, transforming the first position / orientation according to a change from the first imaging direction to the second imaging direction to thereby calculate a second position / orientation, and giving the second position / orientation to the second image to thereby create teacher data; and (f) changing a real position / orientation of the object to repeat steps (b) to (e).SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a method, a system, and a computer program for creating training data for use in training a machine learning model. [Background technology]

[0002] Patent Document 1 discloses a technique for recognizing the position and orientation of an object by matching the features of the object determined from 3D model data with an image of the object. Patent Document 2 discloses a technique for recognizing the position and orientation of an object using a machine learning model. Generally, when a machine learning model is applied to recognizing the position and orientation of an object, training data labeled with the position and orientation is required, and creating the training data requires a huge amount of preparation. Therefore, Patent Document 2 uses a method for generating training data through simulation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-185239 [Patent Document 2] Japanese Patent Application Publication No. 2020-87310 Summary of the Invention [Problem to be solved by the invention]

[0004] When creating training data through simulation, images of objects are generated for a certain scene, and position and orientation labels are assigned to the objects in the scene. However, there is a difference between the images obtained in the simulation and in the real environment, and there is a problem that machine learning models trained with training data created through simulation tend to have a lower accuracy in recognizing position and orientation in the real environment. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a method for generating training data for use in training a machine learning model that recognizes the position and orientation of an object, the method including: (a) performing pre-training of the machine learning model using simulation data of the object; (b) capturing a first image of the object from a first imaging direction using a camera; (c) recognizing a first position and orientation of the object from the first image using the pre-trained machine learning model; (d) determining whether the first position and orientation are correct; (e) if the first position and orientation are determined to be correct, capturing a second image of the object from a second imaging direction different from the first imaging direction using the camera, transforming the first position and orientation in accordance with a change from the first imaging direction to the second imaging direction to calculate a second position and orientation, and assigning the second position and orientation to the second image to generate training data; and (f) changing the actual position and orientation of the object and repeating steps (b) to (e).

[0006] According to a second aspect of the present disclosure, there is provided a system for creating teacher data used in training a machine learning model that recognizes the position and orientation of an object. The system includes a camera that captures an image of the object and a teacher data creation unit that creates the teacher data. The teacher data creation unit executes the following steps: (a) performing pre-training of the machine learning model using simulation data of the object; (b) capturing a first image of the object from a first imaging direction using the camera; (c) recognizing a first position and orientation of the object from the first image using the pre-trained machine learning model; (d) determining whether the first position and orientation are correct; (e) capturing a second image of the object from a second imaging direction different from the first imaging direction using the camera, converting the first position and orientation in accordance with a change from the first imaging direction to the second imaging direction to calculate a second position and orientation, and assigning the second position and orientation to the second image to create teacher data; and (f) changing the actual position and orientation of the object and repeating steps (b) to (e).

[0007] According to a third aspect of the present disclosure, there is provided a computer program that causes a processor to execute a process of creating teacher data to be used in training a machine learning model that recognizes the position and orientation of an object. The computer program causes the processor to execute the following processes: (a) performing pre-training of the machine learning model using simulation data of the object, (b) capturing a first image of the object from a first imaging direction using a camera, (c) recognizing a first position and orientation of the object from the first image using the pre-trained machine learning model, (d) determining whether the first position and orientation are correct, (e) capturing a second image of the object from a second imaging direction different from the first imaging direction using the camera, and calculating a second position and orientation by transforming the first position and orientation in accordance with a change from the first imaging direction to the second imaging direction, and assigning the second position and orientation to the second image to create teacher data; and (f) changing the actual position and orientation of the object and repeating the processes (b) to (e). [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing a configuration of a robot system according to an embodiment. [Figure 2] FIG. 2 is a functional block diagram of an information processing apparatus according to an embodiment. [Figure 3] FIG. 1 is an explanatory diagram showing a function for recognizing the position and orientation of an object using a machine learning model. [Figure 4] 3 is a flowchart showing the overall procedure of processing in the first embodiment. [Figure 5] FIG. 10 is an explanatory diagram of a process for creating training data for pre-learning through simulation. [Figure 6] 10 is a flowchart showing the detailed procedure of step S130 in the first embodiment. [Figure 7] FIG. 4 is an explanatory diagram showing how the imaging direction is changed in the first embodiment. [Figure 8] 10 is a flowchart showing the overall procedure of processing in the second embodiment. [Figure 9]10 is a flowchart showing the detailed procedure of step S125 in the second embodiment. [Figure 10] FIG. 10 is an explanatory diagram showing a method for determining a second imaging direction using statistical data. [Figure 11] 10 is a flowchart showing the detailed procedure of step S135 in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. First embodiment: FIG. 1 is an explanatory diagram showing an example of a robot system according to an embodiment. The robot system includes a robot 100, a control device 200 that controls the robot 100, an information processing device 300, a camera 400, and a platform 500. The information processing device 300 is, for example, a personal computer. FIG. 1 depicts three axes X, Y, and Z that define a Cartesian coordinate system in a three-dimensional space. The X and Y axes are horizontal axes, and the Z axis is a vertical axis. These X, Y, and Z axes are coordinate axes of a robot coordinate system Σr, whose origin is a predetermined position of the robot 100.

[0010] The robot 100 includes a base 110 and a robot arm 120. A hand 150 serving as an end effector is attached to an arm end 122, which is the tip of the robot arm 120. The hand 150 can be realized as a gripper or a suction pad capable of grasping an object OB. A TCP (Tool Center Point) is set at the tip of the hand 150 as a control point of the robot 100. The control point TCP can be set at any position.

[0011] The robot arm 120 is connected in sequence by six joints J1 to J6. Of these joints J1 to J6, three joints J2, J3, and J5 are bending joints, and the other three joints J1, J4, and J6 are torsion joints. In this embodiment, a six-axis robot is exemplified, but a robot having any robot arm mechanism with one or more joints can be used. Furthermore, although the robot 100 in this embodiment is a vertical multi-joint robot, a horizontal multi-joint robot may also be used.

[0012] A first tray 510 and a second tray 520 are installed on the stand 500. A plurality of objects OB are piled up randomly on the first tray 510. The second tray 520 is used as a place to place the objects OB taken out from the first tray 510. The robot 100 performs the task of taking out the objects OB from the first tray 510 and placing them on the second tray 520.

[0013] A camera 400 is installed at the tip of the robot arm 120 to capture an image of the object OB in the first tray 510. The image captured by the camera 400 is used to determine the three-dimensional position and orientation of the object OB. The three-dimensional position and orientation will be referred to as the "position and orientation" hereinafter. The camera 400 may be, for example, an RGBD camera or a stereo camera. An RGBD camera is a camera having an RGB camera that captures RGB images and a D camera that captures depth images. A monochrome camera that captures gray images may be used instead of the RGB camera. A monocular camera may also be used as the camera 400. The camera 400 does not need to be installed on the robot arm 120 and may be installed separately from the robot 100. However, installing the camera 400 on the robot arm 120 has the advantage of making it easier to change the direction in which the camera 400 captures the object OB.

[0014] 2 is a block diagram showing the functions of an information processing device 300. The information processing device 300 has a processor 310, a memory 320, an interface circuit 330, and an input device 340 and a display device 350 connected to the interface circuit 330. The interface circuit 330 is further connected to a control device 200 and a camera 400. Measurement results of the current sensor 180 and joint encoder 190 of the robot 100 are supplied to the information processing device 300 via the control device 200. The current sensor 180 is a sensor that measures the current of a motor provided in each joint of the robot 100. The joint encoder 190 is a sensor that detects the motion position of each joint.

[0015] In this embodiment, the camera 400 includes a first camera 410 that captures two-dimensional images such as RGB images or grayscale images, a second camera 420 that captures depth images, and an illumination unit 430 that irradiates illumination light for the second camera 420. The illumination unit 430 is a projector that irradiates an infrared pattern for capturing depth images.

[0016] The processor 310 functions as a teacher data creation unit 311 that creates teacher data used in training a machine learning model that recognizes the position and orientation of an object OB. The teacher data creation unit 311 includes the functions of a simulation execution unit 312, a learning execution unit 314, an object recognition unit 316, and a correct / incorrect determination unit 318. The simulation execution unit 312 executes a process of creating teacher data for pre-training of the machine learning model by simulating a scene in which an object OB is present in the first tray 510. The learning execution unit 314 executes pre-training and actual training of the machine learning model. The object recognition unit 316 executes a process of recognizing the position and orientation of the object OB from an image captured by the camera 400 using a trained machine learning model. The correct / incorrect determination unit 318 determines whether the position and orientation of the object OB recognized using the machine learning model is correct or incorrect. The function of the teacher data creation unit 311 is realized by the processor 310 executing a computer program stored in the memory 320. However, some or all of the functions of the teacher data creation unit 311 may be implemented using a hardware circuit.

[0017] The memory 320 stores a machine learning model MM that recognizes the position and orientation of the object OB, simulation data SD, pre-learning training data PTD, main training training data RTD, and a robot control program RP. The simulation data SD is data used to simulate a scene in which the objects OB are piled up randomly on the first tray 510, and includes CAD data, which is three-dimensional model data of the objects OB, robot data indicating the position and shape of the robot 100, and camera parameters of the camera 400. The robot control program RP is composed of a plurality of commands that operate the robot 100.

[0018] FIG. 3 is an explanatory diagram showing the recognition function of the position and orientation of an object OB using the machine learning model MM. The camera 400 is calibrated in advance, and the relative relationship between the camera coordinate system Σc and the robot coordinate system Σr is known. The camera 400 creates a two-dimensional image M1 and a depth image M2 by capturing an image of the object OB. Hereinafter, these images M1 and M2 will be collectively referred to as "image IM." The machine learning model MM is a regression model that outputs the position and orientation PA of the object OB in response to the input of the image IM. The position and orientation PA is expressed, for example, by the position (x, y, z) and orientation (w, p, r) in the robot coordinate system Σr. The orientation is expressed by the rotation angles (w, p, r) around three axes.

[0019] There are three possible ways to recognize the position and orientation PA of an object among a plurality of objects in a random pile. (A1) Only the top object is considered. (A2) Not only the topmost object, but also objects partially overlapping other objects are included. Specifically, for example, objects with an obscured area ratio of 20% or less are included. (A3) Applies to all objects. Considering actual bulk picking work, it is preferable to create training data so that the position and orientation of the object in (A2) above can be recognized. In this case, for the bulk pile state generated by the simulation, images labeled with the positions and orientations of multiple objects that satisfy the condition in (A2) above are generated as a training data set. However, for the sake of convenience, in the following explanation, it is assumed that the machine learning model MM recognizes the position and orientation PA of only one object OB in accordance with (A1) above.

[0020] 4 is a flowchart showing the overall procedure of the process in the first embodiment. In step S110, the simulation execution unit 312 creates teacher data for pre-learning by simulation.

[0021] FIG. 5 is an explanatory diagram of the process of creating training data for pre-learning through simulation. The simulation execution unit 312 inputs simulation data SD, including CAD data of objects OB to be recognized and various parameters, into the scene simulator SS. The scene simulator SS creates a scene of a randomly piled object OB by using a physical simulation, for example, by randomly dropping multiple objects OB represented by CAD data from a certain height. Since this is a simulation, the positions and orientations of the randomly piled objects OB are known. The simulation execution unit 312 renders this scene based on camera information set by various parameters to generate an image IM including a 2D image M1 and a depth image M2. The simulation execution unit 312 creates training data PTD for pre-learning by labeling the image IM with the positions and orientations of the objects OB. By generating multiple scenes of randomly piled objects, a large amount of training data PTD can be generated. In this embodiment, a simulation of a randomly piled object OB was performed. However, a simulation of a randomly piled object OB may also be performed, such as a state in which a single object OB is placed on a tray 510 at an arbitrary position and orientation.

[0022] In step S120, the learning execution unit 314 executes pre-learning of the machine learning model MM using the pre-learning teacher data PTD.

[0023] In step S130, the training data creation unit 311 creates training data RTD for the actual training, taking into account the specular reflection of the object OB. The reason for taking into account the specular reflection of the object OB is as follows: In the simulation performed in step S110, it is difficult to accurately reproduce an image IM captured in a real environment in which the specular reflection of the object OB is incident on the camera 400. For this reason, the machine learning model MM trained using the training data for pre-training tends to have reduced accuracy in recognizing the position and orientation of the object OB in a real environment in which the specular reflection of the object OB is incident on the camera 400. In particular, if the object OB is glossy, it is difficult to accurately reflect the degree of light reflection in the simulation, which tends to significantly reduce recognition accuracy. Therefore, in step 130, training data for the actual training is created for a state in which the camera 400 receives the specular reflection of the object OB.

[0024] 6 is a flowchart showing the detailed procedure of step S130. In step S310, the training data creation unit 311 uses the camera 400 to capture a first image of the randomly piled objects OB. The "first image" includes the two-dimensional image M1 and depth image M2 shown in FIG. 3. The capturing direction of the camera 400 at this time is called the "first capturing direction."

[0025] In step S320, the object recognition unit 316 recognizes the first position and orientation of the object OB from the first image using the pre-trained machine learning model MM. In step S330, the accuracy determination unit 318 determines whether the recognition result is accurate. This accuracy determination can be performed using, for example, a reliability score. The reliability score can be calculated by creating a simulation image of the object OB in the first position and orientation by simulation using the recognized first position and orientation of the object OB, and then calculating the reliability score as an index representing the degree to which this simulation image matches the first image captured by the camera 400. For example, the reliability score RS is calculated using the following equation. RS = α × S1 + (1 - α) × S2 (1) Here, α is a coefficient in the range of 0≦α≦1, S1 is the similarity between the 2D image included in the simulation image and the 2D image included in the first image, and S2 is the similarity between the depth image included in the simulation image and the depth image included in the first image. The similarities S1 and S2 can be calculated as image similarity or cosine similarity. According to the above formula (1), the reliability score RS can be calculated by weighted addition of the similarities S1 and S2.

[0026] The accuracy determination unit 318 determines whether the recognition result is accurate by comparing the reliability score RS with a preset reliability threshold. That is, if the reliability score RS is equal to or greater than the reliability threshold, the recognition result is determined to be accurate; if the reliability score RS is less than the reliability threshold, the recognition result is determined to be incorrect. As an alternative method, for example, the simulation image and the outline of the object OB in the first image may be superimposed and displayed on the display device 350 to allow the user to determine whether the recognition result is accurate. If the recognition result is accurate, the process proceeds to step S340, which will be described later. On the other hand, if the recognition result is incorrect, the process proceeds to step S390, where the actual position and orientation of the object OB are changed, and the process returns to step S310. The position and orientation of the object OB may be changed, for example, by the user redoing the stacking of the objects OB.

[0027] In step S340, the teacher data creation unit 311 determines the direction of specular reflection of the illumination light as the second imaging direction, and in step S350, a second image is generated by using the camera 400 to capture an image of the object OB from the second imaging direction.

[0028] FIG. 7 is an explanatory diagram showing how the imaging direction is changed in the first embodiment. The upper part of FIG. 7 shows a state in which an image of the object OB is captured in the first imaging direction in step S310. In this state, diffusely reflected light from the illumination unit 430 is incident on the second camera 420. The depth image captured by the second camera 420 is an image obtained from this diffusely reflected light. In step S340, as shown in the lower part of FIG. 7, the position and orientation of the camera 400 are changed so that specularly reflected light of the illumination light from the illumination unit 430 is incident on the second camera 420. Since the emission direction of the illumination light from the illumination unit 430 is known, the direction of the specularly reflected light from the object OB can be calculated if the position and orientation of the object OB are known. The position and orientation of the camera 400 can be changed by moving the robot arm 120. As described above, it is difficult in a simulation to accurately reproduce an image captured in a real environment in which specularly reflected light from object OB is incident on camera 400, and this tendency is particularly pronounced when object OB is a glossy object. Therefore, in step S340, a second imaging direction is determined in the real environment so that specularly reflected light from object OB is incident on camera 400. If training data is created using an image captured from this second imaging direction, it is possible to obtain training data that is difficult to create by simulation.

[0029] 7, the state in which specularly reflected light from the object OB is incident on the second camera 420 for depth images is determined as the second imaging direction. However, instead, the state in which specularly reflected light from the object OB is incident on the first camera 410 for two-dimensional images may be determined as the second imaging direction. In the latter case, this is feasible if the position of the illumination light source for the two-dimensional images is known. Furthermore, the state in which specularly reflected light from the object OB is incident on both the first camera 410 and the second camera 420 may be set as the second imaging direction for each of them, and imaging may be performed.

[0030] In step S360, the training data creation unit 311 calculates the second position and orientation of the object OB in the second image. The second position and orientation can be calculated according to the following equation. Pb=B H A ×Pa (2) Here, Pb is a 4x4 matrix representing the second position and orientation, Pa is a 4x4 matrix representing the first position and orientation, B H A is a homogeneous transformation matrix that represents the transformation from the first position and orientation Pa to the second position and orientation Pb. B H A is the same as the matrix representing the transformation from the first imaging direction Da to the second imaging direction Db of the camera 400. In other words, the second position and orientation Pb is calculated by performing a transformation on the first position and orientation Pa in accordance with the change from the first imaging direction Da to the second imaging direction Db.

[0031] In step S370, the teacher data creation unit 311 creates images with the positions and orientations as teacher data for the main learning. That is, the teacher data for the main learning is created by labeling the second image captured in step S350 with the second position and orientation. At this time, the teacher data for the main learning may be added by labeling the first image captured in step S310 with the first position and orientation.

[0032] In step S380, the training data creation unit 311 determines whether the expected number of training data for this learning has been obtained. If a sufficient number of training data has not been obtained, the process proceeds to step S390, where the actual position and orientation of the object OB is changed, and the process returns to step S310, where the above-described steps S310 to S380 are executed again. On the other hand, if a sufficient number of training data has been obtained, the process of FIG. 6 ends.

[0033] After the processing of step S130 shown in FIG. 6 is completed, in step S140 of FIG. 4, the learning execution unit 314 performs main learning of the machine learning model MM using the training data for main learning created in step S130. During main learning, not only the training data created in step S130 but also the training data for pre-learning created in step S110 may be used. In step S140, the robot 100 performs a task using the recognition of the object OB by the trained machine learning model MM. This task is performed in accordance with a robot control program RP created in advance.

[0034] As described above, in the first embodiment, a pre-trained machine learning model MM is used to recognize a first position and orientation of an object OB from a first image. If the first position and orientation is determined to be correct, a camera is used to capture a second image from a second imaging direction different from the first imaging direction, and the second position and orientation of the object OB is assigned to the second image, thereby creating training data for this training. As a result, training data for machine learning can be easily created in a real environment, and the difference between the simulation and the real environment can be bridged, thereby improving the performance of the machine learning model MM. Furthermore, in the first embodiment, the second imaging direction is set to the direction of specular reflection, so training data for correctly recognizing the position and orientation of a glossy object can be created.

[0035] B. Second embodiment: Fig. 8 is a flowchart showing the overall procedure of processing in the second embodiment. The only differences from the first embodiment shown in Fig. 4 are that step S125 is added and step S130 is replaced with step S135; the other steps are the same as in the first embodiment. The device configuration of the second embodiment is also the same as in the first embodiment.

[0036] In step S125, the teacher data creation unit 311 creates statistical data using the pre-trained machine learning model MM. This statistical data is data used to determine the second imaging direction.

[0037] FIG. 9 is a flowchart showing the detailed procedure of step S125. In step S210, the training data creation unit 311 uses the camera 400 to capture an image of the randomly piled objects OB. This image includes the two-dimensional image M1 and depth image M2 shown in FIG. 3. In step S220, the object recognition unit 316 recognizes the position and orientation of the object OB from the image obtained in step S210 using the pre-trained machine learning model MM. In step S230, the accuracy determination unit 318 determines whether the recognition result is correct. This accuracy determination can be performed using a confidence score, as in step S330 of FIG. 6 described above. If the recognition result is correct, the process proceeds to step S240, which will be described later. On the other hand, if the recognition result is incorrect, the process proceeds to step S260, where the actual position and orientation of the object OB are changed, and the process returns to step S210. The position and orientation of the object OB can be changed, for example, by the user re-stacking the object OB.

[0038] In step S240, the teacher data creation unit 311 creates images with position and orientation as statistical data. That is, statistical data is created by adding the position and orientation recognized in step S220 to the image captured in step S210. In step S250, the teacher data creation unit 311 determines whether the expected number of statistical data has been obtained. If a sufficient number of statistical data has not been obtained, the process proceeds to step S260, where the actual position and orientation of the object OB is changed, the process returns to step S210, and the above-mentioned steps S210 to S250 are executed again. On the other hand, if a sufficient number of statistical data has been obtained, the process proceeds to step S270.

[0039] In step S270, the teacher data creation unit 311 calculates the missing position and orientation with reference to the statistical data, and registers it in the statistical data.

[0040] FIG. 10 is an explanatory diagram showing missing positions and attitudes in the statistical data. A "missing position and attitude" is an attitude not included in the statistical data created in steps S210 to S260. The upper part of FIG. 10 shows only the rotation angles (w, p) of the positions and attitudes (x, y, z, w, p, r) included in the statistical data in a two-dimensional manner. Ideally, the statistical data would include positions and attitudes uniformly, but positions and attitudes determined to be incorrect in step S230 of FIG. 9 are not included in the statistical data. In the upper part of FIG. 10, black circles indicate positions and attitudes determined to be correct and included in the statistical data. In an incorrect solution area, positions and attitudes determined to be correct are missing. The training data creation unit 311 recognizes one or more positions and attitudes in this incorrect solution area as "missing positions and attitudes" and registers them in the statistical data. The lower part of FIG. 10 shows a state in which the missing positions and attitudes (Desired position and attitude) have been added as white circles. The "missing position and orientation" can be determined, for example, by dividing an omnidirectional sphere around the object OB into certain areas and mapping the positions and orientations included in the statistical data, and then recognizing that there are few recognition results for a particular area. The missing position and orientation is a position and orientation that cannot be accurately recognized by the machine learning model MM that has undergone pre-training through simulation, and is considered to be a position and orientation where there is a large image difference between the simulation and reality. Therefore, by registering one or more positions and orientations as "missing positions and orientations" in an incorrect area where a position and orientation determined to be correct in the statistical data is missing, it is possible to efficiently collect appropriate positions and orientations to fill the difference between the simulation and reality.

[0041] When the process of step S125 is completed in this way, the process proceeds to step S135 in FIG. 8, where the teacher data creation unit 311 creates teacher data for main learning using the statistical data.

[0042] Fig. 11 is a flowchart showing the detailed procedure of step S 135. The procedure in Fig. 11 is the same as that in Fig. 6 in the first embodiment, except that step S340 is replaced with step S345. The other steps are the same as those in Fig. 6.

[0043] In step S345, the teacher data creation unit 311 determines a second imaging direction using statistical data. This second imaging direction is the imaging direction corresponding to the missing position / orientation described in FIG. 10. If there are multiple missing positions / orientations, the imaging direction that obtains any of the missing positions / orientations can be adopted as the second imaging direction. For example, the imaging direction that obtains the missing position / orientation closest to the first position / orientation can be adopted as the second imaging direction. Note that two or more second imaging directions may be determined from one first position / orientation. In step S350, a second image is generated by capturing an image of the object OB from the second imaging direction using the camera 400. The processing from step S350 onwards is the same as in the first embodiment.

[0044] The second imaging direction determined in step S345 of the second embodiment is a direction in which the position and orientation recognized by the pre-trained machine learning model MM is not determined to be correct. In this way, by setting the second imaging direction to a direction that is not determined to be correct by the pre-trained machine learning model MM by simulation, training data that bridges the gap between the simulation and the real environment can be created.

[0045] As described above, in the second embodiment, as in the first embodiment, a pre-trained machine learning model MM is used to recognize a first position and orientation of an object OB from a first image. If the first position and orientation is determined to be correct, a camera is used to capture a second image from a second imaging direction different from the first imaging direction, and the second position and orientation of the object OB is assigned to the second image, thereby creating training data for this training. As a result, training data for machine learning can be easily created in a real environment, and differences between the simulation and the real environment can be bridged, thereby improving the performance of the machine learning model MM. Furthermore, in the second embodiment, the second imaging direction is set to a direction that is not determined to be correct by the pre-trained machine learning model MM, making it possible to create training data that bridges the differences between the simulation and the real environment.

[0046] In the first and second embodiments described above, it is assumed that an object is recognized in a task using the robot 100, but the present disclosure is also applicable to the case of recognizing an object in a system that does not use a robot.

[0047] Other forms: The present disclosure is not limited to the above-described embodiments and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following aspects. The technical features in the above embodiments corresponding to the technical features in each aspect described below can be appropriately replaced or combined to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.

[0048] (1) According to a first aspect of the present disclosure, there is provided a method for generating training data for use in training a machine learning model that recognizes the position and orientation of an object, the method including: (a) performing pre-training of the machine learning model using simulation data of the object; (b) capturing a first image of the object from a first imaging direction using a camera; (c) recognizing a first position and orientation of the object from the first image using the pre-trained machine learning model; (d) determining whether the first position and orientation are correct; (e) if the first position and orientation are determined to be correct, capturing a second image of the object from a second imaging direction different from the first imaging direction using the camera, transforming the first position and orientation in accordance with a change from the first imaging direction to the second imaging direction to calculate a second position and orientation, and assigning the second position and orientation to the second image to generate training data; and (f) changing the actual position and orientation of the object and repeating steps (b) to (e). This method makes it possible to easily create training data for machine learning in a real environment, and also to bridge the gap between the simulation and the real environment, thereby improving the performance of the machine learning model.

[0049] (2) In the above method, the step (e) may include: (e1) determining the direction of specular reflection from the object from the irradiation direction of illumination light onto the object and the first position and orientation; and (e2) setting the specular reflection direction as the second shooting direction. According to this method, the second imaging direction is set to the direction of specular reflection, so that training data for correctly recognizing the position and orientation of a glossy object can be created.

[0050] (3) In the above method, the second shooting direction may be set to a direction in which the position and orientation recognized by the pre-trained machine learning model using an image captured from the second shooting direction is not determined to be correct. According to this method, the second shooting direction is set to a direction that is not determined to be correct by a machine learning model that has been pre-trained through simulation, so that training data that bridges the gap between the simulation and the real environment can be created.

[0051] (4) In the above method, before steps (b) to (f), a step of recognizing the position and orientation of the object using the pre-trained machine learning model, determining whether the position and orientation are correct, and registering a history of the position and orientation determined to be correct may be performed multiple times, and the second shooting direction may be set to a direction that obtains a position and orientation for which there is no history determined to be correct. According to this method, the second shooting direction is set to a direction that has not been determined to be correct by a machine learning model that has been pre-trained using simulation data, making it possible to create training data that bridges the gap between the simulation and the real environment.

[0052] (5) In the above method, the step (d) may include: (d1) creating a simulation image of the object in the first position and posture by simulation; (d2) calculating a reliability score of the first position and posture using the first image and the simulation image; and (d3) determining whether the first position and posture is correct by comparing the reliability score with a threshold value. According to this method, the accuracy of the first position and orientation can be determined based on the reliability score calculated based on the first image and the simulation image.

[0053] (6) According to a second aspect of the present disclosure, there is provided a system for creating teacher data used in training a machine learning model that recognizes the position and orientation of an object. The system includes a camera that captures an image of the object and a teacher data creation unit that creates the teacher data. The teacher data creation unit executes the following processes: (a) performing pre-training of the machine learning model using simulation data of the object; (b) capturing a first image of the object from a first imaging direction using the camera; (c) recognizing a first position and orientation of the object from the first image using the pre-trained machine learning model; (d) determining whether the first position and orientation are correct; (e) capturing a second image of the object from a second imaging direction different from the first imaging direction using the camera, converting the first position and orientation in accordance with a change from the first imaging direction to the second imaging direction to calculate a second position and orientation, and assigning the second position and orientation to the second image to create teacher data; and (f) changing the actual position and orientation of the object and repeating processes (b) to (e).

[0054] (7) According to a third aspect of the present disclosure, there is provided a computer program that causes a processor to execute a process of creating teacher data to be used in training a machine learning model that recognizes the position and orientation of an object. The computer program causes the processor to execute the following processes: (a) performing pre-training of the machine learning model using simulation data of the object, (b) capturing a first image of the object from a first imaging direction using a camera, (c) recognizing a first position and orientation of the object from the first image using the pre-trained machine learning model, (d) determining whether the first position and orientation are true or false, (e) capturing a second image of the object from a second imaging direction different from the first imaging direction using the camera, and calculating a second position and orientation by transforming the first position and orientation in accordance with a change from the first imaging direction to the second imaging direction, and creating teacher data by assigning the second position and orientation to the second image, and (f) changing the actual position and orientation of the object and repeating the processes (b) to (e).

[0055] The present disclosure may be realized in various forms other than those described above, such as a robot system including a robot and a robot control device, a computer program for implementing the functions of the robot control device, or a non-transitory storage medium on which the computer program is recorded. [Explanation of symbols]

[0056] 100...robot, 110...base, 120...robot arm, 122...arm end, 150...hand, 180...current sensor, 190...joint encoder, 200...control device, 300...information processing device, 310...processor, 311...teacher data creation unit, 312...simulation execution unit, 314...learning execution unit, 316...object recognition unit, 318...correctness determination unit, 320...memory, 330...interface circuit, 340...input device, 350...display device, 400...camera, 410...first camera, 420...second camera, 430...illumination unit, 500...stand, 510...first tray, 520...second tray

Claims

1. A method for creating training data used to train a machine learning model that recognizes the position and orientation of an object, comprising: (a) performing pre-training of the machine learning model using simulation data of the object; (b) capturing a first image of the object from a first imaging direction using a camera; (c) recognizing a first position and orientation of the object from the first image using the pre-trained machine learning model; (d) determining whether the first position and posture are correct; (e) when the first position and orientation is determined to be correct, capturing a second image of the object from a second imaging direction different from the first imaging direction using the camera, converting the first position and orientation in accordance with a change from the first imaging direction to the second imaging direction to calculate a second position and orientation, and creating teacher data by assigning the second position and orientation to the second image; (f) changing the actual position and orientation of the object and repeating steps (b) to (e); A method comprising:

2. 10. The method of claim 1, The step (e) (e1) determining a direction of specular reflection from the object based on an irradiation direction of illumination light onto the object and the first position and orientation; (e2) setting the specular reflection direction as the second imaging direction; A method comprising:

3. 10. The method of claim 1, The method, wherein the second shooting direction is set to a direction in which the position and orientation recognized by the pre-trained machine learning model using an image captured from the second shooting direction is not determined to be correct.

4. 4. The method of claim 3, and before steps (b) to (f), a step of performing a process of recognizing the position and orientation of the object using the pre-trained machine learning model, determining whether the position and orientation are correct, and registering a history of the position and orientation determined to be correct multiple times; The method according to claim 1, wherein the second imaging direction is set to a direction that results in a position and orientation for which there is no history determined to be the correct answer.

5. The method according to any one of claims 1 to 4, The step (d) (d1) creating a simulation image of the object in the first position and orientation by simulation; (d2) calculating a reliability score for the first position and posture using the first image and the simulation image; (d3) determining whether the first pose is correct by comparing the confidence score with a threshold; A method comprising:

6. A system for creating training data used to train a machine learning model that recognizes the position and orientation of an object, a camera for photographing the object; A teacher data creation unit that creates the teacher data; Equipped with The teacher data creation unit (a) performing pre-learning of the machine learning model using simulation data of the object; (b) capturing a first image of the object from a first imaging direction using the camera; (c) recognizing a first position and orientation of the object from the first image using the pre-trained machine learning model; (d) a process of determining whether the first position and orientation are correct or not; (e) capturing a second image of the object from a second imaging direction different from the first imaging direction using the camera, calculating a second position and orientation by performing a transformation on the first position and orientation in accordance with a change from the first imaging direction to the second imaging direction, and creating teacher data by assigning the second position and orientation to the second image; (f) changing the actual position and orientation of the object and repeating the steps (b) to (e); To run the system.

7. A computer program that causes a processor to execute a process of creating training data used for training a machine learning model that recognizes the position and orientation of an object, (a) performing pre-learning of the machine learning model using simulation data of the object; (b) capturing a first image of the object from a first imaging direction using a camera; (c) recognizing a first position and orientation of the object from the first image using the pre-trained machine learning model; (d) a process of determining whether the first position and orientation are correct or not; (e) capturing a second image of the object from a second imaging direction different from the first imaging direction using the camera, calculating a second position and orientation by performing a transformation on the first position and orientation in accordance with a change from the first imaging direction to the second imaging direction, and creating teacher data by assigning the second position and orientation to the second image; (f) changing the actual position and orientation of the object and repeating the steps (b) to (e); A computer program that causes the processor to execute the following.

Images