Machine learning methods, machine learning systems, and computer-readable media

Abstract

The present disclosure relates to a machine learning method, a machine learning system, and a computer-readable medium. The machine learning method according to the present embodiment includes training an identifier using a synthetic image, obtaining a labeled actual image based on a captured image captured by a sensor, saving an identification result of the identifier in a case where at least a part of the identification result in a case where the labeled actual image is input to the identifier coincides with a label, generating an image converter by performing machine learning using a dataset group including a plurality of datasets in which an actual image and a synthetic image are paired, converting the synthetic image into the actual image by the image converter, generating a labeled actual image, and training the identifier based on the labeled actual image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a machine learning method, a machine learning system, and a computer-readable medium. BACKGROUND

[0002] Patent Literature 1 (Japanese Patent Application Publication No. 2020-190950) discloses a method of generating learning data for a recognition model. In Patent Literature 1, a processor performs trapezoidal distortion correction on a background image that does not contain a detection object. The processor generates a composite image by superimposing a detection object image on the background image that has been subjected to trapezoidal distortion correction. The processor generates learning data based on the composite image and label information. SUMMARY

[0003] A recognition model generated through machine learning gives results. In order to generate a recognition model with high recognition accuracy, it is desirable to efficiently collect learning data. As learning data, there are image data of a simulation domain such as CG (Computer Graphic) and / or image data of a real domain captured by a camera. Image data of a simulation domain is easy to generate, but it is difficult to improve recognition accuracy. On the other hand, with regard to image data of a real domain, the cost required to collect data and the cost required to label (tag) are large. It is desirable to have a machine learning method that can easily generate a recognition model with high accuracy.

[0004] The present disclosure is made to solve such a problem, and provides a machine learning method, a machine learning system, and a program that can easily generate a recognition model with high accuracy.

[0005] The machine learning method in the present embodiment includes: (1) training a recognition model that recognizes object information including a position and posture of an object using a composite image of the object; (2) acquiring a labeled actual (real) image based on a captured image of the object captured by an image sensor; (3) in a case where at least a part of a recognition result when the labeled actual image is input to the recognition model coincides with the label, saving the actual image and the recognition result of the recognition model; (4) generating a data set in which the actual image and a composite image are paired by generating the composite image using the recognition result of the recognition model; (5) generating an image converter that converts a composite image into an actual image by performing machine learning using a data set group (set) containing a plurality of the data sets; (6) generating a labeled actual image by converting a composite image into an actual image using the image converter; and (7) training the recognition model based on the labeled actual image.

[0006] In the machine learning method described above, the image converter can be a machine learning model capable of converting the first domain and the second domain into each other with the actual image as the first domain and the synthetic image as the second domain.

[0007] In the machine learning method described above, the identifier can be retrained by repeating the processes of (2) to (6) until the identifier reaches a desired performance.

[0008] In the machine learning method described above, in (4), a synthetic image generator capable of changing the position and posture of the object can be used to generate the synthetic image, and the synthetic image generator can generate the synthetic image of the object so as to coincide with the position and posture obtained based on the recognition result in (2).

[0009] In the machine learning method described above, in (1), the synthetic image generated by the synthetic image generator can be used.

[0010] In the machine learning method described above, in (6), the synthetic image of the object in a position and posture different from the position and posture recognized when the captured image is input to the identifier as the labeled actual image can be converted into an actual image.

[0011] In the machine learning method described above, the label of the captured image can be the number of objects or the number of objects classified by category.

[0012] The machine learning system according to the present embodiment is a machine learning system including at least one processor configured to: (1) train an identifier that recognizes object information including a position and posture of an object using a synthetic image of the object; (2) acquire a labeled actual image based on a captured image of the object captured by an image sensor; (3) save the actual image and a recognition result of the identifier in a case where at least a part of the recognition result in a case where the labeled actual image is input to the identifier coincides with the label; (4) generate a data set in which the actual image and a synthetic image are paired by generating the synthetic image using the recognition result of the identifier; (5) generate an image converter that converts the synthetic image into the actual image by performing machine learning using a data set group including a plurality of the data sets; (6) generate a labeled actual image by converting the synthetic image into the actual image using the image converter; and (7) train the identifier based on the labeled actual image.

[0013] In the machine learning system described above, the image converter can be a machine learning model capable of converting the first domain and the second domain into each other with the actual image as the first domain and the synthetic image as the second domain.

[0014] In the machine learning system described above, the identifier can be retrained by repeating the processes of (2) to (6) until the identifier reaches a desired performance.

[0015] In the machine learning system described above, in (4), a synthetic image generator that generates a synthetic image by changing the position and posture of the object can be used, and the synthetic image generator generates a synthetic image of the object so as to coincide with the position and posture obtained from the recognition result in (2).

[0016] In the machine learning system described above, in (1), a synthetic image generated by the synthetic image generator can be used.

[0017] In the machine learning system described above, in (6), a synthetic image of the object of a position and posture different from the position and posture recognized when the captured image is input to the identifier as a labeled actual image can be converted into an actual image.

[0018] In the machine learning system described above, the label of the captured image can be the number of objects or the number of objects classified by category.

[0019] The computer-readable medium according to the present embodiment is a computer-readable medium storing a program that causes a computer to execute a machine learning method including: (1) training an identifier that recognizes object information including a position and posture of an object using a synthetic image of the object; (2) obtaining a labeled actual image based on a captured image of the object captured by an image sensor; (3) saving the actual image and a recognition result of the identifier in a case where at least a part of the recognition result when the labeled actual image is input to the identifier coincides with the label; (4) generating a data set in which the actual image and a synthetic image are paired by generating the synthetic image using the recognition result of the identifier; (5) generating an image converter that converts a synthetic image into an actual image by performing machine learning using a data set group including a plurality of the data sets; (6) generating a labeled actual image by converting a synthetic image into an actual image using the image converter; and (7) training the identifier based on the labeled actual image.

[0020] According to the present disclosure, a machine learning method, a machine learning system, and a program that can easily generate a high-precision identifier can be provided.

[0021] The above and other objects, features and advantages of the present disclosure will be more fully understood from the following detailed description taken in conjunction with the accompanying drawings, which are shown at a merely illustrative scale, and therefore are not intended to limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a block diagram schematically showing the system configuration.

[0023] Figure 2 is a diagram schematically showing an example of a captured image.

[0024] Figure 3 is a diagram for explaining a domain conversion by an image converter.

[0025] Figure 4 is a flowchart showing a learning method to which the present embodiment relates.

[0026] Figure 5 is a block diagram showing a hardware configuration of a processing device. DETAILED DESCRIPTION

[0027] Hereinafter, the present application will be described through embodiments of the application, but the application relating to the claims is not limited to the following embodiments. In addition, not all the configurations described in the embodiments are necessary as means for solving the problem.

[0028] With reference to the drawings, a machine learning system and a method to which the present embodiment relates will be described. Figure 1 is a block diagram showing the configuration of the system 1. The system 1 is provided with a processing device 100, a sensor 200, and a driving mechanism 300.

[0029] The learning system 1 is a system for generating an identifier 130 by machine learning (also simply referred to as learning). The identifier 130 identifies the class information, the number, the position and posture, and the like of an object based on a captured image obtained by capturing the object. That is, the identifier 130 takes the captured image of the object as input data and outputs the recognition result. The recognition result of the identifier 130 is the class information, the number information, the position and posture information, and the like of the object. The recognition result of the object is information that becomes a label described later. The identifier 130 becomes a machine learning model generated by a machine learning method such as deep learning.

[0030] The class information of the object is, for example, information indicating the object name and / or the kind of the object. In the present embodiment, as the objects O1 to O4, a plastic bottle, a glass container (bottle) of a beverage, a container of instant noodles, and a box-type paper container (carton) of a beverage are used. The class information of the object is information indicating the object name and the like of each object. For example, the class information of the object O1 is a plastic bottle and the like indicating the object name. Figure 2

[0031] ​The recognizer 130 recognizes the class information of the objects in the captured image. Further, the recognizer 130 recognizes the number of the objects O1 to O4 included in the captured image. For example, the recognizer 130 can also recognize the number of the objects of each class. The recognizer 130 recognizes the position and posture of each of the objects O1 to O4. Further, the position and posture is, for example, information of XYZ three-dimensional coordinates and roll angle, pitch angle, yaw angle of six degrees of freedom. The recognizer 130 outputs the recognition result of the objects included in the captured image.

[0032] The sensor 200 is a measuring instrument for measuring an object. The sensor 200 is an image sensor and / or a camera (camera) that captures an object. For example, as the sensor 200, an optical sensor such as a CCD (Charge Coupled Device) camera, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like can be used. Specifically, the sensor 200 is a visible light camera that detects visible light. In addition, the sensor 200 is an RGB camera that has RGB pixels and captures a color image. The sensor 200 outputs the captured image to the processing device 100.

[0033] Figure 2 The actual image in the indicates an example of a captured image. Here, the sensor 200 captures a box C that accommodates objects from above. In the box C, four objects O1 to O4 are accommodated. Figure 2 The actual image in the indicates an example of a captured image. Here, the sensor 200 captures a box C that accommodates objects from above. In the box C, four objects O1 to O4 are accommodated.

[0034] The driving mechanism 300 has an actuator for changing the position and posture of the object with respect to the sensor 200. The driving mechanism 300 has, for example, a shaking mechanism that shakes the box C. The driving mechanism 300 changes the position and posture of the objects O1 to O4 by shaking the box C. The driving mechanism 300 is not limited to the shaking mechanism of the box C. For example, the driving mechanism 300 can also be a robot arm or the like that grips or sucks the object. Alternatively, the driving mechanism 300 can be a mechanism such as a turn table. Further, the driving mechanism 300 can also be a mechanism that changes the position and posture of the sensor 200. That is, the driving mechanism 300 can be any structure as long as it can change the relative position and posture of the object with respect to the sensor 200.

[0035] The sensor 200 captures a plurality of captured images of the object. For example, the sensor 200 captures the object before and after the driving mechanism 300 changes the position and posture of the object. In the plurality of captured images, the position and posture of the object with respect to the sensor 200 is different. Alternatively, the number and / or the class of the object can be changed, and the sensor 200 can capture the object image. For example, in the case of the box C, the sensor 200 can capture the object image in which the number of the objects is changed from four to three. Figure 2In the middle, the object put in the box C can also be exchanged for another object. Alternatively, an object can also be added to the box C, and an object can also be removed from the box C. In addition, the number of objects can also be changed. That is, the information contained in the label is changed, and the sensor 200 photographs the inside of the box C.

[0036] The processing device 100 is an information processing device of a personal computer. For example, the processing device 100 is provided with a memory, a processor, various interfaces, input devices, output devices, a monitor, and the like. The processor of the processing device 100 executes a program stored in the memory, thereby performing the processing described later. Further, the processing device 100 is an information processing device capable of communicating in a wireless or wired manner.

[0037] The processing device 100 is provided with a synthetic image generator 110, a first training unit 120, an identifier 130, an image converter 140, an image data acquisition unit 150, a second training unit 160, a storage unit 170, and a determination unit 180.

[0038] The synthetic image generator 110 generates a synthetic image from object data related to an object. The synthetic image is a CG image and / or a rendering image generated from a three-dimensional model or the like of the object. For example, the three-dimensional model has three-dimensional shape data and RGB data of the surface of the object. The three-dimensional shape data indicates the surface shape of the object. The RGB data contains information of the color, pattern, shade, and the like of the surface of the objects O1 to O4. In the three-dimensional model, the gray scale of the RGB data is associated with the three-dimensional coordinates of the surface shape.

[0039] The synthetic image generator 110 generates a synthetic image using the three-dimensional model of each object. The synthetic image generator 110 is, for example, a renderer that performs rendering using the data of the three-dimensional model. The synthetic image is an image in a simulation domain, and thus the synthetic image generator 110 can generate a large number of synthetic images. Further, the synthetic image generator 110 can automatically perform labeling of the synthetic images. That is, the synthetic image generator 110 can attach information of the position and posture information, class information, number information, and the like of the object in the synthetic image to the synthetic image as a label. The synthetic image can be an image containing one or more objects.

[0040] The synthetic image generator 110 changes information included in the label, and generates a plurality of synthetic images. For example, different synthetic images are generated by changing at least one of the number, the category, and the position and posture of the object in the box. The synthetic image generator 110 can generate a large number of synthetic images by randomly changing the position and posture information, the category information, the number information, and the like. The synthetic image generator 110 can generate a synthetic image by changing the position and posture of the object. The synthetic image generator 110 can generate a synthetic image by changing the number of the object. In addition, the synthetic image generator 110 can generate a synthetic image by changing the category of the object.

[0041] The label can be set as information output as a result of recognition by the recognizer 130. In addition, the label can be set as information required to generate a synthetic image. That is, the synthetic image generator 110 can generate a synthetic image based on information given as a label. Also, different synthetic images are generated by changing at least one of the number information, the category information, and the position and posture information.

[0042] The first training unit 120 trains the recognizer 130 using a synthetic image as learning data. The category information, the number information, the position and posture information, and the like are attached to the synthetic image as a label. That is, the first training unit 120 uses the category information, the number information, the position and posture information, and the like attached to the synthetic image as teacher data (also referred to as correct label or correct data), and performs supervised learning. In this way, the first training unit 120 performs machine learning of the recognizer 130. Here, in the first training unit 120, only a synthetic image is used as learning data. That is, the first training unit 120 does not perform machine learning using a captured image. As a result, the parameters of the machine learning model are updated. That is, the parameters are tuned (adjusted) so that the network is optimized.

[0043] The image data acquisition unit 150 acquires image data of a captured image captured by the sensor 200. In addition, the driving mechanism 300 changes the position and posture of the object. As a result, the image data acquisition unit 150 acquires image data of a plurality of captured images different in position and posture. The image data acquisition unit 150 acquires a labeled actual image based on a captured image of an object captured by the sensor 200. Alternatively, the image data acquisition unit 150 acquires image data of a captured image different in category and / or number of objects contained in the box C.

[0044] The label attached to the captured image (actual image) includes one or more of the recognition results of the recognizer 130. For example, the label of the captured image includes the category information and the number information. Here, the category of the object accommodated in the box C and the number of each category become the label. As Figure 2The category information attached to the label of the actual image (captured image) shown is a plastic bottle, a glass container (bottle) for a beverage, a container for instant noodles, and a box-type paper container (carton) for a beverage, and the number information is 1 each.

[0045] In addition, the label of the captured image does not include the position and posture information. The label of the captured image can be only a part of the information included in the recognition result, and thus the cost of labeling can be reduced. For example, the label of the captured image can be only the number information of the objects. The label of the captured image can be only information used in the determination by the determination section 180 described later.

[0046] The recognizer 130 performs recognition processing on the captured image acquired by the image data acquisition section 150. That is, the recognizer 130 performs recognition processing with the image data of the captured image photographed by the sensor 200 as input. Thereby, the recognizer 130 infers the category information, the number information, and the position and posture information of the objects included in the captured image. The recognizer 130 outputs the recognition result including the category information, the number information, and the position and posture information of the objects to the determination section 180.

[0047] The determination section 180 determines whether at least a part of the recognition result of the recognizer 130 coincides with the label of the captured image. Here, the determination section 180 determines whether the number information of the recognition result coincides with the number information of the objects included in the label of the captured image. The determination section 180 can include at least one of the category of the objects, the kind, the number of each category, the number of each object, and the total number of the objects. The user can determine one or more pieces of information used for the determination by the determination section 180 in advance. For example, the determination section 180 determines whether the label and the recognition result coincide with each other with respect to both the category of the objects and the number of each category. Alternatively, the determination section 180 determines whether the label and the recognition result coincide with each other with respect to the category of the objects.

[0048] In a case where the recognition result coincides with the label, the storage section 170 stores the captured image (actual image). In a case where the recognition result coincides with the label, the storage section 170 stores the recognition result. The captured image whose recognition result coincides with the label becomes an actual image used for machine learning in the second training section 160 described later. The storage section 170 stores the recognition result including the position and posture information in association with the captured image. In a case where at least a part of the recognition result coincides with the label in a case where the labeled actual image is input to the recognizer 130, the storage section 170 stores the actual image and the recognition result of the recognizer. The storage section 170 stores all the information of the recognition result. In a case where at least a part of the recognition result coincides with the label, the storage section 170 stores the recognition result with respect to the information that does not coincide. Thus, the storage section 170 stores the position and posture information, the number information, and the category information in association with the actual image.

[0049] In a case where the recognition result does not coincide with the label, the storage section 170 does not store the captured image. That is, regarding the captured image whose recognition result does not coincide with the label at all, the captured image is not used for machine learning in the second training section 160 described later.

[0050] Next, the synthetic image generator 110 generates a synthetic image based on the recognition result stored in the storage section 170. That is, the synthetic image generator 110 generates a synthetic image corresponding to the position and posture information, the category information, and the number information included in the recognition result. Thus, a data set in which the synthetic image and the actual image are paired is generated. Since the synthetic image generator 110 generates a synthetic image corresponding to the recognition result, the synthetic image having a state close to that of the actual image is included in the data set. In this way, the synthetic image generator 110 can generate a data set in which the actual image of the real domain (the first domain) and the synthetic image of the simulation domain (the second domain) are paired. The synthetic image generator 110 generates a plurality of data sets using a plurality of captured images (actual images) and their recognition results.

[0051] The second training section 160 generates the image converter 140 that converts a synthetic image into an actual image by performing machine learning using a data set group including a plurality of data sets. As Figure 2 indicated, the image converter 140 performs domain transformation of an image of the simulation domain to obtain an image of the real domain. That is, the image converter 140 can generate an actual image closer to a captured image than a synthetic image. The second training section 160 generates the image converter 140 using a machine learning method such as deep learning. Thus, the network parameters of the machine learning model of the image converter 140 are updated. That is, the parameters are tuned so that the network is optimized.

[0052] As Figure 3 indicated, the image converter 140 is preferably a machine learning model capable of transforming the simulation domain and the real domain into each other. In Figure 3 , the upper side indicates an actual image as the real domain, and the lower side indicates a synthetic image as the simulation domain. The synthetic image corresponds to an image generated by the synthetic image generator 110. The image converter 140 outputs an actual image of the real domain as an output from a synthetic image of the simulation domain as an input. Alternatively, the image converter 140 outputs a synthetic image of the simulation domain as an output from an actual image of the real domain as an input.

[0053] By using the machine learning model capable of converting to each other, the 2nd training unit 160 can generate a machine learning model with high accuracy. For example, the 2nd training unit 160 alternately and repeatedly trains a neural network that converts an actual image of a real domain to a synthetic image of a simulation domain, and a neural network that converts a synthetic image of a simulation domain to an actual image of a real domain. The 2nd training unit 160 can construct a machine learning model with high accuracy. Since training can be performed efficiently, an actual image closer to a captured image can be generated.

[0054] As the image converter 140 capable of converting to each other, for example, CycleGAN and DRIT shown below can be used.

[0055] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (https: / / junyanz.github.io / CycleGAN / )

[0056] DRIT++: Diverse Image-to-Image Translation via Disentangled Representations (https: / / arxiv.org / abs / 1905.01270)

[0057] Next, the synthetic image is converted to an actual image by the image converter 140, and a labeled actual image is generated. Specifically, the synthetic image generator 110 generates a synthetic image of an object under an arbitrary label. The synthetic image generator 110 generates a synthetic image of an object of an arbitrary class, number, position, and posture. The label here can be generated randomly. That is, the synthetic image generator 110 can generate a synthetic image by randomly changing the position and posture information, the class information, and the number information.

[0058] Further, the image converter 140 performs domain conversion on the synthetic image of the object. Thereby, a labeled actual image can be generated. The label here becomes the position and posture information, the class information, and the number information used in the generation of the synthetic image in the synthetic image generator 110. In this way, a labeled actual image can be efficiently generated. That is, a labeled actual image can be generated without using the captured image of the sensor 200. Thereby, a large number of labeled actual images can be generated.

[0059] The 3rd training section 190 trains the recognizer 130 based on the labeled actual image. That is, the 3rd training section 190 performs machine learning using the labeled actual image as learning data. As a result, the network parameters of the machine learning model are updated. That is, the parameters are tuned so that the network is optimized.

[0060] Therefore, it is possible to improve the recognition accuracy of the recognizer 130. That is, since it is possible to use an actual image closer to the captured image as learning image, it is possible to easily generate a high-accuracy recognizer 130. Here, it is possible to perform supervised learning using the label attached to the actual image as teacher data. As a result, it is possible to further improve the recognition accuracy of the recognizer 130.

[0061] The processing device 100 can repeat the above-described processing until the recognition accuracy of the recognizer 130 reaches the desired performance. That is, until the recognition result satisfies the accuracy of the pre-set reference, the 2nd training section 160, the 3rd training section 190, and the like repeatedly perform training. As a result, the network parameters of the machine learning model are updated. That is, the parameters are tuned so that the network is optimized.

[0062] According to the above-described configuration, it is possible to more efficiently obtain learning data, and to efficiently generate the recognizer 130. It is possible to reduce the cost required to collect learning data. It is also possible to reduce the cost required for labeling the captured image captured by the sensor 200. That is, the image converter 140 generates a labeled actual image based on the synthetic image. As a result, it is possible to easily generate a large number of labeled actual images.

[0063] It is possible to generate a labeled actual image even without using a high-precision driving mechanism such as a robot to control the position and posture. That is, since it is not necessary to control the position and posture of the object, it is possible to reduce the cost required to collect learning data. Since the label of the captured image can be only partial information, it is possible to reduce the cost of labeling. For example, it is also possible to attach only the category information and / or the number information of the object to the captured image.

[0064] Since the image converter 140 is generated using an actual image in which the recognition result obtained by the recognizer 130 coincides with the label, it is possible to generate a high-accuracy image converter 140. Since the synthetic image generator 110 and the image converter 140 are used, it is possible to reduce the cost (working time) of obtaining images of various positions and postures. Since the actual image converted by the image converter 140 is labeled, it is possible to easily generate a large number of labeled actual images.

[0065] Using Figure 4 The method related to the present embodiment will be described. Figure 4 is a flowchart showing the learning method.

[0066] First, the processing device 100 trains the recognizer 130 using the synthetic images of the objects (S101). For example, the synthetic image generator 110 generates the synthetic images of the objects. Here, for the objects of arbitrary position postures, a plurality of synthetic images are generated. The synthetic image generator 110 generates a plurality of synthetic images different in the label information. The synthetic image can be, for example, a CG (Computer Graphics) image, or a rendered image obtained from three-dimensional data of the object. The synthetic image generator 110 can automatically attach the label. Since the synthetic image is an image in a simulation domain, the synthetic image generator 110 can automatically perform the labeling. Further, the first training unit 120 trains the recognizer 130 that recognizes the object information including the position posture of the object using the synthetic images of the objects.

[0067] Next, the processing device 100 acquires the labeled actual images (S102). For example, the sensor 200 photographs the box C containing the objects. Further, the driving mechanism 300 changes the position postures of the objects. By so doing, the sensor 200 can perform the photographing in various position postures. Further, the category and the number of the objects contained in the box C can be changed. Thus, the sensor 200 can photograph various objects. Further, the image data acquisition unit 150 acquires the photographed images from the sensor 200. The image data acquisition unit 150 acquires the photographed images to which a part of the information is attached as the label, as the actual images.

[0068] The processing device 100 saves the recognition result coinciding with the label and the actual image (S103). Specifically, the processing device 100 inputs the photographed images to the recognizer 130, and thus the recognizer 130 outputs the recognition result. Here, the recognizer 130 outputs the recognition result including the category, the number, and the position posture of the objects. The determination unit 180 compares the information (correct data) included in the label with the recognition result. Further, the determination unit 180 determines whether at least one of the information included in the label coincides with the recognition result.

[0069] In a case where at least one of the recognition results coincides with the label, the processing device 100 saves the photographed images in the storage unit 170. The storage unit 170 stores the photographed images in association with the recognition result. In the storage unit 170, the photographed images are stored as the actual images. That is, for the actual images saved in the storage unit 170, the recognition result including the category of the objects, the number per object (per category), and the position posture is associated. Here, the storage unit 170 stores a plurality of photographed images in association with the recognition result.

[0070] The processing device 100 generates a synthetic image by using the recognition result of the recognizer 130, and generates a data set in which the actual image and the synthetic image are paired (S104). Here, the synthetic image generator 110 generates a synthetic image based on the recognition result output in step S103. The synthetic image generator 110 generates a synthetic image using a label attached to the actual image as input. The synthetic image generator 110 generates a synthetic image corresponding to the class, the number, the position and posture of the object included in the recognition result of the captured image. The synthetic image thus generated is paired with the actual image. The storage section 170 stores a data set in which the synthetic image and the actual image are paired. The storage section 170 has obtained the recognition result for a plurality of actual images in step S103, and therefore stores a plurality of data sets. The synthetic image and the label included in the information of the actual image that become the data set are completely identical.

[0071] The processing device 100 generates the image converter 140 that converts the synthetic image into the actual image by using the data set group including a plurality of data sets for machine learning (S105). For example, the second training section 160 performs supervised learning in which the synthetic image is used as input data and the actual image is used as a correct label. The second training section 160 generates a machine learning model that converts the synthetic image into the actual image as the image converter 140. As the image converter 140, a publicly known machine learning model such as a DNN (Deep Neural Network), a CNN (Convolutional Neural Network), or the like can be used.

[0072] The synthetic image generator 110 generates a labeled actual image by converting the synthetic image into the actual image (S106). Therefore, the synthetic image generator 110 generates a synthetic image in various states. Also, the image converter 140 converts the synthetic image into an actual image similar to the captured image. Here, the synthetic image generator 110 can use data different from the data of the label used in S101, S103, and S104. That is, the synthetic image generator 110 can also generate a synthetic image in a state different from the actual image or the synthetic image used in S101, S103, and S104. The synthetic image generator 110 generates a synthetic image using a label of new data. By so doing, an actual image in various states can be generated.

[0073] The synthetic image generator 110 trains the recognizer 130 based on the labeled actual image (S107). That is, the third training section 190 trains the recognizer 130 using the labeled actual image generated in S106 as learning data. By so doing, the recognition accuracy of the recognizer 130 can be improved.

[0074] The processing device 100 determines whether or not the learning is completed (S108). In the case where the learning is not completed (S108: No), the processing returns to step S102, and the recognizer 130 is re-learned. In the case where the learning is completed (S108: Yes), the processing is ended.

[0075] For example, the processing device 100 determines whether or not the recognizer 130 has reached a desired performance. In the case where the recognizer 130 has not reached the desired performance, the processing device 100 repeats the processing of S102 to S107. The sensor 200 newly photographs the object. Also, the processing device 100 performs machine learning based on the newly acquired photographed image. Thereby, the parameters of the recognizer 130 and the image converter 140 are updated. Thereby, the processing device 100 tunes the parameters so that the network of the recognizer 130 and the image converter 140 is optimized. The system 1 re-learns (trains) the recognizer 130 and the image converter 140. By so doing, the processing device 100 can generate a recognizer having a desired performance through machine learning. As for the desired performance, the user can determine in advance. Alternatively, the system 1 can end the learning in the case where the iteration reaches a predetermined number of times.

[0076] The 2nd training unit 160 generates the image converter 140 using the actual image in which the recognition result in the recognizer 130 coincides with the label. Therefore, it is possible to generate the image converter 140 with high precision. The actual image converted by the image converter 140 is tagged with information including the position posture, the number, the category, and the like. Thereby, it is possible to easily generate a large number of actual images with tags. In step S102, it is possible to efficiently acquire the photographed image which becomes the actual image.

[0077] The synthetic image generator 110 and the image converter 140 are trained as separate machine learning models. The processing device 100 uses the synthetic image generator 110 and the image converter 140, and therefore it is possible to reduce the cost (working time) of acquiring images of various position postures. Thereby, it is possible to reduce the number of photographed images required for machine learning, and it is possible to efficiently acquire the sensor data. The label of the photographed image can be only a part of information such as the number of objects. Therefore, it is possible to reduce the cost of annotation.

[0078] The above-described learning method can be implemented by a computer program and / or hardware. That is, the processing device 100 functions as a learning device or a learning system by executing a predetermined program. Figure 5 Fig. 1 shows an example of a hardware configuration of the processing device 100. The processing device 100 is provided with a processor 10, a memory 20, and an interface 30, and the like. The memory 20 stores programs and various parameters, machine learning models, and the like. The processor 10 executes the programs stored in the memory 20. The interface 30 transmits data to the sensor 200 and the driving mechanism 300. In addition, the interface 30 receives data from the sensor 200 and the driving mechanism 300.

[0079] The processor 10 of the processing device 100 is capable of executing the learning method according to the present embodiment by executing a program. The processing device 100 has at least one processor 10. Also, the above processing is implemented by one or more processors 10 executing a program stored in a memory. The processing device 100 is not limited to a physically single (one) device, and can be distributed among a plurality of devices. That is, the above method can be executed by a plurality of devices performing distributed processing.

[0080] Further, to form the machine learning model such as the recognizer 130 and the image converter 140, various methods using AI (Artificial Intelligence) can be applied. In machine learning, deep learning using a multi-layer neural network can be applied. As machine learning, publicly known methods such as supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and the like can be applied. The system 1 can use models using a perceptron, a neocognitron, connectionism. The system 1 can also form a network model such as a CNN, an RNN (Recurrent Neural Network), an LSTM (Long Short Term Memory) network, and the like. As an activation function of a neural network, an S-shaped function, a Softmax function, a step function, a linear function, a nonlinear function, an identity function, and the like can be used.

[0081] Machine learning using backpropagation (error backpropagation) can be applied. As a learning method, publicly known methods such as representation learning, transfer learning, ensemble learning, self-learning, and the like can be used. The system 1 can also use a generative adversarial network, a genetic algorithm, an autoencoder. Of course, the system 1 is not limited to the above methods, and various methods can be used.

[0082] Part or all of the above processing can also be executed by a computer program. That is, the above processing of the processing device 100 is executed by a control computer constituting the processing device 100 executing a program. The above program includes a command group (or software code) for causing a computer to perform one or more functions described in the embodiments when read into the computer.

[0083] The program can be stored and provided to a computer using any type of non-transitory computer readable media. The non-transitory computer readable media include any type of tangible storage media. Examples of the non-transitory computer readable media include a magnetic storage media (e.g., a floppy diskette, a magnetic tape, a hard disk drive, etc.), an opto-magnetic storage media (e.g., a magneto-optical disk), a CD-ROM (compact disc read only memory), a CD-R (compact disc recordable), a CD-R / W (compact disc rewritable), and a semiconductor memory (e.g., a mask ROM, a PROM (programmable ROM), an EPROM (erasable PROM), a flash ROM, a RAM (random access memory), etc.). The program can be provided to a computer using any type of transitory computer readable media. Examples of the transitory computer readable media include an electrical signal, an optical signal, and an electromagnetic wave. The transitory computer readable media can provide the program to a computer via a wired communication line (e.g., an electrical wire and an optical fiber) or a wireless communication line.

[0084] In accordance with the present disclosure thus described, it will be obvious to those skilled in the art that many modifications can be made to the embodiments of the present disclosure. Such modifications should not be considered as departing from the spirit and scope of the present disclosure, and all such modifications as would be obvious to those skilled in the art are intended to be included within the scope of the appended claims.

Claims

1. A machine learning method comprising: (1) training an identifier that identifies object information including a position and posture of an object using a first synthetic image of the object; (2) obtaining a first actual image with a first label based on a captured image of the object captured by an image sensor, the first label not including position and posture information; (3) in a case where at least a part of an identification result of the first actual image with the first label input to the identifier coincides with the first label, saving the first actual image and the identification result of the identifier; (4) generating a data set in which the first actual image and a second synthetic image corresponding to information included in the identification result are paired by using the identification result of the identifier; (5) generating an image converter that converts a synthetic image into an actual image by machine learning using a data set group including a plurality of the data sets; (6) generating a second actual image with a second label by converting a third synthetic image into an actual image by the image converter, the label of the third synthetic image being a second label generated at random, the second label including position and posture information, category information, and number information of the object; (7) training the identifier based on the second actual image with the second label.

2. The machine learning method according to claim 1, the image converter is a machine learning model capable of converting a first domain and a second domain into each other with the actual image as the first domain and the synthetic image as the second domain.

3. The machine learning method according to claim 1 or 2, the identifier is relearned by repeating the processes of (2) to (6) until the identifier reaches a desired performance.

4. The machine learning method according to claim 1 or 2, in (4), a synthetic image generator capable of changing a position and posture of an object is used to generate a synthetic image, the synthetic image generator generating a synthetic image of an object so as to coincide with a position and posture obtained from an identification result in (3).

5. The machine learning method according to claim 4, in (1), a synthetic image generated by the synthetic image generator is used.

6. The machine learning method according to claim 1 or 2, in (6), a synthetic image of an object with a position and posture different from a position and posture identified when the captured image is input to the identifier as a labeled actual image is converted into an actual image.

7. The machine learning method according to claim 1 or 2, the label of the captured image is a number of objects or a number of objects classified by category.

8. A machine learning system provided with at least one processor, the processor, (1) training an identifier that identifies object information including a position and posture of an object using a first synthetic image of the object; (2) obtaining a first actual image with a first label based on a captured image of the object captured by an image sensor, the first label not including position and posture information; (3) in a case where at least a part of the recognition result in a case where the first actual image with the first label is input to the recognizer coincides with the first label, saving the first actual image and the recognition result of the recognizer; (4) generating a second synthetic image corresponding to information included in the recognition result by using the recognition result of the recognizer, and generating a data set in which the first actual image and the second synthetic image are paired; (5) generating an image converter that converts a synthetic image into an actual image by machine learning using a data set group including a plurality of the data sets; (6) generating a second actual image with a second label by converting a third synthetic image into an actual image by the image converter, the label of the third synthetic image being a second label generated at random, the second label including position and posture information, category information, and number information of an object; (7) training the recognizer based on the second actual image with the second label.

9. The machine learning system according to claim 8, the image converter is a machine learning model that can convert a first domain and a second domain into each other with the actual image as the first domain and the synthetic image as the second domain.

10. The machine learning system according to claim 8 or 9, the recognizer is re-learned by repeating the processes of (2) to (6) until the recognizer reaches a desired performance.

11. The machine learning system according to claim 8 or 9, in (4), a synthetic image generator that can change the position and posture of an object is used to generate a synthetic image, and the synthetic image generator generates a synthetic image of an object so as to coincide with the position and posture obtained from the recognition result in (3).

12. The machine learning system according to claim 11, in (1), a synthetic image generated by the synthetic image generator is used.

13. The machine learning system according to claim 8 or 9, in (6), a synthetic image of an object with a position and posture different from the position and posture recognized when the captured image is input to the recognizer as a labeled actual image is converted into an actual image.

14. The machine learning system according to claim 8 or 9, the label of the captured image is the number of objects or the number of objects classified by category.

15. A computer-readable medium storing a program that causes a computer to execute a machine learning method, the machine learning method includes: (1) training a recognizer that recognizes object information including the position and posture of an object using a first synthetic image of the object; (2) obtaining a first actual image with a first label based on a captured image of the object captured by an image sensor, the first label not including position and posture information; (3) in a case where at least a part of the recognition result in a case where the first actual image with the first label is input to the recognizer coincides with the first label, saving the first actual image and the recognition result of the recognizer; (4) generating a data set in which the first actual image and a second synthetic image corresponding to information included in the recognition result are paired, by generating the second synthetic image using the recognition result of the recognizer; (5) generating an image converter that converts a synthetic image into an actual image, by machine learning using a data set group including a plurality of the data sets; (6) generating a second actual image with a second label, by converting a third synthetic image into an actual image using the image converter, the label of the third synthetic image being a second label generated at random, the second label including position and posture information, class information, and number information of an object; (7) training the recognizer based on the second actual image with the second label.

Images