Machine learning method, machine learning system, and program

Abstract

To provide a machine learning method, a machine learning system, and a program which can improve accuracy of a recognizer.SOLUTION: A machine learning method according to the embodiment trains a recognizer 130 by using a composite image, acquires a labeled real image on the basis of a captured image captured by a sensor 200, saves a recognition result of the recognizer in the case where at least a part of a recognition result when the labeled real image is input to the recognizer 130 matches the label, generates an image converter by performing machine learning by using a dataset group including a plurality of datasets in which the real image and the composite image make a pair, generates a labeled real image by converting the composite image into a real image through the image converter, and trains the recognizer 130 on the basis of the labeled real image.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a machine learning method, a machine learning system, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for generating training data used in a discrimination model. In Patent Document 1, a processor performs keystone correction on a background image that does not include a detection target. The processor superimposes a detection target image on the keystone corrected background image to generate a composite image. The processor generates training data based on the composite image and label information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-190950 A Summary of the Invention [Problem to be solved by the invention]

[0004] Recognizers generated by machine learning are producing results. In order to generate a recognizer with high recognition accuracy, it is desirable to efficiently collect training data. Examples of training data include image data from a simulation domain such as CG, and image data from a real domain captured by a camera. Image data from a simulation domain is easy to generate, but it is difficult to improve the recognition accuracy. On the other hand, with real-domain image data, the costs of data collection and labeling are high. A machine learning method that can easily generate a recognizer with high accuracy is desirable.

[0005] The present disclosure has been made to solve such problems, and provides a machine learning method, a machine learning system, and a program that can easily generate a highly accurate recognizer. [Means for solving the problem]

[0006] The machine learning method in this embodiment includes: (1) training a recognizer that recognizes object information including the position and orientation of an object using a synthetic image of the object; (2) acquiring a labeled real image based on an image of the object captured by an image sensor; (3) storing the real image and the recognition result of the recognizer when at least a part of the recognition result when the labeled real image is input to the recognizer matches the label; (4) generating a synthetic image using the recognition result of the recognizer, thereby generating a dataset that pairs the real image and the synthetic image; (5) performing machine learning using a dataset group including a plurality of the datasets, generating an image converter that converts the synthetic image into a real image; (6) generating a labeled real image by the image converter converting the synthetic image into a real image; and (7) training the recognizer based on the labeled real image.

[0007] In the above machine learning method, the image converter may be a machine learning model capable of converting between the first domain and the second domain, with the real image being a first domain and the synthetic image being a second domain.

[0008] In the above machine learning method, the recognizer may be retrained by repeating the steps (2) to (6) until the recognizer reaches a desired performance.

[0009] In the above machine learning method, (4) uses a synthetic image generator that can generate a synthetic image by changing the position and orientation of an object, and the synthetic image generator may generate a synthetic image of the object so that it matches the position and orientation obtained by the recognition result in (2).

[0010] In the above machine learning method, in (1), a synthetic image generated by the synthetic image generator may be used.

[0011] In the above machine learning method, in (6), a synthetic image of an object having a position and orientation different from the position and orientation recognized when the captured image is input to the recognizer as a labeled real image may be converted into a real image.

[0012] In the above machine learning method, the label of the captured image may be the number of objects or the number of each object.

[0013] The machine learning system according to the present embodiment is a machine learning system having at least one processor, in which the processor (1) uses a synthetic image of an object to train a recognizer that recognizes object information including the position and orientation of the object, (2) acquires a labeled real image based on an image of the object captured by an image sensor, (3) saves the real image and the recognition result of the recognizer when at least a part of the recognition result when the labeled real image is input to the recognizer matches the label, (4) generates a synthetic image using the recognition result of the recognizer to generate a dataset that pairs the real image and the synthetic image, (5) performs machine learning using a dataset group including a plurality of the datasets to generate an image converter that converts the synthetic image into a real image, (6) the image converter converts the synthetic image into a real image to generate a labeled real image, and (7) trains the recognizer based on the labeled real image.

[0014] In the above machine learning system, the image converter may be a machine learning model capable of converting between the first domain and the second domain, with the real image being a first domain and the synthetic image being a second domain.

[0015] In the above machine learning system, the recognizer may be retrained by repeating the processes (2) to (6) until the recognizer reaches a desired performance.

[0016] In the above machine learning system, (4) uses a synthetic image generator that can change the position and orientation of an object to generate a synthetic image. The synthetic image generator may generate a synthetic image of the object so that it matches the position and orientation obtained by the recognition result in (2).

[0017] In the above machine learning system, in (1), a synthetic image generated by the synthetic image generator may be used.

[0018] In the above machine learning system, in (6), a synthetic image of an object having a position and orientation different from the position and orientation recognized when the captured image is input to the recognizer as a labeled real image may be converted into a real image.

[0019] In the above machine learning method, the label of the captured image may be the number of objects or the number of each object.

[0020] A program according to this embodiment is a program for causing a computer to execute a machine learning method, the machine learning method including: (1) using a synthetic image of an object, training a recognizer that recognizes object information including a position and orientation of the object; (2) acquiring a labeled real image based on an image of the object captured by an image sensor; (3) storing the real image and the recognition result of the recognizer when at least a part of the recognition result when the labeled real image is input to the recognizer matches the label; (4) generating a synthetic image using the recognition result of the recognizer, thereby generating a dataset that pairs the real image and the synthetic image; (5) performing machine learning using a dataset group including a plurality of the datasets, generating an image converter that converts the synthetic image into a real image; (6) generating a labeled real image by the image converter converting the synthetic image into a real image; and (7) training the recognizer based on the labeled real image.

[0021] The present disclosure provides a machine learning method, a machine learning system, and a program that can easily generate a highly accurate recognizer. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram illustrating a schematic system configuration. [Diagram 2] FIG. 2 is a diagram illustrating an example of a captured image. [Diagram 3] FIG. 13 is a diagram for explaining domain transformation by an image transformer. [Figure 4] 4 is a flowchart showing a learning method according to the present embodiment. [Diagram 5] FIG. 2 is a block diagram showing a hardware configuration of the processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems.

[0024] A machine learning system and method according to the present embodiment will be described with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a system 1. The system 1 includes a processing device 100, a sensor 200, and a driving mechanism 300.

[0025] The learning system 1 is a system for generating the recognizer 130 by machine learning (also simply referred to as learning). The recognizer 130 recognizes the category information, number, position and orientation of an object based on a captured image of the object. That is, the recognizer 130 takes the captured image of the object as input data and outputs a recognition result. The recognition result of the recognizer 130 is the category information, number information, position and orientation information of the object. The recognition result of the object is information that becomes a label, which will be described later. The recognizer 130 is a machine learning model generated by a machine learning method such as deep learning.

[0026] The category information of the object is, for example, information indicating the name or type of the object. In this embodiment, as shown in Fig. 2, a plastic bottle, a glass drink container (bottle), an instant noodle container, and a box-shaped paper drink container (paper carton) are used as the objects O1 to O4. The category information of an object is information indicating the object name of each object, etc. For example, the category information of the object O1 is a plastic bottle indicating the object name.

[0027] The recognizer 130 recognizes category information of objects in the captured image. Furthermore, the recognizer 130 recognizes the number of objects O1 to O4 included in the captured image. For example, the recognizer 130 may recognize the number of objects for each category. The recognizer 130 recognizes the category information of objects in the captured image. O The recognizer 130 recognizes the position and orientation of the object 4. The position and orientation is, for example, information on six degrees of freedom, including XYZ three-dimensional coordinates and roll angle, pitch angle, and yaw angle. The recognizer 130 outputs the recognition result of the object included in the captured image.

[0028] The sensor 200 is a measuring instrument for measuring an object. The sensor 200 is an image sensor or a camera, and captures an image of the object. For example, the sensor 200 may be an optical sensor such as a charge coupled device (CCD) camera or a complementary metal oxide semiconductor (CMOS) image sensor. Specifically, the sensor 200 is a visible light camera that detects visible light. The sensor 200 is also an RGB camera equipped with RGB pixels, and captures a color image. The sensor 200 outputs the captured image to the processing device 100.

[0029] An example of an image is shown in Fig. 2. The actual image in Fig. 2 is an example of a captured image. Here, the sensor 200 captures an image of a case C that contains objects from above. Four objects O1 to O4 are contained in the case C.

[0030] The driving mechanism 300 has an actuator for changing the position and orientation of an object relative to the sensor 200. The driving mechanism 300 has, for example, a rocking mechanism for rocking the case C. The driving mechanism 300 rocks the case C to change the positions and orientations of the objects O1 to O4. The driving mechanism 300 is not limited to a rocking mechanism for rocking the case C. For example, the driving mechanism 300 may be a robot arm that grips or adsorbs an object. Alternatively, the driving mechanism 300 may be something like a turntable. Furthermore, the driving mechanism 300 may change the position and orientation of the sensor 200. In other words, the driving mechanism 300 may have any configuration as long as it can change the relative position and orientation of the object relative to the sensor 200.

[0031] The sensor 200 captures a plurality of captured images of an object. For example, the sensor 200 captures an image of the object before and after the driving mechanism 300 changes the position and orientation of the object. In the plurality of captured images, the position and orientation of the object with respect to the sensor 200 are different. Alternatively, the sensor 200 may capture images by changing the number or category of objects. For example, in FIG. 2, the object contained in case C may be replaced with a different object. Alternatively, an object may be added to case C, or an object may be removed from case C. Also, the number of objects may be changed. In other words, the sensor 200 captures images of the inside of case C by changing the information contained in the label.

[0032] The processing device 100 is an information processing device of a personal computer. For example, the processing device 100 includes a memory, a processor, various interfaces, an input device, an output device, a monitor, and the like. The processor of the processing device 100 executes a program stored in the memory to perform processing described below. Furthermore, the processing device 100 is an information processing device capable of wireless or wired communication.

[0033] The processing device 100 includes a synthetic image generator 110, a first training unit 120, a recognizer 130, an image converter 140, an image data acquisition unit 150, a second training unit 160, a memory unit 170, and a judgment unit 180.

[0034] The composite image generator 110 generates a composite image from object data relating to an object. The composite image is a CG image or a rendering image generated from a three-dimensional model of the object or the like. For example, the three-dimensional model has three-dimensional shape data of the object surface and RGB data. The three-dimensional shape data indicates the surface shape of the object. The RGB data includes information such as the color, pattern, and shading of the surfaces of the objects O1 to O4. In the three-dimensional model, the gradation of the RGB data is associated with the three-dimensional coordinates of the surface shape.

[0035] The composite image generator 110 generates a composite image using a three-dimensional model of each object. The composite image generator 110 is, for example, a renderer that performs rendering using data of the three-dimensional model. Since the composite image is an image in a simulation domain, the composite image generator 110 can generate a large number of composite images. Furthermore, the composite image generator 110 can automatically perform labeling of the composite image. That is, the composite image generator 110 can assign information such as position and orientation information, category information, and number information of objects in the composite image as labels to the composite image. The composite image may be an image that includes one or more objects.

[0036] The composite image generator 110 generates a plurality of composite images by changing information included in the label. For example, different composite images are generated by changing at least one of the number, category, and position and orientation of the objects in the case. The composite image generator 110 can generate a large number of composite images by randomly changing the position and orientation information, category information, number information, and the like. The composite image generator 110 can generate a composite image by changing the position and orientation of the objects. The composite image generator 110 can generate a composite image by changing the number of objects. The composite image generator 110 can also generate a composite image by changing the category of the objects.

[0037] The label can be information output as a recognition result of the recognizer 130. The label can also be information necessary for generating a synthetic image. That is, the synthetic image generator 110 can generate a synthetic image based on information given as a label. Then, a different synthetic image is generated by changing at least one of the number information, the category information, and the position and orientation information.

[0038] The first training unit 120 trains the recognizer 130 using the synthetic image as learning data. The synthetic image is given category information, number information, position and orientation information, etc. as labels. That is, the first training unit 120 performs supervised learning using the category information, number information, position and orientation information, etc. given to the synthetic image as teacher data (also called correct answer label or correct answer data). In this way, the first training unit 120 performs machine learning of the recognizer 130. Here, in the first training unit 120, only the synthetic image is used as learning data. That is, the first training unit 120 performs machine learning without using the captured image. As a result, the parameters of the machine learning model are updated. That is, the parameters are tuned to optimize the network.

[0039] The image data acquisition unit 150 acquires image data of the captured image captured by the sensor 200. Note that the driving mechanism 300 changes the position and orientation of the object. Therefore, the image data acquisition unit 150 acquires image data of a plurality of captured images with different position and orientation. The image data acquisition unit 150 acquires a labeled real image based on the captured image of the object captured by the sensor 200. Alternatively, the image data acquisition unit 150 acquires image data of captured images of different categories or numbers of objects included in the case C.

[0040] The label attached to the captured image (actual image) includes one or more of the recognition results of the recognizer 130. For example, the label of the captured image includes category information and number information. Here, the labels indicate the categories of objects contained in the case C and the number of objects per category. The category information attached as the label of the actual image (captured image) shown in FIG. 2 is PET bottle, glass beverage container (bottle), instant ramen container, and box-shaped paper beverage container (paper carton), and the number information is 1 for each.

[0041] Furthermore, the label of the captured image does not include position and orientation information. Since the label of the captured image may be only a part of the information included in the recognition result, the labeling cost can be reduced. For example, the label of the captured image may be only the number of objects. The label of the captured image may be only the information used for the determination by the determination unit 180 described later.

[0042] The recognizer 130 performs a recognition process on the captured image acquired by the image data acquisition unit 150. That is, the recognizer 130 performs the recognition process using image data of the captured image captured by the sensor 200 as input. In this way, the recognizer 130 infers category information, number information, and position and orientation information of objects included in the captured image. The recognizer 130 outputs the recognition result including the category information, number information, and position and orientation information of the objects to the determination unit 180.

[0043] Judgment section 180 teeth, The determination unit 180 determines whether at least a part of the recognition result of the recognizer 130 matches the label of the captured image. Here, the determination unit 180 determines whether the number information of the recognition result matches the number information of the objects included as the label of the captured image. The determination unit 180 may include at least one of the object category, type, number per category, number per object, and total number of objects. The user can predetermine one or more pieces of information used for the determination by the determination unit 180. For example, the determination unit 180 determines whether the label matches the recognition result for both the object category and the number per category. Alternatively, the determination unit 180 determines whether the label matches the recognition result for the object category.

[0044] If the recognition result matches the label, the storage unit 170 stores the captured image (real image). If the recognition result matches the label, the storage unit 170 stores the recognition result. The captured image whose label matches the recognition result becomes a real image used for machine learning in the second training unit 160 described later. The storage unit 170 stores the recognition result including the position and orientation information in association with the captured image. If at least a part of the recognition result when a labeled real image is input to the recognizer 130 matches the label, the storage unit 170 saves the real image and the recognition result of the recognizer. The storage unit 170 stores all information of the recognition result. If at least a part of the recognition result matches the label, the storage unit 170 also stores the recognition result for information that does not match. Therefore, the storage unit 170 stores the position and orientation information, the number information, and the category information in association with the real image.

[0045] If the recognition result and the label do not match, the captured image is not stored in the storage unit 170. In other words, the captured image for which the recognition result and the label do not match at all is not used for machine learning in the second training unit 160 described later.

[0046] Next, the synthetic image generator 110 generates a synthetic image based on the recognition result stored in the storage unit 170. That is, the synthetic image generator 110 generates a synthetic image corresponding to the position and orientation information, category information, and number information included in the recognition result. As a result, a data set in which the synthetic image and the real image are paired is generated. Since the synthetic image generator 110 generates a synthetic image according to the recognition result, the data set includes a synthetic image in a state close to the state of the real image. In this way, the synthetic image generator 110 can generate a data set in which a real image in the real domain (first domain) and a synthetic image in the simulation domain (second domain) are paired. The synthetic image generator 110 generates multiple data sets using multiple captured images (real images) and their recognition results.

[0047] The second training unit 160 performs machine learning using a data set group including a plurality of data sets to generate an image converter 140 that converts a synthetic image into a real image. As shown in FIG. 2, the image converter 140 performs domain conversion of an image in a simulation domain into an image in a real domain. That is, the image converter 140 can generate a real image that is closer to a captured image than a synthetic image. The second training unit 160 generates the image converter 140 using a machine learning method such as deep learning. This updates the parameters of the network of the machine learning model that becomes the image converter 140. That is, the parameters are tuned to optimize the network.

[0048] As shown in Fig. 3, the image converter 140 is preferably a machine learning model capable of converting between a simulation domain and a real domain. In Fig. 3, a real image in the real domain is shown on the upper side, and a synthetic image in the simulation domain is shown on the lower side. The synthetic image corresponds to an image generated by the synthetic image generator 110. The image converter 140 receives a synthetic image in the simulation domain as an input, and outputs a real image in the real domain. Alternatively, the image converter 140 receives a real image in the real domain as an input, and outputs a synthetic image in the simulation domain.

[0049] By using a mutually convertible machine learning model, the second training unit 160 can generate a highly accurate machine learning model. For example, the second training unit 160 alternately trains a neural network that converts an actual image in the real domain into a synthetic image in the simulation domain and a neural network that converts a synthetic image in the simulation domain into an actual image in the real domain. The second training unit 160 can build a highly accurate machine learning model. Since training can be performed efficiently, it is possible to generate an actual image that is closer to a captured image.

[0050] As the mutually convertible image converter 140, for example, CycleGAN or DRIT shown below can be used. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (https: / / junyanz.github.io / CycleGAN / ) DRIT++: Diverse Image-to-Image Translation via Disentangled Representations (https: / / arxiv.org / abs / 1905.01270)

[0051] Next, the image converter 140 converts the synthetic image into a real image to generate a labeled real image. Specifically, the synthetic image generator 110 generates a synthetic image of an object with an arbitrary label. The synthetic image generator 110 generates a synthetic image of an arbitrary category, number of objects, and position and orientation. The labels here can be generated randomly. In other words, the synthetic image generator 110 can randomly change the position and orientation information, category information, and number information to generate a synthetic image.

[0052] Then, the image converter 140 performs domain conversion on the synthetic image of the object. This allows a labeled real image to be generated. The labels here are the position and orientation information, category information, and number information used in generating the synthetic image by the synthetic image generator 110. In this way, labeled real images can be generated efficiently. That is, labeled real images can be generated without using captured images by the sensor 200. Therefore, a large number of labeled real images can be generated.

[0053] The third training unit 190 trains the recognizer 130 based on the labeled real images. That is, the third training unit 190 performs machine learning using the labeled real images as learning data. This updates the parameters of the network of the machine learning model. That is, the parameters are tuned to optimize the network.

[0054] Therefore, the recognition accuracy of the recognizer 130 can be improved. In other words, since a real image closer to the captured image can be used as a learning image, a highly accurate recognizer 130 can be easily formed. Here, supervised learning can be performed using the labels attached to the real image as teaching data. Therefore, the recognition accuracy of the recognizer 130 can be further improved.

[0055] The processing device 100 can repeat the above process until the recognition accuracy of the recognizer 130 reaches a desired performance. That is, the second training unit 160, the third training unit 190, etc. repeat training until the recognition result meets a preset standard accuracy. This updates the parameters of the network of the machine learning model. That is, the parameters are tuned to optimize the network.

[0056] With the above configuration, learning data can be acquired more efficiently, and the recognizer 130 can be generated efficiently. The cost of collecting learning data can be reduced. The cost of labeling the captured image captured by the sensor 200 can be further reduced. That is, the image converter 140 generates a labeled real image based on the synthetic image. This makes it easy to generate a large amount of labeled real images.

[0057] It is possible to generate labeled real images without controlling the position and orientation using a high-precision driving mechanism such as a robot. In other words, since there is no need to control the position and orientation of an object, the cost of collecting learning data can be reduced. Since only a portion of the information is required for the label of a captured image, the labeling cost can be reduced. For example, only the category information and number information of objects may be attached to the captured image.

[0058] Since the image converter 140 is generated using real images whose recognition results from the recognizer 130 match the labels, it is possible to generate a highly accurate image converter 140. Since the synthetic image generator 110 and the image converter 140 are used, it is possible to reduce the cost (work time) of acquiring images in various positions and orientations. Since the real images converted by the image converter 140 are labeled, it is possible to easily generate a large number of labeled real images.

[0059] The method according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flow chart showing the learning method.

[0060] First, the processing device 100 trains the recognizer 130 using a composite image of an object (S101). For example, the composite image generator 110 generates a composite image of an object. Here, a plurality of composite images are generated for an object at an arbitrary position and orientation. The composite image generator 110 generates a plurality of composite images with different label information. The composite images are generated, for example, using a CG (Computer Graphics) Graphics ) image, or may be a rendering image obtained from three-dimensional data of the object. The synthetic image generator 110 can automatically assign a label. Since the synthetic image is an image in a simulation domain, the synthetic image generator 110 can automatically perform labeling. Then, the first training unit 120 uses the synthetic image of the object to train the recognizer 130 that recognizes object information including the position and orientation of the object.

[0061] Next, the processing device 100 acquires a labeled real image (S102). For example, the sensor 200 captures an image of a case C that contains an object. Furthermore, the driving mechanism 300 changes the position and orientation of the object. In this way, the sensor 200 can capture images in various positions and orientations. Furthermore, it is possible to change the category and number of objects contained in the case C. This allows the sensor 200 to capture images of various objects. Then, the image data acquisition unit 150 acquires the captured image from the sensor 200. The image data acquisition unit 150 acquires the captured image to which some information is attached as a label as a real image.

[0062] The processing device 100 saves the recognition result and the actual image that matches the label (S103). Specifically, the processing device 100 inputs the captured image to the recognizer 130, which outputs the recognition result. Here, the recognizer 130 outputs the recognition result including the category, number, and position and orientation of the object. The determination unit 180 compares the information (correct answer data) included in the label with the recognition result. Then, the determination unit 180 determines whether or not at least one piece of information included in the label matches the recognition result.

[0063] If at least one of the recognition results matches the label, the processing device 100 stores the captured image in the storage unit 170. The storage unit 170 stores the captured image and the recognition results in association with each other. The captured image is stored in the storage unit 170 as a real image. That is, the real image stored in the storage unit 170 is associated with the recognition results including the object category, the number of objects per category, and the positions and orientations. Here, the storage unit 170 stores a plurality of captured images in association with the recognition results.

[0064] The processing device 100 generates a synthetic image using the recognition result of the recognizer 130, thereby generating a data set in which the real image and the synthetic image are paired (S104). Here, the synthetic image generator 110 generates a synthetic image based on the recognition result output in step S103. The synthetic image generator 110 generates a synthetic image using the label attached to the real image as input. The synthetic image generator 110 generates a synthetic image corresponding to the category, number, and position and orientation of the object included in the recognition result of the captured image. The synthetic image generated in this manner is paired with the real image. The storage unit 170 stores a data set in which the synthetic image and the real image are paired. Since the recognition results are obtained for multiple real images in step S103, the storage unit 170 stores multiple data sets. All information included in the labels of the synthetic image and the real image that are the data set is completely consistent.

[0065] The processing device 100 performs machine learning using a data set group including a plurality of data sets to generate an image converter 140 that converts a synthetic image into a real image (S105). For example, the second training unit 160 performs supervised learning using a synthetic image as input data and a real image as a correct answer label. The second training unit 160 generates a machine learning model that converts a synthetic image into a real image as the image converter 140. A known machine learning model such as a DNN (Deep Neural Network) or a CNN (Convolutional Neural Network) can be used as the image converter 140.

[0066] The synthetic image generator 110 converts the synthetic image into a real image to generate a real image with a label (S106). Therefore, the synthetic image generator 110 generates synthetic images in various states. Then, the image converter 140 converts the synthetic image into a real image similar to the captured image. Here, the synthetic image generator 110 can use data with labels different from those used in S101, S103, and S104. That is, the synthetic image generator 110 may generate a synthetic image in a state different from the real image or synthetic image used in S101, S103, and S104. The synthetic image generator 110 generates a synthetic image using the label of the new data. In this way, real images in various states can be generated.

[0067] The synthetic image generator 110 trains the recognizer 130 based on the labeled real images (S107). That is, the third training unit 190 uses the labeled real images generated in S106 as learning data to train the recognizer 130. In this way, the recognition accuracy of the recognizer 130 can be improved.

[0068] The processing device 100 determines whether the learning is completed (S108). If the learning is not completed (NO in S108), the process returns to step S102, and the recognizer 130 is retrained. If the learning is completed (YES in S108), the process ends.

[0069] For example, the processing device 100 determines whether the recognizer 130 has reached the desired performance. If the recognizer 130 has not reached the desired performance, the processing device 100 repeats the processes of S102 to S107. The sensor 200 captures a new image of the object. Then, the processing device 100 performs machine learning based on the newly acquired captured image. As a result, the parameters of the recognizer 130 and the image converter 140 are updated. Therefore, the processing device 100 tunes the parameters so as to optimize the networks of the recognizer 130 and the image converter 140. The system 1 re-learns the recognizer 130 and the image converter 140. In this way, the processing device 100 can generate a recognizer having a desired performance by machine learning. The desired performance can be predetermined by a user. Alternatively, when the number of iterations reaches a predetermined number, the system 1 may end the learning.

[0070] The second training unit 160 generates the image converter 140 by using real images whose recognition results by the recognizer 130 match the labels. Therefore, it is possible to generate a highly accurate image converter 140. The real images converted by the image converter 140 are labeled with information such as position and orientation, number, and category. Therefore, it is possible to easily generate a large number of labeled real images. In step S102, captured images that become real images can be efficiently acquired.

[0071] The composite image generator 110 and the image converter 140 are trained as separate machine learning models. Since the processing device 100 uses the composite image generator 110 and the image converter 140, the cost (work time) of acquiring images of various positions and orientations can be reduced. Therefore, the number of captured images required for machine learning can be reduced, enabling efficient acquisition of sensor data. The labels of the captured images only require partial information such as the number of objects. This allows the labeling cost to be reduced.

[0072] The above learning method can be implemented by a computer program or hardware. In other words, the processing device 100 executes a predetermined program to function as a learning device or a learning system. FIG. 5 shows an example of the hardware configuration of the processing device 100. The processing device 100 includes a processor 10, a memory 20, and an interface 30. The memory 20 stores programs, various parameters, machine learning models, and the like. The processor 10 、 The computer executes a program stored in the memory 20. The interface 30 transmits data to the sensor 200 and the drive mechanism 300. The interface 30 also receives data from the sensor 200 and the drive mechanism 300.

[0073] The processor 10 of the processing device 100 executes a program, thereby executing the learning method according to this embodiment. The processing device 100 only needs to have at least one processor 10. The one or more processors 10 execute a program stored in a memory, thereby carrying out the above-mentioned processing. The processing device 100 is not limited to being a single physical device, and may be distributed among multiple devices. In other words, the above-mentioned method may be carried out by multiple devices performing distributed processing.

[0074] In addition, various methods using AI (Artificial Intelligence) can be applied to the formation of machine learning models such as the recognizer 130 and the image converter 140. Deep learning using a multi-layered neural network can be applied to the machine learning. Known methods such as supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning can be applied to the machine learning. The system 1 can use a model using perceptron, neocognitron, and connectionism. The system 1 may form a network model such as a CNN, an RNN (recurrent neural network), or an LSTM (Long Short Term Memory) network. A sigmoid function, a softmax function, a step function, a linear function, a nonlinear function, an identity function, or the like can be used as an activation function of the neural network.

[0075] Machine learning using backpropagation (error backpropagation) is applicable. As a learning method, known methods such as representation learning, transfer learning, ensemble learning, and self-learning can be used. System 1 may use a generative adversarial network, a genetic algorithm, or an autoencoder. Of course, System 1 is not limited to the above methods and can use various methods.

[0076] A part or all of the above processes may be executed by a computer program. That is, the control of the processing device 100 is executed by a control computer that configures the processing device 100 executing the program. The above-mentioned program includes a group of instructions (or software code) that causes the computer to perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or the tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium includes electrical, optical, acoustic, or other forms of propagating signals.

[0077] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention. [Explanation of symbols]

[0078] 1 System 10 Processors 20 Memory 30 Interface 100 Processing equipment 110 Synthetic Image Generator 120 1st Training Department 130 Recognizer 140 Image Converter 150 Image data acquisition unit 160 2nd Training Department 170 Storage section 180 Judgment section 190 3rd Training Department 200 Sensors 300 Drive mechanism

Claims

1. (1) training a recognizer that uses synthetic images of objects to recognize object information including the positions, orientations, and number of the objects; (2) acquiring a real image with a label including the number of the objects based on an image of the object captured by an optical image sensor; (3) storing the real image and the recognition result of the recognizer when the number of objects included in the recognition result when the labeled real image is input to the recognizer matches the label; (4) A synthetic image generator capable of generating a synthetic image by changing the position and orientation of an object generates a synthetic image so as to match the position and orientation obtained by the recognition result in (3), thereby generating a dataset in which the real image and the synthetic image are paired; (5) performing machine learning using a set of datasets including a plurality of the datasets to generate an image converter that converts a synthetic image into a real image; (6) The image converter converts the synthetic image into a real image to generate a labeled real image; (7) A machine learning method for training the recognizer based on the labeled real images.

2. The machine learning method according to claim 1 , wherein the image converter is a machine learning model capable of converting between the first domain and the second domain, with the real image being a first domain and the synthetic image being a second domain.

3. 3. The machine learning method according to claim 1, further comprising: repeating the steps (2) to (6) to retrain the recognizer until the recognizer reaches a desired performance.

4. A machine learning method as described in claim 1 or 2, wherein in (2), multiple actual images are obtained by changing the position and orientation of the object relative to the optical image sensor or the number of the objects.

5. The machine learning method according to claim 4, wherein in (1), a synthetic image generated by the synthetic image generator is used.

6. (6) The machine learning method according to claim 1 or 2, in which a synthetic image of an object having a position and orientation different from the position and orientation recognized when the captured image is input to the recognizer as a labeled real image is converted into a real image.

7. The recognizer recognizes category information indicating a type of the object and a number of objects for each category, The labels include a count of the objects in each category; (3) The machine learning method according to claim 1 or 2, further comprising storing the actual image and the recognition result of the recognizer when the number of objects for each category included in the recognition result matches the label.

8. 1. A machine learning system comprising at least one processor, The processor, (1) training a recognizer that uses synthetic images of objects to recognize object information including the positions, orientations, and number of the objects; (2) acquiring a real image with a label including the number of the objects based on an image of the object captured by an optical image sensor; (3) storing the real image and the recognition result of the recognizer when the number of objects included in the recognition result when the labeled real image is input to the recognizer matches the label; (4) A synthetic image generator capable of generating a synthetic image by changing the position and orientation of an object generates a synthetic image so as to match the position and orientation obtained by the recognition result in (3), thereby generating a dataset in which the real image and the synthetic image are paired; (5) performing machine learning using a set of datasets including a plurality of the datasets to generate an image converter that converts a synthetic image into a real image; (6) The image converter converts the synthetic image into a real image to generate a labeled real image; (7) A machine learning system that trains the recognizer based on the labeled real images.

9. The machine learning system according to claim 8 , wherein the image converter is a machine learning model capable of converting between the first domain and the second domain, with the real image being a first domain and the synthetic image being a second domain.

10. 10. The machine learning system according to claim 8, further comprising: a processor configured to retrain the recognizer by repeating the steps (2) to (6) until the recognizer reaches a desired performance.

11. A machine learning system as described in claim 8 or 9, wherein in (2), multiple actual images are acquired by changing the position and orientation of the object relative to the optical image sensor or the number of the objects.

12. The machine learning system of claim 11, wherein in (1), a synthetic image generated by the synthetic image generator is used.

13. (6) The machine learning system according to claim 8 or 9, in which a synthetic image of an object having a position and orientation different from the position and orientation recognized when the captured image is input to the recognizer as a labeled real image is converted into a real image.

14. The recognizer recognizes category information indicating a type of the object and a number of objects for each category, The labels include a count of the objects in each category; (3) The machine learning system according to claim 8 or 9, wherein, when the number of objects for each category included in the recognition result matches the label, the actual image and the recognition result of the recognizer are stored.

15. A program for causing a computer to execute a machine learning method, The machine learning method includes: (1) training a recognizer that uses synthetic images of objects to recognize object information including the positions, orientations, and number of the objects; (2) acquiring a real image with a label including the number of the objects based on an image of the object captured by an optical image sensor; (3) storing the real image and the recognition result of the recognizer when the number of objects included in the recognition result when the labeled real image is input to the recognizer matches the label; (4) A synthetic image generator capable of generating a synthetic image by changing the position and orientation of an object generates a synthetic image so as to match the position and orientation obtained by the recognition result in (3), thereby generating a dataset in which the real image and the synthetic image are paired; (5) performing machine learning using a set of datasets including a plurality of the datasets to generate an image converter that converts a synthetic image into a real image; (6) The image converter converts the synthetic image into a real image to generate a labeled real image; (7) A program for training the recognizer based on the labeled real images.

Images