Learning system, estimation system, learning method, estimation method, and program

Abstract

To improve the accuracy of image processing. [Solution] The training data acquisition unit (103) of the learning system (1) includes a training target image and a training reference image as input parts, and acquires training data that includes correct information as a correct part for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object. The first model storage unit (100) stores a first model that calculates a first training target feature and a first training reference feature and outputs first training processing information. The second model storage unit (102) stores a second model that calculates a second training target feature and a second training reference feature and outputs second training processing information based on the first training processing information, the second training target feature, and the second training reference feature. The learning unit (103) performs learning of at least one of the first model and the second model based on the training data.

Claims

1. A training data acquisition unit that includes a training target image representing the object to be trained and a training reference image representing a training reference object as input parts, and acquires training data that includes correct information as a correct part for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object, A first model storage unit stores a first model which calculates a first training target feature of the training target image and a first training reference feature of the training reference image, and outputs first training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training target feature and the first training reference feature, A second model storage unit stores a second model which calculates a second training target feature of the training target image and a second training reference feature of the training reference image, and outputs a second training processing information for processing the training target image so that the training target pose matches the training reference pose, based on the first training processing information, the second training target feature, and the second training reference feature. A learning unit that performs training on at least one of the first model and the second model based on the aforementioned training data, A learning system that includes this.

2. The first model mentioned above is A first computational model that calculates the first training target features based on the training target images and calculates the first training reference features based on the training reference images, A first output model that outputs the first training processing information based on the first training target feature and the first training reference feature, The learning system according to claim 1, including the following:

3. The first computational model is a trained model in which other training objects different from the training target object and the training reference object have been learned, The learning unit performs the learning of the first output model without performing the learning of the first computation model. The learning system according to claim 2.

4. The first model further includes a first encoder that reduces the dimensions of the first trained feature and the first trained reference feature calculated by the first computation model, The first output model outputs the first training processing information based on the first training target feature and the first training reference feature whose dimensions have been reduced by the first encoder. The learning system according to claim 2 or 3.

5. The second model processes the second training target feature based on the first training processing information, and outputs the second training processing information based on the processed second training target feature and the second training reference feature. A learning system according to any one of claims 1 to 3.

6. The second model described above is A second computational model that calculates the second training target feature based on the training target image and calculates the second training reference feature based on the training reference image, A second output model that outputs the second training processing information based on the first training processing information, the second training target features, and the second training reference features, A learning system according to any one of claims 1 to 3, including the following:

7. The second computational model includes a plurality of layers that compute the second training target feature and the second training reference feature, The second output model calculates, layer by layer, an intermediate second training processing information, which is the second intermediate training processing information, based on the first training processing information and the second training target features and second training reference features calculated in each of the plurality of layers, and outputs the final second training processing information, which is the second final training processing information. The learning system according to claim 6.

8. The aforementioned training data includes, as the correct answer information, correct machining information relating to the correct machining process, The learning unit calculates the machining loss based on the second training machining information and the correct machining information, and performs learning on at least one of the first model and the second model based on the machining loss. A learning system according to any one of claims 1 to 3.

9. The training data includes, as the correct answer information, correct answer image information relating to the processed training target image that is the correct answer. The learning unit processes the training target image based on the second training processing information, calculates the image loss based on the processed training target image and the ground truth image information, and performs training on at least one of the first model and the second model based on the image loss. A learning system according to any one of claims 1 to 3.

10. The training data includes, as the correct answer information, correct answer correspondence information relating to the correspondence between each pixel of the training target image and each pixel of the processed training target image that represents the correct answer. The learning unit processes the training target image based on the second training processing information, obtains training correspondence information relating to the correspondence between the training target image before processing and the training target image after processing, calculates the correspondence loss based on the training correspondence information and the correct answer correspondence information, and performs training on at least one of the first model and the second model based on the correspondence loss. A learning system according to any one of claims 1 to 3.

11. After learning by the learning unit described in claim 1 is completed, the system includes an estimation unit that processes the estimated target image based on the estimated target image showing the estimated target object, the estimated reference image showing the estimated reference object, the first model described in claim 1, and the second model described in claim 1, so that the estimated target pose of the estimated target object matches the estimated reference pose of the estimated reference object. Estimation system.

12. A training data acquisition step involves acquiring training data that includes, as input, a training target image representing the object to be trained and a training reference image representing a training reference object, and as a ground truth portion, ground truth information for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object. A learning step that performs learning of at least one of the following: a first model that calculates a first training target feature of the training target image and a first training reference feature of the training reference image based on the training data, and outputs first training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training target feature and the first training reference feature; and a second model that calculates a second training target feature of the training target image and a second training reference feature of the training reference image, and outputs second training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training processing information, the second training target feature, and the second training reference feature; Learning methods that include this.

13. After completing the learning steps described in claim 12, the method includes an estimation step of processing the target image so that the target pose of the target object matches the reference pose of the reference object, based on the target image showing the target object, the reference image showing the reference object, the first model described in claim 12, and the second model described in claim 12. Estimation method.

14. A training data acquisition unit that includes a training target image representing the object to be trained and a training reference image representing a training reference object as input parts, and acquires training data that includes correct information as a correct part for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object. A learning unit that performs learning on at least one of the following: a first model that calculates a first training target feature of the training target image and a first training reference feature of the training reference image based on the training data, and outputs first training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training target feature and the first training reference feature; and a second model that calculates a second training target feature of the training target image and a second training reference feature of the training reference image, and outputs second training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training processing information, the second training target feature, and the second training reference feature. A program that makes a computer function.

15. After learning by the learning unit according to claim 14 is completed, an estimation unit processes the estimated target image based on the estimated target image showing the estimated target object, the estimated reference image showing the estimated reference object, the first model according to claim 1, and the second model according to claim 1, so that the estimated target pose of the estimated target object matches the estimated reference pose of the estimated reference object. A program that makes a computer function.

Images