Learning system, estimation system, learning method, estimation method, and program

Abstract

To improve the accuracy of image processing. [Solution] The training data acquisition unit (103) of the learning system (1) includes a training target image and a training reference image as input parts, and acquires training data that includes correct information as a correct part for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object. The first model storage unit (100) stores a first model that calculates a first training target feature and a first training reference feature and outputs first training processing information. The second model storage unit (102) stores a second model that calculates a second training target feature and a second training reference feature and outputs second training processing information based on the first training processing information, the second training target feature, and the second training reference feature. The learning unit (103) performs learning of at least one of the first model and the second model based on the training data.

Description

[Technical Field]

[0001] This disclosure relates to a learning system, an estimation system, a learning method, an estimation method, and a program. [Background technology]

[0002] Conventionally, techniques for processing target images that represent a predetermined object are known. For example, Patent Document 1 describes a technique for processing a target image by matching a feature point cloud extracted from a target image that represents a document, which is an example of an object, with a feature point cloud extracted from a sample image that represents a sample document, so that the positional relationship of the feature point clouds in the target image becomes or approaches the positional relationship of the feature point clouds in the sample image. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2020 / 008628 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the technology described in Patent Document 1 requires the extraction of numerous feature points from the target image. Therefore, if sufficient feature points cannot be extracted from the target image, the target image cannot be processed with high accuracy. This also applies to the processing of target images that represent objects other than documents, as described in Patent Document 1. For this reason, conventional technologies have not been able to sufficiently improve the accuracy of target image processing.

[0005] One of the purposes of this disclosure is to improve the accuracy of image processing. [Means for solving the problem]

[0006] The learning system relating to this disclosure includes a training target image representing a training target object and a training reference image representing a training reference object as input parts, a training data acquisition unit that acquires training data including correct information as a correct part for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object, a first training target feature of the training target image and a first training reference feature of the training reference image, and calculates the first training target feature and the first training reference feature so that the training target pose matches the training reference pose The system includes: a first model storage unit that stores a first model that outputs first training processing information for processing the training target image; a second model storage unit that stores a second model that calculates second training target features of the training target image and second training reference features of the training reference image, and outputs second training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training processing information, the second training target features, and the second training reference features; and a learning unit that performs learning of at least one of the first model and the second model based on the training data.

[0007] The estimation system relating to this disclosure includes an estimation unit that, after learning by the learning unit is completed, processes the estimated target image, which shows the estimated target object, an estimated reference image, which shows the estimated reference object, the first model described above, and the second model described above, so that the estimated target pose of the estimated target object matches the estimated reference pose of the estimated reference object. [Effects of the Invention]

[0008] This disclosure can improve the accuracy of image processing. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the hardware configuration of a learning system. [Figure 2] This figure shows an example of the functions realized in the learning system of the first embodiment. [Figure 3] This figure shows an example of a model that is the target of training in the first embodiment. [Figure 4] This figure shows an example of a training database. [Figure 5] This figure shows an example of how the processed training images are acquired. [Figure 6] This figure shows an example of the correspondence between each pixel of the training image before and after processing. [Figure 7] This figure shows an example of the processing performed by the learning system of the first embodiment. [Figure 8] This figure shows an example of the functions realized by the estimation system of the second embodiment. [Figure 9] This figure shows an example of the processing performed by the estimation system of the second embodiment. [Modes for carrying out the invention]

[0010] [1. First Embodiment] A first embodiment, which is an example of an embodiment of the learning system, learning method, and program related to this disclosure, will be described.

[0011] [1-1. Hardware configuration of the learning system] Figure 1 shows an example of the hardware configuration of a learning system. For example, learning system 1 includes a learning terminal 10, a server 20, and a user terminal 30. Each of the learning terminal 10, server 20, and user terminal 30 can be connected to a communication network CN such as the Internet or a LAN. In the example described in the second embodiment below, each of these computers is included in estimation system 2, so in Figure 1, the code for estimation system 2 is indicated in parentheses after the code for learning system 1.

[0012] The learning terminal 10 is a computer that performs the learning described below. For example, the learning terminal 10 is a personal computer, a server computer, a smartphone, or a tablet. The learning terminal 10 includes a control unit 11, a storage unit 12, a communication unit 13, an operation unit 14, and a display unit 15. The control unit 11 includes at least one processor. The storage unit 12 includes at least one of volatile memory such as RAM and non-volatile memory such as flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication. The operation unit 14 is an input device such as a touch panel. The display unit 15 is a liquid crystal display or an organic EL display.

[0013] Server 20 is a server computer. Server 20 includes a control unit 21, a storage unit 22, and a communication unit 23. The hardware configurations of the control unit 21, the storage unit 22, and the communication unit 23 may be the same as those of the control unit 11, the storage unit 12, and the communication unit 13, respectively.

[0014] The user terminal 30 is the user's computer. For example, the user terminal 30 is a personal computer, smartphone, tablet, or wearable device. The user terminal 30 includes a control unit 31, a storage unit 32, a communication unit 33, an operation unit 34, a display unit 35, and a camera unit 36. The hardware configurations of the control unit 31, storage unit 32, communication unit 33, operation unit 34, and display unit 35 may be the same as those of the control unit 11, storage unit 12, communication unit 13, operation unit 14, and display unit 15, respectively. The camera unit 36 ​​includes at least one camera.

[0015] The programs stored in the memory units 12, 22, and 32 may be supplied to the learning terminal 10, server 20, or user terminal 30 via the communication network CN. Furthermore, the learning terminal 10, server 20, or user terminal 30 may include a reading unit for reading computer-readable information storage media (e.g., an optical disc drive or memory card slot), or an input / output unit for inputting and outputting data with external devices (e.g., a USB port). For example, a program stored on the information storage media may be supplied to the learning terminal 10, server 20, or user terminal 30 via the reading unit or input / output unit.

[0016] Furthermore, the hardware configuration of learning system 1 is not limited to the example in Figure 1. Learning system 1 may include at least one computer. For example, learning system 1 may include only the learning terminal 10 and the server 20. In this case, the user terminal 30 is located outside of learning system 1. Learning system 1 may include only the learning terminal 10. In this case, the server 20 and the user terminal 30 are located outside of learning system 1. Learning system 1 may include computers not shown in Figure 1.

[0017] [1-2. Overview of the learning system of the first embodiment] In the first embodiment, the learning system 1 trains a model for processing the target image so that the target pose of the target object in the target image matches the reference pose of the reference object in the reference image. The estimation process using the trained model will be described in the second embodiment below. In the first embodiment, we will describe the process up to the creation of the trained model. First, we will explain the meaning of each term.

[0018] The target image is the image to be processed. Processing refers to image processing performed on the target image. Processing can also be described as shaping or transforming the target image. For example, processing may include moving, rotating, enlarging, reducing, cropping, or a combination thereof. Processing may also include changing pixel values, changing brightness, changing file extensions, or other operations. Image processing called affine transformation is also a type of processing. Changing the arrangement of pixels is also a type of processing. In the first embodiment, the case where affine transformation corresponds to processing is given as an example, but processing is not limited to affine transformation. Processing may include all or part of the above examples.

[0019] The target object is the object shown in the target image. For example, if the target image is a photograph generated by a camera, the target object is all or part of the subject photographed by the camera. If the target image is a scanned image generated by a scanner, the target object is all or part of the object scanned by the scanner. If the target image is a computer graphics (CG) image, the target object is all or part of the object created with CG.

[0020] The object's orientation is at least one of the orientation, shape, and position of the object in the target image. Since a change in the positional relationship between the viewpoint (e.g., camera, scanner, or virtual viewpoint) and the object changes at least one of the object's orientation, shape, and position in the target image, the object's orientation can also be defined as the positional relationship between the viewpoint and the object.

[0021] A reference image is an image showing a reference object in a predetermined pose. This predetermined pose is the desirable pose for the target object in the processed image. The predetermined pose can also be described as a target pose or an appropriate pose. A reference image can also be described as an image referenced during the processing of the target image. A reference image can also be described as an image used as a sample during the processing of the target image.

[0022] A reference object is an object shown in a reference image. For example, if a camera-generated image corresponds to a reference image, the reference object is all or part of the subject captured by the camera. If a scanner-generated scanned image corresponds to a reference image, the reference object is all or part of the object scanned by the scanner. If a computer-generated image (CG) corresponds to a reference image, the reference object is all or part of the object created using CG.

[0023] The reference orientation is at least one of the orientation, shape, and position of the reference object in the reference image. The predetermined orientation described above corresponds to the reference orientation. Since changing the positional relationship between the viewpoint (e.g., camera, scanner, or virtual viewpoint) and the reference object changes at least one of the orientation, shape, and position of the reference object in the reference image, the reference orientation can also be defined as the positional relationship between the viewpoint and the reference object.

[0024] In the first embodiment, the target image, target object, target pose, reference image, reference object, and reference pose during training are referred to as the training target image, training target object, training target pose, training reference image, training reference object, and training reference pose, respectively. The learning system 1 performs training on a model for processing the training target image so that the training target pose of the training target object in the training target image matches the training reference pose of the training reference object in the training reference image. The details of the learning system 1 will be described below.

[0025] [1-3. Functions realized in the learning system of the first embodiment] Figure 2 shows an example of the functions implemented in the learning system 1 of the first embodiment. In the first embodiment, the functions implemented in the learning terminal 10 among the functions implemented in the learning system 1 will be described. For example, the learning terminal 10 includes a first model storage unit 100, a second model storage unit 101, a data storage unit 102, a training data acquisition unit 103, and a learning unit 104. The first model storage unit 100, the second model storage unit 101, and the data storage unit 102 are implemented by the storage unit 12. The training data acquisition unit 103 and the learning unit 104 are implemented by the control unit 11.

[0026] Figure 3 shows an example of a model to be trained in the first embodiment. As shown in Figure 3, the model M to be trained in the first embodiment includes the first model M1 and the second model M2. When the first model M1 and the second model M2 are not specifically distinguished, they are simply referred to as model M. Model M can also be considered a concept that encompasses the first model M1 and the second model M2. (Reference numeral I in Figure 3) t ,I r These are the training image and the training reference image, respectively. Hereafter, we will explain the details of each function in Figure 2 with reference to Figure 3. Note that training image I t The machining may be performed within at least one of the first model M1 and the second model M2, or there may be a separate program for machining. In the first embodiment, the case in which machining is performed by a separate program is given as an example.

[0027] In the first embodiment, the training object and the training reference object are each a logo. The logo may be letters, symbols, numbers, shapes, patterns, colors, or a combination thereof. The logo may represent the name of a service, company, local government, or other organization. The logo may be formed on a physical medium such as a credit card or paper, or it may not be formed on a physical medium. The training object and the training reference object may be any object, and are not limited to a logo. For example, the training object and the training reference object may be letters that are not a logo, all or part of an identification document, all or part of a document other than an identification document, or an icon.

[0028] Furthermore, Model 1 M1 and Model 2 M2 can be used for any purpose. For example, Model 1 M1 and Model 2 M2 may be applied to eKYC (electronic Know Your Customer). If a logo is included in the identity verification document in eKYC, Model 1 M1 and Model 2 M2 may be used to correct the orientation of the logo. For example, Model 1 M1 and Model 2 M2 may be used in possession verification for payment services. If a credit card is used in possession verification, Model 1 M1 and Model 2 M2 may be used to correct the orientation of the logo indicating the card company that issued the credit card. Model 1 M1 and Model 2 M2 may be used to determine whether the logo of a merchant participating in a certain service (e.g., a payment service) is in the correct orientation.

[0029] [First Model Memory Unit] The first model storage unit 100 stores the first model M1. The first model M1 includes a program developed using a machine learning technique. The machine learning technique may be any technique used in the field of image processing. The first model M1 may be developed using supervised learning techniques, unsupervised learning techniques, or a combination thereof. For example, the first model M1 may include a program developed using a neural network, ViT (Vision Transformer), SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), HOG (Histogram of Oriented Gradients), or other techniques.

[0030] For example, the first model M1 includes a program that represents a series of information processing steps for an image input to itself, and parameters referenced by the program. The parameters may be incorporated as part of the program. The parameters of the first model M1 may be any parameters used in machine learning techniques. For example, the parameters of the first model M1 may be weights, biases, or other parameters. The parameters of the first model M1 may be any parameters adopted in the aforementioned neural network and other techniques.

[0031] For example, the first model storage unit 100 stores the first model M1 before training (before training is performed by the training unit 104). The first model M1 before training is the first model M1 with initial parameters. All or part of the first model M1 before training may have undergone some degree of pre-training. When training of the first model M1 is performed, the initial parameters are adjusted. Once training of the first model M1 is complete, the first model storage unit 100 stores the trained first model M1. The first model M1 before training may be overwritten with the trained first model M1, or the trained first model M1 may be stored in the first model storage unit 100 separately from the first model M1 before training.

[0032] In the first embodiment, the first model M1 calculates the first training target feature of the training target image I t and the first training reference feature of the training reference image I r , and based on the first training target feature and the first training reference feature, outputs the first training processing information H t for processing the training target image I g so that the training target pose matches the training reference pose. The program of the first model M1 shows these series of information processes. Among these series of information processes, the parameters of the first model M1 are referred to.

[0033] For example, the first model M1 calculates the first training target feature of the training target image I t based on the parameters of the first model M1. The first training target feature is the feature of the training target image I t calculated by the first model M1. The first training target feature can also be said to be information for the computer to recognize the feature of the training target image I t . The first training target feature may also be called an embedded representation or feature amount of the training target image I t . The first training target feature may be in any form. For example, the first training target feature may be a feature map, a vector, an array, a single numerical value, a combination of multiple numerical values, a matrix, or other forms.

[0034] For example, the first model M1 calculates the first training reference feature of the training reference image I r based on the parameters of the first model M1. The first training reference feature is the feature of the training reference image I r calculated by the first model M1. The first training reference feature can also be said to be information for the computer to recognize the feature of the training reference image I r . The first training reference feature may also be called an embedded representation or feature amount of the training reference image I r . The first training reference feature may be in any form. For example, the first training reference feature may be a feature map, a vector, an array, a single numerical value, a combination of multiple numerical values, a matrix, or other forms.

[0035] For example, the first model M1 generates the first training processed information H based on the parameters of the first model M1, the first training target feature, and the first training reference feature. g Outputs the first training processing information H. g The output may be an output to an external entity of the first model M1 (for example, the second model M2), or it may be an output from one configuration of the first model M1 to another configuration (an output from one layer of the first model M1 to another layer).

[0036] 1st training processing information H g This is the machining information calculated by the first model M1. First training machining information H g These can also be called coefficients (parameters) that are referenced during processing. For example, if the affine transformation corresponds to processing, the transformation coefficients in the affine transformation are the first training processing information H g This corresponds to the first training processing information H. When changing the arrangement of each pixel corresponds to processing, the positional relationship of each pixel before and after the change is the first training processing information H. g This corresponds to the first training processing information H. g This refers to any information that is referenced during processing, and is not limited to these examples. First training processing information H g This may be a translation amount, a rotation amount, a scaling factor, a reduction factor, a trimming range, or a combination thereof.

[0037] In the first embodiment, the first model M1 includes a first calculation model M10, a first encoder E11, and a first output model M12. The first model M1 may include only the first calculation model M10 and the first output model M12, without including the first encoder E11. A first model M1 that includes only the first calculation model M10 and the first output model M12, without including the first encoder E11, is also within the scope of this disclosure.

[0038] The first computational model M10 uses the training target image I t Based on this, calculate the first training target feature and training reference image I rThe first training reference feature is calculated based on this. In the example in Figure 3, two first computational models M10 are shown for illustrative purposes, but in the first embodiment, we will take the example where there is only one first computational model M10. That is, the two first computational models M10 in Figure 3 are the same first computational model M10, of which there is only one.

[0039] The first computational model M10 can be any model that computes the features of the image input to it, and any machine learning method may be used. In the first embodiment, the first computational model M10 is given as an example when it is DINO (Distillation of Novel Object Representations), but the first computational model M10 may be a model developed using other machine learning methods. For example, the first computational model M10 may be a model developed using a neural network method other than DINO, such as ViT, SIFT, SURF, or HOG. The first computational model M10 may also be a model developed using a method called a backbone network.

[0040] The first computational model M10 may be a pre-trained model that has learned other training objects different from the training target object and the training reference object. The other training objects are objects that are subject to pre-training. The other training objects may be any object. For example, the other training objects may be letters, numbers, symbols, figures, or other objects. The other training objects may be objects that represent a general shape. In the first embodiment, DINO has learned general characters, so it is a pre-trained model that has learned characters, which are an example of other training objects. The first computational model M10 has learned the features of training images that represent other training objects. The first computational model M10 may be a model that is publicly available, either for a fee or free of charge, by a third-party organization.

[0041] For example, the first computational model M10 uses the training target image I input to itself. tBased on its own parameters, it calculates the first training target feature. The first computational model M10 uses the training target image I t The first training target feature may be calculated by performing convolution on the first training reference image I input to itself. r Based on its own parameters, the first training reference feature is calculated. The first computational model M10 uses the training reference image I r The first training reference feature may be computed by performing a convolution on the first training target feature and the first training reference feature. Any computation method used in machine learning techniques may be used for the computation of the first computation model M10 to compute the first training target feature and the first training reference feature.

[0042] Note that the first model M1 may include multiple first computational models M10. For example, the first model M1 may include training target image I t The input is training reference image I r The first computational model M10, in which no input is provided (the first computational model M10, which only performs calculations for the first training target feature), and the training reference image I r The input is the training target image I t It may include a first computational model M10 in which no input is provided (a first computational model M10 that only performs the calculation of the first training reference feature). The parameters of these two first computational models M10 may be independent of each other. The first model M1 may include three or more first computational models M10.

[0043] The first encoder E11 reduces the dimensionality of the first target feature and the first trained reference feature computed by the first computational model M10. In the example in Figure 3, two first encoders E11 are shown for illustrative purposes, but in the first embodiment, we will use the example where there is only one first encoder E11. That is, the two first encoders E11 in Figure 3 are the same first encoder E11, of which there is only one. In the example in Figure 3, the arrows extending from the first computational model M10 to the first encoder E11 indicate that the first target feature and the first trained reference feature are input from the first computational model M10 to the first encoder E11.

[0044] Reducing the dimensionality of the first target feature and the first trained reference feature may also reduce the size of the first target feature and the first trained reference feature. For example, reducing the size of a feature map or reducing the dimensionality of a vector is equivalent to reducing dimensionality. The first encoder E11 may utilize any machine learning technique for dimensionality reduction. For example, the first encoder E11 may be a convolutional neural network. The first encoder E11 reduces the dimensionality of the first target feature and the first trained reference feature based on the first target feature and the first trained reference feature input to it, and its own parameters.

[0045] The first encoder E11 may include multiple layers that perform information processing to reduce dimensionality. Each layer of the first encoder E11 may have parameters that are referenced in that layer. Each layer of the first encoder E11 reduces the dimensionality of the first target feature and the first trained reference feature calculated by the layer before it and outputs it to the next layer. The first encoder E11 performs dimensionality reduction processing in multiple layers in succession and outputs the final first target feature and the first trained reference feature. In the example in Figure 3, the first encoder E11 has five layers, but the number of layers in the first encoder E11 can be any number. For example, the number of layers in the first encoder E11 may be 1 to 4, or it may be 6 or more.

[0046] Furthermore, the first model M1 may include multiple encoders E11. For example, the first model M1 may include an encoder E11 that reduces the dimension of the first training target feature but not the dimension of the first training reference feature (an encoder E11 that performs processing only on the first training target feature), and an encoder E11 that reduces the dimension of the first training reference feature but not the dimension of the first training target feature (an encoder E11 that performs processing only on the first training reference feature). The parameters of these two encoders E11 may be independent of each other. The first model M1 may include three or more encoders E11.

[0047] The first output model M12 generates the first training processing information H based on the first training target feature and the first training reference feature. g The first output model M12 may utilize any machine learning method in which correlated inputs and outputs are learned. For example, the first output model M12 may be a neural network. Based on the first training target feature and first training reference feature input to itself, and its own parameters, the first training processed information H g Outputs the first training processing information H. g The output may be an output to an external entity (e.g., a second model M2) of the first output model M12, or it may be an output from one configuration of the first output model M12 to another configuration (an output from one layer of the first output model M12 to another layer).

[0048] In the first embodiment, we take as an example the case where the first model M1 further includes not only the first calculation model M10 and the first output model M12, but also the first encoder E11. Therefore, the first output model M12 generates the first training processing information H based on the first training target feature and the first training reference feature whose dimensions have been reduced by the first encoder E11. g This outputs the following. In the example in Figure 3, the arrows extending from the first encoder E11 to the first output model M12 indicate that the dimensionally reduced first trainee feature and first trained reference feature are input from the first encoder E11 to the first output model M12.

[0049] For example, the first output model M12 generates the first training processed information H based on the first training target feature and first training reference feature with reduced dimensions input to it, and its own parameters. g Outputs the first training processing information H. In the example in Figure 3, the first training processing information H g It has five parameters s x ,s y ,t h ,t x ,t y Includes parameters s x ,s y These represent the horizontal and vertical scaling factors, respectively. Parameter t his the amount of rotation. Parameter t x ,t y These are the amounts of movement in the lateral and vertical directions, respectively. In the first embodiment, we take the case where the affine transformation corresponds to machining as an example. These five parameters s x ,s y ,t h ,t x ,t y The affine transformation is controlled by this. First training processing information H g These are the five parameters s x ,s y ,t h ,t x ,t y Only a portion of it may be shown. For example, the first training processing information H g is parameter t h You can just show the parameter t, or x ,t y You may only show that.

[0050] Note that the first model M1 does not necessarily have to include the first encoder E11. If the first model M1 does not include the first encoder E11, the first output model M12 will generate the first training processing information H based on the first training target feature and the first training reference feature (the first training target feature and the first training reference feature whose dimensions have not been reduced by the first encoder E11) calculated by the first calculation model M10. g The output should be as follows. Thus, an embodiment in which the first model M1 does not include the first encoder E11 but includes the first calculation model M10 and the first output model M12 is also within the scope of this disclosure.

[0051] Furthermore, the configuration of the first model M1 is not limited to the example in Figure 3. For example, the first model M1 may not be divided into multiple networks such as the first computation model M10, the first encoder E11, and the first output model M12, but may contain only one network. If the first model M1 contains only one network, that single network will handle the training target image I t and training reference image I rBased on this, the first training target feature and the first training reference feature are calculated, and based on the first training target feature and the first training reference feature, the first training processed information H g The output may be as follows. These series of information processing steps may be defined in the network's program, and the network's parameters may be referenced by that program.

[0052] [Second Model Memory Unit] The second model memory unit 101 stores the second model M2. The second model M2 includes a program developed using a machine learning technique. The machine learning technique may be any technique used in the field of image processing. The second model M2 may be developed using supervised learning techniques, unsupervised learning techniques, or a combination thereof. For example, the second model M2 may be a model developed using a neural network, ViT, SIFT, SURF, HOG, or other techniques.

[0053] For example, the second model M2 includes a program that represents a series of information processing steps for an image input to itself, and parameters referenced by that program. The parameters may be incorporated as part of the program. The parameters of the second model M2 may be any parameters used in machine learning techniques. For example, the parameters of the second model M2 may be weights, biases, or other parameters. The parameters of the second model M2 may be any parameters adopted in the aforementioned neural network and other techniques.

[0054] For example, the second model storage unit 101 stores the second model M2 before training (before training is performed by the training unit 104). The second model M2 before training is the second model M2 with initial parameters. All or part of the second model M2 before training may have undergone some degree of pre-training. When training of the second model M2 is performed, the initial parameters are adjusted. Once training of the second model M2 is complete, the second model storage unit 101 stores the trained second model M2. The second model M2 before training may be overwritten with the trained second model M2, or the trained second model M2 may be stored in the second model storage unit 101 separately from the second model M2 before training.

[0055] In the first embodiment, the second model M2 is the training target image I t The second training target feature and training reference image I r The second training reference feature is calculated, and the first training processed information H is calculated. g Based on the second training target feature and the second training reference feature, the training target image I is adjusted so that the training target posture matches the training reference posture. t Second training processing information H for processing n ··H2H1H g The output is shown. The program for the second model M2 illustrates this series of information processing steps. The parameters of the second model M2 are referenced during this series of information processing steps.

[0056] For example, the second model M2, based on the parameters of the second model M2, uses the training target image I t The second training feature is calculated from the training image I calculated by the second model M2. t This is a characteristic of the second training target feature, which is the computer training target image I t This can also be described as information for recognizing the features of the image. The second training target feature is the training target image I t These are sometimes called embedding representations or features. The second training feature can be in any form. For example, the second training feature may be a feature map, a vector, an array, a single number, a combination of numbers, a matrix, or other form.

[0057] For example, based on the parameters of the second model M2, the second training reference features of the training reference image I r are calculated. The second training reference features are the features of the training reference image I r calculated by the second model M2. The second training reference features can also be said to be information for the computer to recognize the features of the training reference image I r . The second training reference features may also be called an embedded representation or feature amount of the training reference image I r . The second training reference features may be in any form. For example, the second training reference features may be a feature map, a vector, an array, a single numerical value, a combination of multiple numerical values, a matrix, or other forms

[0058] For example, based on the parameters of the second model M2, the first training processing information H g , the second training target features, and the second training reference features, the second training processing information H n ··H2H1H g is output. The output of the second training processing information H n ··H2H1H g may be an output to the outside of the second model M2 (for example, a processing program), or may be an output from one configuration of the second model M2 to another configuration (an output from one layer of the second model M2 to another layer)

[0059] The second training processing information H n ··H2H1H g is information for processing calculated by the second model M2. The second training processing information H n ··H2H1H g can also be said to be a coefficient (parameter) referred to during processing. For example, when an affine transformation corresponds to the processing, the transformation coefficients in the affine transformation correspond to the second training processing information H n ··H2H1H g . When the change in the arrangement of each pixel corresponds to the processing, the positional relationship before and after the change of each pixel corresponds to the second training processing information H n ··H2H1H g The second training processing information Hn ··H2H1H g It may be information referred to during processing, and is not limited to these examples. The second training processing information H n ··H2H1H g may be a translation amount, a rotation amount, an enlargement ratio, a reduction ratio, a trimming range, or a combination thereof.

[0060] For example, the second model M2 processes the second training target feature based on the first training processing information H g and outputs the second training processing information H n ··H2H1H g based on the processed second training target feature and the second training reference feature. The first training processing information H g is information indicating the content of rough processing, and the second training processing information H n ··H2H1H g is information indicating the content of the final processing. Among the second training processing information H n ··H2H1H g the part of H n ··H2H1 is a part for fine adjustment that cannot improve the accuracy only with the first training processing information H g (a part for improving the accuracy of processing by the first training processing information H g ).

[0061] In the first embodiment, the second model M2 includes a second calculation model M20 and a second output model M21. The second calculation model M20 calculates the second training target feature based on the training target image I t and calculates the second training reference feature based on the training reference image I r . In the example of FIG. 3, for the sake of explanation, two second calculation models M20 are shown, but in the first embodiment, the second calculation model M20 is taken as an example where there is one. That is, the two second calculation models M20 in FIG. 3 are the same second calculation model M20 that exists only one.

[0062] The second computational model M20 can be any model that computes the features of the image input to it, and any machine learning method may be used. In the first embodiment, the case in which the second computational model M20 is an encoder is given as an example. For example, the second computational model M20 may be a neural network that performs convolution. The second computational model M20 computes the features of the training target image I input to it. t and training reference image I r Based on its own parameters, the second training target feature and the second training reference feature are calculated.

[0063] Note that the second model M2 may include multiple second computational models M20. For example, the second model M2 may include training image I t The input is training reference image I r The second computation model M20, which does not receive input (the second computation model M20, which only performs calculations for the second training target feature), and the training reference image I r The input is the training target image I t The second computation model M20 may include a second computation model M20 that does not receive input (a second computation model M20 that only performs the calculation of the second training reference feature). The parameters of these two second computation models M20 may be independent of each other. The second model M2 may include three or more second computation models M20.

[0064] In the first embodiment, the second computation model M20 includes multiple layers that compute the second training target feature and the second trained reference feature. In the example in Figure 3, a second computation model M20 with four layers is shown, but the number of layers in the second computation model M20 is not limited to four. For example, the second computation model M20 may have one, two, or three layers, or five or more layers. For example, the second computation model M20 may compute the second training target feature and the second trained reference feature in each of the multiple layers by performing convolutions successively in each layer. In the example in Figure 3, the size of the rectangles representing each layer in the second computation model M20 corresponds to the size of the rectangles representing the second output model M21.

[0065] In the example in Figure 3, the arrow f extends from the second computation model M20 to the second output model M21. t l ,f r l The sign of is the second training target feature and the second training reference feature calculated by the last layer of the multiple layers. t l ,f r l l is any number between 1 and n. n is the number of layers included in the second computational model M20 (4 in the example in Figure 3). The example in Figure 3 shows the case where the values ​​of l and n are the same.

[0066] For example, each layer of the second computation model M20 may successively reduce the dimensions of the second target feature and the second reference feature. Each layer of the second computation model M20 computes features based on the features computed by the layer before it and its own parameters. Each layer of the second computation model M20 is sometimes called a convolutional layer. Each layer of the second computation model M20 may include other layers besides convolutional layers (e.g., activation function layers, pooling layers, or normalization layers). The configuration of the second computation model M20 may be similar to that of known encoders. For example, the second computation model M20 may be a module called a Target-aware feature extractor.

[0067] Furthermore, the second computational model M20 does not need to reduce the dimensions of the second training target feature and the second training reference feature. The second computational model M20 does not need to include multiple layers. The second computational model M20 may include only one layer. The second computational model M20 may be a model that does not have the concept of layers and is capable of computing the features of the image input to itself. Embodiments in which the second computational model M20 does not include multiple layers are also within the scope of this disclosure.

[0068] In the first embodiment, the second output model M21 is the first training processing information H g Based on the second training target features and the second training reference features, the second training processed information H n ··H2H1H gThe second output model M21 may utilize any machine learning method in which correlated inputs and outputs are learned. For example, the second output model M21 may be a neural network. Based on the second training target feature and second training reference feature input to itself, and its own parameters, the second training processed information H n ··H2H1H g Outputs the second training processing information H. n ··H2H1H g The output may be an output to an external source (e.g., a machining program) of the second output model M21, or an output from one configuration of the second output model M21 to another configuration (e.g., the second training machining information H in the second output model M21). n ··H2H1H g From the layer that calculates the second output model M21, the target image I is used for training. t This may also be the output for the layer that undergoes processing.

[0069] In the first embodiment, the second output model M21 is the first training processing information H g Based on the second training target features and second training reference features calculated in each of the multiple layers, the intermediate second training processing information, the second intermediate training processing information, is calculated layer by layer, and the final second training processing information, the second final training processing information, is output. Figure 3 shows H1, H2, ..., H n This is an example of second-intermediate training processing information.

[0070] In the example in Figure 3, the second intermediate training processing information H1, H2, ..., H n This is the first training processing information H g Similarly, the five parameters s x ,s y ,t h ,t x ,t y This shall include: Second Intermediate Training Processing Information H1, H2, ..., H n The number of parameters included and the first training processing information H g The number of parameters included in may be different from that of H in Figure 3. n ··H2H1H gThis is an example of the second final training processing information. The second final training processing information is H g H n The order of multiplication can be arbitrary, such as H2H1.

[0071] For example, the second output model M21 is the second training target feature (in the example in Figure 3, f) calculated by the last layer (in the example in Figure 3, the fourth layer) among the multiple layers of the second computation model M20. t l ) and the second training reference feature (in the example in Figure 3, f r l ) is obtained. The second output model M21 obtains the first training processing information H g Based on this, the second training target feature calculated by the last layer is transformed. If the second training target feature is a feature map, the second output model M21 is the first training processed information H g Based on this, the second training target feature is processed.

[0072] In the example in Figure 3, the processed second training feature is shown by the shaded rectangle. The processing of the second training feature may be performed by a technique called feature warping. The processed second training feature is compared to the unprocessed second training feature with respect to the first training processing information H g This shows the characteristics after the processing indicated. If the second training target feature is information in a format other than a feature map, the second output model M21 will show the first training processing information H g The second training target feature should be transformed so that the processing shown is performed.

[0073] For example, the second output model M21 inputs the processed second training target feature and the second training reference feature into the neural network N210. In the example in Figure 3, the second output model M21 includes multiple neural networks N210. Each neural network N210 is designed to accept input corresponding to the size of the second training target feature and the second training reference feature computed by each layer of the second computation model M20. For example, the second output model M21 may include the same number of neural networks N210 as the number of layers in the second computation model M20. That is, the second output model M21 may include n neural networks N210. The parameters of each neural network N210 are distinct parameters independent of the parameters of other neural networks N210.

[0074] In the example in Figure 3, the leftmost neural network N210 receives the first training information H calculated by the last layer of the second computational model M20. g The processed second training target features, which have been processed by the first neural network, and the second training target features calculated by the last layer of the second computational model M20 are input. The leftmost neural network N210 calculates these features based on its own parameters and produces an output corresponding to those features. In the example in Figure 3, the output of the leftmost neural network N210 is the second intermediate training processing information H1. The second intermediate training processing information H1 shows the processed second training target features input to the leftmost neural network N210, the second training target features, and the coefficients for processing according to those features.

[0075] For example, the second output model M21 uses the first training processing information H. g And the information H1H obtained by multiplying the second intermediate training processing information H1 output from the leftmost neural network N210 by the other information H1H g The second output model M21 calculates the calculated information H1H. g Based on this, the second training target feature (in the example in Figure 3, f) is calculated by the second to last layer of the second computational model M20. t l-1 ) is processed.

[0076] For example, the second output model M21 uses the processed second training target feature and the second training reference feature calculated by the second-to-last layer of the second computation model M20 (in the example in Figure 3, f r l-1 The ) and are input to the neural network N210, the second from the left in Figure 3. The neural network N210, the second from the left, calculates these features based on its own parameters and outputs according to those features. In the example in Figure 3, the output of the neural network N210, the second from the left, is the second intermediate training processing information H2. The second intermediate training processing information H2 shows the processed second training target features input to the neural network N210, the second training target features, and the coefficients for processing according to them.

[0077] Similarly, the second output model M21 successively processes the second intermediate training information H3,··,H up to the first layer of the second calculation model M20. n The second output model M21 calculates the final second final training processing information H. n ··H2H1H g The output is as follows: In this way, the second output model M21 outputs the first training processed information H based on the second training target features and second training reference features calculated by each layer of the second computation model M20. g As additional information for fine-tuning, the second intermediate training processing information H1,··,H n Calculate the second final training processing information H. n ··H2H1H g This is the first training processing information H g In addition, second intermediate training processing information H1,··,H for fine-tuning n This is also reflected in the design, enabling highly precise machining.

[0078] Furthermore, the second output model M21 is particularly the second intermediate training processing information H1,··,H n It is not necessary to calculate this. The second output model M21 is the second intermediate training processing information H1,··,H nThe final second training and processing information may be output without calculating the second training model. Also, the configuration of the second model M2 is not limited to the example in Figure 3. For example, the second model M2 may not be divided into multiple networks such as the second calculation model M20 and the second output model M21, but may consist of only one network.

[0079] For example, if the second model M2 includes only one network, that single network will train on image I. t and training reference image I r Based on this, the second training target feature and the second training reference feature are calculated, and the first training processed information H g Based on the second training target features and the second training reference features, the second training processed information H n ··H2H1H g The output may be as follows. These series of information processing steps may be defined in the network's program, and the network's parameters may be referenced by that program.

[0080] [Data Storage Unit] The data storage unit 102 stores the data necessary for training the first model M1 and the second model M2. For example, the data storage unit 102 stores a training database DB containing multiple training data sets to be trained on the learning model M. The training data includes an input portion that is input to the first model M1 and the second model M2 during training, and a correct answer portion (output portion) that is the correct answer during training. The correct answer portion is not limited to the final output of the second model M2, but may also be an output showing the intermediate results that the second model M2 calculates to obtain the final output. The correct answer portion may also be a result obtained from the final output of the second model M2.

[0081] Figure 4 shows an example of a training database DB. For example, the input section for training data is the training target image I. t and training reference image I r For example, consider a certain training image I. t The object to be trained, and other training images I tThe training target object indicated by may be the same as or different from the training target image I. t The training target pose of the training target object shown by and other training target images I t The training target pose of the training target object shown by may be the same as or different from a certain training reference image I. r The training reference object shown and other training reference images I r The training reference object indicated by may be the same as or different from the training reference image I. r The training reference pose of the training reference object shown and other training reference images I r The training reference pose of the training reference object indicated by and may be the same or different.

[0082] The correct portion of the training data is used as correct information in the processed training image I. t It may include the image itself, or the training target image I t It may include processing information used in the processing, or it may include other information. In the first embodiment, we take the example that the correct portion of the training data is correct processing information, which is the processing information that results in the correct answer. The correct portion of the training data may include other information besides the correct processing information. In the example in Figure 4, the correct portion of the training data includes correct processing information as the correct processing information. In the example in Figure 4, the code for the correct processing information is denoted by a bar, but in the following explanation, the bar in the code will be represented in parentheses, such as H(bar).

[0083] Note that the data stored in the data storage unit 102 is not limited to the above example. For example, the data storage unit 102 may store a program that shows the processing during learning. This program may also define a formula for calculating the loss function.

[0084] [Training Data Acquisition Unit] The training data acquisition unit 103 acquires training data. The training data is a training target image I that represents the training target object. t And, training reference image I which shows the training reference object. rThe input includes and and, so that the training target pose of the training target object matches the training reference pose of the training reference object, the training target image I t The correct answer information for processing is included as the correct answer portion. In the first embodiment, since the training data is stored in the training database DB, the training data acquisition unit 103 acquires the training data from the training database DB. The training data stored in the training database DB is assumed to be prepared by the creator of the learning model M (for example, the person operating the learning terminal 10).

[0085] If training data is stored in a database other than the training database DB, the training data acquisition unit 103 can acquire the training data from the other database. If training data is stored in a computer other than the learning terminal 10, or in an information storage medium, the training data acquisition unit 103 can acquire the training data from the other computer or information storage medium. The training data acquisition unit 103 can acquire any number of training data. For example, the training data acquisition unit 103 can acquire all or part of the training data stored in the training database DB. The training data acquisition unit 103 may repeat the acquisition of training data until the values ​​of each loss function described later become sufficiently small.

[0086] [Learning Department] The learning unit 104 performs training on at least one of the first model M1 and the second model M2 based on the training data. Training is the adjustment of parameters. In the first embodiment, the learning unit 104 performs training on both the first model M1 and the second model M2 as an example, but the learning unit 104 may perform training on only the first model M1 without performing training on the second model M2. The learning unit 104 may perform training on only the second model M2 without performing training on the first model M1. The learning unit 104 may perform training on the entire first model M1, or on only a part of the first model M1. The learning unit 104 may perform training on the entire second model M2, or on only a part of the second model M2.

[0087] For example, the learning unit 104 provides the training target image I, which is the input portion of the training data, to the first model M1 and the second model M2. t and training reference image I r The input is then processed by the learning unit 104, which provides the training target image I to the first model M1 and the second model M2. t and training reference image I r You don't have to input all of them at once; training target image I t and training reference image I r These may be input separately. The learning unit 104 performs training on at least one of the first model M1 and the second model M2 based on the processing results of the first model M1 and the second model M2. The learning unit 104 performs training on at least one of the first model M1 and the second model M2 such that when the input portion of the training data is input, the output portion of the training data is output.

[0088] For example, the first model M1 uses training image I t and training reference image I r When input is received, the first training target feature and the first training reference feature are calculated based on the current parameters. The first model M1 then calculates the first training processing information H based on the current parameters, the first training target feature, and the first training reference feature. g The output is as described above. The learning unit 104 executes this sequence of steps by running the program for the first model M1.

[0089] For example, the second model M2 uses training image I t and training reference image I r When input is received, the second training target feature and the second training reference feature are calculated based on the current parameters. The second model M2 uses the current parameters and the first training processing information H g Based on the second training target feature and the second training reference feature, the second training processed information H n ··H2H1H gIt outputs the following. This sequence of events is as described above. The learning unit 104 executes this sequence of events by running the program for the second model M2.

[0090] For example, the learning unit 104 calculates the loss based on the output of the second model M2, the correct portion of the training data, and a predetermined loss function. The learning unit 104 performs training on at least one of the first model M1 and the second model M2 by adjusting the parameters of at least one of the first model M1 and the second model M2 so that the loss is minimized. When multiple training data sets are acquired one after another by the training data acquisition unit 103, the learning unit 104, for each training data set, calculates the training target image I contained in each of the training data sets. t and training reference image I r The data is input to the first model M1 and the second model M2, the output from the second model M2 is obtained, the loss is calculated based on the loss function, and the parameters are adjusted to minimize the loss. This process is repeated.

[0091] The learning unit 104 may calculate the loss based on the output of the first model M1, the correct portion of the training data, and a predetermined loss function. The learning unit 104 may perform training on at least one of the first model M1 and the second model M2 by adjusting the parameters of at least one of the first model M1 and the second model M2 so that the loss is small. When each of the multiple training data is acquired one after another by the training data acquisition unit 103, the learning unit 104 may, for each training data, acquire the training target image I contained in each of the training data. t and training reference image I r The data may be input to the first model M1 and the second model M2, the output from the first model M1 may be obtained, the loss may be calculated based on the loss function, and the parameters may be adjusted to minimize the loss.

[0092] Furthermore, the learning unit 104 may perform training on at least one of the first model M1 and the second model M2 based on known learning algorithms employed in machine learning methods. For example, the learning unit 104 may train at least one of the first model M1 and the second model M2 with training data based on backpropagation, gradient descent, ADAM (Adaptive Moment) method, momentum method, methods using discriminators and generators employed in GANs, or other methods. The learning unit 104 may repeat the training of at least one of the first model M1 and the second model M2 until the loss falls below a threshold, or it may repeat the training of at least one of the first model M1 and the second model M2 until the number of training iterations reaches a predetermined number. The learning unit 104 may use the same training data for repeated training.

[0093] In the first embodiment, the first model M1 includes a first computation model M10 and a first output model M12. For example, the learning unit 104 may perform training on the first output model M12 without training on the first computation model M10. The parameters of the first computation model M10 are fixed because other training objects have already been trained. The learning unit 104 may also perform training on the first encoder E11.

[0094] In the first embodiment, the training data includes correct machining information H(bar) relating to the correct machining as correct answer information. The learning unit 104 receives the second training machining information H n ··H2H1H g Based on the correct machining information H(bar), the machining loss is calculated, and based on the machining loss, training is performed on at least one of the first model M1 and the second model M2. For example, the learning unit 104 calculates the machining loss L based on the following formula 1. affine l Calculate the machining loss L. affine l This is the second training processing information H n ··H2H1H g This is the loss that shows the magnitude of the difference between the correct processing information H (bar) and the second training processing information H. n ··H2H1H gFurthermore, the greater the difference in the correct machining information H (bar), the greater the machining loss L. affine l The larger these differences become, the smaller the processing loss L affine l It becomes smaller.

[0095]

number

[0096] For example, the training data is the correct processed training image I, which serves as the ground truth information. t It may include correct image information related to the following. The correct image information may be correct processed information H (bar), or training target image I. t This is the image after processing with the correct processing information H (bar) (processed training image I). t ) is also acceptable.

[0097] Figure 5 shows the processed training image I. t This figure shows an example of how the first training processing information H is acquired. As shown in Figure 5, g Processed training image I based on the above t (H g ) is the training image I before processing. t Rather, the first training target posture is closer to the first training reference posture. Second intermediate training processing information H1H g Processed training image I based on the above t (H1H g ) is the training target image I t (H g The first training target posture is closer to the first training reference posture than the second intermediate training processing information H1H. g This is due to fine-tuning of the H1 portion. Similarly, the second intermediate training processing information H2H1H g ,··,H n ··H2H1H g The processed training image I corresponding to each of these t (H2H1H g ),··,I t (H n ··H2H1Hg Through fine-tuning, the first training target posture gradually approaches the first training reference posture.

[0098] For example, the learning unit 104 processes the second training information H l ··H2H1H g Based on this, training target image I t The image is processed. The learning unit 104 processes the processed training target image I. t Based on the ground truth image information, the image loss is calculated, and based on this image loss, training is performed on at least one of the first model M1 and the second model M2. For example, the learning unit 104 calculates the image loss L based on the following formula 2. image l Calculate the following: I of formula 2 t l This is the training target image I t This is a function that shows how to process the image. Image loss L image l This is the processed training image I t (H n ··H2H1H g ) and the correct processed training image I corresponding to the correct image information. t This is the loss that shows the magnitude of the difference between (H(bar)) and . Processed training image I t (H n ··H2H1H g ) and the correct processed training image I corresponding to the correct image information. t The greater the difference between (H(bar)) and , the greater the image loss L. image l The larger these differences become, the smaller the image loss L image l The magnitude of these differences decreases. The magnitude of these differences may be calculated based on the difference in pixel values ​​of each pixel.

[0099]

number

[0100] For example, the training data includes the training target image I as ground truth information. tEach pixel and the correct processed training image I t It may include correct correspondence information regarding the correspondence between each pixel and the target image. The correct image information may be correct processing information H(bar), or the training target image I t This is the image after processing with the correct processing information H (bar) (processed training image I). t ) and the training image I before processing t This information may also indicate a correspondence between pixels. Here, the correspondence refers to information indicating where a pixel moved after processing. For example, the amount of horizontal movement (X-axis direction) of each pixel corresponds to the amount of vertical movement (Y-axis direction) of that pixel. The amount of rotation of each pixel may also correspond to the correspondence, or the magnification ratio of each pixel may also correspond to the correspondence.

[0101] Figure 6 shows the training target image I before and after processing. t This figure shows an example of the correspondence between each pixel. As shown in Figure 6, the learning unit 104 processes the second training information H n ··H2H1H g Based on this, training target image I t The image is processed, and the training target image I before processing is processed. t And the processed training image I t (H n ··H2H1H g The system obtains training correspondence information regarding the correspondence between ( ) and ( ), calculates the correspondence loss based on the training correspondence information and the correct correspondence information, and performs training on at least one of the first model M1 and the second model M2 based on the correspondence loss.

[0102] Training compatibility information is the training target image I before processing. t And the processed training image I t (H n ··H2H1H gThis information shows the correspondence between the original pixel and the new pixel. For example, if each pixel is moved by processing, the training correspondence information will show the amount of horizontal movement (X-axis direction) and the amount of vertical movement (Y-axis direction) of that pixel. If each pixel is rotated by processing, the training correspondence information will show the amount of rotation of each pixel. If each pixel is enlarged or reduced by processing, the training correspondence information will show the enlargement ratio of each pixel.

[0103] For example, the learning unit 104 processes the second training information H l ··H2H1H g Based on this, training response information C t l (H l ··H2H1H g ) calculates training information C t l (H l ··H2H1H g Function C for the calculation of ) t l This may be a publicly known function. Training correspondence information C t l (H l ··H2H1H g This is sometimes called a correspondence map. The learning unit 104 calculates the correspondence loss L based on the following equation 3. corres l Calculate the corresponding loss L. corres l This is training response information C t l (H l ··H2H1H g ) and correct answer correspondence information C t l This is the loss that shows the magnitude of the difference between (H(bar)) and . Training correspondence information C t l (H l ··H2H1H g ) and correct answer correspondence information C t l The greater the difference between (H(bar)) and the corresponding loss L, the larger the corresponding loss L. corres l The larger these differences become, the smaller the corresponding loss L corres l It becomes smaller.

[0104]

number

[0105] For example, the learning unit 104 calculates the total loss L based on the following formula 4. total The processing loss L is calculated. Since the value of l can take on values ​​from 1 to n, in the example of formula 4 below, the learning unit 104 changes the value of l from 1 to n and calculates the processing loss L. affine l , image loss L image l , and corresponding loss L corres l We calculate these one after another, and the sum of these W l (L affine l +L image l +L corres l The learning unit 104 calculates the total W by changing the value of l from 1 to n. l (L affine l +L image l +L corres l By summing these, the total loss L total The learning unit 104 calculates the total loss L. total Train at least one of the first model M1 and the second model M2 so that the value becomes small.

[0106]

number

[0107] Note that the learning method used by the learning unit 104 is not limited to the example above. For example, the learning unit 104 may use the total loss L total Without calculating the processing loss L affine l , image loss L image l , and corresponding loss L corres lBased on any one or two of the above, at least one of the first model M1 and the second model M2 may be trained. The learning unit 104 does not change the value of l from 1 to n, and the total W 1 (L affine 1 +L image 1 +L corres 1 )~W n (L affine n +L image n +L corres n You may calculate only a portion of (for example, just one of these) to train at least one of the first model M1 and the second model M2.

[0108] [1-4. Processing performed by the learning system of the first embodiment] Figure 7 shows an example of processing performed by the learning system 1 of the first embodiment. The processing in Figure 7 is performed when the control unit 11 executes a program stored in the storage unit 12. Each step in Figure 7 is an example of the learning method according to the present disclosure.

[0109] As shown in Figure 7, the learning terminal 10 obtains training data from the training database DB (S100). The learning terminal 10 then retrieves the training target image I t and training reference image I r The data is input to the first model M1 (S101). The learning terminal 10 calculates the first training target feature and the first training reference feature based on the first computation model M10 (S102). The learning terminal 10 reduces the dimensions of the first training target feature and the first training reference feature based on the first encoder E11 (S103). The learning terminal 10 outputs the first training processed information H from the first output model M12 based on the first output model M12 and the first training target feature and first training reference feature whose dimensions have been reduced. g Obtain (S104).

[0110] The learning terminal 10 displays the training target image I t and training reference image I rThe first training processing information H is input to the second model M2 (S105). The learning terminal 10 calculates the second training target feature and the second training reference feature based on the second computation model M20 (S106). The learning terminal 10 inputs the first training processing information H g The second training target feature and the second training reference feature are input to the second output model M21, and the second training processed information H output by the second output model is then processed. n ··H2H1H g Obtain (S107).

[0111] The learning terminal 10 receives the second training processing information H n ··H2H1H g Based on the correct portion of the training data, the processing loss L affine l The calculation is performed (S108). The learning terminal 10 receives the second training processing information H. n ··H2H1H g Based on the correct portion of the training data, the image loss L image l The calculation is performed (S109). The learning terminal 10 receives the second training processing information H. n ··H2H1H g Based on the correct portion of the training data, the corresponding loss L corres l The processing loss L is calculated (S110). The learning terminal 10 calculates the processing loss L. affine l , image loss L image l , and corresponding loss L corres l Based on this, the total loss L total Calculate (S111).

[0112] The learning terminal 10 has a total loss L totalBased on this, training is performed on at least one of the first model M1 and the second model M2 (S112). The training terminal 10 determines whether or not to complete the training (S113). In S113, the training terminal 10 may determine whether each loss has fallen below a threshold, or it may determine whether a predetermined number of training data has been trained on at least one of the first model M1 and the second model M2. If it is determined that training is not complete (S113:N), the process returns to S100 and the next training data is acquired. If it is determined that training is complete (S113:Y), the training terminal 10 sends the trained first model M1 and the second model M2 to the server 20 (S114), and this process ends. The server 20 records the trained first model M1 and the second model M2.

[0113] [1-5. Summary of the First Embodiment] The learning system 1 of the first embodiment acquires training data. The learning system 1 stores a first model M1. The learning system 1 stores a second model M2. Based on the training data, the learning system 1 performs training on at least one of the first model M1 and the second model M2. As a result, the learning system 1 adjusts the training target pose to match the training target image I t To enable high-precision processing, training is performed on at least one of the first model M1 and the second model M2, thereby improving the accuracy of the processing. For example, the learning system 1 trains on image I t Even if sufficient feature points cannot be extracted, at least one of the first model M1 and the second model M2, which can perform highly accurate processing, can be created. Since the learning system 1 can create at least one of the first model M1 and the second model M2, which do not require complex processing such as extracting a large number of feature points, the processing load on the computer used during estimation can be reduced. The learning system 1 can improve the accuracy of estimating the pose of the training target object, which is the same type of object as the training reference object. The learning system 1 can perform processing in various ways, such as scaling, rotation, and translation, like affine transformations, rather than just simple pixel modification.

[0114] Furthermore, the first model M1 includes the first computation model M10 and the first output model M12. The learning system 1 can improve the accuracy of the first model M1 by separating the first computation model M10, which is specialized for feature calculation, from the first output model M12, which is specialized for output for processing.

[0115] Furthermore, the first computational model M10 is a pre-trained model that has learned other training objects different from the training target object and the training reference object. The learning system 1 performs training of the first output model M12 without performing training of the first computational model M10. This allows the learning system 1 to efficiently train the first model M1 by utilizing the pre-trained first computational model M10. For example, if the first computational model M10 has learned features such as common characters, the learning system 1 can handle unknown logos etc. that have not been learned by the second model M2. The learning system 1 can avoid the trouble of retraining at least one of the first model M1 and the second model M2 in order to handle logos different from those learned.

[0116] Furthermore, the first model M1 further includes a first encoder E11. The first output model M12 generates first training processing information H based on the first training target feature and first training reference feature whose dimensions have been reduced by the first encoder E11. g This outputs the first training information H, based on the first training target features and first training reference features, which are easier to recognize. g Since it will be able to output this, the learning system 1 can further improve the precision of the machining.

[0117] Furthermore, the second model M2 uses the first training processing information H g Based on this, the second training target feature is processed, and based on the processed second training target feature and the second training reference feature, the second training processed information H is generated. n ··H2H1H gThis outputs the following: As a result, the learning system 1 can use the second model M2 to perform machining that the first model M1 cannot handle, thereby improving the precision of the machining.

[0118] Furthermore, the second model M2 includes a second computation model M20 and a second output model M21. This allows the learning system 1 to improve the accuracy of the second model M2 by separating the second computation model M20, which is specialized for feature computation, from the second output model M21, which is specialized for output for processing.

[0119] Furthermore, the second computation model M20 includes multiple layers that compute the second training target feature and the second training reference feature. The second output model M21 computes the first training processed information H g Based on the second training target features and second training reference features calculated in each of the multiple layers, the second intermediate training processing information H is generated, which is intermediate second training processing information. n H2 and H1 are calculated layer by layer, and the final second training processing information is the second final training processing information H n ··H2H1H g This outputs the following: As a result, the learning system 1 can successively calculate the information needed to perform machining that is not possible with the first model M1 using the second model M2, thereby improving the accuracy of the machining.

[0120] Furthermore, the training data includes correct processing information H(bar) related to the correct processing as ground truth information. The learning system 1 uses the second training processing information H n ··H2H1H g And based on the correct machining information H (bar), the machining loss L affine l Calculate the processing loss L affine l Based on this, training is performed on at least one of the first model M1 and the second model M2. As a result, the learning system 1 learns the processing loss L affine l Since it is possible to create at least one of the first model M1 and the second model M2 such that the size is reduced, the precision of the machining can be improved.

[0121] Furthermore, the training data includes the correct processed training image I as ground truth information. t This includes correct image information related to the subject. Learning system 1 is second training processing information H n ··H2H1H g Based on this, training target image I t The processed image I of the training target is then processed. t Based on the correct image information, the image loss L image l Calculate the image loss L image l Based on this, training is performed on at least one of the first model M1 and the second model M2. As a result, the learning system 1 learns the image loss L image l Since it is possible to create at least one of the first model M1 and the second model M2 such that the size is reduced, the precision of the machining can be improved.

[0122] Furthermore, the training data includes the training target image I as ground truth information. t Each pixel and the correct processed training image I t This includes correct correspondence information regarding the correspondence between each pixel and . The learning system 1 is the second training processing information H n ··H2H1H g Based on this, training target image I t The image is processed, and the training target image I before processing is processed. t And the processed training image I t We obtain training response information regarding the correspondence with, and based on the training response information and the correct response information, we calculate the correspondence loss L corres l Calculate the corresponding loss L corres l Based on this, training is performed on at least one of the first model M1 and the second model M2. As a result, the learning system 1 has a corresponding loss L corres l Since it is possible to create at least one of the first model M1 and the second model M2 such that the size is reduced, the precision of the machining can be improved.

[0123] [2. Second Embodiment] A second embodiment, which is an example of an embodiment of the estimation system 2, the estimation method, and the program according to the present disclosure, will be described. In the first embodiment, the configurations during the learning of the first model M1 and the second model M2 were described. In the second embodiment, the configuration during the estimation using the learned first model M1 and second model M2 will be described. In the second embodiment, descriptions of the same points as in the first embodiment will be omitted. Note that the estimation system 2 may include only the functions for estimation to be described below without including the functions for learning described in the first embodiment. An aspect in which the estimation system 2 includes only the functions for estimation without including the functions for learning is also within the scope of the present disclosure.

[0124] In the second embodiment, a case where the hardware configuration of the estimation system 2 is the same as that of the learning system 1 will be taken as an example. For example, the estimation system 2 includes the learning terminal 10, the server 20, and the user terminal 30. The hardware configuration of the estimation system 2 is not limited to the example of FIG. 1. The estimation system 2 may include at least one computer. For example, the estimation system 1 may include only the server 20. In this case, the learning terminal 10 and the user terminal 30 exist outside the estimation system 2. The estimation system 2 may include a computer not shown in FIG. 1. For example, estimation using the learned first model M1 and second model M2 may be executed by a computer other than the server 20. <(

[0125] In the second embodiment, the target image, the target object, the target pose, the reference image, the reference object, and the reference pose at the time of estimation are referred to as the estimation target image, the estimation target object, the estimation target pose, the estimation reference image, the estimation reference object, and the estimation reference pose, respectively. The estimation system 2 processes the estimation target image so that the estimation target pose of the estimation target object in the estimation target image matches the estimation reference pose of the estimation reference object in the estimation reference image. Details of the estimation system 2 will be described below.

[0126] [2-1. Functions realized by the estimation system of the second embodiment] FIG. 8 is a diagram showing an example of functions realized by the estimation system 2 of the second embodiment. In the second embodiment, among the functions realized by the estimation system 2, the functions realized by the server 20 will be described. For example, the server 20 includes a first model storage unit 200, a second model storage unit 201, a data storage unit 202, and an estimation unit 203. The first model storage unit 200, the second model storage unit 201, and the data storage unit 202 are realized by the storage unit 22. The estimation unit 203 is realized by the control unit 21.

[0127] [First Model Storage Unit] The first model storage unit 200 stores the learned first model M1. For example, the server 20 acquires the learned first model M1 from the learning terminal 10 and records it in the first model storage unit 200.

[0128] [Second Model Storage Unit] The second model storage unit 201 stores the learned second model M2. For example, the server 20 acquires the learned second model M2 from the learning terminal 10 and records it in the second model storage unit 201.

[0129] [Data Storage Unit] The data storage unit 202 stores the estimation target image and the estimation reference image. For example, the server 20 acquires the estimation target image from the user terminal 30 and records it in the data storage unit 202. It is assumed that the estimation reference image is recorded in the data storage unit 202 in advance.

[0130] [Estimation Unit] After the learning by the learning unit 104 described in the first embodiment is completed, the estimation unit 203 processes the estimation target image based on the estimation target image indicating the estimation target object, the estimation reference image indicating the estimation reference object, the first model M1 described in the first embodiment, and the second model M2 described in the first embodiment, so that the estimated posture of the estimation target object matches the estimated reference posture of the estimation reference object.

[0131] For example, the first model M1 calculates the first estimated target features of the target image and the first estimated reference features of the estimated reference image, and outputs the first estimated processing information based on the first estimated target features and the first estimated reference features. The method for calculating the first estimated target features and the first estimated reference features can be obtained by replacing the word "training" with "estimate" in the description of the calculation method for the first training target features and the first training reference features described in the first embodiment. The method for outputting the first estimated processing information based on the first estimated target features and the first estimated reference features can also be obtained by replacing the word "training" with "estimate" in the description of the first training processing information H described in the first embodiment. g Simply replace the word "training" with "estimation" in the explanation of the output method.

[0132] For example, the second model M2 calculates the second estimated target features of the target image and the second estimated reference features of the estimated reference image, and outputs the second estimated processing information based on the second estimated target features and the second estimated reference features. The method for calculating the second estimated target features and the second estimated reference features can be described by replacing the word "training" with "estimate" in the description of the calculation method for the second training target features and the second training reference features described in the second embodiment. The method for outputting the second estimated processing information based on the second estimated target features and the second estimated reference features can also be described as the second training processing information H described in the second embodiment. n ··H2H1H g Simply replace the word "training" with "estimation" in the explanation of the output method.

[0133] [2-2. Processing performed by the estimation system of the second embodiment] Figure 9 shows an example of the process performed by the estimation system 2 of the second embodiment. The control units 21 and 31 execute the program stored in the storage units 22 and 32, thereby executing the process shown in Figure 8. Each step in Figure 9 is an example of the estimation method according to the present disclosure.

[0134] As shown in Figure 9, the user terminal 30 generates an estimated target image based on the image capture results of the imaging unit 36 ​​and transmits the estimated target image to the server 20 (S200). The server 20 receives the estimated target image from the user terminal 30 (S201). The server 20 retrieves the estimated reference image stored in the storage unit 22 (S202). The estimated reference image is assumed to show the estimated reference object in an appropriate orientation.

[0135] Server 20 inputs the target image and estimated reference image into the trained first model M1 (S203). Server 20 calculates the first target feature and the first estimated reference feature based on the first computation model M10 (S204). Server 20 reduces the dimensionality of the first target feature and the first estimated reference feature based on the first encoder E11 (S205). Server 20 obtains the first estimated processing information output by the first output model M12 based on the first output model M12 and the first target feature and first estimated reference feature with reduced dimensionality (S206).

[0136] Server 20 inputs the target image and estimated reference image into the second model M2 (S207). Server 20 calculates the second target feature and second estimated reference feature based on the second calculation model M20 (S208). Server 20 inputs the first estimated processing information, the second target feature, and the second estimated reference feature into the second output model M21 and obtains the second estimated processing information output by the second output model M21 (S209). Server 20 processes the target image based on the second estimated processing information (S210), and this process ends. The processing of the target image in S210 may be performed within at least one of the first model M1 and the second model M2, or there may be a separate program for processing. If the estimation system 1 is used for eKYC, after the processing in S210 is performed, the eKYC processing is performed based on the processed target image.

[0137] [2-3. Summary of the second embodiment] The estimation system 2 of the second embodiment processes the target image based on the target image, the estimated reference image, the first model M1 described in the first embodiment, and the second model M2 described in the first embodiment, after the learning unit 104 described in the first embodiment has completed learning, so that the estimated target pose matches the estimated reference pose. This allows the learning system 1 to improve the accuracy of processing the target image. For example, the estimation system 2 can perform highly accurate processing even if it cannot extract enough feature points from the target image. Since the estimation system 2 does not need to perform complex processing such as extracting a large number of feature points, it can reduce the processing load on the server 20. The estimation system 2 can improve the accuracy of estimating the pose of the target object, which is the same type of object as the estimated reference object. The estimation system 2 can perform processing not only by changing pixels, but also from various perspectives such as scaling, rotation, and translation, like affine transformations.

[0138] [3. Variant] This disclosure is not limited to the first and second embodiments described above. This disclosure may be modified as appropriate without departing from the spirit of this disclosure.

[0139] For example, Model 1 M1 and Model 2 M2 may be used for purposes other than eKYC or possession authentication. They may be used to process landscape photographs taken by the user, to process images of documents scanned by the user, to process computer graphics created by the user, or for other purposes. The services in which Model 1 M1 and Model 2 M2 are used may also be any services. For example, they may be used in e-commerce services, communication services, travel booking services, financial services, payment services, or other services.

[0140] For example, the training object, training reference object, estimation object, and estimation object may be objects other than the logo. For example, the other object may be a string of characters indicating a credit card company, characters formed on identification documents such as a driver's license, characters formed on other media other than identification documents, a subject such as a road sign or building, or various shapes. The first model M1 and the second model M2 are applicable to any situation where it is necessary to correct the orientation of some object.

[0141] For example, the functions described as being implemented on the learning terminal 10 may be implemented on other computers such as the server 20. The functions described as being implemented on the learning terminal 10 may be shared between the learning terminal 10 and other computers. The functions described as being implemented on the server 20 may be implemented on other computers such as the user terminal 30. The functions described as being implemented on the server 20 may be shared between the server 20 and other computers.

[0142] [4. Addendum] For example, the learning system and estimation system can also be configured as follows: (1) A training data acquisition unit that includes a training target image representing the object to be trained and a training reference image representing a training reference object as input parts, and acquires training data that includes correct information as a correct part for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object, A first model storage unit stores a first model which calculates a first training target feature of the training target image and a first training reference feature of the training reference image, and outputs first training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training target feature and the first training reference feature, Calculate the second training target feature of the training target image and the second training reference feature of the training reference image, and output second training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training processing information, the second training target feature, and the second training reference feature. A second model storage unit that stores the second model; A learning unit that performs learning of at least one of the first model and the second model based on the training data; A learning system including. (2) The first model is A first calculation model that calculates the first training target feature based on the training target image and calculates the first training reference feature based on the training reference image; A first output model that outputs the first training processing information based on the first training target feature and the first training reference feature; The learning system according to (1) including. (3) The first calculation model is a learned model in which another training object different from the training target object and the training reference object is learned, The learning unit performs learning of the first output model without performing learning of the first calculation model. The learning system according to (2). (4) The first model further includes a first encoder that reduces the dimensions of the first training target feature and the first training reference feature calculated by the first calculation model, The first output model outputs the first training processing information based on the first training target feature and the first training reference feature whose dimensions have been reduced by the first encoder. The learning system according to (2) or (3). (5) The second model processes the second training target feature based on the first training processing information, and outputs the second training processing information based on the processed second training target feature and the second training reference feature. A learning system described in any of (1) to (4). (6) The second model is, A second computational model that calculates the second training target feature based on the training target image and calculates the second training reference feature based on the training reference image, A second output model that outputs the second training processing information based on the first training processing information, the second training target feature, and the second training reference feature, A learning system described in any of (1) to (5), including (1) above. (7) The second computational model includes multiple layers that compute the second training target feature and the second training reference feature, The second output model calculates, layer by layer, an intermediate second training information, which is a second intermediate training information, based on the first training information and the second training target feature and second training reference feature calculated in each of the plurality of layers, and outputs the final second training information, which is a second final training information. The learning system described in (6). (8) The aforementioned training data includes, as the correct answer information, correct machining information relating to the correct machining process, The learning unit calculates the machining loss based on the second training machining information and the correct machining information, and performs learning on at least one of the first model and the second model based on the machining loss. A learning system described in any of (1) to (7). (9) The training data includes, as the correct answer information, correct answer image information relating to the processed training target image that is the correct answer. The learning unit processes the training target image based on the second training processing information, calculates the image loss based on the processed training target image and the ground truth image information, and performs training on at least one of the first model and the second model based on the image loss. A learning system described in any of (1) to (8). (10) The training data includes, as the correct answer information, correct answer correspondence information relating to the correspondence between each pixel of the training target image and each pixel of the processed training target image that represents the correct answer. The learning unit processes the training target image based on the second training processing information, obtains training correspondence information relating to the correspondence between the training target image before processing and the training target image after processing, calculates the correspondence loss based on the training correspondence information and the correct answer correspondence information, and performs training on at least one of the first model and the second model based on the correspondence loss. A learning system described in any of (1) to (9). (11) After learning by the learning unit described in any of (1) to (10) is completed, the system includes an estimation unit that processes the estimated target image so that the estimated target pose of the estimated target object matches the estimated reference pose of the estimated reference object, based on the estimated target image showing the estimated target object, the estimated reference image showing the estimated reference object, the first model described in any of (1) to (10), and the second model described in any of (1) to (10). Estimation system. [Explanation of symbols]

[0143] 1 Learning system, 2 Estimation system, 10 Learning terminal, 11, 21, 31 Control unit, 12, 22, 32 Memory unit, 13, 23, 33 Communication unit, 20 Server, 30 User terminal, 14, 34 Operation unit, 15, 35 Display unit, 36 Image capture unit, CN Communication network, DB Training database, I r Training reference image, I t Training target images, 100,200 First model storage unit, 101,201 Second model storage unit, 102,202 Data storage unit, 103 Training data acquisition unit, 104 Learning unit, 203 Estimation unit, M1 First model, M10 First computation model, E11 Encoder, M12 First output model, M2 Second model, M20 Second computation model, M21 Second output model, N210 Neural network, L total Total loss, L image l Image loss, Laffine l Processing loss, L corres l Corresponding loss.

Claims

1. A training data acquisition unit that includes a training target image representing the object to be trained and a training reference image representing a training reference object as input parts, and acquires training data that includes correct information as a correct part for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object, A first model storage unit stores a first model which calculates a first training target feature of the training target image and a first training reference feature of the training reference image, and outputs first training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training target feature and the first training reference feature, A second model storage unit stores a second model which calculates a second training target feature of the training target image and a second training reference feature of the training reference image, and outputs a second training processing information for processing the training target image so that the training target pose matches the training reference pose, based on the first training processing information, the second training target feature, and the second training reference feature. A learning unit that performs training on at least one of the first model and the second model based on the aforementioned training data, A learning system that includes this.

2. The first model described above is A first computational model that calculates the first training target features based on the training target images and calculates the first training reference features based on the training reference images, A first output model that outputs the first training processing information based on the first training target feature and the first training reference feature, The learning system according to claim 1, including the following:

3. The first computational model is a trained model in which other training objects different from the training target object and the training reference object have been learned, The learning unit performs the learning of the first output model without performing the learning of the first computation model. The learning system according to claim 2.

4. The first model further includes a first encoder that reduces the dimensions of the first trained feature and the first trained reference feature calculated by the first computation model, The first output model outputs the first training processing information based on the first training target feature and the first training reference feature whose dimensions have been reduced by the first encoder. The learning system according to claim 2 or 3.

5. The second model processes the second training target feature based on the first training processing information, and outputs the second training processing information based on the processed second training target feature and the second training reference feature. A learning system according to any one of claims 1 to 3.

6. The second model described above is A second computational model that calculates the second training target feature based on the training target image and calculates the second training reference feature based on the training reference image, A second output model that outputs the second training processing information based on the first training processing information, the second training target features, and the second training reference features, A learning system according to any one of claims 1 to 3, including the following:

7. The second computational model includes a plurality of layers that compute the second training target feature and the second training reference feature, The second output model calculates intermediate second training information, which is second intermediate training information, for each layer based on the first training processing information and the second training target features and second training reference features calculated in each of the plurality of layers, and outputs the final second training processing information, which is second final training processing information. The learning system according to claim 6.

8. The aforementioned training data includes, as the correct answer information, correct machining information relating to the correct machining process, The learning unit calculates the machining loss based on the second training machining information and the correct machining information, and performs learning on at least one of the first model and the second model based on the machining loss. A learning system according to any one of claims 1 to 3.

9. The training data includes, as the correct answer information, correct answer image information relating to the processed training target image that is the correct answer. The learning unit processes the training target image based on the second training processing information, calculates the image loss based on the processed training target image and the ground truth image information, and performs training on at least one of the first model and the second model based on the image loss. A learning system according to any one of claims 1 to 3.

10. The training data includes, as the correct answer information, correct answer correspondence information relating to the correspondence between each pixel of the training target image and each pixel of the processed training target image that represents the correct answer. The learning unit processes the training target image based on the second training processing information, obtains training correspondence information relating to the correspondence between the training target image before processing and the training target image after processing, calculates the correspondence loss based on the training correspondence information and the correct answer correspondence information, and performs training on at least one of the first model and the second model based on the correspondence loss. A learning system according to any one of claims 1 to 3.

11. After learning by the learning unit described in claim 1 is completed, the system includes an estimation unit that processes the estimated target image based on the estimated target image showing the estimated target object, the estimated reference image showing the estimated reference object, the first model described in claim 1, and the second model described in claim 1, so that the estimated target pose of the estimated target object matches the estimated reference pose of the estimated reference object. Estimation system.

12. A training data acquisition step involves acquiring training data that includes, as input, a training target image representing the object to be trained and a training reference image representing a training reference object, and as a ground truth portion, ground truth information for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object. A learning step that performs learning of at least one of the following: a first model that calculates a first training target feature of the training target image and a first training reference feature of the training reference image based on the training data, and outputs first training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training target feature and the first training reference feature; and a second model that calculates a second training target feature of the training target image and a second training reference feature of the training reference image, and outputs second training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training processing information, the second training target feature, and the second training reference feature; Learning methods that include this.

13. After completing the learning steps described in claim 12, the method includes an estimation step of processing the target image so that the target pose of the target object matches the reference pose of the reference object, based on the target image showing the target object, the reference image showing the reference object, the first model described in claim 12, and the second model described in claim 12. Estimation method.

14. A training data acquisition unit that includes a training target image representing the object to be trained and a training reference image representing a training reference object as input parts, and acquires training data that includes correct information as a correct part for processing the training target image so that the training target pose of the training target object matches the training reference pose of the training reference object. A learning unit that performs learning on at least one of the following: a first model that calculates a first training target feature of the training target image and a first training reference feature of the training reference image based on the training data, and outputs first training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training target feature and the first training reference feature; and a second model that calculates a second training target feature of the training target image and a second training reference feature of the training reference image, and outputs second training processing information for processing the training target image so that the training target pose matches the training reference pose based on the first training processing information, the second training target feature, and the second training reference feature. A program that makes a computer function.

15. After learning by the learning unit according to claim 14 is completed, an estimation unit processes the estimated target image based on the estimated target image showing the estimated target object, the estimated reference image showing the estimated reference object, the first model according to claim 14, and the second model according to claim 14, so that the estimated target pose of the estimated target object matches the estimated reference pose of the estimated reference object. A program that makes a computer function.

Images