Estimation system, estimation device, and method for generating estimation model

Abstract

An estimation system includes a learning unit that generates an estimation model using a first data set including a plurality of sample data to which correct answers are given and a second data set including a plurality of sample data to which correct answers are not given, and an estimation unit that determines an estimation result. The estimation model includes an encoder that calculates a first feature amount that is independent of the first data set and the second data set and a second feature amount that is dependent on the first data set or the second data set from sample data. The learning unit is configured to learn the encoder so that the same first feature amount is calculated from arbitrary sample data for a pair of a first sample data included in the first data set and a second sample data included in the second data set to which the same correct answer as that given to the first sample data is to be given.

Description

Technical Field

[0001] This invention relates to machine learning methods suitable for practical applications. Background Technology

[0002] Due to the leap in computing power in recent years, solutions utilizing computing power, known as AI (Artificial Intelligence), are being implemented in various fields.

[0003] One such initiative is Cyber-Physical Systems (CPS). Cyber-Physical Systems recognize their connection to the real world and encompass various attempts to leverage computing power to solve social issues.

[0004] To realize cyber-physical systems, various technological elements are required, but one of them is the known unsupervised domain adaptation problem. Non-patent documents 1, 2, and 3 disclose a method called cycleGAN applied to the unsupervised domain adaptation problem.

[0005] Existing technical documents

[0006] Non-patent literature

[0007] Non-patent literature 1: Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, Trevor Darrell, "CyCADA: Cycle-Consistent Adversarial Domain Adaptation," Proceedings of the 35th International Conference on Machine Learning, PMLR 80:1989-1998, 2018, [Retrieved November 8, 2019], Internet<URL:https: / / github.com / jhoffman / cycada_release>

[0008] Non-patent literature 2: Paolo Russo, Fabio M. Carlucci, Tatiana Tommasi, Barbara Caputo, "From Source to Target and Back: Symmetric Bi-Directional Adaptive GAN," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8099-8108, [retrieved November 8, 2019], Internet<URL:http: / / openaccess.thecvf.com / content_cvpr_2018 / html / Russo_From_Source_to_CV PR_2018_paper.html>

[0009] Non-patent literature 3: Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, Russell Webb, "Learning From Simulated and Unsupervised Images Through Adversarial Training," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2107-2116, [Retrieved November 8, 2019], Internet<URL:https: / / arxiv.org / abs / 1612.07828> Summary of the Invention

[0010] The problem that the invention aims to solve

[0011] One object of the present invention is to provide a technique that, compared with the existing methods disclosed in Non-Patent Documents 1, 2 and 3, is capable of generating learned models with further improved estimation performance.

[0012] Methods for solving problems

[0013] An example of the estimation system of the present invention includes: a learning unit that generates an estimation model using a first dataset containing multiple sample data assigned a positive solution and a second dataset containing multiple sample data not assigned a positive solution; and an estimation unit that inputs estimation object data that may belong to the second dataset into the estimation model and determines the estimation result. The estimation model includes an encoder that computes a first feature quantity independent of the first and second datasets and a second feature quantity dependent on either the first or second dataset, based on the sample data. The learning unit is configured to learn the encoder such that for a pair of first sample data contained in the first dataset and second sample data contained in the second dataset that should be assigned the same positive solution as the positive solution assigned to the first sample data, the same first feature quantity is computed based on either the first or second sample data.

[0014] Based on this structure, an encoder is used that computes a first feature independent of both dataset 1 and dataset 2, and a second feature dependent on either dataset 1 or dataset 2, based on the sample data contained in each dataset. Furthermore, by focusing on the first feature rather than the second feature, the estimation accuracy of the estimation model can be improved without knowing the distribution shape of dataset 2.

[0015] The learning unit can also be configured to generate the estimation model using a learning network that acts as an adversarial network. Based on this structure, the estimation model can be learned efficiently using an adversarial network.

[0016] The learning network can also include: a first encoder / decoder consisting of a first encoder and a first decoder; and a second encoder / decoder consisting of a second encoder and a second decoder. Alternatively, during learning in the learning unit, samples from a first dataset are input into the first network configured in the order of the first encoder / decoder and the second encoder / decoder, and samples from a second dataset are input into the second network configured in the order of the second encoder / decoder and the first encoder / decoder. According to this structure, the first encoder / decoder and the second encoder / decoder are shared and their configuration order is different, thereby enabling efficient learning of the estimated model through the adversarial network.

[0017] Alternatively, the learning unit can be configured to optimize the model parameters of the first encoder, first decoder, second encoder, and second decoder by inputting samples contained in the first dataset into the first network to minimize the error between the first feature quantity output from the first encoder and the first pseudo-feature quantity output from the second encoder. The learning unit can also be configured to optimize the model parameters of the first encoder, first decoder, second encoder, and second decoder by inputting samples contained in the second dataset into the second network to minimize the error between the first feature quantity output from the second encoder and the first pseudo-feature quantity output from the first encoder. According to this structure, by minimizing the error between the first feature quantity calculated from the samples and the first pseudo-feature quantity calculated by the decoder and encoder based on the first feature quantity, the model parameters of the estimated model can be learned efficiently.

[0018] The learning unit can also be configured to learn a recognizer based on the sample data and corresponding positive solutions contained in the first dataset, which is input to the output of the encoder. The estimation model may also include a recognizer. According to this structure, learning the estimation (recognition) of the estimation model can be performed efficiently.

[0019] Another example of the estimation apparatus of the present invention includes: a storage unit that holds an estimation model generated by learning using a first dataset containing multiple sample data assigned a positive solution and a second dataset containing multiple sample data not assigned a positive solution; and an estimation unit that inputs estimation object data that may belong to the second dataset into the estimation model and determines an estimation result. The estimation model includes an encoder that calculates a first feature quantity independent of the first and second datasets, and a second feature quantity dependent on either the first or second dataset, based on the sample data. The encoder is learned such that for a pair of first sample data contained in the first dataset and second sample data contained in the second dataset that should be assigned the same positive solution as the positive solution assigned to the first sample data, the same first feature quantity is calculated based on either the first or second sample data.

[0020] Another example of the estimation method of the present invention includes the following steps: generating an estimation model using a first dataset containing multiple sample data assigned a positive solution and a second dataset containing multiple sample data not assigned a positive solution; and inputting estimation object data that may belong to the second dataset into the estimation model to determine the estimation result. The estimation model includes an encoder that, based on the sample data, calculates a first feature quantity independent of the first and second datasets, and a second feature quantity dependent on either the first or second dataset. The step of generating the estimation model includes the following steps: learning the encoder such that for a pair of first sample data contained in the first dataset and second sample data contained in the second dataset that should be assigned the same positive solution as the positive solution assigned to the first sample data, the same first feature quantity is calculated based on either the first or second sample data.

[0021] The effects of the invention

[0022] According to the present invention, compared with existing methods, it is possible to generate learned models with further improved estimation performance. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating a structural example of the robot system according to this embodiment.

[0024] Figure 2 This is a schematic diagram illustrating the processing steps for constructing and applying the estimation model in this embodiment.

[0025] Figure 3 This is a schematic diagram illustrating an example of the hardware structure of the robot system according to this embodiment.

[0026] Figure 4 This is a diagram illustrating an example of a solution to the Unsupervised Domain Adaptation problem in the relevant technology of this invention.

[0027] Figure 5 This is a schematic diagram illustrating an example of a learning network for constructing an encoder using CycleGAN in this embodiment.

[0028] Figure 6 It is used for conceptual explanation Figure 5 The diagram shown illustrates the construction of a CycleGAN-based encoder.

[0029] Figure 7 This is a schematic diagram illustrating an example of a learning network for constructing an encoder using CycleGAN in this embodiment.

[0030] Figure 8 It means that the construction utilizes Figure 7A schematic diagram of an example of a method for a main purpose recognizer of a learning network shown.

[0031] Figure 9 It shows the use of Figure 7 A schematic diagram of an example of a network in the operation stage of an encoder constructed using the learning network shown.

[0032] Figure 10 It is a flowchart showing the processing steps for generating an estimation model in the present embodiment.

[0033] Figure 11 It is a diagram for explaining an example of model parameter sharing in the estimation model of the present embodiment.

[0034] Figure 12 It is a schematic diagram of an experimental example for explaining the performance results of the estimation model of the present embodiment.

[0035] Figure 13 It shows Figure 12 A diagram of an example of performance evaluation in the experiment shown.

[0036] Figure 14 It is a schematic diagram showing a structural example of an estimation system using the estimation model of the present embodiment. Detailed implementation mode

[0037] The embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, the same or corresponding parts in the drawings are labeled with the same reference numerals and their descriptions will not be repeated.

[0038] <A. Application example>

[0039] First, an example of a scenario where the present invention is applied will be described.

[0040] Figure 1 It is a schematic diagram showing a structural example of the robot system 1 of the present embodiment. Referring to Figure 1 , the robot system 1 determines the position and / or type of the workpiece 8 through image recognition, and holds the workpiece 8 by the hand 6 disposed at the end of the arm to perform necessary operations.

[0041] As an example, the robot 2 is a multi-joint robot having a plurality of axes 4 equivalent to joints, and by rotating or moving each axis 4, the hand 6 disposed at the end can be configured at an arbitrary position and an arbitrary posture. A camera 20 is disposed near the hand 6, and by performing recognition processing on the image captured by the camera 20, the position and / or type, etc. of the workpiece 8 are determined (estimated).

[0042] In the robot system 1 of the present embodiment, an estimation model as a learned model pre-constructed by machine learning as described below is used. In order to improve the estimation accuracy of the estimation model, it is necessary to perform machine learning using a learning dataset including a plurality of teacher data.

[0043] On the other hand, it is necessary to assign correct answers (labels) to the teacher data in advance, so it is not easy to prepare a plurality of teacher data. Therefore, the following method is mostly used: workpieces with known positions and types (i.e., correct answers) are pre-configured in simulation, and the workpieces are virtually photographed (rendered or visualized), thereby preparing a plurality of images (i.e., teacher data) assigned with correct answers.

[0044] By constructing an estimation model using such a plurality of teacher data generated by simulation, the estimation accuracy can be improved.

[0045] However, in actual operation, the workpiece 8 is photographed by the camera 20. The shooting conditions of the images generated by simulation and the images actually captured by the camera 20 are not exactly the same. Therefore, in most cases, the estimation model constructed using a plurality of teacher data generated by simulation cannot be directly used in actual operation.

[0046] The present embodiment provides the following structure: an estimation model constructed using teacher data generated under a certain condition can be applied under不一定相同的条件下,运用使用在某个条件下生成的教师数据构建的估计模型。

[0047] Figure 2 is a schematic diagram showing the processing steps of constructing and applying the estimation model of the present embodiment. Refer to Figure 2 , first, an estimation model 40 is constructed using a dataset including images generated by the simulation device 200 (hereinafter, also referred to as the "source dataset 30").

[0048] The constructed estimation model 40 is adapted by using a dataset including images used in actual operation (hereinafter, also referred to as the "target dataset 50") to generate an estimation model 60. In actual operation, by inputting data (hereinafter, also referred to as the "estimation target data 70") to the generated estimation model 60, an estimation result 62 is output. The estimation target data 70 corresponds to sample data that may belong to the target dataset 50.

[0049] In addition, it is not necessarily necessary to construct the estimation model 40, and the estimation model 60 can also be directly constructed based on the source dataset 30 and the target dataset 50.

[0050] Hereinafter, the detailed structure and processing of the robot system 1 as a specific application example of the present invention will be described.

[0051] <B. Hardware Structure of Robot System> It should be noted that the part "不一定相同的条件下" in the original text seems to be an incomplete or incorrect expression. I have translated it as "under conditions that are not necessarily the same" according to the context as accurately as possible. If there are any specific corrections or clarifications for this part, please let me know.

[0052] Next, regarding Figure 1 An example of the hardware structure of the robot system 1 shown will be explained.

[0053] Figure 3 This is a schematic diagram illustrating an example of the hardware structure of the robot system 1 according to this embodiment. (Refer to...) Figure 3 The robot system 1 includes a robot 2 and a robot controller 100 that controls the robot 2.

[0054] In addition to the camera 20, the robot 2 also has a set of servo drives 12 and motors 14 corresponding to the number of axes.

[0055] The robot controller 100 is the device constituting the estimation system of this embodiment, controlling the robot 2 based on images captured by the camera 20. More specifically, the robot controller 100 performs recognition processing on the images captured by the camera 20 and outputs instructions to one or more servo drives 12 according to the recognition results. The servo drives 12 are supplied with power according to the instructions, thereby driving the corresponding motors 14 to rotate, and the joints or arms of the robot 2 mechanically coupled to the motors 14 to perform actions.

[0056] The robot controller 100 is typically implemented using a computer that follows a general architecture (e.g., an industrial personal computer based on a general-purpose personal computer).

[0057] The robot controller 100 includes a processor 102, a main memory 104, a storage device 110, a communication interface 122, an input unit 124, an output unit 126, a camera interface 128, and a motor interface 130 as components.

[0058] The processor 102 consists of a CPU (Central Processing Unit), an MPU (Microprocessing Unit), and a GPU (Graphics Processing Unit). The processor 102 can have a multi-core architecture, or multiple processors 102 can be configured.

[0059] The main memory 104 is composed of volatile storage devices such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The storage device 110 is composed of non-volatile storage devices such as HDD (Hard Disk Drive) or SSD (Solid State Drive). The processor 102 reads various programs stored in the storage device 110, expands and executes them in the main memory 104, thereby realizing the various processes described later.

[0060] In addition to the OS 112 used to implement basic functions, the storage 110 also stores the machine learning program 114, the model parameters 116 specifying the estimated model, and the control application 118 used to control the robot 2. The storage 110 is equivalent to a storage unit for holding the estimated model. In addition, the storage 110 sometimes also stores the source dataset 30.

[0061] The processor 102 performs learning processing by executing the machine learning program 114 to generate an estimation model 60. Additionally, the processor 102 functions as an estimation unit by executing a control application 118, which inputs the estimation object data 70 into the estimation model 60 to determine the estimation result.

[0062] Communication interface 122 mediates data exchange with other devices via any network.

[0063] The input unit 124 consists of a keyboard, mouse, etc., and accepts user operations. The output unit 126 consists of a display, various indicators, printer, etc., and outputs the processing results from the processor 102.

[0064] The camera interface 128 receives images captured by the camera 20 and outputs the required commands to the camera 20.

[0065] The motor interface 130 outputs the required instructions to the servo driver 12 according to the instructions from the processor 102.

[0066] The program for the robot controller 100 can be installed via a computer-readable recording medium (e.g., an optical recording medium such as a DVD (Digital Versatile Disc)) or downloaded from a server device on a network. Furthermore, the functions provided by the robot controller 100 in this embodiment are sometimes implemented as part of a module provided by the OS.

[0067] exist Figure 3In [the figure], a structural example of providing functions required for the robot controller 100 by executing a program by the processor 102 is shown, but a dedicated hardware circuit (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), etc.) can also be used to install a part or all of these provided functions.

[0068] <C. Unsupervised Domain Adaptation Problem>

[0069] First, the Unsupervised Domain Adaptation problem will be described. As shown in Figure 1 and Figure 2 it is assumed that there are data sets (source data set 30 and target data set 50) having different biases (trends) from each other. Here, the bias is typically caused by the difference between the environment for obtaining the source data set 30 (hereinafter, also referred to as "source domain") and the environment for obtaining the target data set 50 (hereinafter, also referred to as "target domain").

[0070] The source data set 30 is basically composed of a plurality of sample data to which correct answers are given. However, the source data set 30 may also include sample data to which no correct answer is given. The source data set 30 corresponds to the first data set including a plurality of sample data to which correct answers are given.

[0071] The target data set 50 is basically composed of a plurality of sample data to which no correct answer is given. However, the target data set 50 may also include sample data to which a correct answer is given. The target data set 50 corresponds to the second data set including a plurality of sample data to which no correct answer is given.

[0072] Referring to Figure 1 and Figure 2 in the shown example, the environment generated by simulation is the source domain, and the environment actually captured by the camera 20 corresponds to the target domain.

[0073] The learning data set for the source domain is given by a set of (x s , y s ), and the learning data set for the target domain is given only by (x t ). Here, x s and x t respectively represent the input vectors of the sample data included in the source domain and the target domain, and y s r refers to the correct answer (label) given to the corresponding x s . That is, it is assumed that no correct answer (label) is given to the sample data included in the target domain.

[0074] Under these conditions, we construct a system capable of estimating the sample data set X that should belong to the target domain. t The correct answer Y is given t The learned model is the goal of the Unsupervised Domain Adaptation problem. That is, the goal is to find the correct domain adaptation model when no positive solution Y exists. t Under the given conditions, construct a system capable of calculating the distribution Pr(y|X). t The learned model.

[0075] The solution to the Unsupervised Domain Adaptation problem in related technologies is to construct sample data x that will become the same estimate. s ~X s and sample data x t ~X t The encoder E(z|x) projects onto the same feature quantity z.

[0076] Here, it is impossible to determine the pair (x) that are the same estimate. s x t ), that is, which pair (x) s x t It is unknown whether the estimated values ​​will be the same. Therefore, instead of learning the encoder in a way that each pair becomes the same feature, the encoder is learned as a group of components that become the same feature. That is, the encoder is learned as a group such that the distribution Pr(y|x) becomes the same feature. s ~X s ) and distribution Pr(y|x t ~X t The shapes are consistent.

[0077] Figure 4 This is a diagram illustrating an example of a solution to the Unsupervised Domain Adaptation problem in the related technology of this invention. (Refer to...) Figure 4 Imagine a source dataset 30 and a target dataset 50. There is an unknown bias (or an unknown difference in bias) between the source dataset 30 and the target dataset 50.

[0078] Source dataset 30 (sample data group X) s It includes one or more sample data 32 that have been assigned the first positive solution (label) and one or more sample data 34 that have been assigned the second positive solution (label).

[0079] On the other hand, the target dataset 50 (sample data group X) tIt includes one or more sample data 52 to be given the first correct answer (label), and one or more sample data 54 to be given the second correct answer (label). However, the correct answers of the sample data included in the target data set 50 are unknown.

[0080] Here, attention is focused on the distribution when the sample data included in the source data set 30 and the target data set 50 are projected onto the feature quantity space of the encoder. The sample data 32 and 34 included in the source data set 30 are projected onto regions 36 and 38 respectively, and the sample data 52 and 54 included in the target data set 50 are projected onto regions 56 and 58 respectively.

[0081] The encoder is learned so that the distributions of regions 36 and 38 are consistent with the shapes of the distributions of regions 56 and 58. On this basis, the main purpose recognizer 64 for recognizing regions 36 and 38 corresponding to the first correct answer (label) and regions 56 and 58 given the second correct answer (label) is also learned. An estimation model 60 is generated from the encoder and the main purpose recognizer 64 (refer to Figure 6 ).

[0082] Even if there is a source data set 30 including sample data given correct answers, for data with different biases, the estimation accuracy will decrease. However, by applying the encoder constructed by the method as described above, such a decrease in estimation accuracy can be prevented.

[0083] <D. Solution means>

[0084] (d1: Discovery of the problem)

[0085] The inventors of the present application conducted in-depth research and as a result, discovered the essential problem in the method as described above and the solution means for this problem. That is, in the above method, it is assumed that the shapes of the distributions Pr(y|x s ~X s ) and Pr(y|x t ~X t ) are consistent, but the actual correct answer Y t is unknown. Therefore, there is an essential problem that such an assumption holds only accidentally except in special cases.

[0086] For example, as described by referring to the above Figure 1 and Figure 2 , it is difficult to make the frequency distributions of the correct answers consistent between the images generated by simulation and the images actually captured by the camera 20.

[0087] As a result, as Figure 4As shown, it is possible to generate sample data 52 that should have been projected onto region 56 but is incorrectly projected onto region 58 (incorrectly projected sample data 53), or sample data 54 that should have been projected onto region 58 but is incorrectly projected onto region 56 (incorrectly projected sample data 55). Furthermore, since the sample data contained in the target dataset 50 are not assigned a correct solution, it is theoretically impossible to detect such incorrect projections.

[0088] (d2: Summary of solutions)

[0089] In the solution of this embodiment, regardless of the distribution Pr(y|x) s ~X s ) and distribution Pr(y|x t ~X t Whether the shapes of the samples x and x are consistent, the encoder E(z|x) is constructed through learning. This encoder E(z|x) will process the sample data x. s ~X s and sample data x t ~X t Projected onto the same feature quantity z.

[0090] More specifically, using a method called CycleGAN, virtual representations are generated that correspond to the sample data x. s First-order pseudo-sample data x' of the same estimate y t Then, minimize the error so that, based on the sample data x s The calculated feature quantity z and the data based on the first pseudo-sample data x' t The calculated characteristic z' represents a similar value.

[0091] In this method, the encoder is learned with the aim of making pairs that will produce the same estimate consistent. That is, using sample data x that will produce the same estimate... s ~X s and sample data x t ~X t The encoder is learned by projecting onto the same feature quantity z. By employing this learning method, even if the distribution Pr(y|x) is... s ~X s ) and distribution Pr(y|x t ~X t Different shapes can also improve estimation accuracy.

[0092] (d3: Basic learning network)

[0093] Figure 5 This is a schematic diagram illustrating an example of a learning network 80 used to construct an encoder via CycleGAN in this embodiment. (Refer to...) Figure 5 The learning network 80 is an adversarial network, which is a network composed of two sets of encoders and decoders. More specifically, the learning network 80 includes a set of encoder 82 and decoder 84, and a set of encoder 86 and decoder 88.

[0094] Encoder 82 is a learned model (StoZ) used to compute feature quantities based on sample data contained in source dataset 30, and decoder 84 is a learned model (ZtoT) used to estimate sample data contained in target dataset 50 based on feature quantities.

[0095] Similarly, encoder 86 is a learned model (TtoZ) for calculating feature quantities based on sample data contained in target dataset 50, and decoder 88 is a learned model (ZtoS) for estimating sample data contained in source dataset 30 based on feature quantities.

[0096] Sample data x contained in source dataset 30 s ~X s The input is fed into the group of encoder 82 and decoder 84, and its output is fed into the group of encoder 86 and decoder 88.

[0097] Similarly, the sample data x contained in the target dataset 50 t ~X t The input is fed into the group of encoder 86 and decoder 88, and its output is fed into the group of encoder 82 and decoder 84.

[0098] Furthermore, although each of the learning networks 80 contains two encoders 82 and 86 and two decoders 84 and 88, they share model parameters. Therefore, encoders and decoders labeled with the same reference number are identical models.

[0099] By using the sample data x contained in the source dataset 30 s The data is input into encoder 82 to calculate the sample data x. s Feature quantity z s By using the characteristic quantity z s The input is fed into decoder 84, which calculates the first-order pseudo-sample data x' that simulates the sample data belonging to the target dataset 50. t .

[0100] Furthermore, by using 1st pseudo-sample data x' t The data is input into encoder 86 to calculate the first pseudo-sample data x'. t Feature quantity z' t By using the characteristic quantity z' t The input is fed into decoder 88, which calculates the pseudo-sample data x” that simulates the sample data belonging to the source dataset 30.s .

[0101] On the other hand, by using the sample data x contained in the target dataset 50 t The input is given to encoder 86 to calculate the sample data x. t Feature quantity z t By using the characteristic quantity z t The input is fed into decoder 88, which calculates the pseudo-sample data x' that simulates the sample data belonging to the source dataset 30. s .

[0102] Furthermore, by using 1st pseudo-sample data x' s The data is input into encoder 82 to calculate the first pseudo-sample data x'. s Feature quantity z' s By using the characteristic quantity z' s The input is fed into decoder 84, which calculates the second-order pseudo-sample data x” that simulates the sample data belonging to the target dataset 50. t .

[0103] As such an adversarial network, network 80 learns through the following steps (1-1), (1-2), (2-1), (2-2), (3-1), and (3-2). Furthermore, the execution of steps (3-1) and (3-2) is arbitrary.

[0104] (1-1) Take the sample data x belonging to the source dataset 30 s Compared with the calculated 2nd pseudo-sample data x” s The error between them is set as the first-to-first loss function, and the model parameters of encoders 82 and 86 and decoders 84 and 88 are optimized in a way that minimizes the value of the first-to-first loss function.

[0105] (1-2) Take the sample data x belonging to the target dataset 50 t Compared with the calculated 2nd pseudo-sample data x” t The error between them is set as the first and second loss functions, and the model parameters of encoders 82 and 86 and decoders 84 and 88 are optimized in a way that minimizes the value of the first and second loss functions.

[0106] (2-1) Using a discriminator (not shown), distinguish the sample data x that belong to the source dataset 30. s and 1st pseudo-sample data x' s The function is used as the second-to-first loss function, and the model parameters of encoders 82 and 86 and decoders 84 and 88 are optimized by the minmax strategy in a way that minimizes the value of the second-to-first loss function (i.e., in an indistinguishable way).

[0107] (2-2) Using the same recognizer as (2-1), sample data x that can distinguish those belonging to the target dataset 50 will be able to be distinguished. t and 1st pseudo-sample data x' t The function is used as the second-second loss function, and the model parameters of encoders 82 and 86 and decoders 84 and 88 are optimized by the minmax strategy in a way that minimizes the value of the second-second loss function (i.e., in an indistinguishable way).

[0108] Here, identity loss can also be used as the loss function to further optimize the model parameters. More specifically, it can be learned through steps (3-1) and (3-2) as follows.

[0109] (3-1) Take the sample data x contained in the source dataset 30 s , and the sample data x s The pseudo-sample data x' input to encoder 86 and decoder 88 is calculated. s The error between them is set as the 3-1 loss function, and the model parameters of encoder 86 and decoder 88 are optimized in a way that minimizes the value of the 3-1 loss function.

[0110] (3-2) Take the sample data x contained in the target dataset 50 t , and the sample data x t The pseudo-sample data x' input to encoder 82 and decoder 84 is calculated. t The error between them is set as the third-second loss function, and the model parameters of encoder 82 and decoder 84 are optimized in a way that minimizes the value of the third-second loss function.

[0111] Through the above steps, the model parameters of Network 80, used as a learning network for adversarial networks, are optimized.

[0112] Figure 6 It is used for conceptual explanation Figure 5 The diagram shown illustrates the construction of the cycleGAN-based encoder. (Refer to...) Figure 6 Imagine a source dataset 30 and a target dataset 50. There is an unknown bias (or an unknown difference in bias) between the source dataset 30 and the target dataset 50.

[0113] Source dataset 30 (sample data group X) s It includes one or more sample data 32 that have been assigned the first positive solution (label) and one or more sample data 34 that have been assigned the second positive solution (label).

[0114] On the other hand, the target dataset 50 (sample data group X) t The target dataset 50 contains one or more sample data points 52 that should be assigned the first correct solution (label), and one or more sample data points 54 that should be assigned the second correct solution (label). However, the correct solutions for the sample data contained in the target dataset 50 are unknown.

[0115] Here, the encoder is trained so that the pairs of sample data 32 and sample data 52, which are estimated with the first positive solution (label) in the sample data contained in the source dataset 30 and the target dataset 50, become the same feature quantity 66.

[0116] Similarly, the encoder is trained so that the pairs of sample data 34 and sample data 54, which are estimated by the second positive solution (label), in the sample data contained in the source dataset 30 and the target dataset 50, become the same feature quantity 68.

[0117] Right now, Figure 5 The learning network 80 shown focuses on the pairs of sample data contained in the source dataset 30 and sample data contained in the target dataset. Starting from the sample data of one side and passing through the feature space, it computes the pseudo-sample data (first-order pseudo-sample data) of the other side, and starting from the pseudo-sample data of the other side (first-order pseudo-sample data) and passing through the feature space, it computes the pseudo-sample data (second-order pseudo-sample data) of one side. Through such bidirectional projection (i.e., computation along bidirectional paths), adversarial learning is performed with the aim of ensuring that the sample data contained in any dataset is the same as the corresponding second-order pseudo-sample data.

[0118] Thus, in Figure 5 The learning network 80 shown is trained to make the encoder consistent with pairs that will produce the same estimate. That is, it learns from sample data x that will produce the same estimate. s ~X s and sample data x t ~X t The encoder is learned with the aim of projecting onto the same feature quantity z. By adopting this learning method, it is not necessary to use the distribution Pr(y|x) s ~X s ) and distribution Pr(y|x t ~X t If the shape is consistent with the shape, a high-precision encoder can be built.

[0119] (d4: Improve the network for learning)

[0120] The inventors of this application conducted further in-depth research and found that... Figure 5 The learning network 80 shown can be further improved in the following aspects.

[0121] Figure 7 This is a schematic diagram illustrating an example of a learning network 90 used to construct an encoder via CycleGAN in this embodiment. (Refer to...) Figure 7 The learning network 90 is an adversarial network, which is a network composed of two encoders and decoders. The estimated model 60 is generated using the learning network 90 as an adversarial network.

[0122] More specifically, the learning network 90 includes a group of encoder 92 and decoder 94 (encoder / decoder), and a group of encoder 96 and decoder 98 (encoder / decoder).

[0123] Figure 7 The learning network shown is 90 and Figure 5 Compared to the learning network 80 shown, the difference lies in the fact that encoders 92 and 96 output two feature quantities z and ζ. The reason for setting the encoding results of encoders 92 and 96 as two feature quantities z and ζ is as follows.

[0124] In the case where variable b is used as teacher data to learn how to estimate variable b based on variable a, regardless of whether there is a dependency relationship between variables a and b, variable a depends on variable b. For Figure 5 The learning network 80 shown should have the same value for feature quantity z regardless of the source and target domains. However, in reality, due to the existence of features z predicting sample data x... s and sample data x t The part, therefore, depends on the sample data x according to the prediction path. s Or sample data x t .

[0125] Therefore, by separating the feature quantity ζ that preserves the dependency and the feature quantity z that is independent of the prediction path, (z, ζ) as a whole, although dependent on the domain, can be set to be domain-independent within the individual feature quantity z. Thus, domain independence can be achieved in minimizing the loss function in sections 4-1 and 4-2 described below.

[0126] In other words, we can consider the feature ζ to represent the information of the content, and the feature z to represent the information of each domain.

[0127] Thus, encoders 92 and 96 calculate, based on the sample data, the feature quantity z (feature quantity z) independent of the source dataset 30 and the target dataset 50. s and z t ), and the feature quantity ζ that depends on the source dataset 30 or the target dataset 50.

[0128] Then, encoders 92 and 96 are trained to train the sample data x contained in the source dataset 30. s The values ​​contained in the target dataset 50 should be assigned to the sample data x. s Sample data x with the same correct answer (i.e., those that become the same estimate) t Yes, according to the sample data x s and sample data x t Each of them computes the same feature z (independent feature z). s and z t ).

[0129] That is, using sample data x that will become the same estimate. s ~X s and sample data x t ~X t Projected onto the same feature z (independent feature z) s and z t The encoder is learned for the purpose of ).

[0130] Thus, in learning network 90, the focus is on the feature quantity z (feature quantity z) that does not depend on the source dataset 30 and the target dataset 50. s and z t ), to optimize the model parameters.

[0131] More specifically, using a method called CycleGAN, virtual representations are generated that correspond to the sample data x. s First-order pseudo-sample data x' of the same estimate y t Then, minimize the error so that, based on the sample data x s The calculated feature quantity z and the data based on the first pseudo-sample data x' t The calculated characteristic z' represents a similar value.

[0132] In the learning process using the learning network 90, the sample data x contained in the source dataset 30 s The input is fed into a network (first network) configured in the order of the group of encoder 92 and decoder 94 (first encoder / decoder) and the group of encoder 96 and decoder 98 (second encoder / decoder).

[0133] On the other hand, the sample data x contained in the target dataset 50 t The input is fed into a network (second network) configured in the order of the group of encoder 96 and decoder 98 (second encoder / decoder) and the group of encoder 92 and decoder 94 (first encoder / decoder).

[0134] Figure 7The learning network 90 shown as an adversarial network learns through the following steps (1-1), (1-2), (2-1), (2-2), (3-1), (3-2), (4-1), (4-2), (5). Furthermore, the execution of steps (3-1) and (3-2) is arbitrary.

[0135] (1-1) Take the sample data x belonging to the source dataset 30 s Compared with the calculated 2nd pseudo-sample data x” s The error between them is set as the first-to-first loss function, and the model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized in a way that minimizes the value of the first-to-first loss function.

[0136] (1-2) Take the sample data x belonging to the target dataset 50 t Compared with the calculated 2nd pseudo-sample data x” t The error between them is set as the first and second loss functions, and the model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized in a way that minimizes the value of the first and second loss functions.

[0137] (2-1) Using a distinguisher (not shown), the sample data x that can be distinguished from those belonging to the source dataset 30 will be... s and 1st pseudo-sample data x' s The function is used as the second-to-first loss function, and the model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized by the minmax strategy in a way that minimizes the value of the second-to-first loss function (i.e., in an indistinguishable way).

[0138] (2-2) Using the same recognizer as (2-1), sample data x that can distinguish those belonging to the target dataset 50 will be able to be distinguished. t and 1st pseudo-sample data x' t The function is used as the second-second loss function, and the model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized by the minmax strategy in a way that minimizes the value of the second-second loss function (i.e., in an indistinguishable way).

[0139] Here, identity loss can also be used as the loss function to further optimize the model parameters. More specifically, it can be learned through steps (3-1) and (3-2) as follows.

[0140] (3-1) Take the sample data x contained in the source dataset 30 s , and the sample data x s The pseudo-sample data x' is input into encoder 96 and decoder 98 and calculated. sThe error between them is set as the 3-1 loss function, and the model parameters of encoder 96 and decoder 98 are optimized in a way that minimizes the value of the 3-1 loss function.

[0141] (3-2) Take the sample data x contained in the target dataset 50 t , and the sample data x t The pseudo-sample data x' input to encoder 92 and decoder 94 is calculated. t The error between them is set as the 3-2 loss function, and the model parameters of encoder 92 and decoder 94 are optimized in a way that minimizes the value of the 3-2 loss function.

[0142] Next, we focus on learning the feature z.

[0143] (4-1) Regarding the sample data x belonging to the source dataset 30 s The feature quantity z output from encoder 92 s The pseudo-feature quantity z' output from the encoder 96 located in the later stage t The error between them is set as the 4-1 loss function, and the model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized in a way that minimizes the value of the 4-1 loss function.

[0144] (4-2) Regarding the sample data x belonging to the target dataset 50 t The feature quantity z output from encoder 96 t The pseudo-feature quantity z' output from the encoder 92 located in the later stage s The error between them is set as the 4-2 loss function, and the model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized in a way that minimizes the value of the 4-2 loss function.

[0145] Then, the learning of the subject-object recognizer 64 is carried out.

[0146] (5) Regarding the sample data x belonging to source dataset 30 s The feature quantity z output from encoder 92 s The estimated value y is input to the primary target recognizer 64 and outputs the sample data x. s Correct solution y s The error between them is set as the fifth loss function, and the model parameters of encoder 92 and main target recognizer 64 are optimized in a way that minimizes the value of the fifth loss function. Thus, based on the sample data x contained in source dataset 30... s And the corresponding positive solution, for the feature quantity z input from encoder 92 output. s The primary target identifier 64 is used for learning.

[0147] Alternatively, the sample data x contained in the target dataset 50 can also be used. t The processing is performed on the sample data in which the correct solution is assigned (5). That is, the main target recognizer 64 can also learn based on the sample data contained in each dataset (source dataset 30 and / or target dataset 50) and the corresponding correct solution.

[0148] Figure 8 It means that the construction utilizes Figure 7 This is a schematic diagram illustrating an example of a method for learning the subject recognition function 64 using network 90. ​​(Refer to...) Figure 8 The processing (5) is performed with regard to the encoder 92 and the main target identifier 64.

[0149] Through the above steps, the model parameters of Network 90, used as a learning network for adversarial networks, are optimized.

[0150] Figure 9 It means to utilize Figure 7 This diagram illustrates an example of a network in the application phase of an encoder constructed using network 90. ​​(Refer to...) Figure 9 In the application phase, the network consisting of encoder 96 and main target recognizer 64 is used as the estimation model. That is, the estimated value y is calculated by inputting the feature quantity z (without using feature quantity ζ) calculated by inputting arbitrary sample data x into encoder 96 into main target recognizer 64.

[0151] Figure 10 This is a flowchart illustrating the processing steps for generating the estimation model in this embodiment. Typically, Figure 10 The steps shown are implemented by processor 102 executing machine learning program 114 (see reference). Figure 3 ).

[0152] Reference Figure 10 Prepare the source dataset 30 and the target dataset 50 (step S100).

[0153] Processor 102 will constitute Figure 7 The learning shown is initialized with model parameters of encoders 92, 96 and decoders 94, 98 of network 90 (step S102). Then, processor 102 performs optimization processing of the model parameters as follows.

[0154] Processor 102 makes the sample data x belonging to source dataset 30 s Compared with the calculated 2nd pseudo-sample data x” s The model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized in a way that minimizes the error between them (step S104). Additionally, processor 102 optimizes the sample data x belonging to the target dataset 50.t Compared with the calculated 2nd pseudo-sample data x” t The model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized in a way that minimizes the error between them (step S106).

[0155] Processor 102 determines whether the optimization processes in steps S104 and S106 have been executed repeatedly a predetermined number of times (step S108). If the optimization processes in steps S104 and S106 have not been executed repeatedly a predetermined number of times ("No" in step S108), then the processes in steps S104 and S106 are repeated.

[0156] If the optimization processes in steps S104 and S106 are repeated a predetermined number of times ("Yes" in step S108), then processor 102 uses the recognizer to identify the sample data x belonging to source dataset 30. s With 1st pseudo-sample data x' s In a manner indistinguishable from each other, the model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized (step S110). Additionally, processor 102 uses a recognizer to analyze sample data x belonging to the target dataset 50. t With 1st pseudo-sample data x' t In a manner in which there is no difference between them, the model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized (step S112).

[0157] Processor 102 determines whether the optimization processes in steps S110 and S112 have been executed repeatedly a predetermined number of times (step S114). If the optimization processes in steps S110 and S112 have not been executed repeatedly a predetermined number of times ("No" in step S114), then the processes in steps S110 and S112 are repeated.

[0158] If the optimization processes in steps S110 and S112 are repeated a predetermined number of times ("yes" in step S114), then the processor 102 will make the sample data x contained in the source dataset 30... s , and the sample data x s The pseudo-sample data x' is input into encoder 96 and decoder 98 and calculated. s The model parameters of encoder 96 and decoder 98 are optimized to minimize the error between them (step S116). Additionally, processor 102 optimizes the sample data x contained in the target dataset 50. t , and the sample data x t The pseudo-sample data x' input to encoder 92 and decoder 94 is calculated. tThe model parameters of encoder 92 and decoder 94 are optimized by minimizing the error between them (step S118).

[0159] Processor 102 determines whether the optimization processes in steps S116 and S118 have been executed repeatedly a predetermined number of times (step S120). If the optimization processes in steps S116 and S118 have not been executed repeatedly a predetermined number of times ("No" in step S120), then the processes in steps S116 and S118 are repeated.

[0160] If the optimization processes in steps S116 and S118 are repeated a predetermined number of times ("Yes" in step S120), then the processor 102 will target the sample data x belonging to the source dataset 30. s So that the feature quantity z output from encoder 92 s The pseudo-feature quantity z' output from the encoder 96 located in the later stage t The model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized by minimizing the error between them (step S122). Additionally, processor 102 optimizes the sample data x belonging to the target dataset 50. t So that the feature quantity z output from encoder 96 t The pseudo-feature quantity z' output from the encoder 92 located in the later stage s The model parameters of encoders 92 and 96 and decoders 94 and 98 are optimized by minimizing the error between them (step S124).

[0161] Processor 102 determines whether the optimization processes in steps S122 and S124 have been executed repeatedly a predetermined number of times (step S126). If the optimization processes in steps S122 and S124 have not been executed repeatedly a predetermined number of times ("No" in step S126), then the processes in steps S122 and S124 are repeated.

[0162] If the optimization processes in steps S122 and S124 are repeated a predetermined number of times ("Yes" in step S126), then the processor 102 will target the sample data x belonging to the source dataset 30. s So that the feature quantity z output from encoder 92 s The estimated value y is input to the primary target recognizer 64 and outputs the sample data x. s Correct solution y s The model parameters of encoder 92 and main target recognizer 64 are optimized by minimizing the error between them (step S128).

[0163] Processor 102 determines whether the optimization process in step S128 has been executed repeatedly a predetermined number of times (step S130). If the optimization process in step S128 has not been executed repeatedly a predetermined number of times ("No" in step S130), then the process in step S128 is performed repeatedly.

[0164] If the optimization process in step S128 is repeated a predetermined number of times ("Yes" in step S130), then the processor 102 outputs the network consisting of the encoder 96 and the main target recognizer 64 as the estimated model (step S132).

[0165] The generation process of the estimation model is now complete after the above steps. Furthermore, the estimation process using the generated estimation model can be applied.

[0166] (d5: Sharing of model parameters)

[0167] It is also possible to share a portion of the model parameters of the estimation model 40 (learning network 90).

[0168] Figure 11 This is a diagram illustrating an example of model parameter sharing in the estimation model of this embodiment. (Refer to...) Figure 11 For example, model parameters related to the calculation of feature quantity ζ can be shared between the group of encoder 92 and decoder 94 (encoder / decoder) and the group of encoder 96 and decoder 98 (encoder / decoder). Additionally, for ease of illustration, the parts related to the calculation of feature quantity z are not illustrated.

[0169] exist Figure 11 In the example shown in (A), a portion of the model parameters are shared between encoder 92 and encoder 96.

[0170] More specifically, encoder 92 includes shared parameters 921 defining internal processing on the input side and independent parameters 922 defining internal processing on the output side. Similarly, encoder 96 includes shared parameters 961 defining internal processing on the input side and independent parameters 962 defining internal processing on the output side.

[0171] In the case where encoders 92 and 96 are constructed from neural networks, the independent parameters 922 and 962 are equivalent to specifying the feature quantity ζ(ζ) in the final layer. s , ζ t The output portion of the encoder 92. Independent parameters 922 and 962 have independent values ​​in encoder 92 and encoder 96, respectively. On the other hand, shared parameters 921 and 961 on the input side are shared with each other.

[0172] In addition, Figure 11In the example shown in (A), a part of the model parameters can also be shared between the decoder 94 and the decoder 98.

[0173] More specifically, the decoder 94 includes independent parameters 941 that define the internal processing on the input side, and shared parameters 942 that define the internal processing on the output side. Similarly, the decoder 98 includes independent parameters 981 that define the internal processing on the input side, and shared parameters 982 that define the internal processing on the output side.

[0174] In the case where the decoder 94 and the decoder 98 are constituted by neural networks, the independent parameters 941 and the independent parameters 981 correspond to parts that absorb the differences in the input feature amounts ζ (ζ s , ζ t ). The independent parameters 941 and the independent parameters 98 have independent values in the decoder 94 and the decoder 98, respectively. On the other hand, the shared parameters 942 on the output side and the shared parameters 98 are shared with each other.

[0175] In Figure 11 the example shown in (B), the model parameters related to the feature amounts ζ (ζ s , ζ t ) are shared between the decoder 94 and the decoder 98. More specifically, the decoder 94 has shared parameters 944, and the decoder 98 has shared parameters 984. The shared parameters 944 and the shared parameters 984 are shared with each other. The shared parameters 944 and the shared parameters 984 are adapted to receive either one of the feature amounts ζ s and ζ t . That is, Figure 11 the structure shown in (B) is equivalent to a structure in which, in the structure shown in (A) of Figure 11 , the independent parameters 941 and the independent parameters 981 are also deformed in a shared manner with each other.

[0176] Thus, by sharing the model parameters related to the feature amount ζ, learning of the estimation model can be performed more efficiently.

[0177] In addition, by sharing the model parameters, the amount of information for determining the output of the feature amounts ζ (ζ s , ζ t ) can be increased. As a result, for inputs having the same estimated value y, it is easy to output the same feature amount z.

[0178] <E. Experimental Example of Performance Results>

[0179] Next, an experimental example of the performance results of the estimation model of the present embodiment will be described.

[0180] Figure 12It is a schematic diagram of an experimental example for explaining the performance results of the estimation model of the present embodiment. In Figure 12 In the illustrated experimental example, the object is the estimation task of human poses. The estimation task of human poses is a type of regression task from images.

[0181] Specifically, the Depth images generated by rendering (304) any pose included in the known motion capture data set 302 are used as the source data set 308. The correct solution 306 (label) of the joint position is manually assigned to each Depth image. The estimation result of the Panoptic Segmentation task is the type and position of the joint positions included in the input image.

[0182] On the other hand, the target data set 314 is composed of Depth images actually captured by the stereo camera 312 of the user 310.

[0183] An estimation model corresponding to the Panoptic Segmentation task is constructed.

[0184] Figure 13 It is a diagram showing Figure 12 an example of the performance evaluation in the illustrated experiment. Figure 13 The illustrated performance evaluation indicates the probability that the estimation result (joint position) of the input estimation target data with respect to the correct solution is within 10 pixels in the 2D space when the estimation target data is input into the estimation model. Figure 13 The illustrated performance evaluation evaluates the deviation (2Dpose) on the image.

[0185] In Figure 13 In the illustrated performance evaluation, a method such as "CyCADA" disclosed in Non-Patent Document 1 is used as a baseline. Referring to Figure 13 it can be seen that the estimation model of the present embodiment has higher estimation accuracy for any part compared with the baseline.

[0186] Thus, it can be seen that by adopting the estimation model and its learning method of the present embodiment, higher estimation performance can be achieved compared with the related art.

[0187] <F. Modified Example>

[0188] (f1: Distributed Configuration)

[0189] Although the structure in which the robot controller 100 of the above embodiment executes the process of collecting the target data set 50 and the process of generating the estimation model 60 is illustrated, it is not limited to such a structure, and any structure can be adopted.

[0190] Figure 14This is a schematic diagram illustrating an example of the structure of the estimation system utilizing the estimation model in this embodiment.

[0191] Figure 14 (A) shows a structural example of the robot controller 100 performing a process 150 of collecting source dataset 30 and target dataset 50, a process 160 of generating an estimation model 60, and an estimation process using the estimation model 60.

[0192] Figure 14 (B) illustrates a structural example of collaboration between a robot controller 100 and an external device 250, such as a server. In this structural example, the robot controller 100 performs a process 150 of collecting a source dataset 30 and a target dataset 50, as well as an estimation process using an estimation model 60, while the external device 250 performs a process 160 of generating the estimation model 60.

[0193] Figure 14 (C) also shows a structural example of the collaboration between the robot controller 100 and external devices 250 such as servers. In this structural example, the external device 250 performs a process 150 of collecting source dataset 30 and target dataset 50, and a process 160 of generating an estimation model 60, while the robot controller 100 performs an estimation process using the estimation model 60.

[0194] in addition, Figure 14 Several typical structural examples are shown, but the scope of the present invention is not limited to these examples. Any installation method can be adopted depending on the required requirements, specifications, and purpose.

[0195] (f2: Data set and estimated object data)

[0196] exist Figure 1 as well as Figure 2 The robot system shown illustrates an example of generating a source dataset from images generated through simulation (CG images) and a target dataset from images actually captured by a camera (live-action images), but it is not limited to this; any information can be used as the dataset. Furthermore, multiple source datasets and / or multiple target datasets can also be used.

[0197] For example, for either a CG image or a live-action image, it is possible to generate or obtain either a 2D color image (hereinafter also referred to as "RGB image") or a depth image. In this case, it is possible to prepare a first source dataset consisting of CG RGB images (with forward resolution) and a second source dataset consisting of CG depth images (with forward resolution), and it is possible to prepare a first target dataset consisting of live-action RGB images (without forward resolution) and a second source dataset consisting of live-action depth images (without forward resolution).

[0198] Using such a source data set and a target data set, an estimation model can be constructed that takes a captured RGB image or Depth image as estimation target data and outputs an identification result.

[0199] Furthermore, a third source data set composed of 3D images (with ground truth) of CGs containing RGB information and Depth information, and a third target data set composed of captured 3D images (without ground truth) can be prepared.

[0200] In addition, data not given ground truth can be included in the source data set, or data given ground truth can be included in the target data set.

[0201] (f3: Learning network corresponding to multiple data sets)

[0202] When using more source data sets and target data sets, the following learning network can be adopted: according to the number of data sets used, the number of groups of encoders and decoders (encoder / decoder) is appropriately increased, and the encoder / decoders are connected to each other in such a way that an adversarial network can be formed between any data sets.

[0203] (f4: Network structure of encoder / decoder)

[0204] The group of encoders and decoders (encoder / decoder) can adopt a network with a data structure corresponding to the dimension of the input sample data and the type of information contained, etc. Therefore, the encoder / decoder for inputting the sample data of the source data set and the encoder / decoder for inputting the sample data of the target data set can adopt the same network structure or different network structures.

[0205] For example, since the number of channels per pixel is different between the encoder / decoder for inputting an RGB image and the encoder / decoder for inputting a Depth image, the dimensions in each layer represented by the input layer are different. Or, when using a source data set composed of image data and a target data set composed of text data, the algorithms themselves can also be different. Specifically, a CNN can be adopted for the encoder / decoder for inputting image data, and an RNN or 1D-CNN, etc. can be adopted for the encoder / decoder for inputting text data.

[0206] <G. Application example>

[0207] The above description illustrates an example of absorbing the discrepancy between an image generated through simulation (CG image) and an image actually captured by a camera (real-world image). However, the estimation model of this embodiment is not limited to this installation example and can be applied to any application. That is, the term "environment" or "domain" can be interpreted as broadly as possible.

[0208] Any information observed by any sensing device can be applied using the method of this embodiment, even if the observation conditions and environment are different. For example, in the field of FA (Factory Automation), by applying the learning method of this embodiment, it is possible to compensate for environmental differences in the factory or equipment of the application object.

[0209] As a specific application example, in a defect inspection model learned using manually annotated image data, additional learning can be performed using unlabeled image data collected on-site as the target dataset, thereby generating a defect inspection model adapted to that specific situation. Alternatively, a product can be photographed, and the image data (labeled with correct answers) can be used as the source dataset, while image data of products from the same system with only slight differences in appearance compared to the original product can be used as the target dataset, generating a defect inspection model that can also be applied to products within the same system.

[0210] Furthermore, the method of this embodiment can also be applied to situations where an estimation model for estimating a worker's posture is generated. For example, in cases where it is necessary to monitor a worker's movements, it is often impossible to place a camera directly in front of the worker. In such cases, it is necessary to place a camera above the worker to capture the worker from directly above. When the camera is placed above the worker, it is difficult to properly annotate the joint positions of the worker within the field of view that are not captured by the camera. However, according to the method of this embodiment, a CG image (source dataset) of the field of view corresponding to the actual position of the camera can be generated. Therefore, by using such a source dataset to generate a posture estimation model, the worker's posture can be easily estimated based on the image actually captured by the camera.

[0211] Furthermore, by using simulation results as the source dataset, sensors that do not exist in reality can be virtually realized. For example, a source dataset can be a dataset containing force field data that has been assigned a state aligned with the coordinate system position within the image and generated through simulation, labeled as a forward retrieval solution. Meanwhile, image data (observation data) captured by a camera or similar device (without forward retrieval labels) can be set as the target domain. Using such source and target datasets, a force field estimation model can be generated, and image data captured at arbitrary intervals can be input into the force field estimation model as the estimation object data, thereby enabling the estimation of force fields generated at arbitrary locations.

[0212] In the simulation, for the parts where objects contact each other to generate acting forces and reaction forces, the generated forces can be expressed as a force field (such a force field is taken as the estimation object). On the other hand, it is difficult to measure such a force field in reality.

[0213] Using the force field estimation model generated by the method of this embodiment, it is possible to estimate the force field that could not be observed only by images in the past. That is, it is also possible to virtually realize a sensor that does not exist in reality.

[0214] In addition, it can be applied not only to the physical information observed in the sensing device, but also to the human-related information such as the sales performance on an EC (electronic commerce) website. For example, consider an application of estimating the sales performance on other EC websites based on the sales performance on a certain EC website.

[0215] In this way, the application objects of the learning method of this embodiment and the estimation model generated by this learning method can be applied to various observable information existing in the real world.

[0216] <H. Supplementary Note>

[0217] The above-described embodiment contains the following technical ideas.

[0218] [Structure 1]

[0219] An estimation system (1), wherein the estimation system (1) has:

[0220] A learning unit (102; 114) that uses a first data set (30) including a plurality of sample data given with correct solutions and a second data set (50) including a plurality of sample data not given with correct solutions to generate an estimation model (60); and

[0221] An estimation unit (102; 118) that inputs estimation object data (70) that may belong to the second data set into the estimation model and determines an estimation result (62),

[0222] The estimation model includes an encoder (96), and the encoder (96) calculates a first feature quantity independent of the first data set and the second data set and a second feature quantity dependent on the first data set or the second data set according to the sample data,

[0223] The learning unit is configured to learn the encoder such that, for a pair of first sample data contained in the first dataset and second sample data contained in the second dataset that should be assigned the same positive solution as the positive solution assigned to the first sample data, the same first feature quantity is calculated based on either the first sample data or the second sample data.

[0224] [Structure 2]

[0225] In the estimation system described in Structure 1,

[0226] The learning unit is configured to generate the estimated model through a learning network (90) that acts as an adversarial network.

[0227] [Structure 3]

[0228] In the estimation system described in structure 2,

[0229] The learning network includes:

[0230] The first encoder / decoder consists of the first encoder (92) and the first decoder (94); and

[0231] The second encoder / decoder consists of the second encoder (96) and the second decoder (98).

[0232] During the learning process in the aforementioned learning department.

[0233] The samples contained in the first dataset are input into a first network (92, 94, 96, 98) configured in the order of the first encoder / decoder and the second encoder / decoder.

[0234] The samples contained in the second dataset are input into a second network (96, 98, 92, 94) configured in the order of the second encoder / decoder and the first encoder / decoder.

[0235] [Structure 4]

[0236] In the estimation system described in structure 3,

[0237] The learning unit is configured to optimize the model parameters of the first encoder, the first decoder, the second encoder, and the second decoder by inputting samples contained in the first dataset into the first network in a manner that minimizes the error between the first feature quantity output from the first encoder and the first pseudo-feature quantity output from the second encoder.

[0238] The learning unit is configured to optimize the model parameters of the first encoder, the first decoder, the second encoder, and the second decoder by inputting samples contained in the second dataset into the second network in a manner that minimizes the error between the first feature quantity output from the second encoder and the first pseudo-feature quantity output from the first encoder.

[0239] [Structure 5]

[0240] In the estimation system described in any of the terms in structures 1 to 4,

[0241] The learning unit is further configured to learn the recognizer (64) input from the encoder's output based on the sample data contained in the first dataset and the corresponding correct solutions.

[0242] The estimation model also includes the recognizer.

[0243] [Structure 6]

[0244] An estimation device, wherein the estimation device comprises:

[0245] Storage unit (110), which holds the estimation model (60), said estimation model (60) is generated by learning using a first dataset (30) containing multiple sample data assigned positive solutions and a second dataset (50) containing multiple sample data not assigned positive solutions; and

[0246] The estimation unit (102; 118) inputs the estimation object data (70) that may belong to the second dataset into the estimation model and determines the estimation result (62).

[0247] The estimation model includes an encoder (96) that, based on sample data, calculates a first feature independent of the first dataset and the second dataset, and a second feature dependent on either the first dataset or the second dataset.

[0248] The encoder is trained such that for a pair of first sample data contained in the first dataset and second sample data contained in the second dataset that should be assigned the same positive solution as the positive solution assigned to the first sample data, the same first feature quantity is calculated based on either the first sample data or the second sample data.

[0249] [Structure 7]

[0250] An estimation method comprising the following steps:

[0251] Generate an estimation model (60) using a first data set (30) including a plurality of sample data to which correct answers are assigned and a second data set (50) including a plurality of sample data to which correct answers are not assigned; and

[0252] Input estimation target data (70) that may belong to the second data set into the estimation model to determine an estimation result (62).

[0253] The estimation model includes an encoder (96), and the encoder (96) calculates a first feature quantity that does not depend on the first data set and the second data set and a second feature quantity that depends on the first data set or the second data set according to the sample data.

[0254] The step of generating the estimation model includes the following steps: learning the encoder so that for a pair of first sample data included in the first data set and second sample data included in the second data set that should be assigned the same correct answer as the correct answer assigned to the first sample data, the same first feature quantity is calculated based on any one of the first sample data and the second sample data.

[0255] <I. Effect>

[0256] According to the present embodiment, a learned model with further improved estimation performance can be generated.

[0257] It should be considered that the embodiments disclosed this time are illustrative in all aspects rather than restrictive. The scope of the present invention is represented by the claims, rather than the above description, and is intended to include meanings equivalent to the claims and all modifications within the scope.

[0258] Reference Signs Explanation

[0259] 1: Robot system; 2: Robot; 4: Axis; 6: Hand; 8: Workpiece; 12: Servo driver; 14: Motor; 20: Camera; 30, 308: Source dataset; 32, 34, 52, 54: Sample data; 36, 56, 58: Region; 40, 60: Estimation model; 50, 314: Target dataset; 53, 55: Projected sample data; 62: Estimation result; 64: Subject-object recognizer; 66, 68: Feature quantity; 70: Estimated object data; 80, 90: Learning network; 82, 86, 92, 96: Encoder; 84, 88, 94, 98: Decoder; 100: Robot Controller; 102: Processor; 104: Main memory; 110: Storage; 114: Machine learning program; 116: Model parameters; 118: Control application; 122: Communication interface; 124: Input unit; 126: Output unit; 128: Camera interface; 130: Motor interface; 150, 160: Processing; 200: Simulation device; 250: External device; 302: Motion capture dataset; 306: Forward solution; 310: User; 312: Stereo camera; 921, 942, 944, 961, 982, 984: Shared parameters; 922, 941, 962, 981: Independent parameters.

Claims

1. An estimation system, wherein, This estimation system has the following characteristics: The learning unit uses a source dataset containing multiple sample data points assigned positive solutions and a target dataset containing multiple sample data points not assigned positive solutions to generate an estimation model. The source dataset consists of images generated through simulation, and the target dataset consists of actually captured images. The estimation department inputs the data of the object to be estimated into the estimation model and determines the estimation result. The learning unit initializes the model parameters of the learning network, which includes a first path input to the source dataset and a second path input to the target dataset. In the first path, the encoder / decoder is configured in the order of a first encoder and a first decoder, and a second encoder / decoder is configured in the order of a second encoder and a second decoder. In the second path, the second encoder / decoder and the first encoder / decoder are configured in that order. The first encoder and the second encoder calculate, based on the sample data, a first feature quantity that is independent of the source dataset and the target dataset, and a second feature quantity that is dependent on either the source dataset or the target dataset. The learning unit optimizes the model parameters of the learning network in a manner that minimizes the error between the sample data belonging to the source dataset and the pseudo-sample data output from the second encoder / decoder of the first path, and minimizes the error between the sample data belonging to the target dataset and the pseudo-sample data output from the first encoder / decoder of the second path. The learning unit optimizes the model parameters of the learning network in a manner that makes it indistinguishable between sample data belonging to the source dataset and pseudo-sample data output from the second encoder / decoder of the second path, and in a manner that makes it indistinguishable between sample data belonging to the target dataset and pseudo-sample data output from the first encoder / decoder of the first path. The learning unit optimizes the model parameters of the learning network in a manner that minimizes the error between the sample data belonging to the source dataset and the pseudo-sample data output from the second encoder / decoder of the second path, and minimizes the error between the sample data belonging to the target dataset and the pseudo-sample data output from the first encoder / decoder of the first path. The learning unit optimizes the model parameters of the learning network in a manner that minimizes the error between the first feature quantity output from the first encoder of the first path and the first feature quantity in the pseudo-sample data output from the second encoder of the first path, and minimizes the error between the first feature quantity output from the second encoder of the second path and the first feature quantity in the pseudo-sample data output from the first encoder of the second path. The learning unit optimizes the model parameters of the first encoder and the main target recognizer by minimizing the error between the estimated value and the positive solution of the sample data. The estimated value is obtained by inputting sample data belonging to the source dataset into the first encoder and inputting the first feature quantity output from the first encoder into the main target recognizer. The estimation model consists of the second encoder and the main target identifier. The estimation unit uses the estimation model to input the estimated object data into the second encoder and input the first feature quantity output from the second encoder into the main target identifier to determine the estimation result.

2. The estimation system according to claim 1, wherein, The learning network mentioned is an adversarial network.

3. The estimation system according to claim 1 or 2, wherein, At least a portion of the model parameters of the first encoder of the first encoder / decoder and at least a portion of the model parameters of the second encoder of the second encoder / decoder are shared.

4. An estimation device, wherein, The estimation device has: The storage unit holds an estimation model generated through learning using a source dataset containing multiple samples assigned positive solutions and a target dataset containing multiple samples not assigned positive solutions, wherein the source dataset consists of images generated through simulation and the target dataset consists of actually captured images; and The estimation department inputs the data of the object to be estimated into the estimation model and determines the estimation result. The estimation model is generated by optimizing the model parameters of the learning network, which includes a first path input to the source dataset and a second path input to the target dataset. In the first path, the encoder / decoder is configured in the order of a first encoder and a first decoder, and a second encoder / decoder is configured in the order of a second encoder and a second decoder. In the second path, the second encoder / decoder and the first encoder / decoder are configured in that order. The first encoder and the second encoder calculate, based on the sample data, a first feature quantity that is independent of the source dataset and the target dataset, and a second feature quantity that is dependent on either the source dataset or the target dataset. The optimization includes: The model parameters of the learning network are optimized in a manner that minimizes the error between the sample data belonging to the source dataset and the pseudo-sample data output from the second encoder / decoder of the first path, and minimizes the error between the sample data belonging to the target dataset and the pseudo-sample data output from the first encoder / decoder of the second path. The model parameters of the learning network are optimized in a manner that makes it indistinguishable between sample data belonging to the source dataset and pseudo-sample data output from the second encoder / decoder of the second path, and in a manner that makes it indistinguishable between sample data belonging to the target dataset and pseudo-sample data output from the first encoder / decoder of the first path. The model parameters of the learning network are optimized in a manner that minimizes the error between the sample data belonging to the source dataset and the pseudo-sample data output from the second encoder / decoder of the second path, and minimizes the error between the sample data belonging to the target dataset and the pseudo-sample data output from the first encoder / decoder of the first path. The model parameters of the learning network are optimized in a manner that minimizes the error between the first feature quantity output from the first encoder of the first path and the first feature quantity in the pseudo-sample data output from the second encoder of the first path, and minimizes the error between the first feature quantity output from the second encoder of the second path and the first feature quantity in the pseudo-sample data output from the first encoder of the second path. The optimization includes optimizing the model parameters of the first encoder and the primary target recognizer in a manner that minimizes the error between the estimated value and the positive solution of the sample data. The estimated value is obtained by inputting sample data belonging to the source dataset into the first encoder and inputting the first feature quantity output from the first encoder into the primary target recognizer. The estimation model consists of the second encoder and the main target identifier. The estimation unit uses the estimation model to input the estimated object data into the second encoder and input the first feature quantity output from the second encoder into the main target identifier to determine the estimation result.

5. The estimation apparatus according to claim 4, wherein, The learning network mentioned is an adversarial network.

6. The estimation apparatus according to claim 4 or 5, wherein, At least a portion of the model parameters of the first encoder of the first encoder / decoder and at least a portion of the model parameters of the second encoder of the second encoder / decoder are shared.

7. A method for generating an estimation model, wherein, The method for generating this estimation model has the following steps: Prepare a source dataset containing multiple sample data points assigned positive solutions, and a target dataset containing multiple sample data points not assigned positive solutions, wherein the source dataset consists of images generated through simulation, and the target dataset consists of actually captured images; and The model parameters of the learning network are optimized. The learning network includes a first path input to the source dataset and a second path input to the target dataset. In the first path, the encoder / decoder is configured in the order of a first encoder and a first decoder, and a second encoder / decoder is configured in the order of a second encoder and a second decoder. In the second path, the second encoder / decoder and the first encoder / decoder are configured in that order. The first encoder and the second encoder calculate, based on the sample data, a first feature quantity that is independent of the source dataset and the target dataset, and a second feature quantity that is dependent on either the source dataset or the target dataset. The optimization steps include the following steps: The model parameters of the learning network are optimized in a manner that minimizes the error between the sample data belonging to the source dataset and the pseudo-sample data output from the second encoder / decoder of the first path, and minimizes the error between the sample data belonging to the target dataset and the pseudo-sample data output from the first encoder / decoder of the second path. The model parameters of the learning network are optimized in a manner that makes it indistinguishable between sample data belonging to the source dataset and pseudo-sample data output from the second encoder / decoder of the second path, and in a manner that makes it indistinguishable between sample data belonging to the target dataset and pseudo-sample data output from the first encoder / decoder of the first path. The model parameters of the learning network are optimized in a manner that minimizes the error between the sample data belonging to the source dataset and the pseudo-sample data output from the second encoder / decoder of the second path, and minimizes the error between the sample data belonging to the target dataset and the pseudo-sample data output from the first encoder / decoder of the first path. The model parameters of the learning network are optimized in a manner that minimizes the error between the first feature quantity output from the first encoder of the first path and the first feature quantity in the pseudo-sample data output from the second encoder of the first path, and minimizes the error between the first feature quantity output from the second encoder of the second path and the first feature quantity in the pseudo-sample data output from the first encoder of the second path. The optimization step includes the following steps: optimizing the model parameters of the first encoder and the main target recognizer in a manner that minimizes the error between the estimated value and the positive solution of the sample data. The estimated value is obtained by inputting sample data belonging to the source dataset into the first encoder and inputting the first feature quantity output from the first encoder into the main target recognizer. The method for generating the estimation model has the following steps: using the second encoder and the main target recognizer to construct an estimation model; using the estimation model, inputting the estimated object data into the second encoder, and inputting the first feature quantity output from the second encoder into the main target recognizer to determine the estimation result.

8. The method for generating the estimation model according to claim 7, wherein, The learning network mentioned is an adversarial network.

9. The method for generating the estimation model according to claim 7 or 8, wherein, At least a portion of the model parameters of the first encoder of the first encoder / decoder and at least a portion of the model parameters of the second encoder of the second encoder / decoder are shared.

Images