Model generation device, estimation device, model generation method, and model generation program

Abstract

An aspect of the present application relates to a model generation device that acquires context data for training data and obtains a difference between the training data and the context data for each learning data set, thereby generating difference data. Then, the model generation device trains a predictor by machine learning so that a result of estimating a feature by the predictor from the generated difference data is adapted to correct answer data for each learning data set.

Description

TECHNICAL FIELD

[0001] The present application relates to a model generation device, an estimation device, a model generation method, and a model generation program. BACKGROUND

[0002] In the past, in a scenario of manufacturing a product in a manufacturing line or the like, a technique of capturing a manufactured product by a capturing device and checking the good or bad of the product based on obtained image data has been utilized. For example, in Patent Literature 1, an inspection device is proposed which determines whether or not an inspection target object represented in an image is normal or abnormal based on a first neural network which has been learned, and in a case where it is determined that the inspection target object is abnormal, classifies the category of the abnormality based on a second neural network which has been learned.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2012-026982

[0006] Patent Literature 2: Japanese Patent Application Publication No. 2017-111806

[0007] Patent Literature 3: Japanese Patent Application Publication No. 2014-049118

[0008] Patent Literature 4: Japanese Patent Application Publication No. 2019-071050

[0009] Patent Literature 5: Japanese Patent No. 6264492 SUMMARY

[0010] PROBLEMS TO BE SOLVED BY THE INVENTION

[0011] With an estimator constituted by a trained machine learning model constructed by machine learning, it is possible to perform an estimation (including prediction) task such as regression, classification, and the like on unknown data of the same kind as training data given. Therefore, according to the method of utilizing a machine learning model such as a neural network in the above-described Patent Literature 1, it is possible to implement appearance inspection of a product based on the output of a trained estimator. However, the present inventors have found that the above-described conventional method has the following problems.

[0012] That is, in the conventional method, in order to implement machine learning which enables a machine learning model to acquire the ability to detect a defect represented in a given image, an image representing a product including a defect is collected in advance as learning data. Detecting a defect can also be identifying the category of a defect. Basically, the more learning data is collected, in other words, the more diverse the conditions of defects represented in learning data, the more it is possible to achieve improvement in the accuracy of appearance inspection.

[0013] However, the more the categories of defects and the categories of the backgrounds including the appearance of products, the more the combinations of the categories of defects and the categories of the backgrounds, and the more the number of pieces (and categories) that need to be collected as samples of learning data. In addition, the background can change when the trained estimator is used. In order to generate a trained estimator that can perform appearance inspection with high accuracy, it is preferable to collect samples with respect to all combinations of the categories of defects and the categories of the backgrounds, but collecting samples with respect to all combinations increases the cost. Therefore, machine learning of the machine learning model is implemented at a stage where learning data is collected to a certain extent, for example, at a time when samples are obtained with respect to all categories of defects of the detection object although samples are not obtained with respect to all combinations.

[0014] However, when there is a combination for which samples are not obtained in the learning data, the following problem can occur. That is, by machine learning, the machine learning model is trained using arbitrary information included in the training data (samples) to perform a desired estimation task. Therefore, in a case where machine learning of the estimator is implemented using training data including defects and backgrounds, the trained estimator generated using the information related to the defects included in the data, sometimes using the information related to the backgrounds, and obtains the ability to perform an estimation task related to the detection of defects. In such a case, since the combination of the category of defects for which samples are not obtained and the category of the background cannot be reflected in machine learning, the estimation accuracy of the trained estimator for detecting defects can decrease with respect to the combination.

[0015] In addition, in a case where the combination of the category of defects and the category of the background appearing in the samples is biased, particularly in a case where the categories of defects and the categories of the backgrounds appear in the samples with the same distribution, the bias of the samples can have a bad influence on the ability of the machine learning model mastered by machine learning. Specifically, by machine learning using samples in which the combination is biased, the estimator can master the ability to recognize the category of the background rather than the ability to detect defects according to the category.

[0016] As an example, assume that two products, a first product and a second product, generate two kinds of defects, a first defect and a second defect. Also, in this example, assume that the first defect is likely to occur in the first product and the second defect is likely to occur in the second product. Further, assume that, in the obtained samples, there is a bias in that images representing the first product including the first defect and images representing the second product including the second defect are included. That is, assume that images representing the first product including the second defect and images representing the second product including the first defect are not obtained as samples for machine learning. Assume that, using the samples obtained under this assumption, machine learning of the machine learning model is performed in order to acquire the ability to detect defects according to the category. In this case, since the obtained samples are biased as described above, the machine learning model can acquire the ability to recognize the category of the background, rather than the ability to detect defects according to the category, through the machine learning. That is, since the sample bias occurs, it is possible to cause the machine learning model to acquire an ability different from the target ability. In a case where the ability to recognize the category of the background is acquired, when an image representing the first product including the second defect is given, the trained inferencer can erroneously detect that the first defect occurs in the first product.

[0017] Therefore, in the conventional method, through the machine learning, the trained inferencer can be constructed to perform the inference task related to the defect detection, taking into account the information related to the background included in the image data. Thus, since the background at the time of use is different from the background at the time of learning, there is a problem that the inference accuracy related to the defect detection of the trained inferencer decreases. In other words, there is a problem that it is difficult to generate a trained inferencer that is robust to the difference in the background.

[0018] Note that such a problem is not unique to the scenario of generating a trained inferencer that can be used for appearance inspection of products. The problem can also occur in various scenarios of inferring certain features included in image data. The scenario of inferring certain features included in image data is, for example, a scenario of performing inference related to the foreground of image data. For example, the inference related to the foreground can be inference of the attribute of an object represented in image data.

[0019] As specific examples, in Patent Literature 2, a scheme of recognizing a license plate using a trained neural network is proposed. In Patent Literature 3, a scheme of recognizing a character using a trained convolutional neural network is proposed. In Patent Literature 4, a scheme of inferring the distribution of the density of a person represented in a captured image and the distribution of a moving vector using a trained neural network is proposed. In Patent Literature 5, a scheme of inferring the degree of concentration of a driver on driving from a captured image using a trained model is proposed.

[0020] In view of these scenarios, in the past, in the case where a trained inferencer for performing an inference task related to a feature of an object is constructed by machine learning, the trained inferencer can also be constructed to perform the inference task taking into account information related to a background included in image data. Therefore, there is a problem that it is difficult to generate a trained inferencer that is robust against a difference in background.

[0021] One aspect of the present application is achieved in view of such a practical situation, and aims to provide a technology for generating a trained inferencer that can perform an inference task related to a feature of an object included in image data and is robust against a difference in background.

[0022] Solution to the problem

[0023] To solve the above problem, the present application adopts the following structure.

[0024] That is, the model generation device according to one aspect of the present application includes: a first data acquisition unit that acquires a plurality of learning data sets each composed of a combination of training data that is image data, and correct answer data that represents a feature included in the training data; a background acquisition unit that acquires, for each of the learning data sets, background data that represents a background of the training data; a difference calculation unit that, for each of the learning data sets, obtains a difference between the acquired background data and the training data, thereby generating difference data that represents the difference between the background data and the training data; and a first training unit that performs machine learning of an inferencer, the machine learning of the inferencer including, for each of the learning data sets, training the inferencer so that a result of inferring the feature by the inferencer from the generated difference data is adapted to the correct answer data.

[0025] In the model generation device according to the structure, for each learning data set, difference data for training excluding a background is generated by subtracting the background data from the training data. The inferencer is trained by machine learning using the generated difference data for training, to infer the feature of the object from the training data excluding information of the background data. Thus, a trained inferencer that has acquired an ability to perform an inference task related to a feature of an object can be generated without using information of the background data excluded in the difference calculation. That is, when an inference task related to a feature of an object is performed by the trained inferencer, information corresponding to the background data can not be taken into account. Therefore, according to the model generation device according to the structure, a trained inferencer that can perform an inference task related to a feature of an object included in image data, that is, a trained inferencer that is robust against a difference in background, can be generated.

[0026] Also, in the model generation device of one aspect described above, the acquiring the background data can include generating the background data with respect to the training data for each of the learning data sets by the generator trained by machine learning. For example, in a case where the same kind of background produces a deviation even if the photographing position is shifted, it is costly to prepare the background data corresponding to each of the deviations in advance. Also, if the difference is calculated while ignoring the deviation, most of the information related to the background can remain in the obtained difference data, and it can be impossible to appropriately generate the trained estimator that is robust to the difference in the background. In contrast, according to this configuration, by using the generator trained by machine learning, it is possible to generate the background data corresponding to the training data even in a case where the background produces a deviation. Therefore, it is possible to reduce the cost of preparing the background data. Also, it is possible to appropriately generate the trained estimator that is robust to the difference in the background.

[0027] Also, the model generation device of one aspect described above can further include a second data acquisition unit that acquires learning background data, and a second training unit that constructs the generator trained by machine learning by performing machine learning using the acquired learning background data, the generator being trained to generate the background data with respect to the training data. According to this configuration, it is possible to appropriately construct the generator trained by machine learning that can generate the background data corresponding to the training data. Also, by using the constructed generator trained by machine learning, it is possible to reduce the cost of preparing the background data corresponding to the training data for each of the learning data sets. Also, it is possible to appropriately generate the trained estimator that is robust to the difference in the background. Note that the learning background data uses image data that represents the same kind of background as the training data.

[0028] Also, in the model generation device of one aspect described above, the difference calculation unit can obtain the difference between each pixel of the training data and each corresponding pixel of the background data based on the correlation between the object region including each pixel of the training data and the surrounding pixels and the corresponding region including each corresponding pixel of the background data and the surrounding pixels, thereby generating the difference data. Due to the influence of noise and the like, it is possible that an error occurs between the background included in the training data and the background data. This possibility is higher particularly in a case where the background data is generated by the generator trained by machine learning. In contrast, in this configuration, by calculating the difference between the object pixels based on the correlation between the regions including the object pixels, it is possible to generate the difference data without leaving the error compared to simple difference calculation. Therefore, according to this configuration, it is possible to appropriately generate the trained estimator that is robust to the difference in the background even in a case where it is possible that an error occurs in the background data.

[0029] Also, in the model generation device according to the above aspect, the training data can be related to a background. According to this configuration, a trained estimator that is robust to a difference in the background and that performs an estimation task related to the foreground can be generated. Note that the background and the foreground can be arbitrarily selected from events that can be represented in image data.

[0030] Also, in the model generation device according to the above aspect, the training data can be image data representing an object, and the feature can be an attribute of the object. According to this configuration, a trained estimator that is robust to a difference in the background and that performs an estimation task related to an attribute of the object can be generated. Note that the object can be arbitrarily selected. The object can be, for example, a product, a package of the product, a license plate, a moving body (e.g., a vehicle, a person, or the like) that can move within a range of a photograph, a driver of a vehicle, or the like.

[0031] Also, in the model generation device according to the above aspect, the object can be a product, and the attribute of the object can be related to a defect of the product. According to this configuration, in a scenario in which appearance inspection is performed, a trained estimator that is robust to a difference in the background and that performs an estimation task related to a defect can be generated.

[0032] In addition, the aspect of the present application can not be limited to the above-described model generation device. One aspect of the present application can be a device that uses a trained estimator generated by the above-described model generation device. For example, one aspect of the present application can be an estimation device configured to perform an estimation task related to a feature of an object on object image data using a trained generator generated by the above-described model generation device. The estimation device can be replaced with an inspection device, an identification device, a monitoring device, a diagnosis device, a prediction device, or the like according to a type of the estimation task in an application scenario.

[0033] For example, the estimation device according to the above aspect can include: a data acquisition unit configured to acquire object image data; a background acquisition unit configured to acquire object background data corresponding to the object image data; a difference calculation unit configured to generate object difference data by taking a difference between the object image data and the object background data; an estimation unit configured to estimate a feature included in the generated object difference data using a trained estimator generated by the model generation device according to any one of the above aspects; and an output unit configured to output information related to a result of estimating the feature. According to this configuration, an estimation task related to a feature of an object can be robustly performed with respect to a difference in a background.

[0034] Furthermore, as another embodiment of the model generation apparatus and estimation apparatus involved in the above-described embodiments, one aspect of the present invention can be an information processing method that implements all or part of the above structures, or a program, or a computer and other readable storage medium such as a machine storing such a program. Here, the readable storage medium such as a computer is a medium that stores information such as programs through electrical, magnetic, optical, mechanical, or chemical processes. Additionally, another aspect of the present invention can be an estimation system composed of the model generation apparatus and estimation apparatus involved in any of the above-described embodiments.

[0035] For example, one aspect of the model generation method relates to an information processing method in which a computer performs the following steps: acquiring multiple learning datasets, each consisting of a combination of training data as image data and positive solution data representing features contained in the training data; acquiring background data representing the background of the training data for each learning dataset; obtaining the difference between the acquired background data and the training data for each learning dataset, thereby generating difference data representing the difference between the background data and the training data; and implementing machine learning for an inferrer, the machine learning of the inferrer comprising: training the inferrer for each learning dataset such that the result of the inferrer inferring the features based on the generated difference data is suitable for the positive solution data.

[0036] For example, one aspect of the present invention relates to a model generation program for causing a computer to perform the following steps: acquiring a plurality of learning datasets, each consisting of a combination of training data as image data and forward solution data representing features contained in the training data; acquiring background data representing the background of the training data with respect to each of the learning datasets; obtaining the difference between the acquired background data and the training data with respect to each of the learning datasets, thereby generating difference data representing the difference between the background data and the training data; and implementing machine learning of an inferr, the machine learning of the inferr comprising: training the inferr with respect to each of the learning datasets such that the result of the inferr inferring the features based on the generated difference data is suitable for the forward solution data.

[0037] Invention Effects

[0038] According to the present invention, a trained inferencer capable of performing an inference task related to the features of objects contained in image data can be generated, i.e., a trained inferencer robust to differences in the background. Attached Figure Description

[0039] Figure 1An example of a scenario in which the present application is applied is schematically illustrated.

[0040] Figure 2 An example of a hardware structure of the model generation device involved in the embodiment is schematically illustrated.

[0041] Figure 3 An example of a hardware structure of the estimation device involved in the embodiment is schematically illustrated.

[0042] Figure 4A An example of a software structure of the model generation device involved in the embodiment is schematically illustrated.

[0043] Figure 4B An example of a software structure of the model generation device involved in the embodiment is schematically illustrated.

[0044] Figure 4C An example of a software structure of the model generation device involved in the embodiment is schematically illustrated.

[0045] Figure 5 An example of a software structure of the estimation device involved in the embodiment is schematically illustrated.

[0046] Figure 6A An example of a processing sequence related to the generation of a trained generator by the model generation device involved in the embodiment is illustrated.

[0047] Figure 6B An example of a processing sequence of machine learning of the generator involved in the embodiment is illustrated.

[0048] Figure 7 An example of a processing sequence related to the generation of a trained input estimator by the model generation device involved in the embodiment is illustrated.

[0049] Figure 8 An example of a processing sequence related to the generation of a trained estimator by the model generation device involved in the embodiment is illustrated.

[0050] Figure 9 An example of a method of calculating a difference between training data and background data is schematically illustrated.

[0051] Figure 10 An example of a processing sequence of the estimation device involved in the embodiment is illustrated.

[0052] Figure 11 An example of another scenario in which the present application is applied is schematically illustrated.

[0053] Figure 12A An example of a hardware structure of the inspection device involved in another embodiment is schematically illustrated.

[0054] Figure 12B An example of a software structure of an inspection device according to another aspect will be schematically illustrated.

[0055] Figure 13 An example of another scenario in which the present application is applied will be schematically illustrated.

[0056] Figure 14 An example of another scenario in which the present application is applied will be schematically illustrated.

[0057] Figure 15 An example of another scenario in which the present application is applied will be schematically illustrated.

[0058] Figure 16 An example of another scenario in which the present application is applied will be schematically illustrated.

[0059] Figure 17 An example of a structure of a generator according to another aspect will be schematically illustrated. DETAILED DESCRIPTION

[0060] Hereinafter, an embodiment according to one aspect of the present application (hereinafter also referred to as "the present embodiment") will be described based on the drawings. However, the present embodiment described below is merely an example of the present application in all respects. Of course, various modifications and changes can be made without departing from the scope of the present application. That is, in implementing the present application, a specific structure corresponding to the embodiment can be appropriately adopted. Note that in the present embodiment, data appearing in natural language is described, and more specifically, is specified by a computer-recognizable analog language, an instruction, a parameter, a machine language, and the like.

[0061] §1 Application Example

[0062] Figure 1 An example of a scenario in which the present application is applied will be schematically illustrated. As shown in Figure 1 The estimation system 100 according to the present embodiment is provided with a model generation device 1 and an estimation device 2.

[0063] The model generation device 1 according to the present embodiment is a computer configured to implement machine learning of the estimator 5. Specifically, the model generation device 1 according to the present embodiment acquires a plurality of learning data sets 30. Each learning data set 30 is constituted by a combination of training data 31 and correct answer data 32 indicating features included in the training data 31.

[0064] The training data 31 is a sample of image data. The training data 31 can be referred to as a "training sample", an "image sample", and simply as a "sample", and so on. The feature can be arbitrarily selected. For example, the feature can be a feature related to a component (element) capable of being directly or indirectly represented in the image data. Direct representation corresponds to being represented in the image data. Indirect representation corresponds to being derived from an event represented in the image data.

[0065] The correct answer data 32 indicates a correct answer of an estimation task related to a feature of an object. The correct answer data 32 can be referred to as a "teaching signal", a "label", and so on. "Estimation" can also be replaced with "inference". Estimation can be, for example, any one of derivation of a discrete value (for example, a class corresponding to a specific feature) by grouping (classification, identification), and derivation of a continuous value (for example, a probability of occurrence of a specific feature) by regression. Estimation can also include some determination such as detection, determination, and so on based on a result of the grouping or the regression. In addition, estimation can also include prediction.

[0066] The model generation device 1 according to the present embodiment acquires, for each of the learning data sets 30, background data 35 indicating a background of the training data 31. The background can be appropriately selected from any component other than the feature of the object of the estimation task included in the image data. The background can be constituted by all or a part of the components other than the feature of the object. The model generation device 1 according to the present embodiment acquires, for each of the learning data sets 30, a difference between the acquired background data 35 and the training data 31, thereby generating difference data 37. The generated difference data 37 indicates a difference between the background data 35 and the training data 31.

[0067] The model generation device 1 according to the present embodiment performs machine learning of the estimator 5 using the generated difference data 37. The estimator 5 is configured to accept input of image data (sample), estimate a feature of an object included in the input image data, and output an output value corresponding to the estimation result. The form of the input and the output of the estimator 5 can be arbitrarily determined according to the estimation task. The estimator 5 is constituted by a machine learning model (learner) having an operation parameter used in operation, that is, an operation parameter adjusted by machine learning. The kind of the machine learning model constituting the estimator 5 is not particularly limited and can be arbitrarily selected according to the embodiment. As described later, in the present embodiment, a neural network is used in the estimator 5.

[0068] The machine learning relating to the present embodiment includes training the estimator 5 with respect to each of the learning data sets 30 so that the result of estimating the feature by the estimator 5 from the generated difference data 37 is fitted to the correct answer data 32. The model generation device 1 relating to the present embodiment is able to generate the trained estimator 5 that has acquired the ability to estimate the feature included in the image data by implementing this machine learning. Note that "trained" can also be referred to as "machine-learned" or simply "learned", and so on. In addition, the model generation device 1 relating to the present embodiment can also be referred to simply as a "generation device", a "learning device", and so on.

[0069] On the other hand, the estimation device 2 relating to the present embodiment is a computer configured to execute an estimation task relating to the feature of the object using the trained estimator 5 generated by the model generation device 1 with respect to the image data. Specifically, the estimation device 2 relating to the present embodiment acquires object image data 41 that is an object of executing the estimation task. The object image data 41 can be referred to as an "object sample", an "image sample", or simply a "sample", and so on. The estimation device 2 acquires object context data 45 corresponding to the object image data 41. The estimation device 2 generates object difference data 47 by taking the difference between the object image data 41 and the object context data 45. The estimation device 2 estimates the feature included in the generated object difference data 47 using the trained estimator 5 generated by the model generation device 1. Then, the estimation device 2 outputs information relating to the result of estimating the feature.

[0070] As described above, in the present embodiment, the estimator 5 is trained so that the feature of the object is fitted to the correct answer data 32 from the training data 31 excluding the information of the context data 35 by machine learning using the difference data 37 for training. As a result, the trained estimator 5 is able to acquire the ability to execute the estimation task relating to the feature of the object without using the information of the context data 35 excluded in the difference calculation. That is, when the estimation task relating to the feature of the object is executed by the trained estimator 5, the information corresponding to the context data 35 can not be taken into account. Therefore, according to the model generation device 1 relating to the present embodiment, it is possible to generate a trained estimator 5 that can execute the estimation task relating to the feature of the object included in the image data, that is, a trained estimator 5 that is robust to the difference in the context. In the estimation device 2 relating to the present embodiment, it is possible to robustly execute the estimation task relating to the feature of the object with respect to the difference in the context by using the generated trained estimator 5.

[0071] Note that, in the present embodiment, the model generation device 1 relating to the present embodiment is able to generate the trained estimator 5 by implementing the machine learning using the difference data 37 for training. However, the model generation device 1 relating to the present embodiment can also be able to generate the trained estimator 5 by implementing the machine learning using the difference data 37 for training and the difference data 37 for validation. Figure 1In the example, the model generation device 1 and the estimation device 2 are connected to each other via a network. The kind of the network can be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, and the like. However, the method of exchanging data between the model generation device 1 and the estimation device 2 is not limited to this example, and can be appropriately selected according to the embodiment. For example, data can be exchanged between the model generation device 1 and the estimation device 2 using a storage medium.

[0072] In addition, in the example, the model generation device 1 and the estimation device 2 are each constituted by a different computer. However, the structure of the estimation system 100 according to the embodiment is not limited to this example, and can be appropriately determined according to the embodiment. For example, the model generation device 1 and the estimation device 2 can also be an integrated computer. In addition, for example, at least one of the model generation device 1 and the estimation device 2 can be constituted by a plurality of computers. Figure 1

[0073] §2 Structure Example

[0074] [Hardware Structure]

[0075] < Model Generation Device >

[0076] Figure 2 An example of the hardware structure of the model generation device 1 according to the embodiment is schematically illustrated. As shown in FIG. 1, the model generation device 1 according to the embodiment is a computer to which a control section 11, a storage section 12, a communication interface 13, an external interface 14, an input device 15, an output device 16, and a driver 17 are electrically connected. Note that, in the example, the communication interface and the external interface are described as “communication I / F” and “external I / F”. Figure 2 Figure 2

[0077] The control section 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like as a hardware processor, and is configured to perform information processing based on a program and various data. The storage section 12 is an example of a storage, and is constituted by, for example, a hard disk drive, a solid state drive, or the like. In the embodiment, the storage section 12 stores various information such as a model generation program 81, a plurality of learning data sets 30, a plurality of pieces of learning background data 39, first to third learning result data 121 to 123, and the like.

[0078] The model generation program 81 is a program for causing the model generation device 1 to perform the information processing described later for generating each of the learned models by implementing machine learning. Figure 6A ,​​​Figure 6B , Figure 7 and Figure 8 ). The model generation program 81 contains a series of commands of the information processing. The model generation program 81 can also be simply referred to as a "generation program", a "learning program", or the like. The learning data set 30 is used for machine learning of the inferencer 5. The learning background data 39 is used for machine learning of the generator 60 described later. The first to third learning result data 121 to 123 respectively represent information related to each of the completed models constructed through machine learning. In the present embodiment, the first to third learning result data 121 to 123 are respectively generated as a result of executing the model generation program 81.

[0079] The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. The model generation apparatus 1 can also use the communication interface 13 to perform data communication via a network with other information processing apparatuses. The external interface 14 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting with an external apparatus. The kind and the number of the external interface 14 can be arbitrarily selected. The model generation apparatus 1 can be connected with a camera for obtaining image samples of the training data 31, the learning background data 39, and the like via at least one of the communication interface 13 and the external interface 14.

[0080] The input apparatus 15 is, for example, an apparatus for inputting a mouse, a keyboard, or the like. In addition, the output apparatus 16 is, for example, an apparatus for outputting a display, a speaker, or the like. An operator such as a user can operate the model generation apparatus 1 by using the input apparatus 15 and the output apparatus 16.

[0081] The driver 17 is, for example, a CD driver, a DVD driver, or the like, and is a drive apparatus for reading various kinds of information such as a program stored in the storage medium 91. The storage medium 91 is a medium that stores various kinds of information such as a program by electric, magnetic, optical, mechanical, or chemical action in a manner that a computer and other apparatuses, machines, or the like can read the stored program. At least any one of the above-described model generation program 81, the plurality of learning data sets 30, and the plurality of pieces of learning background data 39 can also be stored in the storage medium 91. The model generation apparatus 1 can also acquire at least any one of the above-described model generation program 81, the plurality of learning data sets 30, and the plurality of pieces of learning background data 39 from the storage medium 91. Note that, in the present embodiment, the model generation apparatus 1 can acquire at least any one of the above-described model generation program 81, the plurality of learning data sets 30, and the plurality of pieces of learning background data 39 from a server on a network via the communication interface 13. Figure 2In the example shown, storage medium 91 includes disc-type storage media such as CDs and DVDs. However, the type of storage medium 91 is not limited to disc type and can also be other types. Examples of storage media other than disc type include semiconductor memory such as flash memory. The type of drive 17 can be arbitrarily selected depending on the type of storage medium 91.

[0082] It should be noted that the specific hardware structure of the model generation device 1 can be appropriately omitted, substituted, or added depending on the implementation method. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be composed of microprocessors, FPGAs (field-programmable gate arrays), DSPs (digital signal processors), etc. The storage unit 12 may also be composed of RAM and ROM included in the control unit 11. At least one of the communication interface 13, external interface 14, input device 15, output device 16, and driver 17 may be omitted. The model generation device 1 may also be composed of multiple computers. In this case, the hardware structure of each computer may be the same or different. In addition, the model generation device 1 may be designed as an information processing device dedicated to providing services, or it may be a general-purpose server device, PC (personal computer), etc.

[0083] <Predictive Apparatus>

[0084] Figure 3 An example of the hardware structure of the estimation device 2 according to this embodiment is illustrated schematically. For example... Figure 3 As shown, the estimation device 2 involved in this embodiment is a computer electrically connected to a control unit 21, a storage unit 22, a communication interface 23, an external interface 24, an input device 25, an output device 26, and a driver 27.

[0085] The control unit 21, driver 27, and storage medium 92 of the estimation device 2 can be configured similarly to the control unit 11, driver 17, and storage medium 91 of the model generation device 1. The control unit 21 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to perform various information processing based on programs and data. The storage unit 22 is configured, for example, a hard disk drive, a solid-state drive, etc. The storage unit 22 stores various information such as the estimation program 82 and the first to third learning result data 121 to 123. The estimation program 82 is used to enable the estimation device 2 to perform an estimation task related to the features of the object on the image data, as described later, using the trained estimator 5. Figure 10The inference program 82 includes a series of commands of the information processing. The inference program 82 and at least any one of the first to third learning result data 121 to 123 can also be stored in the storage medium 92. In addition, the inference device 2 can acquire the above-mentioned inference program 82 and at least any one of the first to third learning result data 121 to 123 from the storage medium 92.

[0086] Note that, regarding the specific hardware configuration of the inference device 2, omission, replacement, and addition of the constituent elements can be appropriately made according to the embodiment. For example, the control section 21 can include a plurality of hardware processors. The hardware processor can be constituted by a microprocessor, an FPGA, a DSP, or the like. The storage section 22 can also be constituted by a RAM and a ROM included in the control section 21. At least any one of the communication interface 23, the external interface 24, the input device 25, the output device 26, and the driver 27 can also be omitted. The inference device 2 can also be constituted by a plurality of computers. In this case, the hardware structures of the respective computers can be identical or different. In addition, the inference device 2 can be a general-purpose server device, a general-purpose PC, a PLC (programmable logic controller), or the like, in addition to being designed as an information processing device dedicated to the service provided.

[0087] [Software Structure]

[0088] < Model Generation Device >

[0089] Figure 4A - Figure 4C An example of the software structure of the model generation device 1 according to the present embodiment is schematically illustrated.

[0090] The control section 11 of the model generation device 1 expands the model generation program 81 stored in the storage section 12 in the RAM. Then, the control section 11 controls the respective constituent elements by interpreting and executing the commands included in the model generation program 81 expanded in the RAM by the CPU. Thus, as shown in FIG. 1, the model generation device 1 according to the present embodiment functions as a computer that includes the first data acquisition section 111, the background acquisition section 112, the difference calculation section 113, the first training section 114, the second data acquisition section 115, the second training section 116, the third data acquisition section 117, the third training section 118, and the saving processing section 119 as software modules. That is, in the present embodiment, the respective software modules of the model generation device 1 are realized by the control section 11 (CPU). Figure 4A - Figure 4C

[0091] ​The first data acquisition unit 111 acquires multiple learning datasets 30, each consisting of a combination of training data 31 (which serves as image data) and positive solution data 32 representing the features contained in the training data 31. Each learning dataset 30 is generated assuming a scenario where an inference task related to the features of an object is performed. The background acquisition unit 112 acquires background data 35 representing the background of the training data 31 for each learning dataset 30. Background data 35 corresponding to the training data 31 of each learning dataset 30 can be appropriately generated. In this embodiment, acquiring background data 35 may also include generating background data 35 for each learning dataset 30 based on the training data 31 using a generator 60 that has completed machine learning.

[0092] The difference calculation unit 113 calculates the difference between the acquired background data 35 and the training data 31 for each learning dataset 30, thereby generating difference data 37 representing the difference between the background data 35 and the training data 31. The first training unit 114 uses the generated difference data 37 to perform machine learning on the inferr 5. In this machine learning, the first training unit 114 trains the inferr 5 for each learning dataset 30 such that the result of the inferr 5 inferring features based on the generated difference data 37 is suitable for the correct solution data 32.

[0093] The second data acquisition unit 115 acquires multiple pieces of learning background data 39. The learning background data 39 are generated in a manner that reflects the same background as the training data 31 of each learning dataset 30. Preferably, the learning background data 39 are generated such that they do not reflect information related to the features of the object, but only the background. The second training unit 116 constructs a generator 60 trained to generate background data 35 for the training data 31 by implementing machine learning using the acquired learning background data 39. Similar to the estimator 5, any machine learning model can be used in the generator 60. The input and output formats of the generator 60 can be arbitrarily determined.

[0094] As long as background data 35 corresponding to training data 31 can be generated, the structure of generator 60 and the machine learning method can be arbitrarily chosen. For example... Figure 4B As shown, in this embodiment, a recognizer 61 is used in the training of the generator 60. The generator 60 is configured to accept noise 631 (a latent variable) as input, generate simulated background data 633 based on the input noise 631, and output an output value corresponding to the generated simulated background data 633. In contrast, the recognizer 61 is configured to accept background data (samples) as input, identify the source of the input background data (samples), and output an output value corresponding to the identification result. Similar to the generator 60, the recognizer 61 can utilize any machine learning model. The form of the input and output of the recognizer 61 can be arbitrarily determined.

[0095] The machine learning of the generator 60 according to the present embodiment includes a first training step of training the discriminator 61 and a second training step of training the generator 60. In the first training step, the second training unit 116 trains the discriminator 61 to improve the performance of identifying whether the input background data is the learning-use background data 39 or the simulated background data 633 generated by the generator 60 (in other words, identifying the origin of the background data). On the other hand, in the second training step, the second training unit 116 trains the generator 60 to improve the performance of generating the simulated background data 633 that reduces the identification performance of the discriminator 61. The second training unit 116 alternately and repeatedly performs the first training step and the second training step. That is, the machine learning of the generator 60 according to the present embodiment is constituted by adversarial learning between the generator 60 and the discriminator 61.

[0096] Through the above-described machine learning, the trained generator 60 according to the present embodiment acquires the ability to generate simulated background data suitable for the learning-use background data 39 from noise of a latent variable. That is, according to the trained generator 60 according to the present embodiment, it is possible to generate simulated image data from noise, which is highly reproducible for the background given by the learning-use background data 39 and is less reproducible for information other than the same. Therefore, by using this trained generator 60, it is possible to generate the background data 35 corresponding to the training data 31 with respect to each of the learning data sets 30.

[0097] In addition, as described above, in the present embodiment, the trained generator 60 is configured to generate simulated background data from noise. Therefore, in order to generate the background data 35 corresponding to the training data 31 by using this trained generator 60, it is necessary to determine an appropriate input value (noise) to be given to the trained generator 60. The model generation apparatus 1 according to the present embodiment further generates an input estimator 62 of the trained generator 60, which can be used to determine the input value.

[0098] The input estimator 62 is configured to accept the input of image data, estimate the input value (noise) to be given to the trained generator 60 in order to generate (simulated) background data suitable for the background reproduced in the input image data, and output an output value corresponding to the obtained estimated value. As with the estimator 5 described above, an arbitrary machine learning model can be used in the input estimator 62. The form of the input and the output of the input estimator 62 can be arbitrarily determined.

[0099] The third data acquisition unit 117 acquires multiple learning datasets 64. Each learning dataset 64 consists of a combination of noise 641 of the samples used as input values ​​and simulated background data 643 generated by applying noise 641 to a generator 60 that has completed machine learning. The generator 60 that has completed machine learning can be used when generating each learning dataset 64.

[0100] The third training unit 118 uses the acquired multiple learning datasets 64 to perform machine learning on the input estimator 62. In this machine learning, the third training unit 118 trains the input estimator 62 on each learning dataset 64 such that the estimated value obtained by the input estimator 62 based on the simulated background data 643 is suitable for the noise 641. Thus, the input estimator 62 generates (simulated) background data suitable for the background reflected in the image data through the machine learning-completed generator 60, and is able to obtain the ability to estimate the input value (noise) assigned to the machine learning-completed generator 60.

[0101] The storage processing unit 119 generates information related to the trained inferrer 5, generator 60, and input inferrer 62, respectively constructed through each machine learning method, as first to third learning result data 121 to 123. Then, the storage processing unit 119 stores the generated first to third learning result data 123 in a designated storage area. The designated storage area can be arbitrarily selected.

[0102] (Structure of a machine learning model)

[0103] In the inferrer 5, generator 60, recognizer 61, and input inferrer 62, any model capable of implementing machine learning can be utilized. For example... Figure 4A - Figure 4C As shown, in this embodiment, the estimator 5, generator 60, recognizer 61, and input estimator 62 are each composed of a multi-layered fully associative neural network. The estimator 5, generator 60, recognizer 61, and input estimator 62 each have an input layer (51, 601, 611, 621), an intermediate (hidden) layer (52, 602, 612, 622), and an output layer (53, 603, 613, 623).

[0104] In addition, the structures of the inferencer 5, the generator 60, the recognizer 61, and the input inferencer 62 can not be limited to such examples and can be appropriately determined according to the embodiments. For example, the number of intermediate layers (52, 602, 612, 622) is not limited to one and can be two or more. Alternatively, the intermediate layers (52, 602, 612, 622) can be omitted. The number of layers constituting each neural network can be arbitrarily selected. As for the combination of at least any one of the inferencer 5, the generator 60, the recognizer 61, and the input inferencer 62, the configurations can be at least partially the same or different.

[0105] Each layer (51 to 53, 601 to 603, 611 to 613, 621 to 623) has one or more neurons (nodes). The number of neurons (nodes) included in each layer (51 to 53, 601 to 603, 611 to 613, 621 to 623) is not particularly limited and can be appropriately selected according to the embodiments. The neurons of adjacent layers are appropriately combined. In the example of FIG. 6, each neuron is combined with all the neurons of the adjacent layer. However, the combination relationship of each neuron can not be limited to such an example and can be appropriately set according to the embodiments. Figure 4A - Figure 4C

[0106] A weight (combination weight) is set for each combination. A threshold value is set for each neuron, and the output of each neuron is basically determined according to whether the sum of the products of each input and each weight exceeds the threshold value. The threshold value can also be expressed by an activation function. In this case, the sum of the products of each input and each weight is input to the activation function, and the operation of the activation function is performed to determine the output of each neuron. The type of the activation function can be arbitrarily selected. The weight of the combination between each neuron included in each layer (51 to 53, 601 to 603, 611 to 613, 621 to 623) and the threshold value of each neuron are examples of operation parameters used in the operation processing of each of the inferencer 5, the generator 60, the recognizer 61, and the input inferencer 62. The values of the operation parameters of each of the inferencer 5, the generator 60, the recognizer 61, and the input inferencer 62 are adjusted by the machine learning described above to obtain an output suitable for the desired inference.

[0107] (Machine learning of inferencer)

[0108] As described above, the machine learning of the inferencer 5, the generator 60, the recognizer 61, and the input inferencer 62 is performed by the machine learning device 1. The machine learning of the inferencer 5, the generator 60, the recognizer 61, and the input inferencer 62 can be performed by the machine learning device 1 or another machine learning device. Figure 4A ​As shown, the first training unit 114 inputs the generated difference data 37 to the input layer 51 of the inferr 5 for each learning dataset 30, and obtains the output value corresponding to the result of the inferred object's features from the output layer 53. In the machine learning of the inferr 5, the first training unit 114 adjusts the values ​​of the inferr 5's operational parameters to reduce the error between the obtained output value and the correct solution data 32. Thus, for each learning dataset 30, a trained inferr 5 capable of inferring the features of the object based on the difference data 37 can be generated.

[0109] The storage processing unit 119 generates first learning result data 121 for reproducing the trained inferr 5 generated through machine learning. The structure of the first learning result data 121 can be arbitrary, as long as the trained inferr 5 can be reproduced. For example, the storage processing unit 119 generates information representing the values ​​of the operational parameters of the generated trained inferr 5 as the first learning result data 121. Depending on the situation, the first learning result data 121 may also include information representing the structure of the trained inferr 5. The structure can be determined, for example, by the number of layers from the input layer to the output layer in the neural network, the types of each layer, the number of neurons in each layer, and the connection relationships between neurons in adjacent layers. The storage processing unit 119 stores the generated first learning result data 121 in a designated storage area.

[0110] (Machine Learning with Generators)

[0111] like Figure 4B As shown, the machine learning of the generator 60 according to this embodiment includes the first training step and the second training step described above. In the first training step, the second training unit 116 inputs noise 631 to the input layer 601 of the generator 60 and obtains an output value corresponding to the simulated background data 633 generated based on the noise 631 from the output layer 603. The second training unit 116 inputs the generated simulated background data 633 to the input layer 611 of the recognizer 61 and obtains an output value corresponding to the result of recognizing the source of the input background data from the output layer 613. In addition, the second training unit 116 inputs the learning background data 39 to the input layer 611 of the recognizer 61 and obtains an output value corresponding to the result of recognizing the source of the input background data from the output layer 613.

[0112] exist Figure 4BIn the example of FIG. 6, the case from the learning data is expressed as "true" and the case from the generator 60 is expressed as "false". "True" can be replaced with "1" and "false" can be replaced with "0". In addition, the method of expressing each origin is not limited to this example and can be appropriately selected according to the embodiment. In the two scenes in which the result of identifying the origin of the above-described background data is obtained, in the scene in which the generated simulated background data 633 is input to the discriminator 61, the discriminator 61 identifies "true" as the correct answer. On the other hand, in the scene in which the learning background data 39 is input to the discriminator 61, the discriminator 61 identifies "false" as the correct answer. In the first training step, the second training unit 116 adjusts the values of the operation parameters of the discriminator 61 so that the error between the output value obtained from the output layer 613 in each scene and each correct answer becomes small, on the basis of fixing the values of the operation parameters of the generator 60. Thereby, it is possible to train the discriminator 61 to improve the performance of identifying the origin of the input background data.

[0113] On the other hand, in the second training step, the second training unit 116 inputs the noise 631 to the input layer 601 of the generator 60 and obtains the output value corresponding to the simulated background data 633 generated from the noise 631 from the output layer 603. The process of generating this simulated background data 633 can be commonly performed in the first training step and the second training step. That is, the simulated background data 633 generated in the first training step can also be used in the second training step. The second training unit 116 inputs the generated simulated background data 633 to the input layer 611 of the discriminator 61 and obtains the output value corresponding to the result of identifying the origin of the input background data from the output layer 613.

[0114] In the second training step, the identification performance of the discriminator 61 (i.e., the identification result is wrong) is the correct answer. That is, the output value obtained from the output layer 613 corresponds to "true" as the correct answer. The second training unit 116 adjusts the values of the operation parameters of the generator 60 so that the error between the output value obtained from the output layer 613 and the correct answer becomes small, on the basis of fixing the values of the operation parameters of the discriminator 61. Thereby, it is possible to train the generator 60 to improve the performance of generating the simulated background data 633 that reduces the identification performance of the discriminator 61.

[0115] The second training unit 116 can alternately improve the performance of the recognizer 61 and the generator 60 by repeatedly executing the first and second training steps. Therefore, by improving the recognition performance of the recognizer 61, the performance of the generator 60, which generates simulated background data 633 suitable for the learning background data 39, can be improved. Thus, through the aforementioned machine learning, a trained generator 60 can be constructed that can generate simulated background data based on noise. This simulated background data exhibits high background reproducibility in image data obtained from the scene where the inference task is performed, but low reproducibility for other information. It should be noted that the processing order of the first and second training steps can be arbitrary.

[0116] The storage processing unit 119 generates second learning result data 122 for reproducing the trained generator 60 generated through machine learning. The structure of the second learning result data 122 can be the same as that of the first learning result data 121 described above. For example, the second learning result data 122 may include information indicating the values ​​of the operational parameters of the trained generator 60. Depending on the situation, the second learning result data 122 may also include information indicating the structure of the trained generator 60. The storage processing unit 119 stores the generated second learning result data 122 in a predetermined storage area. It should be noted that the processing of the trained recognizer 61 can be arbitrary. The second learning result data 122 may or may not include information for reproducing the trained recognizer 61.

[0117] (Machine learning with input inferrer)

[0118] like Figure 4C As shown, the third training unit 118 inputs simulated background data 643 into the input layer 621 of the input estimator 62 for each learning dataset 64, and obtains the output value from the output layer 623. The output value obtained from the input estimator 62 corresponds to the result of estimating the input value (noise) assigned to the machine learning-completed generator 60 for generating the input simulated background data 643 by the machine learning-completed generator 60. The third training unit 118 adjusts the values ​​of the operation parameters of the input estimator 62 so that the error between the obtained output value and the corresponding noise 641 is reduced. Thus, since the machine learning-completed generator 60 generates (simulated) background data suitable for the background reflected in the image data, it is possible to generate a trained input estimator 62 that has the ability to estimate the input value (noise) assigned to the machine learning-completed generator 60.

[0119] The storage processing unit 119 generates third learning result data 123, which is used to reproduce the trained input estimator 62 generated through machine learning. The structure of the third learning result data 123 may be the same as that of the first learning result data 121 described above. For example, the third learning result data 123 may include information indicating the values ​​of the operation parameters of the trained input estimator 62. Depending on the situation, the third learning result data 123 may also include information indicating the structure of the trained input estimator 62. The storage processing unit 119 stores the generated third learning result data 123 in a designated storage area.

[0120] (Generating background data)

[0121] like Figure 4C As shown, in this embodiment, the background acquisition unit 112 can generate background data 35 corresponding to the training data 31 of each learning dataset 30 by utilizing the generator 60 trained by the above-described machine learning and the input estimator 62.

[0122] Specifically, the background acquisition unit 112 inputs training data 31 into the input layer 621 of the machine learning-completed input inferencer 62 for each learning dataset 30, and obtains output values ​​from the output layer 623. The output values ​​obtained from the machine learning-completed input inferencer 62 correspond to the estimated values ​​of noise assigned to the machine learning-completed generator 60 in order to generate simulated background data suitable for the background reflected in the training data 31.

[0123] Next, the background acquisition unit 112 inputs the estimated value of the acquired noise into the input layer 601 of the generator 60 after machine learning, and obtains the output value corresponding to the simulated background data generated based on the estimated value of the noise from the output layer 603. The background acquisition unit 112 can acquire the generated simulated background data as background data 35 for the training data 31.

[0124] <Predictive Apparatus>

[0125] Figure 5 An example of the software structure of the estimation device 2 according to this embodiment is illustrated schematically.

[0126] The control unit 21 of the estimation device 2 expands the estimation program 82 stored in the storage unit 22 into RAM. Furthermore, the control unit 21, via the CPU, interprets and executes the commands contained in the estimation program 82 expanded in RAM, thereby controlling each component. Thus, as... Figure 5As shown, the estimation device 2 according to the present embodiment functions as a computer that includes the data acquisition section 211, the background acquisition section 212, the difference calculation section 213, the estimation section 214, and the output section 215 as software modules. That is, in the present embodiment, each of the software modules of the estimation device 2 is implemented by the control section 21 (CPU), like the model generation device 1.

[0127] The data acquisition section 211 acquires the object image data 41 that is the execution target of the estimation task. The background acquisition section 212 acquires the object background data 45 that corresponds to the object image data 41. In the present embodiment, the background acquisition section 212 includes the generator 60 and the input estimator 62 that have completed machine learning, by retaining the second learning result data 122 and the third learning result data 123. The background acquisition section 212 can also estimate the noise to be assigned to the generator 60 that has completed machine learning, in order to generate simulated background data that is suitable for the background that is represented in the object image data 41, by assigning the object image data 41 to the input estimator 62 that has completed machine learning. In addition, the background acquisition section 212 can generate simulated background data by assigning the estimated value of the noise that is acquired to the generator 60 that has completed machine learning. The background acquisition section 212 can also acquire the generated simulated background data as the object background data 45.

[0128] The difference calculation section 213 generates the object difference data 47 by taking the difference between the object image data 41 and the object background data 45. The estimation section 214 includes the estimator 5 that has completed machine learning, by retaining the first learning result data 121. The estimation section 214 estimates the feature included in the generated object difference data 47, using the estimator 5 that has completed machine learning. The output section 215 outputs information related to the result of estimating the feature.

[0129] <Other>

[0130] The software modules of the model generation device 1 and the estimation device 2 are described in detail in the operation example described later. Note that, in the present embodiment, an example is described in which each of the software modules of the model generation device 1 and the estimation device 2 is implemented by a general-purpose CPU. However, part or all of the above software modules can also be implemented by one or more dedicated processors. In addition, the software structure of each of the model generation device 1 and the estimation device 2 can also be appropriately omitted, replaced, and added with respect to the software modules, according to the embodiment.

[0131] §3 Operation Example

[0132] [Model Generation Device]

[0133] (A) Machine Learning of Generator

[0134] Figure 6A is a flowchart indicating an example of a processing sequence related to machine learning of the generator 60 by the model generation device 1 according to the present embodiment. The processing sequence described below is merely an example, and each step can be changed as much as possible. Furthermore, regarding the processing sequence described below, omission, replacement, and addition of steps can be appropriately performed according to the embodiment.

[0135] (Step S101)

[0136] In step S101, the control section 11 functions as the second data acquisition section 115 and acquires a plurality of pieces of learning background data 39.

[0137] The learning background data 39 can be appropriately generated. For example, assuming a scene in which a predetermined task is performed in a real space or a virtual space, the learning background data 39 can be generated by capturing a background using a camera under various conditions. At this time, it is preferable to generate the learning background data 39 so that information related to a feature of an object is not reflected and only a background is reflected. The background can be appropriately selected according to the predetermined task. Regarding different kinds of backgrounds such as different bases, it is preferable to collect the learning background data 39 inclusively. On the other hand, regarding a deviation such as a shift in a captured position in the same kind of background, the learning background data 39 can be collected exclusively. The kind of captured background is the same as that of the training data 31. In a case where the training data 31 in which a feature of an object and a background are reflected is obtained, the learning background data 39 can be generated by appropriately removing information related to the feature of the object from the training data 31 using image processing. Alternatively, the learning background data 39 can be acquired separately from the training data 31.

[0138] The learning background data 39 can be automatically generated by an operation of a computer or manually generated by at least partially including an operation of an operator. In addition, the generation of the learning background data 39 can be performed by the model generation device 1 or by another computer other than the model generation device 1. In a case where the model generation device 1 generates the learning background data 39, the control section 11 automatically or manually performs the above-described generation processing by an operation of the operator via the input device 15, and thereby acquires the learning background data 39. On the other hand, in a case where the learning background data 39 is generated by another computer, the control section 11 acquires the learning background data 39 generated by the other computer, for example, via a network, a storage medium 91, or the like. A part of the learning background data 39 can be generated by the model generation device 1, and the other learning background data 39 can be generated by one or more other computers.

[0139] The number of pieces of the acquired learning background data 39 can be arbitrarily selected. When a plurality of pieces of the learning background data 39 are acquired, the control section 11 causes the processing to proceed to the next step S102.

[0140] (Step S102)

[0141] In step S102, the control section 11 functions as the second training section 116 and performs machine learning of the generator 60 using the acquired pieces of learning background data 39. Thereby, the generator 60 that has completed machine learning of the background data 35 with respect to the training data 31 is constructed.

[0142] Figure 6B is a flowchart showing an example of a processing order of a subroutine related to machine learning of the generator 60 according to the present embodiment. The processing of step S102 according to the present embodiment includes the processing of steps S1021 to S1023 below. Note that the processing order described below is merely an example, and each process can be changed as much as possible. In addition, regarding the processing order described below, omission, replacement, and addition of steps can be appropriately performed according to the embodiment.

[0143] In the present embodiment, the discriminator 61 is utilized in the training of the generator 60. The neural networks respectively constituting the generator 60 and the discriminator 61 that become processing targets can be prepared as appropriate. The structure of each of the generator 60 and the discriminator 61 (for example, the number of layers, the number of neurons included in each layer, the binding relationship between the neurons of adjacent layers, and the like), the initial values of the weights of the binding between the neurons, and the initial values of the thresholds of the neurons can be given by a template or by input by an operator. In addition, in the case of performing relearning, the control section 11 can prepare the generator 60 and the discriminator 61 based on learning result data obtained through past machine learning.

[0144] (Step S1021)

[0145] In step S1021, the control section 11 trains the discriminator 61 using the learning background data 39 and the simulated background data 633 generated by the generator 60 to recognize the origin of the input background data. Step S1021 is an example of the above-described first training step. In the training processing of this machine learning, a probabilistic gradient descent method, a mini-batch gradient descent method, or the like can be used. The same applies to the processing of machine learning of other neural networks of step S1022 and the like described below.

[0146] As an example, first, the control section 11 extracts a noise 631 (latent variable) from a prescribed probability distribution. The prescribed probability distribution can be arbitrarily selected. The prescribed probability distribution can be, for example, a known probability distribution such as a Gaussian distribution. Next, the control section 11 assigns the extracted noise 631 to the generator 60, and executes the operation processing of the generator 60. That is, the control section 11 inputs the noise 631 to the input layer 601 of the generator 60, and sequentially performs the firing judgment of each neuron included in each layer 601 to 603 from the input side (that is, performs the operation of the forward propagation). Through this operation processing, the control section 11 acquires the output value corresponding to the simulated background data 633 generated from the noise 631 from the output layer 603. The number of pieces of the generated simulated background data 633 can be arbitrary. Note that this operation processing of the forward propagation is also similarly performed in the operation processing of the other neural networks below.

[0147] Next, the control section 11 inputs the generated simulated background data 633 to the input layer 611 of the discriminator 61, and executes the operation processing of the discriminator 61. Thereby, the control section 11 acquires the output value corresponding to the result of identifying the origin of the input background data from the output layer 613 of the discriminator 61. In this scenario, since the input background data is the simulated background data 633, the discriminator 61 identifies that "false" is the correct answer. The control section 11 calculates the error between the output value obtained from the output layer 613 and this correct answer with respect to each piece of simulated background data 633.

[0148] In addition, the control section 11 inputs each piece of the learning background data 39 to the input layer 611 of the discriminator 61, and executes the operation processing of the discriminator 61. Thereby, the control section 11 acquires the output value corresponding to the result of identifying the origin of the input background data from the output layer 613 of the discriminator 61. In this scenario, since the input background data is the learning background data 39, the discriminator 61 identifies that "true" is the correct answer. The control section 11 calculates the error between the output value obtained from the output layer 613 and this correct answer with respect to each piece of learning background data 39.

[0149] In the calculation of each error (loss), a loss function can be used. The loss function is a function that evaluates the difference (that is, the degree of difference) between the output of the machine learning model and the correct answer, and the larger the difference value between the output value obtained from the output layer 613 and the correct answer, the larger the error value calculated by the loss function. The kind of loss function used to calculate the error is not particularly limited, and can be appropriately selected according to the embodiment.

[0150] The control section 11 calculates errors in the values of the respective operation parameters (weights of the connections between the respective neurons, threshold values of the respective neurons, etc.) of the recognizer 61 using the gradients of the errors in the calculated output values by a back propagation method. The control section 11 updates the values of the respective operation parameters of the recognizer 61 on the basis of the calculated errors. The degree of updating the values of the respective operation parameters can be adjusted by a learning rate. The learning rate can be assigned by designation by an operator or can be assigned as a set value within a program.

[0151] The control section 11 adjusts the values of the respective operation parameters of the recognizer 61 so that the sum of the errors calculated by the above series of update processes becomes smaller on the basis of the values of the respective operation parameters of the generator 60 being fixed. For example, the control section 11 can repeatedly adjust the values of the respective operation parameters of the recognizer 61 by the above series of processes until a prescribed condition, such as the sum of the calculated errors being below a threshold value, being performed a prescribed number of times, etc., is satisfied. Thus, the control section 11 can train the recognizer 61 to improve the performance of recognizing the origins of the input background data. When the training process of this recognizer 61 is completed, the control section 11 causes the process to proceed to the next step S1022.

[0152] (Step S1022)

[0153] In step S1022, the control section 11 trains the generator 60 to generate simulated background data 633 that reduces the recognition performance of the recognizer 61. In other words, the control section 11 trains the generator 60 to generate simulated background data 633 that is recognized as "true" (i.e., recognized as coming from the learning data) by the recognizer 61. Step S1022 is an example of the above second training step.

[0154] As an example, first, the control section 11 generates simulated background data 633 by assigning noise 631 extracted from a prescribed probability distribution to the generator 60, similarly to the above step S1021. In a case where the simulated background data 633 generated by the above step S1021 is also used in this step S1022, the control section 11 can omit the process of generating simulated background data 633 in this step S1022.

[0155] Next, the control section 11 inputs the generated simulated context data 633 to the input layer 611 of the recognizer 61, and executes the operation processing of the recognizer 61. Thereby, the control section 11 acquires, from the output layer 613 of the recognizer 61, an output value corresponding to the result of recognizing the origin of the input context data. In this scenario, if the input context data is "true" (i.e., from the learning data), the recognizer 61 makes a false recognition that is a correct answer. The control section 11 calculates, with respect to each simulated context data 633, an error between the output value obtained from the output layer 613 and this correct answer. As described above, an arbitrary loss function can be used in the calculation of this error.

[0156] The control section 11 calculates, by the error backpropagation method, the gradient of the error of the calculated output value, and inversely propagates it to each operation parameter of the generator 60 via the recognizer 61, to calculate the error of the value of each operation parameter of the generator 60. The control section 11 updates the value of each operation parameter of the generator 60 on the basis of the calculated errors. As described above, the degree of updating the value of each operation parameter can be adjusted by the learning rate.

[0157] The control section 11 adjusts the value of each operation parameter of the generator 60 so that the sum of the errors calculated by the above series of update processing becomes smaller, on the basis of fixing the value of each operation parameter of the recognizer 61. As with the training of the recognizer 61 described above, the control section 11 can repeatedly adjust the value of each operation parameter of the generator 60 by the above series of processing until a prescribed condition is satisfied. Thereby, the control section 11 can train the generator 60 to improve the performance of generating simulated context data 633 that reduces the recognition performance of the recognizer 61. When the training processing of this generator 60 is completed, the control section 11 advances the processing to the next step S1023.

[0158] (Step S1023)

[0159] In step S1023, the control section 11 determines whether to repeat the processing of steps S1021 and S1022. The criterion for repeating the processing can be arbitrarily determined. For example, a prescribed number of times of repeating the processing can be set. The prescribed number of times can be given by the designation of an operator, for example, or can be given as a set value within the program. In this case, the control section 11 determines whether the number of times of executing the processing of step S1021 and step S1022 reaches the prescribed number of times. In the case where it is determined that the number of times of execution does not reach the prescribed number of times, the control section 11 returns the processing to step S1021, and executes the processing of step S1021 and step S1022 again. On the other hand, in the case where it is determined that the number of times of execution reaches the prescribed number of times, the control section 11 completes the processing of the machine learning of the generator 60, and advances the processing to the next step S103.

[0160] As described above, the control section 11 alternately repeats the training step of the recognizer 61 and the training step of the generator 60. Thereby, it is possible to alternately improve the performance of the recognizer 61 and the generator 60. As a result, it is possible to construct the trained generator 60 that has acquired the ability to generate the simulated background data 633 (thereby, generate the background data 35 for the training data 31) suitable for the learning-use background data 39.

[0161] (Step S103)

[0162] Return Figure 6A In step S103, the control section 11 functions as a saving processing section 119, generates information related to the trained generator 60 constructed by machine learning as second learning result data 122. Then, the control section 11 saves the generated second learning result data 122 in a prescribed storage area.

[0163] The prescribed storage area can be, for example, a RAM within the control section 11, the storage section 12, an external storage device, a storage medium, or a combination thereof. The storage medium can be, for example, a CD, a DVD, or the like, and the control section 11 can store the second learning result data 122 in the storage medium via the drive 17. The external storage device can be, for example, a data server such as a NAS (Network Attached Storage) or the like. In this case, the control section 11 can also store the second learning result data 122 in the data server via a network using the communication interface 13. Further, the external storage device can be, for example, an external storage device connected to the model generation apparatus 1 via the external interface 14.

[0164] Thereby, when the saving of the second learning result data 122 is completed, the control section 11 ends the series of processes related to the machine learning of the generator 60.

[0165] Note that the generated second learning result data 122 can be provided to the estimation apparatus 2 at an arbitrary timing. For example, the control section 11 can or can not perform the process of "transferring the second learning result data 122 to the estimation apparatus 2" as the process of step S103. The estimation apparatus 2 can acquire the second learning result data 122 by receiving the transfer. Further, for example, the estimation apparatus 2 can access the model generation apparatus 1 or a data server via a network using the communication interface 23, and thereby acquire the second learning result data 122. Further, for example, the estimation apparatus 2 can acquire the second learning result data 122 via the storage medium 92. Further, for example, the second learning result data 122 can be pre-embedded in the estimation apparatus 2.

[0166] Further, the control section 11 can update or newly generate the second learning result data 122 by repeating the processing of the above-described steps S101 to S103 periodically or non-periodically. At the time of such repetition, the change, the correction, the addition, the deletion, or the like of at least a part of the learning-use background data 39 can be appropriately performed. In addition, the control section 11 can update the second learning result data 122 held by the estimation device 2 by providing the updated or newly generated second learning result data 122 to the estimation device 2 by using an arbitrary method.

[0167] (B) Machine learning of input estimator

[0168] Figure 7 is a flowchart indicating an example of a processing sequence related to the machine learning of the input estimator 62 of the model generation device 1 according to the present embodiment. The processing sequence described below is merely an example, and each step can be changed as much as possible. Further, with respect to the processing sequence described below, omission, replacement, and addition of steps can be appropriately performed according to the embodiment.

[0169] (Step S111)

[0170] In step S111, the control section 11 functions as the third data acquisition section 117 and acquires a plurality of learning data sets 64. Each learning data set 64 is constituted by a combination of a noise 641 of a sample as an input value, and a simulated background data 643 generated by giving the noise 641 to the generator 60 whose machine learning is completed.

[0171] Each learning data set 64 can be appropriately generated by using the generator 60 whose machine learning is completed. As an example, a noise 641 (latent variable) is extracted from a prescribed probability distribution (for example, a Gaussian distribution or the like). The extracted noise 641 is input to the input layer 601 of the generator 60 whose machine learning is completed, and the operation processing of the generator 60 whose machine learning is completed is performed. By this operation processing, it is possible to acquire an output value corresponding to the simulated background data 643 generated from the noise 641 from the output layer 603 of the generator 60 whose machine learning is completed. The input noise 641 is associated with the generated simulated background data 643. Thus, each learning data set 64 can be generated. Note that the structure of the learning data set 64 can not be limited to such an example. For example, the learning data set 64 can include image data instead of the simulated background data 643, the image data further including information other than the background (for example, a feature of an object). In this case, the same noise 641 is associated with the image data representing the same background.

[0172] As with the above-mentioned learning background data 39, each of the learning data sets 64 can be automatically generated by the action of the computer, or can be manually generated by at least partially containing the operation of the operator. In addition, the generation of each of the learning data sets 64 can be performed by the model generation device 1, or can be performed by another computer other than the model generation device 1. It is also possible that a part of the learning data sets 64 is generated by the model generation device 1, and the other learning data sets 64 are generated by one or more other computers.

[0173] The number of pieces of the acquired learning data sets 64 can be arbitrarily selected. When a plurality of learning data sets 64 are acquired, the control section 11 causes the process to proceed to the next step S112.

[0174] (Step S112)

[0175] In step S112, the control section 11 acts as a third training section 118, and performs machine learning of the input estimator 62 using the plurality of acquired learning data sets 64. In this machine learning, the control section 11 trains the input estimator 62 with respect to each of the learning data sets 64 so that the estimated value obtained by estimating the input value to be given to the generator 60 whose machine learning is completed from the input estimator 62 according to the simulated background data 643 is fitted to the noise 641.

[0176] As an example, first, the control section 11 inputs the simulated background data 643 to the input layer 621 of the input estimator 62 with respect to each of the learning data sets 64, and executes the operation processing of the input estimator 62. Thereby, the control section 11 acquires the output value corresponding to the result of estimating the input value (noise) to be given to the generator 60 whose machine learning is completed from the output layer 623 in order to generate the simulated background data 643. The control section 11 calculates the error between the obtained estimated value and the corresponding noise 641. As with the above-mentioned step S1021 and the like, an arbitrary loss function can be used in the calculation of the error.

[0177] The control section 11 calculates the error of the value of each of the operation parameters of the input estimator 62 using the gradient of the calculated error by the error backpropagation method. The control section 11 updates the value of each of the operation parameters of the input estimator 62 on the basis of the calculated errors. As with the above-mentioned step S1021 and the like, the degree of updating the value of each of the operation parameters can be adjusted by the learning rate.

[0178] The control section 11 adjusts the value of each of the operation parameters of the input estimator 62 so that the sum of the errors calculated by the above-mentioned series of update processing becomes smaller. As with the above-mentioned step S1021 and the like, the control section 11 can also repeatedly adjust the value of each of the operation parameters of the input estimator 62 by the above-mentioned series of processing until a prescribed condition is satisfied.

[0179] Through this machine learning, the input estimator 62 is trained with respect to each of the learning data sets 64 so that the noise 641 can be appropriately estimated from the simulation context data 643. That is, the input estimator 62 is trained to estimate the input from the output of the generator 60 that has completed machine learning. The generator 60 that has completed machine learning is trained to generate simulation image data that is high in context reproducibility and low in reproducibility of information other than the context. Thus, the input estimator 62 is trained to be able to estimate the same input value (noise) as long as the image data is of the same context, even if it contains other information. As long as the same noise 641 is associated with the image data of the same context, the input estimator 62 can be trained similarly even if the simulation context data 643 is replaced with image data that also contains other information. Thus, through the above-described machine learning, in order to generate simulation context data of a context that is reproduced in the image data given, a trained input estimator 62 that has acquired the ability to estimate the input value (noise) given to the generator 60 that has completed machine learning can be constructed. When the machine learning of the input estimator 62 is completed, the control section 11 causes the process to proceed to the next step S113.

[0180] (Step S113)

[0181] In step S113, the control section 11 functions as a saving processing section 119 and generates information related to the trained input estimator 62 constructed through machine learning as third learning result data 123. Then, the control section 11 saves the generated third learning result data 123 in a prescribed storage area. The prescribed storage area can be, for example, a RAM within the control section 11, the storage section 12, an external storage device, a storage medium, or a combination thereof.

[0182] Thus, when the saving of the third learning result data 123 is completed, the control section 11 ends the series of processes related to the machine learning of the input estimator 62. Note that, as with the above-described second learning result data 122, the generated third learning result data 123 can be provided to the estimation device 2 at an arbitrary timing. In addition, the above-described processes of steps S111 to S113 can be repeated periodically or aperiodically. Then, the updated or newly generated third learning result data 123 can be provided to the estimation device 2 by an arbitrary method, and thus the third learning result data 123 held by the estimation device 2 can be updated.

[0183] (C) Machine Learning of Estimator

[0184] Figure 8is a flowchart showing an example of a processing procedure related to machine learning of the inferencer 5 performed by the model generation device 1 according to the present embodiment. The processing procedure described below is an example of the "model generation method" of the present application. Note that the processing procedure described below is merely an example, and each step can be changed as much as possible. Furthermore, regarding the processing procedure described below, omission, replacement, and addition of steps can be appropriately performed depending on the embodiment.

[0185] (Step S121)

[0186] In step S121, the control section 11 functions as the first data acquisition section 111 and acquires a plurality of learning data sets 30. Each learning data set 30 is constituted by a combination of the training data 31 which is image data, and the correct answer data 32 which indicates the features of the object included in the training data 31.

[0187] Each learning data set 30 can be generated appropriately. For example, in a real space or a virtual space, by performing photographing of a camera together with a background in a state in which the features of the object are exhibited, the training data 31 can be obtained. As a specific example, in a case in which a trained inferencer which can be used for appearance inspection of a product is generated, by photographing the product with or without a defect using a camera, the training data can be obtained. Then, information indicating the result (correct answer) of inferring the features of the object included in the acquired training data 31 is associated with the acquired training data 31 as the correct answer data 32. The method of inferring the features of the object can be selected arbitrarily. For example, the inference of the features can be performed by an operator. Thus, each learning data set 30 can be generated.

[0188] Like the above-mentioned learning background data 39, each learning data set 30 can be automatically generated by the operation of a computer, or can be manually generated by at least partially including the operation of an operator. In addition, the generation of each learning data set 30 can be performed by the model generation device 1, or can be performed by another computer other than the model generation device 1. It is also possible that a part of the learning data sets 30 are generated by the model generation device 1, and the other learning data sets 30 are generated by one or more other computers.

[0189] The number of pieces of the acquired learning data sets 30 can be selected arbitrarily. When a plurality of learning data sets 30 are acquired, the control section 11 causes the processing to proceed to the next step S122.

[0190] (Step S122)

[0191] In step S122, the control section 11 functions as the background acquisition section 112 and acquires the background data 35 for the training data 31 with respect to each learning data set 30.

[0192] The background data 35 can be generated as appropriate. In the present embodiment, the control section 11 can generate the background data 35 for the training data 31 with respect to each of the learning data sets 30, using the generator 60 for which the machine learning is completed and the input estimator 62. Specifically, the control section 11 inputs the training data 31 to the input layer 621 of the input estimator 62 for which the machine learning is completed, with respect to each of the learning data sets 30, and executes the operation processing of the input estimator 62. Thereby, the control section 11 can acquire the output value corresponding to the result of estimating the input value to be given to the generator 60 for which the machine learning is completed, from the output layer 623 of the input estimator 62 for which the machine learning is completed, in order to generate the simulation background data suitable for the background represented in the input training data 31. Next, the control section 11 inputs the estimated value acquired by the input estimator 62 for which the machine learning is completed, to the input layer 601 of the generator 60 for which the machine learning is completed, and executes the operation processing of the generator 60 for which the machine learning is completed. Thereby, the control section 11 can acquire the output value corresponding to the simulation background data generated from the estimated value, from the output layer 603 of the generator 60 for which the machine learning is completed. The control section 11 can also acquire the generated simulation background data as the background data 35 for the training data 31, with respect to each of the learning data sets 30. When the background data 35 is acquired, the control section 11 causes the processing to proceed to the next step S123.

[0193] (Step S123)

[0194] In step S123, the control section 11 operates as the difference calculation section 113, and calculates the difference between the acquired background data 35 and the training data 31, with respect to each of the learning data sets 30, thereby generating the difference data 37. After the difference data 37 is generated, the control section 11 causes the processing to proceed to the next step S124.

[0195] Note that the method of the difference calculation can not be particularly limited as long as the information related to the background included in the training data 31 can be reduced based on the background data 35, and can be appropriately determined according to the embodiment. As an example, the control section 11 can also compare the values of each pixel at the same position between the training data 31 and the background data 35, set to "0" in the case of the same value, and directly adopt the pixel value of the training data 31 in the case of different values, thereby generating the difference data 37.

[0196] As other examples, the control section 11 can also simply differ the values of each pixel at the same position between the training data 31 and the background data 35 from each other. In this simple difference calculation, a slight background difference between the background data 35 and the training data 31 can remain in the difference data 37. In particular, in the case where the background data 35 is generated by the generator 60 completed by the above-described machine learning, this possibility is high. Therefore, in correspondence thereto, the control section 11 can also binarize the obtained difference data 37. For example, in the case where the pixel value of the difference data 37 is expressed between 0 and 255, the control section 11 can also compare the difference of the value of each pixel calculated between the training data 31 and the background data 35 with a threshold value. The threshold value can be arbitrarily determined. Then, the control section 11 can also generate the difference data 37 by setting the value of the pixel whose difference is smaller than the threshold value to "0" and setting the value of the pixel whose difference is larger than the threshold value to "255".

[0197] Figure 9 An example of another other method of calculating the difference of the training data 31 and the background data 35 is schematically illustrated. In this method, the control section 11 calculates the correlation between the object region 315 of each pixel 311 of the training data 31 and the corresponding region 355 of the corresponding pixel 351 of the background data 35. The object region 315 includes the pixel 311 of the object in the training data 31 and the pixels 312 present around the pixel 311 of the object. The object region 315 is defined for each pixel 311. The corresponding region 355 includes the pixel 351 corresponding to the pixel 311 of the object and the pixels 352 present around the pixel 351. The corresponding pixel 351 is present at the same position as the pixel 311 of the object. The corresponding region 355 is defined at the same position and range as the object region 315. The calculation of the correlation can employ a known calculation method such as a method of calculating a correlation coefficient. Figure 9 In the example, the object region 315 has a size of 3 x 3 by including the pixel 311 of the object and all 8 pixels 312 present around the pixel 311 of the object. In addition, the size and shape of the object region 315 can not be limited to this example, and can be arbitrarily determined as long as at least one pixel 312 present around the pixel 311 of the object is included. The corresponding region 355 includes the pixel 351 corresponding to the pixel 311 of the object and the pixels 352 present around the pixel 351. The corresponding pixel 351 is present at the same position as the pixel 311 of the object. The corresponding region 355 is defined at the same position and range as the object region 315. The calculation of the correlation can employ a known calculation method such as a method of calculating a correlation coefficient.

[0198] Then, the control section 11 calculates the difference between each pixel 311 of the training data 31 and the corresponding pixel 351 of the background data 35 based on the correlation calculated between the object region 315 and the corresponding region 355. That is, the control section 11 determines the difference result between the pixel 311 of the object and the corresponding pixel 351 according to the calculated correlation. The higher the calculated correlation, the more it is indicated that the pixel 311 of the object and the corresponding pixel 351 represent the same image of the background, and the lower the calculated correlation, the more it is indicated that the pixel 311 of the object represents a different image from the corresponding pixel 351 (i.e., an image other than the background such as a feature of the object). Based on this point of view, the relationship between the calculated correlation and the difference result can be arbitrarily defined. For example, the control section 11 can also compare the value of the calculated correlation with a threshold value, and in the case where the value of the correlation is lower than the threshold value, directly adopt the value of each pixel 311, and in the case where the value of the correlation is higher than the threshold value, set the value of the pixel of the object to "0", thereby generating the difference data 37. The threshold value can be arbitrarily determined. According to this method, in addition to the case where the background slightly differs between the background data 35 and the training data 31, even in the case where the background shifts within the range in which the correlation is calculated between the background data 35 and the training data 31, it is possible to reduce the case where the information related to the background remains in the difference data 37.

[0199] (Step S124)

[0200] In step S124, the control section 11 acts as the first training section 114 and performs machine learning of the estimator 5 using the generated difference data 37. In this machine learning, the control section 11 trains the estimator 5 with respect to each learning data set 30 so that the result of estimating the feature by the estimator 5 from the generated difference data 37 is fitted to the correct answer data 32.

[0201] As an example, first, the control section 11 inputs the generated difference data 37 to the input layer 51 of the estimator 5 with respect to each learning data set 30 and executes the operation processing of the estimator 5. Thereby, the control section 11 acquires the output value corresponding to the result of estimating the feature of the object included in the difference data 37 from the output layer 53 of the estimator 5. The control section 11 calculates the error between the output value obtained from the output layer 53 and the corresponding correct answer data 32. As with the above step S1021 and the like, an arbitrary loss function can be used in the calculation of the error.

[0202] The control section 11 calculates the error of the value of each operation parameter of the estimator 5 using the gradient of the calculated error by the error backpropagation method. The control section 11 updates the value of each operation parameter of the estimator 5 based on the calculated error. As with the above step S1021 and the like, the degree of updating the value of each operation parameter can be adjusted by the learning rate.

[0203] The control section 11 adjusts the values of the respective operation parameters of the estimator 5 so that the sum of the errors calculated through the above series of update processes becomes smaller. Similarly to the above step S1021, the control section 11 can also repeatedly adjust the values of the respective operation parameters of the estimator 5 through the above series of processes until a prescribed condition is satisfied.

[0204] Through this machine learning, the estimator 5 is trained with respect to each of the learning data sets 30 so that the features of the object can be appropriately estimated from the difference data 37. Therefore, as a result of this machine learning, when image data obtained through background subtraction is imparted, a trained estimator 5 that has acquired the ability to estimate the features of the object represented in the imparted image data can be constructed. When the machine learning of the estimator 5 is completed, the control section 11 causes the process to proceed to the next step S125.

[0205] (Step S125)

[0206] In step S125, the control section 11 functions as a saving process section 119 and generates information related to the trained estimator 5 constructed through the machine learning as first learning result data 121. Then, the control section 11 saves the generated first learning result data 121 in a prescribed storage area. The prescribed storage area can be, for example, a RAM within the control section 11, the storage section 12, an external storage device, a storage medium, or a combination thereof.

[0207] Thus, when the saving of the first learning result data 121 is completed, the control section 11 ends the series of processes related to the machine learning of the estimator 5. Note that, similarly to the above second learning result data 122, the generated first learning result data 121 can be provided to the estimation device 2 at an arbitrary timing. In addition, the above processes of steps S121 to S125 can be repeated periodically or aperiodically. In addition, the first learning result data 121 that is updated or newly generated can be provided to the estimation device 2 in an arbitrary method, thereby updating the first learning result data 121 held by the estimation device 2.

[0208] [Estimation Device]

[0209] Figure 10 is a flowchart showing an example of a processing sequence of the estimation device 2 according to the present embodiment. The processing sequence described below is an example of an estimation method. In addition, the processing sequence described below is merely an example, and each step can be changed as much as possible. Furthermore, with respect to the processing sequence described below, omission, replacement, and addition of steps can be appropriately performed according to the embodiment.

[0210] (Step S201)

[0211] In step S201, the control section 21 functions as a data acquisition section 211, and acquires the target image data 41.

[0212] The target image data 41 is a sample of image data that becomes a target of performing the estimation task. The target image data 41 can be acquired by any method. For example, by performing photographing related to a feature of a target using a camera, the target image data 41 can be acquired. The kind of the camera can be arbitrarily selected. In addition, the target image data 41 can be any one of a still image and a moving image. When the target image data 41 is acquired, the control section 21 causes the process to proceed to the next step S202.

[0213] (Step S202)

[0214] In step S202, the control section 21 functions as a background acquisition section 212, and acquires the target background data 45 corresponding to the target image data 41.

[0215] The target background data 45 can be appropriately generated. The target background data 45 can be generated by the same method as the background data 35 in the above step S122. In the present embodiment, the control section 21 can also perform the setting of the generator 60 and the input estimator 62 that have completed machine learning by referring to the second learning result data 122 and the third learning result data 123. Then, the control section 21 can also substitute the training data 31 with the target image data 41, and perform the same operation processing as the above step S122, thereby acquiring the target background data 45 corresponding to the target image data 41 using the generator 60 and the input estimator 62 that have completed machine learning. When the target background data 45 is acquired, the control section 21 causes the process to proceed to the next step S203.

[0216] (Step S203)

[0217] In step S203, the control section 21 functions as a difference calculation section 213, and generates the target difference data 47 by taking a difference between the target image data 41 and the target background data 45. The method of the difference calculation in the present step S203 can be arbitrarily determined if the information related to the background included in the target image data 41 can be reduced based on the target background data 45. The method of the difference calculation in the present step S203 can be the same as the above step S123. After the target difference data 47 is generated, the control section 21 causes the process to proceed to the next step S204.

[0218] (Step S204)

[0219] In step S204, the control section 21 functions as a presumption section 214, and presumes the feature included in the generated target difference data 47, using the presumption device 5 whose machine learning is completed. In the present embodiment, the control section 21 refers to the first learning result data 121, and performs the setting of the presumption device 5 whose machine learning is completed. The control section 21 inputs the generated target difference data 47 to the input layer 51 of the presumption device 5 whose machine learning is completed, and executes the operation processing of the presumption device 5 whose machine learning is completed. Thereby, the control section 21 can acquire the output value corresponding to the result of presuming the feature of the target included in the target difference data 47, from the output layer 53 of the presumption device 5 whose machine learning is completed. That is, in the present embodiment, by assigning the target difference data 47 to the presumption device 5 whose machine learning is completed, and acquiring the operation result of the presumption device 5 whose machine learning is completed, it is possible to presume the feature included in the target image data 41. When the presumption processing of the feature of the target is completed, the control section 21 causes the processing to proceed to the next step S205.

[0220] (Step S205)

[0221] In step S205, the control section 21 functions as an output section 215, and outputs the information related to the result of presuming the feature of the target.

[0222] The output destination and the content of the output information can be appropriately determined according to the embodiment, respectively. For example, the control section 21 can also directly output the presumption result of step S204 to the output device 26. In addition, for example, the control section 21 can perform certain information processing based on the result of presuming the feature of the target. In addition, the control section 21 can also output the result of performing the information processing as the information related to the presumption result. The output of the result of performing the information processing can include outputting a certain message according to the presumption result, controlling the operation of a control target device according to the presumption result, and the like. The output destination can be, for example, the output device 26, the output device of another computer, a control target device, and the like.

[0223] When the output of the information related to the presumption result is completed, the control section 21 ends the processing procedure involved in the present action example. Note that the control section 21 can also continuously repeat the series of information processing of steps S201 to S205 within a prescribed period. The timing of the repetition can be arbitrary. Thereby, the presumption device 2 can also continuously perform the presumption task related to the feature of the target.

[0224] [Feature]

[0225] As described above, in the present embodiment, the estimator 5 performs training to estimate the feature of the object from the training data 31 (the difference data 37) excluding the information of the background data 35, like the correct answer data 32, by the above-described steps S123 and S124. As a result, the estimator 5, which has completed machine learning, can acquire the ability to perform the estimation task related to the feature of the object without using the information of the background data 35 excluded in the difference calculation. That is, when the estimation task related to the feature of the object is performed by the trained estimator 5, the information corresponding to the background data 35 can not be considered. Therefore, according to the model generation device 1 related to the present embodiment, it is possible to generate the trained estimator 5, which can perform the estimation task related to the feature of the object included in the image data, that is, the trained estimator 5 robust to the difference in the background. In addition, in the estimation device 2 related to the present embodiment, by using the trained estimator 5 generated in the above-described step S204, it is possible to robustly perform the estimation task related to the feature of the object against the difference in the background.

[0226] In addition, in the training data 31 and the object image data 41 acquired in the steps S121 and S201, for example, there is a possibility that even the same kind of background can have a difference due to a shift in the photographing position or the like. In contrast, in the present embodiment, by the processes of the steps S101 to S103, it is possible to construct the machine-learned generator 60 for generating the background data. By using this machine-learned generator 60, even in the case where a difference occurs in the same kind of background, it is possible to appropriately generate the background data 35 and the object background data 45 corresponding to the training data 31 and the object image data 41, respectively, in the steps S122 and S202. Therefore, it is possible to reduce the cost of preparing the background data 35 and the object background data 45, respectively. In addition, in the step S124, it is possible to appropriately generate the trained estimator 5 robust to the difference in the background. In the step S204, it is possible to robustly perform the estimation task related to the feature of the object against the difference in the background.

[0227] [Embodiments]

[0228] In order to verify the above-described effects, a machine-learned estimator related to the embodiments and comparative examples was constructed. In addition, the present application is not limited to the following embodiments.

[0229] First, images of MNIST (Modified National Institute of Standards and Technology) (28 x 28, gray scale, handwritten characters of 0 to 9) and five kinds of background textures were prepared. By synthesizing the prepared images of MNIST and the background textures, training data used in machine learning and test data used in evaluation were generated. Four of the five kinds of background textures had a stripe pattern, and the position of the pattern was randomly changed at the time of synthesis with the images of MNIST. The number of pieces of the generated training data was 60,000 pieces, and the number of pieces of the test data was 10,000 pieces. By associating the generated training data with correct answer data indicating the synthesized characters (0 to 9), a learning data set was prepared. In addition, by the same method as in the above embodiment, a generator whose machine learning was completed for generating background textures was produced.

[0230] In the embodiment, by the same method as in the above embodiment, a difference between the background data generated by the generator whose machine learning was completed and the training data was calculated, thereby generating difference data. Then, using the generated difference data, a discriminator whose machine learning was completed for recognizing characters was constructed. At the time of evaluation of the discriminator whose machine learning was completed, by the same method as in the above embodiment, a difference between the background data generated by the generator whose machine learning was completed and the test data was calculated, thereby generating object difference data. Then, by the discriminator whose machine learning was completed, a task of recognizing characters was performed on the object difference data. On the other hand, in the comparative example, without removing the background, the training data was directly used, and a discriminator whose machine learning was completed for recognizing characters was constructed. At the time of evaluation, by the discriminator whose machine learning was completed, a task of recognizing characters was performed on the test data. Note that in the embodiment and the comparative example, three kinds of discriminators, a linear recognition model, a 4-layer MLP (Multilayer Perceptron), and a convolutional neural network, were prepared, respectively.

[0231] [Table 1]

[0232] Example Comparative Example Linear identification model 84.7％ 75.6％ 4-layer MLP 89.3％ 75.1％ CNN 97.0％ 96.7％

[0233] Table 1 shows the correct answer rates of character recognition for the test data in the embodiment and the comparative example, respectively. As shown in Table 1, in the case where any of the models was used for the discriminator, the performance of the embodiment was superior to that of the comparative example. In particular, in the case where a simple model such as a linear recognition model and a 4-layer MLP was used, the performance of the discriminator whose machine learning was completed could be greatly improved. According to this result, it could be verified that a discriminator whose machine learning was completed, which was robust to the difference in the background, could be constructed according to the present application.

[0234] §4 Variations

[0235] The above describes embodiments of the present application in detail, but the description up to the above is merely an example of the present application in all respects. Of course, various modifications or variations can be made without departing from the scope of the present application. For example, the following modifications can be made. Note that the same reference numerals are used below for the same constituent elements as those of the above-described embodiments, and the description is appropriately omitted for the same points as those of the above-described embodiments. The following variations can be appropriately combined.

[0236] <4.1>

[0237] The inference system 100 related to the above-described embodiments can be applied to all scenes of performing an inference task related to a feature of an object on image data. For example, the inference system 100 related to the above-described embodiments can be applied to a scene of inferring a feature included in a foreground of image data. The foreground can be constituted by any component that can be extracted from the image data in distinction from a background. The feature can be any information that can be directly or indirectly derived from the foreground. The feature included in the foreground can be constituted by, for example, inferring an attribute of an object material that appears in the image data. In this case, the component related to the derivation of the attribute of the object material is an example of the foreground. The range of the foreground can be appropriately determined to be able to derive the attribute of the object material. For example, the foreground can be a range in which the object material appears. Alternatively, the foreground can be a range including the object material and the surroundings of the object material. Alternatively, the foreground can also be a range of a part of the object material that represents the attribute of the inference object. Hereinafter, variations that limit the application scene are exemplified.

[0238] (A) Scene of appearance inspection

[0239] Figure 11 An example of an application scene of the inspection system 100A related to the first variation is exemplified schematically. This variation is an example of applying the above-described embodiments to a scene of performing appearance inspection of a product using image data in which the product appears. The inspection system 100A related to this variation is provided with the model generation device 1 and the inspection device 2A. As with the above-described embodiments, the model generation device 1 and the inspection device 2A can be connected to each other via a network.

[0240] In the present modification, the image data P10 that reflects the product RA is processed as training data and as object image data. The image data P10 can be obtained by capturing the product RA with the camera SA. The feature that becomes the object of the inference task can be a feature related to the defect P12 of the product RA. The background P11 can be a portion other than the product RA, such as a sheet of the conveyer. The background P11 can also include, for example, a portion that reflects the product RA other than the defect P12, such as a portion that reflects the outer packaging of the product RA that does not have an abnormality. Other than these limitations, the inspection system 100A related to the present modification can be configured similarly to the inference system 100 related to the above-described embodiment.

[0241] Note that the product RA can be, for example, a product conveyed on a manufacturing line, such as an electronic device, an electronic component, an automobile component, a pharmaceutical product, or a food product. The electronic component can be, for example, a substrate, a chip capacitor, a liquid crystal, or a winding of a relay. The automobile component can be, for example, a connecting rod, a shaft, an engine block, a power window switch, or a panel. The pharmaceutical product can be, for example, a packaged tablet, an unpackaged tablet, or the like. The product RA can be a final product generated after a manufacturing process is completed, an intermediate product generated in the middle of a manufacturing process, or an initial product prepared before a manufacturing process. The defect P12 can be, for example, a scratch, dirt, a crack, a dent, a burr, uneven color, or a foreign object. The inference related to the defect P12 can be expressed, for example, by determining whether the product RA includes the defect P12, determining a probability that the product RA includes the defect P12, identifying a type of the defect P12 included in the product RA, determining a range of the defect P12 included in the product RA, or a combination thereof.

[0242] (Model generation device)

[0243] In the present modification, the model generation device 1 acquires the learning background data 39 that reflects the existing background P11 in step S101 described above. Then, the model generation device 1 generates the generator 60 that has completed machine learning that has acquired the ability to generate the simulated background data that reflects the existing background P11 by performing the processes of steps S102 and S103, and saves the second learning result data 122 in a prescribed storage area. The model generation device 1 generates the input inferrer 62 that has completed machine learning by performing the processes of steps S111 to S113 described above, and saves the third learning result data 123 in a prescribed storage area.

[0244] Further, the model generation device 1 acquires a plurality of learning data sets 30 each composed of a combination of the training data 31 representing the product RA and the correct answer data 32 representing information (correct answer) related to the defect P12 of the product RA represented in the training data 31 in the above step S121. The information related to the defect P12 can include, for example, information on the presence or absence of the defect P12, the category of the defect P12, the range in which the defect P12 exists, and the like. The model generation device 1 generates the completed estimator 5 that has acquired the ability to perform estimation related to the defect P12 by performing the processes of the above steps S122 to S125, and stores the first learning result data 121 in a prescribed storage area. The generated first to third learning result data 121 to 123 can be provided to the inspection device 2A at any timing.

[0245] (inspection device)

[0246] Figure 12A An example of a hardware structure of the inspection device 2A related to the present modification example is schematically illustrated. As shown in Figure 12A the inspection device 2A related to the present modification example can be composed of a computer electrically connected with a control section 21, a storage section 22, a communication interface 23, an external interface 24, an input device 25, an output device 26, and a driver 27, like the above-described estimation device 2.

[0247] In the present modification example, the storage section 22 stores various information such as an inspection program 82A, the first to third learning result data 121 to 123. The inspection program 82A corresponds to the above-described estimation program 82. At least any one of the inspection program 82A and the first to third learning result data 121 to 123 can also be stored in the storage medium 92. Further, in the present modification example, the inspection device 2A is connected with a camera SA via the external interface 24. The camera SA can be, for example, a general RGB camera, a depth camera, an infrared camera, or the like. The camera SA can be appropriately disposed at a place where the product RA can be photographed. For example, the camera SA can be disposed in the vicinity of a conveyance device that conveys the product RA.

[0248] Note that the hardware structure of the inspection device 2A can not be limited to such an example. As for the specific hardware structure of the inspection device 2A, omission, substitution, and addition of constituent elements can be appropriately made in accordance with the embodiments. The inspection device 2A can be a general server device, a general PC, a PLC, or the like, in addition to being designed as an information processing device dedicated to the service provided.

[0249] Figure 12BAn example of a software structure of the inspection device 2A related to this modification example is schematically illustrated. As with the above-described embodiment, the software structure of the inspection device 2A is realized by the control section 21 executing the inspection program 82A. The software structure of the inspection device 2A is the same as the above-described estimation device 2 except for the point that the data to be processed is limited as described above. The inspection section 214A corresponds to the estimation section 214. Thus, the inspection device 2A can perform a series of information processing related to appearance inspection as with the estimation processing of the above-described estimation device 2.

[0250] That is, in step S201, the control section 21 acquires the object image data 41A that reflects the product RA that is the inspection target. In step S202, the control section 21 acquires the object background data 45A that corresponds to the object image data 41A. In this modification example, the control section 21 can acquire the object background data 45A that is suitable for the background P11 reflected in the object image data 41A by using the generator 60 and the input estimator 62 for which machine learning is completed. In step S203, the control section 21 generates the object difference data 47A by taking a difference between the object image data 41A and the object background data 45A. In step S204, the control section 21 functions as the inspection section 214A, and gives the generated object difference data 47A to the estimator 5 for which machine learning is completed, and executes the operation processing of the estimator 5 for which machine learning is completed. Thus, the control section 21 acquires the output value that corresponds to the result of the estimation of the defect P12 with respect to the product RA reflected in the object image data 41A from the estimator 5 for which machine learning is completed.

[0251] In step S205, the control section 21 outputs information related to the result of the estimation of the defect P12 with respect to the product RA. For example, the control section 21 can also output the result of the estimation of the defect P12 with respect to the product RA directly to the output device 26. In addition, for example, in a case where it is determined that the product RA includes the defect P12, the control section 21 can also output a warning for notifying this situation to the output device 26. In addition, for example, in a case where the inspection device 2A is connected to a conveyance device that conveys the product RA, the control section 21 can also control the conveyance device to convey the product RA having the defect P12 and the product RA having no defect P12 on different production lines based on the result of the estimation of the defect P12.

[0252] (Features)

[0253] According to the present modification example, the estimator 5 can be trained to perform the estimation task related to the defect P12 without using the information related to the background P11 that is excluded in the difference calculation, by the machine learning of the model generation device 1. Thereby, the estimator 5 that has completed the machine learning for which the difference with respect to the background P11 can robustly perform the estimation task related to the defect P12 can be generated. In the inspection device 2A related to the present modification example, by using the estimator 5 that has completed the machine learning, it is expected that the estimation task related to the defect P12 (i.e., the appearance inspection) can be performed with high accuracy.

[0254] (Other)

[0255] In the above-described first modification example, the product RA is an example of the object, and the information related to the defect P12 is an example of the attribute of the object. The object and the attribute thereof are not limited to the information related to the product RA and the defect P12, and can be appropriately selected depending on the embodiment. That is, the estimation system 100 related to the above-described embodiment can be applied to various scenes of estimating the attribute of an arbitrary object represented by the image data. In this case, in the estimation system 100, the image data in which the object is represented can be processed as the training data 31 and the object image data 41. The feature of the object can be related to the attribute of the object. Thereby, the estimator 5 that has completed the machine learning for which the difference with respect to the background can robustly perform the estimation task related to the attribute of the object can be generated.

[0256] In addition, in a case where a portion other than the product RA is the background P11, the portion in which the product RA is represented is an example of the foreground. Alternatively, in a case where a portion other than the defect P12 of the product RA is the background P11, the range in which the defect P12 is represented is an example of the foreground. The foreground and the background are not limited to such examples, and can be appropriately selected depending on the embodiment. That is, the estimation system 100 related to the above-described embodiment can be applied to various scenes of performing the estimation related to the feature included in an arbitrary foreground represented in the image data. In this case, the kind of data can be the image data including the foreground and the background. In this case, the image data in which the foreground and the background are represented can be processed as the training data 31 and the object image data 41. The feature of the object can be the feature related to the foreground. For the foreground, an arbitrary element that can be extracted from the image data can be selected. Thereby, the estimator 5 that has completed the machine learning for which the difference with respect to the background can robustly perform the estimation task related to the foreground can be generated.

[0257] (B) Scene of recognizing symbols of a license plate

[0258] Figure 13 An example of another scene of estimating the attribute of the object represented in the image data is schematically illustrated. Figure 13The illustrated recognition system has the model generation device 1 and a recognition device 2B. The recognition device 2B corresponds to the above-described inspection device 2A. The structure of the recognition device 2B can be the same as that of the above-described inspection device 2A. In Figure 13 In the example, the image data in which the existing license plate RB is reflected can be processed as the training data and the object image data. In the acquisition of the image data, similarly to the above-described inspection system 100A, the camera SB can be used.

[0259] In this case, the feature that becomes the object of the inference task can be a feature related to the symbol P20 imparted to the license plate RB. The license plate RB is an example of the object. The background can be a region other than the symbol P20. The foreground can be a region in which the existing symbol P20 is reflected. The symbol P20 can be composed of, for example, a number, a character, a mark, a figure, or a combination thereof. A decoration such as a pattern can also be applied to the background of the license plate RB. The inference related to the feature of the object can be the recognition of the symbol P20.

[0260] Thus, the model generation device 1 can generate an inferrer 5 in which machine learning is completed, the inferrer 5 being able to robustly recognize the symbol P20 imparted to the license plate RB against the difference in the background. The recognition device 2B can recognize the symbol P20 of the license plate RB reflected in the object image data by the same sequence as that of the above-described inspection device 2A. The recognition system related to the present modification example can be used for all scenes in which the symbol P20 of the license plate RB is recognized. For example, the recognition system related to the present modification example can be used for recognizing a vehicle traveling on a road from image data obtained by a monitoring camera for monitoring the vehicle traveling on the road.

[0261] Note that, in the scene of monitoring the vehicle traveling on the road, the recognition device 2B can also recognize the symbol P20 of the license plate RB from the object image data obtained in time series by the processing of the above-described steps S201 to S204. Then, the recognition device 2B can also generate history information representing the result of recognizing the vehicle traveling on the object road based on the symbol P20 in time series by the processing of the above-described step S205.

[0262] (C) Scene of Recognizing Symbol Imparted to Packaging of Product

[0263] Figure 14 An example of another scene in which the attribute of the object reflected in the inference image data is schematically illustrated. Figure 14 The illustrated recognition system has the model generation device 1 and a recognition device 2C. The recognition device 2C corresponds to the above-described inspection device 2A. The structure of the recognition device 2C can be the same as that of the above-described inspection device 2A. In Figure 14In the example of the product, image data of a package RC in which the product is reflected can be processed as the training data and the object image data. In the acquisition of the image data, similarly to the inspection system 100A, a camera SC can be used.

[0264] In this case, the feature that becomes the object of the estimation task can be a feature related to a symbol P30 imparted to the package RC. The package RC is an example of the object. The background can be a region other than the symbol P30. The foreground can be a region in which the existing symbol P30 is reflected. The symbol P30 can be constituted by, for example, a number, a character, a mark, a figure, a code, or a combination thereof. The code can be, for example, a one-dimensional code, a two-dimensional code, or the like. A decoration such as a pattern or the like can also be applied to the background of the package RC. The estimation related to the feature of the object can be recognition of the symbol P30.

[0265] Thus, the model generation device 1 can generate the estimator 5 in which the difference in the background can robustly recognize the symbol P30 imparted to the package RC, which is completed by machine learning. The recognition device 2C can recognize the symbol P30 imparted to the package RC reflected in the object image data by the same sequence as the inspection device 2A. The recognition system related to the present modification example can be used for all scenes in which the symbol P30 imparted to the package RC is recognized. For example, the recognition system related to the present modification example can be used for recognizing a product conveyed on a manufacturing production line.

[0266] (D) Scene of Estimating Property of Moving Body

[0267] Figure 15 An example of another scene of estimating the property of the object reflected in the image data is schematically illustrated. Figure 15 The illustrated monitoring system is provided with the model generation device 1 and a monitoring device 2D. The monitoring device 2D corresponds to the inspection device 2A. The structure of the monitoring device 2D can be the same as that of the inspection device 2A. In the monitoring device 2D, the training data and the object image data can be processed in the same manner as in the inspection device 2A. Figure 15 In the example of the product, image data of a package RC in which the product is reflected can be processed as the training data and the object image data. In the acquisition of the image data, similarly to the inspection system 100A, a camera SC can be used.

[0268] In this case, the feature that becomes the object of the estimation task can be a feature related to the attribute of the mobile body RD. The mobile body RD is an example of the object. The attribute of the mobile body RD can include, for example, a range (bounding box) that represents the existing mobile body RD, identification information of the mobile body RD, a state of the mobile body RD, and the like. The state of the mobile body RD can include, for example, a position, a number, a congestion state (degree of congestion, density), a speed, an acceleration, and the like. Further, the state of the mobile body RD can include, for example, a possibility (degree of danger) that the mobile body RD encounters a danger such as an accident or a crime. The kind of danger that the mobile body RD can encounter can be arbitrarily determined. Further, in the case where the mobile body RD is a person, the identification information of the mobile body RD can include, for example, an identifier, a name, a gender, an age, a physical condition, and the like of the person. In addition, the state of the mobile body RD can include, for example, a movement of the person, a posture of the person, and the like. The background can be, for example, a region that represents an object other than the mobile body RD such as an existing road, a building, and the like. The foreground can be a region that represents the existing mobile body RD. Alternatively, the foreground can be a region that includes the mobile body RD and the surroundings of the mobile body RD. In this case, the background can be a region outside the surroundings of the mobile body RD.

[0269] Thus, the model generation device 1 can generate the estimator 5 that is completed with machine learning, which robustly estimates the attribute of the mobile body RD with respect to the difference in the background. The monitoring device 2D can estimate the attribute of the mobile body RD in the photographing range of the camera SD by the same sequence as the above-described checking device 2A. The monitoring system related to the present modification example can be used for all scenes of monitoring the mobile body RD. For example, the monitoring system related to the present modification example can be used for a scene of monitoring the mobile body RD on a road, a scene of monitoring the degree of congestion of the mobile body RD in a building, a scene of monitoring a precursor of a person falling from a platform of a station, a scene of monitoring a precursor of a person walking on a road encountering a crime, and the like. Note that the manner of outputting the estimation result can be arbitrarily determined. For example, in the scene of estimating the degree of danger of the mobile body RD, the monitoring device 2D can output a warning corresponding to the estimated degree of danger in the above-described step S205.

[0270] (E) Scene of monitoring a state of a driver

[0271] Figure 16 An example of another scene of estimating an attribute of an object represented in image data is schematically illustrated. Figure 16 The illustrated monitoring system includes the model generation device 1 and a monitoring device 2E. The monitoring device 2E corresponds to the above-described checking device 2A. The structure of the monitoring device 2E can be the same as that of the above-described checking device 2A. In the case where the model generation device 1 is used for the monitoring device 2E, the model generation device 1 can generate the estimator 5 that is completed with machine learning, which robustly estimates the attribute of the mobile body RD with respect to the difference in the background. The monitoring device 2E can estimate the attribute of the mobile body RD in the photographing range of the camera SD by the same sequence as the above-described checking device 2A. Figure 16In the example of the present embodiment, the image data in which the existing driver RE is imaged can be processed as the training data as well as the target image data. In the acquisition of the image data, similarly to the above-described inspection system 100A, the camera SE can be used. The camera SE can be appropriately arranged to photograph the driver RE assumed to exist at the driver's seat.

[0272] In this case, the feature that becomes the target of the estimation task can be a feature related to the state of the driver RE. The driver RE is an example of the target object. The state of the driver RE can include, for example, a health state, a drowsiness degree, a fatigue degree, a surplus degree, and the like. The health state can be expressed, for example, by a probability of occurrence of a prescribed disease, a probability of occurrence of a change in physical condition, and the like. The drowsiness degree indicates the degree of drowsiness of the driver RE. The fatigue degree indicates the degree of fatigue of the driver RE. The surplus degree indicates the degree of surplus of the driver RE for driving. The background can be, for example, an area other than an area related to the state of the driver RE, such as an in-vehicle scenery. The foreground can be an area related to the state of the driver RE. The area related to the state of the driver RE can be, for example, an area in which the existing driver RE is imaged, an area including the surroundings of the driver RE, an area in which a part (for example, an arbitrary body part) of the existing driver RE is imaged, an area in which a tool (steering wheel, and the like) related to driving of the existing driver RE is imaged, and the like.

[0273] Thus, the model generation device 1 can generate the completed machine learning estimator 5 that robustly estimates the state of the driver RE with respect to the difference in the background. The monitoring device 2E can estimate the state of the driver RE imaged in the target image data by the same sequence as the above-described inspection device 2A. The monitoring system related to the present modification example can be utilized in all scenes of monitoring the state of the driver RE. Further, the monitoring system related to the present modification example can also be utilized for a target other than the driver RE. As an example of the target other than the driver RE, the target can be, for example, an operator who performs work in an office, a factory, or the like.

[0274] Note that, in the above-described step S205, the manner of outputting the result of estimating the state of the driver RE can be arbitrarily determined. As an example, in a case where at least one of the drowsiness degree and the fatigue degree is estimated, the monitoring device 2E can also determine whether the estimated at least one of the drowsiness degree and the fatigue degree exceeds a threshold value. In addition, in a case where at least one of the drowsiness degree and the fatigue degree exceeds the threshold value, the monitoring device 2E can also output a warning to the output device, urging the driver RE to stop the vehicle at a parking lot or the like and take a break. In this way, the monitoring device 2E can also output a specific message such as a warning to the output device in accordance with the estimated state of the driver RE.

[0275] As other examples, in a case where the vehicle is provided with a control device (not shown) that controls the behavior of automatic driving, the monitoring device 2E can also transmit an instruction for instructing the behavior of automatic driving of the vehicle to the control device on the basis of the result of estimating the state of the driver RE. As a specific example, assume that the control device is configured to be able to switch between an automatic driving mode that controls the running of the vehicle and a manual driving mode that controls the running of the vehicle by the steering of the driver RE. In this case, when the vehicle is running in the automatic driving mode, the monitoring device 2E can also determine whether the estimated surplus of the driver RE exceeds a threshold value when a switch from the automatic driving mode to the manual driving mode is received from the driver RE or the system. In addition, in a case where the surplus of the driver RE exceeds the threshold value, the monitoring device 2E can also transmit an instruction to the control device that allows the switch from the automatic driving mode to the manual driving mode. On the other hand, in a case where the surplus of the driver RE is equal to or less than the threshold value, the monitoring device 2E can also transmit a notification to the control device that does not allow the switch from the automatic driving mode to the manual driving mode, and maintain the running in the automatic driving mode.

[0276] In addition, when the vehicle is running in the manual driving mode, the monitoring device 2E can also determine whether at least one of the estimated drowsiness and the estimated fatigue exceeds a threshold value. In addition, in a case where at least one of the drowsiness and the fatigue exceeds the threshold value, the monitoring device 2E can also transmit an instruction to the control device that instructs a switch from the manual driving mode to the automatic driving mode and parking at a safe place such as a parking lot. On the other hand, in a case where this is not so, the monitoring device 2E can also maintain the running of the vehicle on the basis of the manual driving mode.

[0277] In addition, when the vehicle is running in the manual driving mode, the monitoring device 2E can also determine whether the estimated surplus is equal to or less than a threshold value. In addition, in a case where the surplus is equal to or less than the threshold value, the monitoring device 2E can also transmit an instruction to the control device to decelerate. On the other hand, in a case where this is not so, the monitoring device 2E can also maintain the running of the vehicle by the operation of the driver RE. Note that the control device and the monitoring device 2E of the vehicle can also be constituted by an integrated computer.

[0278] <4.2>

[0279] In the above-described embodiments, in the step S122 and the step S202, the background data 35 and the object background data 45 can be acquired using the generator 60 for which the machine learning is completed. However, the method of acquiring each background data (35, 45) is not limited to such an example. For example, background data that can be used as each background data (35, 45) can be prepared in advance. In this case, the background data can be stored in the storage section (12, 22), the storage medium (91, 92), an external storage device, or the like. In the above-described step S122 and step S202, the background data that represents the background of the same kind as the training data 31 and the object image data 41, respectively, can be appropriately selected from the background data prepared in advance as each background data (35, 45). As an example, the background data to be used can be selected by an operator. Alternatively, the background data that is most similar to the training data 31 and the object image data 41, respectively, can be selected as each background data (35, 45).

[0280] <4.3>

[0281] In the above-described embodiments, the discriminator 61 is used in the training of the generator 60. The model generation device 1 constructs the generator 60 for which the machine learning is completed that has the ability to generate the simulated background data 633 suitable for the learning background data 39 by implementing the adversarial learning between the generator 60 and the discriminator 61. However, the method of generating the generator 60 for which the machine learning is completed is not limited to such an example as long as the generator 60 can be trained to generate the simulated background data that represents the background given by the learning background data 39.

[0282] Figure 17 An example of the structure of the generator 60Z related to the present modification example is schematically illustrated. In the present modification example, the generator 60Z is trained together with the encoder 67. The encoder 67 is configured to convert the image data into a feature quantity. The generator 60Z acts as a decoder configured to restore the image data from the feature quantity. The feature quantity obtained by the encoder 67 corresponds to the noise (latent variable) in the above-described embodiments. The encoder 67 corresponds to the above-described input estimator 62. The form of the feature quantity is not particularly limited and can be arbitrarily determined.

[0283] The encoder 67 and the generator 60Z are each configured by a machine learning model. The kind of the machine learning model that configures each encoder and generator can be arbitrarily selected. As in the above-described embodiments, the encoder 67 and the generator 60Z can use a neural network. In the case where the neural network is used for the encoder 67 and the generator 60Z, in the above-described step S102, the model generation device 1 can implement the machine learning as follows.

[0284] That is, the control section 11 inputs the learning background data 39 to the encoder 67, and executes the operation processing of the encoder 67. Thereby, the control section 11 acquires the output value corresponding to the result of converting the learning background data 39 into the feature quantity from the encoder 67. Next, the control section 11 inputs the feature quantity obtained from the encoder 67 to the generator 60Z, and executes the operation processing of the generator 60Z. Thereby, the control section 11 acquires the output value corresponding to the restored background data 633Z restored from the feature quantity from the generator 60Z.

[0285] The control section 11 calculates the error of the restored background data 633Z obtained and the learning background data 39. The control section 11 calculates the error of the value of each operation parameter of the encoder 67 and the generator 60Z using the gradient of the calculated error by the error backpropagation method. The control section 11 updates the value of each operation parameter of the encoder 67 and the generator 60Z on the basis of the calculated error. The control section 11 adjusts the value of each operation parameter of the encoder 67 and the generator 60Z by this series of update processing so that the sum of the calculated errors becomes smaller. Note that, in addition to the above update processing, the control section 11 can calculate the error of the feature quantity obtained from the encoder 67 and the value obtained from a prescribed probability distribution (for example, a Gaussian distribution or the like), and further adjust the value of each operation parameter of the encoder 67 so that the sum of the error becomes smaller. Thereby, the control section 11 can also normalize the output value of the encoder 67.

[0286] By the above machine learning, it is possible to construct the trained encoder 67 that has acquired the ability to convert the image data into an appropriate feature quantity according to the background. At the same time, it is possible to construct the trained generator 60Z that has acquired the ability to generate the simulated background data that reflects the corresponding background according to the feature quantity. In this case, it is possible to use the machine-learned encoder 67 as the above machine-learned input estimator 62. Therefore, in the present modification, it is possible to omit the processing of the above steps S111 to S113.

[0287] In the above step S122, the control section 11 imparts the training data 31 to the machine-learned encoder 67, and executes the operation processing of the machine-learned encoder 67 and the generator 60Z. Thereby, the control section 11 can acquire the background data 35 generated in correspondence with the training data 31 from the machine-learned generator 60Z. Similarly, in the above step S202, the control section 21 imparts the object image data 41 to the machine-learned encoder 67, and executes the operation processing of the machine-learned encoder 67 and the generator 60Z. Thereby, the control section 21 can acquire the object background data 45 generated in correspondence with the object image data 41 from the machine-learned generator 60Z.

[0288] As another example, the machine learning of the generator 60 can be constituted by principal component analysis. In this case, the machine-learned generator 60 can be constituted by intrinsic vectors derived by the principal component analysis. In detail, the machine-learned generator 60 can be constituted by a projection matrix "UU T " that utilizes the intrinsic vectors U. Each component of the intrinsic vectors is an example of the operation parameter of the generator 60. In the above step S122 and step S202, by converting the training data 31 and the target image data 41 respectively using the above projection matrix, the corresponding background data 35 and the target background data 45 can be obtained.

[0289] <4.4>

[0290] In the above embodiment, the estimator 5, the generator 60, the recognizer 61, and the input estimator 62 each use a neural network of a fully connected type. However, the type of the neural network that constitutes them is not limited to this example. In the estimator 5, the generator 60, the recognizer 61, and the input estimator 62, for example, a convolutional neural network, a recurrent neural network, or the like can be used respectively.

[0291] In addition, the type of the machine learning model that constitutes each of the estimator 5, the generator 60, the recognizer 61, and the input estimator 62 can also not be limited to a neural network. For example, as described above, the generator 60 can be constituted by a projection matrix that utilizes the intrinsic vectors U. In addition, for each of the estimator 5, the recognizer 61, and the input estimator 62, in addition to a neural network, for example, a support vector machine, a regression model, a decision tree model, or the like can be used. The method of machine learning can be appropriately selected according to the type of each machine learning model.

[0292] In addition, in the above embodiment, the form of the input and the output of each of the estimator 5, the generator 60, the recognizer 61, and the input estimator 62 can not be particularly limited and can be appropriately determined according to the embodiment. For example, the generator 60 and the recognizer 61 can each be constituted to further accept input of information indicating the category of the background. Thereby, it is possible to control the category of the background that is represented in the simulated background data generated by the machine-learned generator 60 according to the input value (information indicating the category of the background) that is given to the machine-learned generator 60.

[0293] In addition, in the above-described embodiments, the generator 60 and the input estimator 62 that have completed machine learning can also be generated by a computer other than the model generation apparatus 1. In a case where the generator 60 that has completed machine learning is not generated by the model generation apparatus 1, the second data acquisition section 115 and the second training section 116 can be omitted from the software structure of the model generation apparatus 1. The processing of steps S101 to S103 described above can be omitted from the processing sequence of the model generation apparatus 1. In a case where the input estimator 62 that has completed machine learning is not generated by the model generation apparatus 1, the third data acquisition section 117 and the third training section 118 can be omitted from the software structure of the model generation apparatus 1. The processing of steps S111 to S113 described above can be omitted from the processing sequence of the model generation apparatus 1. In a case where information related to the machine learning model that has completed machine learning is not saved, the saving processing section 119 can be omitted from the software structure of the model generation apparatus 1. The processing of steps S103, S113, and S205 described above can be omitted from the processing sequence of the model generation apparatus 1.

[0294] Explanation of Reference Numerals

[0295] 1 … model generation apparatus, 11 … control section, 12 … storage section, 13 … communication interface, 14 … external interface, 15 … input device, 16 … output device, 17 … driver, 111 … first data acquisition section, 112 … context acquisition section, 113 … difference calculation section, 114 … first training section, 115 … second data acquisition section, 116 … second training section, 117 … third data acquisition section, 118 … third training section, 119 … saving processing section, 121 … first learning result data, 122 … second learning result data, 123 … third learning result data, 81 … model generation program, 91 … storage medium, 2 … estimation apparatus, 21 … control section, 22 … storage section, 23 … communication interface, 24 … external interface, 25 … input device, 26 … output device, 27 … driver, 211 … data acquisition section, 212 … context acquisition section, 213 … difference calculation section, 214 … estimation section, 215 … output section, 82 … estimation program, 92 … storage medium, 30 … learning data set, 31 … training data, 32 … correct answer data, 35 … context data, 37 … difference data, 39 … learning context data, 41 … target image data, 45 … target context data, 47 … target difference data, 5 … estimator, 60 … generator, 61 … recognizer, 62 … input estimator.

Claims

1. A model generation device comprising: a first data acquisition section that acquires a plurality of learning data sets each composed of a combination of training data that is image data and correct answer data that represents a feature included in the training data; a context acquisition section that acquires, for each of the learning data sets, context data that represents a context of the training data, the acquisition of the context data including: generating, for each of the learning data sets, the context data for the training data using a generator that has completed machine learning and an input estimator configured to accept input of the image data, estimate an input value to be given to the generator that has completed machine learning in order to generate the context data that is appropriate for the context that is represented in the input image data, and output an output value that corresponds to the estimated value; a difference calculation section that, for each of the learning data sets, obtains a difference between the acquired context data and the training data, thereby generating difference data that represents the difference between the context data and the training data; and a first training section that performs machine learning of the input estimator, the performance of the machine learning of the input estimator including, for each of the learning data sets, training the input estimator so that a result of estimating the feature from the generated difference data by the input estimator is appropriate for the correct answer data, the training data being the image data that represents an object, the feature being an attribute of the object, wherein (a) the object is a product, the attribute of the object being related to a defect of the product; or (b) the object is a license plate, the attribute of the object being related to a symbol given to the license plate; or (c) the object is a package of a product, the attribute of the object being related to a symbol given to the package; or (d) the object is a range of a monitored moving body, the attribute of the object being related to an attribute of the moving body; or (e) the object is a driver, the attribute of the object being related to a state of the driver. 2.The model generation device according to claim 1, wherein the model generation device further comprises: a second data acquisition section that acquires learning context data; and a second training section that constructs the generator that has completed machine learning that generates the context data for the training data by performing machine learning using the acquired learning context data. 3.The model generation device according to claim 1 or 2, wherein the difference calculation section obtains a difference between each pixel of the training data and a corresponding pixel of the context data based on a correlation between a subject region including each pixel of the training data and pixels around each pixel and a corresponding region including the corresponding pixel of the context data and pixels around the corresponding pixel, thereby generating the difference data. 4.The model generation device according to claim 1 or 2, wherein the feature is related to a foreground of the training data. 5.An estimation device comprising: a data acquisition section that acquires object image data; a context acquisition section that acquires object context data that corresponds to the object image data; ​ a difference calculation section that generates object difference data by taking a difference between the object image data and the object background data; an estimation section that estimates a feature included in the generated object difference data using an estimator generated by the model generation device according to any one of claims 1 to 4; and an output section that outputs information related to a result of estimating the feature.

6. A model generation method, comprising the steps of: acquiring a plurality of learning data sets each composed of a combination of training data that is image data and correct answer data that represents a feature included in the training data; for each of the learning data sets, acquiring background data that represents a background of the training data, the acquisition of the background data including: for each of the learning data sets, generating the background data with respect to the training data using a generator that has completed machine learning and an input estimator configured to accept an input of the image data, estimate an input value to be given to the generator that has completed machine learning in order to generate the background data suitable for the background represented in the input image data, and output an output value corresponding to the estimated value obtained; for each of the learning data sets, taking a difference between the acquired background data and the training data, thereby generating difference data that represents a difference between the background data and the training data; and performing machine learning of an estimator, the machine learning of the estimator including, for each of the learning data sets, training the estimator so that a result of estimating the feature from the generated difference data by the estimator is suitable for the correct answer data, the training data is the image data representing an object, and the feature is an attribute of the object, wherein (a) the object is a product, and the attribute of the object is related to a defect of the product; or (b) the object is a license plate, and the attribute of the object is related to a symbol assigned to the license plate; or (c) the object is a package of a product, and the attribute of the object is related to a symbol assigned to the package; or (d) the object is a range of a monitored moving body, and the attribute of the object is related to an attribute of the moving body; or (e) the object is a driver, and the attribute of the object is related to a state of the driver.

7. A program product including a model generation program for causing a computer to execute the steps of: acquiring a plurality of learning data sets each composed of a combination of training data that is image data and correct answer data that represents a feature included in the training data; for each of the learning data sets, acquiring background data that represents a background of the training data, the acquisition of the background data including: for each of the learning data sets, generating the background data with respect to the training data using a generator that has completed machine learning and an input estimator configured to accept an input of the image data, estimate an input value to be given to the generator that has completed machine learning in order to generate the background data suitable for the background represented in the input image data, and output an output value corresponding to the estimated value obtained; The generator completed with machine learning and an input estimator configured to accept input of the image data, estimate an input value to be given to the generator completed with machine learning in order to generate the background data suitable for the background represented in the input image data, and output an output value corresponding to the estimated value are used to generate the background data for the training data with respect to each of the learning data sets. A step of obtaining a difference between the background data and the training data with respect to each of the learning data sets, thereby generating difference data representing the difference between the background data and the training data; and A step of performing machine learning of the estimator, the machine learning of the estimator including training the estimator so that a result of estimating the feature by the estimator from the generated difference data is suitable for the correct answer data with respect to each of the learning data sets, the training data being the image data representing an object, the feature being an attribute of the object, wherein (a) the object is a product, the attribute of the object being related to a defect of the product; or (b) the object is a license plate, the attribute of the object being related to a symbol given to the license plate; or (c) the object is a package of a product, the attribute of the object being related to a symbol given to the package; or (d) the object is a range of a moving body, the attribute of the object being related to an attribute of the moving body; or (e) the object is a driver, the attribute of the object being related to a state of the driver.

Images